Skip to content

The only tool you need to know what is happening and how to fix it.

feat: full otel compatibility (#372)

narrow-adventureebed5cb3fa44
diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml
index f4826db6..560ee552 100644
--- a/.github/workflows/backend.yml
+++ b/.github/workflows/backend.yml
@@ -294,6 +294,8 @@ jobs:
# it uses fakes and needs no live ClickHouse. app/migrations is included
# for the same reason: migrate_logger_test.go is tagged
# `transactional_pg || telemetry_ch` and needs no database either.
+ # app/notifications includes the ClickHouse-only task failure identity
+ # regression, which neither the SQLite nor DuckDB job can run.
#
# -json is written to a file for the tally below and replayed through jq
# so the log still reads like ordinary `go test` output. pipefail is set
@@ -312,7 +314,7 @@ jobs:
run: |
set -o pipefail
go test -json -tags "transactional_pg telemetry_ch" -count=1 \
- ./app/repositories/... ./app/chdb/... ./app/migrations/... \
+ ./app/repositories/... ./app/chdb/... ./app/migrations/... ./app/notifications/... \
| tee "$RUNNER_TEMP/pgch-tests.jsonl" \
| jq -j 'select(.Action == "output") | .Output'
diff --git a/CLAUDE.md b/CLAUDE.md
index 7d2ae351..32ad3cd7 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -312,6 +312,10 @@ SESSION_RECORDING_RETENTION_DAYS=30 # 0 to disable; only applies when STORAGE_
PROFILE_ARCHIVE_RAW=false # native pprof ingest only: write the original pprof bytes to object storage as a lossless archive
PROFILE_RETENTION_DAYS=30 # 0 to disable; on-disk archive TTL, only with PROFILE_ARCHIVE_RAW + STORAGE_TYPE=local
+# V2 trace tables move-over (see "Trace Storage (V2 tables)" below)
+V2_MOVE_OVER= # "true" copies the history of the tables V2 replaced into the V2 tables, in the background, newest day first. Runs at startup and one hour after each pass; resumable (progress in the transactional DB v2_move_over_days, sqlite/0077 + pg/0143), so it can stay set. Finish upgrading old writers before enabling it: completed source-table/date pairs are skipped. With several backend instances set it on exactly one. On SQLite and DuckDB it stops at the telemetry retention window, since older days would be pruned at once.
+V2_MOVE_OVER_OLDEST= # optional YYYY-MM-DD; days before it stay in the old tables. An unparsable value logs and does not start the move.
+
# Session recording uploads (see "Session Recording Uploader" section below)
SESSION_RECORDING_UPLOAD_WORKERS=32 # 0 to disable uploads entirely
SESSION_RECORDING_UPLOAD_QUEUE_SIZE=2048
@@ -530,6 +534,8 @@ const handleClick = createRowClickHandler('/issues/abc123', 'preset', 'from', 't
/endpoints/[endpoint] Single endpoint details
/tasks Background tasks list
/tasks/[task] Single task details
+/spans Span explorer: every OTel span of the project, filtered by name, service, kind, status, duration, trace ID and attribute values (time range defaults to 60m)
+/spans/[traceId] Whole trace waterfall across services and across every project of the organization (?at= is required, ?span= highlights one span), with the current project's correlated logs on backend projects
/dashboards Dashboards page (tabs of org dashboards; /metrics redirects here)
/monitors Monitors (synthetic checks): single sidebar item, TabsRow tabs Monitors | Status Pages | Post-Mortems (?tab=, Status Pages admin-only, Post-Mortems for all members); status pages tab has branding (description, logo upload, custom domain) plus a link to the per-page incidents page (which owns the Record Incident dialog); old /monitors/status-pages redirects to the tab, /monitors/runners to /monitors (runners have no UI — operator infra), bare /monitors/post-mortems to the tab
/monitors/status-pages/[pageId]/incidents Paginated incident management for one status page (timeline updates, titles, manual resolve/delete, post-mortem links); reads for members, mutations admin-gated in the UI
@@ -741,7 +747,7 @@ Templates are DB rows seeded by migrations (`traceway-otel-agent` for the OTel h
**AI Traces & Conversations**
-One `ai_traces` row per LLM call (any OTLP span with `gen_ai.*` attributes). Each row carries a `conversation_id` resolved at ingest (`gen_ai.conversation.id` -> `session.id` span/resource attr -> distributed trace id -> empty), `tool_call_count`/`tool_names` parsed from the completion payload (OpenAI `tool_calls`, Anthropic `tool_use`, OTel output messages, `gen_ai.tool.*` fallback), and `flagged`/`flagged_terms` from an ingest-time word-boundary scan of prompt+completion against per-project selected built-in language packs (`projects.ai_flagged_languages`, default `["en"]`; packs live in `backend/app/services/contentflag/terms/*.txt` — en/de/es/fr/it/pt/sr; empty array = custom terms only) plus per-project custom terms (`projects.ai_flagged_terms`); both edited in the project settings AI tab and cached via the project cache. All three project-creation paths (`Create`, `CreateWithOrganization`, `cmd/seed.go`) must set `AiFlaggedLanguages` explicitly since lit inserts every struct field. `tool_names`/`flagged_terms` are stored comma-separated (values sanitized at ingest); the content matcher lives in `backend/app/services/contentflag/`. Conversation analytics exclude rows with an empty `conversation_id`; user analytics additionally require a non-empty `user_id`.
+One `ai_traces_v2` row per LLM call (any OTLP span with `gen_ai.*` attributes). Each row carries a `conversation_id` resolved at ingest (`gen_ai.conversation.id` -> `session.id` span/resource attr -> trace id), `tool_call_count`/`tool_names` parsed from the completion payload (OpenAI `tool_calls`, Anthropic `tool_use`, OTel output messages, `gen_ai.tool.*` fallback), and `flagged`/`flagged_terms` from an ingest-time word-boundary scan of prompt+completion against per-project selected built-in language packs (`projects.ai_flagged_languages`, default `["en"]`; packs live in `backend/app/services/contentflag/terms/*.txt` — en/de/es/fr/it/pt/sr; empty array = custom terms only) plus per-project custom terms (`projects.ai_flagged_terms`); both edited in the project settings AI tab and cached via the project cache. All three project-creation paths (`Create`, `CreateWithOrganization`, `cmd/seed.go`) must set `AiFlaggedLanguages` explicitly since lit inserts every struct field. `tool_names`/`flagged_terms` are stored comma-separated (values sanitized at ingest); the content matcher lives in `backend/app/services/contentflag/`. Conversation analytics exclude rows with an empty `conversation_id`; user analytics additionally require a non-empty `user_id`.
| Method | Endpoint | Auth | Purpose |
|--------|----------|------|---------|
@@ -892,9 +898,9 @@ func (c *ReportController) Report(ctx *gin.Context) {
| Table | Purpose | Partitioning |
|-------|---------|--------------|
| `transactions` | HTTP request metadata | Monthly (`toYYYYMM(timestamp)`) |
-| `exception_stack_traces` | Exceptions with stack traces | Monthly |
+| `spans_v2`, `endpoints_v2`, `tasks_v2`, `ai_traces_v2`, `exceptions_v2` | Traces, see "Trace Storage (V2 tables)" below | Daily (`toYYYYMMDD(recorded_at)`) |
+| `spans`, `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces` | The tables V2 replaced. History only, read by the move-over | Monthly |
| `metric_records` | Time-series system metrics | Monthly |
-| `endpoints` | Endpoint aggregates (materialized) | None |
| `archived_exceptions` | Archived/resolved exceptions | None |
| `log_records` | OTel logs with 3 attribute maps (resource/scope/log) | Daily (`toDate(timestamp)`) |
@@ -948,8 +954,9 @@ Both embedded builds (SQLite and DuckDB telemetry) are single-instance: the data
- `synthetic_checks`, `check_runs`, `check_incidents`, `incident_updates`, `post_mortems`, `post_mortem_events`, `synthetic_runners`, `status_pages`
**Telemetry DB tables** (`db.TelemetryDB` — non-transactional, uses lit with `db.TelemetryDB` directly):
-- `endpoints`, `tasks`, `exception_stack_traces`, `spans`, `metric_points`
+- `endpoints_v2`, `tasks_v2`, `ai_traces_v2`, `exceptions_v2`, `spans_v2`, `metric_points`
- `session_recordings`, `archived_exceptions`, `slow_endpoints`, `fired_notifications`, `check_results`
+- `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`, `spans`: the tables V2 replaced, kept as history for the move-over
**How to access each database in repository code:**
@@ -986,10 +993,40 @@ Built with `-tags telemetry_duckdb` (`CGO_ENABLED=1` required), this is an alter
- **Writes use the Appender API**, not `INSERT` (`duckdb.NewAppenderFromConn(conn, "", table)` → `AppendRow(...)` → `Close()` flushes). Upserts still go through `ExecContext` with `ON CONFLICT`. The Appender rejects typed `*string` for nullable VARCHAR — use `nullableString()` in `backend/app/repositories/telemetry/duckdb/helpers.go` (returns untyped `nil` or the dereferenced value).
- **Write-path observability:** a row the Appender rejects is dropped rather than failing the whole frame (the SQLite backend 500s instead), so a poison row cannot wedge the SDK's retry loop. Every drop increments a per-table counter (`db.RecordTelemetryRowDropped`) and fires a rate-limited (1/min per table) `traceway.CaptureException`; Appender flush/connect failures still propagate to the request (500, SDK retries) and increment an insert-failure counter. `GET /api/health/deep` (operator endpoint: requires the HEALTH_DEEP_TOKEN bearer secret, unset disables it; all telemetry backends) exposes `telemetryBackend`, `droppedRows` per table, `droppedRowsTotal`, `insertFailures`, `ingestRejected` (requests turned away by the ingest admission gate), and on DuckDB an `engine` object (db/WAL file bytes, `duckdb_memory()` usage, read-pool in-use/wait stats) alongside its existing ClickHouse fields; it 503s only when the configured telemetry backend is ClickHouse and CH is unreachable (the embedded backends answer 200 with `chReachable:false`). The benchmark loadgen polls it before/after every ramp step and fails any step whose drop delta is nonzero; read-probe fills record cumulative `droppedRows` per fill level. When `MONITORING_TRACEWAY_URL` is set, `monitoring.StartTelemetryDBReporter` also emits `traceway.duckdb.*` metrics every 10s: `rows_dropped.delta`, `insert_failures.delta`, `db_size_mb`, `wal_size_mb`, `memory_used_mb`, `read_pool.in_use`, `read_pool.wait_count.delta`, `read_pool.wait_ms.delta`. The hourly retention worker issues a `CHECKPOINT` after its deletes so retention actually reclaims disk (DuckDB otherwise defers reclamation to the WAL checkpoint threshold).
- **`lit` placeholders:** `db.Driver` stays `lit.SQLite`, which emits `?` — DuckDB accepts these, so no separate driver was needed for reads.
-- **Migrations:** `backend/app/migrations/duckdb_telemetry/` (mirrors `sqlite_telemetry/` table-for-table; integer columns are `BIGINT`, JSON is `VARCHAR`, no secondary indexes since it's columnar).
+- **Migrations:** `backend/app/migrations/duckdb_telemetry/` (mirrors `sqlite_telemetry/` table-for-table; integer columns are `BIGINT`, JSON is `VARCHAR`, no secondary indexes since it's columnar, with one exception: `0005_create_v2_tables` adds `idx_spans_v2_trace_key` on `spans_v2(trace_key)` for the span graph lookup. While that index exists `ALTER TABLE spans_v2 ADD COLUMN` still works, but `DROP COLUMN` and `ALTER COLUMN ... TYPE` fail with a dependency error, so a migration that needs either must drop the index first).
- **Dialect gotchas vs SQLite** (the read queries differ): native `quantile_cont(col, p)` for P50/P95/P99 instead of fetch-and-sort; `strftime('%s',col)`→`epoch(col)`; time bucketing via `time_bucket(to_seconds(N), col, TIMESTAMP '1970-01-01')` — the explicit epoch origin is required because DuckDB anchors sub-day buckets at 2000-01-03 by default, which would misalign chart buckets against the SQLite backend's epoch-floored buckets for any interval that doesn't evenly divide a day; `json_extract`→`json_extract_string`; `json_each`→`LATERAL unnest(json_keys(x))`; strict GROUP BY needs `ANY_VALUE`/`arg_max`; `SUM` returns HUGEINT (CAST to BIGINT); `CAST(.. AS REAL)`→`CAST(.. AS DOUBLE)`.
- **Tests:** `backend/app/repositories/telemetry/testhelper_duckdb_test.go` (tagged `telemetry_duckdb`) provides `setupTestDB` so the entire existing telemetry test suite runs against an in-memory DuckDB.
+#### Trace Storage (V2 tables)
+
+A trace is a trace and a span is a span. Five tables hold traces on every backend: `spans_v2`, `endpoints_v2`, `tasks_v2`, `ai_traces_v2`, `exceptions_v2` (migrations `ch/0085`-`0089`, one `CREATE TABLE` each; `sqlite_telemetry/0025_create_v2_tables`; `duckdb_telemetry/0005_create_v2_tables`). They sit next to the tables they replaced (`spans`, `endpoints`, `tasks`, `ai_traces`, `exception_stack_traces`), which nothing reads or writes any more except the move-over and retention. `migrations/v2_tables_test.go` holds the V2 migrations to `CREATE TABLE`/`CREATE INDEX` on `_v2` tables only.
+
+- **Ids.** Every row carries the ids its span arrived with, as lowercase hex: `trace_id` (32 characters), `span_id` (16 for OTel, 32 for a native run or span), `parent_span_id` (`''` for a root). Custom linked-trace identity is retired. Startup cleanup migrations remove `linked_trace_id` and its indexes from all four V2 entity/exception tables; OTLP span links remain intact. Entities and exceptions keep a UUID `id` for URLs and notifications. For OTel rows it is `otelOccurrenceID` = `uuid.NewSHA1(projectId, traceId + spanId)`, so a retried export produces the same id. Nothing about identity lives in the attributes JSON, and ingest adds no `traceway.otel.*` keys.
+- **Identity schema cleanup.** Forward migrations `ch/0091`–`0099`, `sqlite_telemetry/0026`, `duckdb_telemetry/0006`–`0007` replace `sessions.distributed_trace_id` with canonical non-null string `trace_id` (empty = no association), remove V2 `linked_trace_id` columns/indexes, and drop the unused `profiles.distributed_trace_id` column/index. Profiles keep their existing `trace_id` / `span_id`. ClickHouse synchronously materializes session IDs before dropping the source column. SQLite/DuckDB commit each migration and its version atomically; DuckDB adds the session non-null constraint in the next transaction. Stop old backends before the first upgraded start and restore a pre-upgrade database for rollback: old session/profile SQL is incompatible. The five old trace tables remain for the move-over.
+- **Ingest** (`otelcontrollers/trace_converter.go`) validates and indexes spans once per resource, then returns canonical spans, entity projections, exceptions and the invalid-ID count together. Each span is promoted when its own kind and attributes make it an entry point, and each exception event becomes an `exceptions_v2` row with the span's trace id and span id. Canonical span attributes are rendered directly from the OTLP payload at storage time. Trace and log JSON IDs are normalized at the decoder boundary (`otelcontrollers/otlp_json.go`); converters consume decoded bytes without a UUID/base64 repair roundtrip. No owner resolution, no caches, no reads. `trace_type` on an exception is stored only when that span itself was promoted (native rows say it themselves). Every entity is recorded at its span's **start**, OTel tasks included. Spans are inserted **before** the rows promoted from them, and a transient storage error answers 503 + `Retry-After` (`abortIngestStorage`, `telemetry.IsTransientStorageError`) so OTLP exporters retry. Retries are append-only; the span reader deduplicates per span.
+- **Entry points.** An HTTP SERVER span becomes an endpoint unless a local span above it already is one (`isEntryPoint`), which is what keeps a SERVER span nested under another (Next.js under the HTTP instrumentation) from counting as a second request. This is the one check that looks beyond the span, and only inside its `ResourceSpans` block. The walk up the parent chain stops at a remote parent per the OTLP span flags (`0x100` has-is-remote, `0x200` is-remote) and at a CLIENT or PRODUCER span. A missing parent has an unknown kind even when flags mark it local, so it cannot suppress an HTTP entry point. Separately exported nested HTTP spans can create extra projections.
+- **Native `/api/report` rows** go to the same tables (`clientmodels.ClientTrace.RootSpan`, `ClientSpan.ToSpan`): a run is stored as the root span of a trace, `span_id` = the run's id, `trace_id` = the run's id. Its spans hang under it, and a native exception sits on the run's span. The report wire format is unchanged: an exception's `traceId` names its owning run, `distributedTraceId` is ignored on runs, exceptions and sessions until the JS SDK supports canonical trace propagation, and native run/span IDs remain UUIDs.
+- **`spans_v2`** keeps the lossless payload of accepted OTLP spans (`resource_pb`/`scope_pb`/`span_pb` on ClickHouse, `otlp` on SQLite and DuckDB; `shared.AssembleOtelPayload` joins them back) next to the readable columns, built once by `OtelSpanRow.NestedValues`: `span_attributes` is a string map (a `Map(LowCardinality(String), String)` on ClickHouse, a JSON object elsewhere) filled by `shared.StringAttributes` the way the OpenTelemetry Collector's ClickHouse exporter does it, and `resource`, `scope`, `events`, `links` are JSON text. A span that did not arrive as OTLP gets a payload synthesized from its available fields (`NewOtelSpanRow`), including string attributes and service/scope metadata. Native UUID span IDs remain in the API; their exported span/parent IDs use a stable eight-byte hash mapping with original IDs carried in `traceway.native.*` attributes. Native synthesis cannot recover metadata never sent by the SDK. On ClickHouse `recorded_at` is `DateTime64(6)` on endpoint/task/exception tables and `DateTime64(3)` on AI traces (the waterfall's origin is the entity's `recordedAt`) and a plain partition time on `spans_v2`.
+- **Reads.** A waterfall is `spans_v2` by project and trace, the subtree under the entity's `span_id`: `telemetry.SpanRepository.FindGraph(ctx, shared.NewSpanLookup(projectId, traceId, spanId, recordedAt))`, answered by `shared.FindOtelGraphs` (topology first, attributes second). A limit never fails the page: the result carries `spanGraphStatus` (`complete`/`partial`/`unavailable` plus reasons) and spans may have `attributesOmitted`. The subtree includes spans that are entities of their own, such as a CONSUMER span started inside the request.
+- **Distributed trace card** (`controllers/distributed_trace.controller.go`, `POST /api/distributed-traces/:traceId`, hex or dashed). `FindByTraceIds` on the four entity and exception repositories matches only `trace_id IN (...)` across projects the user can open. Every returned node belongs to the requested trace; W3C propagation makes browser and backend share that identity. Nodes are nested by one read of the trace's topology, without attributes (`SpanRepository.FindTraceParents`): each node walks up until it meets another node's span (`parentEntitySpanId`). An exception is attached to the nearest node above its span, else it is a node of its own.
+- **Exceptions.** A detail page reads `exceptions_v2` by trace and keeps those on its own span or a span of its subtree (`findOccurrenceExceptions`). `telemetry.FindExceptionOwner` does the walk the other way, from the exception's span up to the nearest endpoint, task or AI trace of the same project. The exception responses return it as `relatedEntity`, and notification text uses it (`resolveExceptionOwner`). A failed task, for the task failure rule, is a task with an exception on its own span.
+- **Every span read is bound in time.** A lookup reads `recorded_at` within 24 hours either side of the entity it starts from (`shared.OtelLookupWindow`, `shared.TraceWindowBounds`; entity lookups by trace id use 48 hours), which on ClickHouse prunes the read to a few daily partitions. A lookup that names no time is anchored on now. The cost is deliberate: a span recorded more than 24 hours away from its entity is not shown on that entity's page. Tests that store spans under a fixed timestamp must pass that timestamp as the lookup time, never the wall clock.
+- **Orphans.** A span whose parent never arrived has no place in a subtree. The entity promoted from the trace's root span (the one with no parent) shows such spans of its own project and trace, so they are visible exactly once, and the waterfall marks them "Parent span is unavailable". Once the parent arrives they nest normally.
+- **Span explorer** (`controllers/span_explorer.controller.go`, frontend `/spans`). `POST /api/spans/search` (App auth + project access) pages the spans of a project in a required time range of at most 31 days. Body: `fromDate`, `toDate`, `orderBy` (`start_time desc|asc`, `duration desc|asc`), `serviceName`, `name` (case-insensitive contains), `traceId` (32 hex), `kind` (0-5), `status` (0-2), `minDurationMs`, `maxDurationMs`, `attributeFilters` (at most 10 exact `{key, value}` matches on `span_attributes`, value required since a ClickHouse Map answers `''` for a missing key), `pagination`. The response adds `services`, the busiest 100 service names in the range. Validation problems and a search that hits its 15s read limit both answer 422, which the page shows as a warning callout, not an error. The query builder is `shared.OtelSearchQueries` with one small dialect struct per backend. `GET /api/spans/traces/:traceId?at=<RFC 3339>` returns the whole trace (`shared.FindOtelTrace`) and 404s when nothing is recorded within 24 hours of `at`. **It reads every project of the current project's organization**, not only the current one (`organizationProjects` in the controller, one lookup per project, one query): services of one trace usually report to different projects, and organization membership is what grants read access to each. Other organizations of the same user stay out. The response names the projects that hold spans (`projects`), each span keeps its own `projectId`, the page links spans across projects (`buildSpanTree(..., acrossProjects)`), and the popover loads attributes with the span's own project id. Search stays per project. Unlike the detail pages it sends **no attributes**: each span carries only `dbStatement`, a 240 character preview of `db.query.text` (or `db.statement`) that the row label needs, and the page fetches one span's attributes from `GET /api/spans/traces/:traceId/spans/:spanId/attributes?at=` when its popover opens. One query serves both cases of the row cap (`FindTraceOutline` on each backend, ordered by `shared.OtelImportanceOrder`): under `MaxOtelGraphRows` every span comes back, over it the `LIMIT` keeps errors first, then entry points (no parent, SERVER or CONSUMER), then the slowest, and the status is `partial` with reasons `row_limit` and `most_important`. Ordering by duration also tends to keep the path above a span, since a parent usually outlasts its children. Responses are not compressed by the backend: 20,000 spans are about 13 MB of JSON, about 1.1 MB once a proxy gzips them. This is the one place browser and mobile projects see their spans, since they have no endpoint or task pages.
+- `GET /api/otel/spans/:traceId/:spanId?at=<RFC 3339>` (App auth + project access) returns one stored span (16 or 32 hex span id) as an OTLP `ExportTraceServiceRequest` protobuf. `at` is required (400 without it) for the same 24 hour rule.
+- **Frontend: large traces.** A trace over 500 spans opens with every node that has more than 50 children folded (`initiallyCollapsed` in `utils/span-tree.ts`); smaller traces open fully expanded. A folded row shows a pill with the number of hidden spans and an error marker when one of them failed, and the header has Expand all and Collapse all. The waterfall keeps what was clicked as the difference from that default, so there is no effect to reset. Pages must hold span payloads in `$state.raw`: a deep `$state` proxy over 20,000 spans took four minutes to read in the browser, against 0.4 seconds raw. The payload is only ever replaced, never edited.
+- **Frontend.** `span-waterfall.svelte` names the service on each row and colours bars by service once a trace crosses services (one colour per row otherwise), marks error spans and missing parents, renders rows 500 at a time so a 20,000 span trace cannot stall the page, caps the name column to half the visible width on narrow screens, and links to the whole trace in the explorer. The span popover shows service, kind, status and scope (`utils/span-fields.ts`). Span payloads are held in `$state.raw`. The tree is keyed `traceId:spanId` (`utils/span-tree.ts`), and the issue pages build their related entity from the backend's `relatedEntity`.
+- `GET /api/health/deep` also reports `spanPartitionTimeFallbacks`: spans whose source start was missing or unrepresentable, so their partition time is the ingest time (`shared.OtelStorageTimes`).
+- **Move-over** (`repositories/telemetry/move_over.go`, started by `retention.startMoveOver` when `V2_MOVE_OVER=true`). Copies the old tables into the V2 tables inside the backend (an embedded DuckDB file cannot be opened by a second process), newest day first, one UTC day of one table at a time, recorded in the transactional DB `v2_move_over_days` (migrations `sqlite/0077`, `pg/0143`) so it stops and resumes. The scheduler runs at startup and one hour after each pass ends, including failures; passes do not overlap. `started` is committed before telemetry writes, `done` after all batches succeed, with no main-DB transaction held during the copy. `moved_rows` counts scanned source rows, including already-copied rows skipped during recovery. Finish upgrading old writers first: completed table/date pairs are not revisited. A resumed day skips rows already written. It reads through `LegacyRepository` (one per backend, the old ids parked in the V2 model fields, see `shared/legacy.go`) and writes through the V2 repositories, so the id mapping lives in one place: the trace id comes from the `traceway.otel.trace_id` attribute the old ingest kept, else the old `distributed_trace_id`, else the row id; an old zero padded span UUID becomes its last 16 hex characters; spans and exceptions find their old owner by id, nearest in time, and take its trace; each entity also gets its own span row. Slices are halved when a read comes back full, so no day is sorted, and only a single overfull second is paged by stable full-row order and offset. Operator notes: `docs/pages/server/upgrade-v2-tables.mdx`.
+
+ClickHouse `ch/0090_add_span_version` adds a materialized SHA-256 fingerprint over length-prefixed protobuf components. All span readers rank by duration then this fingerprint, so topology never reads payload columns for new parts. Old V2 parts calculate it on read until merged or explicitly materialized; no rewrite is scheduled by migration. The column costs 32 uncompressed bytes per physical row.
+
+#### V2 review corrections
+
+Legacy history paging uses stable full-row ordering and offsets only within overfull one-second slices; source tables must be immutable after cutover. Recovery checks project, occurrence ID, trace/span IDs and destination-precision timestamps. Rejected canonical rows fail the native/migration facade, so a day cannot be marked complete after silent row rejection. An old preview's incorrectly completed day can be set back to `started` for checked recovery; see `docs/pages/server/upgrade-v2-tables.mdx`.
+
+Span search, topology, attributes and export choose one retry version (longest duration, then the stored SHA-256 `span_version` on ClickHouse or stored protobuf bytes on SQLite/DuckDB), with UTF-8 byte budgets applied in SQL. Native report spans are written before projections. OTLP storage returns rejected span identities so their endpoint/task/AI/exception projections and conversation uploads are omitted while reporting partial success. HTTP entry-point classification suppresses only an observed local HTTP ancestor, never an absent parent's assumed kind. Fallback AI conversation IDs retain the historical dashed UUID spelling. Log search accepts `traceId` and optional `spanId`, scoped to the selected project and time window. Retired `distributedTraceId` and `excludeTraceId` selectors return 400. The trace log panel has one paginated list with stale-response protection.
+
#### Data Retention
Retention is handled in several different ways depending on the deployment.
@@ -1008,7 +1045,7 @@ Retention is handled in several different ways depending on the deployment.
| `profiles` (slim metadata) | **30 days** | `0069_add_ttl_profiles.up.sql` |
| `profiling_stacks` (dedup table) | **30 days** | `0070_add_ttl_profiling_stacks.up.sql` |
| `check_results` (synthetic check probes) | **90 days** | `0083_add_ttl_check_results.up.sql` |
-| All other CH tables (`transactions`, `exception_stack_traces`, `tasks`, `spans`, `sessions`, `ai_traces`, `session_recordings`, `fired_notifications`, `archived_exceptions`, `slow_endpoints`, `endpoints`, etc.) | **No TTL — retained indefinitely** | — |
+| All other CH tables (the five V2 trace tables and the five they replaced, `transactions`, `sessions`, `session_recordings`, `fired_notifications`, `archived_exceptions`, `slow_endpoints`, etc.) | **No TTL, retained indefinitely** | none |
The three profiling tables share a 30-day TTL keyed on each table's time column (`start_time` / `recorded_at` / `last_seen`). `profiling_stacks` is a `ReplacingMergeTree(last_seen)` dedup table, so `last_seen` is bumped on every re-ingest that references a stack — a stack only ages out once it has gone unreferenced for the full window, which is exactly when its samples have also expired, so no sample is ever left pointing at a dropped stack.
@@ -1023,7 +1060,7 @@ Tables it prunes (and the column used):
| Database | Table | Time column |
|----------|-------|-------------|
-| Telemetry (`db.TelemetryDB`) | `endpoints`, `tasks`, `exception_stack_traces`, `spans`, `metric_points`, `session_recordings`, `ai_traces` | `recorded_at` |
+| Telemetry (`db.TelemetryDB`) | `endpoints_v2`, `tasks_v2`, `exceptions_v2`, `spans_v2`, `ai_traces_v2`, the five tables they replaced (`endpoints`, `tasks`, `exception_stack_traces`, `spans`, `ai_traces`), `metric_points`, `session_recordings` | `recorded_at` |
| Telemetry | `log_records` | `timestamp` |
| Telemetry | `sessions` | `started_at` |
| Telemetry | `fired_notifications` | `fired_at` |
@@ -1281,9 +1318,16 @@ There is **no Traceway Go SDK**. It was retired, and every backend, Go included,
OpenTelemetry and exports over OTLP/HTTP. The docs carry no Go SDK pages, and `docs/public/_redirects`
301s the old `/client/sdk` and `/client/*-middleware` URLs to `/client/otel`. Do not reintroduce them.
-The `/api/report` endpoint below still exists and is still served. The browser and mobile SDKs
-(`@tracewayapp/*`, the iOS and Android libraries, Flutter) speak it, and the backend uses it to report
-its own telemetry. Keep this section accurate for those clients.
+The native protocol is **deprecated**. Traceway is OpenTelemetry first, and the public docs do not
+describe `/api/report` as a way to send data: there is no protocol page (`/protocol` 301s to
+`/client/otel`) and `docs/pages/learn/data-flow.mdx` is written around OTLP. Do not document it again.
+The URL still shows up in the docs in one role only, as part of the connection string the browser and
+mobile SDKs are configured with.
+
+The endpoint itself is still served, because those SDKs (`@tracewayapp/*`, the iOS and Android
+libraries, Flutter) speak it and the backend uses it to report its own telemetry. This section is the
+engineering reference for keeping it working. Its rows land in the same V2 tables as OTLP: a run is
+stored as the root span of a trace with its spans under it (see "Trace Storage (V2 tables)").
### Data Format (Frame)
@@ -1332,6 +1376,7 @@ Clients send data as gzipped JSON. The wire shape is `ReportRequest` in `backend
Notes:
- Timestamps are `recordedAt` (RFC 3339), never `timestamp`. `duration` is a Go `time.Duration`, i.e. integer nanoseconds (45200000 = 45.2ms).
- Endpoints vs tasks share the `traces` array, split by `isTask`. `CollectionFrame` also carries `sessionRecordings` and `sessions`.
+- Each entry in a trace's `spans` may carry an optional `parentSpanId` (UUID) and `attributes` (string map). A span that names no parent hangs under the run's own span. The ids are stored as 32 hex characters.
- **Unknown fields are silently ignored**: a payload in the wrong shape (e.g. top-level `metrics`) still returns 200 but inserts nothing. When hand-crafting test payloads, confirm ingestion landed via `POST /api/metrics/query` (or the relevant list endpoint) instead of trusting the status code.
---
diff --git a/backend/app/config/config.go b/backend/app/config/config.go
index 9dfefdb1..c47681a0 100644
--- a/backend/app/config/config.go
+++ b/backend/app/config/config.go
@@ -48,6 +48,9 @@ type Cfg struct {
ProfileArchiveRaw string
ProfileRetentionDays string
+ V2MoveOver string
+ V2MoveOverOldest string
+
SourceMapCacheMaxEntries string
SourceMapCacheMaxBytesMB string
SourceMapCacheType string
@@ -242,6 +245,9 @@ func LoadFromEnv() *Cfg {
ProfileArchiveRaw: os.Getenv("PROFILE_ARCHIVE_RAW"),
ProfileRetentionDays: os.Getenv("PROFILE_RETENTION_DAYS"),
+ V2MoveOver: os.Getenv("V2_MOVE_OVER"),
+ V2MoveOverOldest: os.Getenv("V2_MOVE_OVER_OLDEST"),
+
SourceMapCacheMaxEntries: os.Getenv("SOURCEMAP_CACHE_MAX_ENTRIES"),
SourceMapCacheMaxBytesMB: os.Getenv("SOURCEMAP_CACHE_MAX_BYTES_MB"),
SourceMapCacheType: os.Getenv("SOURCEMAP_CACHE_TYPE"),
diff --git a/backend/app/controllers/ai_trace.controller.go b/backend/app/controllers/ai_trace.controller.go
index ae7465dc..2069eee2 100644
--- a/backend/app/controllers/ai_trace.controller.go
+++ b/backend/app/controllers/ai_trace.controller.go
@@ -13,6 +13,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
"github.com/tracewayapp/traceway/backend/app/storage"
traceway "go.tracewayapp.com"
)
@@ -48,8 +49,10 @@ type AiTraceInstancesResponse struct {
}
type AiTraceDetailResponse struct {
- AiTrace *models.AiTrace `json:"aiTrace"`
- Conversation json.RawMessage `json:"conversation,omitempty"`
+ SpanGraphStatus *models.SpanGraphStatus `json:"spanGraphStatus"`
+ Spans []models.Span `json:"spans,omitempty"`
+ AiTrace *models.AiTrace `json:"aiTrace"`
+ Conversation json.RawMessage `json:"conversation,omitempty"`
}
type AiConversationSearchRequest struct {
@@ -210,8 +213,15 @@ func (a aiTraceController) GetAiTraceDetail(c *gin.Context) {
return
}
+ graph, err := telemetry.SpanRepository.FindGraph(c, shared.NewSpanLookup(projectId, aiTrace.TraceId, aiTrace.SpanId, aiTrace.RecordedAt))
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error loading AI trace spans: %w", err))
+ return
+ }
response := AiTraceDetailResponse{
- AiTrace: aiTrace,
+ SpanGraphStatus: &graph.Status,
+ Spans: graph.Spans,
+ AiTrace: aiTrace,
}
if aiTrace.StorageKey != "" {
diff --git a/backend/app/controllers/clientcontrollers/client.controller.go b/backend/app/controllers/clientcontrollers/client.controller.go
index 6eeaf3dd..2c2f309c 100644
--- a/backend/app/controllers/clientcontrollers/client.controller.go
+++ b/backend/app/controllers/clientcontrollers/client.controller.go
@@ -116,6 +116,12 @@ func (e clientController) Report(c *gin.Context) {
return
}
parseSpan.End()
+ for _, frame := range request.CollectionFrames {
+ if err := frame.Validate(); err != nil {
+ c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
+ return
+ }
+ }
bodyBytes := 0
if cb, ok := c.Get(gin.BodyBytesKey); ok {
@@ -154,6 +160,9 @@ func (e clientController) Report(c *gin.Context) {
}
for _, ct := range cf.Traces {
+ if ct == nil {
+ continue
+ }
if ct.IsTask {
t := ct.ToTask(request.AppVersion, request.ServerName)
t.ProjectId = projectId
@@ -167,10 +176,11 @@ func (e clientController) Report(c *gin.Context) {
endpointsToInsert = append(endpointsToInsert, e)
}
+ spansToInsert = append(spansToInsert, ct.RootSpan(projectId, request.ServerName))
for _, cs := range ct.Spans {
- span := cs.ToSpan(ct.ParsedId())
- span.ProjectId = projectId
- spansToInsert = append(spansToInsert, span)
+ if cs != nil {
+ spansToInsert = append(spansToInsert, cs.ToSpan(ct, projectId, request.ServerName))
+ }
}
}
resolveJs := project != nil && project.SourceMapToken != nil && jsFrameworks[project.Framework]
@@ -261,10 +271,13 @@ func (e clientController) Report(c *gin.Context) {
}
convertSpan.End()
- var droppedHealthchecks int
- endpointsToInsert, spansToInsert, droppedHealthchecks = services.FilterHealthchecks(project, endpointsToInsert, spansToInsert, exceptionStackTraceToInsert)
- if droppedHealthchecks > 0 {
- monitoring.RecordHealthchecksDropped(monitoring.SignalNative, droppedHealthchecks)
+ var droppedHealthchecks map[string]bool
+ endpointsToInsert, droppedHealthchecks = services.FilterHealthchecks(project, endpointsToInsert, exceptionStackTraceToInsert)
+ spansToInsert = services.DropSpanSubtrees(spansToInsert, func(span models.Span) (string, string, string) {
+ return span.TraceId, span.SpanId, span.ParentSpanId
+ }, droppedHealthchecks, nil)
+ if len(droppedHealthchecks) > 0 {
+ monitoring.RecordHealthchecksDropped(monitoring.SignalNative, len(droppedHealthchecks))
}
perm := hooks.IngestPermission{Exceptions: true, Data: true, Replay: true}
@@ -300,6 +313,15 @@ func (e clientController) Report(c *gin.Context) {
convertMs := float64(time.Since(convertStart).Microseconds()) / 1000.0
insertStart := time.Now()
+ spanInsertSpan := traceway.StartSpan(c, "report.insert.spans")
+ err = telemetry.SpanRepository.InsertAsync(c, spansToInsert)
+ spanInsertSpan.End()
+
+ if err != nil {
+ c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting spansToInsert: %w", err))
+ return
+ }
+
if len(endpointsToInsert) > 0 {
insertSpan := traceway.StartSpan(c, "report.insert.endpoints")
err := telemetry.EndpointRepository.InsertAsync(c, endpointsToInsert)
@@ -352,15 +374,6 @@ func (e clientController) Report(c *gin.Context) {
go services.AutoRegisterMetrics(projectId, metricNames)
}
- spanInsertSpan := traceway.StartSpan(c, "report.insert.spans")
- err = telemetry.SpanRepository.InsertAsync(c, spansToInsert)
- spanInsertSpan.End()
-
- if err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting spansToInsert: %w", err))
- return
- }
-
insertMs := float64(time.Since(insertStart).Microseconds()) / 1000.0
totalSize := len(endpointsToInsert) + len(tasksToInsert) + len(spansToInsert) + len(exceptionStackTraceToInsert) + len(metricPointsToInsert)
monitoring.RecordIngestBatch(monitoring.SignalNative, "report", convertMs, insertMs, totalSize, bodyBytes)
diff --git a/backend/app/controllers/clientcontrollers/report_spans_test.go b/backend/app/controllers/clientcontrollers/report_spans_test.go
new file mode 100644
index 00000000..0b3dd0a8
--- /dev/null
+++ b/backend/app/controllers/clientcontrollers/report_spans_test.go
@@ -0,0 +1,231 @@
+//go:build !telemetry_ch && !telemetry_duckdb && !transactional_pg
+
+package clientcontrollers
+
+import (
+ "bytes"
+ "context"
+ "encoding/hex"
+ "encoding/json"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models/clientmodels"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestReportIgnoresLegacyDistributedTraceID(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ project, run, distributed, session := uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ at := time.Now().UTC()
+ traceID, spanID := hex.EncodeToString(run[:]), hex.EncodeToString(run[:])
+ body, err := json.Marshal(map[string]any{
+ "collectionFrames": []any{map[string]any{
+ "traces": []any{map[string]any{"id": run, "endpoint": "GET /native", "distributedTraceId": distributed, "recordedAt": at, "duration": 1000}},
+ "stackTraces": []any{map[string]any{
+ "traceId": run, "distributedTraceId": distributed, "stackTrace": "Error: request failed", "recordedAt": at, "sessionId": session,
+ }},
+ "sessions": []any{map[string]any{"id": session, "distributedTraceId": distributed, "startedAt": at}},
+ }},
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ rec := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(rec)
+ c.Request = httptest.NewRequest("POST", "/api/report", bytes.NewReader(body))
+ c.Request.Header.Set("Content-Type", "application/json")
+ c.Set(middleware.ProjectIdContextKey, project)
+ ClientController.Report(c)
+ if rec.Code != 200 || len(c.Errors) > 0 {
+ t.Fatalf("report: %d %s %v", rec.Code, rec.Body.String(), c.Errors)
+ }
+ rows, err := telemetry.ExceptionStackTraceRepository.FindByTraceIds(context.Background(), []string{traceID}, []uuid.UUID{project}, &at)
+ if err != nil || len(rows) != 1 || rows[0].TraceId != traceID || rows[0].SpanId != spanID || rows[0].TraceType != "endpoint" {
+ t.Fatalf("native report lost its trace or owning run: %+v %v", rows, err)
+ }
+ ignored, err := telemetry.ExceptionStackTraceRepository.FindByTraceIds(context.Background(), []string{hex.EncodeToString(distributed[:])}, []uuid.UUID{project}, &at)
+ if err != nil || len(ignored) != 0 {
+ t.Fatalf("legacy field created an association: %+v %v", ignored, err)
+ }
+ owner, err := telemetry.FindExceptionOwner(context.Background(), rows[0])
+ if err != nil || owner == nil || owner.Id != run || rows[0].SessionId == nil || *rows[0].SessionId != session {
+ t.Fatalf("native exception lost its owner or session: %+v %+v %v", owner, rows[0], err)
+ }
+ storedSession, err := telemetry.SessionRepository.FindById(context.Background(), project, session, &at)
+ if err != nil || storedSession == nil || storedSession.TraceId != "" {
+ t.Fatalf("legacy field created a session trace association: %+v %v", storedSession, err)
+ }
+ encoded, err := json.Marshal(rows[0])
+ if err != nil || bytes.Contains(encoded, []byte("linkedTraceId")) || bytes.Contains(encoded, []byte("distributedTraceId")) {
+ t.Fatalf("retired identity exposed: %s %v", encoded, err)
+ }
+}
+
+func TestReportKeepsNativeSpanEdgesAndHistoricalTimes(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ project, owner, parent, child := uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ recorded := time.Now().UTC().Add(-72 * time.Hour)
+ frame := map[string]any{"traces": []any{map[string]any{"id": owner, "distributedTraceId": uuid.New(), "endpoint": "native-task", "isTask": true, "recordedAt": recorded, "duration": 2000000,
+ "spans": []any{
+ map[string]any{"id": child, "name": "child", "parentSpanId": parent, "attributes": map[string]string{"native": "preserved"}, "startTime": recorded, "duration": 1000},
+ map[string]any{"id": parent, "name": "parent", "parentSpanId": "", "startTime": recorded, "duration": 2000},
+ }}}}
+ body, err := json.Marshal(map[string]any{"serverName": "native-host", "collectionFrames": []any{frame}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ rec := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(rec)
+ c.Request = httptest.NewRequest("POST", "/api/report", bytes.NewReader(body))
+ c.Request.Header.Set("Content-Type", "application/json")
+ c.Set(middleware.ProjectIdContextKey, project)
+ ClientController.Report(c)
+ if rec.Code != 200 || len(c.Errors) > 0 {
+ t.Fatalf("report: %d %v", rec.Code, c.Errors)
+ }
+ task, err := telemetry.TaskRepository.FindById(context.Background(), project, owner, &recorded)
+ if err != nil || task == nil {
+ t.Fatalf("task: %+v %v", task, err)
+ }
+ // A native run is the root span of a trace named after it, so it reads like any other.
+ run := hex.EncodeToString(owner[:])
+ if task.TraceId != run || task.SpanId != run || task.ParentSpanId != "" || !task.IsRoot {
+ t.Fatalf("native task ids: %+v", task)
+ }
+ graph, err := telemetry.SpanRepository.FindGraph(context.Background(), shared.NewSpanLookup(project, task.TraceId, task.SpanId, task.RecordedAt))
+ if err != nil {
+ t.Fatal(err)
+ }
+ spans := graph.Spans
+ if len(spans) != 2 {
+ t.Fatalf("native graph: %+v", spans)
+ }
+ for _, span := range spans {
+ switch span.SpanId {
+ case hex.EncodeToString(child[:]):
+ if span.ParentSpanId != hex.EncodeToString(parent[:]) || span.Attributes["native"] != "preserved" {
+ t.Fatalf("lost native edge: %+v", span)
+ }
+ case hex.EncodeToString(parent[:]):
+ if span.ParentSpanId != run {
+ t.Fatalf("a span that names no parent hangs under the run: %+v", span)
+ }
+ default:
+ t.Fatalf("unexpected span: %+v", span)
+ }
+ if !span.StartTime.Equal(recorded) || span.TraceId != run || span.ServiceName != "native-host" {
+ t.Fatalf("native identity/time changed: %+v", span)
+ }
+ }
+ whole, err := telemetry.SpanRepository.FindTrace(context.Background(), []uuid.UUID{project}, run, recorded)
+ if err != nil || len(whole.Spans) != 3 {
+ t.Fatalf("the whole trace is the run's own span and its two children: %+v %v", whole, err)
+ }
+ for _, span := range whole.Spans {
+ if (span.ParentSpanId == "") != (span.Name == "native-task") {
+ t.Fatalf("the run's own span is the only root: %+v", span)
+ }
+ }
+ exported := map[string]*tracepb.Span{}
+ for _, span := range whole.Spans {
+ payload, err := telemetry.OtelSpanRepository.FindOTLP(context.Background(), project, run, span.SpanId, recorded)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var resource tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &resource); err != nil {
+ t.Fatal(err)
+ }
+ if len(resource.ScopeSpans) != 1 || len(resource.ScopeSpans[0].Spans) != 1 || shared.StringAttributes(resource.Resource.GetAttributes())["service.name"] != "native-host" {
+ t.Fatalf("native export lost its envelope: %v", &resource)
+ }
+ source := resource.ScopeSpans[0].Spans[0]
+ if len(source.TraceId) != 16 || len(source.SpanId) != 8 || (len(source.ParentSpanId) != 0 && len(source.ParentSpanId) != 8) || hex.EncodeToString(source.TraceId) != run {
+ t.Fatalf("invalid native OTLP identity: %v", source)
+ }
+ if shared.StringAttributes(source.Attributes)["traceway.native.span_id"] != span.SpanId {
+ t.Fatal("export must retain the native API identity")
+ }
+ exported[span.SpanId] = source
+ }
+ for _, span := range whole.Spans {
+ if span.ParentSpanId != "" && !bytes.Equal(exported[span.SpanId].ParentSpanId, exported[span.ParentSpanId].SpanId) {
+ t.Fatal("native parent and child use different OTLP identity mappings")
+ }
+ }
+ if shared.StringAttributes(exported[hex.EncodeToString(child[:])].Attributes)["native"] != "preserved" {
+ t.Fatal("native attributes were not exported")
+ }
+ // A malformed legacy ID must generate one owner for both projection and children.
+ trace := &clientmodels.ClientTrace{Id: "invalid"}
+ id := trace.ParsedId()
+ if trace.ToTask("", "").Id != id || trace.ToEndpoint("", "").Id != id {
+ t.Fatal("generated owner identity changed between conversions")
+ }
+}
+
+func TestReportZeroDistributedIDKeepsBatchAndExceptionCorrelation(t *testing.T) {
+ for _, isTask := range []bool{false, true} {
+ name := "endpoint"
+ if isTask {
+ name = "task"
+ }
+ t.Run(name, func(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ project, run, healthy := uuid.New(), uuid.New(), uuid.New()
+ at := time.Now().UTC()
+ frame := map[string]any{
+ "traces": []any{
+ map[string]any{"id": run, "endpoint": "zero-header", "isTask": isTask, "distributedTraceId": uuid.Nil, "recordedAt": at, "duration": 1000,
+ "spans": []any{map[string]any{"id": uuid.Nil, "parentSpanId": uuid.Nil, "name": "child", "startTime": at, "duration": 500}}},
+ map[string]any{"id": healthy, "endpoint": "healthy", "recordedAt": at, "duration": 1000},
+ },
+ "stackTraces": []any{map[string]any{"traceId": run, "distributedTraceId": uuid.Nil, "isTask": isTask, "stackTrace": "Error: zero header", "recordedAt": at}},
+ }
+ body, err := json.Marshal(map[string]any{"collectionFrames": []any{frame}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ rec := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(rec)
+ c.Request = httptest.NewRequest("POST", "/api/report", bytes.NewReader(body))
+ c.Request.Header.Set("Content-Type", "application/json")
+ c.Set(middleware.ProjectIdContextKey, project)
+ ClientController.Report(c)
+ if rec.Code != 200 || len(c.Errors) > 0 {
+ t.Fatalf("report: %d %v", rec.Code, c.Errors)
+ }
+ ctx := context.Background()
+ if endpoint, err := telemetry.EndpointRepository.FindById(ctx, project, healthy, &at); err != nil || endpoint == nil {
+ t.Fatalf("healthy request in the batch was lost: %+v %v", endpoint, err)
+ }
+ traceId := hex.EncodeToString(run[:])
+ whole, err := telemetry.SpanRepository.FindTrace(ctx, []uuid.UUID{project}, traceId, at)
+ if err != nil || len(whole.Spans) != 2 {
+ t.Fatalf("native root and child: %+v %v", whole, err)
+ }
+ for _, span := range whole.Spans {
+ if span.TraceId != traceId || span.SpanId == hex.EncodeToString(uuid.Nil[:]) || (span.Name == "child" && span.ParentSpanId != traceId) {
+ t.Fatalf("invalid native identity: %+v", span)
+ }
+ }
+ exceptions, err := telemetry.ExceptionStackTraceRepository.FindAllByTraceId(ctx, project, traceId, &at)
+ if err != nil || len(exceptions) != 1 || exceptions[0].SpanId != traceId {
+ t.Fatalf("exception lost its run: %+v %v", exceptions, err)
+ }
+ owner, err := telemetry.FindExceptionOwner(ctx, exceptions[0])
+ if err != nil || owner == nil || owner.TraceType != name || owner.Name != "zero-header" {
+ t.Fatalf("exception owner: %+v %v", owner, err)
+ }
+ })
+ }
+}
diff --git a/backend/app/controllers/distributed_trace.controller.go b/backend/app/controllers/distributed_trace.controller.go
index c5aead49..a428d6f4 100644
--- a/backend/app/controllers/distributed_trace.controller.go
+++ b/backend/app/controllers/distributed_trace.controller.go
@@ -3,6 +3,9 @@ package controllers
import (
"context"
"database/sql"
+ "errors"
+ "fmt"
+ "io"
"net/http"
"time"
@@ -25,31 +28,41 @@ type distributedTraceRequest struct {
}
type DistributedTraceNode struct {
- ProjectId uuid.UUID `json:"projectId"`
- ProjectName string `json:"projectName"`
- TraceType string `json:"traceType"`
- Endpoint *models.Endpoint `json:"endpoint,omitempty"`
- Task *models.Task `json:"task,omitempty"`
- AiTrace *models.AiTrace `json:"aiTrace,omitempty"`
- Spans []models.Span `json:"spans"`
- Exception *EndpointExceptionInfo `json:"exception,omitempty"`
+ ProjectId uuid.UUID `json:"projectId"`
+ ProjectName string `json:"projectName"`
+ TraceType string `json:"traceType"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ Endpoint *models.Endpoint `json:"endpoint,omitempty"`
+ Task *models.Task `json:"task,omitempty"`
+ AiTrace *models.AiTrace `json:"aiTrace,omitempty"`
+ SpanGraphStatus *models.SpanGraphStatus `json:"spanGraphStatus,omitempty"`
+ Spans []models.Span `json:"spans"`
+ Exception *EndpointExceptionInfo `json:"exception,omitempty"`
+ // ParentEntitySpanId is the span of the nearest endpoint, task or AI trace above this one, found by walking the
+ // whole trace across projects. The hops in between need not belong to any of them, or to a project that has one.
+ ParentEntitySpanId string `json:"parentEntitySpanId,omitempty"`
+
+ recordedAt time.Time
}
type DistributedTraceResponse struct {
- DistributedTraceId string `json:"distributedTraceId"`
- Nodes []DistributedTraceNode `json:"nodes"`
+ TraceId string `json:"traceId"`
+ Nodes []DistributedTraceNode `json:"nodes"`
}
func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {
- distributedTraceIdStr := c.Param("distributedTraceId")
- distributedTraceId, err := uuid.Parse(distributedTraceIdStr)
- if err != nil {
- c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid distributedTraceId"})
+ traceId := shared.NormalizeTraceId(c.Param("traceId"))
+ if !validTraceHex(traceId) {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid traceId"})
return
}
var request distributedTraceRequest
- _ = c.ShouldBindJSON(&request)
+ if err := c.ShouldBindJSON(&request); err != nil && !errors.Is(err, io.EOF) {
+ middleware.RejectBindError(c, err, "Invalid distributed trace request")
+ return
+ }
userId := middleware.GetUserId(c)
@@ -62,10 +75,7 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {
}
if len(projects) == 0 {
- c.JSON(http.StatusOK, DistributedTraceResponse{
- DistributedTraceId: distributedTraceIdStr,
- Nodes: []DistributedTraceNode{},
- })
+ c.JSON(http.StatusOK, DistributedTraceResponse{TraceId: traceId, Nodes: []DistributedTraceNode{}})
return
}
@@ -78,167 +88,236 @@ func (d distributedTraceController) GetDistributedTrace(c *gin.Context) {
ctx := c.Request.Context()
- endpoints, err := telemetry.EndpointRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds, request.RecordedAt)
+ found, err := findTraceEntities(ctx, traceId, projectIds, request.RecordedAt)
if err != nil {
- c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query endpoints: %w", err))
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query the distributed trace: %w", err))
return
}
- tasks, err := telemetry.TaskRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds, request.RecordedAt)
- if err != nil {
- c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query tasks: %w", err))
- return
+ nodes := make([]DistributedTraceNode, 0, len(found.endpoints)+len(found.tasks)+len(found.aiTraces))
+ for _, ep := range found.endpoints {
+ nodes = append(nodes, DistributedTraceNode{
+ ProjectId: ep.ProjectId, ProjectName: projectNameMap[ep.ProjectId], TraceType: "endpoint",
+ TraceId: ep.TraceId, SpanId: ep.SpanId, Endpoint: &ep, Spans: []models.Span{}, recordedAt: ep.RecordedAt,
+ })
}
-
- aiTraces, err := telemetry.AiTraceRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds, request.RecordedAt)
- if err != nil {
- c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query ai traces: %w", err))
- return
+ for _, t := range found.tasks {
+ nodes = append(nodes, DistributedTraceNode{
+ ProjectId: t.ProjectId, ProjectName: projectNameMap[t.ProjectId], TraceType: "task",
+ TraceId: t.TraceId, SpanId: t.SpanId, Task: &t, Spans: []models.Span{}, recordedAt: t.RecordedAt,
+ })
}
-
- exceptions, err := telemetry.ExceptionStackTraceRepository.FindByDistributedTraceId(ctx, distributedTraceId, projectIds, request.RecordedAt)
- if err != nil {
- c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query exceptions: %w", err))
- return
+ for _, a := range found.aiTraces {
+ nodes = append(nodes, DistributedTraceNode{
+ ProjectId: a.ProjectId, ProjectName: projectNameMap[a.ProjectId], TraceType: "ai_trace",
+ TraceId: a.TraceId, SpanId: a.SpanId, AiTrace: &a, Spans: []models.Span{}, recordedAt: a.RecordedAt,
+ })
}
- exceptionByTraceId := make(map[distributedTraceOwner]*EndpointExceptionInfo)
- for _, exc := range exceptions {
- if exc.TraceId != nil {
- if _, exists := exceptionByTraceId[distributedTraceOwner{exc.ProjectId, *exc.TraceId}]; !exists {
- exceptionByTraceId[distributedTraceOwner{exc.ProjectId, *exc.TraceId}] = &EndpointExceptionInfo{
- ExceptionHash: exc.ExceptionHash,
- StackTrace: exc.StackTrace,
- RecordedAt: exc.RecordedAt.Format("2006-01-02T15:04:05Z07:00"),
- }
+ parents := findTraceParents(ctx, nodes, found.exceptions, projectIds)
+ linkDistributedTraceNodes(nodes, parents)
+
+ for _, exc := range found.exceptions {
+ info := &EndpointExceptionInfo{
+ ExceptionHash: exc.ExceptionHash,
+ StackTrace: exc.StackTrace,
+ RecordedAt: exc.RecordedAt.Format("2006-01-02T15:04:05Z07:00"),
+ }
+ if owner := exceptionOwner(nodes, parents, exc); owner >= 0 {
+ if nodes[owner].Exception == nil {
+ nodes[owner].Exception = info
}
+ continue
}
+ nodes = append(nodes, DistributedTraceNode{
+ ProjectId: exc.ProjectId, ProjectName: projectNameMap[exc.ProjectId], TraceType: "exception",
+ TraceId: exc.TraceId, SpanId: exc.SpanId, Spans: []models.Span{}, Exception: info, recordedAt: exc.RecordedAt,
+ })
}
- matchedIds := make(map[distributedTraceOwner]bool)
- for _, ep := range endpoints {
- matchedIds[distributedTraceOwner{ep.ProjectId, ep.Id}] = true
+ if err := loadDistributedTraceSpans(ctx, nodes); err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query distributed trace spans: %w", err))
+ return
}
- for _, t := range tasks {
- matchedIds[distributedTraceOwner{t.ProjectId, t.Id}] = true
+
+ c.JSON(http.StatusOK, DistributedTraceResponse{TraceId: traceId, Nodes: nodes})
+}
+
+func loadDistributedTraceSpans(ctx context.Context, nodes []DistributedTraceNode) error {
+ nodeIndexes := make(map[shared.SpanOwner][]int)
+ lookups := make([]shared.SpanLookup, 0, len(nodes))
+ for i, node := range nodes {
+ if node.TraceId == "" || node.SpanId == "" {
+ continue
+ }
+ nodes[i].SpanGraphStatus = &models.SpanGraphStatus{State: models.SpanGraphComplete}
+ lookup := shared.NewSpanLookup(node.ProjectId, node.TraceId, node.SpanId, node.recordedAt)
+ nodeIndexes[lookup.Owner()] = append(nodeIndexes[lookup.Owner()], i)
+ lookups = append(lookups, lookup)
}
- for _, a := range aiTraces {
- matchedIds[distributedTraceOwner{a.ProjectId, a.Id}] = true
+ graphs, err := telemetry.SpanRepository.FindGraphs(ctx, lookups)
+ if err != nil {
+ return err
+ }
+ for owner, graph := range graphs {
+ for _, i := range nodeIndexes[owner] {
+ nodes[i].SpanGraphStatus = &graph.Status
+ nodes[i].Spans = graph.Spans
+ }
}
+ return nil
+}
- var nodes []DistributedTraceNode
- var lookups []telemetry.SpanLookup
+const (
+ maxEntityLinkDepth = 10000
+)
+
+type traceEntities struct {
+ endpoints []models.Endpoint
+ tasks []models.Task
+ aiTraces []models.AiTrace
+ exceptions []models.ExceptionStackTrace
+}
- for _, ep := range endpoints {
- node := DistributedTraceNode{
- ProjectId: ep.ProjectId,
- ProjectName: projectNameMap[ep.ProjectId],
- TraceType: "endpoint",
- Endpoint: &ep,
- Spans: []models.Span{},
- Exception: exceptionByTraceId[distributedTraceOwner{ep.ProjectId, ep.Id}],
+// findTraceEntities reads one trace across the projects the caller can access.
+func findTraceEntities(ctx context.Context, traceId string, projectIds []uuid.UUID, recordedAt *time.Time) (*traceEntities, error) {
+ if recordedAt == nil || recordedAt.IsZero() {
+ now := time.Now().UTC()
+ recordedAt = &now
+ }
+ found := &traceEntities{}
+ var err error
+ ids := []string{traceId}
+ found.endpoints, err = telemetry.EndpointRepository.FindByTraceIds(ctx, ids, projectIds, recordedAt)
+ if err != nil {
+ return nil, fmt.Errorf("endpoints: %w", err)
+ }
+ found.tasks, err = telemetry.TaskRepository.FindByTraceIds(ctx, ids, projectIds, recordedAt)
+ if err != nil {
+ return nil, fmt.Errorf("tasks: %w", err)
+ }
+ found.aiTraces, err = telemetry.AiTraceRepository.FindByTraceIds(ctx, ids, projectIds, recordedAt)
+ if err != nil {
+ return nil, fmt.Errorf("ai traces: %w", err)
+ }
+ found.exceptions, err = telemetry.ExceptionStackTraceRepository.FindByTraceIds(ctx, ids, projectIds, recordedAt)
+ if err != nil {
+ return nil, fmt.Errorf("exceptions: %w", err)
+ }
+ endpoints, tasks, aiTraces, exceptions := found.endpoints, found.tasks, found.aiTraces, found.exceptions
+ found = &traceEntities{}
+ seen := map[string]bool{}
+ fresh := func(kind string, projectId, id uuid.UUID, traceId, spanId string) bool {
+ key := kind + ":" + projectId.String() + ":" + id.String() + ":" + traceId + ":" + spanId
+ if seen[key] {
+ return false
}
- nodes = append(nodes, node)
- lookups = append(lookups, telemetry.SpanLookup{ProjectId: ep.ProjectId, TraceId: ep.Id, RecordedAt: &ep.RecordedAt})
+ seen[key] = true
+ return true
}
-
- for _, t := range tasks {
- node := DistributedTraceNode{
- ProjectId: t.ProjectId,
- ProjectName: projectNameMap[t.ProjectId],
- TraceType: "task",
- Task: &t,
- Spans: []models.Span{},
- Exception: exceptionByTraceId[distributedTraceOwner{t.ProjectId, t.Id}],
+ for _, endpoint := range endpoints {
+ if fresh("endpoint", endpoint.ProjectId, endpoint.Id, endpoint.TraceId, endpoint.SpanId) {
+ found.endpoints = append(found.endpoints, endpoint)
}
- nodes = append(nodes, node)
- lookups = append(lookups, telemetry.SpanLookup{ProjectId: t.ProjectId, TraceId: t.Id, RecordedAt: &t.RecordedAt})
}
-
- for _, a := range aiTraces {
- node := DistributedTraceNode{
- ProjectId: a.ProjectId,
- ProjectName: projectNameMap[a.ProjectId],
- TraceType: "ai_trace",
- AiTrace: &a,
- Spans: []models.Span{},
- Exception: exceptionByTraceId[distributedTraceOwner{a.ProjectId, a.Id}],
+ for _, task := range tasks {
+ if fresh("task", task.ProjectId, task.Id, task.TraceId, task.SpanId) {
+ found.tasks = append(found.tasks, task)
}
- nodes = append(nodes, node)
- lookups = append(lookups, telemetry.SpanLookup{ProjectId: a.ProjectId, TraceId: a.Id, RecordedAt: &a.RecordedAt})
}
-
- for _, exc := range exceptions {
- if exc.TraceId != nil && matchedIds[distributedTraceOwner{exc.ProjectId, *exc.TraceId}] {
- continue
+ for _, aiTrace := range aiTraces {
+ if fresh("ai_trace", aiTrace.ProjectId, aiTrace.Id, aiTrace.TraceId, aiTrace.SpanId) {
+ found.aiTraces = append(found.aiTraces, aiTrace)
}
- nodes = append(nodes, DistributedTraceNode{
- ProjectId: exc.ProjectId,
- ProjectName: projectNameMap[exc.ProjectId],
- TraceType: "exception",
- Spans: []models.Span{},
- Exception: &EndpointExceptionInfo{
- ExceptionHash: exc.ExceptionHash,
- StackTrace: exc.StackTrace,
- RecordedAt: exc.RecordedAt.Format("2006-01-02T15:04:05Z07:00"),
- },
- })
- lookup := telemetry.SpanLookup{ProjectId: exc.ProjectId, RecordedAt: &exc.RecordedAt}
- if exc.TraceId != nil {
- lookup.TraceId = *exc.TraceId
+ }
+ for _, exception := range exceptions {
+ if fresh("exception", exception.ProjectId, exception.Id, exception.TraceId, exception.SpanId) {
+ found.exceptions = append(found.exceptions, exception)
}
- lookups = append(lookups, lookup)
}
+ return found, nil
+}
- if err := loadDistributedTraceSpans(ctx, nodes, lookups); err != nil {
- c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to query distributed trace spans: %w", err))
- return
+// findTraceParents maps span to parent span for every trace on the card, keyed by trace id. It reads no attributes. The
+// subtrees loaded per node are not enough to nest the nodes: a hop that was never promoted, such as a gRPC server
+// reporting to its own project, belongs to no subtree, and the nodes on either side of it would show side by side.
+func findTraceParents(ctx context.Context, nodes []DistributedTraceNode, exceptions []models.ExceptionStackTrace, projectIds []uuid.UUID) map[string]map[string]string {
+ earliest := map[string]time.Time{}
+ note := func(traceId string, at time.Time) {
+ if traceId == "" {
+ return
+ }
+ if known, ok := earliest[traceId]; !ok || at.Before(known) {
+ earliest[traceId] = at
+ }
}
-
- if nodes == nil {
- nodes = []DistributedTraceNode{}
+ for _, node := range nodes {
+ note(node.TraceId, node.recordedAt)
}
+ for _, exception := range exceptions {
+ note(exception.TraceId, exception.RecordedAt)
+ }
+ parents := map[string]map[string]string{}
+ for traceId, at := range earliest {
+ traceParents, err := telemetry.SpanRepository.FindTraceParents(ctx, projectIds, traceId, at)
+ if err != nil {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("failed to read the span parents of trace %s: %w", traceId, err))
+ continue
+ }
+ parents[traceId] = traceParents
+ }
+ return parents
+}
- c.JSON(http.StatusOK, DistributedTraceResponse{
- DistributedTraceId: distributedTraceIdStr,
- Nodes: nodes,
- })
+// nearestEntity walks up from a span until it meets the span of a node. It prefers a node of the same project, because
+// one payload exported to two projects promotes the same span twice.
+func nearestEntity(nodes []DistributedTraceNode, parents map[string]map[string]string, projectId uuid.UUID, traceId, spanId string, includeSelf bool) int {
+ bySpan := map[string][]int{}
+ for i, node := range nodes {
+ if node.TraceId == traceId && node.SpanId != "" && node.TraceType != "exception" {
+ bySpan[node.SpanId] = append(bySpan[node.SpanId], i)
+ }
+ }
+ pick := func(candidates []int) int {
+ for _, i := range candidates {
+ if nodes[i].ProjectId == projectId {
+ return i
+ }
+ }
+ return candidates[0]
+ }
+ parentOf := parents[traceId]
+ current := spanId
+ if !includeSelf {
+ current = parentOf[spanId]
+ }
+ visited := map[string]bool{}
+ for depth := 0; current != "" && !visited[current] && depth < maxEntityLinkDepth; depth++ {
+ if candidates := bySpan[current]; len(candidates) > 0 {
+ return pick(candidates)
+ }
+ visited[current] = true
+ current = parentOf[current]
+ }
+ return -1
}
-type distributedTraceOwner struct {
- projectId uuid.UUID
- traceId uuid.UUID
+func exceptionOwner(nodes []DistributedTraceNode, parents map[string]map[string]string, exc models.ExceptionStackTrace) int {
+ if exc.TraceId == "" || exc.SpanId == "" {
+ return -1
+ }
+ return nearestEntity(nodes, parents, exc.ProjectId, exc.TraceId, exc.SpanId, true)
}
-func loadDistributedTraceSpans(ctx context.Context, nodes []DistributedTraceNode, lookups []telemetry.SpanLookup) error {
- nodeIndexes := make(map[distributedTraceOwner][]int)
- queries := make([]telemetry.SpanLookup, 0, len(lookups))
- for i, lookup := range lookups {
- if lookup.TraceId == uuid.Nil {
+func linkDistributedTraceNodes(nodes []DistributedTraceNode, parents map[string]map[string]string) {
+ for i, node := range nodes {
+ if node.SpanId == "" {
continue
}
- key := distributedTraceOwner{lookup.ProjectId, lookup.TraceId}
- nodeIndexes[key] = append(nodeIndexes[key], i)
- queries = append(queries, lookup)
- }
- spans, err := telemetry.SpanRepository.FindByTraces(ctx, queries)
- if err != nil {
- return err
- }
- for _, span := range spans {
- for _, i := range nodeIndexes[distributedTraceOwner{span.ProjectId, span.TraceId}] {
- // A bulk result can cover multiple occurrences of the same ID. Keep
- // each node's window consistent with its individual detail endpoint.
- if lookups[i].RecordedAt != nil {
- from, to := shared.TraceWindowBounds(*lookups[i].RecordedAt)
- if span.RecordedAt.Before(from) || span.RecordedAt.After(to) {
- continue
- }
- }
- nodes[i].Spans = append(nodes[i].Spans, span)
+ if parent := nearestEntity(nodes, parents, node.ProjectId, node.TraceId, node.SpanId, false); parent >= 0 {
+ nodes[i].ParentEntitySpanId = nodes[parent].SpanId
}
}
- return nil
}
var DistributedTraceController = distributedTraceController{}
diff --git a/backend/app/controllers/distributed_trace_test.go b/backend/app/controllers/distributed_trace_test.go
index 1148b27e..047ad160 100644
--- a/backend/app/controllers/distributed_trace_test.go
+++ b/backend/app/controllers/distributed_trace_test.go
@@ -17,6 +17,29 @@ import (
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
)
+const (
+ cardTrace = "0102030405060708090a0b0c0d0e0f10"
+ cardBrowser = "a1a2a3a4a5a6a7a8a9aaabacadaeaf00"
+)
+
+func cardSpan(project uuid.UUID, trace, id, parent, name string, at time.Time) models.Span {
+ return models.Span{ProjectId: project, TraceId: trace, SpanId: id, ParentSpanId: parent, Name: name, StartTime: at, RecordedAt: at, Duration: time.Millisecond}
+}
+
+func openCard(t *testing.T, userID int, traceId, body string) (DistributedTraceResponse, int, *gin.Context) {
+ t.Helper()
+ c, recorder := newControllerTestContext(t, nil, userID, http.MethodPost, "/distributed-traces/"+traceId, body)
+ c.Params = gin.Params{{Key: "traceId", Value: traceId}}
+ DistributedTraceController.GetDistributedTrace(c)
+ var response DistributedTraceResponse
+ if recorder.Code == http.StatusOK {
+ if err := json.Unmarshal(recorder.Body.Bytes(), &response); err != nil {
+ t.Fatalf("%v: %s", err, recorder.Body.String())
+ }
+ }
+ return response, recorder.Code, c
+}
+
func TestGetDistributedTraceIncludesOwnedSpans(t *testing.T) {
setupSetupControllerDB(t)
tx, err := db.DB.Begin()
@@ -33,7 +56,7 @@ func TestGetDistributedTraceIncludesOwnedSpans(t *testing.T) {
}
return project.Id
}
- project1, project2 := createProject("API", orgID), createProject("Worker", orgID)
+ api, worker := createProject("API", orgID), createProject("Worker", orgID)
privateProject := createProject("Private", privateOrgID)
if err := tx.Commit(); err != nil {
t.Fatal(err)
@@ -41,66 +64,66 @@ func TestGetDistributedTraceIncludesOwnedSpans(t *testing.T) {
ctx := context.Background()
now := time.Now().UTC().Truncate(time.Millisecond)
- later := now.Add(36 * time.Hour)
- distributedID := uuid.New()
- occurrenceID := uuid.MustParse("00000000-0000-0000-1985-a7abed0024db")
- aiID, emptyID, orphanID := uuid.New(), uuid.New(), uuid.New()
- if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{{
- Id: occurrenceID, ProjectId: project1, Endpoint: "GET /jobs", RecordedAt: now, DistributedTraceId: &distributedID,
- }}); err != nil {
+ // GET /jobs (e1) -> queue publish (c1) -> worker task (k1) -> query (k2). The AI call (a1) and an empty task (k9) hang under e1.
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{
+ {Id: uuid.New(), ProjectId: api, Endpoint: "GET /jobs", RecordedAt: now, TraceId: cardTrace, SpanId: "e100000000000001", IsRoot: true},
+ {Id: uuid.New(), ProjectId: privateProject, Endpoint: "GET /private", RecordedAt: now, TraceId: cardTrace, SpanId: "e900000000000009", IsRoot: true},
+ }); err != nil {
t.Fatal(err)
}
if err := telemetry.TaskRepository.InsertAsync(ctx, []models.Task{
- {Id: occurrenceID, ProjectId: project2, TaskName: "worker", RecordedAt: later, DistributedTraceId: &distributedID},
- {Id: emptyID, ProjectId: project1, TaskName: "empty", RecordedAt: now, DistributedTraceId: &distributedID},
- {Id: occurrenceID, ProjectId: privateProject, TaskName: "private", RecordedAt: now, DistributedTraceId: &distributedID},
+ {Id: uuid.New(), ProjectId: worker, TaskName: "worker", RecordedAt: now, TraceId: cardTrace, SpanId: "7a00000000000001", ParentSpanId: "c100000000000001"},
+ {Id: uuid.New(), ProjectId: api, TaskName: "empty", RecordedAt: now, TraceId: cardTrace, SpanId: "7a00000000000009", ParentSpanId: "e100000000000001"},
}); err != nil {
t.Fatal(err)
}
if err := telemetry.AiTraceRepository.InsertAsync(ctx, []models.AiTrace{{
- Id: aiID, ProjectId: project1, TraceName: "chat", RecordedAt: now, DistributedTraceId: &distributedID,
+ Id: uuid.New(), ProjectId: api, TraceName: "chat", RecordedAt: now, TraceId: cardTrace, SpanId: "a100000000000001", ParentSpanId: "e100000000000001",
}}); err != nil {
t.Fatal(err)
}
if err := telemetry.ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{
- {Id: uuid.New(), ProjectId: project2, TraceId: &occurrenceID, ExceptionHash: "worker-error", RecordedAt: later, DistributedTraceId: &distributedID},
- {Id: uuid.New(), ProjectId: project2, TraceId: &orphanID, ExceptionHash: "orphan", RecordedAt: now, DistributedTraceId: &distributedID},
- {Id: uuid.New(), ProjectId: project1, ExceptionHash: "no-trace", RecordedAt: now, DistributedTraceId: &distributedID},
+ {Id: uuid.New(), ProjectId: worker, TraceId: cardTrace, SpanId: "7b00000000000001", ExceptionHash: "worker-error", RecordedAt: now},
+ {Id: uuid.New(), ProjectId: worker, TraceId: cardTrace, SpanId: "0f00000000000001", ExceptionHash: "orphan", RecordedAt: now},
+ {Id: uuid.New(), ProjectId: api, ExceptionHash: "no-trace", RecordedAt: now},
}); err != nil {
t.Fatal(err)
}
- makeSpan := func(project, owner uuid.UUID, name string, at time.Time) models.Span {
- return models.Span{Id: uuid.New(), ProjectId: project, TraceId: owner, Name: name, StartTime: at, RecordedAt: at, Duration: time.Millisecond}
- }
- parent := uuid.New()
- first := makeSpan(project1, occurrenceID, "first", now)
- first.ParentSpanId = &parent
+ first := cardSpan(api, cardTrace, "c100000000000001", "e100000000000001", "first", now)
first.Attributes = map[string]string{"db.system": "postgresql"}
if err := telemetry.SpanRepository.InsertAsync(ctx, []models.Span{
- makeSpan(project1, occurrenceID, "second", now.Add(time.Second)), first,
- makeSpan(project2, occurrenceID, "worker-child", later),
- makeSpan(project1, aiID, "ai-child", now),
- makeSpan(project2, orphanID, "orphan-child", now),
- makeSpan(privateProject, occurrenceID, "private-child", now),
- makeSpan(project1, occurrenceID, "outside-window", now.Add(-25*time.Hour)),
- makeSpan(project1, distributedID, "distributed-id-is-not-owner", now),
+ cardSpan(api, cardTrace, "e100000000000001", "", "GET /jobs", now),
+ cardSpan(api, cardTrace, "b200000000000001", "e100000000000001", "second", now.Add(time.Second)),
+ first,
+ cardSpan(api, cardTrace, "a100000000000001", "e100000000000001", "chat", now.Add(2*time.Second)),
+ cardSpan(api, cardTrace, "a200000000000001", "a100000000000001", "ai-child", now.Add(2*time.Second)),
+ cardSpan(api, cardTrace, "7a00000000000009", "e100000000000001", "empty", now.Add(3*time.Second)),
+ cardSpan(worker, cardTrace, "7a00000000000001", "c100000000000001", "worker", now),
+ cardSpan(worker, cardTrace, "7b00000000000001", "7a00000000000001", "worker-child", now),
+ cardSpan(worker, cardTrace, "0f00000000000001", "0e00000000000001", "orphan-span", now),
+ cardSpan(worker, cardTrace, "0f00000000000002", "0f00000000000001", "orphan-child", now),
+ cardSpan(privateProject, cardTrace, "e900000000000009", "", "private-child", now),
+ cardSpan(api, cardTrace, "b300000000000001", "e100000000000001", "outside-window", now.Add(-25*time.Hour)),
}); err != nil {
t.Fatal(err)
}
- for _, body := range []string{"{}", `{"recordedAt":"` + now.Format(time.RFC3339Nano) + `"}`} {
- c, recorder := newControllerTestContext(t, nil, userID, http.MethodPost, "/distributed-traces/"+distributedID.String(), body)
- c.Params = gin.Params{{Key: "distributedTraceId", Value: distributedID.String()}}
- DistributedTraceController.GetDistributedTrace(c)
- if recorder.Code != http.StatusOK {
- t.Fatalf("status %d: %s; errors: %v", recorder.Code, recorder.Body.String(), c.Errors)
+ names := func(spans []models.Span) string {
+ out := ""
+ for _, span := range spans {
+ out += span.Name + " "
}
- var response DistributedTraceResponse
- if err := json.Unmarshal(recorder.Body.Bytes(), &response); err != nil {
- t.Fatal(err)
+ return out
+ }
+ // The dashed spelling of the id is what older links carry.
+ dashed := uuid.MustParse(cardTrace).String()
+ for _, request := range [][2]string{{cardTrace, "{}"}, {dashed, `{"recordedAt":"` + now.Format(time.RFC3339Nano) + `"}`}} {
+ response, status, c := openCard(t, userID, request[0], request[1])
+ if status != http.StatusOK || response.TraceId != cardTrace {
+ t.Fatalf("status %d, trace %q, errors: %v", status, response.TraceId, c.Errors)
}
- if len(response.Nodes) != 6 {
- t.Fatalf("got %d nodes, want 6: %s", len(response.Nodes), recorder.Body.String())
+ if len(response.Nodes) != 5 {
+ t.Fatalf("got %d nodes, want the endpoint, two tasks, the AI trace and the orphan exception: %+v", len(response.Nodes), response.Nodes)
}
for _, node := range response.Nodes {
if node.Spans == nil {
@@ -108,51 +131,196 @@ func TestGetDistributedTraceIncludesOwnedSpans(t *testing.T) {
}
switch {
case node.Endpoint != nil:
- if len(node.Spans) != 2 || node.Spans[0].Name != "first" || node.Spans[1].Name != "second" {
- t.Fatalf("endpoint spans: %+v", node.Spans)
+ if got := names(node.Spans); got != "first second chat ai-child empty " {
+ t.Fatalf("the endpoint's subtree stays in its own project and inside the window: %q", got)
}
- if node.Exception != nil || node.Spans[0].ParentSpanId == nil || *node.Spans[0].ParentSpanId != parent || node.Spans[0].Attributes["db.system"] != "postgresql" {
- t.Fatalf("incorrect endpoint metadata: %+v", node)
+ if node.Exception != nil || node.ParentEntitySpanId != "" || node.SpanId != "e100000000000001" || node.Spans[0].Attributes["db.system"] != "postgresql" {
+ t.Fatalf("incorrect endpoint node: %+v", node)
}
- case node.Task != nil && node.Task.Id == occurrenceID:
- if node.ProjectId != project2 || len(node.Spans) != 1 || node.Spans[0].Name != "worker-child" || node.Exception == nil || node.Exception.ExceptionHash != "worker-error" {
+ case node.Task != nil && node.Task.TaskName == "worker":
+ if node.ProjectId != worker || names(node.Spans) != "worker-child " || node.ParentEntitySpanId != "e100000000000001" {
t.Fatalf("worker node: %+v", node)
}
+ if node.Exception == nil || node.Exception.ExceptionHash != "worker-error" {
+ t.Fatalf("an exception below the task's span belongs to the task: %+v", node.Exception)
+ }
+ case node.Task != nil:
+ if len(node.Spans) != 0 || node.ParentEntitySpanId != "e100000000000001" || node.Exception != nil {
+ t.Fatalf("empty task: %+v", node)
+ }
case node.AiTrace != nil:
- if len(node.Spans) != 1 || node.Spans[0].Name != "ai-child" {
- t.Fatalf("AI spans: %+v", node.Spans)
+ if names(node.Spans) != "ai-child " || node.ParentEntitySpanId != "e100000000000001" {
+ t.Fatalf("AI node: %+v", node)
}
- case node.Exception != nil && node.Exception.ExceptionHash == "orphan":
- if len(node.Spans) != 1 || node.Spans[0].Name != "orphan-child" {
- t.Fatalf("orphan spans: %+v", node.Spans)
+ case node.TraceType == "exception":
+ if node.Exception.ExceptionHash != "orphan" || names(node.Spans) != "orphan-child " || node.ProjectName != "Worker" {
+ t.Fatalf("an exception with nothing promoted above it stands alone with the spans under its own: %+v", node)
}
default:
- if len(node.Spans) != 0 {
- t.Fatalf("expected no spans: %+v", node)
- }
+ t.Fatalf("unexpected node: %+v", node)
}
}
}
- if _, err := db.TelemetryDB.Exec("DROP TABLE spans"); err != nil {
+ if _, status, _ := openCard(t, userID, "not-a-trace-id", "{}"); status != http.StatusBadRequest {
+ t.Fatalf("an id that is not 32 hex characters: %d", status)
+ }
+ if _, err := db.TelemetryDB.Exec("DROP TABLE spans_v2"); err != nil {
t.Fatal(err)
}
- for _, id := range []uuid.UUID{uuid.New(), distributedID} {
- c, recorder := newControllerTestContext(t, nil, userID, http.MethodPost, "/distributed-traces/"+id.String(), "{}")
- c.Params = gin.Params{{Key: "distributedTraceId", Value: id.String()}}
- DistributedTraceController.GetDistributedTrace(c)
- if id == distributedID {
- if recorder.Code != http.StatusInternalServerError || len(c.Errors) == 0 {
- t.Fatalf("span lookup failure must return 500, got %d", recorder.Code)
+ if _, status, c := openCard(t, userID, cardTrace, "{}"); status != http.StatusInternalServerError || len(c.Errors) == 0 {
+ t.Fatalf("span lookup failure must return 500, got %d", status)
+ }
+ if response, status, _ := openCard(t, userID, "ffffffffffffffffffffffffffffffff", "{}"); status != http.StatusOK || response.Nodes == nil || len(response.Nodes) != 0 {
+ t.Fatalf("a trace nobody recorded skips the span reads and returns []: %d %+v", status, response)
+ }
+}
+
+func TestDistributedTraceNestsAcrossAnUnpromotedHopInAnotherProject(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ userID, orgID := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ createProject := func(name string) uuid.UUID {
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, name, "opentelemetry", orgID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return project.Id
+ }
+ gateway, inventory, warehouse := createProject("gateway"), createProject("inventory-grpc"), createProject("warehouse")
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Millisecond)
+ // The gRPC hop reports to its own project and is never promoted, so no entity's subtree holds it.
+ if err := telemetry.SpanRepository.InsertAsync(ctx, []models.Span{
+ cardSpan(gateway, cardTrace, "0101010101010101", "", "GET /api/stock", now), cardSpan(gateway, cardTrace, "0202020202020202", "0101010101010101", "inventory.Stock/Get", now),
+ cardSpan(inventory, cardTrace, "0303030303030303", "0202020202020202", "inventory.Stock/Get", now), cardSpan(inventory, cardTrace, "0404040404040404", "0303030303030303", "GET", now),
+ cardSpan(warehouse, cardTrace, "0505050505050505", "0404040404040404", "GET /shelves", now),
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{
+ {Id: uuid.New(), ProjectId: gateway, Endpoint: "GET /api/stock", RecordedAt: now, TraceId: cardTrace, SpanId: "0101010101010101", IsRoot: true},
+ {Id: uuid.New(), ProjectId: warehouse, Endpoint: "GET /shelves", RecordedAt: now, TraceId: cardTrace, SpanId: "0505050505050505", ParentSpanId: "0404040404040404"},
+ }); err != nil {
+ t.Fatal(err)
+ }
+
+ response, status, _ := openCard(t, userID, cardTrace, `{"recordedAt":"`+now.Format(time.RFC3339Nano)+`"}`)
+ if status != http.StatusOK || len(response.Nodes) != 2 {
+ t.Fatalf("status %d: %+v", status, response)
+ }
+ for _, node := range response.Nodes {
+ switch node.Endpoint.Endpoint {
+ case "GET /shelves":
+ if node.ParentEntitySpanId != "0101010101010101" {
+ t.Fatalf("warehouse must nest under gateway through the unpromoted gRPC hop, got parent %q", node.ParentEntitySpanId)
}
- } else {
- var response DistributedTraceResponse
- if err := json.Unmarshal(recorder.Body.Bytes(), &response); err != nil {
- t.Fatal(err)
+ case "GET /api/stock":
+ if node.ParentEntitySpanId != "" {
+ t.Fatalf("the first entity of the trace has nothing above it, got %q", node.ParentEntitySpanId)
}
- if recorder.Code != http.StatusOK || response.Nodes == nil || len(response.Nodes) != 0 {
- t.Fatalf("empty trace should skip span queries and return [], got %s", recorder.Body.String())
+ }
+ }
+}
+
+// The browser SDK sends its own id in `traceway-trace-id`. Only the gateway receives that header, so only its rows
+// link to the browser's trace. Everything keeps the trace id it arrived with, and the card follows the link both ways.
+func TestDistributedTraceIsWholeUnderEitherOfItsIds(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ userID, orgID := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ createProject := func(name, framework string) uuid.UUID {
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, name, framework, orgID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return project.Id
+ }
+ web, gateway, payments, warehouse := createProject("web", "react"), createProject("gateway", "opentelemetry"), createProject("payments", "opentelemetry"), createProject("warehouse", "opentelemetry")
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Millisecond)
+ if err := telemetry.SpanRepository.InsertAsync(ctx, []models.Span{
+ cardSpan(gateway, cardTrace, "0101010101010101", "", "POST /one", now), cardSpan(gateway, cardTrace, "0202020202020202", "0101010101010101", "POST", now),
+ cardSpan(payments, cardTrace, "0303030303030303", "0202020202020202", "POST /two", now), cardSpan(payments, cardTrace, "0404040404040404", "0303030303030303", "POST", now),
+ cardSpan(warehouse, cardTrace, "0505050505050505", "0404040404040404", "POST /three", now),
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{
+ {Id: uuid.New(), ProjectId: gateway, Endpoint: "POST /one", RecordedAt: now, TraceId: cardTrace, SpanId: "0101010101010101", Attributes: map[string]string{"traceway.distributed_trace_id": cardBrowser}, IsRoot: true},
+ {Id: uuid.New(), ProjectId: payments, Endpoint: "POST /two", RecordedAt: now, TraceId: cardTrace, SpanId: "0303030303030303", ParentSpanId: "0202020202020202"},
+ {Id: uuid.New(), ProjectId: warehouse, Endpoint: "POST /three", RecordedAt: now, TraceId: cardTrace, SpanId: "0505050505050505", ParentSpanId: "0404040404040404"},
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{
+ {Id: uuid.New(), ProjectId: web, ExceptionHash: "browser-error", RecordedAt: now, TraceId: cardTrace},
+ }); err != nil {
+ t.Fatal(err)
+ }
+
+ for opened, id := range map[string]string{"the shared W3C trace ID": cardTrace} {
+ response, status, _ := openCard(t, userID, id, `{"recordedAt":"`+now.Format(time.RFC3339Nano)+`"}`)
+ if status != http.StatusOK {
+ t.Fatalf("%s: status %d", opened, status)
+ }
+ parents, browserErrors := map[string]string{}, 0
+ for _, node := range response.Nodes {
+ if node.Endpoint != nil {
+ parents[node.Endpoint.Endpoint] = node.ParentEntitySpanId
+ } else if node.Exception != nil && node.Exception.ExceptionHash == "browser-error" {
+ browserErrors++
}
}
+ if len(response.Nodes) != 4 || browserErrors != 1 || len(parents) != 3 || parents["POST /two"] != "0101010101010101" || parents["POST /three"] != "0303030303030303" || parents["POST /one"] != "" {
+ t.Fatalf("opened by %s: want all three endpoints nested plus the browser error once, got %d nodes, parents %v, browser errors %d", opened, len(response.Nodes), parents, browserErrors)
+ }
+ }
+}
+
+func TestTraceEntityLookupKeepsMigratedIdentitiesAndBoundsMissingTime(t *testing.T) {
+ setupSetupControllerDB(t)
+ ctx := context.Background()
+ project, id := uuid.New(), uuid.New()
+ now := time.Now().UTC()
+ rows := []models.Endpoint{
+ {Id: id, ProjectId: project, TraceId: cardTrace, Attributes: map[string]string{"traceway.distributed_trace_id": cardBrowser}, SpanId: "0102030405060708", RecordedAt: now},
+ {Id: id, ProjectId: project, TraceId: cardBrowser, SpanId: "0102030405060708", RecordedAt: now},
+ {Id: uuid.New(), ProjectId: project, TraceId: cardTrace, SpanId: "0102030405060709", RecordedAt: now.Add(-7 * 24 * time.Hour)},
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, rows); err != nil {
+ t.Fatal(err)
+ }
+ found, err := findTraceEntities(ctx, cardBrowser, []uuid.UUID{project}, nil)
+ if err != nil || len(found.endpoints) != 1 {
+ t.Fatalf("unanchored lookup lost an identity or scanned old history: %+v, %v", found, err)
+ }
+ old := rows[2].RecordedAt
+ found, err = findTraceEntities(ctx, cardTrace, []uuid.UUID{project}, &old)
+ if err != nil || len(found.endpoints) != 1 || found.endpoints[0].Id != rows[2].Id {
+ t.Fatalf("historical anchor: %+v, %v", found, err)
+ }
+}
+
+func TestDistributedTraceRejectsMalformedAnchor(t *testing.T) {
+ _, code, _ := openCard(t, 1, cardTrace, `{"recordedAt":"invalid"}`)
+ if code != http.StatusBadRequest {
+ t.Fatalf("invalid date must not silently become an unanchored scan: %d", code)
}
}
diff --git a/backend/app/controllers/endpoint_detail.controller.go b/backend/app/controllers/endpoint_detail.controller.go
index caeac17a..e9a3d2db 100644
--- a/backend/app/controllers/endpoint_detail.controller.go
+++ b/backend/app/controllers/endpoint_detail.controller.go
@@ -7,6 +7,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
@@ -34,11 +35,12 @@ type EndpointMessageInfo struct {
}
type EndpointDetailResponse struct {
- Endpoint *models.Endpoint `json:"endpoint"`
- Spans []models.Span `json:"spans"`
- HasSpans bool `json:"hasSpans"`
- Exception *EndpointExceptionInfo `json:"exception,omitempty"`
- Messages []EndpointMessageInfo `json:"messages"`
+ Endpoint *models.Endpoint `json:"endpoint"`
+ SpanGraphStatus *models.SpanGraphStatus `json:"spanGraphStatus"`
+ Spans []models.Span `json:"spans"`
+ HasSpans bool `json:"hasSpans"`
+ Exception *EndpointExceptionInfo `json:"exception,omitempty"`
+ Messages []EndpointMessageInfo `json:"messages"`
}
func (t endpointDetailController) GetEndpointDetail(c *gin.Context) {
@@ -77,7 +79,7 @@ func (t endpointDetailController) GetEndpointDetail(c *gin.Context) {
// Get spans (flat list ordered by start_time)
span = traceway.StartSpan(c, "loading spans")
- spans, err := telemetry.SpanRepository.FindByTraceId(c, projectId, endpointId, &recordedAt)
+ graph, err := telemetry.SpanRepository.FindGraph(c, shared.NewSpanLookup(projectId, endpoint.TraceId, endpoint.SpanId, recordedAt))
span.End()
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading spans: %w", err))
@@ -89,7 +91,7 @@ func (t endpointDetailController) GetEndpointDetail(c *gin.Context) {
var messages []EndpointMessageInfo
span = traceway.StartSpan(c, "loading exceptions")
- allExceptions, err := telemetry.ExceptionStackTraceRepository.FindAllByTraceId(c, projectId, endpointId, &recordedAt)
+ allExceptions, err := findOccurrenceExceptions(c, projectId, endpoint.TraceId, endpoint.SpanId, &recordedAt, graph.Spans)
span.End()
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading all exceptions: %w", err))
@@ -121,11 +123,12 @@ func (t endpointDetailController) GetEndpointDetail(c *gin.Context) {
}
c.JSON(http.StatusOK, EndpointDetailResponse{
- Endpoint: endpoint,
- Spans: spans,
- HasSpans: len(spans) > 0,
- Exception: exceptionInfo,
- Messages: messages,
+ Endpoint: endpoint,
+ SpanGraphStatus: &graph.Status,
+ Spans: graph.Spans,
+ HasSpans: len(graph.Spans) > 0,
+ Exception: exceptionInfo,
+ Messages: messages,
})
}
diff --git a/backend/app/controllers/exception_stack_trace.controller.go b/backend/app/controllers/exception_stack_trace.controller.go
index 5f08369b..e4d3d1e1 100644
--- a/backend/app/controllers/exception_stack_trace.controller.go
+++ b/backend/app/controllers/exception_stack_trace.controller.go
@@ -54,6 +54,8 @@ type ExceptionDetailResponse struct {
Occurrences []models.ExceptionStackTrace `json:"occurrences"`
Pagination Pagination `json:"pagination"`
SessionRecording json.RawMessage `json:"sessionRecording,omitempty"`
+ // RelatedEntity is the endpoint, task or AI trace the newest occurrence happened in.
+ RelatedEntity *telemetry.ExceptionOwner `json:"relatedEntity,omitempty"`
// SessionId is set when the exception is linked to a parent session row
// (always-on recording). The frontend swaps SessionRecording for a fetch
// against `/sessions/:sessionId/recording`, which spans every segment.
@@ -189,6 +191,8 @@ func (e exceptionStackTraceController) FindByHash(c *gin.Context) {
}
if len(occurrences) > 0 {
+ response.RelatedEntity = findRelatedEntity(c, occurrences[0])
+
// The per-exception 10 s clip and the parent session are independent
// attachments — surface both whenever they exist. The dashboard plays
// the clip inline and renders a link to the full session.
@@ -308,6 +312,16 @@ func (e exceptionStackTraceController) UnarchiveExceptions(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"unarchived": len(request.Hashes)})
}
+// The related entity is a convenience on the page, so a failed lookup is reported and the page still loads.
+func findRelatedEntity(ctx context.Context, exception models.ExceptionStackTrace) *telemetry.ExceptionOwner {
+ owner, err := telemetry.FindExceptionOwner(ctx, exception)
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to find the related entity of exception %s: %w", exception.Id, err))
+ return nil
+ }
+ return owner
+}
+
type exceptionByIdRequest struct {
RecordedAt *time.Time `json:"recordedAt"`
}
@@ -345,6 +359,9 @@ func (e exceptionStackTraceController) FindById(c *gin.Context) {
}
response := gin.H{"exception": exception}
+ if related := findRelatedEntity(c, *exception); related != nil {
+ response["relatedEntity"] = related
+ }
filePath, err := telemetry.SessionRecordingRepository.FindByExceptionId(c, projectId, exceptionId)
if err == nil && filePath != "" {
diff --git a/backend/app/controllers/health_deep.go b/backend/app/controllers/health_deep.go
index dc5d42ed..d1398ca1 100644
--- a/backend/app/controllers/health_deep.go
+++ b/backend/app/controllers/health_deep.go
@@ -35,14 +35,16 @@ type HealthDeepResponse struct {
MemoryUsageBytes int64 `json:"memoryUsageBytes,omitempty"`
MemoryTotalBytes int64 `json:"memoryTotalBytes,omitempty"`
- TelemetryBackend string `json:"telemetryBackend"`
- DroppedRows map[string]uint64 `json:"droppedRows"`
- DroppedRowsTotal uint64 `json:"droppedRowsTotal"`
- InsertFailures uint64 `json:"insertFailures"`
- IngestRejected uint64 `json:"ingestRejected"`
- Engine *db.TelemetryEngineStats `json:"engine,omitempty"`
- Outbox *outbox.HealthStats `json:"outbox,omitempty"`
- Synthetics *synthetics.HealthStats `json:"synthetics,omitempty"`
+ TelemetryBackend string `json:"telemetryBackend"`
+ DroppedRows map[string]uint64 `json:"droppedRows"`
+ DroppedRowsTotal uint64 `json:"droppedRowsTotal"`
+ InsertFailures uint64 `json:"insertFailures"`
+ IngestRejected uint64 `json:"ingestRejected"`
+ // Spans whose source start was missing or unrepresentable, so their partition time is the ingest time.
+ SpanPartitionTimeFallbacks uint64 `json:"spanPartitionTimeFallbacks"`
+ Engine *db.TelemetryEngineStats `json:"engine,omitempty"`
+ Outbox *outbox.HealthStats `json:"outbox,omitempty"`
+ Synthetics *synthetics.HealthStats `json:"synthetics,omitempty"`
}
// 503 means the configured telemetry backend is down. On the embedded
@@ -57,6 +59,7 @@ func (h healthDeepController) Get(c *gin.Context) {
resp.DroppedRowsTotal = droppedTotal
resp.InsertFailures = insertFailures
resp.IngestRejected = db.GetIngestRejects()
+ resp.SpanPartitionTimeFallbacks = db.GetSpanPartitionTimeFallbacks()
if engine, ok := db.GetTelemetryEngineStats(c.Request.Context()); ok {
resp.Engine = &engine
}
diff --git a/backend/app/controllers/log.controller.go b/backend/app/controllers/log.controller.go
index dfdad7ec..22c85978 100644
--- a/backend/app/controllers/log.controller.go
+++ b/backend/app/controllers/log.controller.go
@@ -1,18 +1,18 @@
package controllers
import (
- "context"
- "encoding/hex"
+ "encoding/json"
+ "fmt"
"net/http"
"time"
"github.com/gin-gonic/gin"
- "github.com/google/uuid"
traceway "go.tracewayapp.com"
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
)
type logController struct{}
@@ -33,22 +33,21 @@ type LogPaginationParams struct {
}
type LogSearchRequest struct {
- FromDate time.Time `json:"fromDate"`
- ToDate time.Time `json:"toDate"`
- OrderBy string `json:"orderBy"`
- SortDirection string `json:"sortDirection"`
- Search string `json:"search"`
- SearchType string `json:"searchType"`
- MinSeverity uint8 `json:"minSeverity"`
- ServiceName string `json:"serviceName"`
- TraceId string `json:"traceId"`
- SpanId string `json:"spanId"`
- ScopeName string `json:"scopeName"`
- Body string `json:"body"`
- DistributedTraceId string `json:"distributedTraceId"`
- ExcludeTraceId string `json:"excludeTraceId"`
- AttributeFilters []LogAttributeFilterRequest `json:"attributeFilters"`
- Pagination LogPaginationParams `json:"pagination"`
+ FromDate time.Time `json:"fromDate"`
+ ToDate time.Time `json:"toDate"`
+ OrderBy string `json:"orderBy"`
+ SortDirection string `json:"sortDirection"`
+ Search string `json:"search"`
+ SearchType string `json:"searchType"`
+ MinSeverity uint8 `json:"minSeverity"`
+ ServiceName string `json:"serviceName"`
+ TraceId string `json:"traceId"`
+ WholeTrace bool `json:"wholeTrace"`
+ SpanId string `json:"spanId"`
+ ScopeName string `json:"scopeName"`
+ Body string `json:"body"`
+ AttributeFilters []LogAttributeFilterRequest `json:"attributeFilters"`
+ Pagination LogPaginationParams `json:"pagination"`
}
// Max time range allowed for body search without any other selector. Keeps a
@@ -71,6 +70,14 @@ func (l logController) List(c *gin.Context) {
return
}
+ if request.TraceId != "" {
+ request.TraceId = shared.NormalizeTraceId(request.TraceId)
+ if !validTraceHex(request.TraceId) {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "invalid traceId"})
+ return
+ }
+ }
+
// Loki-style gate: a body substring search without any selector can scan
// the entire body column across the requested range. Require at least one
// selector (service / severity / trace / attribute) OR a short time range.
@@ -85,7 +92,6 @@ func (l logController) List(c *gin.Context) {
request.SpanId != "" ||
request.ScopeName != "" ||
request.Body != "" ||
- request.DistributedTraceId != "" ||
len(request.AttributeFilters) > 0
rangeTooWide := request.ToDate.Sub(request.FromDate) > bodySearchUnscopedMaxRange
if !hasSelector && rangeTooWide {
@@ -128,32 +134,19 @@ func (l logController) List(c *gin.Context) {
Page: request.Pagination.Page,
PageSize: request.Pagination.PageSize,
}
-
- if request.DistributedTraceId != "" {
- dtid, err := uuid.Parse(request.DistributedTraceId)
- if err != nil {
- c.JSON(http.StatusBadRequest, gin.H{"error": "invalid distributedTraceId"})
- return
- }
- traceIds, err := l.resolveDistributedTraceIds(c, dtid, projectId, request.ExcludeTraceId, request.FromDate)
+ if request.WholeTrace && request.TraceId == "" {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "wholeTrace requires a traceId"})
+ return
+ }
+ if request.WholeTrace {
+ projects, err := organizationProjects(c, projectId)
if err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error resolving distributed trace: %w", err))
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error listing trace projects: %w", err))
return
}
- if len(traceIds) == 0 {
- c.JSON(http.StatusOK, PaginatedResponse[models.LogRecord]{
- Data: []models.LogRecord{},
- Pagination: Pagination{
- Page: request.Pagination.Page,
- PageSize: request.Pagination.PageSize,
- Total: 0,
- TotalPages: 0,
- },
- })
- return
+ for _, project := range projects {
+ params.ProjectIds = append(params.ProjectIds, project.Id)
}
- params.TraceIds = traceIds
- params.TraceId = ""
}
span := traceway.StartSpan(c, "loading logs")
@@ -175,48 +168,16 @@ func (l logController) List(c *gin.Context) {
})
}
-func (l logController) resolveDistributedTraceIds(ctx context.Context, dtid uuid.UUID, projectId uuid.UUID, excludeTraceHex string, recordedAt time.Time) ([]string, error) {
- projectIds := []uuid.UUID{projectId}
-
- var recordedAtHint *time.Time
- if !recordedAt.IsZero() {
- recordedAtHint = &recordedAt
+func (r *LogSearchRequest) UnmarshalJSON(data []byte) error {
+ type request LogSearchRequest
+ var fields map[string]json.RawMessage
+ if err := json.Unmarshal(data, &fields); err != nil {
+ return err
}
-
- endpoints, err := telemetry.EndpointRepository.FindByDistributedTraceId(ctx, dtid, projectIds, recordedAtHint)
- if err != nil {
- return nil, err
- }
- tasks, err := telemetry.TaskRepository.FindByDistributedTraceId(ctx, dtid, projectIds, recordedAtHint)
- if err != nil {
- return nil, err
- }
- aiTraces, err := telemetry.AiTraceRepository.FindByDistributedTraceId(ctx, dtid, projectIds, recordedAtHint)
- if err != nil {
- return nil, err
- }
-
- seen := make(map[string]struct{}, len(endpoints)+len(tasks)+len(aiTraces))
- result := make([]string, 0, len(endpoints)+len(tasks)+len(aiTraces))
- add := func(id uuid.UUID) {
- h := hex.EncodeToString(id[:])
- if _, ok := seen[h]; ok {
- return
- }
- if h == excludeTraceHex {
- return
+ for _, key := range []string{"distributedTraceId", "excludeTraceId"} {
+ if _, exists := fields[key]; exists {
+ return fmt.Errorf("%s is no longer supported; use traceId to select one trace", key)
}
- seen[h] = struct{}{}
- result = append(result, h)
- }
- for _, ep := range endpoints {
- add(ep.Id)
- }
- for _, t := range tasks {
- add(t.Id)
- }
- for _, a := range aiTraces {
- add(a.Id)
}
- return result, nil
+ return json.Unmarshal(data, (*request)(r))
}
diff --git a/backend/app/controllers/log_compatibility_test.go b/backend/app/controllers/log_compatibility_test.go
new file mode 100644
index 00000000..f23b1233
--- /dev/null
+++ b/backend/app/controllers/log_compatibility_test.go
@@ -0,0 +1,147 @@
+//go:build !telemetry_ch && !telemetry_duckdb && !transactional_pg
+
+package controllers
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func TestLogTraceFiltersStayScoped(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ ctx := context.Background()
+ project, other, browser, server := uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ hexID := func(id uuid.UUID) string { return strings.ReplaceAll(id.String(), "-", "") }
+ at := time.Now().UTC().Truncate(time.Second)
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{{Id: uuid.New(), ProjectId: project, TraceId: hexID(server), Attributes: map[string]string{"traceway.distributed_trace_id": hexID(browser)}, SpanId: "0102030405060708", RecordedAt: at}}); err != nil {
+ t.Fatal(err)
+ }
+ logs := []models.LogRecord{
+ {Id: uuid.New(), ProjectId: project, TraceId: hexID(browser), Timestamp: at, Body: "browser"},
+ {Id: uuid.New(), ProjectId: project, TraceId: hexID(server), Timestamp: at, Body: "server"},
+ {Id: uuid.New(), ProjectId: project, TraceId: hexID(uuid.New()), Timestamp: at, Body: "unrelated"},
+ {Id: uuid.New(), ProjectId: other, TraceId: hexID(server), Timestamp: at, Body: "other project"},
+ }
+ if err := telemetry.LogRecordRepository.InsertAsync(ctx, logs); err != nil {
+ t.Fatal(err)
+ }
+ for _, test := range []struct {
+ trace, span, retired string
+ status, want int
+ }{
+ {browser.String(), "", "", 200, 1}, {hexID(server), "", "", 200, 1}, {"invalid", "", "", 400, 0}, {strings.Repeat("0", 32), "", "", 400, 0}, {"", "", "distributedTraceId", 400, 0}, {"", "", "excludeTraceId", 400, 0},
+ } {
+ payload := map[string]any{"fromDate": at.Add(-time.Hour), "toDate": at.Add(time.Hour), "traceId": test.trace, "spanId": test.span, "pagination": map[string]int{"page": 1, "pageSize": 100}}
+ if test.retired != "" {
+ payload[test.retired] = hexID(browser)
+ }
+ body, err := json.Marshal(payload)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rec := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(rec)
+ c.Request = httptest.NewRequest("POST", "/api/logs", bytes.NewReader(body))
+ c.Request.Header.Set("Content-Type", "application/json")
+ c.Set(middleware.ProjectIdContextKey, project)
+ LogController.List(c)
+ if rec.Code != test.status {
+ t.Fatalf("got %d, want %d: %s", rec.Code, test.status, rec.Body.String())
+ }
+ if test.status == 200 {
+ var result PaginatedResponse[models.LogRecord]
+ if err := json.Unmarshal(rec.Body.Bytes(), &result); err != nil {
+ t.Fatal(err)
+ }
+ if len(result.Data) != test.want || result.Pagination.Total != int64(test.want) {
+ t.Fatalf("trace filter widened or lost results: %+v", result)
+ }
+ for _, log := range result.Data {
+ if log.ProjectId != project || log.Body == "unrelated" {
+ t.Fatal("trace filter escaped its scope")
+ }
+ }
+ }
+ }
+}
+
+func TestWholeTraceLogsStayInsideSelectedOrganization(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ user, org := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ _, otherOrg := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ _, strangerOrg := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ if _, err := transactional.OrganizationRepository.AddUser(tx, otherOrg, user, "user"); err != nil {
+ t.Fatal(err)
+ }
+ var projects []*models.Project
+ for i, organization := range []int{org, org, otherOrg, strangerOrg} {
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, fmt.Sprintf("service-%d", i), "opentelemetry", organization)
+ if err != nil {
+ t.Fatal(err)
+ }
+ projects = append(projects, project)
+ }
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+ const trace = "0123456789abcdef0123456789abcdef"
+ at := time.Now().UTC().Truncate(time.Second)
+ for _, project := range projects {
+ if err := telemetry.LogRecordRepository.InsertAsync(context.Background(), []models.LogRecord{{
+ Id: uuid.New(), ProjectId: project.Id, TraceId: trace, Timestamp: at, Body: project.Name,
+ }}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ for _, selected := range projects[:2] {
+ for _, whole := range []bool{false, true} {
+
+ body := fmt.Sprintf(`{"traceId":%q,"wholeTrace":%t,"fromDate":%q,"toDate":%q,"pagination":{"page":1,"pageSize":100}}`, trace, whole, at.Add(-time.Hour).Format(time.RFC3339Nano), at.Add(time.Hour).Format(time.RFC3339Nano))
+ c, response := newControllerTestContext(t, nil, user, "POST", "/logs", body)
+ c.Set(middleware.ProjectIdContextKey, selected.Id)
+ LogController.List(c)
+ var result PaginatedResponse[models.LogRecord]
+ if err := json.Unmarshal(response.Body.Bytes(), &result); err != nil || response.Code != 200 {
+ t.Fatalf("%d %s %v", response.Code, response.Body.String(), err)
+ }
+ want := 1
+ if whole {
+ want = 2
+ }
+ if len(result.Data) != want || result.Pagination.Total != int64(want) {
+ t.Fatalf("whole=%t: %+v", whole, result)
+ }
+ for _, log := range result.Data {
+ if log.ProjectId != projects[0].Id && log.ProjectId != projects[1].Id {
+ t.Fatal("whole trace leaked another organization's log")
+ }
+ }
+ }
+ }
+ c, response := newControllerTestContext(t, nil, user, "POST", "/logs", `{"wholeTrace":true,"pagination":{"page":1,"pageSize":100}}`)
+ c.Set(middleware.ProjectIdContextKey, projects[0].Id)
+ LogController.List(c)
+ if response.Code != 400 {
+ t.Fatalf("wholeTrace without traceId must be rejected: %d", response.Code)
+ }
+}
diff --git a/backend/app/controllers/otel_span.controller.go b/backend/app/controllers/otel_span.controller.go
new file mode 100644
index 00000000..95914e17
--- /dev/null
+++ b/backend/app/controllers/otel_span.controller.go
@@ -0,0 +1,53 @@
+package controllers
+
+import (
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ traceway "go.tracewayapp.com"
+ "google.golang.org/protobuf/proto"
+)
+
+func GetOtelSpan(c *gin.Context) {
+ project, err := middleware.GetProjectId(c)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ traceID, spanID := strings.ToLower(c.Param("traceId")), strings.ToLower(c.Param("spanId"))
+ if !validTraceHex(traceID) || !validSpanHex(spanID) {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid OTLP trace/span ID"})
+ return
+ }
+ at, err := time.Parse(time.RFC3339Nano, c.Query("at"))
+ if err != nil {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "An RFC 3339 `at` time is required: span reads are held to 24 hours either side of it"})
+ return
+ }
+ payload, err := telemetry.OtelSpanRepository.FindOTLP(c, project, traceID, spanID, at)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error loading stored OTLP span: %w", err))
+ return
+ }
+ if len(payload) == 0 {
+ c.JSON(http.StatusNotFound, gin.H{"error": "Span not found"})
+ return
+ }
+ var resource tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &resource); err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("stored OTLP span payload is not a valid ResourceSpans: %w", err))
+ return
+ }
+ response, err := proto.Marshal(&coltracepb.ExportTraceServiceRequest{ResourceSpans: []*tracepb.ResourceSpans{&resource}})
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error encoding OTLP span response: %w", err))
+ return
+ }
+ c.Data(http.StatusOK, "application/x-protobuf", response)
+}
diff --git a/backend/app/controllers/otelcontrollers/android_converter_test.go b/backend/app/controllers/otelcontrollers/android_converter_test.go
index 05573195..25d92922 100644
--- a/backend/app/controllers/otelcontrollers/android_converter_test.go
+++ b/backend/app/controllers/otelcontrollers/android_converter_test.go
@@ -73,7 +73,7 @@ func TestConvertTraces_AndroidR8Retraces(t *testing.T) {
})
req := androidExceptionRequest(string(androidConvFixture(t, "r8", "obfuscated.txt")), proguardUuid, nil)
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), tokenProject(projectId), projectId, req)
+ exceptions := convertTraces(context.Background(), tokenProject(projectId), projectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
@@ -106,7 +106,7 @@ func TestConvertTraces_AndroidStructuredArrays(t *testing.T) {
intArrayKV("exception.structured_stacktrace.lines", 2),
}
req := androidExceptionRequest("", proguardUuid, extra)
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), tokenProject(projectId), projectId, req)
+ exceptions := convertTraces(context.Background(), tokenProject(projectId), projectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
diff --git a/backend/app/controllers/otelcontrollers/canonical_ingest_test.go b/backend/app/controllers/otelcontrollers/canonical_ingest_test.go
new file mode 100644
index 00000000..9cf7bb83
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/canonical_ingest_test.go
@@ -0,0 +1,84 @@
+//go:build !telemetry_ch && !telemetry_duckdb && !transactional_pg
+
+package otelcontrollers
+
+import (
+ "bytes"
+ "context"
+ "encoding/hex"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestExportTracesPersistsLateParentGraphByDefault(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ previousConfig := config.Config
+ config.Config = &config.Cfg{}
+ t.Cleanup(func() { config.Config = previousConfig })
+ gin.SetMode(gin.TestMode)
+ trace := uuid.New()
+ now := uint64(time.Now().UnixNano())
+ parent := &tracepb.Span{TraceId: trace[:], SpanId: []byte{1, 1, 1, 1, 1, 1, 1, 1}, Name: "worker", Kind: tracepb.Span_SPAN_KIND_CONSUMER, StartTimeUnixNano: now, EndTimeUnixNano: now + 1000}
+ child := &tracepb.Span{TraceId: trace[:], SpanId: []byte{2, 2, 2, 2, 2, 2, 2, 2}, ParentSpanId: parent.SpanId, Name: "query", Kind: tracepb.Span_SPAN_KIND_INTERNAL, StartTimeUnixNano: now, EndTimeUnixNano: now + 500}
+ child.Attributes = []*commonpb.KeyValue{strKV("db.system", "postgresql")}
+ invalid := &tracepb.Span{TraceId: []byte{1}, SpanId: child.SpanId}
+ for i, request := range []*coltracepb.ExportTraceServiceRequest{spanRequest(child, invalid), spanRequest(parent)} {
+ payload, err := proto.Marshal(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ recorder := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(recorder)
+ c.Request = httptest.NewRequest(http.MethodPost, "/api/otel/v1/traces", bytes.NewReader(payload))
+ c.Request.Header.Set("Content-Type", "application/x-protobuf")
+ c.Set(middleware.ProjectIdContextKey, testProjectId)
+ OtelController.ExportTraces(c)
+ if recorder.Code != http.StatusOK || len(c.Errors) > 0 {
+ t.Fatalf("export failed: %d %v", recorder.Code, c.Errors)
+ }
+ var response coltracepb.ExportTraceServiceResponse
+ if err := proto.Unmarshal(recorder.Body.Bytes(), &response); err != nil {
+ t.Fatal(err)
+ }
+ if i == 0 && response.GetPartialSuccess().GetRejectedSpans() != 1 {
+ t.Fatalf("missing invalid ID rejection: %v", &response)
+ }
+ }
+ id := otelOccurrenceID(testProjectId, parent)
+ task, err := telemetry.TaskRepository.FindById(context.Background(), testProjectId, id, nil)
+ if err != nil || task == nil {
+ t.Fatalf("missing task projection: %v", err)
+ }
+ if task.TraceId != hex.EncodeToString(trace[:]) || task.SpanId != "0101010101010101" || task.ParentSpanId != "" || !task.IsRoot {
+ t.Fatalf("the task carries its span's ids: %+v", task)
+ }
+ graph, err := telemetry.SpanRepository.FindGraph(context.Background(), shared.NewSpanLookup(testProjectId, task.TraceId, task.SpanId, task.RecordedAt))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if spans := graph.Spans; len(spans) != 1 || spans[0].Name != "query" {
+ t.Fatalf("lost separately exported child: %+v", spans)
+ }
+ if graph.Spans[0].Attributes["db.system"] != "postgresql" {
+ t.Fatalf("lost source span attributes: %v", graph.Spans[0].Attributes)
+ }
+ var stored int
+ if err := db.TelemetryDB.QueryRow("SELECT COUNT(*) FROM spans_v2").Scan(&stored); err != nil || stored != 2 {
+ t.Fatalf("default ingestion did not retain both source spans: %d %v", stored, err)
+ }
+}
diff --git a/backend/app/controllers/otelcontrollers/canonical_spans.go b/backend/app/controllers/otelcontrollers/canonical_spans.go
new file mode 100644
index 00000000..bfd32634
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/canonical_spans.go
@@ -0,0 +1,22 @@
+package otelcontrollers
+
+import (
+ "github.com/google/uuid"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+)
+
+func otelOccurrenceID(projectId uuid.UUID, span *tracepb.Span) uuid.UUID {
+ var identity [24]byte
+ copy(identity[:16], span.TraceId)
+ copy(identity[16:], span.SpanId)
+ return uuid.NewSHA1(projectId, identity[:])
+}
+
+func otelSpanKey(span *tracepb.Span) string {
+ return string(span.TraceId) + string(span.SpanId)
+}
+
+func validSourceSpanIDs(span *tracepb.Span) bool {
+ return span != nil && validOtelID(span.TraceId, 16) && validOtelID(span.SpanId, 8) &&
+ (len(span.ParentSpanId) == 0 || len(span.ParentSpanId) == 8)
+}
diff --git a/backend/app/controllers/otelcontrollers/canonical_spans_test.go b/backend/app/controllers/otelcontrollers/canonical_spans_test.go
new file mode 100644
index 00000000..4c07607c
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/canonical_spans_test.go
@@ -0,0 +1,115 @@
+package otelcontrollers
+
+import (
+ "context"
+ "encoding/hex"
+ "reflect"
+ "testing"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/services"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+)
+
+func spanRequest(spans ...*tracepb.Span) *coltracepb.ExportTraceServiceRequest {
+ return &coltracepb.ExportTraceServiceRequest{ResourceSpans: []*tracepb.ResourceSpans{{ScopeSpans: []*tracepb.ScopeSpans{{Spans: spans}}}}}
+}
+
+func TestCanonicalSpansBatchIndependent(t *testing.T) {
+ trace := uuid.New()
+ root := &tracepb.Span{TraceId: trace[:], SpanId: []byte{1, 1, 1, 1, 1, 1, 1, 1}, Kind: tracepb.Span_SPAN_KIND_CONSUMER, Name: "worker"}
+ child := &tracepb.Span{TraceId: trace[:], SpanId: []byte{2, 2, 2, 2, 2, 2, 2, 2}, ParentSpanId: root.SpanId, Kind: tracepb.Span_SPAN_KIND_INTERNAL, Attributes: []*commonpb.KeyValue{strKV("http.request.method", "GET")}}
+ all := convertTraces(context.Background(), nil, testProjectId, spanRequest(root, child)).Spans
+ split := append(convertTraces(context.Background(), nil, testProjectId, spanRequest(root)).Spans, convertTraces(context.Background(), nil, testProjectId, spanRequest(child)).Spans...)
+ if !reflect.DeepEqual(all, split) {
+ t.Fatal("batching changed canonical graph")
+ }
+ for _, req := range []*coltracepb.ExportTraceServiceRequest{spanRequest(root, child), spanRequest(child)} {
+ endpoints := convertTraces(context.Background(), nil, testProjectId, req).Endpoints
+ if len(endpoints) != 0 {
+ t.Fatal("internal child classification depends on parent arrival")
+ }
+ }
+ req := spanRequest(root)
+ req.ResourceSpans = append(req.ResourceSpans, spanRequest(child).ResourceSpans...)
+ if !reflect.DeepEqual(all, convertTraces(context.Background(), nil, testProjectId, req).Spans) {
+ t.Fatal("resource split changed graph")
+ }
+}
+
+// The entity id is the one derived value: it must differ across projects and traces and stay put across retries. The
+// span keeps the ids it arrived with, and the browser's trace id never replaces them.
+func TestCanonicalIdentityIncludesTraceAndProject(t *testing.T) {
+ trace := uuid.MustParse("01234567-89ab-cdef-0123-456789abcdef")
+ otherTrace, browser := uuid.New(), uuid.New()
+ span := &tracepb.Span{TraceId: trace[:], SpanId: []byte{1, 2, 3, 4, 5, 6, 7, 8}}
+ a := otelOccurrenceID(testProjectId, span)
+ if a.String() != "211293e1-1332-5ff7-8097-c9b3a439f2fc" {
+ t.Fatalf("historical occurrence ID changed: %s", a)
+ }
+ if a == otelOccurrenceID(uuid.New(), span) || a != otelOccurrenceID(testProjectId, span) {
+ t.Fatal("entity id must depend on the project and be stable")
+ }
+ span.TraceId = otherTrace[:]
+ if a == otelOccurrenceID(testProjectId, span) {
+ t.Fatal("entity id collision across traces")
+ }
+ span.Kind = tracepb.Span_SPAN_KIND_SERVER
+ span.Attributes = []*commonpb.KeyValue{strKV("http.request.method", "GET"), strKV("traceway.distributed_trace_id", browser.String())}
+ converted := convertTraces(context.Background(), nil, testProjectId, spanRequest(span))
+ stored := converted.Spans[0]
+ if stored.TraceId != hex.EncodeToString(otherTrace[:]) || stored.SpanId != "0102030405060708" || stored.ParentSpanId != "" {
+ t.Fatalf("a span keeps the ids it arrived with: %+v", stored.Span)
+ }
+ endpoints := converted.Endpoints
+ if len(endpoints) != 1 || endpoints[0].TraceId != stored.TraceId || endpoints[0].SpanId != stored.SpanId || !endpoints[0].IsRoot {
+ t.Fatalf("the endpoint must retain its own trace ID without custom linking: %+v", endpoints)
+ }
+ for _, ignored := range []string{"not-a-uuid", otherTrace.String(), ""} {
+ span.Attributes = []*commonpb.KeyValue{strKV("http.request.method", "GET"), strKV("traceway.distributed_trace_id", ignored)}
+ if endpoints := convertTraces(context.Background(), nil, testProjectId, spanRequest(span)).Endpoints; endpoints[0].TraceId != stored.TraceId {
+ t.Fatalf("%q is no link: %+v", ignored, endpoints[0])
+ }
+ }
+}
+
+func TestCanonicalKeepsUnclassifiedRootsAndRejectsInvalidIDs(t *testing.T) {
+ trace := uuid.New()
+ root := &tracepb.Span{TraceId: trace[:], SpanId: []byte{1, 2, 3, 4, 5, 6, 7, 8}, Kind: tracepb.Span_SPAN_KIND_CLIENT}
+ invalid := &tracepb.Span{TraceId: []byte{1}, SpanId: root.SpanId}
+ zero := &tracepb.Span{TraceId: make([]byte, 16), SpanId: root.SpanId}
+ badParent := &tracepb.Span{TraceId: trace[:], SpanId: root.SpanId, ParentSpanId: []byte{1}}
+ converted := convertTraces(context.Background(), nil, testProjectId, spanRequest(root, invalid, zero, badParent, nil))
+ found := converted.Spans
+ if len(found) != 1 || found[0].ParentSpanId != "" {
+ t.Fatalf("unexpected roots: %+v", found)
+ }
+ if converted.InvalidSpanIDs != 4 {
+ t.Fatalf("invalid span count = %d, want 4", converted.InvalidSpanIDs)
+ }
+ // Frontend projects suppress product entities, but their source graph is retained.
+ converted = convertTraces(context.Background(), &models.Project{Framework: "react"}, testProjectId, spanRequest(root))
+ ep, tasks, ai := converted.Endpoints, converted.Tasks, converted.AiTraces
+ if len(ep)+len(tasks)+len(ai) != 0 || len(converted.Spans) != 1 {
+ t.Fatal("unexpected projection")
+ }
+}
+
+func TestCanonicalHealthcheckFilterKeepsPromotedDescendants(t *testing.T) {
+ trace := uuid.New()
+ root := &tracepb.Span{TraceId: trace[:], SpanId: []byte{1, 1, 1, 1, 1, 1, 1, 1}}
+ child := &tracepb.Span{TraceId: trace[:], SpanId: []byte{2, 2, 2, 2, 2, 2, 2, 2}, ParentSpanId: root.SpanId}
+ promoted := &tracepb.Span{TraceId: trace[:], SpanId: []byte{3, 3, 3, 3, 3, 3, 3, 3}, ParentSpanId: root.SpanId}
+ spans := convertTraces(context.Background(), nil, testProjectId, spanRequest(child, root, promoted)).Spans
+ traceId := hex.EncodeToString(trace[:])
+ kept := services.DropSpanSubtrees(spans, func(span models.OtelSpan) (string, string, string) {
+ return span.TraceId, span.SpanId, span.ParentSpanId
+ },
+ map[string]bool{services.SpanKey(traceId, "0101010101010101"): true}, map[string]bool{services.SpanKey(traceId, "0303030303030303"): true})
+ if len(kept) != 1 || kept[0].SpanId != "0303030303030303" {
+ t.Fatalf("incorrect healthcheck subtree filtering: %+v", kept)
+ }
+}
diff --git a/backend/app/controllers/otelcontrollers/codec.go b/backend/app/controllers/otelcontrollers/codec.go
index 8d744a7c..3da1cd82 100644
--- a/backend/app/controllers/otelcontrollers/codec.go
+++ b/backend/app/controllers/otelcontrollers/codec.go
@@ -87,7 +87,11 @@ func decodeTraceRequest(c *gin.Context) (*coltracepb.ExportTraceServiceRequest,
return nil, 0, fmt.Errorf("failed to unmarshal protobuf: %w", err)
}
} else {
- if err := protojson.Unmarshal(body, req); err != nil {
+ normalized, err := normalizeOTLPJSON(body, req.ProtoReflect().Descriptor())
+ if err != nil {
+ return nil, 0, fmt.Errorf("failed to decode OTLP JSON: %w", err)
+ }
+ if err := (protojson.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(normalized, req); err != nil {
return nil, 0, fmt.Errorf("failed to unmarshal JSON: %w", err)
}
}
@@ -105,15 +109,25 @@ func decodeMetricsRequest(c *gin.Context) (*colmetricspb.ExportMetricsServiceReq
return nil, 0, fmt.Errorf("failed to unmarshal protobuf: %w", err)
}
} else {
- if err := protojson.Unmarshal(body, req); err != nil {
+ if err := (protojson.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(body, req); err != nil {
return nil, 0, fmt.Errorf("failed to unmarshal JSON: %w", err)
}
}
return req, len(body), nil
}
-func writeTraceResponse(c *gin.Context) {
+func writeTraceResponse(c *gin.Context, invalidIDs, notStored int64) {
resp := &coltracepb.ExportTraceServiceResponse{}
+ if invalidIDs+notStored > 0 {
+ var reasons []string
+ if invalidIDs > 0 {
+ reasons = append(reasons, fmt.Sprintf("%d spans need valid non-zero trace and span IDs", invalidIDs))
+ }
+ if notStored > 0 {
+ reasons = append(reasons, fmt.Sprintf("%d spans could not be stored", notStored))
+ }
+ resp.PartialSuccess = &coltracepb.ExportTracePartialSuccess{RejectedSpans: invalidIDs + notStored, ErrorMessage: strings.Join(reasons, "; ")}
+ }
if isProtobuf(c) {
data, _ := proto.Marshal(resp)
c.Data(http.StatusOK, "application/x-protobuf", data)
@@ -143,7 +157,7 @@ func decodeProfilesPayload(c *gin.Context) ([]byte, int, error) {
return body, len(body), nil
}
req := &colprofilespb.ExportProfilesServiceRequest{}
- if err := protojson.Unmarshal(body, req); err != nil {
+ if err := (protojson.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(body, req); err != nil {
return nil, 0, fmt.Errorf("failed to unmarshal JSON: %w", err)
}
binary, err := proto.Marshal(req)
@@ -175,7 +189,11 @@ func decodeLogsRequest(c *gin.Context) (*collogspb.ExportLogsServiceRequest, int
return nil, 0, fmt.Errorf("failed to unmarshal protobuf: %w", err)
}
} else {
- if err := protojson.Unmarshal(body, req); err != nil {
+ normalized, err := normalizeOTLPJSON(body, req.ProtoReflect().Descriptor())
+ if err != nil {
+ return nil, 0, fmt.Errorf("failed to decode OTLP JSON: %w", err)
+ }
+ if err := (protojson.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(normalized, req); err != nil {
return nil, 0, fmt.Errorf("failed to unmarshal JSON: %w", err)
}
}
diff --git a/backend/app/controllers/otelcontrollers/codec_unknown_fields_test.go b/backend/app/controllers/otelcontrollers/codec_unknown_fields_test.go
new file mode 100644
index 00000000..d003bef9
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/codec_unknown_fields_test.go
@@ -0,0 +1,93 @@
+package otelcontrollers
+
+import (
+ "bytes"
+ "net/http/httptest"
+ "strings"
+ "testing"
+
+ "github.com/gin-gonic/gin"
+ colprofilespb "go.opentelemetry.io/proto/otlp/collector/profiles/v1development"
+ "google.golang.org/protobuf/proto"
+)
+
+func jsonContext(path string, body string) *gin.Context {
+ c, _ := gin.CreateTestContext(httptest.NewRecorder())
+ c.Request = httptest.NewRequest("POST", path, bytes.NewReader([]byte(body)))
+ c.Request.Header.Set("Content-Type", "application/json")
+ return c
+}
+
+func TestOTLPJSONSignalsIgnoreUnknownFields(t *testing.T) {
+ metrics, _, err := decodeMetricsRequest(jsonContext("/api/otel/v1/metrics", `{"futureTopLevel":{"a":1},"resourceMetrics":[{"futureResourceField":true,"scopeMetrics":[{"metrics":[{"name":"requests","futureMetricField":[1,2],"gauge":{"dataPoints":[{"asInt":"7","futurePointField":"x"}]}}]}]}]}`))
+ if err != nil {
+ t.Fatalf("metrics: %v", err)
+ }
+ if point := metrics.ResourceMetrics[0].ScopeMetrics[0].Metrics[0].GetGauge().DataPoints[0]; point.GetAsInt() != 7 {
+ t.Fatalf("metrics known fields changed: %v", point)
+ }
+
+ logs, _, err := decodeLogsRequest(jsonContext("/api/otel/v1/logs", `{"futureTopLevel":1,"resourceLogs":[{"futureResourceField":"x","scopeLogs":[{"logRecords":[{"severityText":"ERROR","body":{"stringValue":"boom"},"futureRecordField":{"nested":true}}]}]}]}`))
+ if err != nil {
+ t.Fatalf("logs: %v", err)
+ }
+ if record := logs.ResourceLogs[0].ScopeLogs[0].LogRecords[0]; record.SeverityText != "ERROR" || record.Body.GetStringValue() != "boom" {
+ t.Fatalf("logs known fields changed: %v", record)
+ }
+
+ payload, _, err := decodeProfilesPayload(jsonContext("/api/otel/v1development/profiles", `{"futureTopLevel":true,"resourceProfiles":[{"futureResourceField":1,"scopeProfiles":[{"futureScopeField":"x"}]}]}`))
+ if err != nil {
+ t.Fatalf("profiles: %v", err)
+ }
+ var profiles colprofilespb.ExportProfilesServiceRequest
+ if err := proto.Unmarshal(payload, &profiles); err != nil || len(profiles.ResourceProfiles) != 1 || len(profiles.ResourceProfiles[0].ScopeProfiles) != 1 {
+ t.Fatalf("profiles known fields changed: %v %v", &profiles, err)
+ }
+}
+
+func TestOTLPJSONSignalsStillRejectMalformedKnownFields(t *testing.T) {
+ if _, _, err := decodeMetricsRequest(jsonContext("/api/otel/v1/metrics", `{"resourceMetrics":"not an array"}`)); err == nil {
+ t.Fatal("metrics accepted a malformed known field")
+ }
+ if _, _, err := decodeLogsRequest(jsonContext("/api/otel/v1/logs", `{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"timeUnixNano":{"not":"a number"}}]}]}]}`)); err == nil {
+ t.Fatal("logs accepted a malformed known field")
+ }
+ if _, _, err := decodeProfilesPayload(jsonContext("/api/otel/v1development/profiles", `{"resourceProfiles":{"not":"an array"}}`)); err == nil {
+ t.Fatal("profiles accepted a malformed known field")
+ }
+}
+
+func TestLogJSONNormalizesOnlyContextIDs(t *testing.T) {
+ body := `{"resourceLogs":[{"resource":{"attributes":[{"key":"traceId","value":{"stringValue":"resource ID"}}]},"scopeLogs":[{"scope":{"attributes":[{"key":"spanId","value":{"stringValue":"scope ID"}}]},"logRecords":[{"traceId":"0123456789ABCDEF0123456789ABCDEF","spanId":"ABCDEF0123456789","attributes":[{"key":"traceId","value":{"stringValue":"attribute ID"}}],"body":{"kvlistValue":{"values":[{"key":"spanId","value":{"stringValue":"body ID"}}]}},"futureField":true}]}]}]}`
+ snakeCase := strings.NewReplacer(
+ `"resourceLogs":`, `"resource_logs":`, `"scopeLogs":`, `"scope_logs":`,
+ `"logRecords":`, `"log_records":`, `"traceId":`, `"trace_id":`, `"spanId":`, `"span_id":`,
+ ).Replace(body)
+ for _, input := range []string{body, snakeCase} {
+ req, _, err := decodeLogsRequest(jsonContext("/api/otel/v1/logs", input))
+ if err != nil {
+ t.Fatal(err)
+ }
+ resource := req.ResourceLogs[0]
+ scope := resource.ScopeLogs[0]
+ log := scope.LogRecords[0]
+ if len(log.TraceId) != 16 || len(log.SpanId) != 8 {
+ t.Fatalf("IDs were not decoded from hex: %v", log)
+ }
+ if getStringAttribute(resource.Resource.Attributes, "traceId") != "resource ID" ||
+ getStringAttribute(scope.Scope.Attributes, "spanId") != "scope ID" ||
+ getStringAttribute(log.Attributes, "traceId") != "attribute ID" ||
+ getStringAttribute(log.Body.GetKvlistValue().Values, "spanId") != "body ID" {
+ t.Fatalf("normalization changed user attributes or body: %v", req)
+ }
+ }
+}
+
+func TestLogJSONRejectsMalformedHexIDs(t *testing.T) {
+ for _, id := range []string{`"abc"`, `"gggggggggggggggg"`, `42`} {
+ body := `{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"spanId":` + id + `}]}]}]}`
+ if _, _, err := decodeLogsRequest(jsonContext("/api/otel/v1/logs", body)); err == nil {
+ t.Fatalf("accepted malformed span ID %s", id)
+ }
+ }
+}
diff --git a/backend/app/controllers/otelcontrollers/helpers.go b/backend/app/controllers/otelcontrollers/helpers.go
index a3fc7566..c70d0bc0 100644
--- a/backend/app/controllers/otelcontrollers/helpers.go
+++ b/backend/app/controllers/otelcontrollers/helpers.go
@@ -1,76 +1,21 @@
package otelcontrollers
import (
- "encoding/base64"
- "encoding/hex"
"strconv"
- "time"
- "github.com/google/uuid"
commonpb "go.opentelemetry.io/proto/otlp/common/v1"
)
-// otelTraceIDToUUID converts a 16-byte OTEL trace ID to a UUID.
-// OTLP JSON uses hex encoding for trace_id, but protojson base64-decodes it,
-// producing 24 bytes instead of 16. We roundtrip through base64 to recover the hex string.
-func otelTraceIDToUUID(traceID []byte) uuid.UUID {
- if len(traceID) == 16 {
- var u uuid.UUID
- copy(u[:], traceID)
- return u
+func validOtelID(id []byte, size int) bool {
+ if len(id) != size {
+ return false
}
- if len(traceID) == 24 {
- hexStr := base64.StdEncoding.EncodeToString(traceID)
- if decoded, err := hex.DecodeString(hexStr); err == nil && len(decoded) == 16 {
- var u uuid.UUID
- copy(u[:], decoded)
- return u
+ for _, b := range id {
+ if b != 0 {
+ return true
}
}
- return uuid.Nil
-}
-
-// otelSpanIDToUUID converts an 8-byte OTEL span ID to a UUID by zero-padding the first 8 bytes.
-// Same base64/hex roundtrip as otelTraceIDToUUID for 12-byte inputs.
-func otelSpanIDToUUID(spanID []byte) uuid.UUID {
- if len(spanID) == 8 {
- var u uuid.UUID
- copy(u[8:], spanID)
- return u
- }
- if len(spanID) == 12 {
- hexStr := base64.StdEncoding.EncodeToString(spanID)
- if decoded, err := hex.DecodeString(hexStr); err == nil && len(decoded) == 8 {
- var u uuid.UUID
- copy(u[8:], decoded)
- return u
- }
- }
- return uuid.Nil
-}
-
-func nanoToTime(nanos uint64) time.Time {
- return time.Unix(0, int64(nanos))
-}
-
-// spanUUIDToHex returns the hex representation of the last 8 bytes of a span-derived UUID
-// (those last 8 bytes carry the original OTel span ID). Mirrors otelSpanIDToUUID going the
-// other direction, producing the same format we store in log_records.span_id.
-func spanUUIDToHex(u uuid.UUID) string {
- return hex.EncodeToString(u[8:])
-}
-
-// ptrSpanUUID wraps otelSpanIDToUUID for nullable parent-span-id capture: returns nil when
-// the span has no parent bytes, or when the OTel span ID is malformed.
-func ptrSpanUUID(raw []byte) *uuid.UUID {
- if len(raw) == 0 {
- return nil
- }
- u := otelSpanIDToUUID(raw)
- if u == uuid.Nil {
- return nil
- }
- return &u
+ return false
}
func extractAttributes(attrs []*commonpb.KeyValue) map[string]string {
diff --git a/backend/app/controllers/otelcontrollers/log_correlation_test.go b/backend/app/controllers/otelcontrollers/log_correlation_test.go
index 1562ff4d..e818ec10 100644
--- a/backend/app/controllers/otelcontrollers/log_correlation_test.go
+++ b/backend/app/controllers/otelcontrollers/log_correlation_test.go
@@ -2,21 +2,18 @@ package otelcontrollers
import (
"context"
- "encoding/base64"
"encoding/hex"
"testing"
"github.com/google/uuid"
coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
commonpb "go.opentelemetry.io/proto/otlp/common/v1"
- logspb "go.opentelemetry.io/proto/otlp/logs/v1"
tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
)
func TestPromotedEntityLogCorrelation(t *testing.T) {
const traceHex = "8462dd06157032df3c6c90ac5ff74ed7"
traceBytes, _ := hex.DecodeString(traceHex)
- jsonTraceBytes, _ := base64.StdEncoding.DecodeString(traceHex)
spanBytes, _ := hex.DecodeString("1985a7abed0024db")
groupID := uuid.MustParse("a5100000-0000-4000-8000-000000000001")
@@ -29,55 +26,42 @@ func TestPromotedEntityLogCorrelation(t *testing.T) {
{"endpoint", tracepb.Span_SPAN_KIND_SERVER, []*commonpb.KeyValue{strKV("http.request.method", "GET")}},
{"ai", tracepb.Span_SPAN_KIND_CLIENT, []*commonpb.KeyValue{strKV("gen_ai.system", "openai")}},
} {
- for _, encoding := range []struct {
- name string
- id []byte
- }{
- {"protobuf", traceBytes},
- {"json", jsonTraceBytes},
- } {
+ for _, encoding := range []string{"protobuf", "json"} {
for _, override := range []bool{false, true} {
- t.Run(entity.name+"/"+encoding.name+"/override="+map[bool]string{false: "false", true: "true"}[override], func(t *testing.T) {
+ t.Run(entity.name+"/"+encoding+"/override="+map[bool]string{false: "false", true: "true"}[override], func(t *testing.T) {
attrs := append([]*commonpb.KeyValue{}, entity.attrs...)
if override {
attrs = append(attrs, strKV("traceway.distributed_trace_id", groupID.String()))
}
req := &coltracepb.ExportTraceServiceRequest{ResourceSpans: []*tracepb.ResourceSpans{{
ScopeSpans: []*tracepb.ScopeSpans{{Spans: []*tracepb.Span{{
- TraceId: encoding.id, SpanId: spanBytes, ParentSpanId: []byte{1, 2, 3, 4, 5, 6, 7, 8},
+ TraceId: traceBytes, SpanId: spanBytes, ParentSpanId: []byte{1, 2, 3, 4, 5, 6, 7, 8},
Name: "worker", Kind: entity.kind, Attributes: attrs,
StartTimeUnixNano: 1_700_000_000_000_000_000, EndTimeUnixNano: 1_700_000_000_001_000_000,
}}}},
}}}
- endpoints, tasks, _, _, aiTraces, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, tasks, aiTraces := converted.Endpoints, converted.Tasks, converted.AiTraces
if len(endpoints)+len(tasks)+len(aiTraces) != 1 {
t.Fatal("expected one promoted entity")
}
var id uuid.UUID
- var distributedID *uuid.UUID
- var attributes map[string]string
+ var traceId string
switch entity.name {
case "task":
- id, distributedID, attributes = tasks[0].Id, tasks[0].DistributedTraceId, tasks[0].Attributes
+ id, traceId = tasks[0].Id, tasks[0].TraceId
case "endpoint":
- id, distributedID, attributes = endpoints[0].Id, endpoints[0].DistributedTraceId, endpoints[0].Attributes
+ id, traceId = endpoints[0].Id, endpoints[0].TraceId
case "ai":
- id, distributedID, attributes = aiTraces[0].Id, aiTraces[0].DistributedTraceId, aiTraces[0].Attributes
+ id, traceId = aiTraces[0].Id, aiTraces[0].TraceId
}
- log := toLogRecord(testProjectId, &logspb.LogRecord{TraceId: encoding.id, SpanId: spanBytes}, "worker", "", nil, "", "", "", nil)
- if attributes["traceway.otel.trace_id"] != traceHex || attributes["traceway.otel.trace_id"] != log.TraceId {
- t.Fatalf("entity trace ID %q does not correlate with log trace ID %q", attributes["traceway.otel.trace_id"], log.TraceId)
+ log := toLogRecord(testProjectId, decodeLogIDs(t, encoding, traceBytes, spanBytes), "worker", "", nil, "", "", "", nil)
+ if traceId != traceHex || traceId != log.TraceId {
+ t.Fatalf("entity trace ID %q does not correlate with log trace ID %q", traceId, log.TraceId)
}
- if id != otelSpanIDToUUID(spanBytes) {
+ if id != otelOccurrenceID(testProjectId, &tracepb.Span{TraceId: traceBytes, SpanId: spanBytes}) {
t.Fatalf("occurrence ID changed: %s", id)
}
- wantGroup := otelTraceIDToUUID(traceBytes)
- if override {
- wantGroup = groupID
- }
- if distributedID == nil || *distributedID != wantGroup {
- t.Fatalf("distributed grouping changed: %v", distributedID)
- }
})
}
}
diff --git a/backend/app/controllers/otelcontrollers/logs_converter.go b/backend/app/controllers/otelcontrollers/logs_converter.go
index 4759171c..064e9916 100644
--- a/backend/app/controllers/otelcontrollers/logs_converter.go
+++ b/backend/app/controllers/otelcontrollers/logs_converter.go
@@ -8,11 +8,11 @@ import (
"strings"
"github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
collogspb "go.opentelemetry.io/proto/otlp/collector/logs/v1"
commonpb "go.opentelemetry.io/proto/otlp/common/v1"
logspb "go.opentelemetry.io/proto/otlp/logs/v1"
-
- "github.com/tracewayapp/traceway/backend/app/models"
)
func convertLogs(existingProject *models.Project, ctx context.Context, projectId uuid.UUID, req *collogspb.ExportLogsServiceRequest) []models.LogRecord {
@@ -66,19 +66,18 @@ func toLogRecord(
severityText = severityTextFromNumber(severityNumber)
}
- // JSON-OTLP trace_id/span_id arrive as 24/12 base64 bytes — round-trip through the UUID helpers.
var traceIDHex, spanIDHex string
- if u := otelTraceIDToUUID(lr.TraceId); u != uuid.Nil {
- traceIDHex = hex.EncodeToString(u[:])
+ if validOtelID(lr.TraceId, 16) {
+ traceIDHex = hex.EncodeToString(lr.TraceId)
}
- if u := otelSpanIDToUUID(lr.SpanId); u != uuid.Nil {
- spanIDHex = spanUUIDToHex(u)
+ if validOtelID(lr.SpanId, 8) {
+ spanIDHex = hex.EncodeToString(lr.SpanId)
}
return models.LogRecord{
Id: uuid.New(),
ProjectId: projectId,
- Timestamp: nanoToTime(ts),
+ Timestamp: shared.OtelNanosToTime(ts),
TraceId: traceIDHex,
SpanId: spanIDHex,
TraceFlags: uint8(lr.Flags),
diff --git a/backend/app/controllers/otelcontrollers/logs_converter_test.go b/backend/app/controllers/otelcontrollers/logs_converter_test.go
index d3ed27f1..c2fe248c 100644
--- a/backend/app/controllers/otelcontrollers/logs_converter_test.go
+++ b/backend/app/controllers/otelcontrollers/logs_converter_test.go
@@ -1,12 +1,15 @@
package otelcontrollers
import (
+ "bytes"
"context"
- "encoding/base64"
"encoding/hex"
+ "fmt"
+ "net/http/httptest"
"strings"
"testing"
+ "github.com/gin-gonic/gin"
"github.com/google/uuid"
"github.com/tracewayapp/traceway/backend/app/services"
"github.com/tracewayapp/traceway/backend/app/storage"
@@ -14,6 +17,7 @@ import (
commonpb "go.opentelemetry.io/proto/otlp/common/v1"
logspb "go.opentelemetry.io/proto/otlp/logs/v1"
resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ "google.golang.org/protobuf/proto"
)
const testSourceMap = `{"version":3,"names":["abcd"],"sources":["tests/fixtures/simple/original.js"],"sourcesContent":["// ./node_modules/.bin/terser -c -m --module tests/fixtures/simple/original.js --source-map includeSources -o tests/fixtures/simple/minified.js\nfunction abcd() {}\nexport default abcd;\n"],"mappings":"AACA,SAASA,oBACMA"}`
@@ -111,49 +115,65 @@ func TestConvertLogs_NonJsStacktraceUntouched(t *testing.T) {
}
}
-// Pins the protojson base64-of-hex round-trip — see logs_converter.go.
+func decodeLogIDs(t *testing.T, encoding string, traceID, spanID []byte) *logspb.LogRecord {
+ t.Helper()
+ var body []byte
+ contentType := "application/json"
+ if encoding == "protobuf" {
+ request := &collogspb.ExportLogsServiceRequest{ResourceLogs: []*logspb.ResourceLogs{{ScopeLogs: []*logspb.ScopeLogs{{LogRecords: []*logspb.LogRecord{{TraceId: traceID, SpanId: spanID}}}}}}}
+ var err error
+ body, err = proto.Marshal(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ contentType = "application/x-protobuf"
+ } else {
+ body = []byte(fmt.Sprintf(`{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"traceId":%q,"spanId":%q}]}]}]}`, hex.EncodeToString(traceID), hex.EncodeToString(spanID)))
+ }
+ c, _ := gin.CreateTestContext(httptest.NewRecorder())
+ c.Request = httptest.NewRequest("POST", "/api/otel/v1/logs", bytes.NewReader(body))
+ c.Request.Header.Set("Content-Type", contentType)
+ req, size, err := decodeLogsRequest(c)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if size != len(body) {
+ t.Fatalf("body size = %d, want %d", size, len(body))
+ }
+ return req.ResourceLogs[0].ScopeLogs[0].LogRecords[0]
+}
+
func TestToLogRecord_IDEncoding(t *testing.T) {
const wireTraceHex = "7b873c7bbf35739e79e1f7b9736739f7"
const wireSpanHex = "7dfd3877775ae1bd"
-
- // protojson decodes bytes-typed fields as base64; hex chars are valid base64.
- jsonTrace, err := base64.StdEncoding.DecodeString(wireTraceHex)
- if err != nil || len(jsonTrace) != 24 {
- t.Fatalf("seed: jsonTrace len=%d err=%v", len(jsonTrace), err)
- }
- jsonSpan, err := base64.StdEncoding.DecodeString(wireSpanHex)
- if err != nil || len(jsonSpan) != 12 {
- t.Fatalf("seed: jsonSpan len=%d err=%v", len(jsonSpan), err)
- }
- binTrace, _ := hex.DecodeString(wireTraceHex)
- binSpan, _ := hex.DecodeString(wireSpanHex)
-
- tests := []struct {
- name string
- traceBytes []byte
- spanBytes []byte
- wantTrace string
- wantSpan string
- }{
- {"binary OTLP", binTrace, binSpan, wireTraceHex, wireSpanHex},
- {"JSON OTLP (protojson base64 roundtrip)", jsonTrace, jsonSpan, wireTraceHex, wireSpanHex},
- {"missing trace context", nil, nil, "", ""},
- }
-
- for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
- rec := toLogRecord(
- testProjectId,
- &logspb.LogRecord{TraceId: tt.traceBytes, SpanId: tt.spanBytes},
- "svc", "", nil, "", "scope", "", nil,
- )
- if rec.TraceId != tt.wantTrace {
- t.Errorf("TraceId = %q (len %d), want %q", rec.TraceId, len(rec.TraceId), tt.wantTrace)
- }
- if rec.SpanId != tt.wantSpan {
- t.Errorf("SpanId = %q (len %d), want %q", rec.SpanId, len(rec.SpanId), tt.wantSpan)
- }
- })
+ traceID, _ := hex.DecodeString(wireTraceHex)
+ spanID, _ := hex.DecodeString(wireSpanHex)
+
+ for _, encoding := range []string{"protobuf", "json"} {
+ for _, tt := range []struct {
+ name string
+ trace, span []byte
+ wantTrace, wantSpan string
+ }{
+ {"valid", traceID, spanID, wireTraceHex, wireSpanHex},
+ {"missing", nil, nil, "", ""},
+ {"zero", make([]byte, 16), make([]byte, 8), "", ""},
+ {"short", []byte{1}, []byte{1}, "", ""},
+ {"long", bytes.Repeat([]byte{1}, 24), bytes.Repeat([]byte{1}, 12), "", ""},
+ {"trace only", traceID, nil, wireTraceHex, ""},
+ {"span only", nil, spanID, "", wireSpanHex},
+ } {
+ t.Run(encoding+"/"+tt.name, func(t *testing.T) {
+ lr := decodeLogIDs(t, encoding, tt.trace, tt.span)
+ if !bytes.Equal(lr.TraceId, tt.trace) || !bytes.Equal(lr.SpanId, tt.span) {
+ t.Fatalf("decoder changed ID bytes: %v", lr)
+ }
+ rec := toLogRecord(testProjectId, lr, "svc", "", nil, "", "scope", "", nil)
+ if rec.TraceId != tt.wantTrace || rec.SpanId != tt.wantSpan {
+ t.Fatalf("IDs = %q/%q, want %q/%q", rec.TraceId, rec.SpanId, tt.wantTrace, tt.wantSpan)
+ }
+ })
+ }
}
}
diff --git a/backend/app/controllers/otelcontrollers/metric_converter.go b/backend/app/controllers/otelcontrollers/metric_converter.go
index 08643b5c..e0652db4 100644
--- a/backend/app/controllers/otelcontrollers/metric_converter.go
+++ b/backend/app/controllers/otelcontrollers/metric_converter.go
@@ -2,6 +2,7 @@ package otelcontrollers
import (
"github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
"github.com/google/uuid"
@@ -87,7 +88,7 @@ func convertMetricPoints(projectId uuid.UUID, req *colmetricspb.ExportMetricsSer
}
case *metricspb.Metric_Histogram:
for _, dp := range data.Histogram.GetDataPoints() {
- ts := nanoToTime(dp.TimeUnixNano)
+ ts := shared.OtelNanosToTime(dp.TimeUnixNano)
tags := buildTags(sn, resTags, dp.Attributes)
if dp.Count > 0 && dp.Sum != nil {
points = append(points, models.MetricPoint{
@@ -150,7 +151,7 @@ func appendNumberDataPoints(points []models.MetricPoint, projectId uuid.UUID, na
Name: name,
Value: value,
Tags: tags,
- RecordedAt: nanoToTime(dp.TimeUnixNano),
+ RecordedAt: shared.OtelNanosToTime(dp.TimeUnixNano),
})
}
return points
diff --git a/backend/app/controllers/otelcontrollers/otel.controller.go b/backend/app/controllers/otelcontrollers/otel.controller.go
index e552f730..58b84b71 100644
--- a/backend/app/controllers/otelcontrollers/otel.controller.go
+++ b/backend/app/controllers/otelcontrollers/otel.controller.go
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"net/http"
+ "slices"
"time"
"github.com/gin-gonic/gin"
@@ -14,6 +15,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/monitoring"
"github.com/tracewayapp/traceway/backend/app/profiling"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
"github.com/tracewayapp/traceway/backend/app/services"
"github.com/tracewayapp/traceway/backend/app/storage"
"github.com/tracewayapp/traceway/backend/app/symbolicator/sourcemap/jsstack"
@@ -47,6 +49,29 @@ func otelSymbolicateAndroid(existingProject *models.Project, projectId uuid.UUID
type otelController struct{}
+var traceStore = struct {
+ spans func(context.Context, []models.OtelSpan) ([]shared.SpanOwner, error)
+ endpoints func(context.Context, []models.Endpoint) error
+ tasks func(context.Context, []models.Task) error
+ exceptions func(context.Context, []models.ExceptionStackTrace) error
+ aiTraces func(context.Context, []models.AiTrace) error
+}{
+ spans: telemetry.OtelSpanRepository.InsertWithRejections,
+ endpoints: telemetry.EndpointRepository.InsertAsync,
+ tasks: telemetry.TaskRepository.InsertAsync,
+ exceptions: telemetry.ExceptionStackTraceRepository.InsertAsync,
+ aiTraces: telemetry.AiTraceRepository.InsertAsync,
+}
+
+func abortIngestStorage(c *gin.Context, stage string, err error) {
+ status := http.StatusInternalServerError
+ if telemetry.IsTransientStorageError(err) {
+ status = http.StatusServiceUnavailable
+ c.Header("Retry-After", "2")
+ }
+ c.AbortWithError(status, traceway.NewStackTraceErrorf("error inserting OTEL %s: %w", stage, err))
+}
+
var OtelController = otelController{}
func (o otelController) ExportTraces(c *gin.Context) {
@@ -84,18 +109,35 @@ func (o otelController) ExportTraces(c *gin.Context) {
}
convertStart := time.Now()
- endpoints, tasks, spans, exceptions, aiTraces, aiConversations := convertTraces(c, project, projectId, req)
-
- var droppedHealthchecks int
- endpoints, spans, droppedHealthchecks = services.FilterHealthchecks(project, endpoints, spans, exceptions)
- if droppedHealthchecks > 0 {
- monitoring.RecordHealthchecksDropped(monitoring.SignalTraces, droppedHealthchecks)
+ result := convertTraces(c, project, projectId, req)
+ endpoints, tasks, exceptions := result.Endpoints, result.Tasks, result.Exceptions
+ aiTraces, aiConversations := result.AiTraces, result.AiConversations
+ canonicalSpans := result.Spans
+ if !perm.Data {
+ canonicalSpans = nil
+ }
+ var droppedHealthchecks map[string]bool
+ endpoints, droppedHealthchecks = services.FilterHealthchecks(project, endpoints, exceptions)
+ if len(droppedHealthchecks) > 0 {
+ retained := make(map[string]bool)
+ for _, endpoint := range endpoints {
+ retained[services.SpanKey(endpoint.TraceId, endpoint.SpanId)] = true
+ }
+ for _, task := range tasks {
+ retained[services.SpanKey(task.TraceId, task.SpanId)] = true
+ }
+ for _, ai := range aiTraces {
+ retained[services.SpanKey(ai.TraceId, ai.SpanId)] = true
+ }
+ canonicalSpans = services.DropSpanSubtrees(canonicalSpans, func(span models.OtelSpan) (string, string, string) {
+ return span.TraceId, span.SpanId, span.ParentSpanId
+ }, droppedHealthchecks, retained)
+ monitoring.RecordHealthchecksDropped(monitoring.SignalTraces, len(droppedHealthchecks))
}
if !perm.Data {
endpoints = nil
tasks = nil
- spans = nil
aiTraces = nil
aiConversations = nil
}
@@ -110,33 +152,50 @@ func (o otelController) ExportTraces(c *gin.Context) {
insertStart := time.Now()
- if len(endpoints) > 0 {
- if err := telemetry.EndpointRepository.InsertAsync(c, endpoints); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL endpoints: %w", err))
- return
- }
+ // Spans are the source the other rows are projected from, so a failed span write must not leave projections behind.
+ notStored, err := traceStore.spans(c, canonicalSpans)
+ if err != nil {
+ abortIngestStorage(c, "spans", err)
+ return
}
- if len(tasks) > 0 {
- if err := telemetry.TaskRepository.InsertAsync(c, tasks); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL tasks: %w", err))
- return
+ if len(notStored) > 0 {
+ rejected := make(map[shared.SpanOwner]bool, len(notStored))
+ for _, span := range notStored {
+ rejected[span] = true
}
+ hasRejectedSource := func(traceId, spanId string) bool {
+ return rejected[shared.SpanOwner{ProjectId: projectId, TraceId: traceId, SpanId: spanId}]
+ }
+ endpoints = slices.DeleteFunc(endpoints, func(row models.Endpoint) bool { return hasRejectedSource(row.TraceId, row.SpanId) })
+ tasks = slices.DeleteFunc(tasks, func(row models.Task) bool { return hasRejectedSource(row.TraceId, row.SpanId) })
+ exceptions = slices.DeleteFunc(exceptions, func(row models.ExceptionStackTrace) bool { return hasRejectedSource(row.TraceId, row.SpanId) })
+ aiTraces = slices.DeleteFunc(aiTraces, func(row models.AiTrace) bool { return hasRejectedSource(row.TraceId, row.SpanId) })
+ storedConversations := map[string]bool{}
+ for _, row := range aiTraces {
+ storedConversations[row.StorageKey] = true
+ }
+ aiConversations = slices.DeleteFunc(aiConversations, func(row aiTraceConversation) bool { return !storedConversations[row.StorageKey] })
+ }
+
+ if err := traceStore.endpoints(c, endpoints); err != nil {
+ abortIngestStorage(c, "endpoints", err)
+ return
}
- if err := telemetry.ExceptionStackTraceRepository.InsertAsync(c, exceptions); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL exceptions: %w", err))
+ if err := traceStore.tasks(c, tasks); err != nil {
+ abortIngestStorage(c, "tasks", err)
return
}
- if err := telemetry.SpanRepository.InsertAsync(c, spans); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL spans: %w", err))
+ if err := traceStore.exceptions(c, exceptions); err != nil {
+ abortIngestStorage(c, "exceptions", err)
return
}
if len(aiTraces) > 0 {
- if err := telemetry.AiTraceRepository.InsertAsync(c, aiTraces); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL ai traces: %w", err))
+ if err := traceStore.aiTraces(c, aiTraces); err != nil {
+ abortIngestStorage(c, "ai traces", err)
return
}
@@ -155,7 +214,7 @@ func (o otelController) ExportTraces(c *gin.Context) {
}
insertMs := msSince(insertStart)
- totalSize := len(endpoints) + len(tasks) + len(spans) + len(exceptions) + len(aiTraces)
+ totalSize := len(endpoints) + len(tasks) + len(canonicalSpans) + len(exceptions) + len(aiTraces)
monitoring.RecordIngestBatch(monitoring.SignalTraces, "traces", convertMs, insertMs, totalSize, bodyBytes)
var exceptionHashes []string
@@ -193,7 +252,7 @@ func (o otelController) ExportTraces(c *gin.Context) {
hooks.BroadcastReport(ev)
}
- writeTraceResponse(c)
+ writeTraceResponse(c, result.InvalidSpanIDs, int64(len(notStored)))
}
func (o otelController) ExportMetrics(c *gin.Context) {
@@ -244,7 +303,7 @@ func (o otelController) ExportMetrics(c *gin.Context) {
if len(result.Points) > 0 {
insertStart := time.Now()
if err := telemetry.MetricPointRepository.InsertAsync(c, result.Points); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL metric points: %w", err))
+ abortIngestStorage(c, "metric points", err)
return
}
insertMs = msSince(insertStart)
@@ -315,7 +374,7 @@ func (o otelController) ExportLogs(c *gin.Context) {
if len(records) > 0 {
insertStart := time.Now()
if err := telemetry.LogRecordRepository.InsertAsync(c, records); err != nil {
- c.AbortWithError(500, traceway.NewStackTraceErrorf("error inserting OTEL log records: %w", err))
+ abortIngestStorage(c, "log records", err)
return
}
insertMs = msSince(insertStart)
diff --git a/backend/app/controllers/otelcontrollers/otlp_fidelity_test.go b/backend/app/controllers/otelcontrollers/otlp_fidelity_test.go
new file mode 100644
index 00000000..caef42e7
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/otlp_fidelity_test.go
@@ -0,0 +1,92 @@
+//go:build !telemetry_ch && !telemetry_duckdb && !transactional_pg
+
+package otelcontrollers
+
+import (
+ "bytes"
+ "context"
+ "encoding/hex"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestOTLPFullSpanRoundTrip(t *testing.T) {
+ dbtest.SetupSQLite(t)
+ trace, linkedTrace := uuid.New(), uuid.New()
+ attrs := []*commonpb.KeyValue{
+ strKV("service.name", "api"), strKV("exception.stacktrace", "keep original stack"),
+ {Key: "int", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: 9223372036854775807}}},
+ {Key: "double", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: 1.234}}},
+ {Key: "bool", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: true}}},
+ {Key: "bytes", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BytesValue{BytesValue: []byte{0, 255, 128}}}},
+ {Key: "array", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: []*commonpb.AnyValue{{Value: &commonpb.AnyValue_StringValue{StringValue: "nested"}}}}}}},
+ {Key: "map", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_KvlistValue{KvlistValue: &commonpb.KeyValueList{Values: []*commonpb.KeyValue{strKV("traceId", "this is an attribute")}}}}},
+ }
+ now := uint64(time.Now().UnixNano())
+ source := &tracepb.Span{TraceId: trace[:], SpanId: []byte{128, 2, 3, 4, 5, 6, 7, 255}, ParentSpanId: []byte{8, 7, 6, 5, 4, 3, 2, 1},
+ Name: "original name", Kind: tracepb.Span_SPAN_KIND_INTERNAL, TraceState: "vendor=opaque", Flags: 0xabcdef01,
+ StartTimeUnixNano: now, EndTimeUnixNano: now + 123456789, Attributes: attrs, DroppedAttributesCount: 3, DroppedEventsCount: 4, DroppedLinksCount: 5,
+ Status: &tracepb.Status{Code: tracepb.Status_STATUS_CODE_ERROR, Message: "full status detail"},
+ Events: []*tracepb.Span_Event{{TimeUnixNano: now + 1, Name: "event", Attributes: attrs, DroppedAttributesCount: 6}},
+ Links: []*tracepb.Span_Link{{TraceId: linkedTrace[:], SpanId: []byte{1, 2, 3, 4, 5, 6, 7, 8}, TraceState: "other=state", Flags: 0xff000301, Attributes: attrs, DroppedAttributesCount: 7}},
+ }
+ resource := &tracepb.ResourceSpans{SchemaUrl: "https://resource/schema", Resource: &resourcepb.Resource{Attributes: attrs, DroppedAttributesCount: 8},
+ ScopeSpans: []*tracepb.ScopeSpans{{SchemaUrl: "https://scope/schema", Scope: &commonpb.InstrumentationScope{Name: "scope", Version: "1.2.3", Attributes: attrs, DroppedAttributesCount: 9}, Spans: []*tracepb.Span{source}}}}
+ source.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x01})
+ resource.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x02})
+ resource.ScopeSpans[0].ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x03})
+ req := &coltracepb.ExportTraceServiceRequest{ResourceSpans: []*tracepb.ResourceSpans{resource}}
+ spans := convertTraces(context.Background(), nil, testProjectId, req).Spans
+ // Retries retain each span's full inline context.
+ for range 2 {
+ if _, err := telemetry.OtelSpanRepository.InsertAsync(context.Background(), append(spans, spans...)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ payload, err := telemetry.OtelSpanRepository.FindOTLP(context.Background(), testProjectId, hex.EncodeToString(trace[:]), hex.EncodeToString(source.SpanId), spans[0].StartTime)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var got tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &got); err != nil {
+ t.Fatal(err)
+ }
+ if !proto.Equal(resource, &got) {
+ t.Fatalf("OTLP fields changed:\nwant %v\ngot %v", resource, &got)
+ }
+ var count int
+ if err := db.TelemetryDB.QueryRow("SELECT count(*) FROM spans_v2 WHERE resource_schema_url = 'https://resource/schema'").Scan(&count); err != nil || count != 4 {
+ t.Fatalf("complete span rows: %d %v", count, err)
+ }
+ missing, err := telemetry.OtelSpanRepository.FindOTLP(context.Background(), uuid.New(), spans[0].TraceId, spans[0].SpanId, spans[0].StartTime)
+ if err != nil || len(missing) != 0 {
+ t.Fatalf("cross-project payload leaked: %v", err)
+ }
+}
+
+func TestOTLPJSONHexAndUnknownFields(t *testing.T) {
+ body := []byte(`{"resourceSpans":[{"futureResourceField":true,"scopeSpans":[{"spans":[{"traceId":"0123456789ABCDEF0123456789ABCDEF","spanId":"ABCDEF0123456789","parentSpanId":"FEDCBA9876543210","flags":4294967295,"links":[{"traceId":"1234567890abcdef1234567890abcdef","spanId":"1234567890abcdef"}],"attributes":[{"key":"traceId","value":{"stringValue":"leave this untouched"}}],"futureSpanField":123}]}]}],"futureTopLevelField":{}}`)
+ c, _ := gin.CreateTestContext(httptest.NewRecorder())
+ c.Request = httptest.NewRequest("POST", "/api/otel/v1/traces", bytes.NewReader(body))
+ c.Request.Header.Set("Content-Type", "application/json")
+ req, n, err := decodeTraceRequest(c)
+ if err != nil {
+ t.Fatal(err)
+ }
+ span := req.ResourceSpans[0].ScopeSpans[0].Spans[0]
+ if n != len(body) || len(span.TraceId) != 16 || len(span.SpanId) != 8 || len(span.ParentSpanId) != 8 || len(span.Links[0].TraceId) != 16 || len(span.Links[0].SpanId) != 8 || span.Flags != 0xffffffff || getStringAttribute(span.Attributes, "traceId") != "leave this untouched" {
+ t.Fatalf("incorrect OTLP JSON decoding: %v", span)
+ }
+}
diff --git a/backend/app/controllers/otelcontrollers/otlp_json.go b/backend/app/controllers/otelcontrollers/otlp_json.go
new file mode 100644
index 00000000..c4f05db3
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/otlp_json.go
@@ -0,0 +1,54 @@
+package otelcontrollers
+
+import (
+ "encoding/base64"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+
+ "google.golang.org/protobuf/reflect/protoreflect"
+)
+
+// OTLP uses hex IDs, whereas ordinary protobuf JSON encodes bytes as base64.
+// Walking the descriptor avoids rewriting similarly named user attributes.
+func normalizeOTLPJSON(body []byte, descriptor protoreflect.MessageDescriptor) ([]byte, error) {
+ var object map[string]json.RawMessage
+ if err := json.Unmarshal(body, &object); err != nil {
+ return nil, err
+ }
+ for key, value := range object {
+ field := descriptor.Fields().ByJSONName(key)
+ if field == nil {
+ field = descriptor.Fields().ByName(protoreflect.Name(key))
+ }
+ if field == nil {
+ continue
+ }
+ if field.Kind() == protoreflect.BytesKind && (field.Name() == "trace_id" || field.Name() == "span_id" || field.Name() == "parent_span_id") {
+ var id string
+ if err := json.Unmarshal(value, &id); err != nil {
+ return nil, err
+ }
+ decoded, err := hex.DecodeString(id)
+ if err != nil {
+ return nil, fmt.Errorf("invalid hexadecimal %s", key)
+ }
+ object[key], _ = json.Marshal(base64.StdEncoding.EncodeToString(decoded))
+ } else if field.Kind() == protoreflect.MessageKind && (field.Name() == "resource_spans" || field.Name() == "scope_spans" || field.Name() == "spans" || field.Name() == "links" ||
+ field.Name() == "resource_logs" || field.Name() == "scope_logs" || field.Name() == "log_records") {
+ var values []json.RawMessage
+ if err := json.Unmarshal(value, &values); err != nil {
+ return nil, err
+ }
+ for i := range values {
+ normalized, err := normalizeOTLPJSON(values[i], field.Message())
+ if err != nil {
+ return nil, err
+ }
+ values[i] = normalized
+ }
+ object[key], _ = json.Marshal(values)
+ }
+ }
+ return json.Marshal(object)
+}
diff --git a/backend/app/controllers/otelcontrollers/testdata/anthropic_tool_use_ai_trace.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/anthropic_tool_use_ai_trace.json.golden.json
index d224d00d..e19781b1 100644
--- a/backend/app/controllers/otelcontrollers/testdata/anthropic_tool_use_ai_trace.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/anthropic_tool_use_ai_trace.json.golden.json
@@ -2,8 +2,13 @@
"endpointCount": 0,
"endpoints": [],
"taskCount": 0,
- "spanCount": 0,
- "spans": [],
+ "spanCount": 1,
+ "spans": [
+ {
+ "name": "chat claude-sonnet-5",
+ "linkedToParent": true
+ }
+ ],
"exceptionCount": 0,
"exceptions": [],
"aiTraceCount": 1,
diff --git a/backend/app/controllers/otelcontrollers/testdata/honeycomb_global_error.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/honeycomb_global_error.json.golden.json
index 5c138e7b..41d2e47d 100644
--- a/backend/app/controllers/otelcontrollers/testdata/honeycomb_global_error.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/honeycomb_global_error.json.golden.json
@@ -9,17 +9,26 @@
}
],
"taskCount": 0,
- "spanCount": 0,
- "spans": [],
+ "spanCount": 2,
+ "spans": [
+ {
+ "name": "TTFB",
+ "linkedToParent": true
+ },
+ {
+ "name": "exception",
+ "linkedToParent": false
+ }
+ ],
"exceptionCount": 1,
"exceptions": [
{
"stackTrace": "TypeError: discount rate must be a finite number between 0 and 1, got: NaN\nassertValidDiscountRate()\n http://localhost:4173/assets/app-DWNS_JMv.js:1:321\napplyDiscount()\n http://localhost:4173/assets/app-DWNS_JMv.js:1:199\ncomputeOrderTotal()\n http://localhost:4173/assets/app-DWNS_JMv.js:1:64\nHTMLButtonElement.handleCheckout()\n http://localhost:4173/assets/app-DWNS_JMv.js:1:957",
- "traceType": "task"
+ "traceType": ""
}
],
"aiTraceCount": 0,
"aiTraces": [],
"conversationCount": 0,
- "allSpansLinked": true
+ "allSpansLinked": false
}
\ No newline at end of file
diff --git a/backend/app/controllers/otelcontrollers/testdata/node_better_auth.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/node_better_auth.json.golden.json
index a8951622..2e5993fc 100644
--- a/backend/app/controllers/otelcontrollers/testdata/node_better_auth.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/node_better_auth.json.golden.json
@@ -9,8 +9,12 @@
}
],
"taskCount": 0,
- "spanCount": 4,
+ "spanCount": 5,
"spans": [
+ {
+ "name": "GET /get-session",
+ "linkedToParent": true
+ },
{
"name": "db findOne session",
"linkedToParent": true
diff --git a/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json
index 22ca730f..0d758f64 100644
--- a/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/node_sign_in.json.golden.json
@@ -9,8 +9,12 @@
}
],
"taskCount": 0,
- "spanCount": 5,
+ "spanCount": 6,
"spans": [
+ {
+ "name": "POST /sign-in/email",
+ "linkedToParent": true
+ },
{
"name": "db findMany account",
"linkedToParent": true
@@ -36,7 +40,7 @@
"exceptions": [
{
"stackTrace": "APIError: Invalid email or password",
- "traceType": "endpoint"
+ "traceType": ""
}
],
"aiTraceCount": 0,
diff --git a/backend/app/controllers/otelcontrollers/testdata/openai_tool_calls_ai_trace.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/openai_tool_calls_ai_trace.json.golden.json
index fe66927b..0694506f 100644
--- a/backend/app/controllers/otelcontrollers/testdata/openai_tool_calls_ai_trace.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/openai_tool_calls_ai_trace.json.golden.json
@@ -2,8 +2,13 @@
"endpointCount": 0,
"endpoints": [],
"taskCount": 0,
- "spanCount": 0,
- "spans": [],
+ "spanCount": 1,
+ "spans": [
+ {
+ "name": "chat gpt-4o",
+ "linkedToParent": true
+ }
+ ],
"exceptionCount": 0,
"exceptions": [],
"aiTraceCount": 1,
diff --git a/backend/app/controllers/otelcontrollers/testdata/openrouter_ai_trace.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/openrouter_ai_trace.json.golden.json
index faabeb74..de40618d 100644
--- a/backend/app/controllers/otelcontrollers/testdata/openrouter_ai_trace.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/openrouter_ai_trace.json.golden.json
@@ -2,8 +2,13 @@
"endpointCount": 0,
"endpoints": [],
"taskCount": 0,
- "spanCount": 0,
- "spans": [],
+ "spanCount": 1,
+ "spans": [
+ {
+ "name": "Test Generation",
+ "linkedToParent": true
+ }
+ ],
"exceptionCount": 0,
"exceptions": [],
"aiTraceCount": 1,
diff --git a/backend/app/controllers/otelcontrollers/testdata/spring_boot_exception.json.golden.json b/backend/app/controllers/otelcontrollers/testdata/spring_boot_exception.json.golden.json
index b3058fae..dd4463e4 100644
--- a/backend/app/controllers/otelcontrollers/testdata/spring_boot_exception.json.golden.json
+++ b/backend/app/controllers/otelcontrollers/testdata/spring_boot_exception.json.golden.json
@@ -9,10 +9,14 @@
}
],
"taskCount": 0,
- "spanCount": 1,
+ "spanCount": 2,
"spans": [
{
- "name": "SELECT id, email, name FROM users WHERE id = ?",
+ "name": "GET /api/users/{id}",
+ "linkedToParent": true
+ },
+ {
+ "name": "SELECT users",
"linkedToParent": true
}
],
diff --git a/backend/app/controllers/otelcontrollers/trace_converter.go b/backend/app/controllers/otelcontrollers/trace_converter.go
index 2d7c49a1..e58bb046 100644
--- a/backend/app/controllers/otelcontrollers/trace_converter.go
+++ b/backend/app/controllers/otelcontrollers/trace_converter.go
@@ -12,6 +12,7 @@ import (
"github.com/google/uuid"
"github.com/tracewayapp/traceway/backend/app/controllers/clientcontrollers"
"github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
"github.com/tracewayapp/traceway/backend/app/services/contentflag"
coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
commonpb "go.opentelemetry.io/proto/otlp/common/v1"
@@ -46,14 +47,17 @@ func (k entityKind) traceType() string {
return ""
}
-func convertTraces(ctx context.Context, existingProject *models.Project, projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequest) (
- endpoints []models.Endpoint,
- tasks []models.Task,
- spans []models.Span,
- exceptions []models.ExceptionStackTrace,
- aiTraces []models.AiTrace,
- aiConversations []aiTraceConversation,
-) {
+type convertedTraces struct {
+ Spans []models.OtelSpan
+ Endpoints []models.Endpoint
+ Tasks []models.Task
+ Exceptions []models.ExceptionStackTrace
+ AiTraces []models.AiTrace
+ AiConversations []aiTraceConversation
+ InvalidSpanIDs int64
+}
+
+func convertTraces(ctx context.Context, existingProject *models.Project, projectId uuid.UUID, req *coltracepb.ExportTraceServiceRequest) (result convertedTraces) {
suppressEntities := existingProject != nil && clientcontrollers.IsFrontendFramework(existingProject.Framework)
// nil languages falls back to the default pack; a project that disabled
@@ -64,6 +68,7 @@ func convertTraces(ctx context.Context, existingProject *models.Project, project
customFlagTerms = existingProject.AiFlaggedTerms
}
flagMatcher := contentflag.NewMatcher(flagLanguages, customFlagTerms)
+ ingestedAt := time.Now().UTC()
for _, rs := range req.ResourceSpans {
resourceAttrs := rs.GetResource().GetAttributes()
@@ -80,105 +85,57 @@ func convertTraces(ctx context.Context, existingProject *models.Project, project
}
type spanEntry struct {
- span *tracepb.Span
- scopeName string
+ span *tracepb.Span
+ metadata *tracepb.ResourceSpans
}
var allSpans []spanEntry
- parentMap := map[string]string{}
- spanById := map[string]*tracepb.Span{}
+ spanByKey := map[string]*tracepb.Span{}
for _, ss := range rs.ScopeSpans {
+ metadata := &tracepb.ResourceSpans{Resource: rs.Resource, SchemaUrl: rs.SchemaUrl,
+ ScopeSpans: []*tracepb.ScopeSpans{{Scope: ss.Scope, SchemaUrl: ss.SchemaUrl}}}
+ metadata.ProtoReflect().SetUnknown(rs.ProtoReflect().GetUnknown())
+ metadata.ScopeSpans[0].ProtoReflect().SetUnknown(ss.ProtoReflect().GetUnknown())
for _, span := range ss.Spans {
- allSpans = append(allSpans, spanEntry{span: span, scopeName: ss.GetScope().GetName()})
- spanById[string(span.SpanId)] = span
- if len(span.ParentSpanId) > 0 {
- parentMap[string(span.SpanId)] = string(span.ParentSpanId)
+ if !validSourceSpanIDs(span) {
+ result.InvalidSpanIDs++
+ continue
}
+ allSpans = append(allSpans, spanEntry{span: span, metadata: metadata})
+ spanByKey[otelSpanKey(span)] = span
}
}
-
- // Pass 1: classify each span by Kind/attrs and assign an entity id.
- // Roots get id = otelTraceIDToUUID(trace_id); non-roots get id = otelSpanIDToUUID(span_id).
- // distributed_trace_id = otelTraceIDToUUID(trace_id) for both, unless overridden by
- // the vendor `traceway.distributed_trace_id` attribute.
- type promotion struct {
- kind entityKind
- id uuid.UUID
- isRoot bool
- distributedTraceId *uuid.UUID
+ parentInBatch := func(span *tracepb.Span) *tracepb.Span {
+ if len(span.ParentSpanId) == 0 {
+ return nil
+ }
+ return spanByKey[string(span.TraceId)+string(span.ParentSpanId)]
}
- spanIdToPromotion := map[string]promotion{}
for _, entry := range allSpans {
- if suppressEntities {
- break
- }
span := entry.span
- kind := classifySpan(span, spanById)
- if kind == entityNone {
- continue
- }
-
- isRoot := len(span.ParentSpanId) == 0
-
- var id uuid.UUID
- if isRoot {
- id = otelTraceIDToUUID(span.TraceId)
- if id == uuid.Nil {
- id = otelSpanIDToUUID(span.SpanId)
- }
- } else {
- id = otelSpanIDToUUID(span.SpanId)
- if id == uuid.Nil {
- id = uuid.New()
- }
- }
-
- dtId := otelTraceIDToUUID(span.TraceId)
- var distributedTraceId *uuid.UUID
- if dtId != uuid.Nil {
- distributedTraceId = &dtId
- }
- if override := getStringAttribute(span.Attributes, "traceway.distributed_trace_id"); override != "" {
- if parsed, err := uuid.Parse(override); err == nil {
- distributedTraceId = &parsed
- }
+ scopeName := entry.metadata.ScopeSpans[0].GetScope().GetName()
+ traceId := hex.EncodeToString(span.TraceId)
+ spanId := hex.EncodeToString(span.SpanId)
+ parentSpanId := hex.EncodeToString(span.ParentSpanId)
+ start := shared.OtelNanosToTime(span.StartTimeUnixNano)
+ duration := shared.OtelDuration(span.StartTimeUnixNano, span.EndTimeUnixNano)
+ result.Spans = append(result.Spans, models.OtelSpan{
+ OTLP: span, Context: entry.metadata,
+ Span: models.Span{
+ ProjectId: projectId, TraceId: traceId, SpanId: spanId, ParentSpanId: parentSpanId,
+ Name: span.Name, StartTime: start, RecordedAt: start, Duration: duration,
+ SpanKind: int32(span.Kind), StatusCode: int32(span.GetStatus().GetCode()),
+ ServiceName: serverName, ScopeName: scopeName,
+ },
+ })
+ kind := entityNone
+ if !suppressEntities {
+ kind = classifySpan(span, parentInBatch)
}
-
- spanIdToPromotion[string(span.SpanId)] = promotion{
- kind: kind,
- id: id,
- isRoot: isRoot,
- distributedTraceId: distributedTraceId,
- }
- }
-
- // resolveOwner walks parents until it finds a promoted entity. Returns the
- // promotion (so callers know the kind for trace_type) and whether one was
- // found. Cached per span id within this resource batch.
- ownerCache := map[string]*promotion{}
- var resolveOwner func(spanIdStr string) *promotion
- resolveOwner = func(spanIdStr string) *promotion {
- if cached, ok := ownerCache[spanIdStr]; ok {
- return cached
- }
- if p, ok := spanIdToPromotion[spanIdStr]; ok {
- pp := p
- ownerCache[spanIdStr] = &pp
- return &pp
- }
- parentId, hasParent := parentMap[spanIdStr]
- if !hasParent {
- ownerCache[spanIdStr] = nil
- return nil
+ if kind == entityNone && !spanCarriesException(span) {
+ continue
}
- owner := resolveOwner(parentId)
- ownerCache[spanIdStr] = owner
- return owner
- }
- // Pass 2: emit entity rows + span rows + exceptions.
- for _, entry := range allSpans {
- span := entry.span
spanAttrs := span.Attributes
allAttrs := extractAttributes(spanAttrs)
// A span carrying exception.stacktrace always yields an exception
@@ -186,156 +143,113 @@ func convertTraces(ctx context.Context, existingProject *models.Project, project
// present), so the raw blob is never duplicated onto the
// endpoint/task/span rows or the exception's attribute map.
delete(allAttrs, "exception.stacktrace")
- startTime := nanoToTime(span.StartTimeUnixNano)
- endTime := nanoToTime(span.EndTimeUnixNano)
- duration := endTime.Sub(startTime)
-
- prom, promoted := spanIdToPromotion[string(span.SpanId)]
-
- // Determine the owning entity for span/exception trace_id.
- var owner *promotion
- if promoted {
- p := prom
- owner = &p
- } else {
- owner = resolveOwner(string(span.SpanId))
- }
-
- // trace_id for span rows / exceptions: owning entity id when known;
- // otherwise fall back to the OTel trace_id (orphan path — preserves
- // today's behavior for cross-process children whose parent never
- // matched a promoted entity within this batch).
- var ownerId uuid.UUID
- var ownerTraceType string
- if owner != nil {
- ownerId = owner.id
- ownerTraceType = owner.kind.traceType()
- } else {
- ownerId = otelTraceIDToUUID(span.TraceId)
- if ownerId == uuid.Nil {
- ownerId = uuid.New()
- }
- }
-
- if promoted {
- // Occurrence IDs can come from span_id, and distributed_trace_id can be
- // overridden. Keep the wire trace ID for correlation with OTLP logs.
- if traceId := otelTraceIDToUUID(span.TraceId); traceId != uuid.Nil {
- if allAttrs == nil {
- allAttrs = make(map[string]string)
- }
- allAttrs["traceway.otel.trace_id"] = hex.EncodeToString(traceId[:])
- }
- rootSpanId := otelSpanIDToUUID(span.SpanId)
- switch prom.kind {
- case entityEndpoint:
- ep := buildEndpoint(
- prom.id, projectId, span, spanAttrs, allAttrs,
- startTime, duration, serverName, appVersion,
- )
- ep.DistributedTraceId = prom.distributedTraceId
- ep.SpanId = &rootSpanId
- ep.IsRoot = prom.isRoot
- endpoints = append(endpoints, ep)
- case entityTask:
- t := buildTask(
- prom.id, projectId, span, allAttrs,
- startTime, endTime, duration, serverName, appVersion,
- )
- t.DistributedTraceId = prom.distributedTraceId
- t.SpanId = &rootSpanId
- t.IsRoot = prom.isRoot
- tasks = append(tasks, t)
- case entityAiTrace:
- aiTrace := buildAiTrace(
- prom.id, projectId, span, spanAttrs, allAttrs,
- startTime, duration, serverName, appVersion,
- )
- aiTrace.DistributedTraceId = prom.distributedTraceId
- aiTrace.IsRoot = prom.isRoot
- aiTrace.ConversationId = resolveConversationId(spanAttrs, resourceAttrs, prom.distributedTraceId)
- var convInput, convOutput string
- if conv := extractConversation(spanAttrs, projectId, prom.id); conv != nil {
- convInput, convOutput = conv.Input, conv.Output
- aiConversations = append(aiConversations, *conv)
- }
- aiTrace.ToolCallCount, aiTrace.ToolNames = extractToolCalls(spanAttrs, convOutput)
- if terms := flagMatcher.Scan(convInput, convOutput); len(terms) > 0 {
- aiTrace.Flagged = true
- aiTrace.FlaggedTerms = terms
- }
- aiTraces = append(aiTraces, aiTrace)
+ recordedStart, _, _ := shared.OtelStorageTimes(span.StartTimeUnixNano, ingestedAt)
+ id := otelOccurrenceID(projectId, span)
+
+ switch kind {
+ case entityEndpoint:
+ ep := buildEndpoint(
+ id, projectId, span, spanAttrs, allAttrs,
+ recordedStart, duration, serverName, appVersion,
+ )
+ ep.TraceId, ep.SpanId, ep.ParentSpanId = traceId, spanId, parentSpanId
+ ep.IsRoot = parentSpanId == ""
+ result.Endpoints = append(result.Endpoints, ep)
+ case entityTask:
+ t := buildTask(
+ id, projectId, span, allAttrs,
+ recordedStart, duration, serverName, appVersion,
+ )
+ t.TraceId, t.SpanId, t.ParentSpanId = traceId, spanId, parentSpanId
+ t.IsRoot = parentSpanId == ""
+ result.Tasks = append(result.Tasks, t)
+ case entityAiTrace:
+ aiTrace := buildAiTrace(
+ id, projectId, span, spanAttrs, allAttrs,
+ recordedStart, duration, serverName, appVersion,
+ )
+ aiTrace.TraceId, aiTrace.SpanId, aiTrace.ParentSpanId = traceId, spanId, parentSpanId
+ aiTrace.IsRoot = parentSpanId == ""
+ aiTrace.ConversationId = resolveConversationId(spanAttrs, resourceAttrs, traceId)
+ var convInput, convOutput string
+ if conv := extractConversation(spanAttrs, projectId, id); conv != nil {
+ convInput, convOutput = conv.Input, conv.Output
+ result.AiConversations = append(result.AiConversations, *conv)
}
- } else if !suppressEntities && len(span.ParentSpanId) > 0 {
- // Non-root, unpromoted span → goes to the generic spans table,
- // re-rooted to its nearest enclosing entity (or the OTel
- // trace_id as fallback when the parent chain doesn't reach a
- // promoted span in this batch).
- spanName := span.Name
- if dbQuery := getStringAttribute(spanAttrs, "db.query.text"); dbQuery != "" {
- spanName = dbQuery
- } else if dbStatement := getStringAttribute(spanAttrs, "db.statement"); dbStatement != "" {
- spanName = dbStatement
+ aiTrace.ToolCallCount, aiTrace.ToolNames = extractToolCalls(spanAttrs, convOutput)
+ if terms := flagMatcher.Scan(convInput, convOutput); len(terms) > 0 {
+ aiTrace.Flagged = true
+ aiTrace.FlaggedTerms = terms
}
-
- spans = append(spans, models.Span{
- Id: otelSpanIDToUUID(span.SpanId),
- TraceId: ownerId,
- ProjectId: projectId,
- Name: spanName,
- StartTime: startTime,
- Duration: duration,
- RecordedAt: startTime,
- ParentSpanId: ptrSpanUUID(span.ParentSpanId),
- Attributes: allAttrs,
- })
- } else if !spanCarriesException(span) {
- // Unpromoted root span: match historical behavior and drop it
- // (no entity row, no span row). Common case: CLIENT-kind roots
- // or non-HTTP SERVER roots from custom instrumentation that we
- // don't have a dedicated page for. Spans carrying an exception
- // signal (event or exception.* span attributes, e.g. Honeycomb's
- // global-errors zero-duration root span) fall through so the
- // exception is still captured, without an entity or span row.
- continue
+ result.AiTraces = append(result.AiTraces, aiTrace)
}
- traceType := ownerTraceType
- if traceType == "" {
- traceType = "task"
+ appendException := func(attrs []*commonpb.KeyValue, timeUnixNano uint64) {
+ exc := buildException(
+ ctx, existingProject, projectId, attrs, timeUnixNano,
+ allAttrs, serverName, appVersion, language, proguardUuid, scopeName,
+ )
+ // The kind is stored only when this span says it. An exception further down the request finds its entity at read time.
+ exc.TraceId, exc.SpanId, exc.TraceType = traceId, spanId, kind.traceType()
+ result.Exceptions = append(result.Exceptions, exc)
}
hadExceptionEvent := false
for _, event := range span.Events {
if event.Name == "exception" {
hadExceptionEvent = true
- exc := buildException(
- ctx, existingProject, projectId, ownerId, traceType, event.Attributes, event.TimeUnixNano,
- allAttrs, serverName, appVersion, language, proguardUuid, entry.scopeName,
- )
- if owner != nil {
- exc.DistributedTraceId = owner.distributedTraceId
- }
- exceptions = append(exceptions, exc)
+ appendException(event.Attributes, event.TimeUnixNano)
}
}
-
if !hadExceptionEvent && hasExceptionAttributes(span.Attributes) {
- exc := buildException(
- ctx, existingProject, projectId, ownerId, traceType, span.Attributes, span.StartTimeUnixNano,
- allAttrs, serverName, appVersion, language, proguardUuid, entry.scopeName,
- )
- if owner != nil {
- exc.DistributedTraceId = owner.distributedTraceId
- }
- exceptions = append(exceptions, exc)
+ appendException(span.Attributes, span.StartTimeUnixNano)
}
}
}
return
}
-func classifySpan(span *tracepb.Span, spanById map[string]*tracepb.Span) entityKind {
+const (
+ spanFlagHasIsRemote = uint32(tracepb.SpanFlags_SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK)
+ spanFlagIsRemote = uint32(tracepb.SpanFlags_SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK)
+ // Bounds the walk when malformed input makes the parent chain a cycle.
+ maxEntryPointDepth = 64
+)
+
+func isHTTPServerSpan(span *tracepb.Span) bool {
+ return span.Kind == tracepb.Span_SPAN_KIND_SERVER && hasHTTPAttributes(span.Attributes)
+}
+
+// An HTTP SERVER span is a request of its own unless a local span above it already is one (Next.js under the HTTP
+// instrumentation). The walk stays inside the process: it stops at a remote parent, which OTLP flags mark however
+// the trace is batched, and at a CLIENT or PRODUCER span, which is the calling side. A plain wrapper span above
+// the request does not make it nested. Missing parents have unknown kinds, even
+// when flags establish locality, so they cannot suppress an HTTP entry point.
+func isEntryPoint(span *tracepb.Span, parentOf func(*tracepb.Span) *tracepb.Span) bool {
+ for current, depth := span, 0; depth < maxEntryPointDepth; depth++ {
+ if len(current.ParentSpanId) == 0 {
+ return true
+ }
+ localityKnown := current.Flags&spanFlagHasIsRemote != 0
+ if localityKnown && current.Flags&spanFlagIsRemote != 0 {
+ return true
+ }
+ parent := parentOf(current)
+ if parent == nil {
+ return true
+ }
+ if parent.Kind == tracepb.Span_SPAN_KIND_CLIENT || parent.Kind == tracepb.Span_SPAN_KIND_PRODUCER {
+ return true
+ }
+ if isHTTPServerSpan(parent) {
+ return false
+ }
+ current = parent
+ }
+ return true
+}
+
+func classifySpan(span *tracepb.Span, parentOf func(*tracepb.Span) *tracepb.Span) entityKind {
attrs := span.Attributes
// Honeycomb's browser SDK stamps page context (url.path etc.) on every
// span, including the zero-duration INTERNAL `exception` spans emitted by
@@ -344,14 +258,11 @@ func classifySpan(span *tracepb.Span, spanById map[string]*tracepb.Span) entityK
if span.Kind == tracepb.Span_SPAN_KIND_INTERNAL && hasExceptionAttributes(attrs) {
return entityNone
}
- if (span.Kind == tracepb.Span_SPAN_KIND_SERVER || span.Kind == tracepb.Span_SPAN_KIND_INTERNAL) && hasHTTPAttributes(attrs) {
- if len(span.ParentSpanId) == 0 {
- return entityEndpoint
- }
- if _, parentInBatch := spanById[string(span.ParentSpanId)]; !parentInBatch {
- return entityEndpoint
- }
+ if (isHTTPServerSpan(span) && isEntryPoint(span, parentOf)) ||
+ (span.Kind == tracepb.Span_SPAN_KIND_INTERNAL && len(span.ParentSpanId) == 0 && hasHTTPAttributes(attrs)) {
+ return entityEndpoint
}
+
if span.Kind == tracepb.Span_SPAN_KIND_CONSUMER {
return entityTask
}
@@ -493,7 +404,7 @@ func buildTask(
id, projectId uuid.UUID,
span *tracepb.Span,
allAttrs map[string]string,
- startTime, endTime time.Time,
+ startTime time.Time,
duration time.Duration,
serverName, appVersion string,
) models.Task {
@@ -502,7 +413,7 @@ func buildTask(
ProjectId: projectId,
TaskName: span.Name,
Duration: duration,
- RecordedAt: endTime,
+ RecordedAt: startTime,
Attributes: allAttrs,
AppVersion: appVersion,
ServerName: serverName,
@@ -512,8 +423,7 @@ func buildTask(
func buildException(
ctx context.Context,
existingProject *models.Project,
- projectId, traceId uuid.UUID,
- traceType string,
+ projectId uuid.UUID,
excAttrs []*commonpb.KeyValue,
timeUnixNano uint64,
spanAttrs map[string]string,
@@ -537,6 +447,7 @@ func buildException(
stackTrace = otelSymbolicateAndroid(existingProject, projectId, ctx, stackTrace, language, proguardUuid)
hash := clientcontrollers.ComputeExceptionHash(stackTrace, false)
+ recordedAt, _, _ := shared.OtelStorageTimes(timeUnixNano, time.Now())
attrs := spanAttrs
if isJsLanguage(language) || isAndroidLanguage(language) {
@@ -547,11 +458,9 @@ func buildException(
return models.ExceptionStackTrace{
Id: uuid.New(),
ProjectId: projectId,
- TraceId: &traceId,
- TraceType: traceType,
ExceptionHash: hash,
StackTrace: stackTrace,
- RecordedAt: nanoToTime(timeUnixNano),
+ RecordedAt: recordedAt,
Attributes: attrs,
AppVersion: appVersion,
ServerName: serverName,
@@ -848,9 +757,9 @@ func extractConversation(attrs []*commonpb.KeyValue, projectId, traceId uuid.UUI
// resolveConversationId picks the conversation grouping key for an AI trace:
// an explicit gen_ai.conversation.id, else session.id (span first, then
-// resource — browser SDKs stamp it on the resource), else the distributed
-// trace id so a single agent run still groups its calls.
-func resolveConversationId(spanAttrs, resourceAttrs []*commonpb.KeyValue, distributedTraceId *uuid.UUID) string {
+// resource, where browser SDKs stamp it), else the trace id so a single agent
+// run still groups its calls.
+func resolveConversationId(spanAttrs, resourceAttrs []*commonpb.KeyValue, traceId string) string {
if id := getStringAttribute(spanAttrs, "gen_ai.conversation.id"); id != "" {
return id
}
@@ -860,10 +769,11 @@ func resolveConversationId(spanAttrs, resourceAttrs []*commonpb.KeyValue, distri
if id := getStringAttribute(resourceAttrs, "session.id"); id != "" {
return id
}
- if distributedTraceId != nil {
- return distributedTraceId.String()
+ // Keep fallback conversation keys compatible with history written before V2.
+ if id, err := uuid.Parse(traceId); err == nil {
+ return id.String()
}
- return ""
+ return traceId
}
const maxToolNames = 50
diff --git a/backend/app/controllers/otelcontrollers/trace_converter_test.go b/backend/app/controllers/otelcontrollers/trace_converter_test.go
index 7a044acb..f8111c52 100644
--- a/backend/app/controllers/otelcontrollers/trace_converter_test.go
+++ b/backend/app/controllers/otelcontrollers/trace_converter_test.go
@@ -2,6 +2,7 @@ package otelcontrollers
import (
"context"
+ "encoding/hex"
"encoding/json"
"flag"
"os"
@@ -34,6 +35,31 @@ type snapshotEndpoint struct {
AppVersion string `json:"appVersion"`
}
+func spanKey(traceId, spanId string) string { return traceId + ":" + spanId }
+
+// nearestPromotedAncestor walks the stored parent edges the way a read does, and returns the span id of the first
+// promoted span it meets.
+func nearestPromotedAncestor(canonical []models.OtelSpan, span models.OtelSpan, promoted map[string]bool) (string, bool) {
+ byKey := make(map[string]models.OtelSpan, len(canonical))
+ for _, candidate := range canonical {
+ byKey[spanKey(candidate.TraceId, candidate.SpanId)] = candidate
+ }
+ for range len(canonical) + 1 {
+ if promoted[spanKey(span.TraceId, span.SpanId)] {
+ return span.SpanId, true
+ }
+ if span.ParentSpanId == "" {
+ return "", false
+ }
+ parent, ok := byKey[spanKey(span.TraceId, span.ParentSpanId)]
+ if !ok {
+ return "", false
+ }
+ span = parent
+ }
+ return "", false
+}
+
type snapshotSpan struct {
Name string `json:"name"`
LinkedToParent bool `json:"linkedToParent"`
@@ -105,25 +131,30 @@ func TestConvertTraces_Snapshot(t *testing.T) {
}
req := &coltracepb.ExportTraceServiceRequest{}
- if err := protojson.Unmarshal(raw, req); err != nil {
+ normalized, err := normalizeOTLPJSON(raw, req.ProtoReflect().Descriptor())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := protojson.Unmarshal(normalized, req); err != nil {
t.Fatalf("failed to unmarshal fixture %s: %v", tt.fixture, err)
}
setFakeStore(t, nil)
- endpoints, _, spans, exceptions, aiTraces, aiConversations := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, exceptions, aiTraces, aiConversations := converted.Endpoints, converted.Exceptions, converted.AiTraces, converted.AiConversations
+ spans := converted.Spans
- // Check if all child spans share a trace ID with an endpoint
- endpointIds := map[uuid.UUID]bool{}
+ endpointIds := map[string]bool{}
for _, ep := range endpoints {
- endpointIds[ep.Id] = true
+ endpointIds[spanKey(ep.TraceId, ep.SpanId)] = true
}
for _, at := range aiTraces {
- endpointIds[at.Id] = true
+ endpointIds[spanKey(at.TraceId, at.SpanId)] = true
}
allLinked := true
snapSpans := make([]snapshotSpan, len(spans))
for i, s := range spans {
- linked := endpointIds[s.TraceId]
+ _, linked := nearestPromotedAncestor(spans, s, endpointIds)
snapSpans[i] = snapshotSpan{Name: s.Name, LinkedToParent: linked}
if !linked {
allLinked = false
@@ -454,16 +485,21 @@ func TestTraceIdResolution_CrossScope(t *testing.T) {
},
}
- endpoints, _, spans, _, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints := converted.Endpoints
if len(endpoints) != 1 {
t.Fatalf("expected 1 endpoint, got %d", len(endpoints))
}
- rootTraceId := endpoints[0].Id
+ spans := converted.Spans
+ if len(spans) < 2 {
+ t.Fatalf("expected the root and its descendants to be stored, got %d spans", len(spans))
+ }
+ promoted := map[string]bool{spanKey(endpoints[0].TraceId, endpoints[0].SpanId): true}
for _, s := range spans {
- if s.TraceId != rootTraceId {
- t.Errorf("span %q has traceId %s, want %s (root endpoint)", s.Name, s.TraceId, rootTraceId)
+ if owner, ok := nearestPromotedAncestor(spans, s, promoted); !ok || owner != endpoints[0].SpanId {
+ t.Errorf("span %q does not reach the root endpoint through its stored parent edges", s.Name)
}
}
}
@@ -547,7 +583,8 @@ func TestConvertTraces_ConsumerNonRoot_BecomesTask(t *testing.T) {
}},
}
- endpoints, tasks, spans, _, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, tasks := converted.Endpoints, converted.Tasks
if len(endpoints) != 0 {
t.Fatalf("expected 0 endpoints, got %d", len(endpoints))
@@ -558,23 +595,24 @@ func TestConvertTraces_ConsumerNonRoot_BecomesTask(t *testing.T) {
if tasks[0].IsRoot {
t.Errorf("expected task.IsRoot == false, got true")
}
- wantTaskId := otelSpanIDToUUID(consumerSpanId)
+ wantTaskId := otelOccurrenceID(testProjectId, &tracepb.Span{TraceId: traceId, SpanId: consumerSpanId})
if tasks[0].Id != wantTaskId {
t.Errorf("expected task.Id %s (from span_id), got %s", wantTaskId, tasks[0].Id)
}
- wantDtId := otelTraceIDToUUID(traceId)
- if tasks[0].DistributedTraceId == nil || *tasks[0].DistributedTraceId != wantDtId {
- t.Errorf("expected task.DistributedTraceId %s (from trace_id), got %v", wantDtId, tasks[0].DistributedTraceId)
+ if tasks[0].TraceId != hex.EncodeToString(traceId) || tasks[0].SpanId != hex.EncodeToString(consumerSpanId) || tasks[0].ParentSpanId == "" {
+ t.Errorf("expected the task to carry its span's own ids, got %+v", tasks[0])
}
- if len(spans) != 1 {
- t.Fatalf("expected 1 span row, got %d", len(spans))
+ spans := converted.Spans
+ if len(spans) != 2 {
+ t.Fatalf("expected the consumer and its child to be stored, got %d", len(spans))
}
- if spans[0].TraceId != wantTaskId {
- t.Errorf("expected child span.TraceId == task.Id %s, got %s", wantTaskId, spans[0].TraceId)
+ child := spans[1]
+ if owner, ok := nearestPromotedAncestor(spans, child, map[string]bool{spanKey(tasks[0].TraceId, tasks[0].SpanId): true}); !ok || owner != tasks[0].SpanId {
+ t.Errorf("expected the child to reach task %s through its stored parent edge", wantTaskId)
}
- if spans[0].Attributes["db.system"] != "postgresql" {
- t.Errorf("expected child span attribute db.system 'postgresql', got %q", spans[0].Attributes["db.system"])
+ if getStringAttribute(child.OTLP.Attributes, "db.system") != "postgresql" {
+ t.Errorf("expected child span attribute db.system 'postgresql', got %q", getStringAttribute(child.OTLP.Attributes, "db.system"))
}
}
@@ -597,15 +635,15 @@ func TestConvertTraces_ConsoleCommand_BecomesTask(t *testing.T) {
}},
}
- _, tasks, _, _, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ tasks := convertTraces(context.Background(), nil, testProjectId, req).Tasks
if len(tasks) != 1 {
t.Fatalf("expected 1 task, got %d", len(tasks))
}
if !tasks[0].IsRoot {
t.Errorf("expected task.IsRoot == true, got false")
}
- if tasks[0].Id != otelTraceIDToUUID(traceId) {
- t.Errorf("expected task.Id == otelTraceIDToUUID(trace_id), got %s", tasks[0].Id)
+ if tasks[0].Id != otelOccurrenceID(testProjectId, &tracepb.Span{TraceId: traceId, SpanId: rootSpanId}) {
+ t.Errorf("expected the stable task occurrence ID, got %s", tasks[0].Id)
}
}
@@ -629,14 +667,15 @@ func TestConvertTraces_InlineGenAi_BecomesAiTrace(t *testing.T) {
}},
}
- endpoints, _, _, _, aiTraces, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, aiTraces := converted.Endpoints, converted.AiTraces
if len(endpoints) != 1 {
t.Fatalf("expected 1 endpoint, got %d", len(endpoints))
}
if !endpoints[0].IsRoot {
t.Errorf("expected endpoint.IsRoot == true")
}
- wantEndpointId := otelTraceIDToUUID(traceId)
+ wantEndpointId := otelOccurrenceID(testProjectId, &tracepb.Span{TraceId: traceId, SpanId: rootSpanId})
if endpoints[0].Id != wantEndpointId {
t.Errorf("expected endpoint.Id %s, got %s", wantEndpointId, endpoints[0].Id)
}
@@ -647,12 +686,12 @@ func TestConvertTraces_InlineGenAi_BecomesAiTrace(t *testing.T) {
if aiTraces[0].IsRoot {
t.Errorf("expected aiTrace.IsRoot == false")
}
- wantAiId := otelSpanIDToUUID(llmSpanId)
+ wantAiId := otelOccurrenceID(testProjectId, &tracepb.Span{TraceId: traceId, SpanId: llmSpanId})
if aiTraces[0].Id != wantAiId {
t.Errorf("expected aiTrace.Id %s, got %s", wantAiId, aiTraces[0].Id)
}
- if aiTraces[0].DistributedTraceId == nil || *aiTraces[0].DistributedTraceId != wantEndpointId {
- t.Errorf("expected aiTrace.DistributedTraceId %s, got %v", wantEndpointId, aiTraces[0].DistributedTraceId)
+ if aiTraces[0].TraceId != hex.EncodeToString(traceId) || aiTraces[0].SpanId != hex.EncodeToString(llmSpanId) || aiTraces[0].ParentSpanId == "" {
+ t.Errorf("expected the AI trace to carry its span's own ids, got %+v", aiTraces[0])
}
}
@@ -683,22 +722,23 @@ func TestConvertTraces_ExceptionOnConsumer_TraceTypeIsTask(t *testing.T) {
}},
}
- _, tasks, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ tasks, exceptions := converted.Tasks, converted.Exceptions
if len(tasks) != 1 || len(exceptions) != 1 {
t.Fatalf("expected 1 task + 1 exception, got %d / %d", len(tasks), len(exceptions))
}
if exceptions[0].TraceType != "task" {
t.Errorf("expected exception.TraceType == 'task', got %q", exceptions[0].TraceType)
}
- if exceptions[0].TraceId == nil || *exceptions[0].TraceId != tasks[0].Id {
- t.Errorf("expected exception.TraceId == task.Id %s, got %v", tasks[0].Id, exceptions[0].TraceId)
+ if exceptions[0].TraceId != tasks[0].TraceId || exceptions[0].SpanId != tasks[0].SpanId {
+ t.Errorf("expected the exception on the task's own span, got trace %s span %s", exceptions[0].TraceId, exceptions[0].SpanId)
}
}
-func TestConvertTraces_OrphanSpan_FallsBackToTraceId(t *testing.T) {
+func TestConvertTraces_OrphanSpan_KeepsSourceIdentity(t *testing.T) {
// A non-root span whose parent is not in this batch and the span itself
- // isn't classified as an entity. Should land in the spans table with
- // trace_id == otelTraceIDToUUID(trace_id) (orphan fallback path).
+ // isn't classified as an entity. It is stored with its source trace and
+ // parent IDs untouched, so the edge connects when the parent arrives.
traceId := []byte{0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f}
orphanParentId := []byte{0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67}
orphanSpanId := []byte{0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77}
@@ -717,13 +757,94 @@ func TestConvertTraces_OrphanSpan_FallsBackToTraceId(t *testing.T) {
}},
}
- _, _, spans, _, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, tasks, aiTraces := converted.Endpoints, converted.Tasks, converted.AiTraces
+ if len(endpoints) != 0 || len(tasks) != 0 || len(aiTraces) != 0 {
+ t.Fatalf("an unclassified orphan must not be promoted: %d endpoints / %d tasks / %d aiTraces", len(endpoints), len(tasks), len(aiTraces))
+ }
+ spans := converted.Spans
if len(spans) != 1 {
- t.Fatalf("expected 1 span row, got %d", len(spans))
+ t.Fatalf("expected 1 stored span, got %d", len(spans))
+ }
+ if spans[0].TraceId != hex.EncodeToString(traceId) || spans[0].SpanId != hex.EncodeToString(orphanSpanId) {
+ t.Errorf("source identity changed: trace %s span %s", spans[0].TraceId, spans[0].SpanId)
}
- want := otelTraceIDToUUID(traceId)
- if spans[0].TraceId != want {
- t.Errorf("expected orphan span.TraceId %s (fallback to OTel trace_id), got %s", want, spans[0].TraceId)
+ if spans[0].ParentSpanId != hex.EncodeToString(orphanParentId) {
+ t.Errorf("expected the source parent ID to be kept, got %v", spans[0].ParentSpanId)
+ }
+}
+
+func TestConvertTraces_ExceptionWithoutItsEndpointKeepsItsSpan(t *testing.T) {
+ traceId := []byte{0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, 0x40}
+ rootId := []byte{1, 1, 1, 1, 1, 1, 1, 1}
+ childId := []byte{2, 2, 2, 2, 2, 2, 2, 2}
+ now := uint64(1_700_000_000_000_000_000)
+ child := &tracepb.Span{TraceId: traceId, SpanId: childId, ParentSpanId: rootId, Name: "work", Kind: tracepb.Span_SPAN_KIND_INTERNAL,
+ StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000,
+ Events: []*tracepb.Span_Event{{Name: "exception", TimeUnixNano: now + 500_000, Attributes: []*commonpb.KeyValue{
+ strKV("exception.type", "RuntimeError"), strKV("exception.message", "boom"), strKV("exception.stacktrace", "RuntimeError: boom\n at work (app.js:10:5)"),
+ }}}}
+
+ // The child ends first, so an exporter flush between the two leaves it without its endpoint.
+ exceptions := convertTraces(context.Background(), nil, testProjectId, spanRequest(child)).Exceptions
+ if len(exceptions) != 1 {
+ t.Fatalf("expected 1 exception, got %d", len(exceptions))
+ }
+ // Its endpoint is not in the payload and ingest does not go looking. The trace and the span are all a read needs.
+ if exceptions[0].TraceId != hex.EncodeToString(traceId) || exceptions[0].SpanId != hex.EncodeToString(childId) || exceptions[0].TraceType != "" {
+ t.Fatalf("expected the exception's own trace and span and no kind, got %+v", exceptions[0])
+ }
+ for key := range exceptions[0].Attributes {
+ if strings.HasPrefix(key, "traceway.") {
+ t.Fatalf("ingest must not add attributes of its own: %v", exceptions[0].Attributes)
+ }
+ }
+}
+
+func TestConvertTraces_NestedServerSpanIsOneRequest(t *testing.T) {
+ traceId := []byte{0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, 0x30}
+ outerId := []byte{1, 1, 1, 1, 1, 1, 1, 1}
+ innerId := []byte{2, 2, 2, 2, 2, 2, 2, 2}
+ const hasIsRemote = uint32(tracepb.SpanFlags_SPAN_FLAGS_CONTEXT_HAS_IS_REMOTE_MASK)
+ const isRemote = uint32(tracepb.SpanFlags_SPAN_FLAGS_CONTEXT_IS_REMOTE_MASK)
+ http := []*commonpb.KeyValue{strKV("http.request.method", "GET"), strKV("http.route", "/users")}
+ now := uint64(1_700_000_000_000_000_000)
+ server := func(id, parent []byte, flags uint32) *tracepb.Span {
+ return &tracepb.Span{TraceId: traceId, SpanId: id, ParentSpanId: parent, Name: "GET /users", Kind: tracepb.Span_SPAN_KIND_SERVER,
+ Attributes: http, Flags: flags, StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000}
+ }
+ middleId := []byte{3, 3, 3, 3, 3, 3, 3, 3}
+ wrapper := func(id, parent []byte) *tracepb.Span {
+ return &tracepb.Span{TraceId: traceId, SpanId: id, ParentSpanId: parent, Name: "handle", Kind: tracepb.Span_SPAN_KIND_INTERNAL, StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000}
+ }
+ client := &tracepb.Span{TraceId: traceId, SpanId: outerId, Name: "GET", Kind: tracepb.Span_SPAN_KIND_CLIENT, StartTimeUnixNano: now, EndTimeUnixNano: now + 1_000_000}
+
+ tests := []struct {
+ name string
+ spans []*tracepb.Span
+ want int
+ }{
+ {"local parent by flags, same batch", []*tracepb.Span{server(outerId, nil, 0), server(innerId, outerId, hasIsRemote)}, 1},
+ {"local parent by flags, unknown kind in another batch", []*tracepb.Span{server(innerId, outerId, hasIsRemote)}, 1},
+ {"remote parent by flags, same batch", []*tracepb.Span{server(outerId, nil, 0), server(innerId, outerId, hasIsRemote|isRemote)}, 2},
+ {"no flags, parent in the same batch", []*tracepb.Span{server(outerId, nil, 0), server(innerId, outerId, 0)}, 1},
+ {"no flags, parent in another batch", []*tracepb.Span{server(innerId, outerId, 0)}, 1},
+ {"no flags, calling CLIENT span in the same batch", []*tracepb.Span{client, server(innerId, outerId, 0)}, 1},
+ {"no flags, plain wrapper span above the request", []*tracepb.Span{wrapper(outerId, nil), server(innerId, outerId, 0)}, 1},
+ {"local by flags, plain wrapper span above the request", []*tracepb.Span{wrapper(outerId, nil), server(innerId, outerId, hasIsRemote)}, 1},
+ {"nested through a plain span in between", []*tracepb.Span{server(outerId, nil, 0), wrapper(middleId, outerId), server(innerId, middleId, hasIsRemote)}, 1},
+ {"parent cycle does not hang", []*tracepb.Span{wrapper(outerId, middleId), wrapper(middleId, outerId), server(innerId, middleId, 0)}, 1},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ endpoints := convertTraces(context.Background(), nil, testProjectId, spanRequest(tt.spans...)).Endpoints
+ if len(endpoints) != tt.want {
+ t.Fatalf("expected %d endpoints, got %d", tt.want, len(endpoints))
+ }
+ if stored := convertTraces(context.Background(), nil, testProjectId, spanRequest(tt.spans...)).Spans; len(stored) != len(tt.spans) {
+ t.Fatalf("every span must still be stored, got %d of %d", len(stored), len(tt.spans))
+ }
+ })
}
}
@@ -820,7 +941,7 @@ func TestConvertTraces_HoneycombJsExceptionSymbolicates(t *testing.T) {
}
setFakeStore(t, nil)
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ exceptions := convertTraces(context.Background(), nil, testProjectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
@@ -871,7 +992,7 @@ func TestConvertTraces_JsExceptionResolvesWithSourceMap(t *testing.T) {
}},
}
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), tokenProject(projectId), projectId, req)
+ exceptions := convertTraces(context.Background(), tokenProject(projectId), projectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
@@ -948,20 +1069,21 @@ func honeycombExceptionSpanRequest(spanAttrs []*commonpb.KeyValue, events []*tra
}
}
-func TestConvertTraces_ExceptionSpanAttrs_CapturedAsTask(t *testing.T) {
+func TestConvertTraces_ExceptionSpanAttrs_CapturedWithoutAKind(t *testing.T) {
req := honeycombExceptionSpanRequest(honeycombExceptionSpanAttrs(), nil, strKV("telemetry.sdk.language", "webjs"))
setFakeStore(t, nil)
- endpoints, tasks, spans, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
- if len(endpoints) != 0 || len(tasks) != 0 || len(spans) != 0 {
- t.Fatalf("expected no entity/span rows, got %d endpoints / %d tasks / %d spans", len(endpoints), len(tasks), len(spans))
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, tasks, exceptions := converted.Endpoints, converted.Tasks, converted.Exceptions
+ if len(endpoints) != 0 || len(tasks) != 0 {
+ t.Fatalf("expected no entity rows, got %d endpoints / %d tasks", len(endpoints), len(tasks))
}
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
exc := exceptions[0]
- if exc.TraceType != "task" {
- t.Errorf("expected TraceType 'task', got %q", exc.TraceType)
+ if exc.TraceType != "" || exc.TraceId == "" || exc.SpanId == "" {
+ t.Errorf("an exception on a span that is no endpoint or task keeps its trace and span and names no kind, got %+v", exc)
}
want := "TypeError: discount rate must be finite\nassertValid()\n http://localhost:4173/assets/app.js:1:730\nhandleCheckout()\n http://localhost:4173/assets/app.js:1:1366"
if exc.StackTrace != want {
@@ -1001,7 +1123,8 @@ func TestConvertTraces_ExceptionSpanAttrs_StrippedFromEndpointRow(t *testing.T)
}
setFakeStore(t, nil)
- endpoints, _, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ converted := convertTraces(context.Background(), nil, testProjectId, req)
+ endpoints, exceptions := converted.Endpoints, converted.Exceptions
if len(endpoints) != 1 || len(exceptions) != 1 {
t.Fatalf("expected 1 endpoint + 1 exception, got %d / %d", len(endpoints), len(exceptions))
}
@@ -1029,7 +1152,7 @@ func TestConvertTraces_ExceptionEventAndSpanAttrs_EventWins(t *testing.T) {
req := honeycombExceptionSpanRequest(honeycombExceptionSpanAttrs(), events, strKV("telemetry.sdk.language", "webjs"))
setFakeStore(t, nil)
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ exceptions := convertTraces(context.Background(), nil, testProjectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected exactly 1 exception, got %d", len(exceptions))
}
@@ -1046,7 +1169,7 @@ func TestConvertTraces_ExceptionSpanAttrs_HeaderOnly(t *testing.T) {
req := honeycombExceptionSpanRequest(attrs, nil, strKV("telemetry.sdk.language", "webjs"))
setFakeStore(t, nil)
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ exceptions := convertTraces(context.Background(), nil, testProjectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
@@ -1059,7 +1182,7 @@ func TestConvertTraces_ExceptionSpanAttrs_NoLanguageAttr(t *testing.T) {
req := honeycombExceptionSpanRequest(honeycombExceptionSpanAttrs(), nil)
setFakeStore(t, nil)
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, req)
+ exceptions := convertTraces(context.Background(), nil, testProjectId, req).Exceptions
if len(exceptions) != 1 {
t.Fatalf("expected 1 exception, got %d", len(exceptions))
}
@@ -1128,7 +1251,7 @@ func TestExceptionHash_FourWayStability(t *testing.T) {
"firefox-raw": firefoxRaw,
"structured": structured,
} {
- _, _, _, exceptions, _, _ := convertTraces(context.Background(), nil, testProjectId, fourWayRequest(attrs, nil))
+ exceptions := convertTraces(context.Background(), nil, testProjectId, fourWayRequest(attrs, nil)).Exceptions
if len(exceptions) != 1 {
t.Fatalf("%s: expected 1 exception, got %d", name, len(exceptions))
}
@@ -1151,26 +1274,31 @@ func TestConvertTraces_FrontendFrameworkSuppressesEntityRows(t *testing.T) {
t.Fatal(err)
}
req := &coltracepb.ExportTraceServiceRequest{}
- if err := protojson.Unmarshal(raw, req); err != nil {
+ normalized, err := normalizeOTLPJSON(raw, req.ProtoReflect().Descriptor())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := protojson.Unmarshal(normalized, req); err != nil {
t.Fatal(err)
}
setFakeStore(t, nil)
reactProject := &models.Project{Id: testProjectId, Framework: "react"}
- endpoints, tasks, spans, exceptions, aiTraces, _ := convertTraces(context.Background(), reactProject, testProjectId, req)
- if len(endpoints) != 0 || len(tasks) != 0 || len(spans) != 0 || len(aiTraces) != 0 {
- t.Fatalf("expected no entity/span rows for a frontend-framework project, got %d endpoints / %d tasks / %d spans / %d aiTraces",
- len(endpoints), len(tasks), len(spans), len(aiTraces))
+ converted := convertTraces(context.Background(), reactProject, testProjectId, req)
+ endpoints, tasks, exceptions, aiTraces := converted.Endpoints, converted.Tasks, converted.Exceptions, converted.AiTraces
+ if len(endpoints) != 0 || len(tasks) != 0 || len(aiTraces) != 0 {
+ t.Fatalf("expected no entity rows for a frontend-framework project, got %d endpoints / %d tasks / %d aiTraces",
+ len(endpoints), len(tasks), len(aiTraces))
}
if len(exceptions) != 1 {
t.Fatalf("expected the exception to still be extracted, got %d", len(exceptions))
}
- if exceptions[0].TraceType != "task" {
- t.Errorf("expected TraceType 'task', got %q", exceptions[0].TraceType)
+ if exceptions[0].TraceType != "" {
+ t.Errorf("expected no kind when nothing was promoted, got %q", exceptions[0].TraceType)
}
backendProject := &models.Project{Id: testProjectId, Framework: "gin"}
- endpoints, _, _, _, _, _ = convertTraces(context.Background(), backendProject, testProjectId, req)
+ endpoints = convertTraces(context.Background(), backendProject, testProjectId, req).Endpoints
if len(endpoints) == 0 {
t.Error("expected non-frontend frameworks to keep promoting endpoint rows")
}
diff --git a/backend/app/controllers/otelcontrollers/trace_write_failures_test.go b/backend/app/controllers/otelcontrollers/trace_write_failures_test.go
new file mode 100644
index 00000000..8c99997f
--- /dev/null
+++ b/backend/app/controllers/otelcontrollers/trace_write_failures_test.go
@@ -0,0 +1,223 @@
+//go:build !telemetry_ch && !telemetry_duckdb && !transactional_pg
+
+package otelcontrollers
+
+import (
+ "bytes"
+ "context"
+ "errors"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/dbtest"
+ "github.com/tracewayapp/traceway/backend/app/hooks"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+var traceTables = []string{"spans_v2", "endpoints_v2", "tasks_v2", "exceptions_v2", "ai_traces_v2"}
+
+func traceTableCounts(t *testing.T) []int {
+ t.Helper()
+ counts := make([]int, len(traceTables))
+ for i, table := range traceTables {
+ if err := db.TelemetryDB.QueryRow("SELECT COUNT(*) FROM " + table).Scan(&counts[i]); err != nil {
+ t.Fatal(err)
+ }
+ }
+ return counts
+}
+
+func everyTableRequest() (*coltracepb.ExportTraceServiceRequest, *tracepb.Span, *tracepb.Span) {
+ trace := uuid.New()
+ now := uint64(time.Now().UnixNano())
+ endpoint := &tracepb.Span{TraceId: trace[:], SpanId: []byte{1, 0, 0, 0, 0, 0, 0, 1}, Name: "GET /orders", Kind: tracepb.Span_SPAN_KIND_SERVER,
+ StartTimeUnixNano: now, EndTimeUnixNano: now + 9000, Attributes: []*commonpb.KeyValue{strKV("http.request.method", "GET"), strKV("http.route", "/orders")}}
+ child := &tracepb.Span{TraceId: trace[:], SpanId: []byte{2, 0, 0, 0, 0, 0, 0, 2}, ParentSpanId: endpoint.SpanId, Name: "query", Kind: tracepb.Span_SPAN_KIND_INTERNAL,
+ StartTimeUnixNano: now + 1, EndTimeUnixNano: now + 500, Events: []*tracepb.Span_Event{{Name: "exception", TimeUnixNano: now + 2,
+ Attributes: []*commonpb.KeyValue{strKV("exception.type", "PgError"), strKV("exception.message", "deadlock detected")}}}}
+ task := &tracepb.Span{TraceId: trace[:], SpanId: []byte{3, 0, 0, 0, 0, 0, 0, 3}, ParentSpanId: endpoint.SpanId, Name: "process job", Kind: tracepb.Span_SPAN_KIND_CONSUMER,
+ StartTimeUnixNano: now + 3, EndTimeUnixNano: now + 800}
+ llm := &tracepb.Span{TraceId: trace[:], SpanId: []byte{4, 0, 0, 0, 0, 0, 0, 4}, ParentSpanId: endpoint.SpanId, Name: "chat model", Kind: tracepb.Span_SPAN_KIND_CLIENT,
+ StartTimeUnixNano: now + 4, EndTimeUnixNano: now + 900, Attributes: []*commonpb.KeyValue{strKV("gen_ai.system", "anthropic"), strKV("gen_ai.request.model", "claude-sonnet-5")}}
+ return spanRequest(endpoint, child, task, llm), endpoint, child
+}
+
+func exportTraces(t *testing.T, request *coltracepb.ExportTraceServiceRequest) (*httptest.ResponseRecorder, *coltracepb.ExportTraceServiceResponse) {
+ t.Helper()
+ payload, err := proto.Marshal(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ recorder := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(recorder)
+ c.Request = httptest.NewRequest(http.MethodPost, "/api/otel/v1/traces", bytes.NewReader(payload))
+ c.Request.Header.Set("Content-Type", "application/x-protobuf")
+ c.Set(middleware.ProjectIdContextKey, testProjectId)
+ OtelController.ExportTraces(c)
+ c.Writer.WriteHeaderNow()
+ var response coltracepb.ExportTraceServiceResponse
+ if recorder.Code == http.StatusOK {
+ if err := proto.Unmarshal(recorder.Body.Bytes(), &response); err != nil {
+ t.Fatal(err)
+ }
+ }
+ return recorder, &response
+}
+
+func setupTraceWrites(t *testing.T) {
+ t.Helper()
+ dbtest.SetupSQLite(t)
+ previousConfig, previousStore := config.Config, traceStore
+ config.Config = &config.Cfg{}
+ setFakeStore(t, nil)
+ gin.SetMode(gin.TestMode)
+ t.Cleanup(func() { config.Config, traceStore = previousConfig, previousStore })
+}
+
+func TestExportTracesFailureAtEachWriteStage(t *testing.T) {
+ stages := []struct {
+ name string
+ inject func(error)
+ persisted []int
+ }{
+ {"spans", func(err error) {
+ traceStore.spans = func(context.Context, []models.OtelSpan) ([]shared.SpanOwner, error) { return nil, err }
+ }, []int{0, 0, 0, 0, 0}},
+ {"endpoints", func(err error) { traceStore.endpoints = func(context.Context, []models.Endpoint) error { return err } }, []int{4, 0, 0, 0, 0}},
+ {"tasks", func(err error) { traceStore.tasks = func(context.Context, []models.Task) error { return err } }, []int{4, 1, 0, 0, 0}},
+ {"exceptions", func(err error) {
+ traceStore.exceptions = func(context.Context, []models.ExceptionStackTrace) error { return err }
+ }, []int{4, 1, 1, 0, 0}},
+ {"ai traces", func(err error) { traceStore.aiTraces = func(context.Context, []models.AiTrace) error { return err } }, []int{4, 1, 1, 1, 0}},
+ }
+ failures := []struct {
+ err error
+ status int
+ retryAfter string
+ }{
+ {errors.New("no such column: resource_pb"), http.StatusInternalServerError, ""},
+ {fmt.Errorf("insert timed out: %w", context.DeadlineExceeded), http.StatusServiceUnavailable, "2"},
+ }
+ for _, stage := range stages {
+ for _, failure := range failures {
+ t.Run(fmt.Sprintf("%s/%d", stage.name, failure.status), func(t *testing.T) {
+ setupTraceWrites(t)
+ stage.inject(failure.err)
+ request, _, _ := everyTableRequest()
+ recorder, _ := exportTraces(t, request)
+ if recorder.Code != failure.status || recorder.Header().Get("Retry-After") != failure.retryAfter {
+ t.Fatalf("status %d Retry-After %q, want %d %q", recorder.Code, recorder.Header().Get("Retry-After"), failure.status, failure.retryAfter)
+ }
+ if got := traceTableCounts(t); fmt.Sprint(got) != fmt.Sprint(stage.persisted) {
+ t.Fatalf("persisted rows %v, want %v in %v", got, stage.persisted, traceTables)
+ }
+ })
+ }
+ }
+}
+
+func TestExportTracesRetryDuplicatesAnalyticsButNotTheGraph(t *testing.T) {
+ setupTraceWrites(t)
+ request, endpoint, _ := everyTableRequest()
+ for range 2 {
+ if recorder, _ := exportTraces(t, request); recorder.Code != http.StatusOK {
+ t.Fatalf("export failed: %d", recorder.Code)
+ }
+ }
+ // Writes are separate and append-only, so a retried export is stored twice. Only graph reads deduplicate.
+ if got := traceTableCounts(t); fmt.Sprint(got) != fmt.Sprint([]int{8, 2, 2, 2, 2}) {
+ t.Fatalf("a retried export is at-least-once: got %v in %v", got, traceTables)
+ }
+ id := otelOccurrenceID(testProjectId, endpoint)
+ stored, err := telemetry.EndpointRepository.FindById(context.Background(), testProjectId, id, nil)
+ if err != nil || stored == nil {
+ t.Fatalf("missing endpoint: %v", err)
+ }
+ graph, err := telemetry.SpanRepository.FindGraph(context.Background(), shared.NewSpanLookup(testProjectId, stored.TraceId, stored.SpanId, stored.RecordedAt))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(graph.Spans) != 3 {
+ t.Fatalf("graph reads must deduplicate the retry: %d spans", len(graph.Spans))
+ }
+}
+
+func TestExportTracesReportsRowsStorageRejected(t *testing.T) {
+ setupTraceWrites(t)
+ store := traceStore.spans
+ traceStore.spans = func(ctx context.Context, spans []models.OtelSpan) ([]shared.SpanOwner, error) {
+ stored, err := store(ctx, spans[:len(spans)-2])
+ for _, span := range spans[len(spans)-2:] {
+ stored = append(stored, shared.SpanOwner{ProjectId: span.ProjectId, TraceId: span.TraceId, SpanId: span.SpanId})
+ }
+ return stored, err
+ }
+ request, _, _ := everyTableRequest()
+ request.ResourceSpans[0].ScopeSpans[0].Spans = append(request.ResourceSpans[0].ScopeSpans[0].Spans, &tracepb.Span{TraceId: []byte{1}, SpanId: []byte{1}})
+ recorder, response := exportTraces(t, request)
+ partial := response.GetPartialSuccess()
+ if recorder.Code != http.StatusOK || partial.GetRejectedSpans() != 3 {
+ t.Fatalf("status %d, partial success %v", recorder.Code, partial)
+ }
+ if got := traceTableCounts(t); fmt.Sprint(got) != fmt.Sprint([]int{2, 1, 0, 1, 0}) {
+ t.Fatalf("rejected task/AI spans must not produce projections: %v", got)
+ }
+
+ if !strings.Contains(partial.GetErrorMessage(), "1 spans need valid non-zero trace and span IDs") || !strings.Contains(partial.GetErrorMessage(), "2 spans could not be stored") {
+ t.Fatalf("the exporter must learn why spans were rejected: %q", partial.GetErrorMessage())
+ }
+}
+
+func TestExportTracesCountsInvalidIDsWhenQuotaSuppressesWrites(t *testing.T) {
+ for _, tt := range []struct {
+ name string
+ permission hooks.IngestPermission
+ counts []int
+ }{
+ {"exceptions only", hooks.IngestPermission{Exceptions: true}, []int{0, 0, 0, 1, 0}},
+ {"data only", hooks.IngestPermission{Data: true}, []int{4, 1, 1, 0, 1}},
+ {"neither", hooks.IngestPermission{}, []int{0, 0, 0, 0, 0}},
+ } {
+ t.Run(tt.name, func(t *testing.T) {
+ setupTraceWrites(t)
+ hooks.RegisterIngestPermissionHook(func(int) hooks.IngestPermission { return tt.permission })
+ t.Cleanup(func() { hooks.RegisterIngestPermissionHook(nil) })
+ request, _, _ := everyTableRequest()
+ request.ResourceSpans[0].ScopeSpans[0].Spans = append(request.ResourceSpans[0].ScopeSpans[0].Spans, &tracepb.Span{})
+ payload, err := proto.Marshal(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ recorder := httptest.NewRecorder()
+ c, _ := gin.CreateTestContext(recorder)
+ c.Request = httptest.NewRequest(http.MethodPost, "/api/otel/v1/traces", bytes.NewReader(payload))
+ c.Request.Header.Set("Content-Type", "application/x-protobuf")
+ c.Set(middleware.ProjectIdContextKey, testProjectId)
+ orgID := 1
+ c.Set(middleware.ProjectContextKey, &models.Project{Id: testProjectId, OrganizationId: &orgID})
+ OtelController.ExportTraces(c)
+ var response coltracepb.ExportTraceServiceResponse
+ if recorder.Code != http.StatusOK || proto.Unmarshal(recorder.Body.Bytes(), &response) != nil || response.GetPartialSuccess().GetRejectedSpans() != 1 {
+ t.Fatalf("status %d, response %v", recorder.Code, &response)
+ }
+ if got := traceTableCounts(t); fmt.Sprint(got) != fmt.Sprint(tt.counts) {
+ t.Fatalf("persisted rows %v, want %v", got, tt.counts)
+ }
+ })
+ }
+}
diff --git a/backend/app/controllers/routes.go b/backend/app/controllers/routes.go
index ef6d6b91..9c27c1bc 100644
--- a/backend/app/controllers/routes.go
+++ b/backend/app/controllers/routes.go
@@ -166,7 +166,11 @@ func RegisterControllers(router *gin.RouterGroup) {
router.POST("/ai-conversations/conversation", middleware.UseAppAuth, middleware.RequireProjectAccess, AiTraceController.GetConversationDetail)
router.POST("/ai-users/grouped", middleware.UseAppAuth, middleware.RequireProjectAccess, AiTraceController.FindAiUsers)
- router.POST("/distributed-traces/:distributedTraceId", middleware.UseAppAuth, DistributedTraceController.GetDistributedTrace)
+ router.POST("/distributed-traces/:traceId", middleware.UseAppAuth, DistributedTraceController.GetDistributedTrace)
+ router.GET("/otel/spans/:traceId/:spanId", middleware.UseAppAuth, middleware.RequireProjectAccess, GetOtelSpan)
+ router.POST("/spans/search", middleware.UseAppAuth, middleware.RequireProjectAccess, SpanExplorerController.Search)
+ router.GET("/spans/traces/:traceId", middleware.UseAppAuth, middleware.RequireProjectAccess, SpanExplorerController.GetTrace)
+ router.GET("/spans/traces/:traceId/spans/:spanId/attributes", middleware.UseAppAuth, middleware.RequireProjectAccess, SpanExplorerController.GetSpanAttributes)
router.POST("/logs", middleware.UseAppAuth, middleware.RequireProjectAccess, LogController.List)
diff --git a/backend/app/controllers/span_explorer.controller.go b/backend/app/controllers/span_explorer.controller.go
new file mode 100644
index 00000000..5bc86499
--- /dev/null
+++ b/backend/app/controllers/span_explorer.controller.go
@@ -0,0 +1,279 @@
+package controllers
+
+import (
+ "context"
+ "database/sql"
+ "encoding/hex"
+ "errors"
+ "math"
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ traceway "go.tracewayapp.com"
+)
+
+type spanExplorerController struct{}
+
+var SpanExplorerController = spanExplorerController{}
+
+type SpanAttributeFilterRequest struct {
+ Key string `json:"key"`
+ Value string `json:"value"`
+}
+
+type SpanSearchRequest struct {
+ FromDate time.Time `json:"fromDate"`
+ ToDate time.Time `json:"toDate"`
+ OrderBy string `json:"orderBy"`
+ ServiceName string `json:"serviceName"`
+ Name string `json:"name"`
+ TraceId string `json:"traceId"`
+ Kind *int32 `json:"kind"`
+ Status *int32 `json:"status"`
+ MinDurationMs float64 `json:"minDurationMs"`
+ MaxDurationMs float64 `json:"maxDurationMs"`
+ AttributeFilters []SpanAttributeFilterRequest `json:"attributeFilters"`
+ Pagination PaginationParams `json:"pagination"`
+}
+
+type SpanSearchResponse struct {
+ Data []models.Span `json:"data"`
+ Pagination Pagination `json:"pagination"`
+ Services []string `json:"services"`
+}
+
+type SpanAttributesResponse struct {
+ Attributes map[string]string `json:"attributes"`
+ AttributesOmitted bool `json:"attributesOmitted,omitempty"`
+}
+
+type SpanTraceProject struct {
+ Id uuid.UUID `json:"id"`
+ Name string `json:"name"`
+}
+
+type SpanTraceResponse struct {
+ SpanGraphStatus *models.SpanGraphStatus `json:"spanGraphStatus"`
+ Spans []models.Span `json:"spans"`
+ // Projects names every project that holds spans of the trace, so rows from another project can say where they live.
+ Projects []SpanTraceProject `json:"projects"`
+}
+
+// The spans table is the largest one. A search reads every span of the project in the range, so the range is capped.
+const spanSearchMaxRange = 31*24*time.Hour + 5*time.Minute
+
+const spanSearchTooSlow = "The search took too long. Narrow the time range or add a filter."
+
+func validTraceHex(id string) bool {
+ decoded, err := hex.DecodeString(id)
+ return err == nil && len(decoded) == 16 && strings.Trim(id, "0") != ""
+}
+
+// An OTel span id is 8 bytes. A span of the native protocol keeps its client's 16 byte id.
+func validSpanHex(id string) bool {
+ decoded, err := hex.DecodeString(id)
+ return err == nil && (len(decoded) == 8 || len(decoded) == 16)
+}
+
+func (s spanExplorerController) searchParams(request SpanSearchRequest) (shared.OtelSpanSearch, string) {
+ search := shared.OtelSpanSearch{From: request.FromDate, To: request.ToDate, OrderBy: request.OrderBy, Service: request.ServiceName,
+ Name: request.Name, TraceId: shared.NormalizeTraceId(request.TraceId), Kind: request.Kind, Status: request.Status,
+ MinDuration: time.Duration(request.MinDurationMs * float64(time.Millisecond)), MaxDuration: time.Duration(request.MaxDurationMs * float64(time.Millisecond)),
+ Page: request.Pagination.Page, PageSize: request.Pagination.PageSize}
+ switch {
+ case request.FromDate.IsZero() || request.ToDate.IsZero():
+ return search, "A time range is required."
+ case request.ToDate.Before(request.FromDate):
+ return search, "The time range ends before it starts."
+ case request.ToDate.Sub(request.FromDate) > spanSearchMaxRange:
+ return search, "Span search covers at most 31 days. Narrow the time range."
+ case search.TraceId != "" && !validTraceHex(search.TraceId):
+ return search, "A trace ID is 32 hexadecimal characters."
+ case request.Kind != nil && (*request.Kind < 0 || *request.Kind > 5), request.Status != nil && (*request.Status < 0 || *request.Status > 2):
+ return search, "Unknown span kind or status."
+ case len(request.AttributeFilters) > shared.MaxOtelSearchAttributeFilters:
+ return search, "Too many attribute filters."
+ case request.MinDurationMs < 0 || request.MaxDurationMs < 0 ||
+ math.IsNaN(request.MinDurationMs) || math.IsNaN(request.MaxDurationMs) ||
+ request.MinDurationMs >= float64(math.MaxInt64)/float64(time.Millisecond) ||
+ request.MaxDurationMs >= float64(math.MaxInt64)/float64(time.Millisecond):
+ return search, "Duration limits must be non-negative and fit within 292 years."
+ case request.MaxDurationMs > 0 && request.MinDurationMs > request.MaxDurationMs:
+ return search, "The maximum duration must be at least the minimum duration."
+ }
+ for _, filter := range request.AttributeFilters {
+ if filter.Key == "" {
+ continue
+ }
+ if strings.ContainsAny(filter.Key, `"\`) || len(filter.Key) > 256 {
+ return search, "Attribute keys cannot contain quotes or backslashes."
+ }
+ // A ClickHouse Map answers '' for a key it does not hold, so an empty value would match every span without the attribute there and none elsewhere.
+ if filter.Value == "" {
+ return search, "An attribute filter needs a value."
+ }
+ search.Attributes = append(search.Attributes, shared.OtelAttributeFilter{Key: filter.Key, Value: filter.Value})
+ }
+ return search, ""
+}
+
+func spanSearchTimedOut(ctx context.Context, err error) bool {
+ return errors.Is(err, context.DeadlineExceeded) || ctx.Err() != nil || telemetry.OtelSpanRepository.IsReadLimitError(err)
+}
+
+func (s spanExplorerController) Search(c *gin.Context) {
+ projectId, err := middleware.GetProjectId(c)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ var request SpanSearchRequest
+ if err := c.ShouldBindJSON(&request); err != nil {
+ middleware.RejectBindError(c, err, err.Error())
+ return
+ }
+ search, problem := s.searchParams(request)
+ if problem != "" {
+ c.JSON(http.StatusUnprocessableEntity, gin.H{"error": problem})
+ return
+ }
+ search.ProjectId = projectId
+ ctx, cancel := context.WithTimeout(c.Request.Context(), 20*time.Second)
+ defer cancel()
+
+ span := traceway.StartSpan(c, "searching spans")
+ found, total, err := telemetry.OtelSpanRepository.Search(ctx, search)
+ span.End()
+ if err != nil {
+ if spanSearchTimedOut(ctx, err) {
+ c.JSON(http.StatusUnprocessableEntity, gin.H{"error": spanSearchTooSlow})
+ return
+ }
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error searching spans: %w", err))
+ return
+ }
+ services, err := telemetry.OtelSpanRepository.Services(ctx, projectId, search.From, search.To)
+ if err != nil {
+ if !spanSearchTimedOut(ctx, err) {
+ traceway.CaptureException(traceway.NewStackTraceErrorf("error listing span services: %w", err))
+ }
+ services = []string{}
+ }
+ spans := make([]models.Span, len(found))
+ for i, item := range found {
+ spans[i] = item.Span
+ }
+ c.JSON(http.StatusOK, SpanSearchResponse{Data: spans, Services: services, Pagination: Pagination{
+ Page: search.Page, PageSize: search.PageSize, Total: int64(total), TotalPages: (int64(total) + int64(search.PageSize) - 1) / int64(search.PageSize)}})
+}
+
+// organizationProjects lists what a whole trace is read from: every project of the current project's organization.
+// Services of one trace usually report to different projects, and membership of the organization is what grants
+// read access to each of them. Other organizations of the same user stay out.
+func organizationProjects(c *gin.Context, projectId uuid.UUID) ([]*models.Project, error) {
+ projects, err := db.ExecuteTransaction(func(tx *sql.Tx) ([]*models.Project, error) {
+ return transactional.ProjectRepository.FindByUserId(tx, middleware.GetUserId(c))
+ })
+ if err != nil {
+ return nil, err
+ }
+ var organization *int
+ for _, project := range projects {
+ if project.Id == projectId {
+ organization = project.OrganizationId
+ }
+ }
+ readable := make([]*models.Project, 0, len(projects))
+ for _, project := range projects {
+ if project.Id == projectId || (organization != nil && project.OrganizationId != nil && *project.OrganizationId == *organization) {
+ readable = append(readable, project)
+ }
+ }
+ if len(readable) == 0 {
+ readable = append(readable, &models.Project{Id: projectId})
+ }
+ return readable, nil
+}
+
+func (s spanExplorerController) GetTrace(c *gin.Context) {
+ projectId, err := middleware.GetProjectId(c)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ traceId := strings.ToLower(c.Param("traceId"))
+ at, err := time.Parse(time.RFC3339Nano, c.Query("at"))
+ if !validTraceHex(traceId) || err != nil {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "A trace needs a 32 character hexadecimal ID and an RFC 3339 `at` time."})
+ return
+ }
+ readable, err := organizationProjects(c, projectId)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error listing the projects a trace is read from: %w", err))
+ return
+ }
+ projectIds := make([]uuid.UUID, len(readable))
+ for i, project := range readable {
+ projectIds[i] = project.Id
+ }
+ span := traceway.StartSpan(c, "loading trace spans")
+ graph, err := telemetry.SpanRepository.FindTrace(c, projectIds, traceId, at)
+ span.End()
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error loading trace spans: %w", err))
+ return
+ }
+ if len(graph.Spans) == 0 && graph.Status.State == models.SpanGraphComplete {
+ c.JSON(http.StatusNotFound, gin.H{"error": "Trace not found"})
+ return
+ }
+ holds := make(map[uuid.UUID]bool)
+ for _, found := range graph.Spans {
+ holds[found.ProjectId] = true
+ }
+ projects := []SpanTraceProject{}
+ for _, project := range readable {
+ if holds[project.Id] {
+ projects = append(projects, SpanTraceProject{Id: project.Id, Name: project.Name})
+ }
+ }
+ c.JSON(http.StatusOK, SpanTraceResponse{SpanGraphStatus: &graph.Status, Spans: graph.Spans, Projects: projects})
+}
+
+// GetSpanAttributes serves the popover of one span in the whole trace view, which loads without attributes.
+func (s spanExplorerController) GetSpanAttributes(c *gin.Context) {
+ projectId, err := middleware.GetProjectId(c)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("RequireProjectAccess middleware must be applied: %w", err))
+ return
+ }
+ traceId, spanId := strings.ToLower(c.Param("traceId")), strings.ToLower(c.Param("spanId"))
+ at, err := time.Parse(time.RFC3339Nano, c.Query("at"))
+ if !validTraceHex(traceId) || !validSpanHex(spanId) || err != nil {
+ c.JSON(http.StatusBadRequest, gin.H{"error": "Span attributes need a 32 character trace ID, a 16 or 32 character span ID and an RFC 3339 `at` time."})
+ return
+ }
+ found, err := telemetry.SpanRepository.FindSpanAttributes(c, projectId, traceId, spanId, at)
+ if err != nil {
+ c.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("error loading span attributes: %w", err))
+ return
+ }
+ if found == nil {
+ c.JSON(http.StatusNotFound, gin.H{"error": "Span not found"})
+ return
+ }
+ attributes := found.Attributes
+ if attributes == nil {
+ attributes = map[string]string{}
+ }
+ c.JSON(http.StatusOK, SpanAttributesResponse{Attributes: attributes, AttributesOmitted: found.Omitted != ""})
+}
diff --git a/backend/app/controllers/span_explorer_test.go b/backend/app/controllers/span_explorer_test.go
new file mode 100644
index 00000000..b72e92e3
--- /dev/null
+++ b/backend/app/controllers/span_explorer_test.go
@@ -0,0 +1,175 @@
+//go:build !telemetry_ch && !transactional_pg && !telemetry_duckdb
+
+package controllers
+
+import (
+ "context"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func TestSpanExplorerEndpoints(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ user, org := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "Explorer", "react", org)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+ trace, now := uuid.New(), time.Now().UTC()
+ traceHex := hex.EncodeToString(trace[:])
+ span := models.OtelSpan{Span: models.Span{ProjectId: project.Id, TraceId: traceHex, SpanId: "0102030405060708", Name: "documentLoad",
+ ServiceName: "web", SpanKind: 1, StartTime: now.Add(-time.Minute), Duration: time.Second, Attributes: map[string]string{"url.path": "/checkout"}}}
+ if _, err := telemetry.OtelSpanRepository.InsertAsync(context.Background(), []models.OtelSpan{span}); err != nil {
+ t.Fatal(err)
+ }
+ call := func(method, path, body string, params gin.Params, handler gin.HandlerFunc) (int, []byte) {
+ c, response := newControllerTestContext(t, nil, user, method, path, body)
+ c.Set(middleware.ProjectIdContextKey, project.Id)
+ c.Params = params
+ handler(c)
+ return response.Code, response.Body.Bytes()
+ }
+ search := func(body string) (int, SpanSearchResponse) {
+ code, raw := call(http.MethodPost, "/spans/search", body, nil, SpanExplorerController.Search)
+ var parsed SpanSearchResponse
+ _ = json.Unmarshal(raw, &parsed)
+ return code, parsed
+ }
+ pagination := `"pagination":{"page":1,"pageSize":50}`
+ window := fmt.Sprintf(`"fromDate":%q,"toDate":%q`, now.Add(-time.Hour).Format(time.RFC3339Nano), now.Add(time.Minute).Format(time.RFC3339Nano))
+
+ for name, body := range map[string]string{
+ "no time range": `{` + pagination + `}`,
+ "range over a month": fmt.Sprintf(`{"fromDate":%q,"toDate":%q,%s}`, now.Add(-40*24*time.Hour).Format(time.RFC3339Nano), now.Format(time.RFC3339Nano), pagination),
+ "malformed trace id": `{` + window + `,"traceId":"xyz",` + pagination + `}`,
+ "quoted attribute key": `{` + window + `,"attributeFilters":[{"key":"a\"b","value":"x"}],` + pagination + `}`,
+ "empty attribute value": `{` + window + `,"attributeFilters":[{"key":"url.path","value":""}],` + pagination + `}`,
+ "negative duration": `{` + window + `,"minDurationMs":-1,` + pagination + `}`,
+ "overflowing duration": `{` + window + `,"maxDurationMs":1e20,` + pagination + `}`,
+ "inverted duration": `{` + window + `,"minDurationMs":100,"maxDurationMs":50,` + pagination + `}`,
+ } {
+ if code, _ := search(body); code != http.StatusUnprocessableEntity {
+ t.Fatalf("%s: a search is always bounded and validated, got %d", name, code)
+ }
+ }
+ code, found := search(`{` + window + `,"attributeFilters":[{"key":"url.path","value":"/checkout"}],` + pagination + `}`)
+ if code != http.StatusOK || found.Pagination.Total != 1 || len(found.Data) != 1 || found.Data[0].Name != "documentLoad" || len(found.Services) != 1 || found.Services[0] != "web" {
+ t.Fatalf("a browser project's spans must be searchable: %d %+v", code, found)
+ }
+
+ traceParams := gin.Params{{Key: "traceId", Value: traceHex}}
+ if code, _ := call(http.MethodGet, "/spans/traces/"+traceHex, "", traceParams, SpanExplorerController.GetTrace); code != http.StatusBadRequest {
+ t.Fatalf("a trace read without a time must be refused, got %d", code)
+ }
+ code, raw := call(http.MethodGet, "/spans/traces/"+traceHex+"?at="+now.Format(time.RFC3339Nano), "", traceParams, SpanExplorerController.GetTrace)
+ var whole SpanTraceResponse
+ if err := json.Unmarshal(raw, &whole); err != nil || code != http.StatusOK || len(whole.Spans) != 1 || len(whole.Spans[0].Attributes) != 0 {
+ t.Fatalf("whole trace, without attributes: %d %s", code, raw)
+ }
+ if code, _ := call(http.MethodGet, "/spans/traces/"+traceHex+"?at="+now.Add(-72*time.Hour).Format(time.RFC3339Nano), "", traceParams, SpanExplorerController.GetTrace); code != http.StatusNotFound {
+ t.Fatalf("anchored three days away the trace is outside the window, got %d", code)
+ }
+ spanParams := gin.Params{{Key: "traceId", Value: traceHex}, {Key: "spanId", Value: span.SpanId}}
+ at := "?at=" + now.Format(time.RFC3339Nano)
+ if code, _ := call(http.MethodGet, "/spans/traces/x/spans/y/attributes", "", spanParams, SpanExplorerController.GetSpanAttributes); code != http.StatusBadRequest {
+ t.Fatalf("span attributes need a time, got %d", code)
+ }
+ code, raw = call(http.MethodGet, "/spans/traces/x/spans/y/attributes"+at, "", spanParams, SpanExplorerController.GetSpanAttributes)
+ var popover SpanAttributesResponse
+ if err := json.Unmarshal(raw, &popover); err != nil || code != http.StatusOK || popover.Attributes["url.path"] != "/checkout" {
+ t.Fatalf("the popover loads the attributes the trace read left behind: %d %s", code, raw)
+ }
+ unknown := gin.Params{{Key: "traceId", Value: traceHex}, {Key: "spanId", Value: "ffffffffffffffff"}}
+ if code, _ := call(http.MethodGet, "/spans/traces/x/spans/y/attributes"+at, "", unknown, SpanExplorerController.GetSpanAttributes); code != http.StatusNotFound {
+ t.Fatalf("unknown span, got %d", code)
+ }
+ malformed := gin.Params{{Key: "traceId", Value: traceHex}, {Key: "spanId", Value: "not-hex"}}
+ if code, _ := call(http.MethodGet, "/spans/traces/x/spans/y/attributes"+at, "", malformed, SpanExplorerController.GetSpanAttributes); code != http.StatusBadRequest {
+ t.Fatalf("malformed span id, got %d", code)
+ }
+ if code, _ := call(http.MethodGet, "/otel/spans/x/y", "", spanParams, GetOtelSpan); code != http.StatusBadRequest {
+ t.Fatalf("the OTLP span read needs a time too, got %d", code)
+ }
+ if code, _ := call(http.MethodGet, "/otel/spans/x/y?at="+now.Format(time.RFC3339Nano), "", spanParams, GetOtelSpan); code != http.StatusOK {
+ t.Fatalf("OTLP span with a time: %d", code)
+ }
+}
+
+func TestSpanTraceReadsTheWholeOrganizationAndNothingBeyond(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ user, org := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ _, otherOrg := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ _, strangerOrg := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ // The same person also belongs to a second organization. A trace is read inside one organization at a time.
+ if _, err := transactional.OrganizationRepository.AddUser(tx, otherOrg, user, "user"); err != nil {
+ t.Fatal(err)
+ }
+ createProject := func(name string, organization int) *models.Project {
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, name, "opentelemetry", organization)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return project
+ }
+ gateway, payments := createProject("gateway", org), createProject("payments", org)
+ sameUserOtherOrg, stranger := createProject("elsewhere", otherOrg), createProject("stranger", strangerOrg)
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+
+ trace, now := uuid.New(), time.Now().UTC()
+ traceHex := hex.EncodeToString(trace[:])
+ rootID := "0000000000000001"
+ stored := func(project *models.Project, spanId, name, parent string, offset time.Duration) models.OtelSpan {
+ return models.OtelSpan{Span: models.Span{ProjectId: project.Id, TraceId: traceHex, SpanId: spanId, ParentSpanId: parent, Name: name,
+ ServiceName: project.Name, SpanKind: 2, StartTime: now.Add(offset), Duration: time.Second}}
+ }
+ if _, err := telemetry.OtelSpanRepository.InsertAsync(context.Background(), []models.OtelSpan{
+ stored(gateway, "0000000000000001", "POST /api/checkout", "", -time.Minute),
+ stored(payments, "0000000000000002", "POST /charge", rootID, -59*time.Second),
+ stored(sameUserOtherOrg, "0000000000000003", "another organization", rootID, -58*time.Second),
+ stored(stranger, "0000000000000004", "not a member", rootID, -57*time.Second),
+ }); err != nil {
+ t.Fatal(err)
+ }
+
+ c, response := newControllerTestContext(t, nil, user, http.MethodGet, "/spans/traces/"+traceHex+"?at="+now.Format(time.RFC3339Nano), "")
+ c.Set(middleware.ProjectIdContextKey, payments.Id)
+ c.Params = gin.Params{{Key: "traceId", Value: traceHex}}
+ SpanExplorerController.GetTrace(c)
+ var whole SpanTraceResponse
+ if err := json.Unmarshal(response.Body.Bytes(), &whole); err != nil || response.Code != http.StatusOK {
+ t.Fatalf("%d %v %s", response.Code, err, response.Body.String())
+ }
+ if len(whole.Spans) != 2 || whole.Spans[0].Name != "POST /api/checkout" || whole.Spans[0].ProjectId != gateway.Id || whole.Spans[1].Name != "POST /charge" {
+ t.Fatalf("opened from payments, the trace holds the gateway span too and nothing from outside the organization: %s", response.Body.String())
+ }
+ if len(whole.Projects) != 2 || whole.Projects[0].Name != "gateway" || whole.Projects[1].Name != "payments" {
+ t.Fatalf("projects that hold spans are named: %+v", whole.Projects)
+ }
+}
diff --git a/backend/app/controllers/span_graph_test.go b/backend/app/controllers/span_graph_test.go
new file mode 100644
index 00000000..fcfd3a39
--- /dev/null
+++ b/backend/app/controllers/span_graph_test.go
@@ -0,0 +1,112 @@
+//go:build !telemetry_ch && !transactional_pg && !telemetry_duckdb
+
+package controllers
+
+import (
+ "context"
+ "encoding/hex"
+ "encoding/json"
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func TestTraceResponsesExposePartialGraphs(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ user, org := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "Graph", "opentelemetry", org)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+ ctx, trace, now := context.Background(), uuid.New(), time.Now().UTC()
+ rootID := uuid.MustParse("00000000-0000-0000-0102-030405060708")
+ owner, traceId, rootSpanId := uuid.New(), hex.EncodeToString(trace[:]), hex.EncodeToString(rootID[8:])
+ spans := []models.OtelSpan{}
+ for i, id := range []uuid.UUID{rootID, uuid.New(), uuid.New()} {
+ span := models.OtelSpan{Span: models.Span{ProjectId: project.Id, TraceId: traceId, SpanId: hex.EncodeToString(id[8:]), Name: "operation", StartTime: now.Add(time.Duration(i)), Duration: time.Millisecond}}
+ if i > 0 {
+ span.ParentSpanId = rootSpanId
+ }
+ spans = append(spans, span)
+ }
+ if _, err := telemetry.OtelSpanRepository.InsertAsync(ctx, spans); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{{Id: owner, ProjectId: project.Id, Endpoint: "GET /graph", RecordedAt: now, TraceId: traceId, SpanId: rootSpanId}}); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.TaskRepository.InsertAsync(ctx, []models.Task{{Id: owner, ProjectId: project.Id, TaskName: "graph", RecordedAt: now, TraceId: traceId, SpanId: rootSpanId}}); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.AiTraceRepository.InsertAsync(ctx, []models.AiTrace{{Id: owner, ProjectId: project.Id, TraceName: "graph", RecordedAt: now, TraceId: traceId, SpanId: rootSpanId}}); err != nil {
+ t.Fatal(err)
+ }
+ previous := shared.MaxOtelGraphRows
+ shared.MaxOtelGraphRows = 2
+ t.Cleanup(func() { shared.MaxOtelGraphRows = previous })
+ check := func(spans []models.Span, status *models.SpanGraphStatus) {
+ t.Helper()
+ if len(spans) != 1 || status == nil || status.State != models.SpanGraphPartial || len(status.Reasons) != 1 || status.Reasons[0] != models.SpanGraphRowLimit {
+ t.Fatalf("available spans or completeness lost: %d spans, %+v", len(spans), status)
+ }
+ }
+ for _, route := range []struct {
+ param string
+ handler gin.HandlerFunc
+ }{
+ {"endpointId", EndpointDetailController.GetEndpointDetail},
+ {"taskId", TaskDetailController.GetTaskDetail},
+ {"traceId", AiTraceController.GetAiTraceDetail},
+ } {
+ t.Run(route.param, func(t *testing.T) {
+ c, response := newControllerTestContext(t, nil, user, http.MethodPost, "/detail", "{}")
+ c.Set(middleware.ProjectIdContextKey, project.Id)
+ c.Params = gin.Params{{Key: route.param, Value: owner.String()}}
+ route.handler(c)
+ if response.Code != 200 {
+ t.Fatalf("HTTP %d: %v", response.Code, c.Errors)
+ }
+ var detail struct {
+ Spans []models.Span `json:"spans"`
+ Status *models.SpanGraphStatus `json:"spanGraphStatus"`
+ }
+ if err := json.Unmarshal(response.Body.Bytes(), &detail); err != nil {
+ t.Fatal(err)
+ }
+ check(detail.Spans, detail.Status)
+ })
+ }
+ c, response := newControllerTestContext(t, nil, user, http.MethodPost, "/distributed", "{}")
+ c.Params = gin.Params{{Key: "traceId", Value: traceId}}
+ DistributedTraceController.GetDistributedTrace(c)
+ if response.Code != 200 {
+ t.Fatalf("HTTP %d: %v", response.Code, c.Errors)
+ }
+ var distributed DistributedTraceResponse
+ if err := json.Unmarshal(response.Body.Bytes(), &distributed); err != nil {
+ t.Fatal(err)
+ }
+ if len(distributed.Nodes) != 3 {
+ t.Fatalf("expected three occurrences, got %d", len(distributed.Nodes))
+ }
+ for _, node := range distributed.Nodes {
+ check(node.Spans, node.SpanGraphStatus)
+ }
+}
diff --git a/backend/app/controllers/task_detail.controller.go b/backend/app/controllers/task_detail.controller.go
index 9b4fface..1695df00 100644
--- a/backend/app/controllers/task_detail.controller.go
+++ b/backend/app/controllers/task_detail.controller.go
@@ -7,6 +7,7 @@ import (
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
@@ -34,11 +35,12 @@ type TaskMessageInfo struct {
}
type TaskDetailResponse struct {
- Task *models.Task `json:"task"`
- Spans []models.Span `json:"spans"`
- HasSpans bool `json:"hasSpans"`
- Exception *TaskExceptionInfo `json:"exception,omitempty"`
- Messages []TaskMessageInfo `json:"messages"`
+ Task *models.Task `json:"task"`
+ SpanGraphStatus *models.SpanGraphStatus `json:"spanGraphStatus"`
+ Spans []models.Span `json:"spans"`
+ HasSpans bool `json:"hasSpans"`
+ Exception *TaskExceptionInfo `json:"exception,omitempty"`
+ Messages []TaskMessageInfo `json:"messages"`
}
func (t taskDetailController) GetTaskDetail(c *gin.Context) {
@@ -77,7 +79,7 @@ func (t taskDetailController) GetTaskDetail(c *gin.Context) {
// Get spans (flat list ordered by start_time)
span = traceway.StartSpan(c, "loading spans")
- spans, err := telemetry.SpanRepository.FindByTraceId(c, projectId, taskId, &recordedAt)
+ graph, err := telemetry.SpanRepository.FindGraph(c, shared.NewSpanLookup(projectId, task.TraceId, task.SpanId, recordedAt))
span.End()
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading spans: %w", err))
@@ -89,7 +91,7 @@ func (t taskDetailController) GetTaskDetail(c *gin.Context) {
var messages []TaskMessageInfo
span = traceway.StartSpan(c, "loading exceptions")
- allExceptions, err := telemetry.ExceptionStackTraceRepository.FindAllByTraceId(c, projectId, taskId, &recordedAt)
+ allExceptions, err := findOccurrenceExceptions(c, projectId, task.TraceId, task.SpanId, &recordedAt, graph.Spans)
span.End()
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error loading allExceptions: %w", err))
@@ -121,11 +123,12 @@ func (t taskDetailController) GetTaskDetail(c *gin.Context) {
}
c.JSON(http.StatusOK, TaskDetailResponse{
- Task: task,
- Spans: spans,
- HasSpans: len(spans) > 0,
- Exception: exceptionInfo,
- Messages: messages,
+ Task: task,
+ SpanGraphStatus: &graph.Status,
+ Spans: graph.Spans,
+ HasSpans: len(graph.Spans) > 0,
+ Exception: exceptionInfo,
+ Messages: messages,
})
}
diff --git a/backend/app/controllers/trace_exceptions.go b/backend/app/controllers/trace_exceptions.go
new file mode 100644
index 00000000..8bd65361
--- /dev/null
+++ b/backend/app/controllers/trace_exceptions.go
@@ -0,0 +1,32 @@
+package controllers
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+)
+
+// The exceptions of an endpoint, a task or an AI trace are the ones recorded on its own span or on a span below it.
+func findOccurrenceExceptions(ctx context.Context, projectId uuid.UUID, traceId, spanId string, recordedAt *time.Time, spans []models.Span) ([]models.ExceptionStackTrace, error) {
+ if traceId == "" {
+ return nil, nil
+ }
+ exceptions, err := telemetry.ExceptionStackTraceRepository.FindAllByTraceId(ctx, projectId, traceId, recordedAt)
+ if err != nil {
+ return nil, err
+ }
+ subtree := map[string]bool{spanId: true}
+ for _, span := range spans {
+ subtree[span.SpanId] = true
+ }
+ kept := exceptions[:0]
+ for _, exception := range exceptions {
+ if subtree[exception.SpanId] {
+ kept = append(kept, exception)
+ }
+ }
+ return kept, nil
+}
diff --git a/backend/app/controllers/trace_exceptions_test.go b/backend/app/controllers/trace_exceptions_test.go
new file mode 100644
index 00000000..661c7236
--- /dev/null
+++ b/backend/app/controllers/trace_exceptions_test.go
@@ -0,0 +1,149 @@
+//go:build !telemetry_ch && !transactional_pg && !telemetry_duckdb
+
+package controllers
+
+import (
+ "context"
+ "encoding/hex"
+ "encoding/json"
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/gin-gonic/gin"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/middleware"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+// An exception row says which trace and span it happened on and nothing else. A detail page shows the ones on its own
+// span or below it, whichever arrived first, and the issue page finds its way back the same way.
+func TestDetailShowsTheExceptionsInsideItsSubtree(t *testing.T) {
+ setupSetupControllerDB(t)
+ tx, err := db.DB.Begin()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer tx.Rollback()
+ user, org := createSetupTestAccount(t, tx, "[email protected]", "owner")
+ project, err := transactional.ProjectRepository.CreateWithOrganization(tx, "Exceptions", "opentelemetry", org)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := tx.Commit(); err != nil {
+ t.Fatal(err)
+ }
+
+ ctx, trace, now := context.Background(), uuid.New(), time.Now().UTC()
+ rootID := uuid.MustParse("00000000-0000-0000-0102-030405060708")
+ childID := uuid.MustParse("00000000-0000-0000-1112-131415161718")
+ outsideID := uuid.MustParse("00000000-0000-0000-2122-232425262728")
+ traceHex := hex.EncodeToString(trace[:])
+ spanHex := func(id uuid.UUID) string { return hex.EncodeToString(id[8:]) }
+ owner := uuid.New()
+
+ span := func(id uuid.UUID, parent *uuid.UUID) models.OtelSpan {
+ stored := models.OtelSpan{Span: models.Span{ProjectId: project.Id, TraceId: traceHex, SpanId: spanHex(id), Name: "operation", StartTime: now, Duration: time.Millisecond}}
+ if parent != nil {
+ stored.ParentSpanId = spanHex(*parent)
+ }
+ return stored
+ }
+ if _, err := telemetry.OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span(rootID, nil), span(childID, &rootID), span(outsideID, nil)}); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{{Id: owner, ProjectId: project.Id, Endpoint: "GET /exceptions", RecordedAt: now, TraceId: traceHex, SpanId: spanHex(rootID)}}); err != nil {
+ t.Fatal(err)
+ }
+ if err := telemetry.TaskRepository.InsertAsync(ctx, []models.Task{{Id: owner, ProjectId: project.Id, TaskName: "exceptions", RecordedAt: now, TraceId: traceHex, SpanId: spanHex(rootID)}}); err != nil {
+ t.Fatal(err)
+ }
+ unowned := func(hash string, spanID uuid.UUID, recordedAt time.Time) models.ExceptionStackTrace {
+ return models.ExceptionStackTrace{Id: uuid.New(), ProjectId: project.Id, TraceId: traceHex, SpanId: spanHex(spanID), ExceptionHash: hash, StackTrace: "RuntimeError: " + hash, RecordedAt: recordedAt}
+ }
+ // The exception outside the subtree is older, so only the span match can keep it off the page.
+ if err := telemetry.ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{unowned("outside", outsideID, now.Add(-time.Second)), unowned("inside", childID, now)}); err != nil {
+ t.Fatal(err)
+ }
+
+ for _, route := range []struct {
+ param string
+ handler gin.HandlerFunc
+ }{
+ {"endpointId", EndpointDetailController.GetEndpointDetail},
+ {"taskId", TaskDetailController.GetTaskDetail},
+ } {
+ t.Run(route.param, func(t *testing.T) {
+ c, response := newControllerTestContext(t, nil, user, http.MethodPost, "/detail", "{}")
+ c.Set(middleware.ProjectIdContextKey, project.Id)
+ c.Params = gin.Params{{Key: route.param, Value: owner.String()}}
+ route.handler(c)
+ if response.Code != 200 {
+ t.Fatalf("HTTP %d: %v", response.Code, c.Errors)
+ }
+ var detail struct {
+ Exception *EndpointExceptionInfo `json:"exception"`
+ }
+ if err := json.Unmarshal(response.Body.Bytes(), &detail); err != nil {
+ t.Fatal(err)
+ }
+ if detail.Exception == nil || detail.Exception.ExceptionHash != "inside" {
+ t.Fatalf("expected the exception recorded inside the subtree, got %+v", detail.Exception)
+ }
+ })
+ }
+
+ for hash, related := range map[string]bool{"inside": true, "outside": false} {
+ c, response := newControllerTestContext(t, nil, user, http.MethodPost, "/exception", `{"pagination":{"page":1,"pageSize":20}}`)
+ c.Set(middleware.ProjectIdContextKey, project.Id)
+ c.Params = gin.Params{{Key: "hash", Value: hash}}
+ ExceptionStackTraceController.FindByHash(c)
+ var issue struct {
+ RelatedEntity *telemetry.ExceptionOwner `json:"relatedEntity"`
+ }
+ if err := json.Unmarshal(response.Body.Bytes(), &issue); err != nil || response.Code != 200 {
+ t.Fatalf("%s: HTTP %d %v", hash, response.Code, err)
+ }
+ if !related {
+ if issue.RelatedEntity != nil {
+ t.Fatalf("an exception on a span outside every endpoint names none: %+v", issue.RelatedEntity)
+ }
+ continue
+ }
+ if issue.RelatedEntity == nil || issue.RelatedEntity.Id != owner || issue.RelatedEntity.TraceType != "endpoint" || issue.RelatedEntity.Name != "GET /exceptions" {
+ t.Fatalf("the issue page finds the endpoint above the exception's span: %+v", issue.RelatedEntity)
+ }
+ }
+
+ t.Run("missing descendant exception keeps partial status", func(t *testing.T) {
+ previous := shared.MaxOtelGraphRows
+ shared.MaxOtelGraphRows = 1
+ t.Cleanup(func() { shared.MaxOtelGraphRows = previous })
+ for _, route := range []struct {
+ param string
+ handler gin.HandlerFunc
+ }{
+ {"endpointId", EndpointDetailController.GetEndpointDetail},
+ {"taskId", TaskDetailController.GetTaskDetail},
+ } {
+ c, response := newControllerTestContext(t, nil, user, http.MethodPost, "/detail", "{}")
+ c.Set(middleware.ProjectIdContextKey, project.Id)
+ c.Params = gin.Params{{Key: route.param, Value: owner.String()}}
+ route.handler(c)
+ var detail struct {
+ Exception *EndpointExceptionInfo `json:"exception"`
+ Status *models.SpanGraphStatus `json:"spanGraphStatus"`
+ }
+ if err := json.Unmarshal(response.Body.Bytes(), &detail); err != nil || response.Code != 200 {
+ t.Fatalf("%s: HTTP %d %v", route.param, response.Code, err)
+ }
+ if detail.Exception != nil || detail.Status == nil || detail.Status.State != models.SpanGraphPartial {
+ t.Fatalf("missing exceptions must not imply a complete graph: %+v", detail)
+ }
+ }
+ })
+}
diff --git a/backend/app/db/telemetry_stats.go b/backend/app/db/telemetry_stats.go
index b980d6dd..d87c08ed 100644
--- a/backend/app/db/telemetry_stats.go
+++ b/backend/app/db/telemetry_stats.go
@@ -38,6 +38,8 @@ var (
telemetryDroppedRows = map[string]uint64{}
telemetryInsertFailures uint64
telemetryIngestRejects uint64
+
+ telemetrySpanPartitionFallbacks uint64
)
// RecordIngestRejected counts requests turned away by the ingest admission
@@ -60,6 +62,18 @@ func RecordTelemetryRowDropped(table string) {
telemetryIngestMu.Unlock()
}
+func RecordSpanPartitionTimeFallback() {
+ telemetryIngestMu.Lock()
+ telemetrySpanPartitionFallbacks++
+ telemetryIngestMu.Unlock()
+}
+
+func GetSpanPartitionTimeFallbacks() uint64 {
+ telemetryIngestMu.Lock()
+ defer telemetryIngestMu.Unlock()
+ return telemetrySpanPartitionFallbacks
+}
+
func RecordTelemetryInsertFailure() {
telemetryIngestMu.Lock()
telemetryInsertFailures++
diff --git a/backend/app/migrations/ch/0085_create_spans_v2.up.sql b/backend/app/migrations/ch/0085_create_spans_v2.up.sql
new file mode 100644
index 00000000..6625d978
--- /dev/null
+++ b/backend/app/migrations/ch/0085_create_spans_v2.up.sql
@@ -0,0 +1,35 @@
+CREATE TABLE IF NOT EXISTS spans_v2 (
+ project_id UUID,
+ trace_id String DEFAULT '',
+ span_id String DEFAULT '',
+ parent_span_id String DEFAULT '',
+ name String DEFAULT '',
+ span_kind Int32 DEFAULT 0,
+ status_code Int32 DEFAULT 0,
+ status_message String DEFAULT '',
+ service_name LowCardinality(String) DEFAULT '',
+ scope_name LowCardinality(String) DEFAULT '',
+ scope_version LowCardinality(String) DEFAULT '',
+ start_time_unix_nano UInt64 DEFAULT 0,
+ end_time_unix_nano UInt64 DEFAULT 0,
+ duration Int64 DEFAULT 0,
+ recorded_at DateTime,
+ trace_state String DEFAULT '',
+ flags UInt32 DEFAULT 0,
+ dropped_attributes_count UInt32 DEFAULT 0,
+ dropped_events_count UInt32 DEFAULT 0,
+ dropped_links_count UInt32 DEFAULT 0,
+ resource_schema_url LowCardinality(String) DEFAULT '',
+ scope_schema_url LowCardinality(String) DEFAULT '',
+ span_attributes Map(LowCardinality(String), String),
+ resource LowCardinality(String) DEFAULT '',
+ scope LowCardinality(String) DEFAULT '',
+ events String DEFAULT '[]' CODEC(ZSTD(1)),
+ links String DEFAULT '[]' CODEC(ZSTD(1)),
+ resource_pb LowCardinality(String) DEFAULT '',
+ scope_pb LowCardinality(String) DEFAULT '',
+ span_pb String DEFAULT '' CODEC(ZSTD(1)),
+ INDEX idx_spans_v2_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1
+) ENGINE = MergeTree
+PARTITION BY toYYYYMMDD(recorded_at)
+ORDER BY (project_id, trace_id, start_time_unix_nano)
diff --git a/backend/app/migrations/ch/0086_create_endpoints_v2.up.sql b/backend/app/migrations/ch/0086_create_endpoints_v2.up.sql
new file mode 100644
index 00000000..e7f1abc6
--- /dev/null
+++ b/backend/app/migrations/ch/0086_create_endpoints_v2.up.sql
@@ -0,0 +1,26 @@
+CREATE TABLE IF NOT EXISTS endpoints_v2 (
+ id UUID,
+ project_id UUID,
+ trace_id String DEFAULT '',
+ span_id String DEFAULT '',
+ parent_span_id String DEFAULT '',
+ linked_trace_id String DEFAULT '',
+ endpoint LowCardinality(String),
+ duration Int64 DEFAULT 0,
+ recorded_at DateTime64(6),
+ status_code Int16 DEFAULT 0,
+ body_size Int32 DEFAULT 0,
+ client_ip String DEFAULT '',
+ attributes String DEFAULT '{}',
+ app_version LowCardinality(String) DEFAULT '',
+ server_name LowCardinality(String) DEFAULT '',
+ is_stream UInt8 DEFAULT 0,
+ is_root UInt8 DEFAULT 1,
+ INDEX idx_endpoints_v2_endpoint endpoint TYPE bloom_filter(0.01) GRANULARITY 4,
+ INDEX idx_endpoints_v2_status_code status_code TYPE set(100) GRANULARITY 4,
+ INDEX idx_endpoints_v2_id id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_endpoints_v2_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_endpoints_v2_linked_trace_id linked_trace_id TYPE bloom_filter(0.001) GRANULARITY 1
+) ENGINE = MergeTree
+PARTITION BY toYYYYMMDD(recorded_at)
+ORDER BY (project_id, recorded_at, endpoint)
diff --git a/backend/app/migrations/ch/0087_create_tasks_v2.up.sql b/backend/app/migrations/ch/0087_create_tasks_v2.up.sql
new file mode 100644
index 00000000..a2ae23c0
--- /dev/null
+++ b/backend/app/migrations/ch/0087_create_tasks_v2.up.sql
@@ -0,0 +1,22 @@
+CREATE TABLE IF NOT EXISTS tasks_v2 (
+ id UUID,
+ project_id UUID,
+ trace_id String DEFAULT '',
+ span_id String DEFAULT '',
+ parent_span_id String DEFAULT '',
+ linked_trace_id String DEFAULT '',
+ task_name LowCardinality(String),
+ duration Int64 DEFAULT 0,
+ recorded_at DateTime64(6),
+ client_ip String DEFAULT '',
+ attributes String DEFAULT '{}',
+ app_version LowCardinality(String) DEFAULT '',
+ server_name LowCardinality(String) DEFAULT '',
+ is_root UInt8 DEFAULT 1,
+ INDEX idx_tasks_v2_task_name task_name TYPE bloom_filter(0.01) GRANULARITY 4,
+ INDEX idx_tasks_v2_id id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_tasks_v2_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_tasks_v2_linked_trace_id linked_trace_id TYPE bloom_filter(0.001) GRANULARITY 1
+) ENGINE = MergeTree
+PARTITION BY toYYYYMMDD(recorded_at)
+ORDER BY (project_id, recorded_at, task_name)
diff --git a/backend/app/migrations/ch/0088_create_ai_traces_v2.up.sql b/backend/app/migrations/ch/0088_create_ai_traces_v2.up.sql
new file mode 100644
index 00000000..9d769e27
--- /dev/null
+++ b/backend/app/migrations/ch/0088_create_ai_traces_v2.up.sql
@@ -0,0 +1,44 @@
+CREATE TABLE IF NOT EXISTS ai_traces_v2 (
+ id UUID,
+ project_id UUID,
+ trace_id String DEFAULT '',
+ span_id String DEFAULT '',
+ parent_span_id String DEFAULT '',
+ linked_trace_id String DEFAULT '',
+ recorded_at DateTime64(3),
+ duration Int64 DEFAULT 0,
+ status_code UInt8 DEFAULT 0,
+ model LowCardinality(String) DEFAULT '',
+ response_model LowCardinality(String) DEFAULT '',
+ provider LowCardinality(String) DEFAULT '',
+ operation LowCardinality(String) DEFAULT '',
+ input_tokens Int64 DEFAULT 0,
+ output_tokens Int64 DEFAULT 0,
+ total_tokens Int64 DEFAULT 0,
+ cached_tokens Int64 DEFAULT 0,
+ reasoning_tokens Int64 DEFAULT 0,
+ input_cost Float64 DEFAULT 0,
+ output_cost Float64 DEFAULT 0,
+ total_cost Float64 DEFAULT 0,
+ trace_name LowCardinality(String) DEFAULT '',
+ user_id String DEFAULT '',
+ finish_reason LowCardinality(String) DEFAULT '',
+ server_name LowCardinality(String) DEFAULT '',
+ app_version LowCardinality(String) DEFAULT '',
+ storage_key String DEFAULT '',
+ attributes String DEFAULT '{}',
+ is_root UInt8 DEFAULT 1,
+ conversation_id String DEFAULT '',
+ tool_call_count Int64 DEFAULT 0,
+ tool_names String DEFAULT '',
+ flagged UInt8 DEFAULT 0,
+ flagged_terms String DEFAULT '',
+ INDEX idx_ai_traces_v2_id id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_ai_traces_v2_trace_name trace_name TYPE set(100) GRANULARITY 4,
+ INDEX idx_ai_traces_v2_model model TYPE set(100) GRANULARITY 4,
+ INDEX idx_ai_traces_v2_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_ai_traces_v2_linked_trace_id linked_trace_id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_ai_traces_v2_conversation_id conversation_id TYPE bloom_filter(0.001) GRANULARITY 1
+) ENGINE = MergeTree
+PARTITION BY toYYYYMMDD(recorded_at)
+ORDER BY (project_id, recorded_at, trace_name)
diff --git a/backend/app/migrations/ch/0089_create_exceptions_v2.up.sql b/backend/app/migrations/ch/0089_create_exceptions_v2.up.sql
new file mode 100644
index 00000000..50f969af
--- /dev/null
+++ b/backend/app/migrations/ch/0089_create_exceptions_v2.up.sql
@@ -0,0 +1,25 @@
+CREATE TABLE IF NOT EXISTS exceptions_v2 (
+ id UUID,
+ project_id UUID,
+ trace_id String DEFAULT '',
+ span_id String DEFAULT '',
+ trace_type LowCardinality(String) DEFAULT '',
+ exception_hash String,
+ stack_trace String,
+ recorded_at DateTime64(6),
+ attributes String DEFAULT '{}',
+ app_version LowCardinality(String) DEFAULT '',
+ server_name LowCardinality(String) DEFAULT '',
+ is_message UInt8 DEFAULT 0,
+ linked_trace_id String DEFAULT '',
+ session_id Nullable(UUID) DEFAULT NULL,
+ INDEX idx_exceptions_v2_exception_hash exception_hash TYPE bloom_filter(0.01) GRANULARITY 4,
+ INDEX idx_exceptions_v2_is_message is_message TYPE set(2) GRANULARITY 1,
+ INDEX idx_exceptions_v2_id id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_exceptions_v2_stack_trace stack_trace TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
+ INDEX idx_exceptions_v2_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_exceptions_v2_linked_trace_id linked_trace_id TYPE bloom_filter(0.001) GRANULARITY 1,
+ INDEX idx_exceptions_v2_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1
+) ENGINE = MergeTree
+PARTITION BY toYYYYMMDD(recorded_at)
+ORDER BY (project_id, recorded_at, exception_hash)
diff --git a/backend/app/migrations/ch/0090_add_span_version.up.sql b/backend/app/migrations/ch/0090_add_span_version.up.sql
new file mode 100644
index 00000000..35dfe6e5
--- /dev/null
+++ b/backend/app/migrations/ch/0090_add_span_version.up.sql
@@ -0,0 +1,4 @@
+ALTER TABLE spans_v2 ADD COLUMN IF NOT EXISTS span_version FixedString(32)
+ MATERIALIZED SHA256(concat(toString(length(span_pb)), ':', span_pb,
+ toString(length(resource_pb)), ':', resource_pb,
+ toString(length(scope_pb)), ':', scope_pb))
diff --git a/backend/app/migrations/ch/0091_add_session_trace_id.up.sql b/backend/app/migrations/ch/0091_add_session_trace_id.up.sql
new file mode 100644
index 00000000..c224050a
--- /dev/null
+++ b/backend/app/migrations/ch/0091_add_session_trace_id.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE sessions ADD COLUMN IF NOT EXISTS trace_id String DEFAULT
+ ifNull(nullIf(replaceAll(toString(distributed_trace_id), '-', ''), '00000000000000000000000000000000'), '')
diff --git a/backend/app/migrations/ch/0092_materialize_session_trace_id.up.sql b/backend/app/migrations/ch/0092_materialize_session_trace_id.up.sql
new file mode 100644
index 00000000..af26db1c
--- /dev/null
+++ b/backend/app/migrations/ch/0092_materialize_session_trace_id.up.sql
@@ -0,0 +1 @@
+ALTER TABLE sessions MATERIALIZE COLUMN trace_id SETTINGS mutations_sync = 2
diff --git a/backend/app/migrations/ch/0093_detach_session_trace_id_default.up.sql b/backend/app/migrations/ch/0093_detach_session_trace_id_default.up.sql
new file mode 100644
index 00000000..534f774d
--- /dev/null
+++ b/backend/app/migrations/ch/0093_detach_session_trace_id_default.up.sql
@@ -0,0 +1 @@
+ALTER TABLE sessions MODIFY COLUMN trace_id String DEFAULT ''
diff --git a/backend/app/migrations/ch/0094_drop_session_distributed_trace_id.up.sql b/backend/app/migrations/ch/0094_drop_session_distributed_trace_id.up.sql
new file mode 100644
index 00000000..d26f0609
--- /dev/null
+++ b/backend/app/migrations/ch/0094_drop_session_distributed_trace_id.up.sql
@@ -0,0 +1 @@
+ALTER TABLE sessions DROP COLUMN IF EXISTS distributed_trace_id
diff --git a/backend/app/migrations/ch/0095_clean_endpoints_v2_identity.up.sql b/backend/app/migrations/ch/0095_clean_endpoints_v2_identity.up.sql
new file mode 100644
index 00000000..bbbab41b
--- /dev/null
+++ b/backend/app/migrations/ch/0095_clean_endpoints_v2_identity.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE endpoints_v2
+ DROP INDEX IF EXISTS idx_endpoints_v2_linked_trace_id,
+ DROP COLUMN IF EXISTS linked_trace_id
diff --git a/backend/app/migrations/ch/0096_clean_tasks_v2_identity.up.sql b/backend/app/migrations/ch/0096_clean_tasks_v2_identity.up.sql
new file mode 100644
index 00000000..02da05c9
--- /dev/null
+++ b/backend/app/migrations/ch/0096_clean_tasks_v2_identity.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE tasks_v2
+ DROP INDEX IF EXISTS idx_tasks_v2_linked_trace_id,
+ DROP COLUMN IF EXISTS linked_trace_id
diff --git a/backend/app/migrations/ch/0097_clean_ai_traces_v2_identity.up.sql b/backend/app/migrations/ch/0097_clean_ai_traces_v2_identity.up.sql
new file mode 100644
index 00000000..0c3c98df
--- /dev/null
+++ b/backend/app/migrations/ch/0097_clean_ai_traces_v2_identity.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE ai_traces_v2
+ DROP INDEX IF EXISTS idx_ai_traces_v2_linked_trace_id,
+ DROP COLUMN IF EXISTS linked_trace_id
diff --git a/backend/app/migrations/ch/0098_clean_exceptions_v2_identity.up.sql b/backend/app/migrations/ch/0098_clean_exceptions_v2_identity.up.sql
new file mode 100644
index 00000000..0a9a7d07
--- /dev/null
+++ b/backend/app/migrations/ch/0098_clean_exceptions_v2_identity.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE exceptions_v2
+ DROP INDEX IF EXISTS idx_exceptions_v2_linked_trace_id,
+ DROP COLUMN IF EXISTS linked_trace_id
diff --git a/backend/app/migrations/ch/0099_clean_profile_identity.up.sql b/backend/app/migrations/ch/0099_clean_profile_identity.up.sql
new file mode 100644
index 00000000..21bb1910
--- /dev/null
+++ b/backend/app/migrations/ch/0099_clean_profile_identity.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE profiles
+ DROP INDEX IF EXISTS idx_distributed_trace_id,
+ DROP COLUMN IF EXISTS distributed_trace_id
diff --git a/backend/app/migrations/duckdb_telemetry/0005_create_v2_tables.up.sql b/backend/app/migrations/duckdb_telemetry/0005_create_v2_tables.up.sql
new file mode 100644
index 00000000..5cc09ad5
--- /dev/null
+++ b/backend/app/migrations/duckdb_telemetry/0005_create_v2_tables.up.sql
@@ -0,0 +1,124 @@
+CREATE TABLE IF NOT EXISTS spans_v2 (
+ project_id VARCHAR NOT NULL,
+ trace_id VARCHAR NOT NULL DEFAULT '',
+ span_id VARCHAR NOT NULL DEFAULT '',
+ parent_span_id VARCHAR NOT NULL DEFAULT '',
+ name VARCHAR NOT NULL DEFAULT '',
+ span_kind BIGINT NOT NULL DEFAULT 0,
+ status_code BIGINT NOT NULL DEFAULT 0,
+ status_message VARCHAR NOT NULL DEFAULT '',
+ service_name VARCHAR NOT NULL DEFAULT '',
+ scope_name VARCHAR NOT NULL DEFAULT '',
+ scope_version VARCHAR NOT NULL DEFAULT '',
+ start_time_unix_nano UBIGINT NOT NULL DEFAULT 0,
+ end_time_unix_nano UBIGINT NOT NULL DEFAULT 0,
+ duration BIGINT NOT NULL DEFAULT 0,
+ recorded_at TIMESTAMP NOT NULL,
+ trace_state VARCHAR NOT NULL DEFAULT '',
+ flags BIGINT NOT NULL DEFAULT 0,
+ dropped_attributes_count BIGINT NOT NULL DEFAULT 0,
+ dropped_events_count BIGINT NOT NULL DEFAULT 0,
+ dropped_links_count BIGINT NOT NULL DEFAULT 0,
+ resource_schema_url VARCHAR NOT NULL DEFAULT '',
+ scope_schema_url VARCHAR NOT NULL DEFAULT '',
+ span_attributes VARCHAR NOT NULL DEFAULT '{}',
+ resource VARCHAR NOT NULL DEFAULT '{}',
+ scope VARCHAR NOT NULL DEFAULT '{}',
+ events VARCHAR NOT NULL DEFAULT '[]',
+ links VARCHAR NOT NULL DEFAULT '[]',
+ otlp BLOB,
+ trace_key VARCHAR NOT NULL DEFAULT ''
+);
+
+CREATE TABLE IF NOT EXISTS endpoints_v2 (
+ id VARCHAR NOT NULL,
+ project_id VARCHAR NOT NULL,
+ trace_id VARCHAR NOT NULL DEFAULT '',
+ span_id VARCHAR NOT NULL DEFAULT '',
+ parent_span_id VARCHAR NOT NULL DEFAULT '',
+ linked_trace_id VARCHAR NOT NULL DEFAULT '',
+ endpoint VARCHAR NOT NULL DEFAULT '',
+ duration BIGINT NOT NULL DEFAULT 0,
+ recorded_at TIMESTAMP NOT NULL,
+ status_code BIGINT NOT NULL DEFAULT 0,
+ body_size BIGINT NOT NULL DEFAULT 0,
+ client_ip VARCHAR NOT NULL DEFAULT '',
+ attributes VARCHAR NOT NULL DEFAULT '{}',
+ app_version VARCHAR NOT NULL DEFAULT '',
+ server_name VARCHAR NOT NULL DEFAULT '',
+ is_stream BIGINT NOT NULL DEFAULT 0,
+ is_root BIGINT NOT NULL DEFAULT 1
+);
+
+CREATE TABLE IF NOT EXISTS tasks_v2 (
+ id VARCHAR NOT NULL,
+ project_id VARCHAR NOT NULL,
+ trace_id VARCHAR NOT NULL DEFAULT '',
+ span_id VARCHAR NOT NULL DEFAULT '',
+ parent_span_id VARCHAR NOT NULL DEFAULT '',
+ linked_trace_id VARCHAR NOT NULL DEFAULT '',
+ task_name VARCHAR NOT NULL DEFAULT '',
+ duration BIGINT NOT NULL DEFAULT 0,
+ recorded_at TIMESTAMP NOT NULL,
+ client_ip VARCHAR NOT NULL DEFAULT '',
+ attributes VARCHAR NOT NULL DEFAULT '{}',
+ app_version VARCHAR NOT NULL DEFAULT '',
+ server_name VARCHAR NOT NULL DEFAULT '',
+ is_root BIGINT NOT NULL DEFAULT 1
+);
+
+CREATE TABLE IF NOT EXISTS ai_traces_v2 (
+ id VARCHAR NOT NULL,
+ project_id VARCHAR NOT NULL,
+ trace_id VARCHAR NOT NULL DEFAULT '',
+ span_id VARCHAR NOT NULL DEFAULT '',
+ parent_span_id VARCHAR NOT NULL DEFAULT '',
+ linked_trace_id VARCHAR NOT NULL DEFAULT '',
+ recorded_at TIMESTAMP NOT NULL,
+ duration BIGINT NOT NULL DEFAULT 0,
+ status_code BIGINT NOT NULL DEFAULT 0,
+ model VARCHAR NOT NULL DEFAULT '',
+ response_model VARCHAR NOT NULL DEFAULT '',
+ provider VARCHAR NOT NULL DEFAULT '',
+ operation VARCHAR NOT NULL DEFAULT '',
+ input_tokens BIGINT NOT NULL DEFAULT 0,
+ output_tokens BIGINT NOT NULL DEFAULT 0,
+ total_tokens BIGINT NOT NULL DEFAULT 0,
+ cached_tokens BIGINT NOT NULL DEFAULT 0,
+ reasoning_tokens BIGINT NOT NULL DEFAULT 0,
+ input_cost DOUBLE NOT NULL DEFAULT 0,
+ output_cost DOUBLE NOT NULL DEFAULT 0,
+ total_cost DOUBLE NOT NULL DEFAULT 0,
+ trace_name VARCHAR NOT NULL DEFAULT '',
+ user_id VARCHAR NOT NULL DEFAULT '',
+ finish_reason VARCHAR NOT NULL DEFAULT '',
+ server_name VARCHAR NOT NULL DEFAULT '',
+ app_version VARCHAR NOT NULL DEFAULT '',
+ storage_key VARCHAR NOT NULL DEFAULT '',
+ attributes VARCHAR NOT NULL DEFAULT '{}',
+ is_root BIGINT NOT NULL DEFAULT 1,
+ conversation_id VARCHAR NOT NULL DEFAULT '',
+ tool_call_count BIGINT NOT NULL DEFAULT 0,
+ tool_names VARCHAR NOT NULL DEFAULT '',
+ flagged BIGINT NOT NULL DEFAULT 0,
+ flagged_terms VARCHAR NOT NULL DEFAULT ''
+);
+
+CREATE TABLE IF NOT EXISTS exceptions_v2 (
+ id VARCHAR NOT NULL,
+ project_id VARCHAR NOT NULL,
+ trace_id VARCHAR NOT NULL DEFAULT '',
+ span_id VARCHAR NOT NULL DEFAULT '',
+ trace_type VARCHAR NOT NULL DEFAULT '',
+ exception_hash VARCHAR NOT NULL DEFAULT '',
+ stack_trace VARCHAR NOT NULL DEFAULT '',
+ recorded_at TIMESTAMP NOT NULL,
+ attributes VARCHAR NOT NULL DEFAULT '{}',
+ app_version VARCHAR NOT NULL DEFAULT '',
+ server_name VARCHAR NOT NULL DEFAULT '',
+ is_message BIGINT NOT NULL DEFAULT 0,
+ linked_trace_id VARCHAR NOT NULL DEFAULT '',
+ session_id VARCHAR
+);
+
+CREATE INDEX IF NOT EXISTS idx_spans_v2_trace_key ON spans_v2(trace_key);
diff --git a/backend/app/migrations/duckdb_telemetry/0006_clean_trace_identity.up.sql b/backend/app/migrations/duckdb_telemetry/0006_clean_trace_identity.up.sql
new file mode 100644
index 00000000..cbae0a03
--- /dev/null
+++ b/backend/app/migrations/duckdb_telemetry/0006_clean_trace_identity.up.sql
@@ -0,0 +1,17 @@
+ALTER TABLE sessions RENAME COLUMN distributed_trace_id TO trace_id;
+
+ALTER TABLE sessions ALTER COLUMN trace_id SET DEFAULT '';
+
+UPDATE sessions SET trace_id = CASE
+ WHEN regexp_full_match(lower(replace(COALESCE(trace_id, ''), '-', '')), '[0-9a-f]{32}') AND lower(replace(COALESCE(trace_id, ''), '-', '')) != '00000000000000000000000000000000'
+ THEN lower(replace(COALESCE(trace_id, ''), '-', '')) ELSE '' END;
+
+ALTER TABLE endpoints_v2 DROP COLUMN linked_trace_id;
+
+ALTER TABLE tasks_v2 DROP COLUMN linked_trace_id;
+
+ALTER TABLE ai_traces_v2 DROP COLUMN linked_trace_id;
+
+ALTER TABLE exceptions_v2 DROP COLUMN linked_trace_id;
+
+ALTER TABLE profiles DROP COLUMN distributed_trace_id;
diff --git a/backend/app/migrations/duckdb_telemetry/0007_require_session_trace_id_string.up.sql b/backend/app/migrations/duckdb_telemetry/0007_require_session_trace_id_string.up.sql
new file mode 100644
index 00000000..f55d39c0
--- /dev/null
+++ b/backend/app/migrations/duckdb_telemetry/0007_require_session_trace_id_string.up.sql
@@ -0,0 +1 @@
+ALTER TABLE sessions ALTER COLUMN trace_id SET NOT NULL;
diff --git a/backend/app/migrations/migrations.go b/backend/app/migrations/migrations.go
index f23d1a02..5eb6fbb6 100644
--- a/backend/app/migrations/migrations.go
+++ b/backend/app/migrations/migrations.go
@@ -66,20 +66,29 @@ func runMigrationsOn(target *sql.DB, fsys fs.FS, dir, trackingTable, createTrack
config.Logf("migrations: applying %s/%s", dir, version)
started := time.Now()
+ tx, err := target.Begin()
+ if err != nil {
+ return fmt.Errorf("failed to begin migration %s: %w", file, err)
+ }
statements := splitStatements(string(content))
for _, stmt := range statements {
stmt = strings.TrimSpace(stmt)
if stmt == "" {
continue
}
- if _, err := target.Exec(stmt); err != nil {
+ if _, err := tx.Exec(stmt); err != nil {
+ tx.Rollback()
return fmt.Errorf("failed to execute migration %s: %w", file, err)
}
}
- if _, err := target.Exec(fmt.Sprintf("INSERT INTO %s (version) VALUES (?)", trackingTable), version); err != nil {
+ if _, err := tx.Exec(fmt.Sprintf("INSERT INTO %s (version) VALUES (?)", trackingTable), version); err != nil {
+ tx.Rollback()
return fmt.Errorf("failed to record migration version %s: %w", version, err)
}
+ if err := tx.Commit(); err != nil {
+ return fmt.Errorf("failed to commit migration %s: %w", version, err)
+ }
config.Logf("migrations: applied %s/%s in %s", dir, version, time.Since(started).Round(time.Millisecond))
}
diff --git a/backend/app/migrations/migrations_log_test.go b/backend/app/migrations/migrations_log_test.go
index 20179651..9db0b406 100644
--- a/backend/app/migrations/migrations_log_test.go
+++ b/backend/app/migrations/migrations_log_test.go
@@ -94,3 +94,25 @@ func TestRunMigrationsOnAnnouncesEachMigrationBeforeRunningIt(t *testing.T) {
t.Errorf("re-running applied migrations logged %q, want silence", buf.String())
}
}
+
+func TestMigrationFailureRollsBackSchemaAndVersion(t *testing.T) {
+ target := openMemorySQLite(t)
+ if _, err := target.Exec("CREATE TABLE sessions (distributed_trace_id TEXT); INSERT INTO sessions VALUES ('preserved')"); err != nil {
+ t.Fatal(err)
+ }
+ source := fstest.MapFS{"m/0001_cleanup.up.sql": {Data: []byte("ALTER TABLE sessions RENAME COLUMN distributed_trace_id TO trace_id; INSERT INTO missing_table VALUES (1)")}}
+ if err := runMigrationsOn(target, source, "m", "schema_migrations", sqliteTrackingDDL); err == nil {
+ t.Fatal("expected migration failure")
+ }
+ var id string
+ if err := target.QueryRow("SELECT distributed_trace_id FROM sessions").Scan(&id); err != nil || id != "preserved" {
+ t.Fatalf("failed migration changed the schema/data: %q: %v", id, err)
+ }
+ source["m/0001_cleanup.up.sql"].Data = []byte("ALTER TABLE sessions RENAME COLUMN distributed_trace_id TO trace_id")
+ if err := runMigrationsOn(target, source, "m", "schema_migrations", sqliteTrackingDDL); err != nil {
+ t.Fatal(err)
+ }
+ if err := target.QueryRow("SELECT trace_id FROM sessions").Scan(&id); err != nil || id != "preserved" {
+ t.Fatalf("retry lost the original ID: %q: %v", id, err)
+ }
+}
diff --git a/backend/app/migrations/migrations_telemetry_ch.go b/backend/app/migrations/migrations_telemetry_ch.go
index 30e234eb..6d98751e 100644
--- a/backend/app/migrations/migrations_telemetry_ch.go
+++ b/backend/app/migrations/migrations_telemetry_ch.go
@@ -6,6 +6,7 @@ import (
"database/sql"
"embed"
"fmt"
+ "io/fs"
"net/url"
"github.com/tracewayapp/traceway/backend/app/config"
@@ -19,13 +20,17 @@ import (
var migrationsChFS embed.FS
func runMigrationsClickhouse(connStr string) error {
+ return runMigrationsClickhouseFrom(connStr, migrationsChFS)
+}
+
+func runMigrationsClickhouseFrom(connStr string, sourceFS fs.FS) error {
db, err := sql.Open("clickhouse", connStr)
if err != nil {
return err
}
defer db.Close()
- source, err := iofs.New(migrationsChFS, "ch")
+ source, err := iofs.New(sourceFS, "ch")
if err != nil {
return err
}
diff --git a/backend/app/migrations/pg/0143_create_v2_move_over_days.up.sql b/backend/app/migrations/pg/0143_create_v2_move_over_days.up.sql
new file mode 100644
index 00000000..3dcdcdc5
--- /dev/null
+++ b/backend/app/migrations/pg/0143_create_v2_move_over_days.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS v2_move_over_days (
+ source_table TEXT NOT NULL,
+ day TEXT NOT NULL,
+ state TEXT NOT NULL CHECK (state IN ('started', 'done')),
+ moved_rows BIGINT NOT NULL DEFAULT 0 CHECK (moved_rows >= 0),
+ updated_at TIMESTAMPTZ NOT NULL,
+ PRIMARY KEY (source_table, day)
+);
diff --git a/backend/app/migrations/sqlite/0077_create_v2_move_over_days.up.sql b/backend/app/migrations/sqlite/0077_create_v2_move_over_days.up.sql
new file mode 100644
index 00000000..5cc01210
--- /dev/null
+++ b/backend/app/migrations/sqlite/0077_create_v2_move_over_days.up.sql
@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS v2_move_over_days (
+ source_table TEXT NOT NULL,
+ day TEXT NOT NULL,
+ state TEXT NOT NULL CHECK (state IN ('started', 'done')),
+ moved_rows INTEGER NOT NULL DEFAULT 0 CHECK (moved_rows >= 0),
+ updated_at DATETIME NOT NULL,
+ PRIMARY KEY (source_table, day)
+);
diff --git a/backend/app/migrations/sqlite_telemetry/0025_create_v2_tables.up.sql b/backend/app/migrations/sqlite_telemetry/0025_create_v2_tables.up.sql
new file mode 100644
index 00000000..30caa9b3
--- /dev/null
+++ b/backend/app/migrations/sqlite_telemetry/0025_create_v2_tables.up.sql
@@ -0,0 +1,161 @@
+CREATE TABLE IF NOT EXISTS spans_v2 (
+ project_id TEXT NOT NULL,
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
+ parent_span_id TEXT NOT NULL DEFAULT '',
+ name TEXT NOT NULL DEFAULT '',
+ span_kind INTEGER NOT NULL DEFAULT 0,
+ status_code INTEGER NOT NULL DEFAULT 0,
+ status_message TEXT NOT NULL DEFAULT '',
+ service_name TEXT NOT NULL DEFAULT '',
+ scope_name TEXT NOT NULL DEFAULT '',
+ scope_version TEXT NOT NULL DEFAULT '',
+ start_time_unix_nano TEXT NOT NULL DEFAULT '0',
+ end_time_unix_nano TEXT NOT NULL DEFAULT '0',
+ duration INTEGER NOT NULL DEFAULT 0,
+ recorded_at DATETIME NOT NULL,
+ trace_state TEXT NOT NULL DEFAULT '',
+ flags INTEGER NOT NULL DEFAULT 0,
+ dropped_attributes_count INTEGER NOT NULL DEFAULT 0,
+ dropped_events_count INTEGER NOT NULL DEFAULT 0,
+ dropped_links_count INTEGER NOT NULL DEFAULT 0,
+ resource_schema_url TEXT NOT NULL DEFAULT '',
+ scope_schema_url TEXT NOT NULL DEFAULT '',
+ span_attributes TEXT NOT NULL DEFAULT '{}',
+ resource TEXT NOT NULL DEFAULT '{}',
+ scope TEXT NOT NULL DEFAULT '{}',
+ events TEXT NOT NULL DEFAULT '[]',
+ links TEXT NOT NULL DEFAULT '[]',
+ otlp BLOB
+);
+
+CREATE INDEX IF NOT EXISTS idx_spans_v2_project_id_trace_id_recorded_at ON spans_v2(project_id, trace_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_spans_v2_project_id_recorded_at ON spans_v2(project_id, recorded_at);
+
+CREATE TABLE IF NOT EXISTS endpoints_v2 (
+ id TEXT NOT NULL,
+ project_id TEXT NOT NULL,
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
+ parent_span_id TEXT NOT NULL DEFAULT '',
+ linked_trace_id TEXT NOT NULL DEFAULT '',
+ endpoint TEXT NOT NULL DEFAULT '',
+ duration INTEGER NOT NULL DEFAULT 0,
+ recorded_at DATETIME NOT NULL,
+ status_code INTEGER NOT NULL DEFAULT 0,
+ body_size INTEGER NOT NULL DEFAULT 0,
+ client_ip TEXT NOT NULL DEFAULT '',
+ attributes TEXT NOT NULL DEFAULT '{}',
+ app_version TEXT NOT NULL DEFAULT '',
+ server_name TEXT NOT NULL DEFAULT '',
+ is_stream INTEGER NOT NULL DEFAULT 0,
+ is_root INTEGER NOT NULL DEFAULT 1
+);
+
+CREATE INDEX IF NOT EXISTS idx_endpoints_v2_project_id_endpoint_recorded_at ON endpoints_v2(project_id, endpoint, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_endpoints_v2_project_id_recorded_at ON endpoints_v2(project_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_endpoints_v2_trace_id ON endpoints_v2(trace_id);
+
+CREATE INDEX IF NOT EXISTS idx_endpoints_v2_linked_trace_id ON endpoints_v2(linked_trace_id) WHERE linked_trace_id != '';
+
+CREATE TABLE IF NOT EXISTS tasks_v2 (
+ id TEXT NOT NULL,
+ project_id TEXT NOT NULL,
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
+ parent_span_id TEXT NOT NULL DEFAULT '',
+ linked_trace_id TEXT NOT NULL DEFAULT '',
+ task_name TEXT NOT NULL DEFAULT '',
+ duration INTEGER NOT NULL DEFAULT 0,
+ recorded_at DATETIME NOT NULL,
+ client_ip TEXT NOT NULL DEFAULT '',
+ attributes TEXT NOT NULL DEFAULT '{}',
+ app_version TEXT NOT NULL DEFAULT '',
+ server_name TEXT NOT NULL DEFAULT '',
+ is_root INTEGER NOT NULL DEFAULT 1
+);
+
+CREATE INDEX IF NOT EXISTS idx_tasks_v2_project_id_task_name_recorded_at ON tasks_v2(project_id, task_name, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_tasks_v2_project_id_recorded_at ON tasks_v2(project_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_tasks_v2_trace_id ON tasks_v2(trace_id);
+
+CREATE INDEX IF NOT EXISTS idx_tasks_v2_linked_trace_id ON tasks_v2(linked_trace_id) WHERE linked_trace_id != '';
+
+CREATE TABLE IF NOT EXISTS ai_traces_v2 (
+ id TEXT NOT NULL,
+ project_id TEXT NOT NULL,
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
+ parent_span_id TEXT NOT NULL DEFAULT '',
+ linked_trace_id TEXT NOT NULL DEFAULT '',
+ recorded_at DATETIME NOT NULL,
+ duration INTEGER NOT NULL DEFAULT 0,
+ status_code INTEGER NOT NULL DEFAULT 0,
+ model TEXT NOT NULL DEFAULT '',
+ response_model TEXT NOT NULL DEFAULT '',
+ provider TEXT NOT NULL DEFAULT '',
+ operation TEXT NOT NULL DEFAULT '',
+ input_tokens INTEGER NOT NULL DEFAULT 0,
+ output_tokens INTEGER NOT NULL DEFAULT 0,
+ total_tokens INTEGER NOT NULL DEFAULT 0,
+ cached_tokens INTEGER NOT NULL DEFAULT 0,
+ reasoning_tokens INTEGER NOT NULL DEFAULT 0,
+ input_cost REAL NOT NULL DEFAULT 0,
+ output_cost REAL NOT NULL DEFAULT 0,
+ total_cost REAL NOT NULL DEFAULT 0,
+ trace_name TEXT NOT NULL DEFAULT '',
+ user_id TEXT NOT NULL DEFAULT '',
+ finish_reason TEXT NOT NULL DEFAULT '',
+ server_name TEXT NOT NULL DEFAULT '',
+ app_version TEXT NOT NULL DEFAULT '',
+ storage_key TEXT NOT NULL DEFAULT '',
+ attributes TEXT NOT NULL DEFAULT '{}',
+ is_root INTEGER NOT NULL DEFAULT 1,
+ conversation_id TEXT NOT NULL DEFAULT '',
+ tool_call_count INTEGER NOT NULL DEFAULT 0,
+ tool_names TEXT NOT NULL DEFAULT '',
+ flagged INTEGER NOT NULL DEFAULT 0,
+ flagged_terms TEXT NOT NULL DEFAULT ''
+);
+
+CREATE INDEX IF NOT EXISTS idx_ai_traces_v2_project_id_conversation_id_recorded_at ON ai_traces_v2(project_id, conversation_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_ai_traces_v2_project_id_trace_name_recorded_at ON ai_traces_v2(project_id, trace_name, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_ai_traces_v2_project_id_recorded_at ON ai_traces_v2(project_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_ai_traces_v2_trace_id ON ai_traces_v2(trace_id);
+
+CREATE INDEX IF NOT EXISTS idx_ai_traces_v2_linked_trace_id ON ai_traces_v2(linked_trace_id) WHERE linked_trace_id != '';
+
+CREATE TABLE IF NOT EXISTS exceptions_v2 (
+ id TEXT NOT NULL,
+ project_id TEXT NOT NULL,
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
+ trace_type TEXT NOT NULL DEFAULT '',
+ exception_hash TEXT NOT NULL DEFAULT '',
+ stack_trace TEXT NOT NULL DEFAULT '',
+ recorded_at DATETIME NOT NULL,
+ attributes TEXT NOT NULL DEFAULT '{}',
+ app_version TEXT NOT NULL DEFAULT '',
+ server_name TEXT NOT NULL DEFAULT '',
+ is_message INTEGER NOT NULL DEFAULT 0,
+ linked_trace_id TEXT NOT NULL DEFAULT '',
+ session_id TEXT DEFAULT NULL
+);
+
+CREATE INDEX IF NOT EXISTS idx_exceptions_v2_session_id_recorded_at ON exceptions_v2(session_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_exceptions_v2_project_id_exception_hash_recorded_at ON exceptions_v2(project_id, exception_hash, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_exceptions_v2_project_id_recorded_at ON exceptions_v2(project_id, recorded_at);
+
+CREATE INDEX IF NOT EXISTS idx_exceptions_v2_trace_id ON exceptions_v2(trace_id);
+
+CREATE INDEX IF NOT EXISTS idx_exceptions_v2_linked_trace_id ON exceptions_v2(linked_trace_id) WHERE linked_trace_id != '';
diff --git a/backend/app/migrations/sqlite_telemetry/0026_clean_trace_identity.up.sql b/backend/app/migrations/sqlite_telemetry/0026_clean_trace_identity.up.sql
new file mode 100644
index 00000000..0671fd37
--- /dev/null
+++ b/backend/app/migrations/sqlite_telemetry/0026_clean_trace_identity.up.sql
@@ -0,0 +1,25 @@
+ALTER TABLE sessions ADD COLUMN trace_id TEXT NOT NULL DEFAULT '';
+
+UPDATE sessions SET trace_id = CASE
+ WHEN length(lower(replace(COALESCE(distributed_trace_id, ''), '-', ''))) = 32 AND lower(replace(COALESCE(distributed_trace_id, ''), '-', '')) NOT GLOB '*[^0-9a-f]*' AND lower(replace(COALESCE(distributed_trace_id, ''), '-', '')) != '00000000000000000000000000000000'
+ THEN lower(replace(COALESCE(distributed_trace_id, ''), '-', '')) ELSE '' END;
+
+ALTER TABLE sessions DROP COLUMN distributed_trace_id;
+
+DROP INDEX IF EXISTS idx_endpoints_v2_linked_trace_id;
+
+ALTER TABLE endpoints_v2 DROP COLUMN linked_trace_id;
+
+DROP INDEX IF EXISTS idx_tasks_v2_linked_trace_id;
+
+ALTER TABLE tasks_v2 DROP COLUMN linked_trace_id;
+
+DROP INDEX IF EXISTS idx_ai_traces_v2_linked_trace_id;
+
+ALTER TABLE ai_traces_v2 DROP COLUMN linked_trace_id;
+
+DROP INDEX IF EXISTS idx_exceptions_v2_linked_trace_id;
+
+ALTER TABLE exceptions_v2 DROP COLUMN linked_trace_id;
+
+ALTER TABLE profiles DROP COLUMN distributed_trace_id;
diff --git a/backend/app/migrations/trace_identity_ch_test.go b/backend/app/migrations/trace_identity_ch_test.go
new file mode 100644
index 00000000..4f1105f1
--- /dev/null
+++ b/backend/app/migrations/trace_identity_ch_test.go
@@ -0,0 +1,53 @@
+//go:build telemetry_ch
+
+package migrations
+
+import (
+ "database/sql"
+ "io/fs"
+ "net/url"
+ "os"
+ "strings"
+ "testing"
+
+ _ "github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/google/uuid"
+)
+
+func traceIdentityDatabase(t *testing.T) (*sql.DB, fs.FS, string, string, func(fs.FS) error) {
+ server := os.Getenv("TEST_CLICKHOUSE_SERVER")
+ if server == "" {
+ t.Skip("TEST_CLICKHOUSE_SERVER not set, skipping ClickHouse migration tests")
+ }
+ username := os.Getenv("TEST_CLICKHOUSE_USERNAME")
+ if username == "" {
+ username = "default"
+ }
+ params := url.Values{"username": {username}, "password": {os.Getenv("TEST_CLICKHOUSE_PASSWORD")}}
+ dsn := "clickhouse://" + server + "?"
+ admin, err := sql.Open("clickhouse", dsn+params.Encode())
+ if err != nil {
+ t.Fatal(err)
+ }
+ name := "trace_identity_test_" + strings.ReplaceAll(uuid.NewString(), "-", "")
+ if _, err := admin.Exec("CREATE DATABASE " + name); err != nil {
+ admin.Close()
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ if _, err := admin.Exec("DROP DATABASE " + name + " SYNC"); err != nil {
+ t.Errorf("clean up migration test database: %v", err)
+ }
+ admin.Close()
+ })
+ params.Set("database", name)
+ targetDSN := dsn + params.Encode()
+ target, err := sql.Open("clickhouse", targetDSN)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { target.Close() })
+ return target, migrationsChFS, "ch", "0091", func(source fs.FS) error {
+ return runMigrationsClickhouseFrom(targetDSN, source)
+ }
+}
diff --git a/backend/app/migrations/trace_identity_duckdb_test.go b/backend/app/migrations/trace_identity_duckdb_test.go
new file mode 100644
index 00000000..f2eb9b88
--- /dev/null
+++ b/backend/app/migrations/trace_identity_duckdb_test.go
@@ -0,0 +1,22 @@
+//go:build telemetry_duckdb
+
+package migrations
+
+import (
+ "database/sql"
+ "io/fs"
+ "testing"
+
+ _ "github.com/duckdb/duckdb-go/v2"
+)
+
+func traceIdentityDatabase(t *testing.T) (*sql.DB, fs.FS, string, string, func(fs.FS) error) {
+ target, err := sql.Open("duckdb", "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { target.Close() })
+ return target, migrationsDuckDBTelemetryFS, "duckdb_telemetry", "0006", func(source fs.FS) error {
+ return runMigrationsOn(target, source, "duckdb_telemetry", "schema_migrations", duckdbTrackingDDL)
+ }
+}
diff --git a/backend/app/migrations/trace_identity_sqlite_test.go b/backend/app/migrations/trace_identity_sqlite_test.go
new file mode 100644
index 00000000..3bd79bc2
--- /dev/null
+++ b/backend/app/migrations/trace_identity_sqlite_test.go
@@ -0,0 +1,16 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package migrations
+
+import (
+ "database/sql"
+ "io/fs"
+ "testing"
+)
+
+func traceIdentityDatabase(t *testing.T) (*sql.DB, fs.FS, string, string, func(fs.FS) error) {
+ target := openMemorySQLite(t)
+ return target, migrationsSqliteTelemetryFS, "sqlite_telemetry", "0026", func(source fs.FS) error {
+ return runMigrationsOn(target, source, "sqlite_telemetry", "schema_migrations", sqliteTrackingDDL)
+ }
+}
diff --git a/backend/app/migrations/trace_identity_test.go b/backend/app/migrations/trace_identity_test.go
new file mode 100644
index 00000000..c92cfbd8
--- /dev/null
+++ b/backend/app/migrations/trace_identity_test.go
@@ -0,0 +1,106 @@
+package migrations
+
+import (
+ "database/sql"
+ "fmt"
+ "io/fs"
+ "strings"
+ "testing"
+ "testing/fstest"
+)
+
+func TestTraceIdentityFreshInstall(t *testing.T) {
+ target, source, _, _, run := traceIdentityDatabase(t)
+ if err := run(source); err != nil {
+ t.Fatal(err)
+ }
+ for _, table := range []string{"sessions", "endpoints_v2", "tasks_v2", "ai_traces_v2", "exceptions_v2", "profiles"} {
+ assertCleanTraceColumns(t, target, table)
+ }
+}
+
+func TestTraceIdentityUpgrade(t *testing.T) {
+ target, source, dir, firstCleanup, run := traceIdentityDatabase(t)
+ before := fstest.MapFS{}
+ entries, err := fs.ReadDir(source, dir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, entry := range entries {
+ if entry.Name() < firstCleanup && strings.HasSuffix(entry.Name(), ".up.sql") {
+ path := dir + "/" + entry.Name()
+ data, err := fs.ReadFile(source, path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ before[path] = &fstest.MapFile{Data: data}
+ }
+ }
+ if err := run(before); err != nil {
+ t.Fatal(err)
+ }
+ exec := func(query string) {
+ t.Helper()
+ if _, err := target.Exec(query); err != nil {
+ t.Fatalf("%s: %v", query, err)
+ }
+ }
+ const project = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
+ const trace = "0102030405060708090a0b0c0d0e0f10"
+ const span = "0102030405060708"
+ for i, old := range []string{"'01020304-0506-0708-090A-0B0C0D0E0F10'", "NULL", "'00000000-0000-0000-0000-000000000000'"} {
+ exec(fmt.Sprintf("INSERT INTO sessions (id, project_id, started_at, distributed_trace_id) VALUES ('00000000-0000-0000-0000-%012d', '%s', '2026-09-21 12:00:00', %s)", i+1, project, old))
+ }
+ for _, table := range []string{"endpoints_v2", "tasks_v2", "ai_traces_v2", "exceptions_v2", "profiles"} {
+ legacy := "linked_trace_id"
+ if table == "profiles" {
+ legacy = "distributed_trace_id"
+ }
+ exec(fmt.Sprintf("INSERT INTO %s (id, project_id, recorded_at, trace_id, span_id, %s) VALUES ('11111111-1111-1111-1111-111111111111', '%s', '2026-09-21 12:00:00', '%s', '%s', '99999999-9999-9999-9999-999999999999')", table, legacy, project, trace, span))
+ }
+ if err := run(source); err != nil {
+ t.Fatal(err)
+ }
+ for i, want := range []string{trace, "", ""} {
+ var got string
+ if err := target.QueryRow(fmt.Sprintf("SELECT trace_id FROM sessions WHERE id = '00000000-0000-0000-0000-%012d'", i+1)).Scan(&got); err != nil || got != want {
+ t.Fatalf("session %d: trace_id = %q, want %q: %v", i, got, want, err)
+ }
+ }
+ for _, table := range []string{"endpoints_v2", "tasks_v2", "ai_traces_v2", "exceptions_v2", "profiles"} {
+ var gotTrace, gotSpan string
+ if err := target.QueryRow("SELECT trace_id, span_id FROM "+table).Scan(&gotTrace, &gotSpan); err != nil || gotTrace != trace || gotSpan != span {
+ t.Fatalf("%s lost its actual trace/span identity: %q/%q: %v", table, gotTrace, gotSpan, err)
+ }
+ }
+ for _, table := range []string{"sessions", "endpoints_v2", "tasks_v2", "ai_traces_v2", "exceptions_v2", "profiles"} {
+ assertCleanTraceColumns(t, target, table)
+ }
+ // A restart must not repeat a destructive migration or erase the converted IDs.
+ if err := run(source); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func assertCleanTraceColumns(t *testing.T, target *sql.DB, table string) {
+ t.Helper()
+ rows, err := target.Query("SELECT * FROM " + table + " LIMIT 0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ columns, err := rows.Columns()
+ if err != nil {
+ t.Fatal(err)
+ }
+ found := false
+ for _, column := range columns {
+ if column == "linked_trace_id" || column == "distributed_trace_id" {
+ t.Errorf("%s retains obsolete column %s", table, column)
+ }
+ found = found || column == "trace_id"
+ }
+ if !found {
+ t.Errorf("%s is missing trace_id", table)
+ }
+}
diff --git a/backend/app/migrations/v2_tables_ch_test.go b/backend/app/migrations/v2_tables_ch_test.go
new file mode 100644
index 00000000..aaf95c47
--- /dev/null
+++ b/backend/app/migrations/v2_tables_ch_test.go
@@ -0,0 +1,9 @@
+//go:build telemetry_ch
+
+package migrations
+
+import "testing"
+
+func TestV2MigrationsOnlyCreate(t *testing.T) {
+ assertV2MigrationsOnlyCreate(t, migrationsChFS, "ch")
+}
diff --git a/backend/app/migrations/v2_tables_duckdb_test.go b/backend/app/migrations/v2_tables_duckdb_test.go
new file mode 100644
index 00000000..a1061aef
--- /dev/null
+++ b/backend/app/migrations/v2_tables_duckdb_test.go
@@ -0,0 +1,9 @@
+//go:build telemetry_duckdb
+
+package migrations
+
+import "testing"
+
+func TestV2MigrationsOnlyCreate(t *testing.T) {
+ assertV2MigrationsOnlyCreate(t, migrationsDuckDBTelemetryFS, "duckdb_telemetry")
+}
diff --git a/backend/app/migrations/v2_tables_sqlite_test.go b/backend/app/migrations/v2_tables_sqlite_test.go
new file mode 100644
index 00000000..962896b6
--- /dev/null
+++ b/backend/app/migrations/v2_tables_sqlite_test.go
@@ -0,0 +1,9 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package migrations
+
+import "testing"
+
+func TestV2MigrationsOnlyCreate(t *testing.T) {
+ assertV2MigrationsOnlyCreate(t, migrationsSqliteTelemetryFS, "sqlite_telemetry")
+}
diff --git a/backend/app/migrations/v2_tables_test.go b/backend/app/migrations/v2_tables_test.go
new file mode 100644
index 00000000..f16a5e93
--- /dev/null
+++ b/backend/app/migrations/v2_tables_test.go
@@ -0,0 +1,52 @@
+package migrations
+
+import (
+ "embed"
+ "io/fs"
+ "regexp"
+ "strings"
+ "testing"
+)
+
+// Other tables carry a _v2 suffix of their own (sessions), so the five are named.
+var v2TelemetryMigration = regexp.MustCompile(`_create_(v2_tables|(spans|endpoints|tasks|ai_traces|exceptions)_v2)\.up\.sql$`)
+
+// V2 lives next to the tables it replaces. Its migrations only create, so an upgrade rewrites nothing, a rollback
+// finds the old tables as they were, and the move-over script has an untouched source to read.
+func assertV2MigrationsOnlyCreate(t *testing.T, source embed.FS, dir string) {
+ t.Helper()
+ found := 0
+ err := fs.WalkDir(source, dir, func(path string, entry fs.DirEntry, err error) error {
+ if err != nil || entry.IsDir() || !v2TelemetryMigration.MatchString(path) {
+ return err
+ }
+ found++
+ data, err := source.ReadFile(path)
+ if err != nil {
+ return err
+ }
+ for _, statement := range splitStatements(string(data)) {
+ upper := strings.ToUpper(strings.Join(strings.Fields(statement), " "))
+ if upper == "" {
+ continue
+ }
+ if !strings.HasPrefix(upper, "CREATE TABLE IF NOT EXISTS ") && !strings.HasPrefix(upper, "CREATE INDEX IF NOT EXISTS ") {
+ t.Errorf("%s: a V2 migration only creates tables and indexes, got: %.80s", path, upper)
+ }
+ target := strings.Fields(strings.TrimPrefix(strings.TrimPrefix(upper, "CREATE TABLE IF NOT EXISTS "), "CREATE INDEX IF NOT EXISTS "))
+ if strings.HasPrefix(upper, "CREATE INDEX") && len(target) >= 3 {
+ target = target[2:]
+ }
+ if len(target) == 0 || !strings.Contains(strings.SplitN(target[0], "(", 2)[0], "_V2") {
+ t.Errorf("%s: a V2 migration must not touch the tables V2 replaces: %.80s", path, upper)
+ }
+ }
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if found == 0 {
+ t.Fatalf("no V2 migrations under %s", dir)
+ }
+}
diff --git a/backend/app/models/ai_trace.model.go b/backend/app/models/ai_trace.model.go
index bea5bb6c..ce3f70b9 100644
--- a/backend/app/models/ai_trace.model.go
+++ b/backend/app/models/ai_trace.model.go
@@ -7,37 +7,40 @@ import (
)
type AiTrace struct {
- Id uuid.UUID `json:"id" ch:"id"`
- ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
- RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
- Duration time.Duration `json:"duration" ch:"duration"`
- StatusCode uint8 `json:"statusCode" ch:"status_code"`
- Model string `json:"model" ch:"model"`
- ResponseModel string `json:"responseModel" ch:"response_model"`
- Provider string `json:"provider" ch:"provider"`
- Operation string `json:"operation" ch:"operation"`
- InputTokens int64 `json:"inputTokens" ch:"input_tokens"`
- OutputTokens int64 `json:"outputTokens" ch:"output_tokens"`
- TotalTokens int64 `json:"totalTokens" ch:"total_tokens"`
- CachedTokens int64 `json:"cachedTokens" ch:"cached_tokens"`
- ReasoningTokens int64 `json:"reasoningTokens" ch:"reasoning_tokens"`
- InputCost float64 `json:"inputCost" ch:"input_cost"`
- OutputCost float64 `json:"outputCost" ch:"output_cost"`
- TotalCost float64 `json:"totalCost" ch:"total_cost"`
- TraceName string `json:"traceName" ch:"trace_name"`
- UserId string `json:"userId" ch:"user_id"`
- FinishReason string `json:"finishReason" ch:"finish_reason"`
- ServerName string `json:"serverName" ch:"server_name"`
- AppVersion string `json:"appVersion" ch:"app_version"`
- StorageKey string `json:"storageKey" ch:"storage_key"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
- IsRoot bool `json:"isRoot" ch:"is_root"`
- ConversationId string `json:"conversationId" ch:"conversation_id"`
- ToolCallCount int64 `json:"toolCallCount" ch:"tool_call_count"`
- ToolNames []string `json:"toolNames" ch:"tool_names"`
- Flagged bool `json:"flagged" ch:"flagged"`
- FlaggedTerms []string `json:"flaggedTerms" ch:"flagged_terms"`
+ Id uuid.UUID `json:"id" ch:"id"`
+ ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
+ RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
+ Duration time.Duration `json:"duration" ch:"duration"`
+ StatusCode uint8 `json:"statusCode" ch:"status_code"`
+ Model string `json:"model" ch:"model"`
+ ResponseModel string `json:"responseModel" ch:"response_model"`
+ Provider string `json:"provider" ch:"provider"`
+ Operation string `json:"operation" ch:"operation"`
+ InputTokens int64 `json:"inputTokens" ch:"input_tokens"`
+ OutputTokens int64 `json:"outputTokens" ch:"output_tokens"`
+ TotalTokens int64 `json:"totalTokens" ch:"total_tokens"`
+ CachedTokens int64 `json:"cachedTokens" ch:"cached_tokens"`
+ ReasoningTokens int64 `json:"reasoningTokens" ch:"reasoning_tokens"`
+ InputCost float64 `json:"inputCost" ch:"input_cost"`
+ OutputCost float64 `json:"outputCost" ch:"output_cost"`
+ TotalCost float64 `json:"totalCost" ch:"total_cost"`
+ TraceName string `json:"traceName" ch:"trace_name"`
+ UserId string `json:"userId" ch:"user_id"`
+ FinishReason string `json:"finishReason" ch:"finish_reason"`
+ ServerName string `json:"serverName" ch:"server_name"`
+ AppVersion string `json:"appVersion" ch:"app_version"`
+ StorageKey string `json:"storageKey" ch:"storage_key"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+
+ TraceId string `json:"traceId" ch:"trace_id"`
+ SpanId string `json:"spanId" ch:"span_id"`
+ ParentSpanId string `json:"parentSpanId,omitempty" ch:"parent_span_id"`
+ IsRoot bool `json:"isRoot" ch:"is_root"`
+ ConversationId string `json:"conversationId" ch:"conversation_id"`
+ ToolCallCount int64 `json:"toolCallCount" ch:"tool_call_count"`
+ ToolNames []string `json:"toolNames" ch:"tool_names"`
+ Flagged bool `json:"flagged" ch:"flagged"`
+ FlaggedTerms []string `json:"flaggedTerms" ch:"flagged_terms"`
}
type AiTraceStats struct {
diff --git a/backend/app/models/clientmodels/clientmodels.model.go b/backend/app/models/clientmodels/clientmodels.model.go
index e6d241e2..b091380e 100644
--- a/backend/app/models/clientmodels/clientmodels.model.go
+++ b/backend/app/models/clientmodels/clientmodels.model.go
@@ -1,11 +1,13 @@
package clientmodels
import (
+ "encoding/hex"
"encoding/json"
- "github.com/tracewayapp/traceway/backend/app/models"
+ "fmt"
"time"
"github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
)
type ClientExceptionStackTrace struct {
@@ -17,7 +19,6 @@ type ClientExceptionStackTrace struct {
IsMessage bool `json:"isMessage"`
SessionRecordingId *string `json:"sessionRecordingId"`
SessionId *string `json:"sessionId"`
- DistributedTraceId *string `json:"distributedTraceId"`
DebugIds map[string]string `json:"debugIds"`
}
@@ -27,17 +28,13 @@ func (c *ClientExceptionStackTrace) ToExceptionStackTrace(exceptionHash, appVers
traceType = "task"
}
- var traceId *uuid.UUID
+ // In the native protocol a "trace" is one endpoint or task run, and traceId names it. That run is stored as the root
+ // span of a trace, so the exception sits on that span.
+ var traceId, spanId string
if c.TraceId != nil {
- if parsed, err := uuid.Parse(*c.TraceId); err == nil {
- traceId = &parsed
- }
- }
-
- var distributedTraceId *uuid.UUID
- if c.DistributedTraceId != nil {
- if parsed, err := uuid.Parse(*c.DistributedTraceId); err == nil {
- distributedTraceId = &parsed
+ if parsed, err := uuid.Parse(*c.TraceId); err == nil && parsed != uuid.Nil {
+ spanId = hexId(parsed)
+ traceId = spanId
}
}
@@ -49,17 +46,17 @@ func (c *ClientExceptionStackTrace) ToExceptionStackTrace(exceptionHash, appVers
}
return models.ExceptionStackTrace{
- ExceptionHash: exceptionHash,
- TraceId: traceId,
- TraceType: traceType,
- StackTrace: c.StackTrace,
- RecordedAt: c.RecordedAt,
- Attributes: c.Attributes,
- IsMessage: c.IsMessage,
- AppVersion: appVersion,
- ServerName: serverName,
- DistributedTraceId: distributedTraceId,
- SessionId: sessionId,
+ ExceptionHash: exceptionHash,
+ TraceId: traceId,
+ SpanId: spanId,
+ TraceType: traceType,
+ StackTrace: c.StackTrace,
+ RecordedAt: c.RecordedAt,
+ Attributes: c.Attributes,
+ IsMessage: c.IsMessage,
+ AppVersion: appVersion,
+ ServerName: serverName,
+ SessionId: sessionId,
}
}
@@ -87,98 +84,130 @@ func (c *ClientMetricRecord) ToMetricPoint(serverName string) models.MetricPoint
}
type ClientTrace struct {
- Id string `json:"id"`
- Endpoint string `json:"endpoint"`
- Duration time.Duration `json:"duration"`
- RecordedAt time.Time `json:"recordedAt"`
- StatusCode int `json:"statusCode"`
- BodySize int `json:"bodySize"`
- ClientIP string `json:"clientIP"`
- Attributes map[string]string `json:"attributes"`
- Spans []*ClientSpan `json:"spans"`
- IsTask bool `json:"isTask"`
- DistributedTraceId string `json:"distributedTraceId"`
+ Id string `json:"id"`
+ Endpoint string `json:"endpoint"`
+ Duration time.Duration `json:"duration"`
+ RecordedAt time.Time `json:"recordedAt"`
+ StatusCode int `json:"statusCode"`
+ BodySize int `json:"bodySize"`
+ ClientIP string `json:"clientIP"`
+ Attributes map[string]string `json:"attributes"`
+ Spans []*ClientSpan `json:"spans"`
+ IsTask bool `json:"isTask"`
}
const streamAttributeKey = "traceway.is_stream"
// ParsedId returns the trace ID as uuid.UUID
func (c *ClientTrace) ParsedId() uuid.UUID {
- if parsed, err := uuid.Parse(c.Id); err == nil {
+ if parsed, err := uuid.Parse(c.Id); err == nil && parsed != uuid.Nil {
return parsed
}
- return uuid.New()
+ c.Id = uuid.NewString()
+ return uuid.MustParse(c.Id)
}
-func (c *ClientTrace) parsedDistributedTraceId() *uuid.UUID {
- if c.DistributedTraceId == "" {
- return nil
- }
- if parsed, err := uuid.Parse(c.DistributedTraceId); err == nil {
- return &parsed
- }
- return nil
-}
+func hexId(id uuid.UUID) string { return hex.EncodeToString(id[:]) }
+
+// SpanId is the id of the run itself: a native trace is stored as the root span of a trace.
+func (c *ClientTrace) SpanId() string { return hexId(c.ParsedId()) }
+
+func (c *ClientTrace) TraceId() string { return c.SpanId() }
func (c *ClientTrace) ToEndpoint(appVersion, serverName string) models.Endpoint {
return models.Endpoint{
- Id: c.ParsedId(),
- Endpoint: c.Endpoint,
- Duration: c.Duration,
- RecordedAt: c.RecordedAt,
- StatusCode: int16(c.StatusCode),
- BodySize: int32(c.BodySize),
- ClientIP: c.ClientIP,
- Attributes: c.Attributes,
- AppVersion: appVersion,
- ServerName: serverName,
- DistributedTraceId: c.parsedDistributedTraceId(),
- IsStream: c.Attributes[streamAttributeKey] == "true",
- IsRoot: true,
+ Id: c.ParsedId(),
+ Endpoint: c.Endpoint,
+ Duration: c.Duration,
+ RecordedAt: c.RecordedAt,
+ StatusCode: int16(c.StatusCode),
+ BodySize: int32(c.BodySize),
+ ClientIP: c.ClientIP,
+ Attributes: c.Attributes,
+ AppVersion: appVersion,
+ ServerName: serverName,
+ TraceId: c.TraceId(),
+ SpanId: c.SpanId(),
+ IsStream: c.Attributes[streamAttributeKey] == "true",
+ IsRoot: true,
}
}
func (c *ClientTrace) ToTask(appVersion, serverName string) models.Task {
return models.Task{
- Id: c.ParsedId(),
- TaskName: c.Endpoint,
- Duration: c.Duration,
- RecordedAt: c.RecordedAt,
- ClientIP: c.ClientIP,
- Attributes: c.Attributes,
- AppVersion: appVersion,
- ServerName: serverName,
- DistributedTraceId: c.parsedDistributedTraceId(),
- IsRoot: true,
+ Id: c.ParsedId(),
+ TaskName: c.Endpoint,
+ Duration: c.Duration,
+ RecordedAt: c.RecordedAt,
+ ClientIP: c.ClientIP,
+ Attributes: c.Attributes,
+ AppVersion: appVersion,
+ ServerName: serverName,
+ TraceId: c.TraceId(),
+ SpanId: c.SpanId(),
+ IsRoot: true,
}
}
type ClientSpan struct {
- Id string `json:"id"`
- Name string `json:"name"`
- StartTime time.Time `json:"startTime"`
- Duration time.Duration `json:"duration"`
+ ParentSpanId string `json:"parentSpanId,omitempty"`
+ Attributes map[string]string `json:"attributes,omitempty"`
+ Id string `json:"id"`
+ Name string `json:"name"`
+ StartTime time.Time `json:"startTime"`
+ Duration time.Duration `json:"duration"`
}
// ParsedId returns the span ID as uuid.UUID
func (c *ClientSpan) ParsedId() uuid.UUID {
- if parsed, err := uuid.Parse(c.Id); err == nil {
+ if parsed, err := uuid.Parse(c.Id); err == nil && parsed != uuid.Nil {
return parsed
}
- return uuid.New()
+ c.Id = uuid.NewString()
+ return uuid.MustParse(c.Id)
}
-func (c *ClientSpan) ToSpan(traceId uuid.UUID) models.Span {
+// ToSpan places a client span in its run's trace. A span that names no parent hangs under the run's root span.
+func (c *ClientSpan) ToSpan(run *ClientTrace, projectId uuid.UUID, serverName string) models.Span {
+ parent := run.SpanId()
+ if parsed, err := uuid.Parse(c.ParentSpanId); err == nil && parsed != uuid.Nil {
+ parent = hexId(parsed)
+ }
return models.Span{
- Id: c.ParsedId(),
- TraceId: traceId,
- Name: c.Name,
- StartTime: c.StartTime,
- Duration: c.Duration,
- RecordedAt: time.Now(),
+ ProjectId: projectId,
+ TraceId: run.TraceId(),
+ SpanId: hexId(c.ParsedId()),
+ ParentSpanId: parent,
+ Attributes: c.Attributes,
+ Name: c.Name,
+ StartTime: c.StartTime,
+ Duration: c.Duration,
+ RecordedAt: run.RecordedAt,
+ ServiceName: serverName,
}
}
+const (
+ spanKindInternal = 1
+ spanKindServer = 2
+ spanStatusError = 2
+)
+
+// RootSpan is the run itself as a span, so a native trace reads like any other: a root with its children under it.
+func (c *ClientTrace) RootSpan(projectId uuid.UUID, serverName string) models.Span {
+ span := models.Span{
+ ProjectId: projectId, TraceId: c.TraceId(), SpanId: c.SpanId(), Name: c.Endpoint, StartTime: c.RecordedAt,
+ Duration: c.Duration, RecordedAt: c.RecordedAt, SpanKind: spanKindServer, ServiceName: serverName, Attributes: c.Attributes,
+ }
+ if c.IsTask {
+ span.SpanKind = spanKindInternal
+ }
+ if c.StatusCode >= 500 {
+ span.StatusCode = spanStatusError
+ }
+ return span
+}
+
type ClientSessionRecording struct {
ExceptionId string `json:"exceptionId"`
SessionId string `json:"sessionId,omitempty"`
@@ -197,12 +226,11 @@ type ClientSessionRecording struct {
}
type ClientSession struct {
- Id string `json:"id"`
- StartedAt time.Time `json:"startedAt"`
- EndedAt *time.Time `json:"endedAt,omitempty"`
- ClientIP string `json:"clientIP"`
- Attributes map[string]string `json:"attributes"`
- DistributedTraceId string `json:"distributedTraceId,omitempty"`
+ Id string `json:"id"`
+ StartedAt time.Time `json:"startedAt"`
+ EndedAt *time.Time `json:"endedAt,omitempty"`
+ ClientIP string `json:"clientIP"`
+ Attributes map[string]string `json:"attributes"`
}
func (c *ClientSession) ToSession(appVersion, serverName string) models.Session {
@@ -211,13 +239,6 @@ func (c *ClientSession) ToSession(appVersion, serverName string) models.Session
id = uuid.New()
}
- var distributedTraceId *uuid.UUID
- if c.DistributedTraceId != "" {
- if parsed, err := uuid.Parse(c.DistributedTraceId); err == nil {
- distributedTraceId = &parsed
- }
- }
-
var duration int64
if c.EndedAt != nil {
duration = c.EndedAt.Sub(c.StartedAt).Nanoseconds()
@@ -227,15 +248,14 @@ func (c *ClientSession) ToSession(appVersion, serverName string) models.Session
}
return models.Session{
- Id: id,
- StartedAt: c.StartedAt,
- EndedAt: c.EndedAt,
- Duration: duration,
- ClientIP: c.ClientIP,
- Attributes: c.Attributes,
- AppVersion: appVersion,
- ServerName: serverName,
- DistributedTraceId: distributedTraceId,
+ Id: id,
+ StartedAt: c.StartedAt,
+ EndedAt: c.EndedAt,
+ Duration: duration,
+ ClientIP: c.ClientIP,
+ Attributes: c.Attributes,
+ AppVersion: appVersion,
+ ServerName: serverName,
}
}
@@ -246,3 +266,20 @@ type CollectionFrame struct {
SessionRecordings []*ClientSessionRecording `json:"sessionRecordings"`
Sessions []*ClientSession `json:"sessions"`
}
+
+func (f *CollectionFrame) Validate() error {
+ if f == nil {
+ return fmt.Errorf("collection frame must not be null")
+ }
+ for _, exception := range f.StackTraces {
+ if exception == nil {
+ return fmt.Errorf("exception must not be null")
+ }
+ }
+ for _, session := range f.Sessions {
+ if session == nil {
+ return fmt.Errorf("session must not be null")
+ }
+ }
+ return nil
+}
diff --git a/backend/app/models/clientmodels/clientmodels_test.go b/backend/app/models/clientmodels/clientmodels_test.go
new file mode 100644
index 00000000..20e985a8
--- /dev/null
+++ b/backend/app/models/clientmodels/clientmodels_test.go
@@ -0,0 +1,58 @@
+package clientmodels
+
+import (
+ "encoding/json"
+ "testing"
+
+ "github.com/google/uuid"
+)
+
+func TestNativeMissingIDsAreGeneratedOnce(t *testing.T) {
+ for _, input := range []string{"", "invalid", uuid.Nil.String(), hexId(uuid.Nil)} {
+ t.Run(input, func(t *testing.T) {
+ run := &ClientTrace{Id: input}
+ child := &ClientSpan{Id: input, ParentSpanId: uuid.Nil.String()}
+ project := uuid.New()
+ root := run.RootSpan(project, "")
+ endpoint, task := run.ToEndpoint("", ""), run.ToTask("", "")
+ if endpoint.Id == uuid.Nil || endpoint.Id != task.Id || root.TraceId != hexId(endpoint.Id) || root.SpanId != root.TraceId || endpoint.TraceId != root.TraceId || task.TraceId != root.TraceId {
+ t.Fatalf("generated run identity is inconsistent: root=%+v endpoint=%+v task=%+v", root, endpoint, task)
+ }
+ span := child.ToSpan(run, project, "")
+ if child.ParsedId() == uuid.Nil || span.SpanId != hexId(child.ParsedId()) || span.TraceId != root.TraceId || span.ParentSpanId != root.SpanId {
+ t.Fatalf("generated child identity or parent changed: %+v", span)
+ }
+ })
+ }
+}
+
+func TestNativeExceptionIgnoresZeroTraceIDs(t *testing.T) {
+ run, distributed, zero := uuid.NewString(), uuid.NewString(), uuid.Nil.String()
+ for _, tt := range []struct {
+ name string
+ run, distributed *string
+ wantTrace, wantSpan string
+ }{
+ {"untraced", nil, nil, "", ""},
+ {"zero IDs", &zero, &zero, "", ""},
+ {"zero distributed", &run, &zero, hexId(uuid.MustParse(run)), hexId(uuid.MustParse(run))},
+ {"missing distributed", &run, nil, hexId(uuid.MustParse(run)), hexId(uuid.MustParse(run))},
+ {"valid distributed ignored", &run, &distributed, hexId(uuid.MustParse(run)), hexId(uuid.MustParse(run))},
+ {"zero run", &zero, &distributed, "", ""},
+ } {
+ t.Run(tt.name, func(t *testing.T) {
+ body, err := json.Marshal(map[string]any{"traceId": tt.run, "distributedTraceId": tt.distributed})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var input ClientExceptionStackTrace
+ if err := json.Unmarshal(body, &input); err != nil {
+ t.Fatal(err)
+ }
+ got := input.ToExceptionStackTrace("hash", "", "")
+ if got.TraceId != tt.wantTrace || got.SpanId != tt.wantSpan {
+ t.Fatalf("exception identity = %q/%q, want %q/%q", got.TraceId, got.SpanId, tt.wantTrace, tt.wantSpan)
+ }
+ })
+ }
+}
diff --git a/backend/app/models/clientmodels/trace_context_test.go b/backend/app/models/clientmodels/trace_context_test.go
new file mode 100644
index 00000000..497ec16a
--- /dev/null
+++ b/backend/app/models/clientmodels/trace_context_test.go
@@ -0,0 +1,39 @@
+package clientmodels
+
+import (
+ "encoding/json"
+ "testing"
+)
+
+func TestReportIgnoresLegacyDistributedTraceID(t *testing.T) {
+ const run = "11223344-5566-7788-99aa-bbccddeeff00"
+ const runHex = "112233445566778899aabbccddeeff00"
+ for _, value := range []any{
+ "0102030405060708090a0b0c0d0e0f10",
+ "01020304-0506-0708-090A-0B0C0D0E0F10",
+ "", "00000000000000000000000000000000", "invalid", nil, 42,
+ } {
+ body, err := json.Marshal(map[string]any{"id": run, "traceId": run, "distributedTraceId": value})
+ if err != nil {
+ t.Fatal(err)
+ }
+ var trace ClientTrace
+ var exception ClientExceptionStackTrace
+ var session ClientSession
+ for _, target := range []any{&trace, &exception, &session} {
+ if err := json.Unmarshal(body, target); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if trace.ToEndpoint("", "").TraceId != runHex || trace.ToTask("", "").TraceId != runHex {
+ t.Fatalf("legacy field changed native run identity: %v", value)
+ }
+ got := exception.ToExceptionStackTrace("hash", "", "")
+ if got.TraceId != runHex || got.SpanId != runHex {
+ t.Fatalf("legacy field changed exception ownership: %+v", got)
+ }
+ if got := session.ToSession("", "").TraceId; got != "" {
+ t.Fatalf("legacy field created session association: %q", got)
+ }
+ }
+}
diff --git a/backend/app/models/endpoint.model.go b/backend/app/models/endpoint.model.go
index 0334136b..23ae42ec 100644
--- a/backend/app/models/endpoint.model.go
+++ b/backend/app/models/endpoint.model.go
@@ -10,19 +10,21 @@ type Endpoint struct {
Id uuid.UUID `json:"id" ch:"id"`
ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
// endpoint is the route from the router/does not contain actual params so it's safe to group on it
- Endpoint string `json:"endpoint" ch:"endpoint"`
- Duration time.Duration `json:"duration" ch:"duration"`
- RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
- StatusCode int16 `json:"statusCode" ch:"status_code"`
- BodySize int32 `json:"bodySize" ch:"body_size"`
- ClientIP string `json:"clientIP" ch:"client_ip"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
- AppVersion string `json:"appVersion" ch:"app_version"`
- ServerName string `json:"serverName" ch:"server_name"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
- SpanId *uuid.UUID `json:"spanId,omitempty" ch:"span_id"`
- IsStream bool `json:"isStream" ch:"is_stream"`
- IsRoot bool `json:"isRoot" ch:"is_root"`
+ Endpoint string `json:"endpoint" ch:"endpoint"`
+ Duration time.Duration `json:"duration" ch:"duration"`
+ RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
+ StatusCode int16 `json:"statusCode" ch:"status_code"`
+ BodySize int32 `json:"bodySize" ch:"body_size"`
+ ClientIP string `json:"clientIP" ch:"client_ip"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+ AppVersion string `json:"appVersion" ch:"app_version"`
+ ServerName string `json:"serverName" ch:"server_name"`
+
+ TraceId string `json:"traceId" ch:"trace_id"`
+ SpanId string `json:"spanId" ch:"span_id"`
+ ParentSpanId string `json:"parentSpanId,omitempty" ch:"parent_span_id"`
+ IsStream bool `json:"isStream" ch:"is_stream"`
+ IsRoot bool `json:"isRoot" ch:"is_root"`
}
type EndpointStats struct {
diff --git a/backend/app/models/exception_stack_trace.model.go b/backend/app/models/exception_stack_trace.model.go
index c7a7eb7e..d1769496 100644
--- a/backend/app/models/exception_stack_trace.model.go
+++ b/backend/app/models/exception_stack_trace.model.go
@@ -7,19 +7,21 @@ import (
)
type ExceptionStackTrace struct {
- Id uuid.UUID `json:"id" ch:"id"`
- ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
- TraceId *uuid.UUID `json:"traceId" ch:"trace_id"`
- TraceType string `json:"traceType" ch:"trace_type"` // "endpoint" or "task"
- ExceptionHash string `json:"exceptionHash" ch:"exception_hash"`
- StackTrace string `json:"stackTrace" ch:"stack_trace"`
- RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
- AppVersion string `json:"appVersion" ch:"app_version"`
- ServerName string `json:"serverName" ch:"server_name"`
- IsMessage bool `json:"isMessage" ch:"is_message"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
- SessionId *uuid.UUID `json:"sessionId,omitempty" ch:"session_id"`
+ Id uuid.UUID `json:"id" ch:"id"`
+ ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
+ // TraceId is the trace the exception happened in and SpanId the span it happened on, as lowercase hex.
+ TraceId string `json:"traceId" ch:"trace_id"`
+ SpanId string `json:"spanId" ch:"span_id"`
+ // TraceType is set when it is known without a lookup: native clients say it, and an exception on a promoted span has that span's kind.
+ TraceType string `json:"traceType" ch:"trace_type"`
+ ExceptionHash string `json:"exceptionHash" ch:"exception_hash"`
+ StackTrace string `json:"stackTrace" ch:"stack_trace"`
+ RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+ AppVersion string `json:"appVersion" ch:"app_version"`
+ ServerName string `json:"serverName" ch:"server_name"`
+ IsMessage bool `json:"isMessage" ch:"is_message"`
+ SessionId *uuid.UUID `json:"sessionId,omitempty" ch:"session_id"`
}
type ExceptionTrendPoint struct {
diff --git a/backend/app/models/otel_span.model.go b/backend/app/models/otel_span.model.go
new file mode 100644
index 00000000..1178c4df
--- /dev/null
+++ b/backend/app/models/otel_span.model.go
@@ -0,0 +1,11 @@
+package models
+
+import (
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+)
+
+type OtelSpan struct {
+ Span
+ OTLP *tracepb.Span `json:"-"`
+ Context *tracepb.ResourceSpans `json:"-"`
+}
diff --git a/backend/app/models/profile.model.go b/backend/app/models/profile.model.go
index 3ee131b6..915124c0 100644
--- a/backend/app/models/profile.model.go
+++ b/backend/app/models/profile.model.go
@@ -61,21 +61,20 @@ type ProfileSeriesGroup struct {
}
type Profile struct {
- Id uuid.UUID `json:"id" ch:"id"`
- ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
- RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
- Duration time.Duration `json:"duration" ch:"duration"`
- ServiceName string `json:"serviceName" ch:"service_name"`
- ProfileType string `json:"profileType" ch:"profile_type"`
- Unit string `json:"unit" ch:"unit"`
- IsGauge bool `json:"isGauge" ch:"is_gauge"`
- SampleCount uint64 `json:"sampleCount" ch:"sample_count"`
- TotalValue int64 `json:"totalValue" ch:"total_value"`
- ServerName string `json:"serverName" ch:"server_name"`
- AppVersion string `json:"appVersion" ch:"app_version"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
- StorageKey string `json:"storageKey" ch:"storage_key"`
- TraceId string `json:"traceId" ch:"trace_id"`
- SpanId string `json:"spanId" ch:"span_id"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
+ Id uuid.UUID `json:"id" ch:"id"`
+ ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
+ RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
+ Duration time.Duration `json:"duration" ch:"duration"`
+ ServiceName string `json:"serviceName" ch:"service_name"`
+ ProfileType string `json:"profileType" ch:"profile_type"`
+ Unit string `json:"unit" ch:"unit"`
+ IsGauge bool `json:"isGauge" ch:"is_gauge"`
+ SampleCount uint64 `json:"sampleCount" ch:"sample_count"`
+ TotalValue int64 `json:"totalValue" ch:"total_value"`
+ ServerName string `json:"serverName" ch:"server_name"`
+ AppVersion string `json:"appVersion" ch:"app_version"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+ StorageKey string `json:"storageKey" ch:"storage_key"`
+ TraceId string `json:"traceId" ch:"trace_id"`
+ SpanId string `json:"spanId" ch:"span_id"`
}
diff --git a/backend/app/models/session.model.go b/backend/app/models/session.model.go
index 8e8332d7..fd0886e6 100644
--- a/backend/app/models/session.model.go
+++ b/backend/app/models/session.model.go
@@ -7,14 +7,14 @@ import (
)
type Session struct {
- Id uuid.UUID `json:"id" ch:"id"`
- ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
- StartedAt time.Time `json:"startedAt" ch:"started_at"`
- EndedAt *time.Time `json:"endedAt,omitempty" ch:"ended_at"`
- Duration int64 `json:"duration" ch:"duration"`
- ClientIP string `json:"clientIP" ch:"client_ip"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
- AppVersion string `json:"appVersion" ch:"app_version"`
- ServerName string `json:"serverName" ch:"server_name"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
+ Id uuid.UUID `json:"id" ch:"id"`
+ ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
+ StartedAt time.Time `json:"startedAt" ch:"started_at"`
+ EndedAt *time.Time `json:"endedAt,omitempty" ch:"ended_at"`
+ Duration int64 `json:"duration" ch:"duration"`
+ ClientIP string `json:"clientIP" ch:"client_ip"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+ AppVersion string `json:"appVersion" ch:"app_version"`
+ ServerName string `json:"serverName" ch:"server_name"`
+ TraceId string `json:"traceId,omitempty" ch:"trace_id"`
}
diff --git a/backend/app/models/span.model.go b/backend/app/models/span.model.go
index ec6e7c64..0c0ec9da 100644
--- a/backend/app/models/span.model.go
+++ b/backend/app/models/span.model.go
@@ -6,14 +6,25 @@ import (
"github.com/google/uuid"
)
+// Span is one stored span. TraceId, SpanId and ParentSpanId are the ids it arrived with, as lowercase hex: 32
+// characters for a trace, 16 for an OTel span, 32 for a span from the native protocol.
type Span struct {
- Id uuid.UUID `json:"id" ch:"id"`
- TraceId uuid.UUID `json:"traceId" ch:"trace_id"`
- ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
- Name string `json:"name" ch:"name"`
- StartTime time.Time `json:"startTime" ch:"start_time"`
- Duration time.Duration `json:"duration" ch:"duration"`
- RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
- ParentSpanId *uuid.UUID `json:"parentSpanId,omitempty" ch:"parent_span_id"`
- Attributes map[string]string `json:"attributes,omitempty" ch:"attributes"`
+ ProjectId uuid.UUID `json:"projectId"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ ParentSpanId string `json:"parentSpanId,omitempty"`
+
+ Name string `json:"name"`
+ StartTime time.Time `json:"startTime"`
+ Duration time.Duration `json:"duration"`
+ RecordedAt time.Time `json:"recordedAt"`
+ SpanKind int32 `json:"spanKind,omitempty"`
+ StatusCode int32 `json:"statusCode,omitempty"`
+ ServiceName string `json:"serviceName,omitempty"`
+ ScopeName string `json:"scopeName,omitempty"`
+
+ Attributes map[string]string `json:"attributes,omitempty"`
+ AttributesOmitted bool `json:"attributesOmitted,omitempty"`
+ // DbStatement is a short preview of the database statement, sent when a read leaves the attributes out.
+ DbStatement string `json:"dbStatement,omitempty"`
}
diff --git a/backend/app/models/span_graph.model.go b/backend/app/models/span_graph.model.go
new file mode 100644
index 00000000..23751978
--- /dev/null
+++ b/backend/app/models/span_graph.model.go
@@ -0,0 +1,22 @@
+package models
+
+const (
+ SpanGraphComplete = "complete"
+ SpanGraphPartial = "partial"
+ SpanGraphUnavailable = "unavailable"
+
+ SpanGraphRowLimit = "row_limit"
+ SpanGraphAttributeSize = "attribute_size"
+ SpanGraphAttributeBudget = "attribute_budget"
+ SpanGraphAttributesUnavailable = "attributes_unavailable"
+ SpanGraphReadLimit = "read_limit"
+ // SpanGraphMostImportant accompanies row_limit on a whole trace read: the spans kept are errors, entry points and the slowest, not the earliest.
+ SpanGraphMostImportant = "most_important"
+)
+
+// SpanGraphStatus tells a reader whether the spans beside it are the whole graph.
+type SpanGraphStatus struct {
+ State string `json:"state"`
+ Reasons []string `json:"reasons,omitempty"`
+ OmittedAttributes int `json:"omittedAttributes,omitempty"`
+}
diff --git a/backend/app/models/task.model.go b/backend/app/models/task.model.go
index 0d4f655d..16bc4bfc 100644
--- a/backend/app/models/task.model.go
+++ b/backend/app/models/task.model.go
@@ -7,18 +7,20 @@ import (
)
type Task struct {
- Id uuid.UUID `json:"id" ch:"id"`
- ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
- TaskName string `json:"taskName" ch:"task_name"`
- Duration time.Duration `json:"duration" ch:"duration"`
- RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
- ClientIP string `json:"clientIP" ch:"client_ip"`
- Attributes map[string]string `json:"attributes" ch:"attributes"`
- AppVersion string `json:"appVersion" ch:"app_version"`
- ServerName string `json:"serverName" ch:"server_name"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty" ch:"distributed_trace_id"`
- SpanId *uuid.UUID `json:"spanId,omitempty" ch:"span_id"`
- IsRoot bool `json:"isRoot" ch:"is_root"`
+ Id uuid.UUID `json:"id" ch:"id"`
+ ProjectId uuid.UUID `json:"projectId" ch:"project_id"`
+ TaskName string `json:"taskName" ch:"task_name"`
+ Duration time.Duration `json:"duration" ch:"duration"`
+ RecordedAt time.Time `json:"recordedAt" ch:"recorded_at"`
+ ClientIP string `json:"clientIP" ch:"client_ip"`
+ Attributes map[string]string `json:"attributes" ch:"attributes"`
+ AppVersion string `json:"appVersion" ch:"app_version"`
+ ServerName string `json:"serverName" ch:"server_name"`
+
+ TraceId string `json:"traceId" ch:"trace_id"`
+ SpanId string `json:"spanId" ch:"span_id"`
+ ParentSpanId string `json:"parentSpanId,omitempty" ch:"parent_span_id"`
+ IsRoot bool `json:"isRoot" ch:"is_root"`
}
type TaskStats struct {
diff --git a/backend/app/monitoring/ch_reporter.go b/backend/app/monitoring/ch_reporter.go
index ed977814..cd7c0b36 100644
--- a/backend/app/monitoring/ch_reporter.go
+++ b/backend/app/monitoring/ch_reporter.go
@@ -23,10 +23,17 @@ var telemetryTables = []string{
"traces",
"log_records",
"metric_points",
+ "spans_v2",
+ "exceptions_v2",
+ "endpoints_v2",
+ "tasks_v2",
+ "ai_traces_v2",
+ // Backfill retains the legacy tables; their parts still consume capacity.
"spans",
"exception_stack_traces",
"endpoints",
"tasks",
+ "ai_traces",
}
type chBaselines struct {
diff --git a/backend/app/notifications/evaluator_event.go b/backend/app/notifications/evaluator_event.go
index 7486526c..6dd3403a 100644
--- a/backend/app/notifications/evaluator_event.go
+++ b/backend/app/notifications/evaluator_event.go
@@ -27,7 +27,7 @@ func evaluateNewError(ctx context.Context, rule *models.NotificationRuleWithChan
var count uint64
err := chdb.Conn.QueryRow(ctx,
- "SELECT count() FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ?",
+ "SELECT count() FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ?",
event.ProjectId, hash).Scan(&count)
if err != nil {
traceway.CaptureException(fmt.Errorf("new_error check failed: %w", err))
@@ -45,7 +45,7 @@ func evaluateNewError(ctx context.Context, rule *models.NotificationRuleWithChan
var postArchiveCount uint64
archErr = chdb.Conn.QueryRow(ctx,
- "SELECT count() FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ? AND recorded_at > (SELECT max(archived_at) FROM archived_exceptions FINAL WHERE project_id = ? AND exception_hash = ?)",
+ "SELECT count() FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ? AND recorded_at > (SELECT max(archived_at) FROM archived_exceptions FINAL WHERE project_id = ? AND exception_hash = ?)",
event.ProjectId, hash, event.ProjectId, hash).Scan(&postArchiveCount)
if archErr != nil {
traceway.CaptureException(fmt.Errorf("new_error post-archive count failed: %w", archErr))
@@ -105,13 +105,12 @@ func evaluateErrorRegression(ctx context.Context, rule *models.NotificationRuleW
func getExceptionDetails(ctx context.Context, projectId uuid.UUID, hash string) ExceptionDetails {
var id uuid.UUID
- var traceId *uuid.UUID
- var stackTrace, traceType, appVersion, serverName, attributesJSON string
+ var stackTrace, traceId, spanId, traceType, appVersion, serverName, attributesJSON string
var recordedAt time.Time
err := chdb.Conn.QueryRow(ctx,
- "SELECT id, trace_id, trace_type, stack_trace, attributes, app_version, server_name, recorded_at FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ? ORDER BY recorded_at DESC LIMIT 1",
- projectId, hash).Scan(&id, &traceId, &traceType, &stackTrace, &attributesJSON, &appVersion, &serverName, &recordedAt)
+ "SELECT id, trace_id, span_id, trace_type, stack_trace, attributes, app_version, server_name, recorded_at FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ? ORDER BY recorded_at DESC LIMIT 1",
+ projectId, hash).Scan(&id, &traceId, &spanId, &traceType, &stackTrace, &attributesJSON, &appVersion, &serverName, &recordedAt)
details := ExceptionDetails{
Hash: hash,
@@ -127,8 +126,9 @@ func getExceptionDetails(ctx context.Context, projectId uuid.UUID, hash string)
details.AppVersion = appVersion
details.ServerName = serverName
details.RecordedAt = recordedAt
- details.TraceType = traceType
- details.TraceName = resolveTraceName(ctx, projectId, traceId, traceType, &details.RecordedAt)
+ details.TraceType, details.TraceName = resolveExceptionOwner(ctx, models.ExceptionStackTrace{
+ Id: id, ProjectId: projectId, TraceId: traceId, SpanId: spanId, RecordedAt: recordedAt,
+ })
if attributesJSON != "" && attributesJSON != "{}" {
attrs := make(map[string]string)
diff --git a/backend/app/notifications/evaluator_event_sqlite.go b/backend/app/notifications/evaluator_event_sqlite.go
index 947117a1..e85cd2ff 100644
--- a/backend/app/notifications/evaluator_event_sqlite.go
+++ b/backend/app/notifications/evaluator_event_sqlite.go
@@ -28,7 +28,7 @@ func evaluateNewError(ctx context.Context, rule *models.NotificationRuleWithChan
var count int64
err := db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ?",
+ "SELECT COUNT(*) FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ?",
event.ProjectId.String(), hash).Scan(&count)
if err != nil {
traceway.CaptureException(fmt.Errorf("new_error check failed: %w", err))
@@ -46,7 +46,7 @@ func evaluateNewError(ctx context.Context, rule *models.NotificationRuleWithChan
var postArchiveCount int64
archErr = db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ? AND recorded_at > (SELECT MAX(archived_at) FROM archived_exceptions WHERE project_id = ? AND exception_hash = ?)",
+ "SELECT COUNT(*) FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ? AND recorded_at > (SELECT MAX(archived_at) FROM archived_exceptions WHERE project_id = ? AND exception_hash = ?)",
event.ProjectId.String(), hash, event.ProjectId.String(), hash).Scan(&postArchiveCount)
if archErr != nil {
traceway.CaptureException(fmt.Errorf("new_error post-archive count failed: %w", archErr))
@@ -104,12 +104,11 @@ func evaluateErrorRegression(ctx context.Context, rule *models.NotificationRuleW
}
func getExceptionDetails(ctx context.Context, projectId uuid.UUID, hash string) ExceptionDetails {
- var idStr, stackTrace, traceType, appVersion, serverName, attributesJSON, recordedAtStr string
- var traceIdStr sql.NullString
+ var idStr, stackTrace, traceId, spanId, traceType, appVersion, serverName, attributesJSON, recordedAtStr string
err := db.TelemetryDB.QueryRowContext(ctx,
- "SELECT id, trace_id, trace_type, stack_trace, attributes, app_version, server_name, recorded_at FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ? ORDER BY recorded_at DESC LIMIT 1",
- projectId.String(), hash).Scan(&idStr, &traceIdStr, &traceType, &stackTrace, &attributesJSON, &appVersion, &serverName, &recordedAtStr)
+ "SELECT id, trace_id, span_id, trace_type, stack_trace, attributes, app_version, server_name, recorded_at FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ? ORDER BY recorded_at DESC LIMIT 1",
+ projectId.String(), hash).Scan(&idStr, &traceId, &spanId, &traceType, &stackTrace, &attributesJSON, &appVersion, &serverName, &recordedAtStr)
details := ExceptionDetails{
Hash: hash,
@@ -129,15 +128,10 @@ func getExceptionDetails(ctx context.Context, projectId uuid.UUID, hash string)
details.AppVersion = appVersion
details.ServerName = serverName
details.RecordedAt, _ = time.Parse(time.RFC3339Nano, recordedAtStr)
- details.TraceType = traceType
-
- var traceId *uuid.UUID
- if traceIdStr.Valid {
- if parsed, parseErr := uuid.Parse(traceIdStr.String); parseErr == nil {
- traceId = &parsed
- }
- }
- details.TraceName = resolveTraceName(ctx, projectId, traceId, traceType, &details.RecordedAt)
+ exceptionId, _ := uuid.Parse(idStr)
+ details.TraceType, details.TraceName = resolveExceptionOwner(ctx, models.ExceptionStackTrace{
+ Id: exceptionId, ProjectId: projectId, TraceId: traceId, SpanId: spanId, RecordedAt: details.RecordedAt,
+ })
if attributesJSON != "" && attributesJSON != "{}" {
attrs := make(map[string]string)
diff --git a/backend/app/notifications/evaluator_helpers.go b/backend/app/notifications/evaluator_helpers.go
index bfe6266b..96571859 100644
--- a/backend/app/notifications/evaluator_helpers.go
+++ b/backend/app/notifications/evaluator_helpers.go
@@ -248,51 +248,17 @@ func extractErrorType(stackTrace string) string {
return "Unknown Error"
}
-func resolveTraceName(ctx context.Context, projectId uuid.UUID, traceId *uuid.UUID, traceType string, recordedAt *time.Time) string {
- if traceId == nil {
- return ""
+// resolveExceptionOwner names the endpoint, task or AI trace an exception happened in, for the notification's text.
+func resolveExceptionOwner(ctx context.Context, exception models.ExceptionStackTrace) (traceType, name string) {
+ owner, err := telemetry.FindExceptionOwner(ctx, exception)
+ if err != nil {
+ traceway.CaptureException(fmt.Errorf("failed to resolve the entity of exception %s for a notification: %w", exception.Id, err))
+ return "", ""
}
- switch traceType {
- case "task":
- task, err := telemetry.TaskRepository.FindById(ctx, projectId, *traceId, recordedAt)
- if task == nil && err == nil && recordedAt != nil {
- task, err = telemetry.TaskRepository.FindById(ctx, projectId, *traceId, nil)
- }
- if err != nil {
- traceway.CaptureException(fmt.Errorf("failed to resolve task name for notification: %w", err))
- return ""
- }
- if task == nil {
- return ""
- }
- return task.TaskName
- case "ai_trace":
- trace, err := telemetry.AiTraceRepository.FindById(ctx, projectId, *traceId, recordedAt)
- if trace == nil && err == nil && recordedAt != nil {
- trace, err = telemetry.AiTraceRepository.FindById(ctx, projectId, *traceId, nil)
- }
- if err != nil {
- traceway.CaptureException(fmt.Errorf("failed to resolve ai trace name for notification: %w", err))
- return ""
- }
- if trace == nil {
- return ""
- }
- return trace.TraceName
- default:
- endpoint, err := telemetry.EndpointRepository.FindById(ctx, projectId, *traceId, recordedAt)
- if endpoint == nil && err == nil && recordedAt != nil {
- endpoint, err = telemetry.EndpointRepository.FindById(ctx, projectId, *traceId, nil)
- }
- if err != nil {
- traceway.CaptureException(fmt.Errorf("failed to resolve endpoint name for notification: %w", err))
- return ""
- }
- if endpoint == nil {
- return ""
- }
- return endpoint.Endpoint
+ if owner == nil {
+ return "", ""
}
+ return owner.TraceType, owner.Name
}
func getProjectName(projectId uuid.UUID) string {
diff --git a/backend/app/notifications/evaluator_polled.go b/backend/app/notifications/evaluator_polled.go
index f07cda2a..78783b7d 100644
--- a/backend/app/notifications/evaluator_polled.go
+++ b/backend/app/notifications/evaluator_polled.go
@@ -60,7 +60,7 @@ func evaluateErrorRateThreshold(ctx context.Context, rule *models.NotificationRu
var total, errors uint64
err := chdb.Conn.QueryRow(ctx,
- "SELECT count() as total, countIf(status_code >= 500) as errors FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT count() as total, countIf(status_code >= 500) as errors FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
projectId, from, now).Scan(&total, &errors)
if err != nil {
return nil, err
@@ -101,7 +101,7 @@ func evaluateEndpointP95Threshold(ctx context.Context, rule *models.Notification
from := now.Add(-time.Duration(cfg.LookbackMinutes) * time.Minute)
// duration is stored in nanoseconds, convert to milliseconds
- query := "SELECT quantile(0.95)(duration / 1000000) as p95 FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT quantile(0.95)(duration / 1000000) as p95 FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId, from, now}
if cfg.Endpoint != "" && cfg.Endpoint != "*" {
query += " AND endpoint = ?"
@@ -141,7 +141,7 @@ func evaluateEndpointP99Threshold(ctx context.Context, rule *models.Notification
now := time.Now().UTC()
from := now.Add(-time.Duration(cfg.LookbackMinutes) * time.Minute)
- query := "SELECT quantile(0.99)(duration / 1000000) as p99 FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT quantile(0.99)(duration / 1000000) as p99 FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId, from, now}
if cfg.Endpoint != "" && cfg.Endpoint != "*" {
query += " AND endpoint = ?"
@@ -193,7 +193,7 @@ func evaluateApdexDrop(ctx context.Context, rule *models.NotificationRule, proje
`SELECT count() as total,
countIf(duration <= 750000000 AND status_code < 500) as satisfied,
countIf(duration > 750000000 AND duration <= 1500000000 AND status_code < 500) as tolerating
- FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?`,
+ FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?`,
projectId, from, now).Scan(&total, &satisfied, &tolerating)
if err != nil {
return nil, err
@@ -306,7 +306,7 @@ func evaluateNoData(ctx context.Context, rule *models.NotificationRule, projectI
threshold := time.Now().UTC().Add(-time.Duration(cfg.SilenceMinutes) * time.Minute)
if cfg.DataType == "any" {
- tables := []string{"endpoints", "exception_stack_traces", "metric_points", "tasks"}
+ tables := []string{"endpoints_v2", "exceptions_v2", "metric_points", "tasks_v2"}
for _, t := range tables {
var maxTs time.Time
err := chdb.Conn.QueryRow(ctx,
@@ -324,13 +324,13 @@ func evaluateNoData(ctx context.Context, rule *models.NotificationRule, projectI
table := ""
switch cfg.DataType {
case "endpoints":
- table = "endpoints"
+ table = "endpoints_v2"
case "exceptions":
- table = "exception_stack_traces"
+ table = "exceptions_v2"
case "metrics":
table = "metric_points"
case "tasks":
- table = "tasks"
+ table = "tasks_v2"
default:
return nil, fmt.Errorf("unknown data type: %s", cfg.DataType)
}
@@ -373,7 +373,7 @@ func evaluateErrorCountThreshold(ctx context.Context, rule *models.NotificationR
var count uint64
err := chdb.Conn.QueryRow(ctx,
- "SELECT count() FROM exception_stack_traces WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_message = 0",
+ "SELECT count() FROM exceptions_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_message = 0",
projectId, from, now).Scan(&count)
if err != nil {
return nil, err
@@ -409,7 +409,7 @@ func evaluateTaskDurationThreshold(ctx context.Context, rule *models.Notificatio
from := now.Add(-time.Duration(cfg.LookbackMinutes) * time.Minute)
// duration is stored in nanoseconds, convert to milliseconds
- query := "SELECT quantile(0.95)(duration / 1000000) as p95 FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT quantile(0.95)(duration / 1000000) as p95 FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId, from, now}
if cfg.TaskName != "" && cfg.TaskName != "*" {
query += " AND task_name = ?"
@@ -438,7 +438,7 @@ func evaluateTaskDurationThreshold(ctx context.Context, rule *models.Notificatio
// --- Task Failure Rate ---
func countTaskExecutions(ctx context.Context, projectId uuid.UUID, taskName string, named bool, from, to time.Time) (int64, error) {
- query := "SELECT count() FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT count() FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId, from, to}
if named {
query += " AND task_name = ?"
@@ -453,14 +453,15 @@ func countTaskExecutions(ctx context.Context, projectId uuid.UUID, taskName stri
}
func countFailedTaskExecutions(ctx context.Context, projectId uuid.UUID, taskName string, named bool, from, to time.Time) (int64, error) {
- query := "SELECT countDistinct(trace_id) FROM exception_stack_traces WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?" +
- " AND trace_id IN (SELECT id FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ // A task failed when an exception was recorded on its own span. One caught further down did not fail it.
+ query := "SELECT count() FROM (SELECT DISTINCT trace_id, span_id FROM exceptions_v2 WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?" +
+ " AND (trace_id, span_id) IN (SELECT trace_id, span_id FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId, from, to, projectId, from, to}
if named {
query += " AND task_name = ?"
args = append(args, taskName)
}
- query += ")"
+ query += ")) AS failed_tasks"
var failed uint64
if err := chdb.Conn.QueryRow(ctx, query, args...).Scan(&failed); err != nil {
@@ -495,7 +496,7 @@ func evaluateThroughputDrop(ctx context.Context, rule *models.NotificationRule,
var currentCount uint64
err := chdb.Conn.QueryRow(ctx,
- "SELECT count() FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT count() FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
projectId, lookbackFrom, now).Scan(&currentCount)
if err != nil {
return nil, err
@@ -503,7 +504,7 @@ func evaluateThroughputDrop(ctx context.Context, rule *models.NotificationRule,
var baselineCount uint64
err = chdb.Conn.QueryRow(ctx,
- "SELECT count() FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT count() FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
projectId, baselineFrom, lookbackFrom).Scan(&baselineCount)
if err != nil {
return nil, err
@@ -551,7 +552,7 @@ func evaluateEndpointErrorRate(ctx context.Context, rule *models.NotificationRul
var total, errors uint64
err := chdb.Conn.QueryRow(ctx,
- "SELECT count() as total, countIf(status_code >= 500) as errors FROM endpoints WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT count() as total, countIf(status_code >= 500) as errors FROM endpoints_v2 WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?",
projectId, cfg.Endpoint, from, now).Scan(&total, &errors)
if err != nil {
return nil, err
@@ -624,7 +625,7 @@ func computeImpactEndpoints(ctx context.Context, projectId uuid.UUID, minRequest
FROM (
SELECT e.endpoint, e.duration, e.status_code, e.recorded_at,
s.offset_ms as offset_ms
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN (SELECT * FROM slow_endpoints FINAL) AS s
ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE e.project_id = ? AND e.recorded_at >= ? AND e.recorded_at <= ? AND e.is_stream = 0
diff --git a/backend/app/notifications/evaluator_polled_sqlite.go b/backend/app/notifications/evaluator_polled_sqlite.go
index f418c355..4fe874f1 100644
--- a/backend/app/notifications/evaluator_polled_sqlite.go
+++ b/backend/app/notifications/evaluator_polled_sqlite.go
@@ -62,7 +62,7 @@ func evaluateErrorRateThreshold(ctx context.Context, rule *models.NotificationRu
var total, errors int64
err := db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) as total, COALESCE(SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END), 0) as errors FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT COUNT(*) as total, COALESCE(SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END), 0) as errors FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
projectId.String(), from.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)).Scan(&total, &errors)
if err != nil {
return nil, err
@@ -147,7 +147,7 @@ func evaluateEndpointP99Threshold(ctx context.Context, rule *models.Notification
}
func queryPercentile(ctx context.Context, projectId uuid.UUID, endpoint string, from, to time.Time, pct float64) float64 {
- query := "SELECT duration FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT duration FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId.String(), from.Format(time.RFC3339Nano), to.Format(time.RFC3339Nano)}
if endpoint != "" && endpoint != "*" {
query += " AND endpoint = ?"
@@ -215,7 +215,7 @@ func evaluateApdexDrop(ctx context.Context, rule *models.NotificationRule, proje
`SELECT COUNT(*) as total,
COALESCE(SUM(CASE WHEN duration <= 750000000 AND status_code < 500 THEN 1 ELSE 0 END), 0) as satisfied,
COALESCE(SUM(CASE WHEN duration > 750000000 AND duration <= 1500000000 AND status_code < 500 THEN 1 ELSE 0 END), 0) as tolerating
- FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?`,
+ FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?`,
projectId.String(), from.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)).Scan(&total, &satisfied, &tolerating)
if err != nil {
return nil, err
@@ -359,7 +359,7 @@ func evaluateNoData(ctx context.Context, rule *models.NotificationRule, projectI
pid := projectId.String()
if cfg.DataType == "any" {
- tables := []string{"endpoints", "exception_stack_traces", "metric_points", "tasks"}
+ tables := []string{"endpoints_v2", "exceptions_v2", "metric_points", "tasks_v2"}
for _, t := range tables {
var maxTs sql.NullString
err := db.TelemetryDB.QueryRowContext(ctx,
@@ -379,13 +379,13 @@ func evaluateNoData(ctx context.Context, rule *models.NotificationRule, projectI
table := ""
switch cfg.DataType {
case "endpoints":
- table = "endpoints"
+ table = "endpoints_v2"
case "exceptions":
- table = "exception_stack_traces"
+ table = "exceptions_v2"
case "metrics":
table = "metric_points"
case "tasks":
- table = "tasks"
+ table = "tasks_v2"
default:
return nil, fmt.Errorf("unknown data type: %s", cfg.DataType)
}
@@ -430,7 +430,7 @@ func evaluateErrorCountThreshold(ctx context.Context, rule *models.NotificationR
var count int64
err := db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) FROM exception_stack_traces WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_message = 0",
+ "SELECT COUNT(*) FROM exceptions_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_message = 0",
projectId.String(), from.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)).Scan(&count)
if err != nil {
return nil, err
@@ -465,7 +465,7 @@ func evaluateTaskDurationThreshold(ctx context.Context, rule *models.Notificatio
now := time.Now().UTC()
from := now.Add(-time.Duration(cfg.LookbackMinutes) * time.Minute)
- query := "SELECT duration FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT duration FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId.String(), from.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)}
if cfg.TaskName != "" && cfg.TaskName != "*" {
query += " AND task_name = ?"
@@ -506,7 +506,7 @@ func evaluateTaskDurationThreshold(ctx context.Context, rule *models.Notificatio
// --- Task Failure Rate ---
func countTaskExecutions(ctx context.Context, projectId uuid.UUID, taskName string, named bool, from, to time.Time) (int64, error) {
- query := "SELECT COUNT(*) FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ query := "SELECT COUNT(*) FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{projectId.String(), from.Format(time.RFC3339Nano), to.Format(time.RFC3339Nano)}
if named {
query += " AND task_name = ?"
@@ -525,14 +525,15 @@ func countFailedTaskExecutions(ctx context.Context, projectId uuid.UUID, taskNam
fromStr := from.Format(time.RFC3339Nano)
toStr := to.Format(time.RFC3339Nano)
- query := "SELECT COUNT(DISTINCT trace_id) FROM exception_stack_traces WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?" +
- " AND trace_id IN (SELECT id FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
+ // A task failed when an exception was recorded on its own span. One caught further down did not fail it.
+ query := "SELECT COUNT(*) FROM (SELECT DISTINCT trace_id, span_id FROM exceptions_v2 WHERE project_id = ? AND trace_type = 'task' AND recorded_at >= ? AND recorded_at <= ?" +
+ " AND (trace_id, span_id) IN (SELECT trace_id, span_id FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?"
args := []interface{}{pid, fromStr, toStr, pid, fromStr, toStr}
if named {
query += " AND task_name = ?"
args = append(args, taskName)
}
- query += ")"
+ query += ")) AS failed_tasks"
var failed int64
if err := db.TelemetryDB.QueryRowContext(ctx, query, args...).Scan(&failed); err != nil {
@@ -568,7 +569,7 @@ func evaluateThroughputDrop(ctx context.Context, rule *models.NotificationRule,
var currentCount int64
err := db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT COUNT(*) FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
pid, lookbackFrom.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)).Scan(&currentCount)
if err != nil {
return nil, err
@@ -576,7 +577,7 @@ func evaluateThroughputDrop(ctx context.Context, rule *models.NotificationRule,
var baselineCount int64
err = db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT COUNT(*) FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?",
pid, baselineFrom.Format(time.RFC3339Nano), lookbackFrom.Format(time.RFC3339Nano)).Scan(&baselineCount)
if err != nil {
return nil, err
@@ -625,7 +626,7 @@ func evaluateEndpointErrorRate(ctx context.Context, rule *models.NotificationRul
var total, errors int64
err := db.TelemetryDB.QueryRowContext(ctx,
- "SELECT COUNT(*) as total, COALESCE(SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END), 0) as errors FROM endpoints WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?",
+ "SELECT COUNT(*) as total, COALESCE(SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END), 0) as errors FROM endpoints_v2 WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?",
pid, cfg.Endpoint, from.Format(time.RFC3339Nano), now.Format(time.RFC3339Nano)).Scan(&total, &errors)
if err != nil {
return nil, err
@@ -662,7 +663,7 @@ func computeImpactEndpoints(ctx context.Context, projectId uuid.UUID, minRequest
COALESCE(SUM(CASE WHEN e.duration > (750000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.duration <= (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.status_code < 500 THEN 1 ELSE 0 END), 0) as tolerating_count,
COALESCE(SUM(CASE WHEN e.duration > (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) OR e.status_code >= 500 THEN 1 ELSE 0 END), 0) as bad_count,
COALESCE(SUM(CASE WHEN e.status_code >= 400 AND e.status_code < 500 THEN 1 ELSE 0 END), 0) as client_error_count
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE e.project_id = ? AND e.recorded_at >= ? AND e.recorded_at <= ? AND e.is_stream = 0
GROUP BY e.endpoint, COALESCE(s.offset_ms, 0)`,
@@ -707,11 +708,11 @@ func computeImpactEndpoints(ctx context.Context, projectId uuid.UUID, minRequest
SELECT endpoint, duration,
ROW_NUMBER() OVER (PARTITION BY endpoint ORDER BY duration) AS rn,
COUNT(*) OVER (PARTITION BY endpoint) AS cnt
- FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0
+ FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0
) WHERE rn = CAST(0.99 * (cnt - 1) AS INTEGER) + 1`
if db.IsDuckDBTelemetry() {
p99Query = `SELECT endpoint, CAST(quantile_cont(duration, 0.99) AS BIGINT) as duration
- FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0
+ FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0
GROUP BY endpoint`
}
diff --git a/backend/app/notifications/notifications_sqlite_test.go b/backend/app/notifications/notifications_sqlite_test.go
index 40a16287..84a19859 100644
--- a/backend/app/notifications/notifications_sqlite_test.go
+++ b/backend/app/notifications/notifications_sqlite_test.go
@@ -6,6 +6,7 @@ import (
"context"
"database/sql"
"encoding/json"
+ "fmt"
"path/filepath"
"sort"
"strings"
@@ -187,24 +188,27 @@ func setupNotificationsTestDB(t *testing.T) {
db.Driver = lit.SQLite
models.Init(db.Driver)
- if _, err := telemetryDB.Exec(`CREATE TABLE tasks (
+ if _, err := telemetryDB.Exec(`CREATE TABLE tasks_v2 (
id TEXT NOT NULL,
project_id TEXT NOT NULL,
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
task_name TEXT NOT NULL DEFAULT '',
duration INTEGER NOT NULL DEFAULT 0,
recorded_at DATETIME NOT NULL
)`); err != nil {
- t.Fatalf("failed to create tasks table: %v", err)
+ t.Fatalf("failed to create tasks_v2 table: %v", err)
}
- if _, err := telemetryDB.Exec(`CREATE TABLE exception_stack_traces (
+ if _, err := telemetryDB.Exec(`CREATE TABLE exceptions_v2 (
id TEXT NOT NULL,
project_id TEXT NOT NULL,
- trace_id TEXT,
- trace_type TEXT NOT NULL DEFAULT 'endpoint',
+ trace_id TEXT NOT NULL DEFAULT '',
+ span_id TEXT NOT NULL DEFAULT '',
+ trace_type TEXT NOT NULL DEFAULT '',
recorded_at DATETIME NOT NULL
)`); err != nil {
- t.Fatalf("failed to create exception_stack_traces table: %v", err)
+ t.Fatalf("failed to create exceptions_v2 table: %v", err)
}
}
@@ -216,26 +220,30 @@ func TestEvaluateTaskFailureRate(t *testing.T) {
pid := projectId.String()
now := time.Now().UTC()
- var taskIds []string
+ trace := func(i int) string { return fmt.Sprintf("%032x", i+1) }
+ span := func(i int) string { return fmt.Sprintf("%016x", i+1) }
for i := 0; i < 5; i++ {
- id := uuid.New().String()
- taskIds = append(taskIds, id)
ts := now.Add(-time.Duration(i+1) * time.Minute).Format(time.RFC3339Nano)
if _, err := db.TelemetryDB.Exec(
- "INSERT INTO tasks (id, project_id, task_name, duration, recorded_at) VALUES (?, ?, ?, ?, ?)",
- id, pid, "etl-job", 1000000, ts); err != nil {
+ "INSERT INTO tasks_v2 (id, project_id, trace_id, span_id, task_name, duration, recorded_at) VALUES (?, ?, ?, ?, ?, ?, ?)",
+ uuid.New().String(), pid, trace(i), span(i), "etl-job", 1000000, ts); err != nil {
t.Fatalf("failed to insert task: %v", err)
}
}
- for i := 0; i < 2; i++ {
+ exception := func(i int, spanId, traceType string) {
ts := now.Add(-time.Duration(i+1) * time.Minute).Format(time.RFC3339Nano)
if _, err := db.TelemetryDB.Exec(
- "INSERT INTO exception_stack_traces (id, project_id, trace_id, trace_type, recorded_at) VALUES (?, ?, ?, ?, ?)",
- uuid.New().String(), pid, taskIds[i], "task", ts); err != nil {
+ "INSERT INTO exceptions_v2 (id, project_id, trace_id, span_id, trace_type, recorded_at) VALUES (?, ?, ?, ?, ?, ?)",
+ uuid.New().String(), pid, trace(i), spanId, traceType, ts); err != nil {
t.Fatalf("failed to insert exception: %v", err)
}
}
+ // Two tasks failed, one of them twice on a retried export. A third caught an exception further down and finished.
+ exception(0, span(0), "task")
+ exception(0, span(0), "task")
+ exception(1, span(1), "task")
+ exception(2, "00000000000000ff", "")
t.Run("fires above threshold", func(t *testing.T) {
rule := &models.NotificationRule{
diff --git a/backend/app/notifications/task_failure_identity_ch_test.go b/backend/app/notifications/task_failure_identity_ch_test.go
new file mode 100644
index 00000000..2c4eb2e8
--- /dev/null
+++ b/backend/app/notifications/task_failure_identity_ch_test.go
@@ -0,0 +1,84 @@
+//go:build telemetry_ch
+
+package notifications
+
+import (
+ "context"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ ch "github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+)
+
+func TestTaskFailuresKeepTraceIdentityClickHouse(t *testing.T) {
+ server := os.Getenv("TEST_CLICKHOUSE_SERVER")
+ if server == "" {
+ t.Skip("TEST_CLICKHOUSE_SERVER not set")
+ }
+ ctx := context.Background()
+ auth := ch.Auth{Database: "default", Username: os.Getenv("TEST_CLICKHOUSE_USERNAME"), Password: os.Getenv("TEST_CLICKHOUSE_PASSWORD")}
+ if auth.Username == "" {
+ auth.Username = "default"
+ }
+ admin, err := ch.Open(&ch.Options{Addr: []string{server}, Auth: auth})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer admin.Close()
+ auth.Database = "task_identity_" + strings.ReplaceAll(uuid.NewString(), "-", "")
+ if err := admin.Exec(ctx, "CREATE DATABASE "+auth.Database); err != nil {
+ t.Fatal(err)
+ }
+ defer admin.Exec(ctx, "DROP DATABASE "+auth.Database)
+ conn, err := ch.Open(&ch.Options{Addr: []string{server}, Auth: auth})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close()
+ previous := chdb.Conn
+ chdb.Conn = conn
+ defer func() { chdb.Conn = previous }()
+ for _, query := range []string{
+ "CREATE TABLE tasks_v2 (project_id UUID, trace_id String, span_id String, task_name String, recorded_at DateTime64(6)) ENGINE=Memory",
+ "CREATE TABLE exceptions_v2 (project_id UUID, trace_id String, span_id String, trace_type String, recorded_at DateTime64(6)) ENGINE=Memory",
+ } {
+ if err := conn.Exec(ctx, query); err != nil {
+ t.Fatal(err)
+ }
+ }
+ project, at := uuid.New(), time.Now().UTC().Truncate(time.Second)
+ for _, trace := range []string{"trace-a", "trace-b"} {
+ if err := conn.Exec(ctx, "INSERT INTO tasks_v2 VALUES (?, ?, 'reused', 'task', ?)", project, trace, at); err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.Exec(ctx, "INSERT INTO exceptions_v2 VALUES (?, ?, 'reused', 'task', ?)", project, trace, at); err != nil {
+ t.Fatal(err)
+ }
+ }
+ for _, named := range []bool{true, false} {
+ got, err := countFailedTaskExecutions(ctx, project, "task", named, at.Add(-time.Hour), at.Add(time.Hour))
+ if err != nil || got != 2 {
+ t.Fatalf("named=%v: got %d, %v; want 2", named, got, err)
+ }
+ }
+ if err := conn.Exec(ctx, "TRUNCATE TABLE exceptions_v2"); err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.Exec(ctx, "TRUNCATE TABLE tasks_v2"); err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.Exec(ctx, "INSERT INTO tasks_v2 VALUES (?, 'trace-a', 'reused', 'task', ?), (?, 'trace-b', 'reused', 'other', ?)", project, at, project, at); err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.Exec(ctx, "INSERT INTO exceptions_v2 VALUES (?, 'trace-b', 'reused', 'task', ?)", project, at); err != nil {
+ t.Fatal(err)
+ }
+ got, err := countFailedTaskExecutions(ctx, project, "task", true, at.Add(-time.Hour), at.Add(time.Hour))
+ if err != nil || got != 0 {
+ t.Fatalf("successful task inherited another trace's failure: %d, %v", got, err)
+ }
+}
diff --git a/backend/app/notifications/task_failure_identity_test.go b/backend/app/notifications/task_failure_identity_test.go
new file mode 100644
index 00000000..0c92d1e6
--- /dev/null
+++ b/backend/app/notifications/task_failure_identity_test.go
@@ -0,0 +1,59 @@
+//go:build !telemetry_ch
+
+package notifications
+
+import (
+ "context"
+ "database/sql"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+)
+
+func TestRegressionTaskFailuresKeepTraceIdentity(t *testing.T) {
+ c, err := sql.Open("sqlite", ":memory:")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer c.Close()
+ previous := db.TelemetryDB
+ db.TelemetryDB = c
+ defer func() { db.TelemetryDB = previous }()
+ for _, q := range []string{"CREATE TABLE tasks_v2 (project_id TEXT,trace_id TEXT,span_id TEXT,task_name TEXT,recorded_at TEXT)", "CREATE TABLE exceptions_v2 (project_id TEXT,trace_id TEXT,span_id TEXT,trace_type TEXT,recorded_at TEXT)"} {
+ if _, err := c.Exec(q); err != nil {
+ t.Fatal(err)
+ }
+ }
+ project := uuid.New()
+ at := time.Now().UTC().Truncate(time.Second)
+ for _, trace := range []string{"trace-a", "trace-b"} {
+ if _, err := c.Exec("INSERT INTO tasks_v2 VALUES (?,?,?,?,?)", project.String(), trace, "reused-span", "task", at.Format(time.RFC3339Nano)); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := c.Exec("INSERT INTO exceptions_v2 VALUES (?,?,?,?,?)", project.String(), trace, "reused-span", "task", at.Format(time.RFC3339Nano)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ got, err := countFailedTaskExecutions(context.Background(), project, "task", true, at.Add(-time.Hour), at.Add(time.Hour))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got != 2 {
+ t.Errorf("two failed tasks in different traces counted as %d; want 2", got)
+ }
+ if _, err := c.Exec("DELETE FROM exceptions_v2 WHERE trace_id = 'trace-a'"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := c.Exec("UPDATE tasks_v2 SET task_name='other' WHERE trace_id = 'trace-b'"); err != nil {
+ t.Fatal(err)
+ }
+ got, err = countFailedTaskExecutions(context.Background(), project, "task", true, at.Add(-time.Hour), at.Add(time.Hour))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got != 0 {
+ t.Errorf("successful named task inherits another trace's exception: got %d; want 0", got)
+ }
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/ai_trace.repository.go b/backend/app/repositories/telemetry/clickhouse/ai_trace.repository.go
index 7da22726..a9238078 100644
--- a/backend/app/repositories/telemetry/clickhouse/ai_trace.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/ai_trace.repository.go
@@ -20,7 +20,7 @@ type aiTraceRepository struct{}
func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTrace) error {
batch, err := chdb.Conn.PrepareBatch(chdb.BatchCtx(),
- "INSERT INTO ai_traces (id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes, distributed_trace_id, is_root, conversation_id, tool_call_count, tool_names, flagged, flagged_terms)")
+ "INSERT INTO ai_traces_v2 (id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes, trace_id, span_id, parent_span_id, is_root, conversation_id, tool_call_count, tool_names, flagged, flagged_terms)")
if err != nil {
return err
}
@@ -45,7 +45,7 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTr
t.InputTokens, t.OutputTokens, t.TotalTokens, t.CachedTokens, t.ReasoningTokens,
t.InputCost, t.OutputCost, t.TotalCost,
t.TraceName, t.UserId, t.FinishReason, t.ServerName, t.AppVersion,
- t.StorageKey, attributesJSON, t.DistributedTraceId, isRoot,
+ t.StorageKey, attributesJSON, t.TraceId, t.SpanId, t.ParentSpanId, isRoot,
t.ConversationId, t.ToolCallCount, shared.JoinCSV(t.ToolNames), flagged, shared.JoinCSV(t.FlaggedTerms),
); err != nil {
return err
@@ -70,7 +70,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
}
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT uniq(trace_name) FROM ai_traces WHERE "+whereClause, args...).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT uniq(trace_name) FROM ai_traces_v2 WHERE "+whereClause, args...).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -109,7 +109,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
max(recorded_at) as last_seen,
max(is_root) as has_root,
max(if(is_root = 0, 1, 0)) as has_non_root
- FROM ai_traces
+ FROM ai_traces_v2
WHERE ` + whereClause + `
GROUP BY trace_name
ORDER BY ` + orderExpr + ` ` + sortDir + `
@@ -144,7 +144,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.UUID, traceName string, fromDate, toDate time.Time, page, pageSize int, orderBy, sortDirection string) ([]models.AiTrace, int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM ai_traces WHERE project_id = ? AND trace_name = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, traceName, fromDate, toDate).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM ai_traces_v2 WHERE project_id = ? AND trace_name = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, traceName, fromDate, toDate).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -173,9 +173,9 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = ? AND trace_name = ? AND recorded_at >= ? AND recorded_at <= ?
ORDER BY ` + orderBy + ` ` + sortDir + `
LIMIT ? OFFSET ?`
@@ -210,7 +210,7 @@ func scanAiTrace(scan func(dest ...any) error) (models.AiTrace, error) {
&t.InputTokens, &t.OutputTokens, &t.TotalTokens, &t.CachedTokens, &t.ReasoningTokens,
&t.InputCost, &t.OutputCost, &t.TotalCost,
&t.TraceName, &t.UserId, &t.FinishReason, &t.ServerName, &t.AppVersion,
- &t.StorageKey, &attributesJSON, &t.DistributedTraceId, &isRoot,
+ &t.StorageKey, &attributesJSON, &t.TraceId, &t.SpanId, &t.ParentSpanId, &isRoot,
&t.ConversationId, &t.ToolCallCount, &toolNames, &flagged, &flaggedTerms,
); err != nil {
return t, err
@@ -242,7 +242,7 @@ func (r *aiTraceRepository) GetTraceNameStats(ctx context.Context, projectId uui
sum(total_cost) as total_cost,
avg(input_tokens) as avg_input_tokens,
avg(output_tokens) as avg_output_tokens
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = ? AND trace_name = ? AND recorded_at >= ? AND recorded_at <= ?`
var stats models.AiTraceDetailStats
@@ -274,9 +274,9 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = ? AND id = ?`
args := []any{projectId, traceId}
if recordedAt != nil {
@@ -298,26 +298,23 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
return &t, nil
}
-func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.AiTrace, error) {
- if len(projectIds) == 0 {
+func (r *aiTraceRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.AiTrace, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- placeholders := make([]string, len(projectIds))
- args := []interface{}{distributedTraceId}
- for i, pid := range projectIds {
- placeholders[i] = "?"
- args = append(args, pid)
- _ = i
+ if len(projectIds) == 0 || len(traceIds) == 0 {
+ return nil, nil
}
+ args := []any{traceIds, projectIds}
query := `SELECT id, project_id, recorded_at, duration, status_code,
model, response_model, provider, operation,
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
- WHERE distributed_trace_id = ? AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ FROM ai_traces_v2
+ WHERE trace_id IN (?) AND project_id IN (?)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= ? AND recorded_at <= ?`
@@ -346,7 +343,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
// WHERE columns are table-qualified: the select list aliases aggregates to
// the same names (max(user_id) AS user_id), and ClickHouse resolves bare
// names in WHERE against those aliases.
- whereClause := "ai_traces.project_id = ? AND ai_traces.recorded_at >= ? AND ai_traces.recorded_at <= ? AND ai_traces.conversation_id != ''"
+ whereClause := "ai_traces_v2.project_id = ? AND ai_traces_v2.recorded_at >= ? AND ai_traces_v2.recorded_at <= ? AND ai_traces_v2.conversation_id != ''"
args := []interface{}{projectId, fromDate, toDate}
// Row-level filters use a semi-join on conversation_id: a conversation
@@ -374,7 +371,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
rowArgs = append(rowArgs, toolName)
}
if len(rowPredicates) > 0 {
- whereClause += " AND ai_traces.conversation_id IN (SELECT DISTINCT conversation_id FROM ai_traces WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND conversation_id != '' AND " +
+ whereClause += " AND ai_traces_v2.conversation_id IN (SELECT DISTINCT conversation_id FROM ai_traces_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND conversation_id != '' AND " +
strings.Join(rowPredicates, " AND ") + ")"
args = append(args, projectId, fromDate, toDate)
args = append(args, rowArgs...)
@@ -400,7 +397,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
arrayStringConcat(groupUniqArray(flagged_terms), ',') AS flagged_terms,
min(recorded_at) AS first_seen,
max(recorded_at) AS last_seen
- FROM ai_traces
+ FROM ai_traces_v2
WHERE ` + whereClause + `
GROUP BY conversation_id` + havingClause
@@ -475,9 +472,9 @@ func (r *aiTraceRepository) FindByConversationId(ctx context.Context, projectId
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = ? AND conversation_id = ?`
args := []interface{}{projectId, conversationId}
if !fromDate.IsZero() {
@@ -517,7 +514,7 @@ func (r *aiTraceRepository) FindUserStats(ctx context.Context, projectId uuid.UU
var count uint64
if err := chdb.Conn.QueryRow(ctx,
- "SELECT uniq(user_id) FROM ai_traces WHERE "+whereClause, args...).Scan(&count); err != nil {
+ "SELECT uniq(user_id) FROM ai_traces_v2 WHERE "+whereClause, args...).Scan(&count); err != nil {
return nil, 0, err
}
@@ -562,7 +559,7 @@ func (r *aiTraceRepository) FindUserStats(ctx context.Context, projectId uuid.UU
sum(total_tokens) AS conv_tokens,
max(flagged) AS conv_flagged,
max(recorded_at) AS conv_last_seen
- FROM ai_traces
+ FROM ai_traces_v2
WHERE ` + whereClause + `
GROUP BY user_id, conversation_id
)
@@ -612,7 +609,7 @@ func (r *aiTraceRepository) GetConversationCosts(ctx context.Context, projectId
rows, err := chdb.Conn.Query(ctx,
`SELECT conversation_id, sum(total_cost)
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = ? AND recorded_at >= ? AND conversation_id IN (`+strings.Join(placeholders, ",")+`)
GROUP BY conversation_id`, args...)
if err != nil {
@@ -633,7 +630,7 @@ func (r *aiTraceRepository) GetConversationCosts(ctx context.Context, projectId
func (r *aiTraceRepository) ListModels(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time) ([]string, error) {
rows, err := chdb.Conn.Query(ctx,
- `SELECT DISTINCT model FROM ai_traces
+ `SELECT DISTINCT model FROM ai_traces_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND model != ''
ORDER BY model LIMIT 200`, projectId, fromDate, toDate)
if err != nil {
@@ -654,7 +651,7 @@ func (r *aiTraceRepository) ListModels(ctx context.Context, projectId uuid.UUID,
func (r *aiTraceRepository) ListToolNames(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time) ([]string, error) {
rows, err := chdb.Conn.Query(ctx,
- `SELECT DISTINCT tool_names FROM ai_traces
+ `SELECT DISTINCT tool_names FROM ai_traces_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND tool_names != ''
LIMIT 500`, projectId, fromDate, toDate)
if err != nil {
diff --git a/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go b/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go
index 004b56fb..f068219f 100644
--- a/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/endpoint.repository.go
@@ -23,7 +23,7 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
if len(lines) == 0 {
return nil
}
- return chdb.SendBatch("INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream, is_root)", func(batch driver.Batch) error {
+ return chdb.SendBatch("INSERT INTO endpoints_v2 (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_stream, is_root)", func(batch driver.Batch) error {
for _, t := range lines {
attributesJSON := "{}"
if len(t.Attributes) != 0 {
@@ -39,7 +39,7 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
if t.IsRoot {
isRoot = 1
}
- if err := batch.Append(t.Id, t.ProjectId, t.Endpoint, int64(t.Duration), t.RecordedAt, t.StatusCode, t.BodySize, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isStream, isRoot); err != nil {
+ if err := batch.Append(t.Id, t.ProjectId, t.Endpoint, int64(t.Duration), t.RecordedAt, t.StatusCode, t.BodySize, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.TraceId, t.SpanId, t.ParentSpanId, isStream, isRoot); err != nil {
return err
}
}
@@ -49,13 +49,13 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
func (e *endpointRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, start, end).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, start, end).Scan(&count)
return int64(count), err
}
func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string) ([]models.Endpoint, int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, fromDate, toDate).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, fromDate, toDate).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -73,7 +73,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
orderBy = "recorded_at"
}
- query := "SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id FROM endpoints WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " DESC LIMIT ? OFFSET ?"
+ query := "SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id FROM endpoints_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " DESC LIMIT ? OFFSET ?"
rows, err := chdb.Conn.Query(ctx, query, projectId, fromDate, toDate, pageSize, offset)
if err != nil {
return nil, 0, err
@@ -84,7 +84,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
for rows.Next() {
var t models.Endpoint
var attributesJSON string
- if err := rows.Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt, &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId); err != nil {
+ if err := rows.Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt, &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId); err != nil {
return nil, 0, err
}
if attributesJSON != "" && attributesJSON != "{}" {
@@ -129,7 +129,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
// Count unique endpoints
var count uint64
- countQuery := "SELECT uniq(endpoint) FROM endpoints WHERE " + whereClause
+ countQuery := "SELECT uniq(endpoint) FROM endpoints_v2 WHERE " + whereClause
err := chdb.Conn.QueryRow(ctx, countQuery, args...).Scan(&count)
if err != nil {
return nil, 0, err
@@ -242,7 +242,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
FROM (
SELECT e.endpoint, e.duration, e.status_code, e.recorded_at, e.is_stream, e.is_root,
s.offset_ms as offset_ms
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN (SELECT * FROM slow_endpoints FINAL) AS s
ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE ` + joinWhereClause + `
@@ -297,7 +297,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.UUID, endpoint string, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string) ([]models.Endpoint, int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM endpoints WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, endpoint, fromDate, toDate).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM endpoints_v2 WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, endpoint, fromDate, toDate).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -321,7 +321,7 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
sortDir = "ASC"
}
- query := "SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id FROM endpoints WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " " + sortDir + " LIMIT ? OFFSET ?"
+ query := "SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id FROM endpoints_v2 WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " " + sortDir + " LIMIT ? OFFSET ?"
rows, err := chdb.Conn.Query(ctx, query, projectId, endpoint, fromDate, toDate, pageSize, offset)
if err != nil {
return nil, 0, err
@@ -332,7 +332,7 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
for rows.Next() {
var t models.Endpoint
var attributesJSON string
- if err := rows.Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt, &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId); err != nil {
+ if err := rows.Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt, &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId); err != nil {
return nil, 0, err
}
if attributesJSON != "" && attributesJSON != "{}" {
@@ -348,8 +348,8 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
// FindById returns a single endpoint by ID
func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId uuid.UUID, recordedAt *time.Time) (*models.Endpoint, error) {
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
- FROM endpoints
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM endpoints_v2
WHERE project_id = ? AND id = ?`
args := []any{projectId, endpointId}
if recordedAt != nil {
@@ -365,7 +365,7 @@ func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId
err := chdb.Conn.QueryRow(ctx, query, args...).Scan(
&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt,
- &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId, &isRoot)
+ &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId, &isRoot)
t.IsRoot = isRoot == 1
if err != nil {
@@ -389,7 +389,7 @@ func (e *endpointRepository) CountByHour(ctx context.Context, projectId uuid.UUI
query := `SELECT
toStartOfHour(recorded_at) as hour,
toFloat64(count()) as count
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY hour
ORDER BY hour ASC`
@@ -417,7 +417,7 @@ func (e *endpointRepository) AvgDurationByHour(ctx context.Context, projectId uu
query := `SELECT
toStartOfHour(recorded_at) as hour,
avg(duration) / 1000000 as avg_duration_ms
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0
GROUP BY hour
ORDER BY hour ASC`
@@ -445,7 +445,7 @@ func (e *endpointRepository) ErrorRateByHour(ctx context.Context, projectId uuid
query := `SELECT
toStartOfHour(recorded_at) as hour,
countIf(status_code >= 500) * 100.0 / count() as error_rate
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY hour
ORDER BY hour ASC`
@@ -473,7 +473,7 @@ func (e *endpointRepository) CountByInterval(ctx context.Context, projectId uuid
query := `SELECT
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
toFloat64(count()) as count
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY bucket
ORDER BY bucket ASC`
@@ -501,7 +501,7 @@ func (e *endpointRepository) AvgDurationByInterval(ctx context.Context, projectI
query := `SELECT
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
avg(duration) / 1000000 as avg_duration_ms
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0
GROUP BY bucket
ORDER BY bucket ASC`
@@ -529,7 +529,7 @@ func (e *endpointRepository) ErrorRateByInterval(ctx context.Context, projectId
query := `SELECT
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
countIf(status_code >= 500) * 100.0 / count() as error_rate
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY bucket
ORDER BY bucket ASC`
@@ -605,7 +605,7 @@ func (e *endpointRepository) FindWorstEndpoints(ctx context.Context, projectId u
FROM (
SELECT e.endpoint, e.duration, e.status_code, e.recorded_at,
s.offset_ms as offset_ms
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN (SELECT * FROM slow_endpoints FINAL) AS s
ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE e.project_id = ? AND e.recorded_at >= ? AND e.recorded_at <= ? AND e.is_stream = 0
@@ -664,7 +664,7 @@ func (e *endpointRepository) GetEndpointStats(ctx context.Context, projectId uui
0) as satisfied_tolerating,
max(is_stream) as has_stream,
countIf(is_stream = 0) as non_stream_count
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND endpoint = ? AND recorded_at >= ? AND recorded_at <= ?`
var stats models.EndpointDetailStats
@@ -726,7 +726,7 @@ func (e *endpointRepository) GetEndpointStackedChart(ctx context.Context, projec
// Step 1: Get top 5 endpoints ranked by selected metric (streams excluded)
rankQuery := `SELECT endpoint, ` + metricExpr + ` as metric_value
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0` + filterClause + `
GROUP BY endpoint
ORDER BY metric_value DESC
@@ -768,7 +768,7 @@ func (e *endpointRepository) GetEndpointStackedChart(ctx context.Context, projec
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
` + caseExpr + ` as endpoint_category,
` + metricExpr + ` as metric_value
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND is_stream = 0` + filterClause + `
GROUP BY bucket, endpoint_category
ORDER BY bucket ASC, endpoint_category ASC`
@@ -836,11 +836,14 @@ func (e *endpointRepository) UpsertSlowEndpoint(ctx context.Context, projectId u
return batch.Send()
}
-func (e *endpointRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Endpoint, error) {
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints
- WHERE distributed_trace_id = ? AND project_id IN (?)`
- args := []any{distributedTraceId, projectIds}
+func (e *endpointRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Endpoint, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
+ return nil, nil
+ }
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM endpoints_v2
+ WHERE trace_id IN (?) AND project_id IN (?)`
+ args := []any{traceIds, projectIds}
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= ? AND recorded_at <= ?`
@@ -859,7 +862,7 @@ func (e *endpointRepository) FindByDistributedTraceId(ctx context.Context, distr
var t models.Endpoint
var attributesJSON string
if err := rows.Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt,
- &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId); err != nil {
+ &t.StatusCode, &t.BodySize, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId, &t.IsRoot); err != nil {
return nil, err
}
if attributesJSON != "" && attributesJSON != "{}" {
diff --git a/backend/app/repositories/telemetry/clickhouse/exception_stack_trace.repository.go b/backend/app/repositories/telemetry/clickhouse/exception_stack_trace.repository.go
index 1bb46cf1..e35aa70f 100644
--- a/backend/app/repositories/telemetry/clickhouse/exception_stack_trace.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/exception_stack_trace.repository.go
@@ -23,7 +23,7 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
if len(lines) == 0 {
return nil
}
- return chdb.SendBatch("INSERT INTO exception_stack_traces (id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id)", func(batch driver.Batch) error {
+ return chdb.SendBatch("INSERT INTO exceptions_v2 (id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id)", func(batch driver.Batch) error {
for _, est := range lines {
attributesJSON := "{}"
if len(est.Attributes) != 0 {
@@ -35,11 +35,7 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
if est.IsMessage {
isMessage = 1
}
- traceType := est.TraceType
- if traceType == "" {
- traceType = "endpoint"
- }
- if err := batch.Append(est.Id, est.ProjectId, est.TraceId, traceType, est.ExceptionHash, est.StackTrace, est.RecordedAt, attributesJSON, est.AppVersion, est.ServerName, isMessage, est.DistributedTraceId, est.SessionId); err != nil {
+ if err := batch.Append(est.Id, est.ProjectId, est.TraceId, est.SpanId, est.TraceType, est.ExceptionHash, est.StackTrace, est.RecordedAt, attributesJSON, est.AppVersion, est.ServerName, isMessage, est.SessionId); err != nil {
return err
}
}
@@ -50,8 +46,8 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
// FindAllBySessionId returns all exceptions/messages stamped with the given session_id.
func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context, projectId, sessionId uuid.UUID) ([]models.ExceptionStackTrace, error) {
rows, err := chdb.Conn.Query(ctx,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2
WHERE project_id = ? AND session_id = ?
ORDER BY recorded_at ASC`,
projectId, sessionId)
@@ -65,8 +61,8 @@ func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context,
var est models.ExceptionStackTrace
var attributesJSON string
var isMessage uint8
- if err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
- &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage, &est.DistributedTraceId, &est.SessionId); err != nil {
+ if err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.SpanId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
+ &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage, &est.SessionId); err != nil {
return nil, err
}
est.IsMessage = isMessage == 1
@@ -84,7 +80,7 @@ func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context,
func (e *exceptionStackTraceRepository) GetSessionIdForException(ctx context.Context, projectId, exceptionId uuid.UUID) (*uuid.UUID, error) {
var sessionId *uuid.UUID
err := chdb.Conn.QueryRow(ctx,
- "SELECT session_id FROM exception_stack_traces WHERE project_id = ? AND id = ? LIMIT 1",
+ "SELECT session_id FROM exceptions_v2 WHERE project_id = ? AND id = ? LIMIT 1",
projectId, exceptionId).Scan(&sessionId)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
@@ -97,7 +93,7 @@ func (e *exceptionStackTraceRepository) GetSessionIdForException(ctx context.Con
func (e *exceptionStackTraceRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM exception_stack_traces WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, start, end).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM exceptions_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, start, end).Scan(&count)
return int64(count), err
}
@@ -155,7 +151,7 @@ func (e *exceptionStackTraceRepository) FindGrouped(ctx context.Context, project
// Count query needs to wrap the grouped query to apply HAVING filter correctly
countQuery := `SELECT count() FROM (
SELECT e.exception_hash
- FROM exception_stack_traces e
+ FROM exceptions_v2 e
` + archiveSubquery + `
WHERE ` + whereClause + `
GROUP BY e.exception_hash` + havingClause + `
@@ -170,7 +166,7 @@ func (e *exceptionStackTraceRepository) FindGrouped(ctx context.Context, project
// Main query with archive-aware filtering
fullQuery := `SELECT e.exception_hash, any(e.stack_trace), max(e.recorded_at) as last_seen, min(e.recorded_at) as first_seen, count() as count
- FROM exception_stack_traces e
+ FROM exceptions_v2 e
` + archiveSubquery + `
WHERE ` + whereClause + `
GROUP BY e.exception_hash` + havingClause + `
@@ -202,7 +198,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
// Get grouped info
var group models.ExceptionGroup
err := chdb.Conn.QueryRow(ctx,
- "SELECT exception_hash, any(stack_trace), max(recorded_at) as last_seen, min(recorded_at) as first_seen, count() as count FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ? GROUP BY exception_hash",
+ "SELECT exception_hash, any(stack_trace), max(recorded_at) as last_seen, min(recorded_at) as first_seen, count() as count FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ? GROUP BY exception_hash",
projectId, exceptionHash).Scan(&group.ExceptionHash, &group.StackTrace, &group.LastSeen, &group.FirstSeen, &group.Count)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
@@ -212,7 +208,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
}
rows, err := chdb.Conn.Query(ctx,
- "SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id FROM exception_stack_traces WHERE project_id = ? AND exception_hash = ? ORDER BY recorded_at DESC LIMIT ? OFFSET ?",
+ "SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message FROM exceptions_v2 WHERE project_id = ? AND exception_hash = ? ORDER BY recorded_at DESC LIMIT ? OFFSET ?",
projectId, exceptionHash, pageSize, offset)
if err != nil {
return nil, nil, 0, err
@@ -224,7 +220,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
var o models.ExceptionStackTrace
var attributesJSON string
var isMessage uint8
- if err := rows.Scan(&o.Id, &o.ProjectId, &o.TraceId, &o.TraceType, &o.ExceptionHash, &o.StackTrace, &o.RecordedAt, &attributesJSON, &o.AppVersion, &o.ServerName, &isMessage, &o.DistributedTraceId); err != nil {
+ if err := rows.Scan(&o.Id, &o.ProjectId, &o.TraceId, &o.SpanId, &o.TraceType, &o.ExceptionHash, &o.StackTrace, &o.RecordedAt, &attributesJSON, &o.AppVersion, &o.ServerName, &isMessage); err != nil {
return nil, nil, 0, err
}
o.IsMessage = isMessage == 1
@@ -244,7 +240,7 @@ func (e *exceptionStackTraceRepository) CountByHour(ctx context.Context, project
query := `SELECT
toStartOfHour(recorded_at) as hour,
toFloat64(count()) as count
- FROM exception_stack_traces
+ FROM exceptions_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY hour
ORDER BY hour ASC`
@@ -272,7 +268,7 @@ func (e *exceptionStackTraceRepository) CountByInterval(ctx context.Context, pro
query := `SELECT
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
toFloat64(count()) as count
- FROM exception_stack_traces
+ FROM exceptions_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY bucket
ORDER BY bucket ASC`
@@ -305,7 +301,7 @@ func (e *exceptionStackTraceRepository) GetHourlyTrendForHashes(ctx context.Cont
exception_hash,
toStartOfHour(recorded_at) as hour,
count() as count
- FROM exception_stack_traces
+ FROM exceptions_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND exception_hash IN (?)
GROUP BY exception_hash, hour
ORDER BY exception_hash, hour ASC`
@@ -373,39 +369,10 @@ func (e *exceptionStackTraceRepository) IsArchived(ctx context.Context, projectI
return count > 0, nil
}
-func (e *exceptionStackTraceRepository) FindExceptionByTraceId(ctx context.Context, projectId uuid.UUID, traceId uuid.UUID) (*models.ExceptionStackTrace, error) {
- var est models.ExceptionStackTrace
- var attributesJSON string
- var isMessage uint8
-
- err := chdb.Conn.QueryRow(ctx,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id
- FROM exception_stack_traces
- WHERE project_id = ? AND trace_id = ? AND is_message = false
- LIMIT 1`,
- projectId, traceId).Scan(
- &est.Id, &est.ProjectId, &est.TraceId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
- &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage, &est.DistributedTraceId)
-
- if err != nil {
- // No exception found for this trace
- return nil, nil
- }
-
- est.IsMessage = isMessage == 1
- if attributesJSON != "" && attributesJSON != "{}" {
- if err := json.Unmarshal([]byte(attributesJSON), &est.Attributes); err != nil {
- est.Attributes = nil
- }
- }
-
- return &est, nil
-}
-
// FindAllByTraceId returns all exceptions and messages associated with a specific trace
-func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, projectId uuid.UUID, traceId uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id
- FROM exception_stack_traces
+func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, projectId uuid.UUID, traceId string, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message
+ FROM exceptions_v2
WHERE project_id = ? AND trace_id = ?`
args := []any{projectId, traceId}
if recordedAt != nil {
@@ -428,8 +395,8 @@ func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, pr
var attributesJSON string
var isMessage uint8
- if err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
- &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage, &est.DistributedTraceId); err != nil {
+ if err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.SpanId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
+ &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage); err != nil {
return nil, err
}
@@ -451,8 +418,8 @@ func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId
var attributesJSON string
var isMessage uint8
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id
- FROM exception_stack_traces
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message
+ FROM exceptions_v2
WHERE project_id = ? AND id = ?`
args := []any{projectId, id}
if recordedAt != nil {
@@ -463,8 +430,8 @@ func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId
query += ` LIMIT 1`
err := chdb.Conn.QueryRow(ctx, query, args...).Scan(
- &est.Id, &est.ProjectId, &est.TraceId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
- &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage, &est.DistributedTraceId)
+ &est.Id, &est.ProjectId, &est.TraceId, &est.SpanId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
+ &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
@@ -483,11 +450,14 @@ func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId
return &est, nil
}
-func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id
- FROM exception_stack_traces
- WHERE distributed_trace_id = ? AND project_id IN (?) AND is_message = false`
- args := []any{distributedTraceId, projectIds}
+func (e *exceptionStackTraceRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
+ return nil, nil
+ }
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message
+ FROM exceptions_v2
+ WHERE trace_id IN (?) AND project_id IN (?) AND is_message = false`
+ args := []any{traceIds, projectIds}
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= ? AND recorded_at <= ?`
@@ -506,8 +476,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
var est models.ExceptionStackTrace
var attributesJSON string
var isMessage uint8
- if err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
- &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage, &est.DistributedTraceId); err != nil {
+ if err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.SpanId, &est.TraceType, &est.ExceptionHash, &est.StackTrace,
+ &est.RecordedAt, &attributesJSON, &est.AppVersion, &est.ServerName, &isMessage); err != nil {
return nil, err
}
est.IsMessage = isMessage == 1
diff --git a/backend/app/repositories/telemetry/clickhouse/legacy.repository.go b/backend/app/repositories/telemetry/clickhouse/legacy.repository.go
new file mode 100644
index 00000000..e0f102ed
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/legacy.repository.go
@@ -0,0 +1,223 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "context"
+ "encoding/json"
+ "slices"
+ "time"
+
+ "github.com/ClickHouse/clickhouse-go/v2/lib/driver"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+// legacyRepository reads the tables V2 replaced, for the move-over. See shared/legacy.go for what the rows carry.
+type legacyRepository struct{}
+
+const legacyIdChunk = 1000
+
+const (
+ legacyEndpointColumns = `id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name,
+ ifNull(toString(distributed_trace_id), ''), ifNull(toString(span_id), ''), is_stream, is_root`
+ legacyTaskColumns = `id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name,
+ ifNull(toString(distributed_trace_id), ''), ifNull(toString(span_id), ''), is_root`
+ legacyAiTraceColumns = `id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens,
+ total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version,
+ storage_key, attributes, ifNull(toString(distributed_trace_id), ''), '', '', is_root, conversation_id, tool_call_count, tool_names, flagged, flagged_terms`
+)
+
+func legacyAttributes(raw string) map[string]string {
+ var attributes map[string]string
+ if raw != "" && raw != "{}" && json.Unmarshal([]byte(raw), &attributes) != nil {
+ return nil
+ }
+ return attributes
+}
+
+// legacyWindow sorts only overfull one-second slices; see shared.LegacyPageOrder.
+func legacyWindow(columns, table string, from, to time.Time, limit int, offset *int) (string, []any) {
+ if offset != nil {
+ return "SELECT " + columns + " FROM " + table + " WHERE recorded_at >= ? AND recorded_at < ? ORDER BY " + shared.LegacyPageOrder(table, true) + " LIMIT ? OFFSET ?", []any{from, to, limit, *offset}
+ }
+ return "SELECT " + columns + " FROM " + table + " WHERE recorded_at >= ? AND recorded_at < ? LIMIT ?", []any{from, to, limit}
+}
+
+func legacyByIds(columns, table string, ids []uuid.UUID, from, to time.Time) (string, []any) {
+ return "SELECT " + columns + " FROM " + table + " WHERE id IN (?) AND recorded_at >= ? AND recorded_at < ?", []any{ids, from, to}
+}
+
+func legacyRows[T any](ctx context.Context, query string, args []any, scan func(driver.Rows) (T, error)) ([]T, error) {
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var found []T
+ for rows.Next() {
+ row, err := scan(rows)
+ if err != nil {
+ return nil, err
+ }
+ found = append(found, row)
+ }
+ return found, rows.Err()
+}
+
+func scanLegacyEndpoint(rows driver.Rows) (models.Endpoint, error) {
+ var t models.Endpoint
+ var attributes string
+ var isStream, isRoot uint8
+ err := rows.Scan(&t.Id, &t.ProjectId, &t.Endpoint, &t.Duration, &t.RecordedAt, &t.StatusCode, &t.BodySize, &t.ClientIP, &attributes, &t.AppVersion,
+ &t.ServerName, &t.TraceId, &t.SpanId, &isStream, &isRoot)
+ t.Attributes, t.IsStream, t.IsRoot = legacyAttributes(attributes), isStream == 1, isRoot == 1
+ return t, err
+}
+
+func scanLegacyTask(rows driver.Rows) (models.Task, error) {
+ var t models.Task
+ var attributes string
+ var isRoot uint8
+ err := rows.Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt, &t.ClientIP, &attributes, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &isRoot)
+ t.Attributes, t.IsRoot = legacyAttributes(attributes), isRoot == 1
+ return t, err
+}
+
+func scanLegacyAiTrace(rows driver.Rows) (models.AiTrace, error) {
+ return scanAiTrace(rows.Scan)
+}
+
+func (legacyRepository) FindEndpoints(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Endpoint, error) {
+ query, args := legacyWindow(legacyEndpointColumns, "endpoints", from, to, limit, offset)
+ return legacyRows(ctx, query, args, scanLegacyEndpoint)
+}
+
+func (legacyRepository) FindTasks(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Task, error) {
+ query, args := legacyWindow(legacyTaskColumns, "tasks", from, to, limit, offset)
+ return legacyRows(ctx, query, args, scanLegacyTask)
+}
+
+func (legacyRepository) FindAiTraces(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.AiTrace, error) {
+ query, args := legacyWindow(legacyAiTraceColumns, "ai_traces", from, to, limit, offset)
+ return legacyRows(ctx, query, args, scanLegacyAiTrace)
+}
+
+func (legacyRepository) FindExceptions(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacyException, error) {
+ query, args := legacyWindow(`id, project_id, ifNull(toString(trace_id), ''), trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version,
+ server_name, is_message, ifNull(toString(distributed_trace_id), ''), session_id`, "exception_stack_traces", from, to, limit, offset)
+ return legacyRows(ctx, query, args, func(rows driver.Rows) (shared.LegacyException, error) {
+ var est shared.LegacyException
+ var attributes string
+ var isMessage uint8
+ err := rows.Scan(&est.Id, &est.ProjectId, &est.TraceId, &est.TraceType, &est.ExceptionHash, &est.StackTrace, &est.RecordedAt, &attributes,
+ &est.AppVersion, &est.ServerName, &isMessage, &est.DistributedTraceId, &est.SessionId)
+ est.Attributes, est.IsMessage = legacyAttributes(attributes), isMessage == 1
+ return est, err
+ })
+}
+
+func (legacyRepository) FindSpans(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacySpan, error) {
+ query, args := legacyWindow(`project_id, id, trace_id, ifNull(toString(parent_span_id), ''), name, start_time, duration, recorded_at, attributes`, "spans", from, to, limit, offset)
+ return legacyRows(ctx, query, args, func(rows driver.Rows) (shared.LegacySpan, error) {
+ var span shared.LegacySpan
+ var attributes string
+ var duration int64
+ err := rows.Scan(&span.ProjectId, &span.Id, &span.OwnerId, &span.ParentSpanId, &span.Name, &span.StartTime, &duration, &span.RecordedAt, &attributes)
+ span.Duration, span.Attributes = time.Duration(duration), legacyAttributes(attributes)
+ return span, err
+ })
+}
+
+// FindOwners looks the owners of spans and exceptions up by id, in all three tables an owner can live in.
+func (legacyRepository) FindOwners(ctx context.Context, ids []uuid.UUID, from, to time.Time) ([]models.Endpoint, []models.Task, []models.AiTrace, error) {
+ var endpoints []models.Endpoint
+ var tasks []models.Task
+ var aiTraces []models.AiTrace
+ for chunk := range slices.Chunk(ids, legacyIdChunk) {
+ query, args := legacyByIds(legacyEndpointColumns, "endpoints", chunk, from, to)
+ foundEndpoints, err := legacyRows(ctx, query, args, scanLegacyEndpoint)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ query, args = legacyByIds(legacyTaskColumns, "tasks", chunk, from, to)
+ foundTasks, err := legacyRows(ctx, query, args, scanLegacyTask)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ query, args = legacyByIds(legacyAiTraceColumns, "ai_traces", chunk, from, to)
+ foundAiTraces, err := legacyRows(ctx, query, args, scanLegacyAiTrace)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ endpoints, tasks, aiTraces = append(endpoints, foundEndpoints...), append(tasks, foundTasks...), append(aiTraces, foundAiTraces...)
+ }
+ return endpoints, tasks, aiTraces, nil
+}
+
+func (legacyRepository) FindBounds(ctx context.Context, table string) (oldest, newest time.Time, found bool, err error) {
+ if table, err = shared.LegacyTable(table); err != nil {
+ return oldest, newest, false, err
+ }
+ var count uint64
+ if err = chdb.Conn.QueryRow(ctx, "SELECT count(), toDateTime64(min(recorded_at), 3, 'UTC'), toDateTime64(max(recorded_at), 3, 'UTC') FROM "+table).Scan(&count, &oldest, &newest); err != nil {
+ return oldest, newest, false, err
+ }
+ return oldest, newest, count > 0, nil
+}
+
+// FindMovedIds returns full occurrence keys, never IDs shared by unrelated traces.
+func (legacyRepository) FindMovedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time) (map[shared.MovedOccurrence]bool, error) {
+ table, err := shared.LegacyTable(table)
+ if err != nil {
+ return nil, err
+ }
+ moved := map[shared.MovedOccurrence]bool{}
+ for chunk := range slices.Chunk(ids, legacyIdChunk) {
+ found, err := legacyRows(ctx, "SELECT project_id, id, trace_id, span_id, recorded_at FROM "+table+" WHERE recorded_at >= ? AND recorded_at < ? AND id IN (?)", []any{from, to, chunk}, func(rows driver.Rows) (shared.MovedOccurrence, error) {
+ var project, id uuid.UUID
+ var trace, span string
+ var at time.Time
+ err := rows.Scan(&project, &id, &trace, &span, &at)
+ return shared.MovedOccurrenceKey(project, id, trace, span, at), err
+ })
+ if err != nil {
+ return nil, err
+ }
+ for _, id := range found {
+ moved[id] = true
+ }
+ }
+ return moved, nil
+}
+
+func (legacyRepository) FindMovedSpans(ctx context.Context, traceIds []string, from, to time.Time) (map[string]bool, error) {
+ moved := map[string]bool{}
+ for chunk := range slices.Chunk(traceIds, legacyIdChunk) {
+ found, err := legacyRows(ctx, "SELECT project_id, trace_id, span_id FROM "+shared.SpansTable+" WHERE recorded_at >= ? AND recorded_at < ? AND trace_id IN (?)", []any{from, to, chunk}, func(rows driver.Rows) (string, error) {
+ var project uuid.UUID
+ var trace, span string
+ err := rows.Scan(&project, &trace, &span)
+ return shared.MovedSpanKey(project, trace, span), err
+ })
+ if err != nil {
+ return nil, err
+ }
+ for _, key := range found {
+ moved[key] = true
+ }
+ }
+ return moved, nil
+}
+
+// OccurrenceTime matches the destination column precision for recovery keys.
+func (legacyRepository) OccurrenceTime(table string, at time.Time) time.Time {
+ if table == "ai_traces" {
+ return at.Truncate(time.Millisecond)
+ }
+ return at.Truncate(time.Microsecond)
+}
+
+var LegacyRepository = legacyRepository{}
diff --git a/backend/app/repositories/telemetry/clickhouse/log_record.repository.go b/backend/app/repositories/telemetry/clickhouse/log_record.repository.go
index 7f867a8f..2f51d84a 100644
--- a/backend/app/repositories/telemetry/clickhouse/log_record.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/log_record.repository.go
@@ -188,6 +188,16 @@ func (r *logRecordRepository) FindByTraceId(ctx context.Context, projectId uuid.
func (r *logRecordRepository) buildWhere(params shared.LogSearchParams) (string, []interface{}) {
clauses := []string{"project_id = ?", "timestamp >= ?", "timestamp <= ?"}
args := []interface{}{params.ProjectId, params.FromDate, params.ToDate}
+ if len(params.ProjectIds) > 0 {
+ placeholders := make([]string, len(params.ProjectIds))
+ args = make([]interface{}, 0, len(params.ProjectIds)+2)
+ for i, id := range params.ProjectIds {
+ placeholders[i] = "?"
+ args = append(args, id)
+ }
+ clauses[0] = "project_id IN (" + strings.Join(placeholders, ", ") + ")"
+ args = append(args, params.FromDate, params.ToDate)
+ }
if params.MinSeverity > 0 {
clauses = append(clauses, "severity_number >= ?")
diff --git a/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go b/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go
new file mode 100644
index 00000000..e0b3ff9e
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/otel_span.repository.go
@@ -0,0 +1,319 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "cmp"
+ "context"
+ "database/sql"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "slices"
+ "strings"
+ "time"
+
+ ch "github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/ClickHouse/clickhouse-go/v2/lib/driver"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+type otelSpanRepository struct{}
+
+const otelInsertColumns = `INSERT INTO ` + shared.SpansTable + ` (` + shared.OtelScalarColumns + `, span_attributes, resource, scope, events, links, resource_pb, scope_pb, span_pb)`
+
+var clickhouseOtelCodec = shared.OtelValueCodec{
+ UUID: func(id uuid.UUID) any { return id },
+ Time: func(t time.Time) any { return t.UTC() },
+ Nanos: func(n uint64) any { return n },
+ Uint32: func(n uint32) any { return n },
+}
+
+// Order must match the columns otelInsertColumns adds after shared.OtelScalarColumns.
+func otelNestedValues(row *shared.OtelSpanRow) ([]any, error) {
+ values, err := row.NestedValues()
+ if err != nil {
+ return nil, err
+ }
+ return append(values, string(row.Group.ResourcePB), string(row.Group.ScopePB), string(row.SpanPB)), nil
+}
+
+// ClickHouse rejects one insert that touches more than max_partitions_per_insert_block daily partitions (100 by default).
+const (
+ clickhouseMaxPartitionsPerInsert = 100
+ clickhouseMaxPartitionedInserts = 8
+)
+
+type clickhouseOtelRow struct {
+ owner shared.SpanOwner
+ values []any
+ day int64
+}
+
+func clickhouseOtelValues(span models.OtelSpan, groups shared.OtelSpanGroups) (clickhouseOtelRow, error) {
+ row, err := shared.NewOtelSpanRow(span, groups)
+ if err != nil {
+ return clickhouseOtelRow{}, err
+ }
+ values := row.ScalarValues(clickhouseOtelCodec)
+ nested, err := otelNestedValues(row)
+ if err != nil {
+ return clickhouseOtelRow{}, err
+ }
+ return clickhouseOtelRow{owner: shared.SpanOwner{ProjectId: span.ProjectId, TraceId: span.TraceId, SpanId: span.SpanId}, values: append(values, nested...), day: row.PartitionTime.Unix() / 86400}, nil
+}
+
+// partitionedInserts keeps each insert within the partition limit and bounds how many inserts one request may cause.
+func partitionedInserts(rows []clickhouseOtelRow) (inserts [][]clickhouseOtelRow, overflow []clickhouseOtelRow) {
+ days := make(map[int64]bool)
+ for _, row := range rows {
+ days[row.day] = true
+ }
+ if len(days) <= clickhouseMaxPartitionsPerInsert {
+ return [][]clickhouseOtelRow{rows}, nil
+ }
+ slices.SortStableFunc(rows, func(a, b clickhouseOtelRow) int { return cmp.Compare(a.day, b.day) })
+ start, distinct := 0, 0
+ for i, row := range rows {
+ if i > 0 && row.day == rows[i-1].day {
+ continue
+ }
+ if distinct == clickhouseMaxPartitionsPerInsert {
+ inserts = append(inserts, rows[start:i])
+ start, distinct = i, 0
+ if len(inserts) == clickhouseMaxPartitionedInserts {
+ return inserts, rows[i:]
+ }
+ }
+ distinct++
+ }
+ return append(inserts, rows[start:]), nil
+}
+
+func (r *otelSpanRepository) InsertAsync(ctx context.Context, spans []models.OtelSpan) (int, error) {
+ rejected, err := r.InsertWithRejections(ctx, spans)
+ return len(rejected), err
+}
+
+func (r *otelSpanRepository) InsertWithRejections(ctx context.Context, spans []models.OtelSpan) ([]shared.SpanOwner, error) {
+ var rejected []shared.SpanOwner
+ groups, rows := shared.OtelSpanGroups{}, make([]clickhouseOtelRow, 0, len(spans))
+ for _, span := range spans {
+ row, err := clickhouseOtelValues(span, groups)
+ if err != nil {
+ shared.RecordRejectedOtelSpan(err)
+ rejected = append(rejected, shared.SpanOwner{ProjectId: span.ProjectId, TraceId: span.TraceId, SpanId: span.SpanId})
+ continue
+ }
+ rows = append(rows, row)
+ }
+ if len(rows) == 0 {
+ return rejected, nil
+ }
+ inserts, overflow := partitionedInserts(rows)
+ for _, row := range overflow {
+ rejected = append(rejected, row.owner)
+ shared.RecordRejectedOtelSpan(fmt.Errorf("one request spans more than %d daily partitions", clickhouseMaxPartitionsPerInsert*clickhouseMaxPartitionedInserts))
+ }
+ for _, insert := range inserts {
+ err := chdb.SendBatch(otelInsertColumns, func(batch driver.Batch) error {
+ for _, row := range insert {
+ if err := batch.Append(row.values...); err != nil {
+ return err
+ }
+ }
+ return nil
+ })
+ if err != nil {
+ return nil, err
+ }
+ }
+ return rejected, nil
+}
+
+const otelReadSettings = ` SETTINGS max_execution_time=10, max_rows_to_read=1000000, max_bytes_to_read=268435456, max_memory_usage=268435456, read_overflow_mode='throw', timeout_overflow_mode='throw'`
+
+// One lookup uses equality so index analysis keys on project_id and trace_id with binary search. Several lookups
+// name both key sets for the same reason and keep the pair condition, which stops a project matching another's trace.
+func clickhouseTracePredicate(lookups []shared.SpanLookup) (string, []any) {
+ projects, traces := make([]uuid.UUID, 0, len(lookups)), make([]string, 0, len(lookups))
+ pairs := make([]any, 0, len(lookups)*2)
+ for _, lookup := range lookups {
+ if !slices.Contains(projects, lookup.ProjectId) {
+ projects = append(projects, lookup.ProjectId)
+ }
+ if !slices.Contains(traces, lookup.TraceId) {
+ traces = append(traces, lookup.TraceId)
+ }
+ pairs = append(pairs, lookup.ProjectId, lookup.TraceId)
+ }
+ if len(lookups) == 1 {
+ return "project_id = ? AND trace_id = ?", pairs
+ }
+ tuples := strings.TrimSuffix(strings.Repeat("(?, ?),", len(lookups)), ",")
+ return "project_id IN (?) AND trace_id IN (?) AND (project_id, trace_id) IN (" + tuples + ")", append([]any{projects, traces}, pairs...)
+}
+
+func (r *otelSpanRepository) FindTraceTopology(ctx context.Context, lookups []shared.SpanLookup, fromUnixNano uint64) ([]models.OtelSpan, error) {
+ if len(lookups) == 0 {
+ return nil, nil
+ }
+ predicate, args := clickhouseTracePredicate(lookups)
+ predicate, args = shared.OtelWindowPredicate(predicate, args, lookups, clickhouseOtelCodec.Time)
+ query := "SELECT " + shared.OtelTopologyColumns + " FROM " + shared.OtelWinningRows(shared.OtelTopologyColumns, predicate, shared.OtelClickHouseWinnerOrder)
+ if fromUnixNano > 0 {
+ query += " WHERE start_time_unix_nano >= ?"
+ args = append(args, fromUnixNano)
+ }
+ query += fmt.Sprintf(" ORDER BY start_time_unix_nano ASC, span_id ASC LIMIT %d", shared.MaxOtelGraphRows+1) + otelReadSettings
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return scanClickhouseTopology(rows, false)
+}
+
+const clickhouseStatementPreview = `substringUTF8(if(span_attributes['db.query.text'] != '', span_attributes['db.query.text'], span_attributes['db.statement']), 1, 240)`
+
+func (r *otelSpanRepository) FindTraceOutline(ctx context.Context, lookups []shared.SpanLookup) ([]models.OtelSpan, error) {
+ predicate, args := clickhouseTracePredicate(lookups)
+ predicate, args = shared.OtelWindowPredicate(predicate, args, lookups, clickhouseOtelCodec.Time)
+ query := "SELECT " + shared.OtelTopologyColumns + ", " + clickhouseStatementPreview + " FROM " + shared.OtelWinningRows(shared.OtelTopologyColumns+", span_attributes", predicate, shared.OtelClickHouseWinnerOrder) + shared.OtelImportanceOrder + fmt.Sprintf(" LIMIT %d", shared.MaxOtelGraphRows+1) + otelReadSettings
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return scanClickhouseTopology(rows, true)
+}
+
+func scanClickhouseTopology(rows driver.Rows, withStatement bool) ([]models.OtelSpan, error) {
+ result := make([]models.OtelSpan, 0)
+ for rows.Next() {
+ var span models.OtelSpan
+ var nanos uint64
+ columns := []any{&span.ProjectId, &span.TraceId, &span.SpanId, &span.ParentSpanId,
+ &span.Name, &span.Duration, &span.SpanKind, &span.StatusCode, &span.ServiceName, &span.ScopeName, &nanos, &span.RecordedAt}
+ if withStatement {
+ columns = append(columns, &span.DbStatement)
+ }
+ if err := rows.Scan(columns...); err != nil {
+ return nil, err
+ }
+ span.StartTime = shared.OtelNanosToTime(nanos)
+ result = append(result, span)
+ }
+ return result, rows.Err()
+}
+
+// A search reads every span of the project in the range, so it is held by time and memory, not by a row count.
+const otelSearchSettings = ` SETTINGS max_execution_time=15, timeout_overflow_mode='throw', max_memory_usage=1073741824`
+
+var clickhouseSearchDialect = shared.OtelSearchDialect{
+ WinnerOrder: shared.OtelClickHouseWinnerOrder,
+ Project: func(id uuid.UUID) any { return id },
+ Time: clickhouseOtelCodec.Time,
+ Trace: func(_ uuid.UUID, traceId string) (string, any) { return "trace_id = ?", traceId },
+ NameLike: "positionCaseInsensitive(name, ?) > 0",
+ Attribute: "span_attributes[?] = ?",
+ StartOrder: "start_time_unix_nano",
+ ReadSetting: otelSearchSettings,
+}
+
+func (r *otelSpanRepository) Search(ctx context.Context, search shared.OtelSpanSearch) ([]models.OtelSpan, uint64, error) {
+ page, pageArgs, count, countArgs := shared.OtelSearchQueries(search, clickhouseSearchDialect)
+ var total uint64
+ if err := chdb.Conn.QueryRow(ctx, count, countArgs...).Scan(&total); err != nil {
+ return nil, 0, err
+ }
+ rows, err := chdb.Conn.Query(ctx, page, pageArgs...)
+ if err != nil {
+ return nil, 0, err
+ }
+ defer rows.Close()
+ spans, err := scanClickhouseTopology(rows, false)
+ return spans, total, err
+}
+
+func (r *otelSpanRepository) Services(ctx context.Context, project uuid.UUID, from, to time.Time) ([]string, error) {
+ query, args := shared.OtelServicesQuery(project, from, to, clickhouseSearchDialect)
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ services := make([]string, 0)
+ for rows.Next() {
+ var service string
+ if err := rows.Scan(&service); err != nil {
+ return nil, err
+ }
+ services = append(services, service)
+ }
+ return services, rows.Err()
+}
+
+func (r *otelSpanRepository) FindSpanAttributes(ctx context.Context, lookup shared.SpanLookup, spanIds []string, limits shared.OtelAttributeLimits) (map[string]shared.OtelSpanAttributes, error) {
+ predicate, args := shared.OtelWindowPredicate("project_id = ? AND trace_id = ? AND span_id IN (?)",
+ []any{lookup.ProjectId, lookup.TraceId, spanIds}, []shared.SpanLookup{lookup}, clickhouseOtelCodec.Time)
+ query := shared.OtelAttributeQuery("toJSONString(span_attributes)", "length(toJSONString(span_attributes))", "start_time_unix_nano", predicate, shared.OtelClickHouseWinnerOrder, limits) + otelReadSettings
+ rows, err := chdb.Conn.Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ result := make(map[string]shared.OtelSpanAttributes)
+ for rows.Next() {
+ var id, encoded string
+ var flag uint8
+ if err := rows.Scan(&id, &encoded, &flag); err != nil {
+ return nil, err
+ }
+ if _, seen := result[id]; seen {
+ continue
+ }
+ attributes := shared.OtelSpanAttributes{Omitted: shared.OtelAttributeOmission(int(flag)), Bytes: len(encoded)}
+ if attributes.Omitted == "" && encoded != "" {
+ if err := json.Unmarshal([]byte(encoded), &attributes.Attributes); err != nil {
+ return nil, err
+ }
+ }
+ result[id] = attributes
+ }
+ return result, rows.Err()
+}
+
+var clickhouseReadLimitCodes = map[int32]bool{
+ 158: true, // TOO_MANY_ROWS
+ 159: true, // TIMEOUT_EXCEEDED
+ 160: true, // TOO_SLOW
+ 241: true, // MEMORY_LIMIT_EXCEEDED
+ 307: true, // TOO_MANY_BYTES
+ 396: true, // TOO_MANY_ROWS_OR_BYTES
+}
+
+func (r *otelSpanRepository) IsReadLimitError(err error) bool {
+ var exception *ch.Exception
+ return errors.As(err, &exception) && clickhouseReadLimitCodes[exception.Code]
+}
+
+var OtelSpanRepository = &otelSpanRepository{}
+
+func (r *otelSpanRepository) FindOTLP(ctx context.Context, project uuid.UUID, traceID, spanID string, at time.Time) ([]byte, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ from, to := shared.TraceWindowBounds(at.UTC())
+ var resourcePB, scopePB, spanPB string
+ err := chdb.Conn.QueryRow(ctx, "SELECT resource_pb, scope_pb, span_pb FROM "+shared.SpansTable+" WHERE project_id = ? AND trace_id = ? AND span_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY "+shared.OtelClickHouseWinnerOrder+" LIMIT 1"+otelReadSettings, project, traceID, spanID, from, to).Scan(&resourcePB, &scopePB, &spanPB)
+ if errors.Is(err, sql.ErrNoRows) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, err
+ }
+ return shared.AssembleOtelPayload([]byte(resourcePB), []byte(scopePB), []byte(spanPB)), nil
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/otel_span_adversarial_test.go b/backend/app/repositories/telemetry/clickhouse/otel_span_adversarial_test.go
new file mode 100644
index 00000000..a3db6e7c
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/otel_span_adversarial_test.go
@@ -0,0 +1,185 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "fmt"
+ "math"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestOtelSpanAdversarialAttributes(t *testing.T) {
+ ctx, conn := scratchDatabase(t)
+ previous := chdb.Conn
+ chdb.Conn = conn
+ defer func() { chdb.Conn = previous }()
+ createSpansTable(t, ctx, conn)
+
+ double := func(v float64) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: v}}
+ }
+ array := func(v ...*commonpb.AnyValue) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: v}}}
+ }
+ kvlist := func(v ...*commonpb.KeyValue) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_KvlistValue{KvlistValue: &commonpb.KeyValueList{Values: v}}}
+ }
+ deep := benchValue("leaf")
+ for range 120 {
+ deep = kvlist(benchKV("n", deep))
+ }
+ many := make([]*commonpb.KeyValue, 3000)
+ for i := range many {
+ many[i] = benchKV(fmt.Sprintf("bulk.key.%d", i), benchInt(int64(i)))
+ }
+ cases := map[string][]*commonpb.KeyValue{
+ "baseline": {benchKV("http.method", benchValue("GET"))},
+ "empty key": {benchKV("", benchValue("x"))},
+ "dot only key": {benchKV(".", benchValue("x"))},
+ "double dot key": {benchKV("a..b", benchValue("x"))},
+ "trailing dot key": {benchKV("a.", benchValue("x"))},
+ "leading dot key": {benchKV(".a", benchValue("x"))},
+ "scalar and dotted prefix": {benchKV("a", benchValue("x")), benchKV("a.b", benchValue("y"))},
+ "nested and dotted prefix": {benchKV("p", kvlist(benchKV("q", benchValue("x")))), benchKV("p.q", benchValue("y"))},
+ "kvlist with empty key": {benchKV("obj", kvlist(benchKV("", benchValue("x"))))},
+ "empty kvlist": {benchKV("obj", kvlist())},
+ "empty array": {benchKV("arr", array())},
+ "array with nulls": {benchKV("arr", array(&commonpb.AnyValue{}, benchInt(1)))},
+ "array of mixed values": {benchKV("arr", array(kvlist(benchKV("k", benchInt(1))), benchValue("s"), benchInt(2)))},
+ "array of arrays": {benchKV("arr", array(array(benchInt(1)), array(benchValue("a"))))},
+ "array of different maps": {benchKV("arr", array(kvlist(benchKV("a", benchInt(1))), kvlist(benchKV("a", benchValue("s")), benchKV("b", benchInt(2)))))},
+ "unset value": {benchKV("nothing", &commonpb.AnyValue{})},
+ "nil value": {{Key: "nilval"}},
+ "minimum integer": {benchKV("n", benchInt(math.MinInt64))},
+ "extreme doubles": {benchKV("d", double(math.MaxFloat64)), benchKV("tiny", double(5e-324)), benchKV("negzero", double(math.Copysign(0, -1)))},
+ "not a number": {benchKV("d", double(math.NaN())), benchKV("inf", double(math.Inf(-1)))},
+ "quotes and control chars": {benchKV("we`ird \"key\"\n\twith'quotes\\and/slash", benchValue("va\"l\nue"))},
+ "nul bytes": {benchKV("nul\x00key", benchValue("va\x00lue"))},
+ "unicode": {benchKV("ключ.名前.🔥", benchValue("значение"))},
+ "html sensitive": {benchKV("a<b>&c", benchValue("<script>&amp;"))},
+ "long key": {benchKV(strings.Repeat("k", 5000), benchValue("x"))},
+ "one megabyte value": {benchKV("big", benchValue(strings.Repeat("v", 1<<20)))},
+ "deep nesting": {benchKV("deep", deep)},
+ "three thousand keys": many,
+ "duplicate keys": {benchKV("dup", benchValue("first")), benchKV("dup", benchInt(2))},
+ "percent and escapes": {benchKV("100%", benchValue("x")), benchKV("a%2Eb", benchValue("y")), benchKV("a.b", benchValue("z"))},
+ "numeric keys": {benchKV("0", benchValue("x")), benchKV("1.5", benchValue("y"))},
+ }
+ project, start, index := uuid.New(), time.Now().UTC().Truncate(time.Microsecond), 0
+ for name, attributes := range cases {
+ index++
+ trace := fmt.Sprintf("%032x", index)
+ source := &tracepb.Span{TraceId: make([]byte, 16), SpanId: []byte{0, 0, 0, 0, 0, 0, 0, byte(index)}, Name: name, Attributes: attributes,
+ StartTimeUnixNano: uint64(start.UnixNano()), EndTimeUnixNano: uint64(start.Add(time.Millisecond).UnixNano()),
+ Events: []*tracepb.Span_Event{{Name: "event", TimeUnixNano: uint64(start.UnixNano()), Attributes: attributes}},
+ Links: []*tracepb.Span_Link{{TraceId: []byte{1}, SpanId: []byte{2}, Attributes: attributes}}}
+ source.TraceId[15] = byte(index)
+ span := models.OtelSpan{OTLP: source, Span: models.Span{ProjectId: project, TraceId: trace, SpanId: fmt.Sprintf("%016x", index), Name: name, StartTime: start, Duration: time.Millisecond},
+ Context: &tracepb.ResourceSpans{Resource: &resourcepb.Resource{Attributes: attributes}, ScopeSpans: []*tracepb.ScopeSpans{{Scope: &commonpb.InstrumentationScope{Name: "scope", Attributes: attributes}}}}}
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span}); err != nil {
+ t.Errorf("%s: insert rejected: %v", name, err)
+ continue
+ }
+ found, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: project, TraceId: trace, SpanId: span.SpanId}}, 0)
+ if err != nil || len(found) != 1 {
+ t.Errorf("%s: graph read: %d rows, %v", name, len(found), err)
+ continue
+ }
+ display, err := OtelSpanRepository.FindSpanAttributes(ctx, shared.SpanLookup{ProjectId: project, TraceId: trace}, []string{span.SpanId}, shared.OtelAttributeLimits{PerSpanBytes: shared.MaxOtelSpanAttributeBytes, BudgetBytes: shared.MaxOtelGraphBytes})
+ if err != nil {
+ t.Fatalf("%s: display attributes: %v", name, err)
+ }
+ if name == "one megabyte value" && display[span.SpanId].Omitted != models.SpanGraphAttributeSize {
+ t.Fatal("large attributes must be marked omitted")
+ }
+ payload, err := OtelSpanRepository.FindOTLP(ctx, project, trace, span.SpanId, start)
+ if err != nil {
+ t.Errorf("%s: payload read: %v", name, err)
+ continue
+ }
+ expected := proto.Clone(span.Context).(*tracepb.ResourceSpans)
+ expected.ScopeSpans[0].Spans = []*tracepb.Span{source}
+ var restored tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &restored); err != nil || !proto.Equal(expected, &restored) {
+ t.Errorf("%s: lossless payload changed: %v", name, err)
+ }
+ var validText uint8
+ if err := conn.QueryRow(ctx, "SELECT isValidJSON(resource) AND isValidJSON(scope) AND isValidJSON(events) AND isValidJSON(links) FROM spans_v2 WHERE project_id = ? AND span_id = ?", project, span.SpanId).Scan(&validText); err != nil || validText != 1 {
+ t.Errorf("%s: text columns are not valid JSON: %d %v", name, validText, err)
+ }
+ }
+}
+
+func TestOtelTopologyDoesNotReadWidePayloads(t *testing.T) {
+ ctx, conn := scratchDatabase(t)
+ previous := chdb.Conn
+ chdb.Conn = conn
+ defer func() { chdb.Conn = previous }()
+ createSpansTable(t, ctx, conn)
+ project := uuid.New()
+ trace := "0123456789abcdef0123456789abcdef"
+ // Payloads exceed the graph's 256 MiB read cap; topology must leave them on disk.
+ for batch := 0; batch < 6; batch++ {
+ err := conn.Exec(ctx, `INSERT INTO spans_v2 (project_id, trace_id, span_id, recorded_at, start_time_unix_nano, span_pb)
+ SELECT ?, ?, leftPad(toString(number + ?), 16, '0'), now(), toUInt64(toUnixTimestamp(now())) * 1000000000, repeat('x', 1000000) FROM numbers(50)`, project, trace, batch*50+1)
+ if err != nil {
+ t.Fatal(err)
+ }
+ }
+ spans, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: project, TraceId: trace}}, 0)
+ if err != nil || len(spans) != 300 {
+ t.Fatalf("wide payloads must not break scalar topology: %d spans, %v", len(spans), err)
+ }
+ search := shared.OtelSpanSearch{ProjectId: project, From: time.Now().Add(-time.Hour), To: time.Now().Add(time.Hour), Page: 1, PageSize: 50}
+ rows, total, err := OtelSpanRepository.Search(ctx, search)
+ if err != nil || len(rows) != 50 || total != 300 {
+ t.Fatalf("wide payload search: %d rows, total %d, %v", len(rows), total, err)
+ }
+}
+
+func TestOtelSpanVersionUpgradeReadsExistingParts(t *testing.T) {
+ ctx, conn := scratchDatabase(t)
+ previous := chdb.Conn
+ chdb.Conn = conn
+ defer func() { chdb.Conn = previous }()
+ apply := func(name string) {
+ migration, err := os.ReadFile("../../../migrations/ch/" + name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.Exec(ctx, string(migration)); err != nil {
+ t.Fatal(err)
+ }
+ }
+ apply("0085_create_spans_v2.up.sql")
+ span := models.OtelSpan{Span: models.Span{ProjectId: uuid.New(), TraceId: "0123456789abcdef0123456789abcdef", SpanId: "0123456789abcdef", Name: "before upgrade", StartTime: time.Now().UTC(), Duration: time.Second}}
+ insert := func() {
+ if rejected, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span}); err != nil || rejected != 0 {
+ t.Fatalf("insert: %d, %v", rejected, err)
+ }
+ }
+ insert()
+ apply("0090_add_span_version.up.sql")
+ apply("0090_add_span_version.up.sql")
+ insert()
+ var versions, count uint64
+ if err := conn.QueryRow(ctx, "SELECT uniqExact(span_version), count() FROM spans_v2").Scan(&versions, &count); err != nil || versions != 1 || count != 2 {
+ t.Fatalf("mixed old/new parts: %d versions, %d rows, %v", versions, count, err)
+ }
+ found, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: span.ProjectId, TraceId: span.TraceId}}, 0)
+ if err != nil || len(found) != 1 || found[0].Name != span.Name {
+ t.Fatalf("upgrade lost a span: %+v, %v", found, err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/otel_span_bench_fixture_test.go b/backend/app/repositories/telemetry/clickhouse/otel_span_bench_fixture_test.go
new file mode 100644
index 00000000..18897846
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/otel_span_bench_fixture_test.go
@@ -0,0 +1,70 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "context"
+ "os"
+ "strings"
+ "testing"
+
+ ch "github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/ClickHouse/clickhouse-go/v2/lib/driver"
+ "github.com/google/uuid"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+)
+
+func scratchDatabase(t *testing.T) (context.Context, driver.Conn) {
+ server := os.Getenv("TEST_CLICKHOUSE_SERVER")
+ if server == "" {
+ t.Skip("TEST_CLICKHOUSE_SERVER not set")
+ }
+ ctx := context.Background()
+ auth := ch.Auth{Database: "default", Username: os.Getenv("TEST_CLICKHOUSE_USERNAME"), Password: os.Getenv("TEST_CLICKHOUSE_PASSWORD")}
+ if auth.Username == "" {
+ auth.Username = "default"
+ }
+ admin, err := ch.Open(&ch.Options{Addr: []string{server}, Auth: auth})
+ if err != nil {
+ t.Fatal(err)
+ }
+ database := "span_scratch_" + strings.ReplaceAll(uuid.NewString(), "-", "")
+ if err := admin.Exec(ctx, "CREATE DATABASE "+database); err != nil {
+ t.Fatal(err)
+ }
+ auth.Database = database
+ conn, err := ch.Open(&ch.Options{Addr: []string{server}, Auth: auth})
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ conn.Close()
+ admin.Exec(ctx, "DROP DATABASE "+database)
+ admin.Close()
+ })
+ return ctx, conn
+}
+
+func createSpansTable(t *testing.T, ctx context.Context, conn driver.Conn) {
+ for _, name := range []string{"0085_create_spans_v2.up.sql", "0090_add_span_version.up.sql"} {
+ migration, err := os.ReadFile("../../../migrations/ch/" + name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := conn.Exec(ctx, string(migration)); err != nil {
+ t.Fatal(err)
+ }
+ }
+}
+
+func benchValue(v string) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: v}}
+}
+
+func benchInt(v int64) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: v}}
+}
+
+func benchKV(key string, value *commonpb.AnyValue) *commonpb.KeyValue {
+ return &commonpb.KeyValue{Key: key, Value: value}
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/otel_span_partitions_test.go b/backend/app/repositories/telemetry/clickhouse/otel_span_partitions_test.go
new file mode 100644
index 00000000..85adade6
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/otel_span_partitions_test.go
@@ -0,0 +1,94 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "fmt"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+)
+
+func partitionRows(days, perDay int) []clickhouseOtelRow {
+ rows := make([]clickhouseOtelRow, 0, days*perDay)
+ for i := range perDay {
+ for day := days; day > 0; day-- {
+ rows = append(rows, clickhouseOtelRow{values: []any{day, i}, day: int64(day)})
+ }
+ }
+ return rows
+}
+
+func TestPartitionedInsertsRespectThePartitionLimit(t *testing.T) {
+ if inserts, overflow := partitionedInserts(partitionRows(clickhouseMaxPartitionsPerInsert, 3)); len(inserts) != 1 || len(overflow) != 0 || len(inserts[0]) != 300 {
+ t.Fatalf("a request within the limit must stay one insert: %d inserts, %d overflow", len(inserts), len(overflow))
+ }
+ inserts, overflow := partitionedInserts(partitionRows(130, 3))
+ if len(inserts) != 2 || len(overflow) != 0 {
+ t.Fatalf("130 days need two inserts: %d inserts, %d overflow", len(inserts), len(overflow))
+ }
+ stored, seen := 0, map[int64]int{}
+ for index, insert := range inserts {
+ days := map[int64]bool{}
+ for _, row := range insert {
+ days[row.day] = true
+ }
+ if len(days) > clickhouseMaxPartitionsPerInsert {
+ t.Fatalf("insert %d touches %d partitions", index, len(days))
+ }
+ for day := range days {
+ seen[day]++
+ }
+ stored += len(insert)
+ }
+ if stored != 390 {
+ t.Fatalf("rows lost while splitting: %d", stored)
+ }
+ for day, inserts := range seen {
+ if inserts != 1 {
+ t.Fatalf("day %d was split across %d inserts", day, inserts)
+ }
+ }
+ limit := clickhouseMaxPartitionsPerInsert * clickhouseMaxPartitionedInserts
+ inserts, overflow = partitionedInserts(partitionRows(limit, 1))
+ if len(inserts) != clickhouseMaxPartitionedInserts || len(overflow) != 0 {
+ t.Fatalf("the bound itself must fit: %d inserts, %d overflow", len(inserts), len(overflow))
+ }
+ inserts, overflow = partitionedInserts(partitionRows(limit+50, 2))
+ if len(inserts) != clickhouseMaxPartitionedInserts || len(overflow) != 100 {
+ t.Fatalf("one request must not cause unbounded inserts: %d inserts, %d overflow", len(inserts), len(overflow))
+ }
+}
+
+func TestOtelSpanInsertSpanningManyDays(t *testing.T) {
+ ctx, conn := scratchDatabase(t)
+ previous := chdb.Conn
+ chdb.Conn = conn
+ defer func() { chdb.Conn = previous }()
+ createSpansTable(t, ctx, conn)
+
+ const days = 130
+ project, first := uuid.New(), time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)
+ spans := make([]models.OtelSpan, days)
+ for day := range spans {
+ start := uint64(first.AddDate(0, 0, day).UnixNano())
+ source := &tracepb.Span{TraceId: make([]byte, 16), SpanId: []byte{0, 0, 0, 0, 0, 0, 1, byte(day + 1)}, Name: "buffered", StartTimeUnixNano: start, EndTimeUnixNano: start + 1000}
+ source.TraceId[15] = byte(day + 1)
+ spans[day] = models.OtelSpan{OTLP: source, Span: models.Span{ProjectId: project, TraceId: fmt.Sprintf("%032x", day+1), SpanId: fmt.Sprintf("%016x", 256+day+1), Name: "buffered"}}
+ }
+ rejected, err := OtelSpanRepository.InsertAsync(ctx, spans)
+ if err != nil || rejected != 0 {
+ t.Fatalf("a request spanning %d days must be stored: rejected=%d err=%v", days, rejected, err)
+ }
+ var rows, partitions uint64
+ if err := conn.QueryRow(ctx, "SELECT count(), uniqExact(toYYYYMMDD(recorded_at)) FROM spans_v2 WHERE project_id = ?", project).Scan(&rows, &partitions); err != nil {
+ t.Fatal(err)
+ }
+ if rows != days || partitions != days {
+ t.Fatalf("stored %d rows in %d daily partitions, want %d in %d", rows, partitions, days, days)
+ }
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/otel_span_repository_test.go b/backend/app/repositories/telemetry/clickhouse/otel_span_repository_test.go
new file mode 100644
index 00000000..0bdd8a48
--- /dev/null
+++ b/backend/app/repositories/telemetry/clickhouse/otel_span_repository_test.go
@@ -0,0 +1,172 @@
+//go:build telemetry_ch
+
+package clickhouse
+
+import (
+ "context"
+ "encoding/hex"
+ "fmt"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+ "math"
+ "os"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ ch "github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func TestOtelSpanWireRoundTrip(t *testing.T) {
+ server := os.Getenv("TEST_CLICKHOUSE_SERVER")
+ if server == "" {
+ t.Skip("TEST_CLICKHOUSE_SERVER not set")
+ }
+ ctx := context.Background()
+ auth := ch.Auth{Database: "default", Username: os.Getenv("TEST_CLICKHOUSE_USERNAME"), Password: os.Getenv("TEST_CLICKHOUSE_PASSWORD")}
+ if auth.Username == "" {
+ auth.Username = "default"
+ }
+ admin, err := ch.Open(&ch.Options{Addr: []string{server}, Auth: auth})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer admin.Close()
+ database := "span_graph_test_" + strings.ReplaceAll(uuid.NewString(), "-", "")
+ if err := admin.Exec(ctx, "CREATE DATABASE "+database); err != nil {
+ t.Fatal(err)
+ }
+ defer admin.Exec(ctx, "DROP DATABASE "+database)
+ auth.Database = database
+ conn, err := ch.Open(&ch.Options{Addr: []string{server}, Auth: auth})
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer conn.Close()
+ previous := chdb.Conn
+ chdb.Conn = conn
+ defer func() { chdb.Conn = previous }()
+ createSpansTable(t, ctx, conn)
+ project, start := uuid.New(), time.Now().UTC().Truncate(time.Microsecond)
+ span := models.OtelSpan{Span: models.Span{ProjectId: project, TraceId: "0123456789abcdef0123456789abcdef", SpanId: "abcdef1234567890",
+ ParentSpanId: "fedcba9876543210", Name: "query", StartTime: start, RecordedAt: start, Duration: 123456789,
+ Attributes: map[string]string{"db.system": "postgresql"}, SpanKind: 3, StatusCode: 2, ServiceName: "api", ScopeName: "test"}}
+ span.OTLP = &tracepb.Span{Name: "original name", Flags: 0xffffffff, Status: &tracepb.Status{Message: "original status"}, Links: []*tracepb.Span_Link{{Flags: 0x100}}}
+ span.OTLP.Attributes = []*commonpb.KeyValue{{Key: "db.system", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: "postgresql"}}}}
+ span.OTLP.StartTimeUnixNano = uint64(span.StartTime.UnixNano())
+ span.Context = &tracepb.ResourceSpans{SchemaUrl: "resource-schema", ScopeSpans: []*tracepb.ScopeSpans{{SchemaUrl: "scope-schema"}}}
+ span.OTLP.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x01})
+ span.Context.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x02})
+ span.Context.ScopeSpans[0].ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x03})
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span}); err != nil {
+ t.Fatal(err)
+ }
+ var storedJSON string
+ if err := conn.QueryRow(ctx, "SELECT toJSONString(span_attributes) FROM spans_v2").Scan(&storedJSON); err != nil {
+ t.Fatal(err)
+ }
+ t.Log("stored attributes", storedJSON)
+ found, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: project, TraceId: span.TraceId, SpanId: span.SpanId}}, 0)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(found) != 1 {
+ t.Fatalf("expected one span, got %d", len(found))
+ }
+ attributes, err := OtelSpanRepository.FindSpanAttributes(ctx, shared.SpanLookup{ProjectId: project, TraceId: span.TraceId}, []string{span.SpanId}, shared.OtelAttributeLimits{PerSpanBytes: shared.MaxOtelSpanAttributeBytes, BudgetBytes: shared.MaxOtelGraphBytes})
+ if err != nil {
+ t.Fatal(err)
+ }
+ found[0].Attributes = attributes[span.SpanId].Attributes
+ expectedSpan := span.Span
+ expectedSpan.RecordedAt = start.Truncate(time.Second)
+ if !reflect.DeepEqual(found[0].Span, expectedSpan) {
+ t.Fatalf("stored span mismatch: want %+v; got %+v", expectedSpan, found)
+ }
+ payload, err := OtelSpanRepository.FindOTLP(ctx, project, span.TraceId, span.SpanId, span.StartTime)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var restored tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &restored); err != nil {
+ t.Fatal(err)
+ }
+ expected := proto.Clone(span.Context).(*tracepb.ResourceSpans)
+ expected.ScopeSpans[0].Spans = []*tracepb.Span{span.OTLP}
+ if !proto.Equal(expected, &restored) {
+ t.Fatal("OTLP fidelity changed")
+ }
+ other, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: uuid.New(), TraceId: span.TraceId, SpanId: span.SpanId}}, 0)
+ if err != nil || len(other) != 0 {
+ t.Fatalf("project isolation failed: %+v %v", other, err)
+ }
+ stringValue := func(value string) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: value}}
+ }
+ attrs := []*commonpb.KeyValue{
+ {Key: "large", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: 9223372036854775807}}},
+ {Key: "enabled", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: true}}},
+ {Key: "http.method", Value: stringValue("POST")},
+ {Key: "http%2Emethod", Value: stringValue("literal-percent")},
+ {Key: "http", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_KvlistValue{KvlistValue: &commonpb.KeyValueList{Values: []*commonpb.KeyValue{{Key: "method", Value: stringValue("nested")}}}}}},
+ {Key: "bytes", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BytesValue{BytesValue: []byte{0, 255}}}},
+ {Key: "nonfinite", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: math.Inf(1)}}},
+ {Key: "array", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: []*commonpb.AnyValue{stringValue("first"), {Value: &commonpb.AnyValue_IntValue{IntValue: 123}}}}}}},
+ }
+ for i, mixed := range []*commonpb.AnyValue{stringValue("text"), {Value: &commonpb.AnyValue_IntValue{IntValue: 123}}, {Value: &commonpb.AnyValue_BoolValue{BoolValue: true}}} {
+ sample := span
+ sample.SpanId = fmt.Sprintf("%016x", i+1)
+ sample.OTLP = proto.Clone(span.OTLP).(*tracepb.Span)
+ sample.OTLP.SpanId, _ = hex.DecodeString(sample.SpanId)
+ sample.OTLP.Attributes = append(append([]*commonpb.KeyValue{}, attrs...), &commonpb.KeyValue{Key: "mixed", Value: mixed})
+ sample.OTLP.Events = []*tracepb.Span_Event{{Name: "event", TimeUnixNano: sample.OTLP.StartTimeUnixNano + 1, Attributes: attrs, DroppedAttributesCount: 4}}
+ sample.OTLP.Links = []*tracepb.Span_Link{{TraceId: []byte{1, 2, 3}, SpanId: []byte{4, 5}, TraceState: "vendor=state", Flags: 0xffffffff, Attributes: attrs, DroppedAttributesCount: 5}}
+ sample.Context = &tracepb.ResourceSpans{Resource: &resourcepb.Resource{Attributes: []*commonpb.KeyValue{{Key: "service.name", Value: stringValue("api")}}}, ScopeSpans: []*tracepb.ScopeSpans{{Scope: &commonpb.InstrumentationScope{Name: "scope", Attributes: attrs}}}}
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{sample}); err != nil {
+ t.Fatal(err)
+ }
+ payload, err := OtelSpanRepository.FindOTLP(ctx, project, sample.TraceId, sample.SpanId, sample.StartTime)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var decoded tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &decoded); err != nil {
+ t.Fatal(err)
+ }
+ if !proto.Equal(decoded.ScopeSpans[0].Spans[0], sample.OTLP) || !proto.Equal(decoded.Resource, sample.Context.Resource) {
+ t.Fatal("the attribute projection changed lossless span export")
+ }
+ }
+ var large int64
+ var enabled, dotted, nested, percent, array, service, eventName, linkedTrace string
+ var eventTime, linkFlags uint64
+ query := "SELECT toInt64(span_attributes['large']), span_attributes['enabled'], span_attributes['http.method'], span_attributes['http'], span_attributes['http%2Emethod'], span_attributes['array'], JSONExtractString(resource, 'attributes', 'service.name'), JSONExtractString(events, 1, 'name'), JSONExtractUInt(events, 1, 'time_unix_nano'), JSONExtractString(links, 1, 'trace_id'), JSONExtractUInt(links, 1, 'flags') FROM spans_v2 WHERE span_id = '0000000000000001'"
+ if err := conn.QueryRow(ctx, query).Scan(&large, &enabled, &dotted, &nested, &percent, &array, &service, &eventName, &eventTime, &linkedTrace, &linkFlags); err != nil {
+ t.Fatal(err)
+ }
+ if large != 9223372036854775807 || enabled != "true" || dotted != "POST" || nested != `{"method":"nested"}` || percent != "literal-percent" || array != `["first",123]` || service != "api" || eventName != "event" || eventTime != span.OTLP.StartTimeUnixNano+1 || linkedTrace != "010203" || linkFlags != 0xffffffff {
+ t.Fatalf("queryable values changed: %d %s %s %s %s %s %s %s %d %s %d", large, enabled, dotted, nested, percent, array, service, eventName, eventTime, linkedTrace, linkFlags)
+ }
+ var literalKey string
+ if err := conn.QueryRow(ctx, "SELECT JSONExtractString(events, 1, 'attributes', 'http%2Emethod') FROM spans_v2 WHERE span_id = '0000000000000001'").Scan(&literalKey); err != nil || literalKey != "literal-percent" {
+ t.Fatalf("text columns must keep attribute keys verbatim: %q %v", literalKey, err)
+ }
+ var distinctResources, distinctEnvelopes, matched uint64
+ if err := conn.QueryRow(ctx, "SELECT uniqExact(resource), uniqExact(resource_pb), countIf(resource LIKE '%\"service.name\":\"api\"%') FROM spans_v2 WHERE span_id IN ('0000000000000001','0000000000000002','0000000000000003')").Scan(&distinctResources, &distinctEnvelopes, &matched); err != nil {
+ t.Fatal(err)
+ }
+ if distinctResources != 1 || distinctEnvelopes != 1 || matched != 3 {
+ t.Fatalf("equal resources must share one dictionary entry and match a LIKE predicate: %d %d %d", distinctResources, distinctEnvelopes, matched)
+ }
+ var types uint64
+ if err := conn.QueryRow(ctx, "SELECT uniqExact(span_attributes['mixed']) FROM spans_v2 WHERE span_id IN ('0000000000000001','0000000000000002','0000000000000003') AND span_attributes['mixed'] IN ('text', '123', 'true')").Scan(&types); err != nil || types != 3 {
+ t.Fatalf("one key must hold values of any source type: %d %v", types, err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/clickhouse/profile.repository.go b/backend/app/repositories/telemetry/clickhouse/profile.repository.go
index 1cb1e277..b27e7899 100644
--- a/backend/app/repositories/telemetry/clickhouse/profile.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/profile.repository.go
@@ -73,7 +73,7 @@ func (r *profileRepository) InsertProfilesAsync(ctx context.Context, profiles []
return nil
}
batch, err := chdb.Conn.PrepareBatch(chdb.BatchCtx(),
- "INSERT INTO profiles (id, project_id, recorded_at, duration, service_name, profile_type, unit, is_gauge, sample_count, total_value, server_name, app_version, attributes, storage_key, trace_id, span_id, distributed_trace_id)")
+ "INSERT INTO profiles (id, project_id, recorded_at, duration, service_name, profile_type, unit, is_gauge, sample_count, total_value, server_name, app_version, attributes, storage_key, trace_id, span_id)")
if err != nil {
return err
}
@@ -87,7 +87,7 @@ func (r *profileRepository) InsertProfilesAsync(ctx context.Context, profiles []
if err := batch.Append(
p.Id, p.ProjectId, p.RecordedAt, int64(p.Duration), p.ServiceName, p.ProfileType, p.Unit, boolToUInt8(p.IsGauge),
p.SampleCount, p.TotalValue, p.ServerName, p.AppVersion, attributesJSON, p.StorageKey,
- p.TraceId, p.SpanId, p.DistributedTraceId,
+ p.TraceId, p.SpanId,
); err != nil {
return err
}
diff --git a/backend/app/repositories/telemetry/clickhouse/session.repository.go b/backend/app/repositories/telemetry/clickhouse/session.repository.go
index 05272dfa..ea07a4ce 100644
--- a/backend/app/repositories/telemetry/clickhouse/session.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/session.repository.go
@@ -22,13 +22,13 @@ func (r *sessionRepository) Upsert(ctx context.Context, sessions []models.Sessio
if len(sessions) == 0 {
return nil
}
- return chdb.SendBatch("INSERT INTO sessions (id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id, version)", func(batch driver.Batch) error {
+ return chdb.SendBatch("INSERT INTO sessions (id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id, version)", func(batch driver.Batch) error {
for _, s := range sessions {
attrs := s.Attributes
if attrs == nil {
attrs = map[string]string{}
}
- if err := batch.Append(s.Id, s.ProjectId, s.StartedAt, s.EndedAt, s.Duration, s.ClientIP, attrs, s.AppVersion, s.ServerName, s.DistributedTraceId, time.Now()); err != nil {
+ if err := batch.Append(s.Id, s.ProjectId, s.StartedAt, s.EndedAt, s.Duration, s.ClientIP, attrs, s.AppVersion, s.ServerName, s.TraceId, time.Now()); err != nil {
return err
}
}
@@ -69,7 +69,7 @@ func (r *sessionRepository) FindAll(ctx context.Context, projectId uuid.UUID, fr
offset := (page - 1) * pageSize
- query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id FROM sessions FINAL WHERE project_id = ? AND started_at >= ? AND started_at <= ?" + whereExtra + " ORDER BY " + orderBy + " " + sortDir + " LIMIT ? OFFSET ?"
+ query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id FROM sessions FINAL WHERE project_id = ? AND started_at >= ? AND started_at <= ?" + whereExtra + " ORDER BY " + orderBy + " " + sortDir + " LIMIT ? OFFSET ?"
queryArgs := []interface{}{projectId, fromDate, toDate}
queryArgs = append(queryArgs, extraArgs...)
queryArgs = append(queryArgs, pageSize, offset)
@@ -82,7 +82,7 @@ func (r *sessionRepository) FindAll(ctx context.Context, projectId uuid.UUID, fr
var sessions []models.Session
for rows.Next() {
var s models.Session
- if err := rows.Scan(&s.Id, &s.ProjectId, &s.StartedAt, &s.EndedAt, &s.Duration, &s.ClientIP, &s.Attributes, &s.AppVersion, &s.ServerName, &s.DistributedTraceId); err != nil {
+ if err := rows.Scan(&s.Id, &s.ProjectId, &s.StartedAt, &s.EndedAt, &s.Duration, &s.ClientIP, &s.Attributes, &s.AppVersion, &s.ServerName, &s.TraceId); err != nil {
return nil, 0, err
}
sessions = append(sessions, s)
@@ -93,7 +93,7 @@ func (r *sessionRepository) FindAll(ctx context.Context, projectId uuid.UUID, fr
func (r *sessionRepository) FindById(ctx context.Context, projectId, sessionId uuid.UUID, startedAt *time.Time) (*models.Session, error) {
var s models.Session
- query := `SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id
+ query := `SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id
FROM sessions FINAL
WHERE project_id = ? AND id = ?`
args := []any{projectId, sessionId}
@@ -105,7 +105,7 @@ func (r *sessionRepository) FindById(ctx context.Context, projectId, sessionId u
query += ` LIMIT 1`
err := chdb.Conn.QueryRow(ctx, query, args...).Scan(
- &s.Id, &s.ProjectId, &s.StartedAt, &s.EndedAt, &s.Duration, &s.ClientIP, &s.Attributes, &s.AppVersion, &s.ServerName, &s.DistributedTraceId,
+ &s.Id, &s.ProjectId, &s.StartedAt, &s.EndedAt, &s.Duration, &s.ClientIP, &s.Attributes, &s.AppVersion, &s.ServerName, &s.TraceId,
)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
diff --git a/backend/app/repositories/telemetry/clickhouse/span.repository.go b/backend/app/repositories/telemetry/clickhouse/span.repository.go
deleted file mode 100644
index 5a752d7a..00000000
--- a/backend/app/repositories/telemetry/clickhouse/span.repository.go
+++ /dev/null
@@ -1,112 +0,0 @@
-//go:build telemetry_ch
-
-package clickhouse
-
-import (
- "context"
- "encoding/json"
- "slices"
- "time"
-
- "github.com/tracewayapp/traceway/backend/app/chdb"
- "github.com/tracewayapp/traceway/backend/app/models"
- "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
-
- "github.com/ClickHouse/clickhouse-go/v2/lib/driver"
- "github.com/google/uuid"
-)
-
-type spanRepository struct{}
-
-func (r *spanRepository) InsertAsync(ctx context.Context, spans []models.Span) error {
- if len(spans) == 0 {
- return nil
- }
-
- return chdb.SendBatch("INSERT INTO spans (id, trace_id, project_id, name, start_time, duration, recorded_at, parent_span_id, attributes)", func(batch driver.Batch) error {
- for _, s := range spans {
- attributesJSON := "{}"
- if len(s.Attributes) != 0 {
- if attributesBytes, err := json.Marshal(s.Attributes); err == nil {
- attributesJSON = string(attributesBytes)
- }
- }
- if err := batch.Append(
- s.Id,
- s.TraceId,
- s.ProjectId,
- s.Name,
- s.StartTime,
- int64(s.Duration),
- s.RecordedAt,
- s.ParentSpanId,
- attributesJSON,
- ); err != nil {
- return err
- }
- }
- return nil
- })
-}
-
-func (r *spanRepository) FindByTraceId(ctx context.Context, projectId, traceId uuid.UUID, recordedAt *time.Time) ([]models.Span, error) {
- query, args := shared.SpanLookupQuery([]shared.SpanLookup{{ProjectId: projectId, TraceId: traceId, RecordedAt: recordedAt}},
- func(t time.Time) any { return t.UTC() })
- return r.querySpans(ctx, query, args)
-}
-
-func (r *spanRepository) FindByTraces(ctx context.Context, lookups []shared.SpanLookup) ([]models.Span, error) {
- spans := make([]models.Span, 0)
- type spanKey struct {
- projectId, traceId, id uuid.UUID
- recordedAt time.Time
- }
- seen := make(map[spanKey]bool)
- for batch := range slices.Chunk(lookups, shared.SpanLookupBatchSize) {
- query, args := shared.SpanLookupQuery(batch, func(t time.Time) any { return t.UTC() })
- found, err := r.querySpans(ctx, query, args)
- if err != nil {
- return nil, err
- }
- for _, span := range found {
- key := spanKey{span.ProjectId, span.TraceId, span.Id, span.RecordedAt}
- if !seen[key] {
- seen[key] = true
- spans = append(spans, span)
- }
- }
- }
- slices.SortStableFunc(spans, func(a, b models.Span) int { return a.StartTime.Compare(b.StartTime) })
- return spans, nil
-}
-
-func (r *spanRepository) querySpans(ctx context.Context, query string, args []any) ([]models.Span, error) {
- rows, err := chdb.Conn.Query(ctx, query, args...)
- if err != nil {
- return nil, err
- }
- defer rows.Close()
-
- var spans []models.Span
- for rows.Next() {
- var s models.Span
- var attributesJSON string
- if err := rows.Scan(
- &s.Id, &s.TraceId, &s.ProjectId,
- &s.Name, &s.StartTime, &s.Duration, &s.RecordedAt, &s.ParentSpanId,
- &attributesJSON,
- ); err != nil {
- return nil, err
- }
- if attributesJSON != "" && attributesJSON != "{}" {
- if err := json.Unmarshal([]byte(attributesJSON), &s.Attributes); err != nil {
- s.Attributes = nil
- }
- }
- spans = append(spans, s)
- }
-
- return spans, rows.Err()
-}
-
-var SpanRepository = &spanRepository{}
diff --git a/backend/app/repositories/telemetry/clickhouse/task.repository.go b/backend/app/repositories/telemetry/clickhouse/task.repository.go
index 33efcc35..37b146b8 100644
--- a/backend/app/repositories/telemetry/clickhouse/task.repository.go
+++ b/backend/app/repositories/telemetry/clickhouse/task.repository.go
@@ -22,7 +22,7 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
if len(lines) == 0 {
return nil
}
- return chdb.SendBatch("INSERT INTO tasks (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root)", func(batch driver.Batch) error {
+ return chdb.SendBatch("INSERT INTO tasks_v2 (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root)", func(batch driver.Batch) error {
for _, t := range lines {
attributesJSON := "{}"
if len(t.Attributes) != 0 {
@@ -34,7 +34,7 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
if t.IsRoot {
isRoot = 1
}
- if err := batch.Append(t.Id, t.ProjectId, t.TaskName, int64(t.Duration), t.RecordedAt, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.DistributedTraceId, t.SpanId, isRoot); err != nil {
+ if err := batch.Append(t.Id, t.ProjectId, t.TaskName, int64(t.Duration), t.RecordedAt, t.ClientIP, attributesJSON, t.AppVersion, t.ServerName, t.TraceId, t.SpanId, t.ParentSpanId, isRoot); err != nil {
return err
}
}
@@ -44,13 +44,13 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
func (e *taskRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, start, end).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, start, end).Scan(&count)
return int64(count), err
}
func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string) ([]models.Task, int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, fromDate, toDate).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, fromDate, toDate).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -66,7 +66,7 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromD
orderBy = "recorded_at"
}
- query := "SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id FROM tasks WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " DESC LIMIT ? OFFSET ?"
+ query := "SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id FROM tasks_v2 WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " DESC LIMIT ? OFFSET ?"
rows, err := chdb.Conn.Query(ctx, query, projectId, fromDate, toDate, pageSize, offset)
if err != nil {
return nil, 0, err
@@ -77,7 +77,7 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromD
for rows.Next() {
var t models.Task
var attributesJSON string
- if err := rows.Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId); err != nil {
+ if err := rows.Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId); err != nil {
return nil, 0, err
}
if attributesJSON != "" && attributesJSON != "{}" {
@@ -106,7 +106,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
}
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT uniq(task_name) FROM tasks WHERE "+whereClause, args...).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT uniq(task_name) FROM tasks_v2 WHERE "+whereClause, args...).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -143,7 +143,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
max(recorded_at) as last_seen,
max(is_root) as has_root,
max(if(is_root = 0, 1, 0)) as has_non_root
- FROM tasks
+ FROM tasks_v2
WHERE ` + whereClause + `
GROUP BY task_name
ORDER BY ` + orderExpr + ` ` + sortDir + `
@@ -177,7 +177,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID, taskName string, fromDate, toDate time.Time, page, pageSize int, orderBy string, sortDirection string) ([]models.Task, int64, error) {
var count uint64
- err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM tasks WHERE project_id = ? AND task_name = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, taskName, fromDate, toDate).Scan(&count)
+ err := chdb.Conn.QueryRow(ctx, "SELECT count() FROM tasks_v2 WHERE project_id = ? AND task_name = ? AND recorded_at >= ? AND recorded_at <= ?", projectId, taskName, fromDate, toDate).Scan(&count)
if err != nil {
return nil, 0, err
}
@@ -199,7 +199,7 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
sortDir = "ASC"
}
- query := "SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id FROM tasks WHERE project_id = ? AND task_name = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " " + sortDir + " LIMIT ? OFFSET ?"
+ query := "SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id FROM tasks_v2 WHERE project_id = ? AND task_name = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY " + orderBy + " " + sortDir + " LIMIT ? OFFSET ?"
rows, err := chdb.Conn.Query(ctx, query, projectId, taskName, fromDate, toDate, pageSize, offset)
if err != nil {
return nil, 0, err
@@ -210,7 +210,7 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
for rows.Next() {
var t models.Task
var attributesJSON string
- if err := rows.Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId); err != nil {
+ if err := rows.Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt, &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId); err != nil {
return nil, 0, err
}
if attributesJSON != "" && attributesJSON != "{}" {
@@ -226,8 +226,8 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
// FindById returns a single task by ID
func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID, recordedAt *time.Time) (*models.Task, error) {
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
- FROM tasks
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM tasks_v2
WHERE project_id = ? AND id = ?`
args := []any{projectId, taskId}
if recordedAt != nil {
@@ -243,7 +243,7 @@ func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UU
err := chdb.Conn.QueryRow(ctx, query, args...).Scan(
&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt,
- &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId, &t.SpanId, &isRoot)
+ &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId, &isRoot)
t.IsRoot = isRoot == 1
if err != nil {
@@ -267,7 +267,7 @@ func (e *taskRepository) CountByHour(ctx context.Context, projectId uuid.UUID, s
query := `SELECT
toStartOfHour(recorded_at) as hour,
toFloat64(count()) as count
- FROM tasks
+ FROM tasks_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY hour
ORDER BY hour ASC`
@@ -295,7 +295,7 @@ func (e *taskRepository) AvgDurationByHour(ctx context.Context, projectId uuid.U
query := `SELECT
toStartOfHour(recorded_at) as hour,
avg(duration) / 1000000 as avg_duration_ms
- FROM tasks
+ FROM tasks_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY hour
ORDER BY hour ASC`
@@ -323,7 +323,7 @@ func (e *taskRepository) CountByInterval(ctx context.Context, projectId uuid.UUI
query := `SELECT
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
toFloat64(count()) as count
- FROM tasks
+ FROM tasks_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY bucket
ORDER BY bucket ASC`
@@ -351,7 +351,7 @@ func (e *taskRepository) AvgDurationByInterval(ctx context.Context, projectId uu
query := `SELECT
toStartOfInterval(recorded_at, INTERVAL ? MINUTE) as bucket,
avg(duration) / 1000000 as avg_duration_ms
- FROM tasks
+ FROM tasks_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY bucket
ORDER BY bucket ASC`
@@ -383,7 +383,7 @@ func (e *taskRepository) FindWorstTasks(ctx context.Context, projectId uuid.UUID
quantile(0.95)(duration) as p95_duration,
avg(duration) as avg_duration,
max(recorded_at) as last_seen
- FROM tasks
+ FROM tasks_v2
WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ?
GROUP BY task_name
ORDER BY count * (p95_duration - p50_duration) DESC
@@ -425,7 +425,7 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
quantile(0.5)(duration) / 1000000 as p50_duration_ms,
quantile(0.95)(duration) / 1000000 as p95_duration_ms,
quantile(0.99)(duration) / 1000000 as p99_duration_ms
- FROM tasks
+ FROM tasks_v2
WHERE project_id = ? AND task_name = ? AND recorded_at >= ? AND recorded_at <= ?`
var stats models.TaskDetailStats
@@ -449,11 +449,14 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
return &stats, nil
}
-func (e *taskRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Task, error) {
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks
- WHERE distributed_trace_id = ? AND project_id IN (?)`
- args := []any{distributedTraceId, projectIds}
+func (e *taskRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Task, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
+ return nil, nil
+ }
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM tasks_v2
+ WHERE trace_id IN (?) AND project_id IN (?)`
+ args := []any{traceIds, projectIds}
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= ? AND recorded_at <= ?`
@@ -472,7 +475,7 @@ func (e *taskRepository) FindByDistributedTraceId(ctx context.Context, distribut
var t models.Task
var attributesJSON string
if err := rows.Scan(&t.Id, &t.ProjectId, &t.TaskName, &t.Duration, &t.RecordedAt,
- &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.DistributedTraceId); err != nil {
+ &t.ClientIP, &attributesJSON, &t.AppVersion, &t.ServerName, &t.TraceId, &t.SpanId, &t.ParentSpanId, &t.IsRoot); err != nil {
return nil, err
}
if attributesJSON != "" && attributesJSON != "{}" {
diff --git a/backend/app/repositories/telemetry/complete_span_test.go b/backend/app/repositories/telemetry/complete_span_test.go
new file mode 100644
index 00000000..43cdc1e8
--- /dev/null
+++ b/backend/app/repositories/telemetry/complete_span_test.go
@@ -0,0 +1,142 @@
+//go:build !telemetry_ch
+
+package telemetry
+
+import (
+ "context"
+ "encoding/hex"
+ "encoding/json"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestCompleteSpanFieldsAndNativeSpans(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, trace := uuid.New(), uuid.New()
+ sourceID := uuid.New()
+ span := canonicalSpan(project, trace, sourceID, nil)
+ span.StartTime = time.Unix(1789747200, 123456789).UTC()
+ attrs := []*commonpb.KeyValue{
+ {Key: "large", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: 9223372036854775807}}},
+ {Key: "enabled", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: true}}},
+ {Key: "bytes", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BytesValue{BytesValue: []byte{0, 255}}}},
+ {Key: "nested", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_KvlistValue{KvlistValue: &commonpb.KeyValueList{Values: []*commonpb.KeyValue{{Key: "array", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: []*commonpb.AnyValue{{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: 1.25}}}}}}}}}}}},
+ }
+ source := &tracepb.Span{TraceId: trace[:], SpanId: sourceID[8:], Name: span.Name, StartTimeUnixNano: uint64(span.StartTime.UnixNano()), EndTimeUnixNano: uint64(span.StartTime.Add(span.Duration).UnixNano()), TraceState: "vendor=value", Flags: 0xffffffff, Attributes: attrs, Status: &tracepb.Status{Code: tracepb.Status_STATUS_CODE_ERROR, Message: "detail"}, DroppedAttributesCount: 1, DroppedEventsCount: 2, DroppedLinksCount: 3,
+ Events: []*tracepb.Span_Event{{Name: "exception", TimeUnixNano: uint64(span.StartTime.UnixNano() + 1), Attributes: attrs, DroppedAttributesCount: 4}},
+ Links: []*tracepb.Span_Link{{TraceId: trace[:], SpanId: sourceID[8:], TraceState: "other=value", Flags: 0xffffffff, Attributes: attrs, DroppedAttributesCount: 5}}}
+ span.OTLP = source
+ span.Context = &tracepb.ResourceSpans{Resource: &resourcepb.Resource{Attributes: attrs, DroppedAttributesCount: 6}, SchemaUrl: "resource-schema", ScopeSpans: []*tracepb.ScopeSpans{{Scope: &commonpb.InstrumentationScope{Name: "scope", Version: "v1", Attributes: attrs, DroppedAttributesCount: 7}, SchemaUrl: "scope-schema"}}}
+ // Unknown protobuf fields must survive even though the query columns only expose known fields.
+ source.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x01})
+ span.Context.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x02})
+ span.Context.ScopeSpans[0].ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x03})
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span}); err != nil {
+ t.Fatal(err)
+ }
+ var attrJSON, eventsJSON, linksJSON, resourceJSON, scopeJSON, traceState, message, resourceSchema, scopeSchema, scopeVersion string
+ var flags, attrDrops, eventDrops, linkDrops int64
+ if err := db.TelemetryDB.QueryRow(`SELECT span_attributes,events,links,resource,scope,trace_state,flags,status_message,resource_schema_url,scope_schema_url,scope_version,dropped_attributes_count,dropped_events_count,dropped_links_count FROM spans_v2`).Scan(&attrJSON, &eventsJSON, &linksJSON, &resourceJSON, &scopeJSON, &traceState, &flags, &message, &resourceSchema, &scopeSchema, &scopeVersion, &attrDrops, &eventDrops, &linkDrops); err != nil {
+ t.Fatal(err)
+ }
+ // The SQL-visible columns are the readable form every backend stores. Exact types live in the payload, checked below.
+ var attributes map[string]string
+ if err := json.Unmarshal([]byte(attrJSON), &attributes); err != nil {
+ t.Fatal(err)
+ }
+ if want := map[string]string{"large": "9223372036854775807", "enabled": "true", "bytes": "AP8=", "nested": `{"array":[1.25]}`}; !reflect.DeepEqual(attributes, want) {
+ t.Fatalf("span_attributes:\nwant %v\ngot %v", want, attributes)
+ }
+ var events []struct {
+ Name string `json:"name"`
+ Time uint64 `json:"time_unix_nano"`
+ Attributes map[string]any `json:"attributes"`
+ Dropped uint32 `json:"dropped_attributes_count"`
+ }
+ var links []struct {
+ TraceId string `json:"trace_id"`
+ SpanId string `json:"span_id"`
+ TraceState string `json:"trace_state"`
+ Flags uint32 `json:"flags"`
+ Attributes map[string]any `json:"attributes"`
+ Dropped uint32 `json:"dropped_attributes_count"`
+ }
+ if err := json.Unmarshal([]byte(eventsJSON), &events); err != nil {
+ t.Fatal(err)
+ }
+ if err := json.Unmarshal([]byte(linksJSON), &links); err != nil {
+ t.Fatal(err)
+ }
+ if len(events) != 1 || events[0].Name != "exception" || events[0].Time != uint64(span.StartTime.UnixNano()+1) || events[0].Dropped != 4 || events[0].Attributes["enabled"] != true {
+ t.Fatalf("events column changed: %s", eventsJSON)
+ }
+ if len(links) != 1 || links[0].TraceId != hex.EncodeToString(trace[:]) || links[0].SpanId != hex.EncodeToString(sourceID[8:]) || links[0].TraceState != "other=value" || links[0].Flags != 0xffffffff || links[0].Dropped != 5 {
+ t.Fatalf("links column changed: %s", linksJSON)
+ }
+ var resource, scope struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+ Attributes map[string]any `json:"attributes"`
+ Dropped uint32 `json:"droppedAttributesCount"`
+ }
+ if err := json.Unmarshal([]byte(resourceJSON), &resource); err != nil {
+ t.Fatal(err)
+ }
+ if err := json.Unmarshal([]byte(scopeJSON), &scope); err != nil {
+ t.Fatal(err)
+ }
+ if len(resource.Attributes) != 4 || resource.Dropped != 6 || scope.Name != "scope" || scope.Version != "v1" || len(scope.Attributes) != 4 || scope.Dropped != 7 {
+ t.Fatalf("resource/scope columns changed:\n%s\n%s", resourceJSON, scopeJSON)
+ }
+ if traceState != "vendor=value" || flags != 0xffffffff || message != "detail" || resourceSchema != "resource-schema" || scopeSchema != "scope-schema" || scopeVersion != "v1" || attrDrops != 1 || eventDrops != 2 || linkDrops != 3 {
+ t.Fatal("scalar OTLP fields changed")
+ }
+ payload, err := OtelSpanRepository.FindOTLP(ctx, project, span.TraceId, span.SpanId, span.StartTime)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var restored tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &restored); err != nil {
+ t.Fatal(err)
+ }
+ expected := proto.Clone(span.Context).(*tracepb.ResourceSpans)
+ expected.ScopeSpans[0].Spans = []*tracepb.Span{source}
+ if !proto.Equal(expected, &restored) {
+ t.Fatal("inline OTLP round trip changed source data")
+ }
+ // A span of the native protocol has no OTLP source. It is stored in the same table and read the same way.
+ run := uuid.New()
+ native := makeSpan(project, run, "native", span.StartTime, time.Second)
+ native.SpanKind, native.StatusCode = 3, 2
+ if err := SpanRepository.InsertAsync(ctx, []models.Span{native}); err != nil {
+ t.Fatal(err)
+ }
+ stored, err := findRunSpans(ctx, project, run, &span.StartTime)
+ if err != nil || len(stored) != 1 || stored[0].SpanId != native.SpanId || stored[0].SpanKind != 3 || stored[0].StatusCode != 2 {
+ t.Fatalf("native span: %+v %v", stored, err)
+ }
+ synthesized, err := OtelSpanRepository.FindOTLP(ctx, project, native.TraceId, native.SpanId, span.StartTime)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var nativePayload tracepb.ResourceSpans
+ if err := proto.Unmarshal(synthesized, &nativePayload); err != nil || nativePayload.ScopeSpans[0].Spans[0].Name != "native" {
+ t.Fatalf("a native span downloads as OTLP too: %v %v", &nativePayload, err)
+ }
+ // No Endpoint, Task, or AI row exists: source lookup still returns the complete trace.
+ found, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: project, TraceId: hex.EncodeToString(trace[:]), RecordedAt: &span.StartTime}}, 0)
+ if err != nil || len(found) != 1 || !found[0].StartTime.Equal(span.StartTime) {
+ t.Fatalf("source lookup or nanosecond precision failed: %+v %v", found, err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/duckdb/ai_trace.repository.go b/backend/app/repositories/telemetry/duckdb/ai_trace.repository.go
index 0c62c31a..efbed396 100644
--- a/backend/app/repositories/telemetry/duckdb/ai_trace.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/ai_trace.repository.go
@@ -21,41 +21,43 @@ import (
type aiTraceRowNaming struct{ lit.DefaultDbNamingStrategy }
func (aiTraceRowNaming) GetTableNameFromStructName(string) string {
- return "ai_traces"
+ return "ai_traces_v2"
}
type aiTraceRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- Duration int64 `lit:"duration"`
- StatusCode uint8 `lit:"status_code"`
- Model string `lit:"model"`
- ResponseModel string `lit:"response_model"`
- Provider string `lit:"provider"`
- Operation string `lit:"operation"`
- InputTokens int64 `lit:"input_tokens"`
- OutputTokens int64 `lit:"output_tokens"`
- TotalTokens int64 `lit:"total_tokens"`
- CachedTokens int64 `lit:"cached_tokens"`
- ReasoningTokens int64 `lit:"reasoning_tokens"`
- InputCost float64 `lit:"input_cost"`
- OutputCost float64 `lit:"output_cost"`
- TotalCost float64 `lit:"total_cost"`
- TraceName string `lit:"trace_name"`
- UserId string `lit:"user_id"`
- FinishReason string `lit:"finish_reason"`
- ServerName string `lit:"server_name"`
- AppVersion string `lit:"app_version"`
- StorageKey string `lit:"storage_key"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- IsRoot bool `lit:"is_root"`
- ConversationId string `lit:"conversation_id"`
- ToolCallCount int64 `lit:"tool_call_count"`
- ToolNames string `lit:"tool_names"`
- Flagged bool `lit:"flagged"`
- FlaggedTerms string `lit:"flagged_terms"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ Duration int64 `lit:"duration"`
+ StatusCode uint8 `lit:"status_code"`
+ Model string `lit:"model"`
+ ResponseModel string `lit:"response_model"`
+ Provider string `lit:"provider"`
+ Operation string `lit:"operation"`
+ InputTokens int64 `lit:"input_tokens"`
+ OutputTokens int64 `lit:"output_tokens"`
+ TotalTokens int64 `lit:"total_tokens"`
+ CachedTokens int64 `lit:"cached_tokens"`
+ ReasoningTokens int64 `lit:"reasoning_tokens"`
+ InputCost float64 `lit:"input_cost"`
+ OutputCost float64 `lit:"output_cost"`
+ TotalCost float64 `lit:"total_cost"`
+ TraceName string `lit:"trace_name"`
+ UserId string `lit:"user_id"`
+ FinishReason string `lit:"finish_reason"`
+ ServerName string `lit:"server_name"`
+ AppVersion string `lit:"app_version"`
+ StorageKey string `lit:"storage_key"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ ParentSpanId string `lit:"parent_span_id"`
+ IsRoot bool `lit:"is_root"`
+ ConversationId string `lit:"conversation_id"`
+ ToolCallCount int64 `lit:"tool_call_count"`
+ ToolNames string `lit:"tool_names"`
+ Flagged bool `lit:"flagged"`
+ FlaggedTerms string `lit:"flagged_terms"`
}
type groupedAiTraceRow struct {
@@ -136,36 +138,38 @@ func init() {
func (r *aiTraceRow) toModel() models.AiTrace {
t := models.AiTrace{
- Id: r.Id,
- ProjectId: r.ProjectId,
- RecordedAt: r.RecordedAt.Time,
- Duration: time.Duration(r.Duration),
- StatusCode: r.StatusCode,
- Model: r.Model,
- ResponseModel: r.ResponseModel,
- Provider: r.Provider,
- Operation: r.Operation,
- InputTokens: r.InputTokens,
- OutputTokens: r.OutputTokens,
- TotalTokens: r.TotalTokens,
- CachedTokens: r.CachedTokens,
- ReasoningTokens: r.ReasoningTokens,
- InputCost: r.InputCost,
- OutputCost: r.OutputCost,
- TotalCost: r.TotalCost,
- TraceName: r.TraceName,
- UserId: r.UserId,
- FinishReason: r.FinishReason,
- ServerName: r.ServerName,
- AppVersion: r.AppVersion,
- StorageKey: r.StorageKey,
- DistributedTraceId: r.DistributedTraceId,
- IsRoot: r.IsRoot,
- ConversationId: r.ConversationId,
- ToolCallCount: r.ToolCallCount,
- ToolNames: shared.SplitCSV(r.ToolNames),
- Flagged: r.Flagged,
- FlaggedTerms: shared.SplitCSV(r.FlaggedTerms),
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ RecordedAt: r.RecordedAt.Time,
+ Duration: time.Duration(r.Duration),
+ StatusCode: r.StatusCode,
+ Model: r.Model,
+ ResponseModel: r.ResponseModel,
+ Provider: r.Provider,
+ Operation: r.Operation,
+ InputTokens: r.InputTokens,
+ OutputTokens: r.OutputTokens,
+ TotalTokens: r.TotalTokens,
+ CachedTokens: r.CachedTokens,
+ ReasoningTokens: r.ReasoningTokens,
+ InputCost: r.InputCost,
+ OutputCost: r.OutputCost,
+ TotalCost: r.TotalCost,
+ TraceName: r.TraceName,
+ UserId: r.UserId,
+ FinishReason: r.FinishReason,
+ ServerName: r.ServerName,
+ AppVersion: r.AppVersion,
+ StorageKey: r.StorageKey,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ ParentSpanId: r.ParentSpanId,
+ IsRoot: r.IsRoot,
+ ConversationId: r.ConversationId,
+ ToolCallCount: r.ToolCallCount,
+ ToolNames: shared.SplitCSV(r.ToolNames),
+ Flagged: r.Flagged,
+ FlaggedTerms: shared.SplitCSV(r.FlaggedTerms),
}
if r.Attributes != nil {
t.Attributes = map[string]string(r.Attributes)
@@ -180,27 +184,17 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTr
return nil
}
- return withAppender(ctx, "ai_traces", func(appender *duckdb.Appender) {
+ return withAppenderColumns(ctx, "ai_traces_v2", strings.Split("id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes, is_root, trace_id, span_id, parent_span_id, conversation_id, tool_call_count, tool_names, flagged, flagged_terms", ", "), func(appender *duckdb.Appender) {
for _, t := range lines {
attributesJSON, err := attrJSON(t.Attributes)
if err != nil {
- captureDroppedRow("ai_traces", err)
+ captureDroppedRow("ai_traces_v2", err)
continue
}
- var distributedTraceId *string
- if t.DistributedTraceId != nil {
- s := t.DistributedTraceId.String()
- distributedTraceId = &s
- }
-
isRoot := boolToInt(t.IsRoot)
- // Column order follows the ai_traces DDL exactly: is_root precedes
- // distributed_trace_id, and the 0002 migration's conversation columns
- // (conversation_id, tool_call_count, tool_names, flagged, flagged_terms)
- // come last in that order.
if err := appender.AppendRow(
t.Id.String(),
t.ProjectId.String(),
@@ -227,14 +221,16 @@ func (r *aiTraceRepository) InsertAsync(ctx context.Context, lines []models.AiTr
t.StorageKey,
attributesJSON,
isRoot,
- nullableString(distributedTraceId),
+ t.TraceId,
+ t.SpanId,
+ t.ParentSpanId,
t.ConversationId,
t.ToolCallCount,
shared.JoinCSV(t.ToolNames),
boolToInt(t.Flagged),
shared.JoinCSV(t.FlaggedTerms),
); err != nil {
- captureDroppedRow("ai_traces", err)
+ captureDroppedRow("ai_traces_v2", err)
}
}
@@ -252,7 +248,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
whereClause += shared.RootFilterClause("is_root", rootFilter)
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(DISTINCT trace_name) AS count FROM ai_traces WHERE "+whereClause, params)
+ "SELECT COUNT(DISTINCT trace_name) AS count FROM ai_traces_v2 WHERE "+whereClause, params)
if err != nil {
return nil, 0, err
}
@@ -273,7 +269,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
MAX(recorded_at) AS last_seen,
MAX(is_root) AS has_root,
MAX(CASE WHEN is_root = 0 THEN 1 ELSE 0 END) AS has_non_root
- FROM ai_traces WHERE `+whereClause+`
+ FROM ai_traces_v2 WHERE `+whereClause+`
GROUP BY trace_name`, params)
if err != nil {
return nil, 0, err
@@ -335,7 +331,7 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.
params := lit.P{"project_id": projectId, "trace_name": traceName, "from": fromDate.UTC(), "to": toDate.UTC()}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM ai_traces WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM ai_traces_v2 WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -365,9 +361,9 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
lit.P{"project_id": projectId, "trace_name": traceName, "from": fromDate.UTC(), "to": toDate.UTC(), "limit": pageSize, "offset": offset})
@@ -400,7 +396,7 @@ func (r *aiTraceRepository) GetTraceNameStats(ctx context.Context, projectId uui
COALESCE(SUM(total_cost), 0) AS total_cost,
COALESCE(AVG(input_tokens), 0) AS avg_input_tokens,
COALESCE(AVG(output_tokens), 0) AS avg_output_tokens
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
@@ -429,9 +425,9 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": traceId}
if recordedAt != nil {
@@ -453,11 +449,12 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
return &result, nil
}
-func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.AiTrace, error) {
- if len(projectIds) == 0 {
+func (r *aiTraceRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.AiTrace, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
@@ -469,9 +466,9 @@ func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distri
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ FROM ai_traces_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -499,7 +496,7 @@ func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distri
&row.InputTokens, &row.OutputTokens, &row.TotalTokens, &row.CachedTokens, &row.ReasoningTokens,
&row.InputCost, &row.OutputCost, &row.TotalCost,
&row.TraceName, &row.UserId, &row.FinishReason, &row.ServerName, &row.AppVersion,
- &row.StorageKey, &row.Attributes, &row.DistributedTraceId, &row.IsRoot,
+ &row.StorageKey, &row.Attributes, &row.TraceId, &row.SpanId, &row.ParentSpanId, &row.IsRoot,
&row.ConversationId, &row.ToolCallCount, &row.ToolNames, &row.Flagged, &row.FlaggedTerms,
); err != nil {
return nil, err
@@ -539,7 +536,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
whereClause := baseWhere
if len(rowPredicates) > 0 {
- whereClause += " AND conversation_id IN (SELECT DISTINCT conversation_id FROM ai_traces WHERE " +
+ whereClause += " AND conversation_id IN (SELECT DISTINCT conversation_id FROM ai_traces_v2 WHERE " +
baseWhere + " AND " + strings.Join(rowPredicates, " AND ") + ")"
}
@@ -561,7 +558,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
string_agg(DISTINCT flagged_terms, ',') AS flagged_terms,
MIN(recorded_at) AS first_seen,
MAX(recorded_at) AS last_seen
- FROM ai_traces WHERE `+whereClause+`
+ FROM ai_traces_v2 WHERE `+whereClause+`
GROUP BY conversation_id`+havingClause, params)
if err != nil {
return nil, 0, nil, err
@@ -597,9 +594,9 @@ func (r *aiTraceRepository) FindByConversationId(ctx context.Context, projectId
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND conversation_id = :conversation_id`
params := lit.P{"project_id": projectId, "conversation_id": conversationId}
if !fromDate.IsZero() {
@@ -640,7 +637,7 @@ func (r *aiTraceRepository) FindUserStats(ctx context.Context, projectId uuid.UU
CAST(SUM(total_tokens) AS BIGINT) AS conv_tokens,
MAX(flagged) AS conv_flagged,
MAX(recorded_at) AS last_seen
- FROM ai_traces WHERE `+whereClause+`
+ FROM ai_traces_v2 WHERE `+whereClause+`
GROUP BY user_id, conversation_id`, params)
if err != nil {
return nil, 0, err
@@ -678,7 +675,7 @@ func (r *aiTraceRepository) GetConversationCosts(ctx context.Context, projectId
}
rows, err := lit.SelectNamed[conversationCostRow](db.TelemetryDB,
`SELECT conversation_id, SUM(total_cost) AS total_cost
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND recorded_at >= :since AND conversation_id IN (`+strings.Join(placeholders, ",")+`)
GROUP BY conversation_id`, params)
if err != nil {
@@ -692,7 +689,7 @@ func (r *aiTraceRepository) GetConversationCosts(ctx context.Context, projectId
func (r *aiTraceRepository) ListModels(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time) ([]string, error) {
rows, err := lit.SelectNamed[modelNameRow](db.TelemetryDB,
- `SELECT DISTINCT model FROM ai_traces
+ `SELECT DISTINCT model FROM ai_traces_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND model != ''
ORDER BY model LIMIT 200`,
lit.P{"project_id": projectId, "from": fromDate.UTC(), "to": toDate.UTC()})
@@ -708,7 +705,7 @@ func (r *aiTraceRepository) ListModels(ctx context.Context, projectId uuid.UUID,
func (r *aiTraceRepository) ListToolNames(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time) ([]string, error) {
rows, err := lit.SelectNamed[toolNamesRow](db.TelemetryDB,
- `SELECT DISTINCT tool_names FROM ai_traces
+ `SELECT DISTINCT tool_names FROM ai_traces_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND tool_names != ''
LIMIT 500`,
lit.P{"project_id": projectId, "from": fromDate.UTC(), "to": toDate.UTC()})
diff --git a/backend/app/repositories/telemetry/duckdb/appender_test.go b/backend/app/repositories/telemetry/duckdb/appender_test.go
new file mode 100644
index 00000000..4cbda240
--- /dev/null
+++ b/backend/app/repositories/telemetry/duckdb/appender_test.go
@@ -0,0 +1,80 @@
+//go:build telemetry_duckdb
+
+package duckdb
+
+import (
+ "context"
+ "database/sql/driver"
+ "testing"
+
+ "github.com/duckdb/duckdb-go/v2"
+ "github.com/tracewayapp/traceway/backend/app/db"
+)
+
+func openAppenderTestTable(t *testing.T) driver.Conn {
+ t.Helper()
+ connector, err := duckdb.NewConnector("", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ previous := db.DuckDBConnector
+ db.DuckDBConnector = connector
+ conn, err := connector.Connect(context.Background())
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() {
+ conn.Close()
+ connector.Close()
+ db.DuckDBConnector = previous
+ })
+ if _, err := conn.(driver.ExecerContext).ExecContext(context.Background(), "CREATE TABLE probe (id VARCHAR, n BIGINT)", nil); err != nil {
+ t.Fatal(err)
+ }
+ return conn
+}
+
+func probeIds(t *testing.T, conn driver.Conn) string {
+ t.Helper()
+ rows, err := conn.(driver.QueryerContext).QueryContext(context.Background(), "SELECT coalesce(string_agg(id, ',' ORDER BY id), '') FROM probe", nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ values := make([]driver.Value, 1)
+ if err := rows.Next(values); err != nil {
+ t.Fatal(err)
+ }
+ return values[0].(string)
+}
+
+func TestAppenderKeepsRowsAroundARejectedOne(t *testing.T) {
+ conn := openAppenderTestTable(t)
+ var rejected error
+ err := withAppenderColumns(context.Background(), "probe", []string{"id", "n"}, func(appender *duckdb.Appender) {
+ if err := appender.AppendRow("first", int64(1)); err != nil {
+ t.Fatal(err)
+ }
+ rejected = appender.AppendRow("bad", "not a number")
+ if err := appender.AppendRow("last", int64(3)); err != nil {
+ t.Fatalf("the appender must stay usable after rejecting a row: %v", err)
+ }
+ })
+ if err != nil || rejected == nil {
+ t.Fatalf("flush error %v, rejected row error %v", err, rejected)
+ }
+ if ids := probeIds(t, conn); ids != "first,last" {
+ t.Fatalf("rows around a rejected one must persist, got %q", ids)
+ }
+}
+
+func TestAppenderFailureIsABatchFailure(t *testing.T) {
+ openAppenderTestTable(t)
+ _, _, before := db.GetTelemetryIngestCounters()
+ ran := false
+ err := withAppenderColumns(context.Background(), "missing_table", []string{"id"}, func(*duckdb.Appender) { ran = true })
+ _, _, after := db.GetTelemetryIngestCounters()
+ if err == nil || ran || after != before+1 {
+ t.Fatalf("a batch that cannot start must fail and be counted: err=%v ran=%v failures %d -> %d", err, ran, before, after)
+ }
+}
diff --git a/backend/app/repositories/telemetry/duckdb/endpoint.repository.go b/backend/app/repositories/telemetry/duckdb/endpoint.repository.go
index f47e01f8..1cbaf391 100644
--- a/backend/app/repositories/telemetry/duckdb/endpoint.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/endpoint.repository.go
@@ -21,21 +21,22 @@ import (
)
type endpoint struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- Endpoint string `lit:"endpoint"`
- Duration int64 `lit:"duration"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- StatusCode int16 `lit:"status_code"`
- BodySize int32 `lit:"body_size"`
- ClientIP string `lit:"client_ip"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- SpanId *uuid.UUID `lit:"span_id"`
- IsStream bool `lit:"is_stream"`
- IsRoot bool `lit:"is_root"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ Endpoint string `lit:"endpoint"`
+ Duration int64 `lit:"duration"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ StatusCode int16 `lit:"status_code"`
+ BodySize int32 `lit:"body_size"`
+ ClientIP string `lit:"client_ip"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ ParentSpanId string `lit:"parent_span_id"`
+ IsStream bool `lit:"is_stream"`
+ IsRoot bool `lit:"is_root"`
}
type groupedEndpointRow struct {
@@ -90,20 +91,21 @@ func init() {
func (r *endpoint) toModel() models.Endpoint {
e := models.Endpoint{
- Id: r.Id,
- ProjectId: r.ProjectId,
- Endpoint: r.Endpoint,
- Duration: time.Duration(r.Duration),
- RecordedAt: r.RecordedAt.Time,
- StatusCode: r.StatusCode,
- BodySize: r.BodySize,
- ClientIP: r.ClientIP,
- AppVersion: r.AppVersion,
- ServerName: r.ServerName,
- DistributedTraceId: r.DistributedTraceId,
- SpanId: r.SpanId,
- IsStream: r.IsStream,
- IsRoot: r.IsRoot,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ Endpoint: r.Endpoint,
+ Duration: time.Duration(r.Duration),
+ RecordedAt: r.RecordedAt.Time,
+ StatusCode: r.StatusCode,
+ BodySize: r.BodySize,
+ ClientIP: r.ClientIP,
+ AppVersion: r.AppVersion,
+ ServerName: r.ServerName,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ ParentSpanId: r.ParentSpanId,
+ IsStream: r.IsStream,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
e.Attributes = map[string]string(r.Attributes)
@@ -121,26 +123,15 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
return nil
}
- return withAppender(ctx, "endpoints", func(appender *duckdb.Appender) {
+ return withAppenderColumns(ctx, "endpoints_v2", strings.Split("id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_stream, is_root", ", "), func(appender *duckdb.Appender) {
for _, ep := range lines {
attributesJSON, err := attrJSON(ep.Attributes)
if err != nil {
- captureDroppedRow("endpoints", err)
+ captureDroppedRow("endpoints_v2", err)
continue
}
- var distributedTraceId *string
- if ep.DistributedTraceId != nil {
- v := ep.DistributedTraceId.String()
- distributedTraceId = &v
- }
- var spanId *string
- if ep.SpanId != nil {
- v := ep.SpanId.String()
- spanId = &v
- }
-
isStream := boolToInt(ep.IsStream)
isRoot := boolToInt(ep.IsRoot)
@@ -156,12 +147,13 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
attributesJSON,
ep.AppVersion,
ep.ServerName,
- nullableString(distributedTraceId),
- nullableString(spanId),
+ ep.TraceId,
+ ep.SpanId,
+ ep.ParentSpanId,
isStream,
isRoot,
); err != nil {
- captureDroppedRow("endpoints", err)
+ captureDroppedRow("endpoints_v2", err)
}
}
@@ -170,7 +162,7 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
func (e *endpointRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
result, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
return 0, err
@@ -185,7 +177,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
params := lit.P{"project_id": projectId, "from": fromDate.UTC(), "to": toDate.UTC()}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -203,8 +195,8 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
}
rows, err := lit.SelectNamed[endpoint](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s DESC LIMIT :limit OFFSET :offset`, orderBy),
lit.P{"project_id": projectId, "from": fromDate.UTC(), "to": toDate.UTC(), "limit": pageSize, "offset": offset})
if err != nil {
@@ -256,7 +248,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
quantile_cont(e.duration, 0.50) FILTER (WHERE e.is_stream = 0) AS p50,
quantile_cont(e.duration, 0.95) FILTER (WHERE e.is_stream = 0) AS p95,
quantile_cont(e.duration, 0.99) FILTER (WHERE e.is_stream = 0) AS p99
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE ` + whereClause + `
GROUP BY e.endpoint, s.offset_ms`
@@ -351,7 +343,7 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
params := lit.P{"project_id": projectId, "endpoint": endpointName, "from": fromDate.UTC(), "to": toDate.UTC()}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -374,8 +366,8 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
}
rows, err := lit.SelectNamed[endpoint](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
lit.P{"project_id": projectId, "endpoint": endpointName, "from": fromDate.UTC(), "to": toDate.UTC(), "limit": pageSize, "offset": offset})
if err != nil {
@@ -391,8 +383,8 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
}
func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId uuid.UUID, recordedAt *time.Time) (*models.Endpoint, error) {
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id
- FROM endpoints WHERE project_id = :project_id AND id = :id`
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root, is_stream
+ FROM endpoints_v2 WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": endpointId}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -416,7 +408,7 @@ func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId
func (e *endpointRepository) CountByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime(recorded_at, '%Y-%m-%d %H:00:00') as bucket, CAST(COUNT(*) AS DOUBLE) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -428,7 +420,7 @@ func (e *endpointRepository) CountByHour(ctx context.Context, projectId uuid.UUI
func (e *endpointRepository) AvgDurationByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime(recorded_at, '%Y-%m-%d %H:00:00') as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -441,7 +433,7 @@ func (e *endpointRepository) ErrorRateByHour(ctx context.Context, projectId uuid
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime(recorded_at, '%Y-%m-%d %H:00:00') as bucket,
SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -453,7 +445,7 @@ func (e *endpointRepository) ErrorRateByHour(ctx context.Context, projectId uuid
func (e *endpointRepository) CountByInterval(ctx context.Context, projectId uuid.UUID, start, end time.Time, intervalMinutes int) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT strftime(%s, '%%Y-%%m-%%d %%H:%%M:%%S') as bucket, CAST(COUNT(*) AS DOUBLE) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, timeBucketExpr("recorded_at", intervalMinutes*60)),
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -465,7 +457,7 @@ func (e *endpointRepository) CountByInterval(ctx context.Context, projectId uuid
func (e *endpointRepository) AvgDurationByInterval(ctx context.Context, projectId uuid.UUID, start, end time.Time, intervalMinutes int) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT strftime(%s, '%%Y-%%m-%%d %%H:%%M:%%S') as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
GROUP BY bucket ORDER BY bucket ASC`, timeBucketExpr("recorded_at", intervalMinutes*60)),
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -478,7 +470,7 @@ func (e *endpointRepository) ErrorRateByInterval(ctx context.Context, projectId
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT strftime(%s, '%%Y-%%m-%%d %%H:%%M:%%S') as bucket,
SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, timeBucketExpr("recorded_at", intervalMinutes*60)),
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -505,7 +497,7 @@ func (e *endpointRepository) FindWorstEndpoints(ctx context.Context, projectId u
quantile_cont(e.duration, 0.50) FILTER (WHERE e.is_stream = 0) AS p50,
quantile_cont(e.duration, 0.95) FILTER (WHERE e.is_stream = 0) AS p95,
quantile_cont(e.duration, 0.99) FILTER (WHERE e.is_stream = 0) AS p99
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE e.project_id = :project_id AND e.recorded_at >= :from AND e.recorded_at <= :to AND e.is_stream = 0
GROUP BY e.endpoint, s.offset_ms`
@@ -583,7 +575,7 @@ func (e *endpointRepository) GetEndpointStats(ctx context.Context, projectId uui
CASE WHEN COUNT(*) > 0 THEN SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) ELSE 0 END as error_rate,
COALESCE(SUM(CASE WHEN duration <= 500000000 AND status_code < 500 THEN 1 ELSE 0 END) +
SUM(CASE WHEN duration > 500000000 AND duration <= 2000000000 AND status_code < 500 THEN 1 ELSE 0 END) * 0.5, 0) as satisfied_tolerating
- FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
+ FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
return nil, err
@@ -593,7 +585,7 @@ func (e *endpointRepository) GetEndpointStats(ctx context.Context, projectId uui
}
isStreamRow, err := lit.SelectSingleNamed[isStreamFlagRow](db.TelemetryDB,
- `SELECT COALESCE(MAX(is_stream), 0) as is_stream FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
+ `SELECT COALESCE(MAX(is_stream), 0) as is_stream FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
return nil, err
@@ -614,7 +606,7 @@ func (e *endpointRepository) GetEndpointStats(ctx context.Context, projectId uui
pctQuery, pctArgs, err := lit.ParseNamedQuery(db.Driver,
`SELECT quantile_cont(duration, 0.5) AS p50, quantile_cont(duration, 0.95) AS p95, quantile_cont(duration, 0.99) AS p99
- FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`,
+ FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`,
params)
if err != nil {
return nil, err
@@ -675,7 +667,7 @@ func (e *endpointRepository) GetEndpointStackedChart(ctx context.Context, projec
%s as bucket,
%s as endpoint_category,
%s as metric_value
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0%s
GROUP BY bucket, endpoint_category
ORDER BY bucket ASC, endpoint_category ASC`, timeBucketExpr("recorded_at", secs), caseExpr, metricExpr, filterClause)
@@ -743,7 +735,7 @@ func (e *endpointRepository) getStackedChartWithPercentiles(ctx context.Context,
%s as bucket,
%s as endpoint_category,
quantile_cont(duration, %g) / 1000000.0 as metric_value
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0%s
GROUP BY bucket, endpoint_category
ORDER BY bucket ASC, endpoint_category ASC`, timeBucketExpr("recorded_at", bucketSecs), caseExpr, percentile, filterClause)
@@ -852,7 +844,7 @@ func getTopEndpointsByMetric(ctx context.Context, projectId uuid.UUID, from, to
if metricType == "total_time" {
results, err := lit.SelectNamed[endpointMetricRow](db.TelemetryDB,
`SELECT endpoint, COUNT(*) * AVG(duration) / 1000000.0 as metric_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`+filterClause+`
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`+filterClause+`
GROUP BY endpoint ORDER BY metric_value DESC LIMIT 5`,
params)
if err != nil {
@@ -876,7 +868,7 @@ func getTopEndpointsByMetric(ctx context.Context, projectId uuid.UUID, from, to
results, err := lit.SelectNamed[endpointMetricRow](db.TelemetryDB,
fmt.Sprintf(`SELECT endpoint, quantile_cont(duration, %g) / 1000000.0 as metric_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0%s
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0%s
GROUP BY endpoint ORDER BY metric_value DESC LIMIT 5`, percentile, filterClause),
params)
if err != nil {
@@ -890,19 +882,20 @@ func getTopEndpointsByMetric(ctx context.Context, projectId uuid.UUID, from, to
return endpoints, nil
}
-func (e *endpointRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Endpoint, error) {
- if len(projectIds) == 0 {
+func (e *endpointRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Endpoint, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
placeholders[i] = ":" + key
params[key] = pid
}
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM endpoints_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -926,7 +919,7 @@ func (e *endpointRepository) FindByDistributedTraceId(ctx context.Context, distr
for sqlRows.Next() {
var row endpoint
if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.Endpoint, &row.Duration, &row.RecordedAt,
- &row.StatusCode, &row.BodySize, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.DistributedTraceId); err != nil {
+ &row.StatusCode, &row.BodySize, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.TraceId, &row.SpanId, &row.ParentSpanId, &row.IsRoot); err != nil {
return nil, err
}
endpoints = append(endpoints, row.toModel())
diff --git a/backend/app/repositories/telemetry/duckdb/exception_stack_trace.repository.go b/backend/app/repositories/telemetry/duckdb/exception_stack_trace.repository.go
index 206e3b2d..8f2edf45 100644
--- a/backend/app/repositories/telemetry/duckdb/exception_stack_trace.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/exception_stack_trace.repository.go
@@ -19,25 +19,25 @@ import (
)
type exceptionRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- TraceId *uuid.UUID `lit:"trace_id"`
- TraceType string `lit:"trace_type"`
- ExceptionHash string `lit:"exception_hash"`
- StackTrace string `lit:"stack_trace"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- IsMessage bool `lit:"is_message"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- SessionId *uuid.UUID `lit:"session_id"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ TraceType string `lit:"trace_type"`
+ ExceptionHash string `lit:"exception_hash"`
+ StackTrace string `lit:"stack_trace"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ IsMessage bool `lit:"is_message"`
+ SessionId *uuid.UUID `lit:"session_id"`
}
type exceptionRowNaming struct{ lit.DefaultDbNamingStrategy }
func (exceptionRowNaming) GetTableNameFromStructName(string) string {
- return "exception_stack_traces"
+ return "exceptions_v2"
}
// In DuckDB the aggregated timestamp/count columns come back as native
@@ -75,18 +75,18 @@ func init() {
func (r *exceptionRow) toModel() models.ExceptionStackTrace {
est := models.ExceptionStackTrace{
- Id: r.Id,
- ProjectId: r.ProjectId,
- TraceId: r.TraceId,
- TraceType: r.TraceType,
- ExceptionHash: r.ExceptionHash,
- StackTrace: r.StackTrace,
- RecordedAt: r.RecordedAt.Time,
- AppVersion: r.AppVersion,
- ServerName: r.ServerName,
- IsMessage: r.IsMessage,
- DistributedTraceId: r.DistributedTraceId,
- SessionId: r.SessionId,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ TraceType: r.TraceType,
+ ExceptionHash: r.ExceptionHash,
+ StackTrace: r.StackTrace,
+ RecordedAt: r.RecordedAt.Time,
+ AppVersion: r.AppVersion,
+ ServerName: r.ServerName,
+ IsMessage: r.IsMessage,
+ SessionId: r.SessionId,
}
if r.Attributes != nil {
est.Attributes = map[string]string(r.Attributes)
@@ -109,30 +109,15 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
return nil
}
- return withAppender(ctx, "exception_stack_traces", func(appender *duckdb.Appender) {
+ return withAppenderColumns(ctx, "exceptions_v2", strings.Split("id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id", ", "), func(appender *duckdb.Appender) {
for _, est := range lines {
- traceType := est.TraceType
- if traceType == "" {
- traceType = "endpoint"
- }
-
attributesJSON, err := attrJSON(est.Attributes)
if err != nil {
- captureDroppedRow("exception_stack_traces", err)
+ captureDroppedRow("exceptions_v2", err)
continue
}
- var traceId *string
- if est.TraceId != nil {
- v := est.TraceId.String()
- traceId = &v
- }
- var distributedTraceId *string
- if est.DistributedTraceId != nil {
- v := est.DistributedTraceId.String()
- distributedTraceId = &v
- }
var sessionId *string
if est.SessionId != nil {
v := est.SessionId.String()
@@ -144,8 +129,9 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
if err := appender.AppendRow(
est.Id.String(),
est.ProjectId.String(),
- nullableString(traceId),
- traceType,
+ est.TraceId,
+ est.SpanId,
+ est.TraceType,
est.ExceptionHash,
est.StackTrace,
est.RecordedAt.UTC(),
@@ -153,10 +139,9 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
est.AppVersion,
est.ServerName,
isMessage,
- nullableString(distributedTraceId),
nullableString(sessionId),
); err != nil {
- captureDroppedRow("exception_stack_traces", err)
+ captureDroppedRow("exceptions_v2", err)
}
}
@@ -165,7 +150,7 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
func (e *exceptionStackTraceRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
result, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM exception_stack_traces WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM exceptions_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
return 0, err
@@ -218,7 +203,7 @@ func (e *exceptionStackTraceRepository) FindGrouped(ctx context.Context, project
// COUNT(*) OVER () is evaluated on the post-GROUP BY/HAVING rows before
// LIMIT, so one scan yields both the page and the total group count.
fullQuery := `SELECT e.exception_hash, ANY_VALUE(e.stack_trace) as stack_trace, MAX(e.recorded_at) as last_seen, MIN(e.recorded_at) as first_seen, COUNT(*) as count, MAX(a.archived_at) as max_archived_at, COUNT(*) OVER () as total
- FROM exception_stack_traces e ` + archiveSubquery + `
+ FROM exceptions_v2 e ` + archiveSubquery + `
WHERE ` + whereClause + `
GROUP BY e.exception_hash` + havingClause + `
ORDER BY ` + orderBy + ` ` + sortDirection + `, e.exception_hash LIMIT :limit OFFSET :offset`
@@ -238,7 +223,7 @@ func (e *exceptionStackTraceRepository) FindGrouped(ctx context.Context, project
// total; only then fall back to a separate count scan.
countQuery := `SELECT COUNT(*) AS count FROM (
SELECT e.exception_hash, MAX(a.archived_at) as max_archived_at
- FROM exception_stack_traces e ` + archiveSubquery + `
+ FROM exceptions_v2 e ` + archiveSubquery + `
WHERE ` + whereClause + `
GROUP BY e.exception_hash` + havingClause + `) AS sub`
@@ -271,7 +256,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
groupRow, err := lit.SelectSingleNamed[exceptionGroupRow](db.TelemetryDB,
`SELECT exception_hash, ANY_VALUE(stack_trace) as stack_trace, MAX(recorded_at) as last_seen, MIN(recorded_at) as first_seen, COUNT(*) as count, NULL as max_archived_at
- FROM exception_stack_traces WHERE project_id = :project_id AND exception_hash = :exception_hash
+ FROM exceptions_v2 WHERE project_id = :project_id AND exception_hash = :exception_hash
GROUP BY exception_hash`,
params)
if err != nil {
@@ -290,8 +275,8 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
}
rows, err := lit.SelectNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND exception_hash = :exception_hash
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE project_id = :project_id AND exception_hash = :exception_hash
ORDER BY recorded_at DESC LIMIT :limit OFFSET :offset`,
lit.P{"project_id": projectId, "exception_hash": exceptionHash, "limit": pageSize, "offset": offset})
if err != nil {
@@ -309,7 +294,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
func (e *exceptionStackTraceRepository) CountByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[exceptionBucketRow](db.TelemetryDB,
`SELECT `+timeBucketExpr("recorded_at", 3600)+` as bucket, CAST(COUNT(*) AS DOUBLE) as agg_value
- FROM exception_stack_traces WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM exceptions_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -321,7 +306,7 @@ func (e *exceptionStackTraceRepository) CountByHour(ctx context.Context, project
func (e *exceptionStackTraceRepository) CountByInterval(ctx context.Context, projectId uuid.UUID, start, end time.Time, intervalMinutes int) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[exceptionBucketRow](db.TelemetryDB,
`SELECT `+timeBucketExpr("recorded_at", intervalMinutes*60)+` as bucket, CAST(COUNT(*) AS DOUBLE) as agg_value
- FROM exception_stack_traces WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM exceptions_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
@@ -344,7 +329,7 @@ func (e *exceptionStackTraceRepository) GetHourlyTrendForHashes(ctx context.Cont
}
query := `SELECT exception_hash, ` + timeBucketExpr("recorded_at", 3600) + ` as hour, COUNT(*) as count
- FROM exception_stack_traces
+ FROM exceptions_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
AND exception_hash IN (` + strings.Join(placeholders, ",") + `)
GROUP BY exception_hash, hour ORDER BY exception_hash, hour ASC`
@@ -430,24 +415,9 @@ func (e *exceptionStackTraceRepository) IsArchived(ctx context.Context, projectI
return result.Count > 0, nil
}
-func (e *exceptionStackTraceRepository) FindExceptionByTraceId(ctx context.Context, projectId uuid.UUID, traceId uuid.UUID) (*models.ExceptionStackTrace, error) {
- row, err := lit.SelectSingleNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND trace_id = :trace_id AND is_message = 0 LIMIT 1`,
- lit.P{"project_id": projectId, "trace_id": traceId})
- if err != nil {
- return nil, err
- }
- if row == nil {
- return nil, nil
- }
- est := row.toModel()
- return &est, nil
-}
-
-func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, projectId uuid.UUID, traceId uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND trace_id = :trace_id`
+func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, projectId uuid.UUID, traceId string, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE project_id = :project_id AND trace_id = :trace_id`
params := lit.P{"project_id": projectId, "trace_id": traceId}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -470,8 +440,8 @@ func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, pr
}
func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId uuid.UUID, id uuid.UUID, recordedAt *time.Time) (*models.ExceptionStackTrace, error) {
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND id = :id`
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": id}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -492,11 +462,12 @@ func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId
return &est, nil
}
-func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
- if len(projectIds) == 0 {
+func (e *exceptionStackTraceRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
@@ -504,8 +475,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
params[key] = pid
}
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `) AND is_message = 0`
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `) AND is_message = 0`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -528,8 +499,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
var results []models.ExceptionStackTrace
for sqlRows.Next() {
var row exceptionRow
- if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TraceId, &row.TraceType, &row.ExceptionHash, &row.StackTrace,
- &row.RecordedAt, &row.Attributes, &row.AppVersion, &row.ServerName, &row.IsMessage, &row.DistributedTraceId, &row.SessionId); err != nil {
+ if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TraceId, &row.SpanId, &row.TraceType, &row.ExceptionHash, &row.StackTrace,
+ &row.RecordedAt, &row.Attributes, &row.AppVersion, &row.ServerName, &row.IsMessage, &row.SessionId); err != nil {
return nil, err
}
results = append(results, row.toModel())
@@ -540,8 +511,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
// FindAllBySessionId returns all exceptions/messages stamped with the given session_id, ordered by time.
func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context, projectId, sessionId uuid.UUID) ([]models.ExceptionStackTrace, error) {
rows, err := lit.SelectNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2
WHERE project_id = :project_id AND session_id = :session_id
ORDER BY recorded_at ASC`,
lit.P{"project_id": projectId, "session_id": sessionId})
@@ -558,8 +529,8 @@ func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context,
// GetSessionIdForException returns the session_id for the given exception or nil when not linked.
func (e *exceptionStackTraceRepository) GetSessionIdForException(ctx context.Context, projectId, exceptionId uuid.UUID) (*uuid.UUID, error) {
row, err := lit.SelectSingleNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2
WHERE project_id = :project_id AND id = :id LIMIT 1`,
lit.P{"project_id": projectId, "id": exceptionId})
if err != nil || row == nil {
diff --git a/backend/app/repositories/telemetry/duckdb/helpers.go b/backend/app/repositories/telemetry/duckdb/helpers.go
index af14e424..ef925b33 100644
--- a/backend/app/repositories/telemetry/duckdb/helpers.go
+++ b/backend/app/repositories/telemetry/duckdb/helpers.go
@@ -76,6 +76,10 @@ func attrJSON(m map[string]string) (string, error) {
// Appender API is not reachable through database/sql, and Close flushes
// the appended rows.
func withAppender(ctx context.Context, table string, fn func(*duckdb.Appender)) error {
+ return withAppenderColumns(ctx, table, nil, fn)
+}
+
+func withAppenderColumns(ctx context.Context, table string, columns []string, fn func(*duckdb.Appender)) error {
conn, err := db.DuckDBConnector.Connect(ctx)
if err != nil {
db.RecordTelemetryInsertFailure()
@@ -83,7 +87,12 @@ func withAppender(ctx context.Context, table string, fn func(*duckdb.Appender))
}
defer conn.Close()
- appender, err := duckdb.NewAppenderFromConn(conn, "", table)
+ var appender *duckdb.Appender
+ if len(columns) == 0 {
+ appender, err = duckdb.NewAppenderFromConn(conn, "", table)
+ } else {
+ appender, err = duckdb.NewAppenderWithColumns(conn, "", "", table, columns)
+ }
if err != nil {
db.RecordTelemetryInsertFailure()
return err
diff --git a/backend/app/repositories/telemetry/duckdb/legacy.repository.go b/backend/app/repositories/telemetry/duckdb/legacy.repository.go
new file mode 100644
index 00000000..0018ff71
--- /dev/null
+++ b/backend/app/repositories/telemetry/duckdb/legacy.repository.go
@@ -0,0 +1,271 @@
+//go:build telemetry_duckdb
+
+package duckdb
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+// legacyRepository reads the tables V2 replaced, for the move-over. See shared/legacy.go for what the rows carry.
+type legacyRepository struct{}
+
+const legacyIdChunk = 500
+
+const (
+ legacyEndpointColumns = `id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name,
+ COALESCE(distributed_trace_id, '') AS trace_id, COALESCE(span_id, '') AS span_id, '' AS parent_span_id, is_stream, is_root`
+ legacyTaskColumns = `id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name,
+ COALESCE(distributed_trace_id, '') AS trace_id, COALESCE(span_id, '') AS span_id, '' AS parent_span_id, is_root`
+ legacyAiTraceColumns = `id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens,
+ total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version,
+ storage_key, attributes, COALESCE(distributed_trace_id, '') AS trace_id, '' AS span_id, '' AS parent_span_id, is_root,
+ conversation_id, tool_call_count, tool_names, flagged, flagged_terms`
+ legacyExceptionColumns = `id, project_id, COALESCE(trace_id, '') AS trace_id, '' AS span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes,
+ app_version, server_name, is_message, COALESCE(distributed_trace_id, ''), session_id`
+)
+
+// legacyWindow sorts only overfull one-second slices; see shared.LegacyPageOrder.
+func legacyWindow(columns, table string, from, to time.Time, limit int, offset *int) (string, lit.P) {
+ if offset != nil {
+ query, params := legacyWindow(columns, table, from, to, limit, nil)
+ params["offset"] = *offset
+ return strings.Replace(query, " LIMIT :limit", " ORDER BY "+shared.LegacyPageOrder(table, false)+" LIMIT :limit OFFSET :offset", 1), params
+ }
+ return "SELECT " + columns + " FROM " + table + " WHERE recorded_at >= :from AND recorded_at < :to LIMIT :limit",
+ lit.P{"from": from.UTC(), "to": to.UTC(), "limit": limit}
+}
+
+func legacyByIds(columns, table string, ids []uuid.UUID, from, to time.Time) (string, lit.P) {
+ params := lit.P{"from": from.UTC(), "to": to.UTC()}
+ names := make([]string, len(ids))
+ for i, id := range ids {
+ key := fmt.Sprintf("id_%d", i)
+ names[i], params[key] = ":"+key, id
+ }
+ return "SELECT " + columns + " FROM " + table + " WHERE id IN (" + strings.Join(names, ",") + ") AND recorded_at >= :from AND recorded_at < :to", params
+}
+
+func legacyEndpoints(ctx context.Context, query string, params lit.P) ([]models.Endpoint, error) {
+ rows, err := shared.SelectLegacy[endpoint](ctx, db.TelemetryDB, query, params)
+ if err != nil {
+ return nil, err
+ }
+ found := make([]models.Endpoint, len(rows))
+ for i, row := range rows {
+ found[i] = row.toModel()
+ }
+ return found, nil
+}
+
+func legacyTasks(ctx context.Context, query string, params lit.P) ([]models.Task, error) {
+ rows, err := shared.SelectLegacy[task](ctx, db.TelemetryDB, query, params)
+ if err != nil {
+ return nil, err
+ }
+ found := make([]models.Task, len(rows))
+ for i, row := range rows {
+ found[i] = row.toModel()
+ }
+ return found, nil
+}
+
+func legacyAiTraces(ctx context.Context, query string, params lit.P) ([]models.AiTrace, error) {
+ rows, err := shared.SelectLegacy[aiTraceRow](ctx, db.TelemetryDB, query, params)
+ if err != nil {
+ return nil, err
+ }
+ found := make([]models.AiTrace, len(rows))
+ for i, row := range rows {
+ found[i] = row.toModel()
+ }
+ return found, nil
+}
+
+func (legacyRepository) FindEndpoints(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Endpoint, error) {
+ query, params := legacyWindow(legacyEndpointColumns, "endpoints", from, to, limit, offset)
+ return legacyEndpoints(ctx, query, params)
+}
+
+func (legacyRepository) FindTasks(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Task, error) {
+ query, params := legacyWindow(legacyTaskColumns, "tasks", from, to, limit, offset)
+ return legacyTasks(ctx, query, params)
+}
+
+func (legacyRepository) FindAiTraces(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.AiTrace, error) {
+ query, params := legacyWindow(legacyAiTraceColumns, "ai_traces", from, to, limit, offset)
+ return legacyAiTraces(ctx, query, params)
+}
+
+func (legacyRepository) FindExceptions(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacyException, error) {
+ query, params := legacyWindow(legacyExceptionColumns, "exception_stack_traces", from, to, limit, offset)
+ query, args, err := lit.ParseNamedQuery(db.Driver, query, params)
+ if err != nil {
+ return nil, err
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var found []shared.LegacyException
+ for rows.Next() {
+ var row exceptionRow
+ var traceId string
+ if err := rows.Scan(&row.Id, &row.ProjectId, &row.TraceId, &row.SpanId, &row.TraceType, &row.ExceptionHash, &row.StackTrace, &row.RecordedAt, &row.Attributes, &row.AppVersion, &row.ServerName, &row.IsMessage, &traceId, &row.SessionId); err != nil {
+ return nil, err
+ }
+ found = append(found, shared.LegacyException{ExceptionStackTrace: row.toModel(), DistributedTraceId: traceId})
+ }
+ return found, rows.Err()
+}
+
+func (legacyRepository) FindSpans(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacySpan, error) {
+ query, params := legacyWindow("project_id, id, trace_id, COALESCE(parent_span_id, ''), name, start_time, duration, recorded_at, attributes", "spans", from, to, limit, offset)
+ query, args, err := lit.ParseNamedQuery(db.Driver, query, params)
+ if err != nil {
+ return nil, err
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var found []shared.LegacySpan
+ for rows.Next() {
+ var span shared.LegacySpan
+ var start, recorded sqlitetypes.SQLiteTime
+ var attributes sqlitetypes.SQLiteJSONMap
+ var duration int64
+ if err := rows.Scan(&span.ProjectId, &span.Id, &span.OwnerId, &span.ParentSpanId, &span.Name, &start, &duration, &recorded, &attributes); err != nil {
+ return nil, err
+ }
+ span.StartTime, span.RecordedAt, span.Duration, span.Attributes = start.Time, recorded.Time, time.Duration(duration), attributes
+ found = append(found, span)
+ }
+ return found, rows.Err()
+}
+
+// FindOwners looks the owners of spans and exceptions up by id, in all three tables an owner can live in.
+func (legacyRepository) FindOwners(ctx context.Context, ids []uuid.UUID, from, to time.Time) ([]models.Endpoint, []models.Task, []models.AiTrace, error) {
+ var endpoints []models.Endpoint
+ var tasks []models.Task
+ var aiTraces []models.AiTrace
+ for chunk := range slices.Chunk(ids, legacyIdChunk) {
+ query, params := legacyByIds(legacyEndpointColumns, "endpoints", chunk, from, to)
+ foundEndpoints, err := legacyEndpoints(ctx, query, params)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ query, params = legacyByIds(legacyTaskColumns, "tasks", chunk, from, to)
+ foundTasks, err := legacyTasks(ctx, query, params)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ query, params = legacyByIds(legacyAiTraceColumns, "ai_traces", chunk, from, to)
+ foundAiTraces, err := legacyAiTraces(ctx, query, params)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ endpoints, tasks, aiTraces = append(endpoints, foundEndpoints...), append(tasks, foundTasks...), append(aiTraces, foundAiTraces...)
+ }
+ return endpoints, tasks, aiTraces, nil
+}
+
+func (legacyRepository) FindBounds(ctx context.Context, table string) (oldest, newest time.Time, found bool, err error) {
+ if table, err = shared.LegacyTable(table); err != nil {
+ return oldest, newest, false, err
+ }
+ var low, high sqlitetypes.SQLiteTime
+ var count int64
+ if err = db.TelemetryDB.QueryRowContext(ctx, "SELECT COUNT(*), MIN(recorded_at), MAX(recorded_at) FROM "+table).Scan(&count, &low, &high); err != nil {
+ return oldest, newest, false, err
+ }
+ return low.Time, high.Time, count > 0, nil
+}
+
+// FindMovedIds returns full occurrence keys, never IDs shared by unrelated traces.
+func (legacyRepository) FindMovedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time) (map[shared.MovedOccurrence]bool, error) {
+ table, err := shared.LegacyTable(table)
+ if err != nil {
+ return nil, err
+ }
+ moved := map[shared.MovedOccurrence]bool{}
+ for chunk := range slices.Chunk(ids, legacyIdChunk) {
+ args := []any{from.UTC(), to.UTC()}
+ for _, id := range chunk {
+ args = append(args, id)
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, "SELECT project_id, id, trace_id, span_id, recorded_at FROM "+table+" WHERE recorded_at >= ? AND recorded_at < ? AND id IN (?"+strings.Repeat(",?", len(chunk)-1)+")", args...)
+ if err != nil {
+ return nil, err
+ }
+ if err := scanMovedIds(rows, moved); err != nil {
+ return nil, err
+ }
+ }
+ return moved, nil
+}
+
+func scanMovedIds(rows *sql.Rows, moved map[shared.MovedOccurrence]bool) error {
+ defer rows.Close()
+ for rows.Next() {
+ var project, id uuid.UUID
+ var trace, span string
+ var at sqlitetypes.SQLiteTime
+ if err := rows.Scan(&project, &id, &trace, &span, &at); err != nil {
+ return err
+ }
+ moved[shared.MovedOccurrenceKey(project, id, trace, span, at.Time)] = true
+ }
+ return rows.Err()
+}
+
+func (legacyRepository) FindMovedSpans(ctx context.Context, traceIds []string, from, to time.Time) (map[string]bool, error) {
+ moved := map[string]bool{}
+ for chunk := range slices.Chunk(traceIds, legacyIdChunk) {
+ args := []any{from.UTC(), to.UTC()}
+ for _, id := range chunk {
+ args = append(args, id)
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, "SELECT project_id, trace_id, span_id FROM "+shared.SpansTable+" WHERE recorded_at >= ? AND recorded_at < ? AND trace_id IN (?"+strings.Repeat(",?", len(chunk)-1)+")", args...)
+ if err != nil {
+ return nil, err
+ }
+ for rows.Next() {
+ var project uuid.UUID
+ var trace, span string
+ if err := rows.Scan(&project, &trace, &span); err != nil {
+ rows.Close()
+ return nil, err
+ }
+ moved[shared.MovedSpanKey(project, trace, span)] = true
+ }
+ if err := rows.Err(); err != nil {
+ rows.Close()
+ return nil, err
+ }
+ if err := rows.Close(); err != nil {
+ return nil, err
+ }
+ }
+ return moved, nil
+}
+
+// OccurrenceTime matches the destination column precision for recovery keys.
+func (legacyRepository) OccurrenceTime(table string, at time.Time) time.Time {
+ return at.Truncate(time.Microsecond)
+}
+
+var LegacyRepository = legacyRepository{}
diff --git a/backend/app/repositories/telemetry/duckdb/log_record.repository.go b/backend/app/repositories/telemetry/duckdb/log_record.repository.go
index 91b7a512..e3a1d064 100644
--- a/backend/app/repositories/telemetry/duckdb/log_record.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/log_record.repository.go
@@ -214,6 +214,15 @@ func (r *logRecordRepository) buildWhere(params shared.LogSearchParams) (string,
"from": params.FromDate.UTC(),
"to": params.ToDate.UTC(),
}
+ if len(params.ProjectIds) > 0 {
+ placeholders := make([]string, len(params.ProjectIds))
+ for i, id := range params.ProjectIds {
+ key := fmt.Sprintf("project%d", i)
+ placeholders[i] = ":" + key
+ args[key] = id
+ }
+ clauses[0] = "project_id IN (" + strings.Join(placeholders, ", ") + ")"
+ }
if params.MinSeverity > 0 {
clauses = append(clauses, "severity_number >= :min_severity")
diff --git a/backend/app/repositories/telemetry/duckdb/otel_span.repository.go b/backend/app/repositories/telemetry/duckdb/otel_span.repository.go
new file mode 100644
index 00000000..82d90d22
--- /dev/null
+++ b/backend/app/repositories/telemetry/duckdb/otel_span.repository.go
@@ -0,0 +1,197 @@
+//go:build telemetry_duckdb
+
+package duckdb
+
+import (
+ "context"
+ "database/sql"
+ "database/sql/driver"
+ "errors"
+ "strings"
+ "time"
+
+ "github.com/duckdb/duckdb-go/v2"
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+type otelSpanRepository struct{}
+
+var duckdbOtelCodec = shared.OtelValueCodec{
+ UUID: func(id uuid.UUID) any { return id.String() },
+ Time: func(t time.Time) any { return t.UTC() },
+ Nanos: func(n uint64) any { return n },
+ Uint32: func(n uint32) any { return int64(n) },
+}
+
+func duckdbOtelValues(span models.OtelSpan, groups shared.OtelSpanGroups) ([]driver.Value, error) {
+ row, err := shared.NewOtelSpanRow(span, groups)
+ if err != nil {
+ return nil, err
+ }
+ values := row.ScalarValues(duckdbOtelCodec)
+ text, err := row.TextValues()
+ if err != nil {
+ return nil, err
+ }
+ values = append(append(values, text...), shared.OtelKey(span.ProjectId, span.TraceId))
+ driverValues := make([]driver.Value, len(values))
+ for i, value := range values {
+ driverValues[i] = value
+ }
+ return driverValues, nil
+}
+
+func (r *otelSpanRepository) InsertAsync(ctx context.Context, spans []models.OtelSpan) (int, error) {
+ rejected, err := r.InsertWithRejections(ctx, spans)
+ return len(rejected), err
+}
+
+func (r *otelSpanRepository) InsertWithRejections(ctx context.Context, spans []models.OtelSpan) ([]shared.SpanOwner, error) {
+ if len(spans) == 0 {
+ return nil, nil
+ }
+ var rejected []shared.SpanOwner
+ err := withAppenderColumns(ctx, shared.SpansTable, strings.Split(shared.OtelTextStorageColumns+", trace_key", ", "), func(appender *duckdb.Appender) {
+ groups := shared.OtelSpanGroups{}
+ for _, span := range spans {
+ values, err := duckdbOtelValues(span, groups)
+ if err == nil {
+ err = appender.AppendRow(values...)
+ }
+ if err != nil {
+ captureDroppedRow(shared.SpansTable, err)
+ rejected = append(rejected, shared.SpanOwner{ProjectId: span.ProjectId, TraceId: span.TraceId, SpanId: span.SpanId})
+ }
+ }
+ })
+ if err != nil {
+ return nil, err
+ }
+ return rejected, nil
+}
+
+var OtelSpanRepository = &otelSpanRepository{}
+
+func scanDuckdbTopology(rows *sql.Rows, withStatement bool) ([]models.OtelSpan, error) {
+ result := make([]models.OtelSpan, 0)
+ for rows.Next() {
+ var span models.OtelSpan
+ var nanos uint64
+ columns := []any{&span.ProjectId, &span.TraceId, &span.SpanId, &span.ParentSpanId,
+ &span.Name, &span.Duration, &span.SpanKind, &span.StatusCode, &span.ServiceName, &span.ScopeName, &nanos, &span.RecordedAt}
+ if withStatement {
+ columns = append(columns, &span.DbStatement)
+ }
+ if err := rows.Scan(columns...); err != nil {
+ return nil, err
+ }
+ span.StartTime = shared.OtelNanosToTime(nanos)
+ result = append(result, span)
+ }
+ return result, rows.Err()
+}
+
+func (r *otelSpanRepository) FindTraceTopology(ctx context.Context, lookups []shared.SpanLookup, fromUnixNano uint64) ([]models.OtelSpan, error) {
+ if len(lookups) == 0 {
+ return nil, nil
+ }
+ query, args := shared.OtelTopologyQuery(lookups, true, fromUnixNano, duckdbOtelCodec.Time)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return scanDuckdbTopology(rows, false)
+}
+
+const duckdbStatementPreview = `substr(coalesce(json_extract_string(span_attributes, '$."db.query.text"'), json_extract_string(span_attributes, '$."db.statement"'), ''), 1, 240)`
+
+func (r *otelSpanRepository) FindTraceOutline(ctx context.Context, lookups []shared.SpanLookup) ([]models.OtelSpan, error) {
+ query, args := shared.OtelOutlineQuery(lookups, true, duckdbStatementPreview, duckdbOtelCodec.Time)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return scanDuckdbTopology(rows, true)
+}
+
+func (r *otelSpanRepository) FindSpanAttributes(ctx context.Context, lookup shared.SpanLookup, spanIds []string, limits shared.OtelAttributeLimits) (map[string]shared.OtelSpanAttributes, error) {
+ args := []any{shared.OtelKey(lookup.ProjectId, lookup.TraceId)}
+ for _, id := range spanIds {
+ args = append(args, id)
+ }
+ predicate := "trace_key = ? AND span_id IN (" + shared.OtelPlaceholders(len(spanIds)) + ")"
+ predicate, args = shared.OtelWindowPredicate(predicate, args, []shared.SpanLookup{lookup}, duckdbOtelCodec.Time)
+ rows, err := db.TelemetryDB.QueryContext(ctx, shared.OtelAttributeQuery("span_attributes", "octet_length(encode(span_attributes))", "start_time_unix_nano", predicate, shared.OtelTextWinnerOrder, limits), args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanOtelAttributeRows(rows)
+}
+
+func (r *otelSpanRepository) IsReadLimitError(error) bool { return false }
+
+var duckdbSearchDialect = shared.OtelSearchDialect{
+ WinnerOrder: shared.OtelTextWinnerOrder,
+ Project: func(id uuid.UUID) any { return id.String() },
+ Time: duckdbOtelCodec.Time,
+ Trace: func(project uuid.UUID, traceHex string) (string, any) {
+ return "trace_key = ?", shared.OtelKey(project, traceHex)
+ },
+ NameLike: "contains(lower(name), lower(?))",
+ Attribute: `json_extract_string(span_attributes, '$."' || ? || '"') = ?`,
+ StartOrder: "start_time_unix_nano",
+}
+
+func (r *otelSpanRepository) Search(ctx context.Context, search shared.OtelSpanSearch) ([]models.OtelSpan, uint64, error) {
+ page, pageArgs, count, countArgs := shared.OtelSearchQueries(search, duckdbSearchDialect)
+ var total uint64
+ if err := db.TelemetryDB.QueryRowContext(ctx, count, countArgs...).Scan(&total); err != nil {
+ return nil, 0, err
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, page, pageArgs...)
+ if err != nil {
+ return nil, 0, err
+ }
+ defer rows.Close()
+ spans, err := scanDuckdbTopology(rows, false)
+ return spans, total, err
+}
+
+func (r *otelSpanRepository) Services(ctx context.Context, project uuid.UUID, from, to time.Time) ([]string, error) {
+ query, args := shared.OtelServicesQuery(project, from, to, duckdbSearchDialect)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ services := make([]string, 0)
+ for rows.Next() {
+ var service string
+ if err := rows.Scan(&service); err != nil {
+ return nil, err
+ }
+ services = append(services, service)
+ }
+ return services, rows.Err()
+}
+
+func (r *otelSpanRepository) FindOTLP(ctx context.Context, project uuid.UUID, traceID, spanID string, at time.Time) ([]byte, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ var value []byte
+ from, to := shared.TraceWindowBounds(at.UTC())
+ err := db.TelemetryDB.QueryRowContext(ctx, "SELECT otlp FROM "+shared.SpansTable+" WHERE trace_key = ? AND span_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY "+shared.OtelTextWinnerOrder+" LIMIT 1", shared.OtelKey(project, traceID), spanID, from.UTC(), to.UTC()).Scan(&value)
+ if errors.Is(err, sql.ErrNoRows) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, err
+ }
+ return value, nil
+}
diff --git a/backend/app/repositories/telemetry/duckdb/profile.repository.go b/backend/app/repositories/telemetry/duckdb/profile.repository.go
index febefeb8..65dba7b1 100644
--- a/backend/app/repositories/telemetry/duckdb/profile.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/profile.repository.go
@@ -147,12 +147,6 @@ func (r *profileRepository) InsertProfilesAsync(ctx context.Context, profiles []
continue
}
- var distributedTraceId *string
- if p.DistributedTraceId != nil {
- v := p.DistributedTraceId.String()
- distributedTraceId = &v
- }
-
if err := appender.AppendRow(
p.Id.String(),
p.ProjectId.String(),
@@ -168,7 +162,6 @@ func (r *profileRepository) InsertProfilesAsync(ctx context.Context, profiles []
p.StorageKey,
p.TraceId,
p.SpanId,
- nullableString(distributedTraceId),
p.Unit,
boolToInt(p.IsGauge),
); err != nil {
diff --git a/backend/app/repositories/telemetry/duckdb/session.repository.go b/backend/app/repositories/telemetry/duckdb/session.repository.go
index 7f0d9963..c51de09b 100644
--- a/backend/app/repositories/telemetry/duckdb/session.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/session.repository.go
@@ -18,16 +18,16 @@ import (
)
type sessionRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- StartedAt sqlitetypes.SQLiteTime `lit:"started_at"`
- EndedAt *sqlitetypes.SQLiteTime `lit:"ended_at"`
- Duration int64 `lit:"duration"`
- ClientIP string `lit:"client_ip"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ StartedAt sqlitetypes.SQLiteTime `lit:"started_at"`
+ EndedAt *sqlitetypes.SQLiteTime `lit:"ended_at"`
+ Duration int64 `lit:"duration"`
+ ClientIP string `lit:"client_ip"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ TraceId string `lit:"trace_id"`
}
type sessionRowNaming struct{ lit.DefaultDbNamingStrategy }
@@ -44,14 +44,14 @@ func init() {
func (row *sessionRow) toModel() models.Session {
s := models.Session{
- Id: row.Id,
- ProjectId: row.ProjectId,
- StartedAt: row.StartedAt.Time,
- Duration: row.Duration,
- ClientIP: row.ClientIP,
- AppVersion: row.AppVersion,
- ServerName: row.ServerName,
- DistributedTraceId: row.DistributedTraceId,
+ Id: row.Id,
+ ProjectId: row.ProjectId,
+ StartedAt: row.StartedAt.Time,
+ Duration: row.Duration,
+ ClientIP: row.ClientIP,
+ AppVersion: row.AppVersion,
+ ServerName: row.ServerName,
+ TraceId: row.TraceId,
}
if row.EndedAt != nil {
t := row.EndedAt.Time
@@ -73,15 +73,15 @@ func (r *sessionRepository) Upsert(ctx context.Context, sessions []models.Sessio
return nil
}
const stmt = `INSERT INTO sessions
- (id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id)
- VALUES (:id, :project_id, :started_at, :ended_at, :duration, :client_ip, :attributes, :app_version, :server_name, :distributed_trace_id)
+ (id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id)
+ VALUES (:id, :project_id, :started_at, :ended_at, :duration, :client_ip, :attributes, :app_version, :server_name, :trace_id)
ON CONFLICT (id) DO UPDATE SET
ended_at = COALESCE(excluded.ended_at, sessions.ended_at),
duration = CASE WHEN excluded.duration > 0 THEN excluded.duration ELSE sessions.duration END,
attributes = excluded.attributes,
app_version = excluded.app_version,
server_name = excluded.server_name,
- distributed_trace_id = excluded.distributed_trace_id`
+ trace_id = excluded.trace_id`
tx, err := db.TelemetryDB.BeginTx(ctx, nil)
if err != nil {
@@ -103,22 +103,17 @@ func (r *sessionRepository) Upsert(ctx context.Context, sessions []models.Sessio
t := s.EndedAt.UTC()
endedAt = &t
}
- var distTraceId *string
- if s.DistributedTraceId != nil {
- v := s.DistributedTraceId.String()
- distTraceId = &v
- }
query, args, err := lit.ParseNamedQuery(db.Driver, stmt, lit.P{
- "id": s.Id.String(),
- "project_id": s.ProjectId.String(),
- "started_at": s.StartedAt.UTC(),
- "ended_at": endedAt,
- "duration": s.Duration,
- "client_ip": s.ClientIP,
- "attributes": attrs,
- "app_version": s.AppVersion,
- "server_name": s.ServerName,
- "distributed_trace_id": distTraceId,
+ "id": s.Id.String(),
+ "project_id": s.ProjectId.String(),
+ "started_at": s.StartedAt.UTC(),
+ "ended_at": endedAt,
+ "duration": s.Duration,
+ "client_ip": s.ClientIP,
+ "attributes": attrs,
+ "app_version": s.AppVersion,
+ "server_name": s.ServerName,
+ "trace_id": s.TraceId,
})
if err != nil {
return err
@@ -177,7 +172,7 @@ func (r *sessionRepository) FindAll(ctx context.Context, projectId uuid.UUID, fr
}
offset := (page - 1) * pageSize
- query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id FROM sessions WHERE project_id = :project_id AND started_at >= :start AND started_at <= :end" + whereExtra + " ORDER BY " + orderBy + " " + sortDir + " LIMIT :limit OFFSET :offset"
+ query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id FROM sessions WHERE project_id = :project_id AND started_at >= :start AND started_at <= :end" + whereExtra + " ORDER BY " + orderBy + " " + sortDir + " LIMIT :limit OFFSET :offset"
queryParams := lit.P{
"project_id": projectId,
@@ -231,7 +226,7 @@ func buildSessionFilterClauseDuckDB(search string, filters []shared.SessionAttri
}
func (r *sessionRepository) FindById(ctx context.Context, projectId, sessionId uuid.UUID, startedAt *time.Time) (*models.Session, error) {
- query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id FROM sessions WHERE project_id = :project_id AND id = :id"
+ query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id FROM sessions WHERE project_id = :project_id AND id = :id"
params := lit.P{"project_id": projectId, "id": sessionId}
if startedAt != nil {
from, to := shared.TraceWindowBounds(*startedAt)
diff --git a/backend/app/repositories/telemetry/duckdb/span.repository.go b/backend/app/repositories/telemetry/duckdb/span.repository.go
deleted file mode 100644
index b6b82c5d..00000000
--- a/backend/app/repositories/telemetry/duckdb/span.repository.go
+++ /dev/null
@@ -1,144 +0,0 @@
-//go:build telemetry_duckdb
-
-package duckdb
-
-import (
- "context"
- "slices"
- "time"
-
- "github.com/duckdb/duckdb-go/v2"
- "github.com/google/uuid"
- "github.com/tracewayapp/lit/v2"
- "github.com/tracewayapp/traceway/backend/app/db"
- "github.com/tracewayapp/traceway/backend/app/models"
- "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
- "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
-)
-
-type span struct {
- Id uuid.UUID `lit:"id"`
- TraceId uuid.UUID `lit:"trace_id"`
- ProjectId uuid.UUID `lit:"project_id"`
- Name string `lit:"name"`
- StartTime sqlitetypes.SQLiteTime `lit:"start_time"`
- Duration int64 `lit:"duration"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- ParentSpanId *uuid.UUID `lit:"parent_span_id"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
-}
-
-func init() {
- models.ExtensionModelRegistrations = append(models.ExtensionModelRegistrations, func(driver lit.Driver) {
- lit.RegisterModel[span](driver)
- })
-}
-
-func (r *span) toModel() models.Span {
- s := models.Span{
- Id: r.Id,
- TraceId: r.TraceId,
- ProjectId: r.ProjectId,
- Name: r.Name,
- StartTime: r.StartTime.Time,
- Duration: time.Duration(r.Duration),
- RecordedAt: r.RecordedAt.Time,
- ParentSpanId: r.ParentSpanId,
- }
- if r.Attributes != nil {
- s.Attributes = map[string]string(r.Attributes)
- }
- return s
-}
-
-type spanRepository struct{}
-
-func (r *spanRepository) InsertAsync(ctx context.Context, spans []models.Span) error {
- if len(spans) == 0 {
- return nil
- }
-
- return withAppender(ctx, "spans", func(appender *duckdb.Appender) {
-
- for _, s := range spans {
- attributesJSON, err := attrJSON(s.Attributes)
- if err != nil {
- captureDroppedRow("spans", err)
- continue
- }
-
- var parentSpanId *string
- if s.ParentSpanId != nil {
- v := s.ParentSpanId.String()
- parentSpanId = &v
- }
-
- if err := appender.AppendRow(
- s.Id.String(),
- s.TraceId.String(),
- s.ProjectId.String(),
- s.Name,
- s.StartTime.UTC(),
- int64(s.Duration),
- s.RecordedAt.UTC(),
- nullableString(parentSpanId),
- attributesJSON,
- ); err != nil {
- captureDroppedRow("spans", err)
- }
- }
-
- })
-}
-
-func (r *spanRepository) FindByTraceId(ctx context.Context, projectId, traceId uuid.UUID, recordedAt *time.Time) ([]models.Span, error) {
- query, args := shared.SpanLookupQuery([]shared.SpanLookup{{ProjectId: projectId, TraceId: traceId, RecordedAt: recordedAt}},
- func(t time.Time) any { return t.UTC() })
- return r.querySpans(ctx, query, args)
-}
-
-func (r *spanRepository) FindByTraces(ctx context.Context, lookups []shared.SpanLookup) ([]models.Span, error) {
- spans := make([]models.Span, 0)
- type spanKey struct {
- projectId, traceId, id uuid.UUID
- recordedAt time.Time
- }
- seen := make(map[spanKey]bool)
- for batch := range slices.Chunk(lookups, shared.SpanLookupBatchSize) {
- query, args := shared.SpanLookupQuery(batch, func(t time.Time) any { return t.UTC() })
- found, err := r.querySpans(ctx, query, args)
- if err != nil {
- return nil, err
- }
- for _, span := range found {
- key := spanKey{span.ProjectId, span.TraceId, span.Id, span.RecordedAt}
- if !seen[key] {
- seen[key] = true
- spans = append(spans, span)
- }
- }
- }
- slices.SortStableFunc(spans, func(a, b models.Span) int { return a.StartTime.Compare(b.StartTime) })
- return spans, nil
-}
-
-func (r *spanRepository) querySpans(ctx context.Context, query string, args []any) ([]models.Span, error) {
- rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
- if err != nil {
- return nil, err
- }
- defer rows.Close()
-
- spans := make([]models.Span, 0)
- for rows.Next() {
- var row span
- if err := rows.Scan(&row.Id, &row.TraceId, &row.ProjectId, &row.Name, &row.StartTime,
- &row.Duration, &row.RecordedAt, &row.ParentSpanId, &row.Attributes); err != nil {
- return nil, err
- }
- spans = append(spans, row.toModel())
- }
- return spans, rows.Err()
-}
-
-var SpanRepository = &spanRepository{}
diff --git a/backend/app/repositories/telemetry/duckdb/task.repository.go b/backend/app/repositories/telemetry/duckdb/task.repository.go
index 71d30be7..d29ca1a5 100644
--- a/backend/app/repositories/telemetry/duckdb/task.repository.go
+++ b/backend/app/repositories/telemetry/duckdb/task.repository.go
@@ -20,18 +20,19 @@ import (
)
type task struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- TaskName string `lit:"task_name"`
- Duration int64 `lit:"duration"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- ClientIP string `lit:"client_ip"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- SpanId *uuid.UUID `lit:"span_id"`
- IsRoot bool `lit:"is_root"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ TaskName string `lit:"task_name"`
+ Duration int64 `lit:"duration"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ ClientIP string `lit:"client_ip"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ ParentSpanId string `lit:"parent_span_id"`
+ IsRoot bool `lit:"is_root"`
}
type taskGroupRow struct {
@@ -60,17 +61,18 @@ func init() {
func (r *task) toModel() models.Task {
t := models.Task{
- Id: r.Id,
- ProjectId: r.ProjectId,
- TaskName: r.TaskName,
- Duration: time.Duration(r.Duration),
- RecordedAt: r.RecordedAt.Time,
- ClientIP: r.ClientIP,
- AppVersion: r.AppVersion,
- ServerName: r.ServerName,
- DistributedTraceId: r.DistributedTraceId,
- SpanId: r.SpanId,
- IsRoot: r.IsRoot,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ TaskName: r.TaskName,
+ Duration: time.Duration(r.Duration),
+ RecordedAt: r.RecordedAt.Time,
+ ClientIP: r.ClientIP,
+ AppVersion: r.AppVersion,
+ ServerName: r.ServerName,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ ParentSpanId: r.ParentSpanId,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
t.Attributes = map[string]string(r.Attributes)
@@ -85,27 +87,15 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
return nil
}
- return withAppender(ctx, "tasks", func(appender *duckdb.Appender) {
+ return withAppenderColumns(ctx, "tasks_v2", strings.Split("id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root", ", "), func(appender *duckdb.Appender) {
for _, t := range lines {
attributesJSON, err := attrJSON(t.Attributes)
if err != nil {
- captureDroppedRow("tasks", err)
+ captureDroppedRow("tasks_v2", err)
continue
}
- var distributedTraceId *string
- if t.DistributedTraceId != nil {
- s := t.DistributedTraceId.String()
- distributedTraceId = &s
- }
-
- var spanId *string
- if t.SpanId != nil {
- s := t.SpanId.String()
- spanId = &s
- }
-
isRoot := boolToInt(t.IsRoot)
if err := appender.AppendRow(
@@ -118,11 +108,12 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
attributesJSON,
t.AppVersion,
t.ServerName,
- nullableString(distributedTraceId),
- nullableString(spanId),
+ t.TraceId,
+ t.SpanId,
+ t.ParentSpanId,
isRoot,
); err != nil {
- captureDroppedRow("tasks", err)
+ captureDroppedRow("tasks_v2", err)
}
}
@@ -131,7 +122,7 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
func (e *taskRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
result, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
if err != nil {
return 0, err
@@ -144,7 +135,7 @@ func (e *taskRepository) CountBetween(ctx context.Context, projectId uuid.UUID,
func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string) ([]models.Task, int64, error) {
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": fromDate.UTC(), "to": toDate.UTC()})
if err != nil {
return nil, 0, err
@@ -162,8 +153,8 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromD
}
rows, err := lit.SelectNamed[task](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s DESC LIMIT :limit OFFSET :offset`, orderBy),
lit.P{"project_id": projectId, "from": fromDate.UTC(), "to": toDate.UTC(), "limit": pageSize, "offset": offset})
if err != nil {
@@ -195,7 +186,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
totalCount := int64(0)
if !needsGoSort {
totalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(DISTINCT task_name) AS count FROM tasks WHERE "+whereClause,
+ "SELECT COUNT(DISTINCT task_name) AS count FROM tasks_v2 WHERE "+whereClause,
params)
if err != nil {
return nil, 0, err
@@ -220,7 +211,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
if needsGoSort {
baseQuery = `SELECT ` + groupedCols + `
- FROM tasks WHERE ` + whereClause + `
+ FROM tasks_v2 WHERE ` + whereClause + `
GROUP BY task_name`
} else {
orderExpr := map[string]string{"count": "count", "last_seen": "last_seen"}
@@ -229,7 +220,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
expr = "count"
}
baseQuery = fmt.Sprintf(`SELECT `+groupedCols+`
- FROM tasks WHERE `+whereClause+`
+ FROM tasks_v2 WHERE `+whereClause+`
GROUP BY task_name ORDER BY %s %s LIMIT :limit OFFSET :offset`, expr, sortDir)
params["limit"] = pageSize
params["offset"] = offset
@@ -300,7 +291,7 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
params := lit.P{"project_id": projectId, "task_name": taskName, "from": fromDate.UTC(), "to": toDate.UTC()}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -323,8 +314,8 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
}
rows, err := lit.SelectNamed[task](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
lit.P{"project_id": projectId, "task_name": taskName, "from": fromDate.UTC(), "to": toDate.UTC(), "limit": pageSize, "offset": offset})
if err != nil {
@@ -340,8 +331,8 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
}
func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID, recordedAt *time.Time) (*models.Task, error) {
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
- FROM tasks WHERE project_id = :project_id AND id = :id`
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM tasks_v2 WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": taskId}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -365,7 +356,7 @@ func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UU
func (e *taskRepository) CountByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
return queryTaskTimeSeries(ctx,
`SELECT `+timeBucketExpr("recorded_at", 3600)+` as bucket, CAST(COUNT(*) AS DOUBLE) as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
}
@@ -373,7 +364,7 @@ func (e *taskRepository) CountByHour(ctx context.Context, projectId uuid.UUID, s
func (e *taskRepository) AvgDurationByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
return queryTaskTimeSeries(ctx,
`SELECT `+timeBucketExpr("recorded_at", 3600)+` as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
}
@@ -381,7 +372,7 @@ func (e *taskRepository) AvgDurationByHour(ctx context.Context, projectId uuid.U
func (e *taskRepository) CountByInterval(ctx context.Context, projectId uuid.UUID, start, end time.Time, intervalMinutes int) ([]models.TimeSeriesPoint, error) {
return queryTaskTimeSeries(ctx,
`SELECT `+timeBucketExpr("recorded_at", intervalMinutes*60)+` as bucket, CAST(COUNT(*) AS DOUBLE) as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
}
@@ -389,7 +380,7 @@ func (e *taskRepository) CountByInterval(ctx context.Context, projectId uuid.UUI
func (e *taskRepository) AvgDurationByInterval(ctx context.Context, projectId uuid.UUID, start, end time.Time, intervalMinutes int) ([]models.TimeSeriesPoint, error) {
return queryTaskTimeSeries(ctx,
`SELECT `+timeBucketExpr("recorded_at", intervalMinutes*60)+` as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": start.UTC(), "to": end.UTC()})
}
@@ -400,7 +391,7 @@ func (e *taskRepository) FindWorstTasks(ctx context.Context, projectId uuid.UUID
groups, err := lit.SelectNamed[taskGroupRow](db.TelemetryDB,
`SELECT task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen,
quantile_cont(duration, 0.5) as p50, quantile_cont(duration, 0.95) as p95
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY task_name`,
params)
if err != nil {
@@ -440,7 +431,7 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
}
statsRow, err := lit.SelectSingleNamed[taskCountStatsRow](db.TelemetryDB,
- "SELECT COUNT(*) AS count, CASE WHEN COUNT(*) > 0 THEN AVG(duration) / 1000000.0 ELSE 0 END AS avg_dur_ms FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count, CASE WHEN COUNT(*) > 0 THEN AVG(duration) / 1000000.0 ELSE 0 END AS avg_dur_ms FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, err
@@ -451,7 +442,7 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
pctQuery, pctArgs, err := lit.ParseNamedQuery(db.Driver,
`SELECT quantile_cont(duration, 0.5) AS p50, quantile_cont(duration, 0.95) AS p95, quantile_cont(duration, 0.99) AS p99
- FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to`,
+ FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
return nil, err
@@ -474,19 +465,20 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
}, nil
}
-func (e *taskRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Task, error) {
- if len(projectIds) == 0 {
+func (e *taskRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Task, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
placeholders[i] = ":" + key
params[key] = pid
}
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM tasks_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -509,7 +501,7 @@ func (e *taskRepository) FindByDistributedTraceId(ctx context.Context, distribut
var tasks []models.Task
for sqlRows.Next() {
var row task
- if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TaskName, &row.Duration, &row.RecordedAt, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.DistributedTraceId); err != nil {
+ if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TaskName, &row.Duration, &row.RecordedAt, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.TraceId, &row.SpanId, &row.ParentSpanId, &row.IsRoot); err != nil {
return nil, err
}
tasks = append(tasks, row.toModel())
diff --git a/backend/app/repositories/telemetry/exception_owner.go b/backend/app/repositories/telemetry/exception_owner.go
new file mode 100644
index 00000000..4c35aece
--- /dev/null
+++ b/backend/app/repositories/telemetry/exception_owner.go
@@ -0,0 +1,82 @@
+package telemetry
+
+import (
+ "context"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+// ExceptionOwner is the endpoint, task or AI trace an exception happened in.
+type ExceptionOwner struct {
+ TraceType string `json:"traceType"`
+ Id uuid.UUID `json:"id"`
+ Name string `json:"name"`
+ StatusCode int16 `json:"statusCode"`
+ Duration time.Duration `json:"duration"`
+ RecordedAt time.Time `json:"recordedAt"`
+ TraceId string `json:"traceId"`
+}
+
+const maxExceptionOwnerDepth = 10000
+
+// FindExceptionOwner finds the endpoint, task or AI trace of the exception's project whose span the exception was
+// recorded on, or the nearest one above that span. Ingest stores the exception's trace and span and nothing more, so
+// the owner is worked out here, when somebody looks. nil means the exception stands alone.
+func FindExceptionOwner(ctx context.Context, exception models.ExceptionStackTrace) (*ExceptionOwner, error) {
+ if exception.TraceId == "" || exception.SpanId == "" {
+ return nil, nil
+ }
+ traceIds := []string{exception.TraceId}
+ projectIds := []uuid.UUID{exception.ProjectId}
+ at := exception.RecordedAt
+
+ owners := map[string]*ExceptionOwner{}
+ endpoints, err := EndpointRepository.FindByTraceIds(ctx, traceIds, projectIds, &at)
+ if err != nil {
+ return nil, err
+ }
+ tasks, err := TaskRepository.FindByTraceIds(ctx, traceIds, projectIds, &at)
+ if err != nil {
+ return nil, err
+ }
+ aiTraces, err := AiTraceRepository.FindByTraceIds(ctx, traceIds, projectIds, &at)
+ if err != nil {
+ return nil, err
+ }
+ for _, ai := range aiTraces {
+ if ai.TraceId == exception.TraceId {
+ owners[ai.SpanId] = &ExceptionOwner{TraceType: "ai_trace", Id: ai.Id, Name: ai.TraceName, Duration: ai.Duration, RecordedAt: ai.RecordedAt, TraceId: ai.TraceId}
+ }
+ }
+ for _, task := range tasks {
+ if task.TraceId == exception.TraceId {
+ owners[task.SpanId] = &ExceptionOwner{TraceType: "task", Id: task.Id, Name: task.TaskName, Duration: task.Duration, RecordedAt: task.RecordedAt, TraceId: task.TraceId}
+ }
+ }
+ for _, endpoint := range endpoints {
+ if endpoint.TraceId == exception.TraceId {
+ owners[endpoint.SpanId] = &ExceptionOwner{TraceType: "endpoint", Id: endpoint.Id, Name: endpoint.Endpoint, StatusCode: endpoint.StatusCode, Duration: endpoint.Duration, RecordedAt: endpoint.RecordedAt, TraceId: endpoint.TraceId}
+ }
+ }
+ if len(owners) == 0 {
+ return nil, nil
+ }
+ if owner := owners[exception.SpanId]; owner != nil {
+ return owner, nil
+ }
+
+ parents, err := SpanRepository.FindTraceParents(ctx, projectIds, exception.TraceId, at)
+ if err != nil {
+ return nil, err
+ }
+ visited := map[string]bool{exception.SpanId: true}
+ for current, depth := parents[exception.SpanId], 0; current != "" && !visited[current] && depth < maxExceptionOwnerDepth; current, depth = parents[current], depth+1 {
+ if owner := owners[current]; owner != nil {
+ return owner, nil
+ }
+ visited[current] = true
+ }
+ return nil, nil
+}
diff --git a/backend/app/repositories/telemetry/helpers_test.go b/backend/app/repositories/telemetry/helpers_test.go
index 31458cee..784a2605 100644
--- a/backend/app/repositories/telemetry/helpers_test.go
+++ b/backend/app/repositories/telemetry/helpers_test.go
@@ -1,12 +1,14 @@
package telemetry
import (
+ "context"
"math"
"testing"
"time"
"github.com/google/uuid"
"github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
)
func assertApproxEqual(t *testing.T, name string, got, want, tolerance float64) {
@@ -90,16 +92,27 @@ func makeException(projectId uuid.UUID, hash, stackTrace string, recordedAt time
}
}
-func makeSpan(projectId, traceId uuid.UUID, name string, startTime time.Time, duration time.Duration) models.Span {
+// makeSpan is a span of the native protocol: a child of the run it was recorded in, which is the root span of its trace.
+func makeSpan(projectId, runId uuid.UUID, name string, startTime time.Time, duration time.Duration) models.Span {
return models.Span{
- Id: uuid.New(),
- TraceId: traceId,
- ProjectId: projectId,
- Name: name,
- StartTime: startTime,
- Duration: duration,
- RecordedAt: startTime,
+ ProjectId: projectId,
+ TraceId: hexId(runId),
+ SpanId: hexId(uuid.New()),
+ ParentSpanId: hexId(runId),
+ Name: name,
+ StartTime: startTime,
+ Duration: duration,
+ RecordedAt: startTime,
+ }
+}
+
+// findRunSpans reads the spans under a native run the way its detail page does.
+func findRunSpans(ctx context.Context, projectId, runId uuid.UUID, recordedAt *time.Time) ([]models.Span, error) {
+ graph, err := SpanRepository.FindGraph(ctx, shared.SpanLookup{ProjectId: projectId, TraceId: hexId(runId), SpanId: hexId(runId), RecordedAt: recordedAt})
+ if err != nil {
+ return nil, err
}
+ return graph.Spans, nil
}
func makeSessionRecording(projectId, exceptionId uuid.UUID, filePath string, recordedAt time.Time) models.SessionRecording {
diff --git a/backend/app/repositories/telemetry/log_record_repository_test.go b/backend/app/repositories/telemetry/log_record_repository_test.go
index ac18ae3a..8b91d2d6 100644
--- a/backend/app/repositories/telemetry/log_record_repository_test.go
+++ b/backend/app/repositories/telemetry/log_record_repository_test.go
@@ -223,3 +223,42 @@ func TestLogRecordRepository_SearchByTraceIdsPastBranchCap(t *testing.T) {
bodies, _ = searchBodies(t, params)
assertBodies(t, "page 2", bodies, []string{"older"})
}
+
+func TestLogRecordRepository_WholeTracePaginatesAcrossProjects(t *testing.T) {
+ setupTestDB(t)
+ projects := []uuid.UUID{uuid.New(), uuid.New()}
+ other := uuid.New()
+ now := truncateMs(time.Now().UTC())
+ const trace = "0123456789abcdef0123456789abcdef"
+ var records []models.LogRecord
+ for i := 0; i < 7; i++ {
+ record := makeLogRecord(projects[i%2], 9, fmt.Sprintf("row %d", i), now.Add(time.Duration(i)*time.Second))
+ record.TraceId = trace
+ records = append(records, record)
+ }
+ outside := makeLogRecord(other, 9, "outside", now)
+ outside.TraceId = trace
+ records = append(records, outside, makeLogRecord(projects[0], 9, "another trace", now))
+ if err := LogRecordRepository.InsertAsync(context.Background(), records); err != nil {
+ t.Fatal(err)
+ }
+ params := shared.LogSearchParams{ProjectId: projects[0], ProjectIds: projects, TraceId: trace,
+ FromDate: now.Add(-time.Hour), ToDate: now.Add(time.Hour), OrderBy: "timestamp", SortDirection: "asc", PageSize: 3}
+ var got []string
+ for page := 1; page <= 3; page++ {
+ params.Page = page
+ bodies, total := searchBodies(t, params)
+ if total != 7 {
+ t.Fatalf("page %d: total = %d", page, total)
+ }
+ got = append(got, bodies...)
+ }
+ assertBodies(t, "whole trace", got, []string{"row 0", "row 1", "row 2", "row 3", "row 4", "row 5", "row 6"})
+ params.ProjectIds = nil
+ params.Page, params.PageSize = 1, 10
+ bodies, total := searchBodies(t, params)
+ if total != 4 {
+ t.Fatalf("project scope widened: %d", total)
+ }
+ assertBodies(t, "selected project", bodies, []string{"row 0", "row 2", "row 4", "row 6"})
+}
diff --git a/backend/app/repositories/telemetry/move_over.go b/backend/app/repositories/telemetry/move_over.go
new file mode 100644
index 00000000..1072a192
--- /dev/null
+++ b/backend/app/repositories/telemetry/move_over.go
@@ -0,0 +1,575 @@
+package telemetry
+
+import (
+ "context"
+ "fmt"
+ "log"
+ "strings"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+// MoveOverOptions tunes RunMoveOver.
+type MoveOverOptions struct {
+ // PageSize bounds the rows held in memory at once. Zero means the default.
+ PageSize int
+ // Oldest, when set, is the last day moved. History before it stays in the old tables.
+ Oldest time.Time
+ Log func(format string, args ...any)
+}
+
+const (
+ defaultPageSize = 20000
+ moveOverDay = 24 * time.Hour
+ // An owner is recorded near its spans and exceptions, not at the same instant: an endpoint at its start, an old
+ // OTel task at its end.
+ ownerWindow = moveOverDay
+)
+
+// Entities before what hangs off them, so a day is usable as soon as its endpoints are in.
+var tableOrder = []string{"endpoints", "tasks", "ai_traces", "exception_stack_traces", "spans"}
+
+type mover struct {
+ MoveOverOptions
+ legacy interface {
+ OccurrenceTime(table string, at time.Time) time.Time
+ FindEndpoints(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Endpoint, error)
+ FindTasks(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Task, error)
+ FindAiTraces(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.AiTrace, error)
+ FindExceptions(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacyException, error)
+ FindSpans(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacySpan, error)
+ FindOwners(ctx context.Context, ids []uuid.UUID, from, to time.Time) ([]models.Endpoint, []models.Task, []models.AiTrace, error)
+ FindBounds(ctx context.Context, table string) (oldest, newest time.Time, found bool, err error)
+ FindMovedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time) (map[shared.MovedOccurrence]bool, error)
+ FindMovedSpans(ctx context.Context, traceIds []string, from, to time.Time) (map[string]bool, error)
+ }
+ progress interface {
+ FindProgress(ctx context.Context) ([]transactional.MoveOverDay, error)
+ SaveProgress(ctx context.Context, day transactional.MoveOverDay) error
+ }
+}
+
+// RunMoveOver copies history from the tables V2 replaced into the V2 tables. It runs inside the backend, because the
+// embedded DuckDB file cannot be opened by a second process. It goes newest day first, so recent data is back first,
+// one day of one table at a time, and records each finished day so it can stop and resume. It returns when history is
+// exhausted, the context ends, or a read or a write fails. Running it again picks up where it stopped.
+func RunMoveOver(ctx context.Context, options MoveOverOptions) error {
+ m := &mover{MoveOverOptions: options, legacy: LegacyRepository, progress: moveOverProgress{}}
+ return m.run(ctx)
+}
+
+func (m *mover) run(ctx context.Context) error {
+ if err := ctx.Err(); err != nil {
+ return err
+ }
+ if m.PageSize <= 0 {
+ m.PageSize = defaultPageSize
+ }
+ if m.Log == nil {
+ m.Log = func(format string, args ...any) { log.Printf("[tracewaybackend] move-over: "+format, args...) }
+ }
+ saved, err := m.progress.FindProgress(ctx)
+ if err != nil {
+ return fmt.Errorf("move-over progress: %w", err)
+ }
+ state := map[string]string{}
+ for _, entry := range saved {
+ state[entry.Table+"|"+entry.Day] = entry.State
+ }
+
+ type span struct{ oldest, newest time.Time }
+ bounds := map[string]span{}
+ var oldest, newest time.Time
+ for _, table := range tableOrder {
+ low, high, found, err := m.legacy.FindBounds(ctx, table)
+ if err != nil {
+ return fmt.Errorf("move-over bounds of %s: %w", table, err)
+ }
+ if !found {
+ continue
+ }
+ low, high = low.UTC().Truncate(moveOverDay), high.UTC().Truncate(moveOverDay)
+ bounds[table] = span{low, high}
+ if oldest.IsZero() || low.Before(oldest) {
+ oldest = low
+ }
+ if high.After(newest) {
+ newest = high
+ }
+ }
+ if len(bounds) == 0 {
+ m.Log("the old tables are empty, nothing to move")
+ return nil
+ }
+ if limit := m.Oldest.UTC().Truncate(moveOverDay); !m.Oldest.IsZero() && limit.After(oldest) {
+ oldest = limit
+ }
+
+ for current := newest; !current.Before(oldest); current = current.Add(-moveOverDay) {
+ for _, table := range tableOrder {
+ held, known := bounds[table]
+ label := current.Format(time.DateOnly)
+ if !known || current.Before(held.oldest) || current.After(held.newest) || state[table+"|"+label] == transactional.MoveOverDone {
+ continue
+ }
+ if err := ctx.Err(); err != nil {
+ return err
+ }
+ resumed := state[table+"|"+label] == transactional.MoveOverStarted
+ if err := m.progress.SaveProgress(ctx, transactional.MoveOverDay{Table: table, Day: label, State: transactional.MoveOverStarted}); err != nil {
+ return fmt.Errorf("move-over progress of %s %s: %w", table, label, err)
+ }
+ began := time.Now()
+ rows, err := m.moveDay(ctx, table, current, resumed)
+ if err != nil {
+ return fmt.Errorf("move-over of %s %s: %w", table, label, err)
+ }
+ if err := m.progress.SaveProgress(ctx, transactional.MoveOverDay{Table: table, Day: label, State: transactional.MoveOverDone, Rows: rows}); err != nil {
+ return fmt.Errorf("move-over progress of %s %s: %w", table, label, err)
+ }
+ m.Log("%s %s: %d rows in %s", label, table, rows, time.Since(began).Round(time.Millisecond))
+ }
+ }
+ m.Log("done, every day from %s back to %s is in the V2 tables", newest.Format(time.DateOnly), oldest.Format(time.DateOnly))
+ return nil
+}
+
+// moveDay walks one day in slices narrow enough to hold in memory. A slice that comes back full is halved and read
+// again, so no read sorts a day. Only a single second that is still too full is paged, in a stable full-row order.
+func (m *mover) moveDay(ctx context.Context, table string, start time.Time, resumed bool) (int64, error) {
+ var total int64
+ width, end := time.Hour, start.Add(moveOverDay)
+ for cursor := start; cursor.Before(end); {
+ if err := ctx.Err(); err != nil {
+ return total, err
+ }
+ to := cursor.Add(width)
+ if to.After(end) {
+ to = end
+ }
+ var moved int64
+ if to.Sub(cursor) > time.Second {
+ var full bool
+ var err error
+ if moved, full, err = m.move(ctx, table, cursor, to, nil, resumed); err != nil {
+ return total, err
+ }
+ if full {
+ width = max(to.Sub(cursor)/2, time.Second).Truncate(time.Second)
+ continue
+ }
+ } else {
+ for offset, full := 0, true; full; {
+ page, more, err := m.move(ctx, table, cursor, to, &offset, resumed)
+ if err != nil {
+ return total, err
+ }
+ moved, full, offset = moved+page, more, offset+int(page)
+ }
+ }
+ total, cursor = total+moved, to
+ // A sparse stretch widens the slice again, or one busy second would leave the rest of the day read second by second.
+ if moved*2 <= int64(m.PageSize) {
+ width = min(width*2, moveOverDay)
+ }
+ }
+ return total, nil
+}
+
+// move probes a slice without ordering before narrowing it. Only a paged read
+// writes a full page, since a full probe may have omitted rows.
+func (m *mover) move(ctx context.Context, table string, from, to time.Time, offset *int, resumed bool) (int64, bool, error) {
+ var count int
+ var write func() error
+ switch table {
+ case "endpoints":
+ rows, err := m.legacy.FindEndpoints(ctx, from, to, m.PageSize, offset)
+ if err != nil {
+ return 0, false, err
+ }
+ count = len(rows)
+ write = func() error { return m.writeEndpoints(ctx, rows, from, to, resumed) }
+ case "tasks":
+ rows, err := m.legacy.FindTasks(ctx, from, to, m.PageSize, offset)
+ if err != nil {
+ return 0, false, err
+ }
+ count = len(rows)
+ write = func() error { return m.writeTasks(ctx, rows, from, to, resumed) }
+ case "ai_traces":
+ rows, err := m.legacy.FindAiTraces(ctx, from, to, m.PageSize, offset)
+ if err != nil {
+ return 0, false, err
+ }
+ count = len(rows)
+ write = func() error { return m.writeAiTraces(ctx, rows, from, to, resumed) }
+ case "exception_stack_traces":
+ rows, err := m.legacy.FindExceptions(ctx, from, to, m.PageSize, offset)
+ if err != nil {
+ return 0, false, err
+ }
+ count = len(rows)
+ write = func() error { return m.writeExceptions(ctx, rows, from, to, resumed) }
+ case "spans":
+ rows, err := m.legacy.FindSpans(ctx, from, to, m.PageSize, offset)
+ if err != nil {
+ return 0, false, err
+ }
+ count = len(rows)
+ write = func() error { return m.writeSpans(ctx, rows, from, to, resumed) }
+ default:
+ return 0, false, fmt.Errorf("unknown table %q", table)
+ }
+ full := count >= m.PageSize
+ if count == 0 || (full && offset == nil) {
+ return 0, full, nil
+ }
+ return int64(count), full, write()
+}
+
+func (m *mover) movedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time, resumed bool) (map[shared.MovedOccurrence]bool, error) {
+ if !resumed || len(ids) == 0 {
+ return nil, nil
+ }
+ return m.legacy.FindMovedIds(ctx, shared.LegacyTables[table], ids, from.Truncate(time.Second), to.Add(time.Second))
+}
+
+// insertSpans writes the spans of a page. On a resumed day it leaves out the ones the interrupted run already wrote.
+func (m *mover) insertSpans(ctx context.Context, spans []models.Span, from, to time.Time, resumed bool) error {
+ if resumed && len(spans) > 0 {
+ seen := map[string]bool{}
+ var traces []string
+ for _, span := range spans {
+ if span.StartTime.Before(from) {
+ from = span.StartTime
+ }
+ if span.StartTime.After(to) {
+ to = span.StartTime
+ }
+ if !seen[span.TraceId] {
+ seen[span.TraceId] = true
+ traces = append(traces, span.TraceId)
+ }
+ }
+ moved, err := m.legacy.FindMovedSpans(ctx, traces, from.Truncate(time.Second), to.Add(time.Second))
+ if err != nil {
+ return err
+ }
+ kept := spans[:0]
+ for _, span := range spans {
+ if !moved[shared.MovedSpanKey(span.ProjectId, span.TraceId, span.SpanId)] {
+ kept = append(kept, span)
+ }
+ }
+ spans = kept
+ }
+ if len(spans) == 0 {
+ return nil
+ }
+ return SpanRepository.InsertAsync(ctx, spans)
+}
+
+func (m *mover) writeEndpoints(ctx context.Context, rows []models.Endpoint, from, to time.Time, resumed bool) error {
+ ids := make([]uuid.UUID, len(rows))
+ for i, row := range rows {
+ ids[i] = row.Id
+ }
+ moved, err := m.movedIds(ctx, "endpoints", ids, from, to, resumed)
+ if err != nil {
+ return err
+ }
+ var entities []models.Endpoint
+ var spans []models.Span
+ for _, row := range rows {
+ entity, span := mapEndpoint(row)
+ if moved[shared.MovedOccurrenceKey(entity.ProjectId, entity.Id, entity.TraceId, entity.SpanId, m.legacy.OccurrenceTime("endpoints", entity.RecordedAt))] {
+ continue
+ }
+ entities, spans = append(entities, entity), append(spans, span)
+ }
+ if err := m.insertSpans(ctx, spans, from, to, resumed); err != nil || len(entities) == 0 {
+ return err
+ }
+ return EndpointRepository.InsertAsync(ctx, entities)
+}
+
+func (m *mover) writeTasks(ctx context.Context, rows []models.Task, from, to time.Time, resumed bool) error {
+ ids := make([]uuid.UUID, len(rows))
+ for i, row := range rows {
+ ids[i] = row.Id
+ // Legacy OTel tasks stored their end time, with no upper bound on duration.
+ if start := row.RecordedAt.Add(-row.Duration); row.SpanId != "" && start.Before(from) {
+ from = start
+ }
+ }
+ moved, err := m.movedIds(ctx, "tasks", ids, from, to, resumed)
+ if err != nil {
+ return err
+ }
+ var entities []models.Task
+ var spans []models.Span
+ for _, row := range rows {
+ entity, span := mapTask(row)
+ if moved[shared.MovedOccurrenceKey(entity.ProjectId, entity.Id, entity.TraceId, entity.SpanId, m.legacy.OccurrenceTime("tasks", entity.RecordedAt))] {
+ continue
+ }
+ entities, spans = append(entities, entity), append(spans, span)
+ }
+ if err := m.insertSpans(ctx, spans, from, to, resumed); err != nil || len(entities) == 0 {
+ return err
+ }
+ return TaskRepository.InsertAsync(ctx, entities)
+}
+
+func (m *mover) writeAiTraces(ctx context.Context, rows []models.AiTrace, from, to time.Time, resumed bool) error {
+ ids := make([]uuid.UUID, len(rows))
+ for i, row := range rows {
+ ids[i] = row.Id
+ }
+ moved, err := m.movedIds(ctx, "ai_traces", ids, from, to, resumed)
+ if err != nil {
+ return err
+ }
+ var entities []models.AiTrace
+ var spans []models.Span
+ for _, row := range rows {
+ entity, span := mapAiTrace(row)
+ if moved[shared.MovedOccurrenceKey(entity.ProjectId, entity.Id, entity.TraceId, entity.SpanId, m.legacy.OccurrenceTime("ai_traces", entity.RecordedAt))] {
+ continue
+ }
+ entities, spans = append(entities, entity), append(spans, span)
+ }
+ if err := m.insertSpans(ctx, spans, from, to, resumed); err != nil || len(entities) == 0 {
+ return err
+ }
+ return AiTraceRepository.InsertAsync(ctx, entities)
+}
+
+func (m *mover) owners(ctx context.Context, ids map[uuid.UUID]bool, from, to time.Time) (ownerIndex, error) {
+ found := ownerIndex{}
+ if len(ids) == 0 {
+ return found, nil
+ }
+ list := make([]uuid.UUID, 0, len(ids))
+ for id := range ids {
+ list = append(list, id)
+ }
+ endpoints, tasks, aiTraces, err := m.legacy.FindOwners(ctx, list, from.Add(-ownerWindow), to.Add(ownerWindow))
+ if err != nil {
+ return nil, err
+ }
+ for _, row := range endpoints {
+ entity, _ := mapEndpoint(row)
+ found.add(row.ProjectId, row.Id, owner{entity.TraceId, entity.SpanId, entity.ServerName, row.RecordedAt})
+ }
+ for _, row := range tasks {
+ entity, _ := mapTask(row)
+ found.add(row.ProjectId, row.Id, owner{entity.TraceId, entity.SpanId, entity.ServerName, row.RecordedAt})
+ }
+ for _, row := range aiTraces {
+ entity, _ := mapAiTrace(row)
+ found.add(row.ProjectId, row.Id, owner{entity.TraceId, entity.SpanId, entity.ServerName, row.RecordedAt})
+ }
+ return found, nil
+}
+
+func (m *mover) writeExceptions(ctx context.Context, rows []shared.LegacyException, from, to time.Time, resumed bool) error {
+ ids, ownerIds := make([]uuid.UUID, len(rows)), map[uuid.UUID]bool{}
+ for i, row := range rows {
+ ids[i] = row.Id
+ if id, err := uuid.Parse(row.TraceId); err == nil {
+ ownerIds[id] = true
+ }
+ }
+ moved, err := m.movedIds(ctx, "exception_stack_traces", ids, from, to, resumed)
+ if err != nil {
+ return err
+ }
+ owners, err := m.owners(ctx, ownerIds, from, to)
+ if err != nil {
+ return err
+ }
+ var exceptions []models.ExceptionStackTrace
+ for _, row := range rows {
+ entity := mapException(row, owners)
+ if !moved[shared.MovedOccurrenceKey(entity.ProjectId, entity.Id, entity.TraceId, entity.SpanId, m.legacy.OccurrenceTime("exception_stack_traces", entity.RecordedAt))] {
+ exceptions = append(exceptions, entity)
+ }
+ }
+ if len(exceptions) == 0 {
+ return nil
+ }
+ return ExceptionStackTraceRepository.InsertAsync(ctx, exceptions)
+}
+
+func (m *mover) writeSpans(ctx context.Context, rows []shared.LegacySpan, from, to time.Time, resumed bool) error {
+ ownerIds := map[uuid.UUID]bool{}
+ for _, row := range rows {
+ ownerIds[row.OwnerId] = true
+ }
+ owners, err := m.owners(ctx, ownerIds, from, to)
+ if err != nil {
+ return err
+ }
+ spans := make([]models.Span, len(rows))
+ for i, row := range rows {
+ spans[i] = mapSpan(row, owners)
+ }
+ return m.insertSpans(ctx, spans, from, to, resumed)
+}
+
+type ownerKey struct {
+ projectId uuid.UUID
+ id uuid.UUID
+}
+
+type owner struct {
+ traceId, spanId, serverName string
+ recordedAt time.Time
+}
+
+// ownerIndex holds every old row an id answers to. An OTel id was a span id, which repeats across traces, and a retried
+// export stored the same row twice, so the owner of a span or an exception is the one recorded nearest to it.
+type ownerIndex map[ownerKey][]owner
+
+func (index ownerIndex) add(projectId, id uuid.UUID, found owner) {
+ key := ownerKey{projectId, id}
+ index[key] = append(index[key], found)
+}
+
+func (index ownerIndex) nearest(projectId, id uuid.UUID, at time.Time) (owner, bool) {
+ var best owner
+ candidates := index[ownerKey{projectId, id}]
+ for i, candidate := range candidates {
+ if i == 0 || candidate.recordedAt.Sub(at).Abs() < best.recordedAt.Sub(at).Abs() {
+ best = candidate
+ }
+ }
+ return best, len(candidates) > 0
+}
+
+const (
+ identityPrefix = "traceway.otel."
+ traceIdAttribute = "traceway.otel.trace_id"
+ zeroPadding = "0000000000000000"
+
+ spanKindInternal = 1
+ spanKindServer = 2
+ spanKindClient = 3
+ spanKindConsumer = 5
+ spanStatusError = 2
+)
+
+func hexId(id uuid.UUID) string { return shared.NormalizeTraceId(id.String()) }
+
+func validTraceId(id string) bool {
+ return len(id) == 32 && id != zeroPadding+zeroPadding && strings.Trim(id, "0123456789abcdef") == ""
+}
+
+// spanHex undoes the old storage of an OTel span id, which was kept as a UUID with eight zero bytes in front. A native
+// id is a real UUID and stays 32 characters.
+func spanHex(id string) string {
+ id = shared.NormalizeTraceId(id)
+ if len(id) == 32 && strings.HasPrefix(id, zeroPadding) {
+ return id[16:]
+ }
+ return id
+}
+
+// Prefer the original OTel trace ID over the old browser correlation attribute.
+func traceIdFromLegacy(attributes map[string]string, distributed string, id uuid.UUID) string {
+ if source := shared.NormalizeTraceId(attributes[traceIdAttribute]); validTraceId(source) {
+ return source
+ }
+ if source := shared.NormalizeTraceId(distributed); validTraceId(source) {
+ return source
+ }
+ return hexId(id)
+}
+
+func withoutIdentity(attributes map[string]string) map[string]string {
+ for key := range attributes {
+ if strings.HasPrefix(key, identityPrefix) {
+ delete(attributes, key)
+ }
+ }
+ return attributes
+}
+
+func mapEndpoint(row models.Endpoint) (models.Endpoint, models.Span) {
+ row.TraceId = traceIdFromLegacy(row.Attributes, row.TraceId, row.Id)
+ if row.SpanId = spanHex(row.SpanId); row.SpanId == "" {
+ row.SpanId = hexId(row.Id)
+ }
+ row.ParentSpanId, row.Attributes = "", withoutIdentity(row.Attributes)
+ span := models.Span{ProjectId: row.ProjectId, TraceId: row.TraceId, SpanId: row.SpanId, Name: row.Endpoint, StartTime: row.RecordedAt,
+ RecordedAt: row.RecordedAt, Duration: row.Duration, SpanKind: spanKindServer, ServiceName: row.ServerName, Attributes: row.Attributes}
+ if row.StatusCode >= 500 {
+ span.StatusCode = spanStatusError
+ }
+ return row, span
+}
+
+func mapTask(row models.Task) (models.Task, models.Span) {
+ row.TraceId = traceIdFromLegacy(row.Attributes, row.TraceId, row.Id)
+ kind := int32(spanKindConsumer)
+ if row.SpanId = spanHex(row.SpanId); row.SpanId == "" {
+ row.SpanId, kind = hexId(row.Id), spanKindInternal
+ } else {
+ // The old ingest recorded an OTel task at its end. V2 records everything at its span's start.
+ row.RecordedAt = row.RecordedAt.Add(-row.Duration)
+ }
+ row.ParentSpanId, row.Attributes = "", withoutIdentity(row.Attributes)
+ return row, models.Span{ProjectId: row.ProjectId, TraceId: row.TraceId, SpanId: row.SpanId, Name: row.TaskName, StartTime: row.RecordedAt,
+ RecordedAt: row.RecordedAt, Duration: row.Duration, SpanKind: kind, ServiceName: row.ServerName, Attributes: row.Attributes}
+}
+
+func mapAiTrace(row models.AiTrace) (models.AiTrace, models.Span) {
+ row.TraceId = traceIdFromLegacy(row.Attributes, row.TraceId, row.Id)
+ // The old table kept no span id. A call below the root was stored under its span id. The root was stored under the
+ // trace id, which serves as its span id here: it only has to be the same on the row and on the span.
+ row.SpanId, row.ParentSpanId, row.Attributes = spanHex(row.Id.String()), "", withoutIdentity(row.Attributes)
+ return row, models.Span{ProjectId: row.ProjectId, TraceId: row.TraceId, SpanId: row.SpanId, Name: row.TraceName, StartTime: row.RecordedAt,
+ RecordedAt: row.RecordedAt, Duration: row.Duration, SpanKind: spanKindClient, ServiceName: row.ServerName, Attributes: row.Attributes}
+}
+
+// mapException puts an exception on its owner's span. The old row named the owner, not the span it happened on, and
+// the owner's span is where a detail page and the issue page look first.
+func mapException(legacy shared.LegacyException, owners ownerIndex) models.ExceptionStackTrace {
+ row := legacy.ExceptionStackTrace
+ distributed := shared.NormalizeTraceId(legacy.DistributedTraceId)
+ if !validTraceId(distributed) {
+ distributed = ""
+ }
+ ownerId, err := uuid.Parse(row.TraceId)
+ row.Attributes, row.SpanId = withoutIdentity(row.Attributes), ""
+ switch found, known := owners.nearest(row.ProjectId, ownerId, row.RecordedAt); {
+ case err != nil:
+ row.TraceId, row.TraceType = distributed, ""
+ case known:
+ row.TraceId, row.SpanId = found.traceId, found.spanId
+ default:
+ row.TraceId, row.TraceType = hexId(ownerId), ""
+ }
+ return row
+}
+
+func mapSpan(row shared.LegacySpan, owners ownerIndex) models.Span {
+ span := models.Span{ProjectId: row.ProjectId, TraceId: hexId(row.OwnerId), SpanId: spanHex(row.Id.String()), ParentSpanId: spanHex(row.ParentSpanId),
+ Name: row.Name, StartTime: row.StartTime, RecordedAt: row.RecordedAt, Duration: row.Duration, Attributes: withoutIdentity(row.Attributes)}
+ found, known := owners.nearest(row.ProjectId, row.OwnerId, row.RecordedAt)
+ if known {
+ span.TraceId, span.ServiceName = found.traceId, found.serverName
+ }
+ if span.ParentSpanId == "" {
+ // Only a native span was stored without a parent. It hangs under its run, whose span id is the run's own id.
+ if span.ParentSpanId = hexId(row.OwnerId); known {
+ span.ParentSpanId = found.spanId
+ }
+ }
+ return span
+}
diff --git a/backend/app/repositories/telemetry/move_over_progress.go b/backend/app/repositories/telemetry/move_over_progress.go
new file mode 100644
index 00000000..dfdea9cc
--- /dev/null
+++ b/backend/app/repositories/telemetry/move_over_progress.go
@@ -0,0 +1,24 @@
+package telemetry
+
+import (
+ "context"
+ "database/sql"
+
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+type moveOverProgress struct{}
+
+func (moveOverProgress) FindProgress(ctx context.Context) ([]transactional.MoveOverDay, error) {
+ return db.ExecuteTransaction(func(tx *sql.Tx) ([]transactional.MoveOverDay, error) {
+ return transactional.MoveOverRepository.FindProgress(ctx, tx)
+ })
+}
+
+func (moveOverProgress) SaveProgress(ctx context.Context, day transactional.MoveOverDay) error {
+ _, err := db.ExecuteTransaction(func(tx *sql.Tx) (struct{}, error) {
+ return struct{}{}, transactional.MoveOverRepository.SaveProgress(ctx, tx, day)
+ })
+ return err
+}
diff --git a/backend/app/repositories/telemetry/move_over_test.go b/backend/app/repositories/telemetry/move_over_test.go
new file mode 100644
index 00000000..b9706b4c
--- /dev/null
+++ b/backend/app/repositories/telemetry/move_over_test.go
@@ -0,0 +1,368 @@
+package telemetry
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "os"
+ "sort"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func paddedSpan(last byte) uuid.UUID {
+ return uuid.UUID{8: last, 9: last, 10: last, 11: last, 12: last, 13: last, 14: last, 15: last}
+}
+
+func setupMoveOverProgress(t *testing.T) {
+ t.Helper()
+ path := "../../migrations/sqlite/0077_create_v2_move_over_days.up.sql"
+ if db.Driver == lit.PostgreSQL {
+ path = "../../migrations/pg/0143_create_v2_move_over_days.up.sql"
+ }
+ ddl, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.DB.Exec(string(ddl)); err != nil {
+ t.Fatal(err)
+ }
+ reset := func() {
+ if _, err := db.DB.Exec("DELETE FROM v2_move_over_days"); err != nil {
+ t.Fatal(err)
+ }
+ }
+ reset()
+ t.Cleanup(reset)
+}
+
+type failingMoveOverProgress struct {
+ moveOverProgress
+ failState string
+}
+
+var errProgressUnavailable = errors.New("progress store unavailable")
+
+func (p *failingMoveOverProgress) SaveProgress(ctx context.Context, day transactional.MoveOverDay) error {
+ if day.State == p.failState {
+ p.failState = ""
+ return errProgressUnavailable
+ }
+ return p.moveOverProgress.SaveProgress(ctx, day)
+}
+
+func TestMoveOverProgressFailureIsResumable(t *testing.T) {
+ for _, failState := range []string{transactional.MoveOverStarted, transactional.MoveOverDone} {
+ t.Run(failState, func(t *testing.T) {
+ setupTestDB(t)
+ setupMoveOverProgress(t)
+ legacyReset(t)
+ t.Cleanup(func() { legacyReset(t) })
+ ctx := context.Background()
+ at := time.Now().UTC().Truncate(moveOverDay).Add(10 * time.Hour)
+ seedLegacyDay(t, uuid.New(), at, uuid.New(), "GET /retry")
+ progress := &failingMoveOverProgress{failState: failState}
+ m := &mover{MoveOverOptions: MoveOverOptions{PageSize: 2, Log: func(string, ...any) {}}, legacy: LegacyRepository, progress: progress}
+ if err := m.run(ctx); !errors.Is(err, errProgressUnavailable) {
+ t.Fatalf("expected injected progress failure: %v", err)
+ }
+ saved, err := progress.FindProgress(ctx)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if failState == transactional.MoveOverStarted {
+ if len(saved) != 0 || moveOverRowCount(t, "endpoints_v2") != 0 || moveOverRowCount(t, "spans_v2") != 0 {
+ t.Fatal("data must not be written without a durable started marker")
+ }
+ } else if len(saved) != 1 || saved[0].State != transactional.MoveOverStarted || saved[0].Table != "endpoints" {
+ t.Fatalf("failed completion must leave the day resumable in the main DB: %+v", saved)
+ }
+ if err := m.run(ctx); err != nil {
+ t.Fatal(err)
+ }
+ for table, want := range map[string]int64{"endpoints_v2": 1, "tasks_v2": 1, "exceptions_v2": 1, "spans_v2": 5, "endpoints": 1, "tasks": 1, "exception_stack_traces": 1, "spans": 3} {
+ if got := moveOverRowCount(t, table); got != want {
+ t.Errorf("%s physical rows after resume = %d, want %d", table, got, want)
+ }
+ }
+ saved, err = progress.FindProgress(ctx)
+ if err != nil || len(saved) != 4 {
+ t.Fatalf("progress in main DB: %+v, %v", saved, err)
+ }
+ for _, day := range saved {
+ wantRows := int64(1)
+ if day.Table == "spans" {
+ wantRows = 3
+ }
+ if day.State != transactional.MoveOverDone || day.Day != at.Format(time.DateOnly) || day.Rows != wantRows {
+ t.Errorf("incomplete day: %+v", day)
+ }
+ }
+ })
+ }
+}
+
+func TestMoveOverIds(t *testing.T) {
+ trace, browser, run := uuid.MustParse("01020304-0506-0708-090a-0b0c0d0e0f10"), uuid.MustParse("a1a2a3a4-a5a6-a7a8-a9aa-abacadaeaf00"), uuid.New()
+ traceHex, browserHex := "0102030405060708090a0b0c0d0e0f10", "a1a2a3a4a5a6a7a8a9aaabacadaeaf00"
+
+ for name, test := range map[string]struct {
+ attributes map[string]string
+ distributed string
+ id uuid.UUID
+ wantTrace string
+ }{
+ "an OTel row names its trace in the attributes": {map[string]string{"traceway.otel.trace_id": traceHex}, trace.String(), trace, traceHex},
+ "the browser's id was filed as the distributed id": {map[string]string{"traceway.otel.trace_id": traceHex}, browser.String(), trace, traceHex},
+ "an older OTel row only has the distributed id": {nil, trace.String(), paddedSpan(7), traceHex},
+ "a native run without a distributed trace": {nil, "", run, hexId(run)},
+ "a distributed id that is not an id is ignored": {nil, "not-a-uuid", run, hexId(run)},
+ "a zero distributed id is ignored": {nil, uuid.Nil.String(), run, hexId(run)},
+ "a zero hex distributed id is ignored": {nil, hexId(uuid.Nil), run, hexId(run)},
+ "a zero OTel id falls back to the distributed id": {map[string]string{traceIdAttribute: hexId(uuid.Nil)}, trace.String(), run, traceHex},
+ "zero source ids fall back to the run": {map[string]string{traceIdAttribute: hexId(uuid.Nil)}, uuid.Nil.String(), run, hexId(run)},
+ "a zero distributed id does not become a link": {map[string]string{traceIdAttribute: traceHex}, uuid.Nil.String(), run, traceHex},
+ } {
+ if gotTrace := traceIdFromLegacy(test.attributes, test.distributed, test.id); gotTrace != test.wantTrace {
+ t.Errorf("%s: trace %q", name, gotTrace)
+ }
+ }
+ if got := spanHex(paddedSpan(0xab).String()); got != "abababababababab" {
+ t.Errorf("an OTel span id loses its zero padding: %q", got)
+ }
+ if got := spanHex(run.String()); got != hexId(run) {
+ t.Errorf("a native span id stays whole: %q", got)
+ }
+ if got := spanHex(""); got != "" {
+ t.Errorf("no id stays no id: %q", got)
+ }
+
+ recorded := time.Date(2026, 9, 1, 12, 0, 0, 0, time.UTC)
+ task, span := mapTask(models.Task{Id: paddedSpan(5), TaskName: "settle", TraceId: trace.String(), SpanId: paddedSpan(5).String(), RecordedAt: recorded, Duration: time.Minute,
+ Attributes: map[string]string{"traceway.otel.trace_id": traceHex, "messaging.system": "kafka"}})
+ if task.TraceId != traceHex || task.SpanId != "0505050505050505" || !task.RecordedAt.Equal(recorded.Add(-time.Minute)) || len(task.Attributes) != 1 {
+ t.Errorf("an OTel task moves to its start and sheds the identity attribute: %+v", task)
+ }
+ if span.SpanId != task.SpanId || span.TraceId != task.TraceId || span.SpanKind != spanKindConsumer || !span.StartTime.Equal(task.RecordedAt) {
+ t.Errorf("the task's own span: %+v", span)
+ }
+ native, _ := mapTask(models.Task{Id: run, RecordedAt: recorded, Duration: time.Minute})
+ if native.SpanId != hexId(run) || native.TraceId != hexId(run) || !native.RecordedAt.Equal(recorded) {
+ t.Errorf("a native task was recorded at its start already: %+v", native)
+ }
+
+ project := uuid.New()
+ owners := ownerIndex{}
+ owners.add(project, trace, owner{"ffffffffffffffffffffffffffffffff", "f1f1f1f1f1f1f1f1", "yesterday", recorded.Add(-20 * time.Hour)})
+ owners.add(project, trace, owner{traceHex, "e1e1e1e1e1e1e1e1", "gateway", recorded})
+ onEndpoint := mapException(shared.LegacyException{ExceptionStackTrace: models.ExceptionStackTrace{ProjectId: project, TraceId: trace.String(), TraceType: "endpoint", RecordedAt: recorded.Add(time.Second)}, DistributedTraceId: browser.String()}, owners)
+ if onEndpoint.TraceId != traceHex || onEndpoint.SpanId != "e1e1e1e1e1e1e1e1" || onEndpoint.TraceType != "endpoint" {
+ t.Errorf("an exception lands on the span of the owner recorded nearest to it: %+v", onEndpoint)
+ }
+ unowned := mapException(shared.LegacyException{ExceptionStackTrace: models.ExceptionStackTrace{ProjectId: project, TraceId: browser.String(), TraceType: "task"}}, owners)
+ if unowned.TraceId != browserHex || unowned.SpanId != "" || unowned.TraceType != "" {
+ t.Errorf("an exception whose owner is gone keeps the trace and claims no kind: %+v", unowned)
+ }
+ inBrowser := mapException(shared.LegacyException{ExceptionStackTrace: models.ExceptionStackTrace{ProjectId: project, TraceType: "endpoint"}, DistributedTraceId: browser.String()}, owners)
+ if inBrowser.TraceId != browserHex || inBrowser.SpanId != "" || inBrowser.TraceType != "" {
+ t.Errorf("a browser exception belongs to the browser's trace: %+v", inBrowser)
+ }
+
+ child := mapSpan(shared.LegacySpan{ProjectId: project, Id: paddedSpan(9), OwnerId: trace, ParentSpanId: paddedSpan(8).String(), RecordedAt: recorded}, owners)
+ if child.TraceId != traceHex || child.SpanId != "0909090909090909" || child.ParentSpanId != "0808080808080808" || child.ServiceName != "gateway" {
+ t.Errorf("an OTel span keeps its own parent and takes its owner's trace: %+v", child)
+ }
+ nativeChild := mapSpan(shared.LegacySpan{ProjectId: project, Id: uuid.New(), OwnerId: run}, owners)
+ if nativeChild.TraceId != hexId(run) || nativeChild.ParentSpanId != hexId(run) {
+ t.Errorf("a native span without a parent hangs under its run: %+v", nativeChild)
+ }
+}
+
+func TestMoveOverZeroDistributedIDKeepsNativeGraph(t *testing.T) {
+ setupTestDB(t)
+ setupMoveOverProgress(t)
+ legacyReset(t)
+ t.Cleanup(func() { legacyReset(t) })
+ ctx := context.Background()
+ at := time.Now().UTC().Truncate(moveOverDay).Add(10 * time.Hour)
+ project, run, child, exception := uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ legacyExec(t, `INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream, is_root)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, run, project, "GET /zero-header", int64(time.Second), at, 200, 0, "", "{}", "1.0", "api", &uuid.Nil, (*uuid.UUID)(nil), false, true)
+ legacyExec(t, `INSERT INTO spans (id, trace_id, project_id, name, start_time, duration, recorded_at, parent_span_id, attributes)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, child, run, project, "child", at, int64(time.Millisecond), at, (*uuid.UUID)(nil), "{}")
+ legacyExec(t, `INSERT INTO exception_stack_traces (id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, exception, project, &run, "endpoint", "zero-header", "Error: zero header", at, "{}", "1.0", "api", false, &uuid.Nil, (*uuid.UUID)(nil))
+ if err := RunMoveOver(ctx, MoveOverOptions{Log: func(string, ...any) {}}); err != nil {
+ t.Fatal(err)
+ }
+ endpoint, err := EndpointRepository.FindById(ctx, project, run, &at)
+ if err != nil || endpoint == nil || endpoint.TraceId != hexId(run) || endpoint.SpanId != hexId(run) {
+ t.Fatalf("migrated endpoint: %+v %v", endpoint, err)
+ }
+ graph, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, hexId(run), at)
+ if err != nil || len(graph.Spans) != 2 {
+ t.Fatalf("migrated root and child: %+v %v", graph, err)
+ }
+ for _, span := range graph.Spans {
+ if span.TraceId != hexId(run) || (span.SpanId == hexId(child) && span.ParentSpanId != hexId(run)) {
+ t.Fatalf("migrated span identity: %+v", span)
+ }
+ }
+ exc, err := ExceptionStackTraceRepository.FindById(ctx, project, exception, &at)
+ if err != nil || exc == nil || exc.TraceId != hexId(run) || exc.SpanId != hexId(run) {
+ t.Fatalf("migrated exception: %+v %v", exc, err)
+ }
+ owner, err := FindExceptionOwner(ctx, *exc)
+ if err != nil || owner == nil || owner.Name != endpoint.Endpoint {
+ t.Fatalf("migrated exception owner: %+v %v", owner, err)
+ }
+}
+
+// seedLegacyDay writes one request the way the released ingest stored it: a root endpoint filed under the trace id, a
+// worker task below it filed under its span id, their spans filed under their owner, and an exception on the task.
+func seedLegacyDay(t *testing.T, project uuid.UUID, at time.Time, trace uuid.UUID, endpointName string) {
+ t.Helper()
+ traceHex := hexId(trace)
+ taskId, endpointSpan := paddedSpan(0x7a), paddedSpan(0xe1)
+ attributes := fmt.Sprintf(`{"traceway.otel.trace_id":"%s","http.route":"/checkout"}`, traceHex)
+ legacyExec(t, `INSERT INTO endpoints (id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream, is_root)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, trace, project, endpointName, int64(time.Second), at, 500, 10, "10.0.0.1", attributes, "1.0.0", "gateway", &trace, &endpointSpan, false, true)
+ legacyExec(t, `INSERT INTO tasks (id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, taskId, project, "settle", int64(time.Minute), at.Add(time.Minute), "", attributes, "1.0.0", "worker", &trace, &taskId, false)
+ span := func(id, owner uuid.UUID, parent *uuid.UUID, name string) {
+ legacyExec(t, `INSERT INTO spans (id, trace_id, project_id, name, start_time, duration, recorded_at, parent_span_id, attributes) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
+ id, owner, project, name, at, int64(time.Millisecond), at, parent, `{"db.system":"postgresql"}`)
+ }
+ query, publish := paddedSpan(0xb1), paddedSpan(0xc1)
+ span(query, trace, &endpointSpan, "SELECT orders")
+ span(publish, trace, &endpointSpan, "publish settle")
+ span(paddedSpan(0x7b), taskId, &taskId, "UPDATE ledger")
+ legacyExec(t, `INSERT INTO exception_stack_traces (id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, uuid.New(), project, &taskId, "task", "ledger-locked", "LockError: ledger", at.Add(time.Second), "{}", "1.0.0", "worker", false, &trace, (*uuid.UUID)(nil))
+}
+
+func TestMoveOverBringsHistoryIntoTheV2Tables(t *testing.T) {
+ setupTestDB(t)
+ setupMoveOverProgress(t)
+ legacyReset(t)
+ t.Cleanup(func() { legacyReset(t) })
+ ctx := context.Background()
+ project := uuid.New()
+ today := time.Now().UTC().Truncate(24 * time.Hour).Add(10 * time.Hour)
+ traces := []uuid.UUID{uuid.New(), uuid.New(), uuid.New()}
+ for age, trace := range traces {
+ seedLegacyDay(t, project, today.AddDate(0, 0, -age), trace, fmt.Sprintf("GET /checkout/%d", age))
+ }
+
+ var moved []string
+ options := MoveOverOptions{PageSize: 2, Oldest: today.AddDate(0, 0, -1), Log: func(format string, args ...any) {
+ moved = append(moved, strings.SplitN(fmt.Sprintf(format, args...), ":", 2)[0])
+ }}
+ if err := RunMoveOver(ctx, options); err != nil {
+ t.Fatal(err)
+ }
+ var order []string
+ for _, line := range moved {
+ if strings.Contains(line, " endpoints") {
+ order = append(order, strings.Fields(line)[0])
+ }
+ }
+ if want := []string{today.Format(time.DateOnly), today.AddDate(0, 0, -1).Format(time.DateOnly)}; fmt.Sprint(order) != fmt.Sprint(want) {
+ t.Fatalf("newest day first, and nothing older than asked for: %v", order)
+ }
+
+ window := today
+ endpoints, err := EndpointRepository.FindByTraceIds(ctx, []string{hexId(traces[0])}, []uuid.UUID{project}, &window)
+ if err != nil || len(endpoints) != 1 {
+ t.Fatalf("moved endpoint: %+v %v", endpoints, err)
+ }
+ endpoint := endpoints[0]
+ if endpoint.Id != traces[0] || endpoint.SpanId != "e1e1e1e1e1e1e1e1" || endpoint.ParentSpanId != "" || !endpoint.IsRoot ||
+ endpoint.StatusCode != 500 || endpoint.ServerName != "gateway" || len(endpoint.Attributes) != 1 || endpoint.Attributes["http.route"] != "/checkout" {
+ t.Fatalf("the endpoint keeps its row id and gains real ids, and the identity attribute is gone: %+v", endpoint)
+ }
+ graph, err := SpanRepository.FindGraph(ctx, shared.NewSpanLookup(project, endpoint.TraceId, endpoint.SpanId, endpoint.RecordedAt))
+ if err != nil {
+ t.Fatal(err)
+ }
+ names := func(spans []models.Span) string {
+ out := make([]string, len(spans))
+ for i, span := range spans {
+ out[i] = span.Name
+ }
+ sort.Strings(out)
+ return strings.Join(out, ", ")
+ }
+ if got := names(graph.Spans); got != "SELECT orders, publish settle" {
+ t.Fatalf("the endpoint's waterfall survives the move: %q", got)
+ }
+ if graph.Spans[0].ServiceName != "gateway" || graph.Spans[0].Attributes["db.system"] != "postgresql" {
+ t.Fatalf("spans take their owner's service and keep their attributes: %+v", graph.Spans[0])
+ }
+
+ tasks, err := TaskRepository.FindByTraceIds(ctx, []string{endpoint.TraceId}, []uuid.UUID{project}, &window)
+ if err != nil || len(tasks) != 1 || tasks[0].SpanId != "7a7a7a7a7a7a7a7a" || !tasks[0].RecordedAt.Equal(today) {
+ t.Fatalf("the task joins the endpoint's trace and is recorded at its start: %+v %v", tasks, err)
+ }
+ taskGraph, err := SpanRepository.FindGraph(ctx, shared.NewSpanLookup(project, tasks[0].TraceId, tasks[0].SpanId, tasks[0].RecordedAt))
+ if err != nil || names(taskGraph.Spans) != "UPDATE ledger" {
+ t.Fatalf("a span filed under a task that was not the root finds the trace through its owner: %+v %v", taskGraph, err)
+ }
+ exceptions, err := ExceptionStackTraceRepository.FindAllByTraceId(ctx, project, endpoint.TraceId, &window)
+ if err != nil || len(exceptions) != 1 || exceptions[0].SpanId != tasks[0].SpanId || exceptions[0].TraceType != "task" {
+ t.Fatalf("the exception lands on its task's span: %+v %v", exceptions, err)
+ }
+ owner, err := FindExceptionOwner(ctx, exceptions[0])
+ if err != nil || owner == nil || owner.TraceType != "task" || owner.Name != "settle" {
+ t.Fatalf("the issue page finds the task: %+v %v", owner, err)
+ }
+ whole, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, endpoint.TraceId, today)
+ if err != nil || names(whole.Spans) != fmt.Sprintf("%s, SELECT orders, UPDATE ledger, publish settle, settle", endpoint.Endpoint) {
+ t.Fatalf("the whole trace holds the entities' own spans too: %q %v", names(whole.Spans), err)
+ }
+ if older, err := EndpointRepository.FindByTraceIds(ctx, []string{hexId(traces[2])}, []uuid.UUID{project}, nil); err != nil || len(older) != 0 {
+ t.Fatalf("a day before the oldest asked for stays where it is: %+v %v", older, err)
+ }
+
+ count := func() int {
+ found, err := EndpointRepository.FindByTraceIds(ctx, []string{hexId(traces[0]), hexId(traces[1]), hexId(traces[2])}, []uuid.UUID{project}, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ whole, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, hexId(traces[0]), today)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return len(found)*100 + len(whole.Spans)
+ }
+ before := count()
+ // A finished day is left alone.
+ if err := RunMoveOver(ctx, options); err != nil || count() != before {
+ t.Fatalf("running again must change nothing: %d -> %d %v", before, count(), err)
+ }
+ // A day that was interrupted is finished without writing its first rows twice.
+ for _, table := range []string{"endpoints", "tasks", "exception_stack_traces", "spans"} {
+ if err := (moveOverProgress{}).SaveProgress(ctx, transactional.MoveOverDay{Table: table, Day: today.Format(time.DateOnly), State: transactional.MoveOverStarted}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ options.Oldest = time.Time{}
+ if err := RunMoveOver(ctx, options); err != nil {
+ t.Fatal(err)
+ }
+ if after := count(); after != before+100 {
+ t.Fatalf("resuming adds the day that was left out and duplicates nothing: %d -> %d", before, after)
+ }
+ retried, err := ExceptionStackTraceRepository.FindAllByTraceId(ctx, project, endpoint.TraceId, &window)
+ if err != nil || len(retried) != 1 {
+ t.Fatalf("a resumed day must not write an exception twice: %d %v", len(retried), err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/otel_span_explorer_test.go b/backend/app/repositories/telemetry/otel_span_explorer_test.go
new file mode 100644
index 00000000..2a125de2
--- /dev/null
+++ b/backend/app/repositories/telemetry/otel_span_explorer_test.go
@@ -0,0 +1,366 @@
+package telemetry
+
+import (
+ "context"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func lookupUnder(span models.OtelSpan) shared.SpanLookup {
+ return shared.NewSpanLookup(span.ProjectId, span.TraceId, span.SpanId, span.RecordedAt)
+}
+
+func spanNames(spans []models.Span) []string {
+ names := make([]string, len(spans))
+ for i, span := range spans {
+ names[i] = span.Name
+ }
+ return names
+}
+
+func TestOtelGraphRootTakesOrphanedSubtrees(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, trace := uuid.New(), uuid.New()
+ rootID, childID, droppedID, orphanID, belowOrphanID := uuid.New(), uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ named := func(name string, id uuid.UUID, parent *uuid.UUID) models.OtelSpan {
+ span := canonicalSpan(project, trace, id, parent)
+ span.Name = name
+ return span
+ }
+ zeroPadded := func(id uuid.UUID) *uuid.UUID {
+ parent := uuid.UUID{}
+ copy(parent[8:], id[8:])
+ return &parent
+ }
+ root, child := named("root", rootID, nil), named("child", childID, zeroPadded(rootID))
+ // The span between the root and the orphan was dropped by a collector and never arrives.
+ orphan, belowOrphan := named("orphan", orphanID, zeroPadded(droppedID)), named("below orphan", belowOrphanID, zeroPadded(orphanID))
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{root, child, orphan, belowOrphan}); err != nil {
+ t.Fatal(err)
+ }
+ found, err := findSpans(ctx, root, &root.RecordedAt)
+ if err != nil || len(found) != 3 {
+ t.Fatalf("the trace root must show the subtree whose parent never arrived: %v %v", spanNames(found), err)
+ }
+ found, err = findSpans(ctx, child, &child.RecordedAt)
+ if err != nil || len(found) != 0 {
+ t.Fatalf("an occurrence below the root could be the wrong owner and must not take orphans: %v %v", spanNames(found), err)
+ }
+}
+
+func TestOtelReadsAreBoundToTheLookupWindow(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, trace := uuid.New(), uuid.New()
+ rootID := uuid.New()
+ parent := uuid.UUID{}
+ copy(parent[8:], rootID[8:])
+ root := canonicalSpan(project, trace, rootID, nil)
+ near, far := canonicalSpan(project, trace, uuid.New(), &parent), canonicalSpan(project, trace, uuid.New(), &parent)
+ near.Name, far.Name = "near", "three days later"
+ far.StartTime = far.StartTime.Add(72 * time.Hour)
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{root, near, far}); err != nil {
+ t.Fatal(err)
+ }
+ found, err := findSpans(ctx, root, &root.RecordedAt)
+ if err != nil || len(found) != 1 || found[0].Name != "near" {
+ t.Fatalf("a read is held to 24 hours either side of the occurrence: %v %v", spanNames(found), err)
+ }
+ if payload, err := OtelSpanRepository.FindOTLP(ctx, project, far.TraceId, far.SpanId, root.RecordedAt); err != nil || payload != nil {
+ t.Fatalf("the payload read must use the same window: %d bytes %v", len(payload), err)
+ }
+ if payload, err := OtelSpanRepository.FindOTLP(ctx, project, far.TraceId, far.SpanId, far.StartTime); err != nil || payload == nil {
+ t.Fatalf("anchored on its own time the span is found: %v", err)
+ }
+}
+
+func TestOtelSpanSearch(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, otherProject := uuid.New(), uuid.New()
+ now := time.Now().UTC().Truncate(time.Microsecond)
+ request, job := uuid.New(), uuid.New()
+ build := func(owner, trace uuid.UUID, name, service string, kind, status int32, duration time.Duration, start time.Time, attributes map[string]string) models.OtelSpan {
+ span := canonicalSpan(owner, trace, uuid.New(), nil)
+ span.Name, span.ServiceName, span.SpanKind, span.StatusCode, span.Duration, span.StartTime, span.Attributes = name, service, kind, status, duration, start, attributes
+ return span
+ }
+ spans := []models.OtelSpan{
+ build(project, request, "GET /users", "api", 2, 0, 50*time.Millisecond, now.Add(-time.Minute), map[string]string{"http.route": "/users"}),
+ build(project, request, "SELECT users", "api", 3, 2, 5*time.Millisecond, now.Add(-50*time.Second), map[string]string{"db.system": "postgresql"}),
+ build(project, job, "process job", "worker", 5, 0, 2*time.Second, now.Add(-30*time.Second), nil),
+ build(project, uuid.New(), "three days ago", "api", 2, 0, time.Millisecond, now.Add(-72*time.Hour), nil),
+ build(otherProject, uuid.New(), "another tenant", "api", 2, 0, time.Millisecond, now.Add(-time.Minute), nil),
+ }
+ if _, err := OtelSpanRepository.InsertAsync(ctx, spans); err != nil {
+ t.Fatal(err)
+ }
+ kind, status := int32(5), int32(2)
+ hour := shared.OtelSpanSearch{ProjectId: project, From: now.Add(-time.Hour), To: now.Add(time.Minute), Page: 1, PageSize: 50}
+ with := func(change func(*shared.OtelSpanSearch)) shared.OtelSpanSearch {
+ search := hour
+ change(&search)
+ return search
+ }
+ for _, tt := range []struct {
+ name string
+ search shared.OtelSpanSearch
+ want []string
+ total uint64
+ }{
+ {"newest first, held to the range and the project", hour, []string{"process job", "SELECT users", "GET /users"}, 3},
+ {"a wider range reaches older spans", with(func(s *shared.OtelSpanSearch) { s.From = now.Add(-7 * 24 * time.Hour) }), []string{"process job", "SELECT users", "GET /users", "three days ago"}, 4},
+ {"service", with(func(s *shared.OtelSpanSearch) { s.Service = "worker" }), []string{"process job"}, 1},
+ {"name, any case", with(func(s *shared.OtelSpanSearch) { s.Name = "select" }), []string{"SELECT users"}, 1},
+ {"kind", with(func(s *shared.OtelSpanSearch) { s.Kind = &kind }), []string{"process job"}, 1},
+ {"errors", with(func(s *shared.OtelSpanSearch) { s.Status = &status }), []string{"SELECT users"}, 1},
+ {"slower than", with(func(s *shared.OtelSpanSearch) { s.MinDuration = time.Second }), []string{"process job"}, 1},
+ {"faster than", with(func(s *shared.OtelSpanSearch) { s.MaxDuration = 10 * time.Millisecond }), []string{"SELECT users"}, 1},
+ {"attribute with a dotted key", with(func(s *shared.OtelSpanSearch) {
+ s.Attributes = []shared.OtelAttributeFilter{{Key: "http.route", Value: "/users"}}
+ }), []string{"GET /users"}, 1},
+ {"one trace", with(func(s *shared.OtelSpanSearch) { s.TraceId = spans[0].TraceId }), []string{"SELECT users", "GET /users"}, 2},
+ {"slowest first", with(func(s *shared.OtelSpanSearch) { s.OrderBy = "duration desc" }), []string{"process job", "GET /users", "SELECT users"}, 3},
+ {"second page", with(func(s *shared.OtelSpanSearch) { s.Page, s.PageSize = 2, 2 }), []string{"GET /users"}, 3},
+ } {
+ t.Run(tt.name, func(t *testing.T) {
+ found, total, err := OtelSpanRepository.Search(ctx, tt.search)
+ if err != nil {
+ t.Fatal(err)
+ }
+ names := make([]string, len(found))
+ for i, span := range found {
+ names[i] = span.Name
+ }
+ if total != tt.total || len(names) != len(tt.want) {
+ t.Fatalf("got %v of %d, want %v of %d", names, total, tt.want, tt.total)
+ }
+ for i := range names {
+ if names[i] != tt.want[i] {
+ t.Fatalf("got %v, want %v", names, tt.want)
+ }
+ }
+ })
+ }
+ first, _, err := OtelSpanRepository.Search(ctx, with(func(s *shared.OtelSpanSearch) { s.Service = "worker" }))
+ if err != nil || len(first) != 1 || first[0].ServiceName != "worker" || first[0].SpanKind != 5 || first[0].Duration != 2*time.Second || !first[0].StartTime.Equal(spans[2].StartTime) {
+ t.Fatalf("a result carries the fields the list shows: %+v %v", first, err)
+ }
+ services, err := OtelSpanRepository.Services(ctx, project, hour.From, hour.To)
+ if err != nil || len(services) != 2 || services[0] != "api" || services[1] != "worker" {
+ t.Fatalf("services in range, busiest first: %v %v", services, err)
+ }
+ trace, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, spans[0].TraceId, now)
+ if err != nil || len(trace.Spans) != 2 || trace.Status.State != models.SpanGraphComplete || trace.Spans[0].Name != "GET /users" {
+ t.Fatalf("the whole trace, oldest first: %+v %v", trace, err)
+ }
+ if elsewhere, err := SpanRepository.FindTrace(ctx, []uuid.UUID{otherProject}, spans[0].TraceId, now); err != nil || len(elsewhere.Spans) != 0 {
+ t.Fatalf("another project must not see the trace: %+v %v", elsewhere, err)
+ }
+}
+
+func TestOtelWholeTraceKeepsTheMostImportantSpansPastTheCap(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ previous := shared.MaxOtelGraphRows
+ shared.MaxOtelGraphRows = 5
+ t.Cleanup(func() { shared.MaxOtelGraphRows = previous })
+
+ project, trace := uuid.New(), uuid.New()
+ now := time.Now().UTC().Truncate(time.Microsecond)
+ rootID := uuid.New()
+ build := func(name string, id uuid.UUID, parent *uuid.UUID, kind, status int32, duration time.Duration, offset time.Duration) models.OtelSpan {
+ span := canonicalSpan(project, trace, id, parent)
+ span.Name, span.SpanKind, span.StatusCode, span.Duration, span.StartTime = name, kind, status, duration, now.Add(offset)
+ return span
+ }
+ spans := []models.OtelSpan{
+ build("root", rootID, nil, 1, 0, time.Second, 0),
+ build("fast 1", uuid.New(), &rootID, 1, 0, time.Millisecond, time.Millisecond),
+ build("fast 2", uuid.New(), &rootID, 1, 0, 2*time.Millisecond, 2*time.Millisecond),
+ build("fast 3", uuid.New(), &rootID, 1, 0, 3*time.Millisecond, 3*time.Millisecond),
+ build("slow", uuid.New(), &rootID, 1, 0, 400*time.Millisecond, 4*time.Millisecond),
+ build("slower", uuid.New(), &rootID, 3, 0, 500*time.Millisecond, 5*time.Millisecond),
+ build("consumer", uuid.New(), &rootID, 5, 0, time.Microsecond, 6*time.Millisecond),
+ build("short error", uuid.New(), &rootID, 1, 2, time.Microsecond, 7*time.Millisecond),
+ }
+ if _, err := OtelSpanRepository.InsertAsync(ctx, spans); err != nil {
+ t.Fatal(err)
+ }
+ graph, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, spans[0].TraceId, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // Cut by start time, the earliest five would be the root and the fast spans, and the error would be gone.
+ want := []string{"root", "slow", "slower", "consumer", "short error"}
+ if got := spanNames(graph.Spans); len(got) != len(want) {
+ t.Fatalf("kept %v, want %v", got, want)
+ } else {
+ for i := range want {
+ if got[i] != want[i] {
+ t.Fatalf("kept %v, want %v in start order", got, want)
+ }
+ }
+ }
+ status := graph.Status
+ if status.State != models.SpanGraphPartial || len(status.Reasons) != 2 || status.Reasons[0] != models.SpanGraphRowLimit || status.Reasons[1] != models.SpanGraphMostImportant {
+ t.Fatalf("status %+v", status)
+ }
+
+ shared.MaxOtelGraphRows = previous
+ whole, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, spans[0].TraceId, now)
+ if err != nil || len(whole.Spans) != len(spans) || whole.Status.State != models.SpanGraphComplete {
+ t.Fatalf("a trace under the cap is returned whole: %d spans, %+v, %v", len(whole.Spans), whole.Status, err)
+ }
+ if whole.Spans[0].Name != "root" || whole.Spans[7].Name != "short error" {
+ t.Fatalf("spans come back in start order whatever order they were read in: %v", spanNames(whole.Spans))
+ }
+}
+
+func TestOtelWholeTraceLeavesAttributesToThePopover(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, trace := uuid.New(), uuid.New()
+ now := time.Now().UTC().Truncate(time.Microsecond)
+ statement := "SELECT * FROM orders WHERE note = 'žž' AND " + strings.Repeat("id = 1 OR ", 40)
+ query := canonicalSpan(project, trace, uuid.New(), nil)
+ query.Name, query.StartTime, query.Attributes = "SELECT shop.orders", now, map[string]string{"db.query.text": statement, "db.system.name": "postgresql"}
+ legacy := canonicalSpan(project, trace, uuid.New(), nil)
+ legacy.Name, legacy.StartTime, legacy.Attributes = "old driver", now.Add(time.Millisecond), map[string]string{"db.statement": "DELETE FROM carts"}
+ plain := canonicalSpan(project, trace, uuid.New(), nil)
+ plain.Name, plain.StartTime = "no attributes", now.Add(2*time.Millisecond)
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{query, legacy, plain}); err != nil {
+ t.Fatal(err)
+ }
+
+ graph, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, query.TraceId, now)
+ if err != nil || len(graph.Spans) != 3 {
+ t.Fatalf("%v %v", graph, err)
+ }
+ for _, span := range graph.Spans {
+ if len(span.Attributes) != 0 || span.AttributesOmitted {
+ t.Fatalf("%s: the whole trace read carries no attributes: %+v", span.Name, span)
+ }
+ }
+ preview := []rune(graph.Spans[0].DbStatement)
+ if len(preview) != shared.OtelStatementPreviewLength || string(preview) != string([]rune(statement)[:shared.OtelStatementPreviewLength]) {
+ t.Fatalf("statement preview is the first %d characters, got %d: %q", shared.OtelStatementPreviewLength, len(preview), graph.Spans[0].DbStatement)
+ }
+ if graph.Spans[1].DbStatement != "DELETE FROM carts" || graph.Spans[2].DbStatement != "" {
+ t.Fatalf("db.statement is the fallback and other spans have none: %q %q", graph.Spans[1].DbStatement, graph.Spans[2].DbStatement)
+ }
+
+ loaded, err := SpanRepository.FindSpanAttributes(ctx, project, query.TraceId, query.SpanId, now)
+ if err != nil || loaded == nil || loaded.Attributes["db.query.text"] != statement || loaded.Attributes["db.system.name"] != "postgresql" {
+ t.Fatalf("popover attributes: %+v %v", loaded, err)
+ }
+ if empty, err := SpanRepository.FindSpanAttributes(ctx, project, plain.TraceId, plain.SpanId, now); err != nil || empty == nil || len(empty.Attributes) != 0 {
+ t.Fatalf("a span without attributes is found and empty: %+v %v", empty, err)
+ }
+ if missing, err := SpanRepository.FindSpanAttributes(ctx, project, query.TraceId, "ffffffffffffffff", now); err != nil || missing != nil {
+ t.Fatalf("unknown span: %+v %v", missing, err)
+ }
+ if outside, err := SpanRepository.FindSpanAttributes(ctx, project, query.TraceId, query.SpanId, now.Add(-72*time.Hour)); err != nil || outside != nil {
+ t.Fatalf("outside the 24 hour window: %+v %v", outside, err)
+ }
+ if other, err := SpanRepository.FindSpanAttributes(ctx, uuid.New(), query.TraceId, query.SpanId, now); err != nil || other != nil {
+ t.Fatalf("another project: %+v %v", other, err)
+ }
+}
+
+func TestOtelGraphRootAdoptsOnlyItsOwnOrphans(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ gateway, warehouse, trace, otherTrace := uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ named := func(project, trace uuid.UUID, name string, id uuid.UUID, parent *uuid.UUID) models.OtelSpan {
+ span := canonicalSpan(project, trace, id, parent)
+ span.Name = name
+ return span
+ }
+ rootID, elsewhereID, unrelatedRootID, missingHop := uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ root := named(gateway, trace, "GET /api/stock", rootID, nil)
+ // Same trace, another project: its parent is a hop that reports to a third project, so it looks like an orphan.
+ elsewhere := named(warehouse, trace, "GET /shelves", elsewhereID, &missingHop)
+ unrelatedRoot := named(gateway, otherTrace, "GET /health", unrelatedRootID, nil)
+ unrelatedOrphan := named(gateway, otherTrace, "lost child", uuid.New(), &missingHop)
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{root, elsewhere, unrelatedRoot, unrelatedOrphan}); err != nil {
+ t.Fatal(err)
+ }
+ // The distributed trace card reads entities of several projects and traces in one batch.
+ graphs, err := SpanRepository.FindGraphs(ctx, []shared.SpanLookup{
+ lookupUnder(root),
+ lookupUnder(elsewhere),
+ lookupUnder(unrelatedRoot),
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := spanNames(graphs[lookupUnder(root).Owner()].Spans); len(got) != 0 {
+ t.Fatalf("a root must not adopt another project's or another trace's spans, got %v", got)
+ }
+ if got := spanNames(graphs[lookupUnder(unrelatedRoot).Owner()].Spans); len(got) != 1 || got[0] != "lost child" {
+ t.Fatalf("a root still adopts the orphans of its own trace, got %v", got)
+ }
+}
+
+func TestOtelWholeTraceReadsEveryProjectItIsGiven(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ gateway, inventory, warehouse, stranger, trace := uuid.New(), uuid.New(), uuid.New(), uuid.New(), uuid.New()
+ now := time.Now().UTC().Truncate(time.Microsecond)
+ ids := []uuid.UUID{uuid.New(), uuid.New(), uuid.New(), uuid.New(), uuid.New()}
+ hop := func(project uuid.UUID, name string, index int) models.OtelSpan {
+ var parent *uuid.UUID
+ if index > 0 {
+ parent = &ids[index-1]
+ }
+ span := canonicalSpan(project, trace, ids[index], parent)
+ span.Name, span.StartTime = name, now.Add(time.Duration(index)*time.Millisecond)
+ return span
+ }
+ outsider := canonicalSpan(stranger, trace, uuid.New(), &ids[0])
+ outsider.Name = "not one of the projects asked for"
+ spans := []models.OtelSpan{hop(gateway, "GET /api/stock", 0), hop(gateway, "grpc client", 1), hop(inventory, "inventory.Stock/Get", 2),
+ hop(inventory, "http client", 3), hop(warehouse, "GET /shelves", 4), outsider}
+ if _, err := OtelSpanRepository.InsertAsync(ctx, spans); err != nil {
+ t.Fatal(err)
+ }
+
+ whole, err := SpanRepository.FindTrace(ctx, []uuid.UUID{gateway, inventory, warehouse}, spans[0].TraceId, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []string{"GET /api/stock", "grpc client", "inventory.Stock/Get", "http client", "GET /shelves"}
+ if got := spanNames(whole.Spans); len(got) != len(want) {
+ t.Fatalf("one trace over three projects, in start order: got %v", got)
+ } else {
+ for i := range want {
+ if got[i] != want[i] {
+ t.Fatalf("one trace over three projects, in start order: got %v", got)
+ }
+ }
+ }
+ if whole.Spans[2].ProjectId != inventory || whole.Spans[4].ProjectId != warehouse {
+ t.Fatalf("each span keeps the project it was reported to: %+v", whole.Spans)
+ }
+ // The distributed trace nests its entities from the same read, without attributes.
+ parents, err := SpanRepository.FindTraceParents(ctx, []uuid.UUID{gateway, inventory, warehouse}, spans[0].TraceId, now)
+ if err != nil || len(parents) != 4 || parents[spans[4].SpanId] != spans[3].SpanId || parents[spans[2].SpanId] != spans[1].SpanId {
+ t.Fatalf("every hop points at its parent, across projects: %v %v", parents, err)
+ }
+ if _, isRoot := parents[spans[0].SpanId]; isRoot {
+ t.Fatalf("the root has no parent: %v", parents)
+ }
+ if one, err := SpanRepository.FindTrace(ctx, []uuid.UUID{gateway}, spans[0].TraceId, now); err != nil || len(one.Spans) != 2 {
+ t.Fatalf("a single project still reads only itself: %v %v", one, err)
+ }
+ if none, err := SpanRepository.FindTrace(ctx, nil, spans[0].TraceId, now); err != nil || len(none.Spans) != 0 || none.Status.State != models.SpanGraphComplete {
+ t.Fatalf("no projects, no spans: %v %v", none, err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/otel_span_plan_duckdb_test.go b/backend/app/repositories/telemetry/otel_span_plan_duckdb_test.go
new file mode 100644
index 00000000..44d890b6
--- /dev/null
+++ b/backend/app/repositories/telemetry/otel_span_plan_duckdb_test.go
@@ -0,0 +1,71 @@
+//go:build telemetry_duckdb
+
+package telemetry
+
+import (
+ "context"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func TestOtelSpanIndexedQueries(t *testing.T) {
+ setupTestDB(t)
+ project := uuid.New()
+ spans := make([]models.OtelSpan, 10000)
+ for i := range spans {
+ spans[i] = canonicalSpan(project, uuid.New(), uuid.New(), nil)
+ }
+ start := time.Now()
+ if _, err := OtelSpanRepository.InsertAsync(context.Background(), spans); err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("DuckDB canonical ingest: %d spans in %s", len(spans), time.Since(start))
+ lookup := shared.SpanLookup{ProjectId: project, TraceId: spans[0].TraceId, SpanId: spans[0].SpanId, RecordedAt: &spans[0].RecordedAt}
+ query, args := shared.OtelTopologyQuery([]shared.SpanLookup{lookup}, true, 0, func(t time.Time) any { return t.UTC() })
+ rows, err := db.TelemetryDB.Query("EXPLAIN ANALYZE "+query, args...)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var plan string
+ for rows.Next() {
+ var key, value string
+ if err := rows.Scan(&key, &value); err != nil {
+ t.Fatal(err)
+ }
+ plan += value
+ }
+ err = rows.Err()
+ rows.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(plan, "Index Scan") {
+ t.Fatalf("unindexed source lookup: %s", plan)
+ }
+ t.Log("source lookup uses Index Scan")
+
+ start = time.Now()
+ for range 100 {
+ if _, err := findSpans(context.Background(), spans[0], &spans[0].RecordedAt); err != nil {
+ t.Fatal(err)
+ }
+ }
+ t.Logf("DuckDB canonical lookup: %s per detail (100 reads, 10000 stored spans)", time.Since(start)/100)
+ for _, statement := range []string{"DROP INDEX idx_spans_v2_trace_key", "DELETE FROM spans_v2"} {
+ if _, err := db.TelemetryDB.Exec(statement); err != nil {
+ t.Fatal(err)
+ }
+ }
+ start = time.Now()
+ if _, err := OtelSpanRepository.InsertAsync(context.Background(), spans); err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("canonical ingest without lookup indexes: %d spans in %s", len(spans), time.Since(start))
+
+}
diff --git a/backend/app/repositories/telemetry/otel_span_plan_sqlite_test.go b/backend/app/repositories/telemetry/otel_span_plan_sqlite_test.go
new file mode 100644
index 00000000..8c678a90
--- /dev/null
+++ b/backend/app/repositories/telemetry/otel_span_plan_sqlite_test.go
@@ -0,0 +1,76 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package telemetry
+
+import (
+ "context"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+func TestOtelSpanIndexedQueries(t *testing.T) {
+ setupTestDB(t)
+ project := uuid.New()
+ spans := make([]models.OtelSpan, 10000)
+ for i := range spans {
+ spans[i] = canonicalSpan(project, uuid.New(), uuid.New(), nil)
+ }
+ start := time.Now()
+ if _, err := OtelSpanRepository.InsertAsync(context.Background(), spans); err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("SQLite canonical ingest: %d spans in %s", len(spans), time.Since(start))
+ if _, err := db.TelemetryDB.Exec("ANALYZE spans_v2"); err != nil {
+ t.Fatal(err)
+ }
+ lookup := shared.SpanLookup{ProjectId: project, TraceId: spans[0].TraceId, SpanId: spans[0].SpanId, RecordedAt: &spans[0].RecordedAt}
+ query, args := shared.OtelTopologyQuery([]shared.SpanLookup{lookup}, false, 0, func(t time.Time) any { return sqlitetypes.NewSQLiteTime(t) })
+ rows, err := db.TelemetryDB.Query("EXPLAIN QUERY PLAN "+query, args...)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var plan string
+ for rows.Next() {
+ var id, parent, unused int
+ var detail string
+ if err := rows.Scan(&id, &parent, &unused, &detail); err != nil {
+ t.Fatal(err)
+ }
+ plan += detail
+ }
+ err = rows.Err()
+ rows.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(plan, "USING INDEX idx_spans_v2_project_id_trace_id_recorded_at") {
+ t.Fatalf("unindexed source lookup: %s", plan)
+ }
+ t.Log("source lookup uses USING INDEX idx_spans_v2_project_id_trace_id_recorded_at")
+
+ start = time.Now()
+ for range 100 {
+ if _, err := findSpans(context.Background(), spans[0], &spans[0].RecordedAt); err != nil {
+ t.Fatal(err)
+ }
+ }
+ t.Logf("SQLite canonical lookup: %s per detail (100 reads, 10000 stored spans)", time.Since(start)/100)
+ for _, statement := range []string{"DROP INDEX idx_spans_v2_project_id_trace_id_recorded_at", "DELETE FROM spans_v2"} {
+ if _, err := db.TelemetryDB.Exec(statement); err != nil {
+ t.Fatal(err)
+ }
+ }
+ start = time.Now()
+ if _, err := OtelSpanRepository.InsertAsync(context.Background(), spans); err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("canonical ingest without lookup indexes: %d spans in %s", len(spans), time.Since(start))
+
+}
diff --git a/backend/app/repositories/telemetry/otel_span_rejected_rows_test.go b/backend/app/repositories/telemetry/otel_span_rejected_rows_test.go
new file mode 100644
index 00000000..624b2cdb
--- /dev/null
+++ b/backend/app/repositories/telemetry/otel_span_rejected_rows_test.go
@@ -0,0 +1,34 @@
+package telemetry
+
+import (
+ "context"
+ "testing"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+func TestOtelSpanInsertSkipsAndCountsAnUnstorableRow(t *testing.T) {
+ setupTestDB(t)
+ project, trace := uuid.New(), uuid.New()
+ first := canonicalSpan(project, trace, uuid.MustParse("00000000-0000-0000-0101-010101010101"), nil)
+ last := canonicalSpan(project, trace, uuid.MustParse("00000000-0000-0000-0303-030303030303"), nil)
+ unstorable := canonicalSpan(project, trace, uuid.MustParse("00000000-0000-0000-0202-020202020202"), nil)
+ unstorable.SpanId = "not hexadecimal"
+
+ before, _, _ := db.GetTelemetryIngestCounters()
+ rejected, err := OtelSpanRepository.InsertAsync(context.Background(), []models.OtelSpan{first, unstorable, last})
+ after, _, _ := db.GetTelemetryIngestCounters()
+ if err != nil || rejected != 1 {
+ t.Fatalf("one bad row must not fail the batch: rejected=%d err=%v", rejected, err)
+ }
+ if after[shared.SpansTable] != before[shared.SpansTable]+1 {
+ t.Fatalf("the rejected row must be counted: %d -> %d", before[shared.SpansTable], after[shared.SpansTable])
+ }
+ stored, err := OtelSpanRepository.FindTraceTopology(context.Background(), []shared.SpanLookup{{ProjectId: project, TraceId: first.TraceId}}, 0)
+ if err != nil || len(stored) != 2 {
+ t.Fatalf("rows around the rejected one must persist: %d %v", len(stored), err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/otel_span_repository_test.go b/backend/app/repositories/telemetry/otel_span_repository_test.go
new file mode 100644
index 00000000..0a165b78
--- /dev/null
+++ b/backend/app/repositories/telemetry/otel_span_repository_test.go
@@ -0,0 +1,131 @@
+package telemetry
+
+import (
+ "context"
+ "encoding/hex"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+// canonicalSpan takes its ids as UUIDs for the fixtures' convenience: the trace id is all 16 bytes, a span id the last 8.
+func canonicalSpan(project, trace, id uuid.UUID, parent *uuid.UUID) models.OtelSpan {
+ span := models.OtelSpan{Span: models.Span{
+ ProjectId: project, TraceId: hex.EncodeToString(trace[:]), SpanId: hex.EncodeToString(id[8:]),
+ Name: "operation", StartTime: time.Now().UTC(), RecordedAt: time.Now().UTC(), Duration: time.Second,
+ }}
+ if parent != nil {
+ span.ParentSpanId = hex.EncodeToString(parent[8:])
+ }
+ return span
+}
+
+func TestOtelGraphLateParentsAndIsolation(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, trace := uuid.New(), uuid.New()
+ rootID := uuid.MustParse("00000000-0000-0000-0101-010101010101")
+ childID := uuid.MustParse("00000000-0000-0000-0202-020202020202")
+ grandchildID := uuid.MustParse("00000000-0000-0000-0303-030303030303")
+ root := canonicalSpan(project, trace, rootID, nil)
+ child := canonicalSpan(project, trace, childID, &rootID)
+ grandchild := canonicalSpan(project, trace, grandchildID, &childID)
+ // Still inside the 24 hours either side of the root that every read is held to.
+ grandchild.StartTime = root.StartTime.Add(23 * time.Hour)
+ grandchild.RecordedAt = grandchild.StartTime
+ reusedTrace := canonicalSpan(project, uuid.New(), childID, &rootID)
+ reusedProject := canonicalSpan(uuid.New(), trace, childID, &rootID)
+ // Child exports first; unrelated reused span IDs must not attach to this root.
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{grandchild, reusedTrace, reusedProject}); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{root}); err != nil {
+ t.Fatal(err)
+ }
+ // Its parent has not arrived, so the trace's root entity holds it until the child does.
+ found, err := findSpans(ctx, root, &root.RecordedAt)
+ if err != nil || len(found) != 1 || found[0].SpanId != grandchild.SpanId {
+ t.Fatalf("orphan under the root entity: %v, %v", found, err)
+ }
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{child, grandchild}); err != nil {
+ t.Fatal(err)
+ }
+ found, err = findSpans(ctx, root, &root.RecordedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(found) != 2 {
+ t.Fatalf("want child and long-running grandchild, got %d", len(found))
+ }
+ if found[0].SpanId != child.SpanId || found[1].SpanId != grandchild.SpanId {
+ t.Fatalf("wrong tree: %+v", found)
+ }
+ for _, span := range found {
+ if span.ProjectId != project || span.TraceId != root.TraceId || span.ParentSpanId == "" {
+ t.Fatalf("lost identity: %+v", span)
+ }
+ }
+ found, err = findSpans(ctx, child, &child.RecordedAt)
+ if err != nil || len(found) != 1 || found[0].SpanId != grandchild.SpanId {
+ t.Fatalf("nested promoted subtree: %v %v", found, err)
+ }
+}
+
+func TestOtelGraphCyclesAndDuplicateExports(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ project, trace := uuid.New(), uuid.New()
+ aID := uuid.MustParse("00000000-0000-0000-0101-010101010101")
+ bID := uuid.MustParse("00000000-0000-0000-0202-020202020202")
+ a, b := canonicalSpan(project, trace, aID, &bID), canonicalSpan(project, trace, bID, &aID)
+ duplicate := b
+ duplicate.Duration = 2 * time.Second
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{a, b, duplicate, a, b}); err != nil {
+ t.Fatal(err)
+ }
+ found, err := findSpans(ctx, a, &a.RecordedAt)
+ if err != nil || len(found) != 1 || found[0].Duration != 2*time.Second {
+ t.Fatalf("cycle or duplicate: %+v %v", found, err)
+ }
+}
+
+func TestOtelPayloadContextRoundTrip(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ span := canonicalSpan(uuid.New(), uuid.New(), uuid.New(), nil)
+ span.OTLP = &tracepb.Span{Name: "original", Flags: 0xffffffff, StartTimeUnixNano: 1789747200123456789,
+ Events: []*tracepb.Span_Event{{Name: "event", TimeUnixNano: 1789747200123456790}},
+ Status: &tracepb.Status{Code: tracepb.Status_STATUS_CODE_ERROR, Message: "status detail"}}
+ span.Context = &tracepb.ResourceSpans{SchemaUrl: "resource-schema", ScopeSpans: []*tracepb.ScopeSpans{{SchemaUrl: "scope-schema"}}}
+ for range 2 {
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span, span}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ // The row is filed under the OTLP start, not under the wall clock the fixture was built at.
+ payload, err := OtelSpanRepository.FindOTLP(ctx, span.ProjectId, span.TraceId, span.SpanId, shared.OtelNanosToTime(span.OTLP.StartTimeUnixNano))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var resource tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &resource); err != nil {
+ t.Fatal(err)
+ }
+ if resource.SchemaUrl != span.Context.SchemaUrl || !proto.Equal(resource.ScopeSpans[0].Spans[0], span.OTLP) {
+ t.Fatalf("OTLP data changed: %v", &resource)
+ }
+}
+
+// findSpans reads the spans under one span the way the detail pages do, through the graph facade.
+func findSpans(ctx context.Context, under models.OtelSpan, recordedAt *time.Time) ([]models.Span, error) {
+ graph, err := SpanRepository.FindGraph(ctx, shared.SpanLookup{ProjectId: under.ProjectId, TraceId: under.TraceId, SpanId: under.SpanId, RecordedAt: recordedAt})
+ if err != nil {
+ return nil, err
+ }
+ return graph.Spans, nil
+}
diff --git a/backend/app/repositories/telemetry/otel_span_timestamps_test.go b/backend/app/repositories/telemetry/otel_span_timestamps_test.go
new file mode 100644
index 00000000..74cacaf0
--- /dev/null
+++ b/backend/app/repositories/telemetry/otel_span_timestamps_test.go
@@ -0,0 +1,94 @@
+package telemetry
+
+import (
+ "context"
+ "encoding/hex"
+ "math"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestOtelSpanTimestampBoundariesKeepExactSourceValues(t *testing.T) {
+ setupTestDB(t)
+ ctx, project := context.Background(), uuid.New()
+ historical := uint64(time.Date(2019, 3, 4, 5, 6, 7, 123456789, time.UTC).UnixNano())
+ cases := map[string][2]uint64{
+ "missing start and end": {0, 0},
+ "historical": {historical, historical + 1_000_000},
+ "reversed interval": {historical + 5_000, historical},
+ "beyond signed nanoseconds": {math.MaxInt64 + 1_000, math.MaxInt64 + 2_000},
+ "largest unsigned timestamp": {math.MaxUint64 - 1, math.MaxUint64},
+ }
+ fallbacksBefore := db.GetSpanPartitionTimeFallbacks()
+ index := byte(0)
+ for name, times := range cases {
+ index++
+ trace := uuid.New()
+ source := &tracepb.Span{TraceId: trace[:], SpanId: []byte{index, 2, 3, 4, 5, 6, 7, 8}, Name: name, StartTimeUnixNano: times[0], EndTimeUnixNano: times[1]}
+ source.Attributes = []*commonpb.KeyValue{{Key: "test.case", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: name}}}}
+ span := models.OtelSpan{OTLP: source, Span: models.Span{ProjectId: project, TraceId: hex.EncodeToString(trace[:]), SpanId: hex.EncodeToString(source.SpanId),
+ Name: name, StartTime: shared.OtelNanosToTime(times[0]), Duration: shared.OtelDuration(times[0], times[1])}}
+ if rejected, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span}); err != nil || rejected != 0 {
+ t.Fatalf("%s: rejected=%d err=%v", name, rejected, err)
+ }
+ // Reads are anchored on the partition time, which is the ingest time when the source start cannot be stored.
+ partition, _, _ := shared.OtelStorageTimes(times[0], time.Now())
+ payload, err := OtelSpanRepository.FindOTLP(ctx, project, span.TraceId, span.SpanId, partition)
+ if err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ var restored tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &restored); err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ stored := restored.ScopeSpans[0].Spans[0]
+ if stored.StartTimeUnixNano != times[0] || stored.EndTimeUnixNano != times[1] {
+ t.Errorf("%s: exact source times changed to %d %d", name, stored.StartTimeUnixNano, stored.EndTimeUnixNano)
+ }
+ found, err := OtelSpanRepository.FindTraceTopology(ctx, []shared.SpanLookup{{ProjectId: project, TraceId: span.TraceId, RecordedAt: &partition}}, 0)
+ if err != nil || len(found) != 1 {
+ t.Fatalf("%s: by-ID retrieval must not depend on the start hint: %d rows %v", name, len(found), err)
+ }
+ if found[0].Duration < 0 {
+ t.Errorf("%s: derived duration %d is negative", name, found[0].Duration)
+ }
+ if !found[0].StartTime.Equal(span.StartTime) || found[0].RecordedAt.Sub(partition).Abs() >= time.Minute {
+ t.Errorf("%s: topology must keep source start %s and storage time near %s, got %s and %s", name, span.StartTime, partition, found[0].StartTime, found[0].RecordedAt)
+ }
+ results, total, err := OtelSpanRepository.Search(ctx, shared.OtelSpanSearch{
+ ProjectId: project, TraceId: span.TraceId, From: partition.Add(-time.Minute), To: partition.Add(time.Minute), Page: 1, PageSize: 10,
+ })
+ if err != nil || total != 1 || len(results) != 1 {
+ t.Fatalf("%s: search must find the stored span: %d rows, total %d, %v", name, len(results), total, err)
+ }
+ result := results[0]
+ graph, err := SpanRepository.FindTrace(ctx, []uuid.UUID{project}, result.TraceId, result.RecordedAt)
+ if err != nil || graph == nil || len(graph.Spans) != 1 {
+ t.Fatalf("%s: search result at=%s must reopen its trace: %+v %v", name, result.RecordedAt, graph, err)
+ }
+ reopened := graph.Spans[0]
+ if !reopened.RecordedAt.Equal(result.RecordedAt) || !reopened.StartTime.Equal(span.StartTime) {
+ t.Errorf("%s: trace outline must retain storage and source times: %+v", name, reopened)
+ }
+ attributes, err := SpanRepository.FindSpanAttributes(ctx, project, reopened.TraceId, reopened.SpanId, reopened.RecordedAt)
+ if err != nil || attributes == nil || attributes.Attributes["test.case"] != name {
+ t.Fatalf("%s: reopened span must load its attributes: %+v %v", name, attributes, err)
+ }
+ reopenedPayload, err := OtelSpanRepository.FindOTLP(ctx, project, reopened.TraceId, reopened.SpanId, reopened.RecordedAt)
+ var reopenedSource tracepb.ResourceSpans
+ if err != nil || len(reopenedPayload) == 0 || proto.Unmarshal(reopenedPayload, &reopenedSource) != nil || !proto.Equal(&restored, &reopenedSource) {
+ t.Fatalf("%s: reopening must export the same original payload: %v", name, err)
+ }
+ }
+ if fallbacks := db.GetSpanPartitionTimeFallbacks() - fallbacksBefore; fallbacks != 3 {
+ t.Errorf("missing and unrepresentable starts must be counted, got %d fallbacks", fallbacks)
+ }
+}
diff --git a/backend/app/repositories/telemetry/profile_identity_test.go b/backend/app/repositories/telemetry/profile_identity_test.go
new file mode 100644
index 00000000..3e371703
--- /dev/null
+++ b/backend/app/repositories/telemetry/profile_identity_test.go
@@ -0,0 +1,28 @@
+package telemetry
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+func TestProfileInsertAfterIdentityCleanup(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Second)
+ profile := models.Profile{
+ Id: uuid.New(), ProjectId: uuid.New(), RecordedAt: now,
+ ServiceName: "trace-identity", ProfileType: "cpu", Unit: "nanoseconds",
+ SampleCount: 1, TotalValue: 123, TraceId: "0102030405060708090a0b0c0d0e0f10", SpanId: "0102030405060708",
+ }
+ if err := ProfileRepository.InsertProfilesAsync(ctx, []models.Profile{profile}); err != nil {
+ t.Fatal(err)
+ }
+ groups, total, err := ProfileRepository.FindGroupedByService(ctx, profile.ProjectId, now.Add(-time.Second), now.Add(time.Second), 1, 10, "profile_count", "desc", "")
+ if err != nil || total != 1 || len(groups) != 1 || groups[0].ServiceName != profile.ServiceName || groups[0].ProfileCount != 1 {
+ t.Fatalf("profile was not stored after obsolete column removal: %+v total=%d: %v", groups, total, err)
+ }
+}
diff --git a/backend/app/repositories/telemetry/profile_repository_test.go b/backend/app/repositories/telemetry/profile_repository_test.go
index 7102b59b..27a72745 100644
--- a/backend/app/repositories/telemetry/profile_repository_test.go
+++ b/backend/app/repositories/telemetry/profile_repository_test.go
@@ -48,10 +48,15 @@ func TestProfileRepository_InsertAndReadBack(t *testing.T) {
Id: profileId, ProjectId: projectId, RecordedAt: now, Duration: 30 * time.Second,
ServiceName: "checkout", ProfileType: "cpu",
SampleCount: 1, TotalValue: 300, ServerName: "pod-a", AppVersion: "1.2.3",
+ TraceId: "0102030405060708090a0b0c0d0e0f10", SpanId: "0102030405060708",
}
if err := ProfileRepository.InsertProfilesAsync(ctx, []models.Profile{prof}); err != nil {
t.Fatalf("InsertProfilesAsync: %v", err)
}
+ var traceID, spanID string
+ if err := db.TelemetryDB.QueryRowContext(ctx, "SELECT trace_id, span_id FROM profiles WHERE id = ?", profileId.String()).Scan(&traceID, &spanID); err != nil || traceID != prof.TraceId || spanID != prof.SpanId {
+ t.Fatalf("profile trace/span identity = %q/%q: %v", traceID, spanID, err)
+ }
var stackCount int
if err := db.TelemetryDB.QueryRowContext(ctx, "SELECT COUNT(*) FROM profiling_stacks").Scan(&stackCount); err != nil {
diff --git a/backend/app/repositories/telemetry/review_regression_test.go b/backend/app/repositories/telemetry/review_regression_test.go
new file mode 100644
index 00000000..3cd5f75e
--- /dev/null
+++ b/backend/app/repositories/telemetry/review_regression_test.go
@@ -0,0 +1,329 @@
+package telemetry
+
+import (
+ "context"
+ "fmt"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestRegressionMoveOverPageKeyIncludesProject(t *testing.T) {
+ setupTestDB(t)
+ setupMoveOverProgress(t)
+ legacyReset(t)
+ t.Cleanup(func() { legacyReset(t) })
+ at := time.Now().UTC().Truncate(moveOverDay).Add(10 * time.Hour)
+ id := uuid.New()
+ seedLegacyDay(t, uuid.New(), at, id, "GET /project-a")
+ seedLegacyDay(t, uuid.New(), at, id, "GET /project-b")
+ for _, project := range []uuid.UUID{uuid.New(), uuid.New()} {
+ legacyExec(t, "INSERT INTO ai_traces (id, project_id, recorded_at, attributes) VALUES (?, ?, ?, ?)", id, project, at, "{}")
+ }
+ if err := RunMoveOver(context.Background(), MoveOverOptions{PageSize: 1, Log: func(string, ...any) {}}); err != nil {
+ t.Fatal(err)
+ }
+ for table, want := range map[string]int64{"endpoints_v2": 2, "tasks_v2": 2, "ai_traces_v2": 2, "exceptions_v2": 2, "spans_v2": 12} {
+ if got := moveOverRowCount(t, table); got != want {
+ t.Fatalf("%s: got %d rows, want %d", table, got, want)
+ }
+ }
+}
+
+func TestRegressionMoveOverResumeKeyIncludesProject(t *testing.T) {
+ setupTestDB(t)
+ setupMoveOverProgress(t)
+ legacyReset(t)
+ t.Cleanup(func() { legacyReset(t) })
+ ctx := context.Background()
+ at := time.Now().UTC().Truncate(moveOverDay).Add(10 * time.Hour)
+ id := uuid.New()
+ first := uuid.New()
+ seedLegacyDay(t, first, at, id, "GET /project-a")
+ seedLegacyDay(t, uuid.New(), at, id, "GET /project-b")
+ rows, err := LegacyRepository.FindEndpoints(ctx, at, at.Add(time.Second), 10, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, row := range rows {
+ if row.ProjectId == first {
+ entity, span := mapEndpoint(row)
+ if err := SpanRepository.InsertAsync(ctx, []models.Span{span}); err != nil {
+ t.Fatal(err)
+ }
+ if err := EndpointRepository.InsertAsync(ctx, []models.Endpoint{entity}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ if err := (moveOverProgress{}).SaveProgress(ctx, transactional.MoveOverDay{Table: "endpoints", Day: at.Format(time.DateOnly), State: transactional.MoveOverStarted}); err != nil {
+ t.Fatal(err)
+ }
+ if err := RunMoveOver(ctx, MoveOverOptions{PageSize: 100, Log: func(string, ...any) {}}); err != nil {
+ t.Fatal(err)
+ }
+ if got := moveOverRowCount(t, "endpoints_v2"); got != 2 {
+ t.Fatalf("resuming treats another project's ID as migrated: got %d endpoints, want 2", got)
+ }
+}
+
+func TestRegressionSearchRetriesHaveUniqueUIKeys(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ span := canonicalSpan(uuid.New(), uuid.New(), uuid.New(), nil)
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span, span}); err != nil {
+ t.Fatal(err)
+ }
+ rows, total, err := OtelSpanRepository.Search(ctx, shared.OtelSpanSearch{ProjectId: span.ProjectId, From: span.StartTime.Add(-time.Hour), To: span.StartTime.Add(time.Hour), Page: 1, PageSize: 50})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if total != 1 || len(rows) != 1 {
+ t.Fatalf("duplicate retries changed search count: %d / %d", len(rows), total)
+ }
+ seen := map[string]bool{}
+ for _, r := range rows {
+ key := r.TraceId + ":" + r.SpanId
+ if seen[key] {
+ t.Fatalf("search returns duplicate Svelte row key %s: %d rows, total %d", key, len(rows), total)
+ }
+ seen[key] = true
+ }
+}
+
+func TestRegressionGraphDuplicateAttributesMatchChosenSpan(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ root := uuid.New()
+ span := canonicalSpan(uuid.New(), uuid.New(), uuid.New(), &root)
+ span.Attributes = map[string]string{"version": "first"}
+ best := span
+ best.Duration = 2 * time.Second
+ best.Attributes = map[string]string{"version": "longest"}
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span, best}); err != nil {
+ t.Fatal(err)
+ }
+ graph, err := SpanRepository.FindGraph(ctx, shared.SpanLookup{ProjectId: span.ProjectId, TraceId: span.TraceId, SpanId: span.ParentSpanId, RecordedAt: &span.StartTime})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(graph.Spans) != 1 {
+ t.Fatalf("graph rows: %d", len(graph.Spans))
+ }
+ if got := graph.Spans[0]; got.Duration != best.Duration || got.Attributes["version"] != "longest" {
+ t.Fatalf("graph mixes duplicate versions: duration=%s attributes=%v; want 2s and longest", got.Duration, got.Attributes)
+ }
+}
+
+func TestRegressionAttributeLimitCountsUTF8Bytes(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ span := canonicalSpan(uuid.New(), uuid.New(), uuid.New(), nil)
+ span.Attributes = map[string]string{"large": strings.Repeat("界", 90000)}
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{span}); err != nil {
+ t.Fatal(err)
+ }
+ attrs, err := OtelSpanRepository.FindSpanAttributes(ctx, shared.SpanLookup{ProjectId: span.ProjectId, TraceId: span.TraceId, RecordedAt: &span.StartTime}, []string{span.SpanId}, shared.OtelAttributeLimits{PerSpanBytes: 256 << 10, BudgetBytes: 32 << 20})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := attrs[span.SpanId]; got.Omitted == "" {
+ t.Fatalf("accepted %d UTF-8 bytes over 262144-byte per-span cap", got.Bytes)
+ }
+}
+
+func TestMoveOverRecoveryKeepsTraceAndOccurrenceIdentity(t *testing.T) {
+ for _, table := range []string{"endpoints", "tasks", "ai_traces", "exception_stack_traces"} {
+ for _, partial := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%s/span-only=%v", table, partial), func(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ at := time.Now().UTC().Truncate(time.Second)
+ project, id := uuid.New(), uuid.New()
+ traces := []uuid.UUID{uuid.New(), uuid.New()}
+ m := &mover{legacy: LegacyRepository}
+ write := func(firstOnly, resumed bool) error {
+ var endpoints []models.Endpoint
+ var tasks []models.Task
+ var calls []models.AiTrace
+ var exceptions []shared.LegacyException
+ for i, trace := range traces {
+ if firstOnly && i > 0 {
+ break
+ }
+ for _, recorded := range []time.Time{at, at.Add(time.Millisecond)} {
+ switch table {
+ case "endpoints":
+ endpoints = append(endpoints, models.Endpoint{ProjectId: project, Id: id, TraceId: trace.String(), RecordedAt: recorded, Duration: time.Second})
+ case "tasks":
+ tasks = append(tasks, models.Task{ProjectId: project, Id: id, TraceId: trace.String(), SpanId: paddedSpan(3).String(), RecordedAt: recorded, Duration: 8 * 24 * time.Hour})
+ case "ai_traces":
+ calls = append(calls, models.AiTrace{ProjectId: project, Id: id, TraceId: trace.String(), RecordedAt: recorded, Duration: time.Second, Attributes: map[string]string{}})
+ case "exception_stack_traces":
+ exceptions = append(exceptions, shared.LegacyException{ExceptionStackTrace: models.ExceptionStackTrace{ProjectId: project, Id: id, RecordedAt: recorded, Attributes: map[string]string{}}, DistributedTraceId: trace.String()})
+ }
+ }
+ }
+ if firstOnly && partial && table != "exception_stack_traces" {
+ var spans []models.Span
+ for _, row := range endpoints {
+ _, span := mapEndpoint(row)
+ spans = append(spans, span)
+ }
+ for _, row := range tasks {
+ _, span := mapTask(row)
+ spans = append(spans, span)
+ }
+ for _, row := range calls {
+ _, span := mapAiTrace(row)
+ spans = append(spans, span)
+ }
+ return m.insertSpans(ctx, spans, at, at.Add(time.Second), false)
+ }
+ switch table {
+ case "endpoints":
+ return m.writeEndpoints(ctx, endpoints, at, at.Add(time.Second), resumed)
+ case "tasks":
+ return m.writeTasks(ctx, tasks, at, at.Add(time.Second), resumed)
+ case "ai_traces":
+ return m.writeAiTraces(ctx, calls, at, at.Add(time.Second), resumed)
+ default:
+ return m.writeExceptions(ctx, exceptions, at, at.Add(time.Second), resumed)
+ }
+ }
+ if err := write(true, false); err != nil {
+ t.Fatal(err)
+ }
+ if err := write(false, true); err != nil {
+ t.Fatal(err)
+ }
+ if err := write(false, true); err != nil {
+ t.Fatal(err)
+ }
+ if got := moveOverRowCount(t, shared.LegacyTables[table]); got != 4 {
+ t.Fatalf("%s: got %d occurrences, want 4", table, got)
+ }
+ })
+ }
+ }
+}
+
+func TestSpanRetryWinnerMatchesGraphAttributesAndExport(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ root := uuid.New()
+ first := canonicalSpan(uuid.New(), uuid.New(), uuid.New(), &root)
+ first.Name, first.Attributes = "a-version", map[string]string{"version": "a"}
+ last := first
+ last.Name, last.Attributes = "z-version", map[string]string{"version": "z"}
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{last, first, last, first}); err != nil {
+ t.Fatal(err)
+ }
+ lookup := shared.NewSpanLookup(first.ProjectId, first.TraceId, first.ParentSpanId, first.StartTime)
+ graph, err := SpanRepository.FindGraph(ctx, lookup)
+ if err != nil || len(graph.Spans) != 1 {
+ t.Fatalf("graph: %+v, %v", graph, err)
+ }
+ payload, err := OtelSpanRepository.FindOTLP(ctx, first.ProjectId, first.TraceId, first.SpanId, first.StartTime)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var envelope tracepb.ResourceSpans
+ if err := proto.Unmarshal(payload, &envelope); err != nil {
+ t.Fatal(err)
+ }
+ exported := envelope.ScopeSpans[0].Spans[0]
+ attributes := shared.StringAttributes(exported.Attributes)
+ if graph.Spans[0].Name != exported.Name || graph.Spans[0].Attributes["version"] != attributes["version"] {
+ t.Fatalf("graph and export chose different retry versions: %+v / %v", graph.Spans[0], exported)
+ }
+ search := shared.OtelSpanSearch{ProjectId: first.ProjectId, From: first.StartTime.Add(-time.Hour), To: first.StartTime.Add(time.Hour), Page: 1, PageSize: 1}
+ rows, total, err := OtelSpanRepository.Search(ctx, search)
+ if err != nil || total != 1 || len(rows) != 1 || rows[0].Name != exported.Name {
+ t.Fatalf("search winner/count: %v %d %v", rows, total, err)
+ }
+ search.Page = 2
+ rows, total, err = OtelSpanRepository.Search(ctx, search)
+ if err != nil || total != 1 || len(rows) != 0 {
+ t.Fatalf("retry leaked onto next page: %v %d %v", rows, total, err)
+ }
+ search.Page = 1
+ discarded := "a"
+ if attributes["version"] == discarded {
+ discarded = "z"
+ }
+ search.Attributes = []shared.OtelAttributeFilter{{Key: "version", Value: discarded}}
+ rows, total, err = OtelSpanRepository.Search(ctx, search)
+ if err != nil || total != 0 || len(rows) != 0 {
+ t.Fatalf("filter matched a discarded retry: %v %d %v", rows, total, err)
+ }
+}
+
+func TestAttributeBudgetCountsUTF8BytesAfterDeduplication(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ first := canonicalSpan(uuid.New(), uuid.New(), uuid.New(), nil)
+ first.Attributes = map[string]string{"text": strings.Repeat("界", 200)}
+ second := first
+ second.SpanId = hexId(uuid.New())[16:]
+ second.StartTime = first.StartTime.Add(time.Millisecond)
+ if _, err := OtelSpanRepository.InsertAsync(ctx, []models.OtelSpan{first, first, second}); err != nil {
+ t.Fatal(err)
+ }
+ lookup := shared.NewSpanLookup(first.ProjectId, first.TraceId, first.SpanId, first.StartTime)
+ attributes, err := OtelSpanRepository.FindSpanAttributes(ctx, lookup, []string{first.SpanId, second.SpanId}, shared.OtelAttributeLimits{PerSpanBytes: 700, BudgetBytes: 900})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if attributes[first.SpanId].Omitted != "" || attributes[second.SpanId].Omitted != models.SpanGraphAttributeBudget {
+ t.Fatalf("incorrect UTF-8 budget or retry charged twice: %+v", attributes)
+ }
+ if attributes[first.SpanId].Bytes > 900 || attributes[second.SpanId].Bytes != 0 {
+ t.Fatal("omitted data crossed the database boundary")
+ }
+}
+
+func TestMoveOverRejectedSpanDoesNotCompleteDay(t *testing.T) {
+ setupTestDB(t)
+ setupMoveOverProgress(t)
+ legacyReset(t)
+ t.Cleanup(func() { legacyReset(t) })
+ at := time.Now().UTC().Truncate(moveOverDay).Add(10 * time.Hour)
+ seedLegacyDay(t, uuid.New(), at, uuid.Nil, "GET /invalid-trace")
+ if err := RunMoveOver(context.Background(), MoveOverOptions{Log: func(string, ...any) {}}); err == nil {
+ t.Fatal("rejected canonical span was reported as a successful migration")
+ }
+ progress, err := (moveOverProgress{}).FindProgress(context.Background())
+ if err != nil || len(progress) != 1 || progress[0].State != transactional.MoveOverStarted {
+ t.Fatalf("rejected day must stay started: %+v, %v", progress, err)
+ }
+ if moveOverRowCount(t, "endpoints_v2") != 0 {
+ t.Fatal("rejected root must not leave an endpoint projection")
+ }
+}
+
+func TestLegacySourceReadsHonorCancellation(t *testing.T) {
+ setupTestDB(t)
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ at := time.Now().UTC()
+ if _, err := LegacyRepository.FindEndpoints(ctx, at.Add(-time.Hour), at, 10, nil); err == nil {
+ t.Fatal("cancelled endpoint scan succeeded")
+ }
+ if _, err := LegacyRepository.FindTasks(ctx, at.Add(-time.Hour), at, 10, nil); err == nil {
+ t.Fatal("cancelled task scan succeeded")
+ }
+ if _, err := LegacyRepository.FindAiTraces(ctx, at.Add(-time.Hour), at, 10, nil); err == nil {
+ t.Fatal("cancelled AI scan succeeded")
+ }
+ if _, err := LegacyRepository.FindExceptions(ctx, at.Add(-time.Hour), at, 10, nil); err == nil {
+ t.Fatal("cancelled exception scan succeeded")
+ }
+}
diff --git a/backend/app/repositories/telemetry/session_repository_test.go b/backend/app/repositories/telemetry/session_repository_test.go
index 39ad6733..b1e19125 100644
--- a/backend/app/repositories/telemetry/session_repository_test.go
+++ b/backend/app/repositories/telemetry/session_repository_test.go
@@ -2,6 +2,8 @@ package telemetry
import (
"context"
+ "encoding/json"
+ "strings"
"testing"
"time"
@@ -9,6 +11,29 @@ import (
"github.com/tracewayapp/traceway/backend/app/models"
)
+func TestSessionTraceIdentityRoundTrip(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ now := time.Now().UTC().Truncate(time.Second)
+ const traceID = "0102030405060708090a0b0c0d0e0f10"
+ session := models.Session{Id: uuid.New(), ProjectId: uuid.New(), StartedAt: now, TraceId: traceID}
+ if err := SessionRepository.Upsert(ctx, []models.Session{session}); err != nil {
+ t.Fatal(err)
+ }
+ got, err := SessionRepository.FindById(ctx, session.ProjectId, session.Id, &now)
+ if err != nil || got == nil || got.TraceId != traceID {
+ t.Fatalf("session trace did not round trip: %+v: %v", got, err)
+ }
+ encoded, err := json.Marshal(got)
+ if err != nil || !strings.Contains(string(encoded), `"traceId":"`+traceID+`"`) || strings.Contains(string(encoded), "distributedTraceId") {
+ t.Fatalf("session JSON must expose the canonical trace ID: %s: %v", encoded, err)
+ }
+ rows, total, err := SessionRepository.FindAll(ctx, session.ProjectId, now.Add(-time.Second), now.Add(time.Second), 1, 10, "started_at", "asc", "", nil)
+ if err != nil || total != 1 || len(rows) != 1 || rows[0].TraceId != traceID {
+ t.Fatalf("session search lost trace identity: %+v total=%d: %v", rows, total, err)
+ }
+}
+
func TestSessionRepository_SearchAttributes(t *testing.T) {
setupTestDB(t)
ctx := context.Background()
diff --git a/backend/app/repositories/telemetry/shared/legacy.go b/backend/app/repositories/telemetry/shared/legacy.go
new file mode 100644
index 00000000..d91c8bda
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/legacy.go
@@ -0,0 +1,106 @@
+package shared
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+type legacyReadExecutor struct {
+ *sql.DB
+ ctx context.Context
+}
+
+func (r legacyReadExecutor) Query(query string, args ...any) (*sql.Rows, error) {
+ return r.DB.QueryContext(r.ctx, query, args...)
+}
+
+// lit.SelectNamed uses Query without a context; the adapter keeps shutdown from
+// waiting for an uncancellable legacy scan.
+func SelectLegacy[T any](ctx context.Context, conn *sql.DB, query string, params lit.P) ([]*T, error) {
+ return lit.SelectNamed[T](legacyReadExecutor{conn, ctx}, query, params)
+}
+
+// The move-over reads the tables V2 replaced. Endpoints, tasks, AI traces and exceptions come back in their V2 model
+// with the old ids parked in the new fields exactly as they were stored: on an entity TraceId holds the old distributed
+// trace id and SpanId the old span id, on a LegacyException TraceId holds the old owner id and DistributedTraceId the old
+// distributed trace id. The moveover package turns them into V2 ids, in one place for every backend.
+
+type LegacyException struct {
+ models.ExceptionStackTrace
+ DistributedTraceId string
+}
+
+// LegacySpan is a row of the old spans table, where trace_id named the owning endpoint, task or AI trace.
+type LegacySpan struct {
+ ProjectId uuid.UUID
+ Id uuid.UUID
+ OwnerId uuid.UUID
+ ParentSpanId string
+ Name string
+ StartTime time.Time
+ Duration time.Duration
+ RecordedAt time.Time
+ Attributes map[string]string
+}
+
+// LegacyTables maps each table V2 replaced to the table its rows move into.
+var LegacyTables = map[string]string{
+ "endpoints": "endpoints_v2",
+ "tasks": "tasks_v2",
+ "ai_traces": "ai_traces_v2",
+ "exception_stack_traces": "exceptions_v2",
+ "spans": SpansTable,
+}
+
+// LegacyTable guards the table names the move-over puts into SQL text.
+func LegacyTable(name string) (string, error) {
+ if _, known := LegacyTables[name]; known {
+ return name, nil
+ }
+ for _, v2 := range LegacyTables {
+ if v2 == name {
+ return name, nil
+ }
+ }
+ return "", fmt.Errorf("%q is not a table the move-over reads or writes", name)
+}
+
+// MovedSpanKey identifies a span already present in the V2 table when a day is resumed.
+func MovedSpanKey(projectId uuid.UUID, traceId, spanId string) string {
+ return OtelKey(projectId, traceId) + ":" + spanId
+}
+
+// Legacy history is immutable after cutover. Offset paging within a single second
+// preserves even identical retries; ordering every copied value makes unequal rows
+// stable across reads. An ID alone cannot order occurrences from different traces.
+func LegacyPageOrder(table string, clickhouse bool) string {
+ columns := map[string]string{
+ "endpoints": "id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_stream, is_root",
+ "tasks": "id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root",
+ "ai_traces": "id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version, storage_key, attributes, distributed_trace_id, is_root, conversation_id, tool_call_count, tool_names, flagged, flagged_terms",
+ "exception_stack_traces": "id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id",
+ "spans": "project_id, id, trace_id, parent_span_id, name, start_time, duration, recorded_at, attributes",
+ }[table]
+ if clickhouse {
+ // Arrays and nullable UUIDs need a common, totally ordered representation.
+ return "toJSONString(tuple(" + columns + "))"
+ }
+ return columns
+}
+
+type MovedOccurrence struct {
+ ProjectId uuid.UUID
+ Id uuid.UUID
+ TraceId, SpanId string
+ RecordedAt string
+}
+
+func MovedOccurrenceKey(project, id uuid.UUID, trace, span string, at time.Time) MovedOccurrence {
+ return MovedOccurrence{project, id, trace, span, at.UTC().Format(time.RFC3339Nano)}
+}
diff --git a/backend/app/repositories/telemetry/shared/native_otlp_test.go b/backend/app/repositories/telemetry/shared/native_otlp_test.go
new file mode 100644
index 00000000..ae7a4598
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/native_otlp_test.go
@@ -0,0 +1,83 @@
+package shared
+
+import (
+ "bytes"
+ "encoding/hex"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestRegressionNativeOTLPConformance(t *testing.T) {
+ s := models.OtelSpan{Span: models.Span{ProjectId: uuid.New(), TraceId: "0123456789abcdef0123456789abcdef", SpanId: "abcdef0123456789abcdef0123456789", Name: "GET /probe", StartTime: time.Now(), Duration: time.Second, ServiceName: "native-service", ScopeName: "native-scope", ParentSpanId: "fedcba9876543210fedcba9876543210", Attributes: map[string]string{"http.request.method": "GET"}}}
+ r, err := NewOtelSpanRow(s, OtelSpanGroups{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(r.Source.SpanId) != 8 {
+ t.Errorf("native export span ID has %d bytes, OTLP requires 8", len(r.Source.SpanId))
+ }
+ if len(r.Source.Attributes) == 0 {
+ t.Error("native span attributes missing from exported protobuf")
+ }
+ if len(r.Group.Resource.GetAttributes()) == 0 {
+ t.Error("native service resource missing from exported protobuf")
+ }
+}
+
+func TestNativeOTLPIdentityMappingAndMetadata(t *testing.T) {
+ traceID := "0123456789abcdef0123456789abcdef"
+ parent := models.OtelSpan{Span: models.Span{TraceId: traceID, SpanId: "fedcba9876543210fedcba9876543210", ServiceName: "parent-service", ScopeName: "parent-scope", StartTime: time.Now(), Duration: time.Second}}
+ child := parent
+ child.SpanId, child.ParentSpanId = "abcdef0123456789abcdef0123456789", parent.SpanId
+ child.ServiceName, child.ScopeName = "child-service", "child-scope"
+ child.Attributes = map[string]string{"z": "last", "a": "first", "unicode": "界"}
+ groups := OtelSpanGroups{}
+ parentRow, err := NewOtelSpanRow(parent, groups)
+ if err != nil {
+ t.Fatal(err)
+ }
+ childRow, err := NewOtelSpanRow(child, groups)
+ if err != nil {
+ t.Fatal(err)
+ }
+ repeated, err := NewOtelSpanRow(child, groups)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !bytes.Equal(childRow.Source.ParentSpanId, parentRow.Source.SpanId) || !bytes.Equal(childRow.Payload(), repeated.Payload()) {
+ t.Fatal("native mapping or encoding is not deterministic")
+ }
+ var envelope tracepb.ResourceSpans
+ if err := proto.Unmarshal(childRow.Payload(), &envelope); err != nil {
+ t.Fatal(err)
+ }
+ scope := envelope.ScopeSpans[0]
+ exported := scope.Spans[0]
+ if hex.EncodeToString(exported.TraceId) != traceID || scope.Scope.Name != "child-scope" || StringAttributes(envelope.Resource.Attributes)["service.name"] != "child-service" {
+ t.Fatal("metadata leaked between native spans sharing a batch")
+ }
+ attributes := StringAttributes(exported.Attributes)
+ for key, value := range child.Attributes {
+ if attributes[key] != value {
+ t.Fatalf("attribute %s was lost", key)
+ }
+ }
+ if attributes["traceway.native.span_id"] != child.SpanId || attributes["traceway.native.parent_span_id"] != parent.SpanId {
+ t.Fatal("native identity mapping was not exported")
+ }
+ for _, id := range []string{"", "00", "0000000000000000", "00000000000000000000000000000000", "invalid"} {
+ if _, err := nativeOTLPSpanID(id); err == nil {
+ t.Errorf("accepted invalid span ID %q", id)
+ }
+ }
+ existing := "0123456789abcdef"
+ raw, err := nativeOTLPSpanID(existing)
+ if err != nil || hex.EncodeToString(raw) != existing {
+ t.Fatal("an existing eight-byte ID must be unchanged")
+ }
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_graph.go b/backend/app/repositories/telemetry/shared/otel_graph.go
new file mode 100644
index 00000000..de8748fc
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_graph.go
@@ -0,0 +1,362 @@
+package shared
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ traceway "go.tracewayapp.com"
+)
+
+var (
+ MaxOtelGraphRows = 20000
+ MaxOtelGraphBytes = 32 << 20
+ MaxOtelSpanAttributeBytes = 256 << 10
+)
+
+const (
+ maxAnchoredTopologyReads = 8
+ otelAttributeChunk = 4000
+)
+
+const OtelTopologyColumns = `project_id, trace_id, span_id, parent_span_id, name, duration, span_kind, status_code, service_name, scope_name, start_time_unix_nano, recorded_at`
+
+type OtelAttributeLimits struct{ PerSpanBytes, BudgetBytes int }
+
+type OtelSpanAttributes struct {
+ Attributes map[string]string
+ Omitted string
+ Bytes int
+}
+
+type OtelGraphReader interface {
+ // FindTraceTopology returns scalar span rows in (start, span ID) order, at most MaxOtelGraphRows+1 of them.
+ FindTraceTopology(ctx context.Context, lookups []SpanLookup, fromUnixNano uint64) ([]models.OtelSpan, error)
+ // FindTraceOutline returns one whole trace across the projects of the lookups, most important spans first, at most MaxOtelGraphRows+1 rows, each with its statement preview and no attributes.
+ FindTraceOutline(ctx context.Context, lookups []SpanLookup) ([]models.OtelSpan, error)
+ FindSpanAttributes(ctx context.Context, lookup SpanLookup, spanIds []string, limits OtelAttributeLimits) (map[string]OtelSpanAttributes, error)
+ IsReadLimitError(err error) bool
+}
+
+type SpanGraph struct {
+ Spans []models.Span
+ Status models.SpanGraphStatus
+}
+
+// OtelAttributeQuery applies both byte limits inside the database, so an oversized value is never transferred.
+func OtelAttributeQuery(attributes, byteLength, startOrder, predicate, winnerOrder string, limits OtelAttributeLimits) string {
+ return fmt.Sprintf(`SELECT span_id,
+ CASE WHEN size > %[1]d OR running > %[2]d THEN '' ELSE attributes END,
+ CASE WHEN size > %[1]d THEN 2 WHEN running > %[2]d THEN 1 ELSE 0 END
+ FROM (SELECT span_id, attributes, size, start_order,
+ SUM(CASE WHEN size > %[1]d THEN 0 ELSE size END) OVER (ORDER BY start_order, span_id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running
+ FROM (SELECT span_id, %[3]s AS attributes, %[4]s AS size, %[5]s AS start_order FROM %[6]s) sized) budgeted
+ ORDER BY start_order, span_id`, limits.PerSpanBytes, limits.BudgetBytes, attributes, byteLength, startOrder,
+ OtelWinningRows("project_id, trace_id, span_id, span_attributes, start_time_unix_nano", predicate, winnerOrder))
+}
+
+func OtelAttributeOmission(flag int) string {
+ switch flag {
+ case 2:
+ return models.SpanGraphAttributeSize
+ case 1:
+ return models.SpanGraphAttributeBudget
+ }
+ return ""
+}
+
+type otelGraphIndex struct {
+ byId map[string]models.OtelSpan
+ children map[string][]string
+}
+
+func otelSpanKey(project uuid.UUID, traceId, spanId string) string {
+ return project.String() + ":" + traceId + ":" + spanId
+}
+
+func newOtelGraphIndex(topology []models.OtelSpan) otelGraphIndex {
+ index := otelGraphIndex{make(map[string]models.OtelSpan, len(topology)), make(map[string][]string)}
+ for _, span := range topology {
+ key := otelSpanKey(span.ProjectId, span.TraceId, span.SpanId)
+ // Export retries are append-only. Choose the same completed span regardless of arrival order.
+ if old, exists := index.byId[key]; !exists || preferOtelSpan(span, old) {
+ index.byId[key] = span
+ }
+ }
+ for key, span := range index.byId {
+ if span.ParentSpanId == "" {
+ continue
+ }
+ if parentKey := otelSpanKey(span.ProjectId, span.TraceId, span.ParentSpanId); parentKey != key {
+ index.children[parentKey] = append(index.children[parentKey], key)
+ }
+ }
+ return index
+}
+
+// subtree returns the spans under the span the lookup names, without that span itself.
+func (index otelGraphIndex) subtree(lookup SpanLookup) []models.Span {
+ rootKey := otelSpanKey(lookup.ProjectId, lookup.TraceId, lookup.SpanId)
+ spans := []models.Span{}
+ visited := map[string]bool{rootKey: true}
+ pending := append([]string{}, index.children[rootKey]...)
+ if root, stored := index.byId[rootKey]; stored && root.ParentSpanId == "" {
+ // A span whose parent never arrived, because a collector dropped it, would hide everything below it. The
+ // trace's own root takes those subtrees. Any other span could be the wrong owner, so it does not. A batch can
+ // hold other projects and other traces, so only this trace's own orphans belong to its root.
+ for parentKey, orphans := range index.children {
+ if _, arrived := index.byId[parentKey]; arrived {
+ continue
+ }
+ for _, orphan := range orphans {
+ if span := index.byId[orphan]; span.ProjectId == root.ProjectId && span.TraceId == root.TraceId {
+ pending = append(pending, orphan)
+ }
+ }
+ }
+ }
+ for len(pending) > 0 {
+ key := pending[len(pending)-1]
+ pending = pending[:len(pending)-1]
+ if visited[key] {
+ continue
+ }
+ visited[key] = true
+ spans = append(spans, index.byId[key].Span)
+ pending = append(pending, index.children[key]...)
+ }
+ sortOtelSpans(spans)
+ return spans
+}
+
+func sortOtelSpans(spans []models.Span) {
+ slices.SortFunc(spans, func(a, b models.Span) int {
+ if c := a.StartTime.Compare(b.StartTime); c != 0 {
+ return c
+ }
+ return strings.Compare(a.SpanId, b.SpanId)
+ })
+}
+
+func addReason(status *models.SpanGraphStatus, state, reason string) {
+ if status.State != models.SpanGraphUnavailable {
+ status.State = state
+ }
+ if !slices.Contains(status.Reasons, reason) {
+ status.Reasons = append(status.Reasons, reason)
+ }
+}
+
+func readLimitReached(ctx context.Context, reader OtelGraphReader, err error) bool {
+ // The client went away; nobody reads the result and there is nothing to report.
+ if errors.Is(ctx.Err(), context.Canceled) {
+ return true
+ }
+ if !errors.Is(err, context.DeadlineExceeded) && ctx.Err() == nil && !reader.IsReadLimitError(err) {
+ return false
+ }
+ traceway.CaptureException(fmt.Errorf("otel span graph read hit a limit: %w", err))
+ return true
+}
+
+// FindOtelGraphs reads topology first and attributes second. A limit never fails the caller: the graph comes back
+// partial or unavailable and says why. A lookup that names no trace or no span is skipped.
+func FindOtelGraphs(ctx context.Context, lookups []SpanLookup, reader OtelGraphReader) (map[SpanOwner]*SpanGraph, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ result := make(map[SpanOwner]*SpanGraph)
+ budget, anchoredReads := MaxOtelGraphBytes, 0
+ for batch := range slices.Chunk(lookups, SpanLookupBatchSize) {
+ traceKeys := make(map[string]bool)
+ traces, wanted := make([]SpanLookup, 0, len(batch)), make([]SpanLookup, 0, len(batch))
+ for _, lookup := range batch {
+ if lookup.TraceId == "" || lookup.SpanId == "" {
+ continue
+ }
+ wanted = append(wanted, lookup)
+ if key := OtelKey(lookup.ProjectId, lookup.TraceId); !traceKeys[key] {
+ traceKeys[key] = true
+ traces = append(traces, lookup)
+ }
+ }
+ if len(traces) == 0 {
+ continue
+ }
+ topology, err := reader.FindTraceTopology(ctx, traces, 0)
+ if err != nil {
+ if !readLimitReached(ctx, reader, err) {
+ return nil, err
+ }
+ for _, lookup := range wanted {
+ result[lookup.Owner()] = &SpanGraph{Spans: []models.Span{}, Status: models.SpanGraphStatus{State: models.SpanGraphUnavailable, Reasons: []string{models.SpanGraphReadLimit}}}
+ }
+ continue
+ }
+ truncated := len(topology) > MaxOtelGraphRows
+ index := newOtelGraphIndex(topology[:min(len(topology), MaxOtelGraphRows)])
+ for _, lookup := range wanted {
+ if _, exists := result[lookup.Owner()]; exists {
+ continue
+ }
+ graph := &SpanGraph{Status: models.SpanGraphStatus{State: models.SpanGraphComplete}}
+ graph.Spans = index.subtree(lookup)
+ if truncated {
+ addReason(&graph.Status, models.SpanGraphPartial, models.SpanGraphRowLimit)
+ // An unanchored cap keeps the earliest spans of the trace, which can exclude a span that starts late.
+ if lookup.StartUnixNano > 0 && anchoredReads < maxAnchoredTopologyReads {
+ anchoredReads++
+ anchored, err := reader.FindTraceTopology(ctx, []SpanLookup{lookup}, lookup.StartUnixNano)
+ if err != nil && !readLimitReached(ctx, reader, err) {
+ return nil, err
+ }
+ if err == nil {
+ if spans := newOtelGraphIndex(anchored[:min(len(anchored), MaxOtelGraphRows)]).subtree(lookup); len(spans) > len(graph.Spans) {
+ graph.Spans = spans
+ }
+ }
+ }
+ }
+ if err := loadOtelAttributes(ctx, reader, lookup, graph, &budget); err != nil {
+ return nil, err
+ }
+ result[lookup.Owner()] = graph
+ }
+ }
+ return result, nil
+}
+
+// FindOtelTrace reads a whole trace for the explorer, one lookup per project the caller may read. Attributes stay behind: the page asks for one span's attributes
+// when its popover opens, so a 20,000 span trace is a few megabytes and not the full attribute budget.
+func FindOtelTrace(ctx context.Context, lookups []SpanLookup, reader OtelGraphReader) (*SpanGraph, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ graph := &SpanGraph{Spans: []models.Span{}, Status: models.SpanGraphStatus{State: models.SpanGraphComplete}}
+ if len(lookups) == 0 {
+ return graph, nil
+ }
+ outline, err := reader.FindTraceOutline(ctx, lookups)
+ if err != nil {
+ if !readLimitReached(ctx, reader, err) {
+ return nil, err
+ }
+ graph.Status = models.SpanGraphStatus{State: models.SpanGraphUnavailable, Reasons: []string{models.SpanGraphReadLimit}}
+ return graph, nil
+ }
+ if len(outline) > MaxOtelGraphRows {
+ addReason(&graph.Status, models.SpanGraphPartial, models.SpanGraphRowLimit)
+ addReason(&graph.Status, models.SpanGraphPartial, models.SpanGraphMostImportant)
+ outline = outline[:MaxOtelGraphRows]
+ }
+ for _, span := range newOtelGraphIndex(outline).byId {
+ graph.Spans = append(graph.Spans, span.Span)
+ }
+ sortOtelSpans(graph.Spans)
+ return graph, nil
+}
+
+// FindOtelTraceParents maps every span of a trace to its parent, across the projects of the lookups. It reads the
+// topology columns and leaves the attributes on disk. Entities of a trace are joined by the spans between them, and those spans
+// need not belong to any entity: a gRPC hop reporting to its own project is one. A trace the read limits cut short
+// yields what was read.
+func FindOtelTraceParents(ctx context.Context, lookups []SpanLookup, reader OtelGraphReader) (map[string]string, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ parents := map[string]string{}
+ if len(lookups) == 0 {
+ return parents, nil
+ }
+ topology, err := reader.FindTraceTopology(ctx, lookups, 0)
+ if err != nil {
+ if readLimitReached(ctx, reader, err) {
+ return parents, nil
+ }
+ return nil, err
+ }
+ for _, span := range topology {
+ if span.ParentSpanId != "" {
+ parents[span.SpanId] = span.ParentSpanId
+ }
+ }
+ return parents, nil
+}
+
+// FindOtelSpanAttributes loads the attributes of one span under the same per-span size limit as a graph read. It
+// returns nil when the span is not stored inside the lookup window.
+func FindOtelSpanAttributes(ctx context.Context, lookup SpanLookup, spanId string, reader OtelGraphReader) (*OtelSpanAttributes, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ loaded, err := reader.FindSpanAttributes(ctx, lookup, []string{spanId}, OtelAttributeLimits{MaxOtelSpanAttributeBytes, MaxOtelGraphBytes})
+ if err != nil {
+ if readLimitReached(ctx, reader, err) {
+ return &OtelSpanAttributes{Omitted: models.SpanGraphAttributesUnavailable}, nil
+ }
+ return nil, err
+ }
+ if found, ok := loaded[spanId]; ok {
+ return &found, nil
+ }
+ return nil, nil
+}
+
+func loadOtelAttributes(ctx context.Context, reader OtelGraphReader, lookup SpanLookup, graph *SpanGraph, budget *int) error {
+ positions := make(map[string][]int, len(graph.Spans))
+ ids := make([]string, 0, len(graph.Spans))
+ for i, span := range graph.Spans {
+ if _, seen := positions[span.SpanId]; !seen {
+ ids = append(ids, span.SpanId)
+ }
+ positions[span.SpanId] = append(positions[span.SpanId], i)
+ }
+ omit := func(spanIds []string, reason string) {
+ for _, id := range spanIds {
+ for _, i := range positions[id] {
+ graph.Spans[i].AttributesOmitted = true
+ graph.Status.OmittedAttributes++
+ }
+ }
+ if len(spanIds) > 0 {
+ addReason(&graph.Status, models.SpanGraphPartial, reason)
+ }
+ }
+ for chunk := range slices.Chunk(ids, otelAttributeChunk) {
+ if *budget <= 0 {
+ omit(chunk, models.SpanGraphAttributeBudget)
+ continue
+ }
+ loaded, err := reader.FindSpanAttributes(ctx, lookup, chunk, OtelAttributeLimits{MaxOtelSpanAttributeBytes, *budget})
+ if err != nil {
+ if !readLimitReached(ctx, reader, err) {
+ return err
+ }
+ omit(chunk, models.SpanGraphAttributesUnavailable)
+ continue
+ }
+ for _, id := range chunk {
+ attributes := loaded[id]
+ if attributes.Omitted != "" {
+ omit([]string{id}, attributes.Omitted)
+ continue
+ }
+ *budget -= attributes.Bytes
+ for _, i := range positions[id] {
+ graph.Spans[i].Attributes = attributes.Attributes
+ }
+ }
+ }
+ return nil
+}
+
+// SpanOwner keys a graph by the span it hangs under.
+type SpanOwner struct {
+ ProjectId uuid.UUID
+ TraceId, SpanId string
+}
+
+func (lookup SpanLookup) Owner() SpanOwner {
+ return SpanOwner{lookup.ProjectId, lookup.TraceId, lookup.SpanId}
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_graph_sql.go b/backend/app/repositories/telemetry/shared/otel_graph_sql.go
new file mode 100644
index 00000000..759fe247
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_graph_sql.go
@@ -0,0 +1,90 @@
+package shared
+
+import (
+ "database/sql"
+ "encoding/json"
+ "fmt"
+ "math"
+ "strings"
+ "time"
+)
+
+const OtelTextWinnerOrder = "duration DESC, otlp DESC"
+
+// A stored digest keeps ClickHouse topology reads independent of payload size.
+const OtelClickHouseWinnerOrder = "duration DESC, span_version DESC"
+
+// Apply the same winner before filtering or limiting every logical-span read.
+// Otherwise topology, attributes and export can describe different retry versions.
+func OtelWinningRows(columns, predicate, order string) string {
+ return "(SELECT " + columns + " FROM (SELECT " + columns +
+ ", ROW_NUMBER() OVER (PARTITION BY project_id, trace_id, span_id ORDER BY " + order +
+ ") AS otel_version FROM " + SpansTable + " WHERE " + predicate + ") ranked WHERE otel_version = 1) winners"
+}
+
+func OtelTopologyQuery(lookups []SpanLookup, combinedKey bool, fromUnixNano uint64, timeValue func(time.Time) any) (string, []any) {
+ predicate, args := OtelTracePredicate(lookups, combinedKey)
+ predicate, args = OtelWindowPredicate(predicate, args, lookups, timeValue)
+ startOrder := "start_time_unix_nano"
+ if !combinedKey {
+ startOrder = "CAST(start_time_unix_nano AS INTEGER)"
+ }
+ query := "SELECT " + OtelTopologyColumns + " FROM " + OtelWinningRows(OtelTopologyColumns, predicate, OtelTextWinnerOrder)
+ if fromUnixNano > 0 {
+ query += " WHERE " + startOrder + " >= ?"
+ args = append(args, OtelNanosArgument(fromUnixNano))
+ }
+ return query + fmt.Sprintf(" ORDER BY %s, span_id LIMIT %d", startOrder, MaxOtelGraphRows+1), args
+}
+
+// OtelImportanceOrder decides which spans survive the row cap on a whole trace read: errors, then entry points, then
+// the slowest. A parent usually outlasts its children, so ordering by duration also tends to keep the path above a span.
+const OtelImportanceOrder = " ORDER BY (status_code = 2) DESC, (parent_span_id = '' OR span_kind IN (2, 5)) DESC, duration DESC, span_id"
+
+const OtelStatementPreviewLength = 240
+
+// OtelOutlineQuery reads one whole trace without its attributes, from every project named by the lookups: services of
+// one trace often report to different projects. statement is the backend's expression for the preview
+// of the database statement, the one attribute a row label needs.
+func OtelOutlineQuery(lookups []SpanLookup, combinedKey bool, statement string, timeValue func(time.Time) any) (string, []any) {
+ predicate, args := OtelTracePredicate(lookups, combinedKey)
+ predicate, args = OtelWindowPredicate(predicate, args, lookups, timeValue)
+ return "SELECT " + OtelTopologyColumns + ", " + statement + " FROM " + OtelWinningRows(OtelTopologyColumns+", span_attributes", predicate, OtelTextWinnerOrder) + OtelImportanceOrder + fmt.Sprintf(" LIMIT %d", MaxOtelGraphRows+1), args
+}
+
+// OtelWindowPredicate adds the lookup window to a predicate on the spans table.
+func OtelWindowPredicate(predicate string, args []any, lookups []SpanLookup, timeValue func(time.Time) any) (string, []any) {
+ from, to := OtelLookupWindow(lookups)
+ return predicate + " AND recorded_at >= ? AND recorded_at <= ?", append(args, timeValue(from), timeValue(to))
+}
+
+func OtelPlaceholders(count int) string {
+ return strings.TrimSuffix(strings.Repeat("?,", count), ",")
+}
+
+// OtelNanosArgument keeps a start bound inside what database/sql can bind.
+func OtelNanosArgument(nanos uint64) int64 {
+ return int64(min(nanos, uint64(math.MaxInt64)))
+}
+
+func ScanOtelAttributeRows(rows *sql.Rows) (map[string]OtelSpanAttributes, error) {
+ result := make(map[string]OtelSpanAttributes)
+ for rows.Next() {
+ var id, encoded string
+ var flag int
+ if err := rows.Scan(&id, &encoded, &flag); err != nil {
+ return nil, err
+ }
+ if _, seen := result[id]; seen {
+ continue
+ }
+ attributes := OtelSpanAttributes{Omitted: OtelAttributeOmission(flag), Bytes: len(encoded)}
+ if attributes.Omitted == "" && encoded != "" {
+ if err := json.Unmarshal([]byte(encoded), &attributes.Attributes); err != nil {
+ return nil, err
+ }
+ }
+ result[id] = attributes
+ }
+ return result, rows.Err()
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_json.go b/backend/app/repositories/telemetry/shared/otel_json.go
new file mode 100644
index 00000000..eccbc9ff
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_json.go
@@ -0,0 +1,132 @@
+package shared
+
+import (
+ "encoding/base64"
+ "encoding/hex"
+ "encoding/json"
+ "math"
+ "strconv"
+
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/encoding/protojson"
+ "google.golang.org/protobuf/proto"
+)
+
+func JSONAttributes(attributes []*commonpb.KeyValue) map[string]any {
+ result := make(map[string]any, len(attributes))
+ for _, attribute := range attributes {
+ result[attribute.Key] = jsonAttributeValue(attribute.Value)
+ }
+ return result
+}
+
+// StringAttributes is the span_attributes value on every backend, a Map on ClickHouse and a JSON object elsewhere. It
+// follows the OpenTelemetry Collector's ClickHouse exporter: keys verbatim, scalars as text, arrays and key/value lists
+// as JSON. Exact types stay in the lossless payload.
+func StringAttributes(attributes []*commonpb.KeyValue) map[string]string {
+ result := make(map[string]string, len(attributes))
+ for _, attribute := range attributes {
+ if attribute.Key == "exception.stacktrace" {
+ continue
+ }
+ switch value := jsonAttributeValue(attribute.Value).(type) {
+ case nil:
+ case string:
+ result[attribute.Key] = value
+ default:
+ encoded, _ := json.Marshal(value)
+ result[attribute.Key] = string(encoded)
+ }
+ }
+ return result
+}
+
+func jsonAttributeValue(value *commonpb.AnyValue) any {
+ if value == nil {
+ return nil
+ }
+ switch v := value.Value.(type) {
+ case *commonpb.AnyValue_StringValue:
+ return v.StringValue
+ case *commonpb.AnyValue_IntValue:
+ return v.IntValue
+ case *commonpb.AnyValue_DoubleValue:
+ if math.IsNaN(v.DoubleValue) || math.IsInf(v.DoubleValue, 0) {
+ return strconv.FormatFloat(v.DoubleValue, 'g', -1, 64)
+ }
+ return v.DoubleValue
+ case *commonpb.AnyValue_BoolValue:
+ return v.BoolValue
+ case *commonpb.AnyValue_BytesValue:
+ return base64.StdEncoding.EncodeToString(v.BytesValue)
+ case *commonpb.AnyValue_ArrayValue:
+ values := make([]any, len(v.ArrayValue.Values))
+ for i, value := range v.ArrayValue.Values {
+ values[i] = jsonAttributeValue(value)
+ }
+ return values
+ case *commonpb.AnyValue_KvlistValue:
+ return JSONAttributes(v.KvlistValue.Values)
+ default:
+ return nil
+ }
+}
+
+func MetadataJSON(message proto.Message, attributes []*commonpb.KeyValue) (string, error) {
+ encoded, err := protojson.Marshal(message)
+ if err != nil {
+ return "", err
+ }
+ var fields map[string]json.RawMessage
+ if err := json.Unmarshal(encoded, &fields); err != nil {
+ return "", err
+ }
+ text, err := json.Marshal(JSONAttributes(attributes))
+ if err != nil {
+ return "", err
+ }
+ fields["attributes"] = text
+ encoded, err = json.Marshal(fields)
+ return string(encoded), err
+}
+
+type eventText struct {
+ TimeUnixNano uint64 `json:"time_unix_nano"`
+ Name string `json:"name"`
+ Attributes map[string]any `json:"attributes"`
+ DroppedAttributesCount uint32 `json:"dropped_attributes_count"`
+}
+
+type linkText struct {
+ TraceId string `json:"trace_id"`
+ SpanId string `json:"span_id"`
+ TraceState string `json:"trace_state"`
+ Flags uint32 `json:"flags"`
+ Attributes map[string]any `json:"attributes"`
+ DroppedAttributesCount uint32 `json:"dropped_attributes_count"`
+}
+
+func EventsJSON(events []*tracepb.Span_Event) (string, error) {
+ if len(events) == 0 {
+ return "[]", nil
+ }
+ values := make([]eventText, len(events))
+ for i, event := range events {
+ values[i] = eventText{event.TimeUnixNano, event.Name, JSONAttributes(event.Attributes), event.DroppedAttributesCount}
+ }
+ encoded, err := json.Marshal(values)
+ return string(encoded), err
+}
+
+func LinksJSON(links []*tracepb.Span_Link) (string, error) {
+ if len(links) == 0 {
+ return "[]", nil
+ }
+ values := make([]linkText, len(links))
+ for i, link := range links {
+ values[i] = linkText{hex.EncodeToString(link.TraceId), hex.EncodeToString(link.SpanId), link.TraceState, link.Flags, JSONAttributes(link.Attributes), link.DroppedAttributesCount}
+ }
+ encoded, err := json.Marshal(values)
+ return string(encoded), err
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_json_test.go b/backend/app/repositories/telemetry/shared/otel_json_test.go
new file mode 100644
index 00000000..a3761f54
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_json_test.go
@@ -0,0 +1,75 @@
+package shared
+
+import (
+ "reflect"
+ "strings"
+ "testing"
+
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+)
+
+func TestOtelResourceTextIsDeterministic(t *testing.T) {
+ values := map[string]string{"service.name": "api", "k8s.pod.name": "cart-1", "note": `a<b & "q" 100%_done`}
+ build := func(order ...string) string {
+ resource := &resourcepb.Resource{DroppedAttributesCount: 3}
+ for _, key := range order {
+ resource.Attributes = append(resource.Attributes, &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: values[key]}}})
+ }
+ text, err := MetadataJSON(resource, resource.Attributes)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return text
+ }
+ want := `{"attributes":{"k8s.pod.name":"cart-1","note":"a\u003cb \u0026 \"q\" 100%_done","service.name":"api"},"droppedAttributesCount":3}`
+ for range 50 {
+ if got := build("note", "k8s.pod.name", "service.name"); got != want || got != build("service.name", "note", "k8s.pod.name") {
+ t.Fatalf("dictionary columns and LIKE filters need stable resource text:\nwant %s\ngot %s", want, got)
+ }
+ }
+}
+
+func TestOtelTextColumnsKeepAttributeKeysVerbatim(t *testing.T) {
+ attributes := []*commonpb.KeyValue{{Key: "rate%", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: "high"}}}}
+ events, err := EventsJSON([]*tracepb.Span_Event{{Name: "event", TimeUnixNano: 18446744073709551615, Attributes: attributes, DroppedAttributesCount: 2}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ links, err := LinksJSON([]*tracepb.Span_Link{{TraceId: []byte{1, 2}, SpanId: []byte{3}, TraceState: "vendor=state", Flags: 0xffffffff, Attributes: attributes}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if events != `[{"time_unix_nano":18446744073709551615,"name":"event","attributes":{"rate%":"high"},"dropped_attributes_count":2}]` ||
+ links != `[{"trace_id":"0102","span_id":"03","trace_state":"vendor=state","flags":4294967295,"attributes":{"rate%":"high"},"dropped_attributes_count":0}]` {
+ t.Fatalf("text columns changed:\n%s\n%s", events, links)
+ }
+ if stored := StringAttributes(attributes); stored["rate%"] != "high" || strings.Contains(events, "%25") {
+ t.Fatal("attribute keys must be stored verbatim everywhere")
+ }
+ if empty, err := EventsJSON(nil); err != nil || empty != "[]" {
+ t.Fatalf("empty events: %q %v", empty, err)
+ }
+}
+
+func TestOtelSpanAttributesMapFollowsTheCollectorConvention(t *testing.T) {
+ text := func(v string) *commonpb.AnyValue {
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: v}}
+ }
+ stored := StringAttributes([]*commonpb.KeyValue{
+ {Key: "http.method", Value: text("POST")},
+ {Key: "http%2Emethod", Value: text("literal")},
+ {Key: "large", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: 9223372036854775807}}},
+ {Key: "enabled", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: true}}},
+ {Key: "ratio", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: 1.25}}},
+ {Key: "list", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: []*commonpb.AnyValue{text("a"), {Value: &commonpb.AnyValue_IntValue{IntValue: 2}}}}}}},
+ {Key: "nested", Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_KvlistValue{KvlistValue: &commonpb.KeyValueList{Values: []*commonpb.KeyValue{{Key: "k", Value: text("v")}}}}}},
+ {Key: "unset"},
+ {Key: "exception.stacktrace", Value: text("Error: boom\n at f")},
+ })
+ want := map[string]string{"http.method": "POST", "http%2Emethod": "literal", "large": "9223372036854775807", "enabled": "true", "ratio": "1.25", "list": `["a",2]`, "nested": `{"k":"v"}`}
+ if !reflect.DeepEqual(stored, want) {
+ t.Fatalf("span_attributes map:\nwant %v\ngot %v", want, stored)
+ }
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_search.go b/backend/app/repositories/telemetry/shared/otel_search.go
new file mode 100644
index 00000000..1b208eec
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_search.go
@@ -0,0 +1,100 @@
+package shared
+
+import (
+ "strings"
+ "time"
+
+ "github.com/google/uuid"
+)
+
+const MaxOtelSearchAttributeFilters = 10
+
+type OtelAttributeFilter struct{ Key, Value string }
+
+// OtelSpanSearch is one page of the span explorer. From and To are required: a search is always bounded by time.
+type OtelSpanSearch struct {
+ ProjectId uuid.UUID
+ From, To time.Time
+ Service, Name, TraceId string
+ Kind, Status *int32
+ MinDuration, MaxDuration time.Duration
+ Attributes []OtelAttributeFilter
+ OrderBy string
+ Page, PageSize int
+}
+
+// OtelSearchDialect holds what differs between the three backends. Every fragment uses ? placeholders.
+type OtelSearchDialect struct {
+ Project func(uuid.UUID) any
+ Time func(time.Time) any
+ Trace func(project uuid.UUID, traceHex string) (string, any)
+ NameLike string
+ Attribute string
+ StartOrder string
+ ReadSetting string
+ WinnerOrder string
+}
+
+var otelSearchOrders = map[string]string{
+ "start_time desc": "%s DESC, span_id DESC",
+ "start_time asc": "%s ASC, span_id ASC",
+ "duration desc": "duration DESC, %s DESC",
+ "duration asc": "duration ASC, %s DESC",
+}
+
+func otelSearchPredicate(search OtelSpanSearch, dialect OtelSearchDialect) (string, []any) {
+ conditions := []string{"project_id = ?", "recorded_at >= ?", "recorded_at <= ?"}
+ args := []any{dialect.Project(search.ProjectId), dialect.Time(search.From), dialect.Time(search.To)}
+ add := func(condition string, values ...any) {
+ conditions, args = append(conditions, condition), append(args, values...)
+ }
+ if search.TraceId != "" {
+ condition, value := dialect.Trace(search.ProjectId, search.TraceId)
+ add(condition, value)
+ }
+ if search.Service != "" {
+ add("service_name = ?", search.Service)
+ }
+ if search.Name != "" {
+ add(dialect.NameLike, search.Name)
+ }
+ if search.Kind != nil {
+ add("span_kind = ?", *search.Kind)
+ }
+ if search.Status != nil {
+ add("status_code = ?", *search.Status)
+ }
+ if search.MinDuration > 0 {
+ add("duration >= ?", int64(search.MinDuration))
+ }
+ if search.MaxDuration > 0 {
+ add("duration <= ?", int64(search.MaxDuration))
+ }
+ for _, filter := range search.Attributes {
+ add(dialect.Attribute, filter.Key, filter.Value)
+ }
+ return strings.Join(conditions, " AND "), args
+}
+
+// OtelSearchQueries returns the page query, which selects OtelTopologyColumns, and the matching count query.
+func OtelSearchQueries(search OtelSpanSearch, dialect OtelSearchDialect) (page string, pageArgs []any, count string, countArgs []any) {
+ predicate, args := otelSearchPredicate(search, dialect)
+ order, known := otelSearchOrders[search.OrderBy]
+ if !known {
+ order = otelSearchOrders["start_time desc"]
+ }
+ order = strings.ReplaceAll(order, "%s", dialect.StartOrder)
+ order += ", trace_id, span_id"
+ scope, scopeArgs := otelSearchPredicate(OtelSpanSearch{ProjectId: search.ProjectId, From: search.From, To: search.To, TraceId: search.TraceId}, dialect)
+ winners := OtelWinningRows("*", scope, dialect.WinnerOrder)
+ args = append(scopeArgs, args...)
+ page = "SELECT " + OtelTopologyColumns + " FROM " + winners + " WHERE " + predicate + " ORDER BY " + order + " LIMIT ? OFFSET ?" + dialect.ReadSetting
+ pageArgs = append(append([]any{}, args...), search.PageSize, (search.Page-1)*search.PageSize)
+ return page, pageArgs, "SELECT count(*) FROM " + winners + " WHERE " + predicate + dialect.ReadSetting, args
+}
+
+// OtelServicesQuery lists the services that reported spans in the range, busiest first, for the explorer's filter.
+func OtelServicesQuery(project uuid.UUID, from, to time.Time, dialect OtelSearchDialect) (string, []any) {
+ return "SELECT service_name FROM " + SpansTable + " WHERE project_id = ? AND recorded_at >= ? AND recorded_at <= ? AND service_name != '' GROUP BY service_name ORDER BY count(*) DESC LIMIT 100" + dialect.ReadSetting,
+ []any{dialect.Project(project), dialect.Time(from), dialect.Time(to)}
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_span.go b/backend/app/repositories/telemetry/shared/otel_span.go
new file mode 100644
index 00000000..b0263a93
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_span.go
@@ -0,0 +1,46 @@
+package shared
+
+import (
+ "encoding/json"
+ "strings"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+const SpansTable = "spans_v2"
+
+// OtelKey is the DuckDB trace_key: that backend keeps one index, and it has to cover project and trace together.
+func OtelKey(project uuid.UUID, id string) string { return project.String() + ":" + id }
+
+func OtelTracePredicate(lookups []SpanLookup, combinedKey bool) (string, []any) {
+ conditions := make([]string, 0, len(lookups))
+ args := make([]any, 0, len(lookups)*2)
+ for _, lookup := range lookups {
+ if combinedKey {
+ conditions = append(conditions, "?")
+ args = append(args, OtelKey(lookup.ProjectId, lookup.TraceId))
+ } else {
+ conditions = append(conditions, "(project_id = ? AND trace_id = ?)")
+ args = append(args, lookup.ProjectId.String(), lookup.TraceId)
+ }
+ }
+ if combinedKey {
+ return "trace_key IN (" + strings.Join(conditions, ",") + ")", args
+ }
+ return "(" + strings.Join(conditions, " OR ") + ")", args
+}
+
+func preferOtelSpan(a, b models.OtelSpan) bool {
+ if a.Duration != b.Duration {
+ return a.Duration > b.Duration
+ }
+ aa, _ := json.Marshal(a)
+ bb, _ := json.Marshal(b)
+ return string(aa) > string(bb)
+}
+
+// NormalizeTraceId turns any spelling of an id (a UUID with dashes, upper case hex) into the stored one: lowercase hex.
+func NormalizeTraceId(id string) string {
+ return strings.ToLower(strings.ReplaceAll(strings.TrimSpace(id), "-", ""))
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_span_row.go b/backend/app/repositories/telemetry/shared/otel_span_row.go
new file mode 100644
index 00000000..7923281f
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_span_row.go
@@ -0,0 +1,290 @@
+package shared
+
+import (
+ "bytes"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "slices"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ traceway "go.tracewayapp.com"
+ "google.golang.org/protobuf/encoding/protowire"
+ "google.golang.org/protobuf/proto"
+)
+
+const OtelScalarColumns = `project_id, trace_id, span_id, parent_span_id, name, span_kind, status_code, status_message, service_name, scope_name, scope_version, start_time_unix_nano, end_time_unix_nano, duration, recorded_at, trace_state, flags, dropped_attributes_count, dropped_events_count, dropped_links_count, resource_schema_url, scope_schema_url`
+
+const OtelTextStorageColumns = OtelScalarColumns + `, span_attributes, resource, scope, events, links, otlp`
+
+var OtelTextInsertSQL = `INSERT INTO ` + SpansTable + ` (` + OtelTextStorageColumns + `) VALUES (` + strings.TrimSuffix(strings.Repeat("?,", len(strings.Split(OtelTextStorageColumns, ","))), ",") + `)`
+
+var (
+ deterministicProto = proto.MarshalOptions{Deterministic: true}
+ resourceSpansScopeTag = (&tracepb.ResourceSpans{}).ProtoReflect().Descriptor().Fields().ByName("scope_spans").Number()
+ scopeSpansSpanTag = (&tracepb.ScopeSpans{}).ProtoReflect().Descriptor().Fields().ByName("spans").Number()
+)
+
+type OtelSpanGroup struct {
+ Resource *resourcepb.Resource
+ Scope *commonpb.InstrumentationScope
+ ResourceSchemaUrl string
+ ScopeSchemaUrl string
+ ResourcePB []byte
+ ScopePB []byte
+ memo map[string]string
+}
+
+func (g *OtelSpanGroup) Memo(key string, build func() (string, error)) (string, error) {
+ if value, ok := g.memo[key]; ok {
+ return value, nil
+ }
+ value, err := build()
+ if err != nil {
+ return "", err
+ }
+ g.memo[key] = value
+ return value, nil
+}
+
+type OtelSpanGroups map[*tracepb.ResourceSpans]*OtelSpanGroup
+
+func (groups OtelSpanGroups) group(context *tracepb.ResourceSpans) (*OtelSpanGroup, error) {
+ if group, ok := groups[context]; ok {
+ return group, nil
+ }
+ group := &OtelSpanGroup{memo: make(map[string]string)}
+ if context != nil {
+ if len(context.ScopeSpans) != 1 {
+ return nil, fmt.Errorf("expected one instrumentation scope per stored span")
+ }
+ scopeEnvelope := context.ScopeSpans[0]
+ if len(scopeEnvelope.Spans) > 0 {
+ scopeEnvelope = proto.Clone(scopeEnvelope).(*tracepb.ScopeSpans)
+ scopeEnvelope.Spans = nil
+ }
+ resourceEnvelope := proto.Clone(context).(*tracepb.ResourceSpans)
+ resourceEnvelope.ScopeSpans = nil
+ var err error
+ if group.ResourcePB, err = deterministicProto.Marshal(resourceEnvelope); err != nil {
+ return nil, err
+ }
+ if group.ScopePB, err = deterministicProto.Marshal(scopeEnvelope); err != nil {
+ return nil, err
+ }
+ group.Resource, group.ResourceSchemaUrl = context.Resource, context.SchemaUrl
+ group.Scope, group.ScopeSchemaUrl = scopeEnvelope.Scope, scopeEnvelope.SchemaUrl
+ }
+ groups[context] = group
+ return group, nil
+}
+
+// Valid because protobuf accepts a message's fields in any order: each envelope only lacks its one nested field.
+func AssembleOtelPayload(resourcePB, scopePB, spanPB []byte) []byte {
+ scopeSpans := make([]byte, 0, len(scopePB)+len(spanPB)+8)
+ scopeSpans = append(scopeSpans, scopePB...)
+ scopeSpans = protowire.AppendTag(scopeSpans, scopeSpansSpanTag, protowire.BytesType)
+ scopeSpans = protowire.AppendBytes(scopeSpans, spanPB)
+ payload := make([]byte, 0, len(resourcePB)+len(scopeSpans)+8)
+ payload = append(payload, resourcePB...)
+ payload = protowire.AppendTag(payload, resourceSpansScopeTag, protowire.BytesType)
+ return protowire.AppendBytes(payload, scopeSpans)
+}
+
+const rejectedSpanReportInterval = time.Minute
+
+var rejectedSpans struct {
+ sync.Mutex
+ pending uint64
+ reported time.Time
+}
+
+func RecordRejectedOtelSpan(err error) {
+ db.RecordTelemetryRowDropped(SpansTable)
+ rejectedSpans.Lock()
+ rejectedSpans.pending++
+ var report uint64
+ if time.Since(rejectedSpans.reported) >= rejectedSpanReportInterval {
+ report, rejectedSpans.pending, rejectedSpans.reported = rejectedSpans.pending, 0, time.Now()
+ }
+ rejectedSpans.Unlock()
+ if report > 0 {
+ traceway.CaptureException(fmt.Errorf("otel span insert: rejected %d spans since last report, latest: %w", report, err))
+ }
+}
+
+type OtelSpanRow struct {
+ Span models.Span
+ Source *tracepb.Span
+ Group *OtelSpanGroup
+ SpanPB []byte
+ Synthesized bool
+ PartitionTime time.Time
+}
+
+// Native UUIDs stay unchanged in Traceway's API. OTLP requires eight-byte span
+// IDs, so export uses this stable, domain-separated mapping for both ends of an edge.
+func nativeOTLPSpanID(id string) ([]byte, error) {
+ raw, err := hex.DecodeString(id)
+ if err != nil || (len(raw) != 8 && len(raw) != 16) || bytes.Equal(raw, make([]byte, len(raw))) {
+ return nil, fmt.Errorf("invalid span ID %q", id)
+ }
+ if len(raw) == 8 {
+ return raw, nil
+ }
+ hash := sha256.Sum256(append([]byte("traceway.native.span-id.v1:"), raw...))
+ result := hash[:8]
+ if bytes.Equal(result, make([]byte, 8)) {
+ result[7] = 1
+ }
+ return result, nil
+}
+
+func nativeStringAttribute(key, value string) *commonpb.KeyValue {
+ return &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: value}}}
+}
+
+// NewOtelSpanRow prepares one span for storage. A span that did not arrive as OTLP, one from the native protocol, gets
+// a payload built from its fields, so every stored span can be returned in the same format.
+func NewOtelSpanRow(span models.OtelSpan, groups OtelSpanGroups) (*OtelSpanRow, error) {
+ traceID, err := hex.DecodeString(span.TraceId)
+ if err != nil || len(traceID) != 16 || bytes.Equal(traceID, make([]byte, 16)) {
+ return nil, fmt.Errorf("invalid trace ID %q", span.TraceId)
+ }
+ sourceID, err := hex.DecodeString(span.SpanId)
+ if err != nil || (len(sourceID) != 8 && (span.OTLP != nil || len(sourceID) != 16)) {
+ return nil, fmt.Errorf("invalid span ID %q", span.SpanId)
+ }
+ row := &OtelSpanRow{Span: span.Span, Source: span.OTLP}
+ if row.Source == nil {
+ row.Synthesized = true
+ sourceID, err = nativeOTLPSpanID(span.SpanId)
+ if err != nil {
+ return nil, err
+ }
+ row.Source = &tracepb.Span{TraceId: traceID, SpanId: sourceID, Name: span.Name, Kind: tracepb.Span_SpanKind(span.SpanKind),
+ StartTimeUnixNano: uint64(span.StartTime.UnixNano()), EndTimeUnixNano: uint64(span.StartTime.Add(span.Duration).UnixNano())}
+ if span.StatusCode != 0 {
+ row.Source.Status = &tracepb.Status{Code: tracepb.Status_StatusCode(span.StatusCode)}
+ }
+ if span.ParentSpanId != "" {
+ row.Source.ParentSpanId, err = nativeOTLPSpanID(span.ParentSpanId)
+ if err != nil {
+ return nil, err
+ }
+ }
+ keys := make([]string, 0, len(span.Attributes))
+ for key := range span.Attributes {
+ keys = append(keys, key)
+ }
+ slices.Sort(keys)
+ for _, key := range keys {
+ if key != "traceway.native.span_id" && key != "traceway.native.parent_span_id" {
+ row.Source.Attributes = append(row.Source.Attributes, nativeStringAttribute(key, span.Attributes[key]))
+ }
+ }
+ if len(span.SpanId) == 32 {
+ row.Source.Attributes = append(row.Source.Attributes, nativeStringAttribute("traceway.native.span_id", span.SpanId))
+ }
+ if len(span.ParentSpanId) == 32 {
+ row.Source.Attributes = append(row.Source.Attributes, nativeStringAttribute("traceway.native.parent_span_id", span.ParentSpanId))
+ }
+ if span.Context == nil {
+ resource := &resourcepb.Resource{}
+ if span.ServiceName != "" {
+ resource.Attributes = []*commonpb.KeyValue{nativeStringAttribute("service.name", span.ServiceName)}
+ }
+ span.Context = &tracepb.ResourceSpans{Resource: resource, ScopeSpans: []*tracepb.ScopeSpans{{Scope: &commonpb.InstrumentationScope{Name: span.ScopeName}}}}
+ }
+ }
+ if row.Group, err = groups.group(span.Context); err != nil {
+ return nil, err
+ }
+ var fallback bool
+ if row.PartitionTime, _, fallback = OtelStorageTimes(row.Source.StartTimeUnixNano, time.Now()); fallback {
+ db.RecordSpanPartitionTimeFallback()
+ }
+ if row.SpanPB, err = deterministicProto.Marshal(row.Source); err != nil {
+ return nil, err
+ }
+ return row, nil
+}
+
+func (row *OtelSpanRow) Payload() []byte {
+ return AssembleOtelPayload(row.Group.ResourcePB, row.Group.ScopePB, row.SpanPB)
+}
+
+type OtelValueCodec struct {
+ UUID func(uuid.UUID) any
+ Time func(time.Time) any
+ Nanos func(uint64) any
+ Uint32 func(uint32) any
+}
+
+// Order must match OtelScalarColumns.
+func (row *OtelSpanRow) ScalarValues(codec OtelValueCodec) []any {
+ span, source, group := row.Span, row.Source, row.Group
+ return []any{
+ codec.UUID(span.ProjectId), span.TraceId, span.SpanId, span.ParentSpanId, span.Name, span.SpanKind, span.StatusCode, source.GetStatus().GetMessage(),
+ span.ServiceName, span.ScopeName, group.Scope.GetVersion(), codec.Nanos(source.StartTimeUnixNano), codec.Nanos(source.EndTimeUnixNano),
+ int64(span.Duration), codec.Time(row.PartitionTime), source.TraceState, codec.Uint32(source.Flags),
+ codec.Uint32(source.DroppedAttributesCount), codec.Uint32(source.DroppedEventsCount), codec.Uint32(source.DroppedLinksCount),
+ group.ResourceSchemaUrl, group.ScopeSchemaUrl,
+ }
+}
+
+// NestedValues holds the variable-size parts of a span the way every backend stores them: the attributes as a string
+// map, then resource, scope, events and links as JSON text. Resource and scope are encoded once per group.
+func (row *OtelSpanRow) NestedValues() ([]any, error) {
+ attributes := StringAttributes(row.Source.GetAttributes())
+ if row.Synthesized {
+ attributes = make(map[string]string, len(row.Span.Attributes))
+ for key, value := range row.Span.Attributes {
+ attributes[key] = value
+ }
+ }
+ resource, err := row.Group.Memo("resource", func() (string, error) {
+ return MetadataJSON(row.Group.Resource, row.Group.Resource.GetAttributes())
+ })
+ if err != nil {
+ return nil, err
+ }
+ scope, err := row.Group.Memo("scope", func() (string, error) {
+ return MetadataJSON(row.Group.Scope, row.Group.Scope.GetAttributes())
+ })
+ if err != nil {
+ return nil, err
+ }
+ events, err := EventsJSON(row.Source.GetEvents())
+ if err != nil {
+ return nil, err
+ }
+ links, err := LinksJSON(row.Source.GetLinks())
+ if err != nil {
+ return nil, err
+ }
+ return []any{attributes, resource, scope, events, links}, nil
+}
+
+// Order must match the columns OtelTextStorageColumns adds after OtelScalarColumns.
+func (row *OtelSpanRow) TextValues() ([]any, error) {
+ values, err := row.NestedValues()
+ if err != nil {
+ return nil, err
+ }
+ attributes, err := json.Marshal(values[0])
+ if err != nil {
+ return nil, err
+ }
+ values[0] = string(attributes)
+ return append(values, row.Payload()), nil
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_span_row_test.go b/backend/app/repositories/telemetry/shared/otel_span_row_test.go
new file mode 100644
index 00000000..e9a898e1
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_span_row_test.go
@@ -0,0 +1,180 @@
+package shared
+
+import (
+ "fmt"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+var testOtelCodec = OtelValueCodec{
+ UUID: func(id uuid.UUID) any { return id },
+ Time: func(t time.Time) any { return t },
+ Nanos: func(n uint64) any { return n },
+ Uint32: func(n uint32) any { return n },
+}
+
+func stringAttribute(key, value string) *commonpb.KeyValue {
+ return &commonpb.KeyValue{Key: key, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: value}}}
+}
+
+func testOtelContext(attributes int) *tracepb.ResourceSpans {
+ resource := &resourcepb.Resource{DroppedAttributesCount: 6}
+ for i := 0; i < attributes; i++ {
+ resource.Attributes = append(resource.Attributes, stringAttribute(fmt.Sprintf("k8s.resource.attribute.%d", i), fmt.Sprintf("value-%d-prod-eu-central-1", i)))
+ }
+ return &tracepb.ResourceSpans{Resource: resource, SchemaUrl: "resource-schema", ScopeSpans: []*tracepb.ScopeSpans{{
+ Scope: &commonpb.InstrumentationScope{Name: "scope", Version: "v1", Attributes: []*commonpb.KeyValue{stringAttribute("scope.key", "value")}}, SchemaUrl: "scope-schema"}}}
+}
+
+func testOtelSpan(context *tracepb.ResourceSpans, spanID byte) models.OtelSpan {
+ trace := uuid.MustParse("01020304-0506-0708-090a-0b0c0d0e0f10")
+ start := time.Unix(1789747200, 123456789).UTC()
+ source := &tracepb.Span{TraceId: trace[:], SpanId: []byte{spanID, 2, 3, 4, 5, 6, 7, 8}, Name: "GET /orders", Kind: tracepb.Span_SPAN_KIND_SERVER,
+ StartTimeUnixNano: uint64(start.UnixNano()), EndTimeUnixNano: uint64(start.Add(time.Second).UnixNano()), TraceState: "vendor=value", Flags: 0x301,
+ Status: &tracepb.Status{Code: tracepb.Status_STATUS_CODE_ERROR, Message: "detail"},
+ Attributes: []*commonpb.KeyValue{stringAttribute("http.request.method", "GET"), stringAttribute("http.route", "/orders")},
+ Events: []*tracepb.Span_Event{{Name: "exception", TimeUnixNano: uint64(start.UnixNano()) + 1}},
+ Links: []*tracepb.Span_Link{{TraceId: trace[:], SpanId: []byte{9, 9, 9, 9, 9, 9, 9, 9}}}}
+ return models.OtelSpan{OTLP: source, Context: context, Span: models.Span{ProjectId: uuid.MustParse("00000000-0000-4000-8000-000000000001"),
+ TraceId: strings.ReplaceAll(trace.String(), "-", ""), SpanId: fmt.Sprintf("%02x02030405060708", spanID), Name: source.Name, StartTime: start, Duration: time.Second}}
+}
+
+func TestOtelPayloadAssemblyMatchesMessageAssembly(t *testing.T) {
+ unknown := testOtelContext(3)
+ unknown.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x02})
+ unknown.ScopeSpans[0].ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x03})
+ carriesSpans := testOtelContext(2)
+ carriesSpans.ScopeSpans[0].Spans = []*tracepb.Span{{Name: "must not be stored with the scope"}}
+ for name, context := range map[string]*tracepb.ResourceSpans{
+ "populated": testOtelContext(29), "unknown fields": unknown, "no resource or scope": {ScopeSpans: []*tracepb.ScopeSpans{{}}},
+ "missing context": nil, "scope already carries spans": carriesSpans,
+ } {
+ span := testOtelSpan(context, 1)
+ span.OTLP.ProtoReflect().SetUnknown([]byte{0xa0, 0x06, 0x01})
+ row, err := NewOtelSpanRow(span, OtelSpanGroups{})
+ if err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ expected := &tracepb.ResourceSpans{ScopeSpans: []*tracepb.ScopeSpans{{}}}
+ if context != nil {
+ expected = proto.Clone(context).(*tracepb.ResourceSpans)
+ }
+ expected.ScopeSpans[0].Spans = []*tracepb.Span{span.OTLP}
+ var restored tracepb.ResourceSpans
+ if err := proto.Unmarshal(row.Payload(), &restored); err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ if !proto.Equal(expected, &restored) {
+ t.Fatalf("%s: assembled payload differs from the source message\nwant %v\ngot %v", name, expected, &restored)
+ }
+ }
+}
+
+func TestOtelSpanGroupsEncodeSharedContextOnce(t *testing.T) {
+ context, groups := testOtelContext(29), OtelSpanGroups{}
+ first, err := NewOtelSpanRow(testOtelSpan(context, 1), groups)
+ if err != nil {
+ t.Fatal(err)
+ }
+ second, err := NewOtelSpanRow(testOtelSpan(context, 2), groups)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if first.Group != second.Group || len(groups) != 1 {
+ t.Fatal("spans sharing a context must share one encoded group")
+ }
+ builds := 0
+ for range 3 {
+ if _, err := first.Group.Memo("key", func() (string, error) { builds++; return "value", nil }); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if builds != 1 {
+ t.Fatalf("group encoding ran %d times", builds)
+ }
+ other, err := NewOtelSpanRow(testOtelSpan(testOtelContext(29), 3), OtelSpanGroups{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if string(other.Group.ResourcePB) != string(first.Group.ResourcePB) || string(other.Group.ScopePB) != string(first.Group.ScopePB) {
+ t.Fatal("equal resources from separate requests must encode to identical bytes so dictionary columns deduplicate them")
+ }
+ if len(first.Group.ResourcePB) == 0 || strings.Contains(string(first.SpanPB), "k8s.resource.attribute") {
+ t.Fatal("the span payload must not repeat the resource")
+ }
+}
+
+func TestOtelSpanRowValuesMatchColumns(t *testing.T) {
+ row, err := NewOtelSpanRow(testOtelSpan(testOtelContext(2), 1), OtelSpanGroups{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ scalars := row.ScalarValues(testOtelCodec)
+ text, err := row.TextValues()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if want := len(strings.Split(OtelScalarColumns, ",")); len(scalars) != want {
+ t.Fatalf("scalar values %d, columns %d", len(scalars), want)
+ }
+ if want := len(strings.Split(OtelTextStorageColumns, ",")); len(scalars)+len(text) != want || strings.Count(OtelTextInsertSQL, "?") != want {
+ t.Fatalf("text storage values %d, columns %d", len(scalars)+len(text), want)
+ }
+ multiple := testOtelContext(1)
+ multiple.ScopeSpans = append(multiple.ScopeSpans, &tracepb.ScopeSpans{})
+ if _, err := NewOtelSpanRow(testOtelSpan(multiple, 1), OtelSpanGroups{}); err == nil {
+ t.Fatal("a stored span must belong to exactly one scope")
+ }
+}
+
+func benchmarkOtelSpans(count int) []models.OtelSpan {
+ context := testOtelContext(29)
+ spans := make([]models.OtelSpan, count)
+ for i := range spans {
+ spans[i] = testOtelSpan(context, byte(i%250+1))
+ }
+ return spans
+}
+
+func BenchmarkOtelSpanRowScalars(b *testing.B) {
+ spans := benchmarkOtelSpans(512)
+ b.ReportAllocs()
+ for b.Loop() {
+ groups := OtelSpanGroups{}
+ for _, span := range spans {
+ row, err := NewOtelSpanRow(span, groups)
+ if err != nil {
+ b.Fatal(err)
+ }
+ row.ScalarValues(testOtelCodec)
+ }
+ }
+ b.ReportMetric(float64(b.Elapsed().Microseconds())/float64(b.N*len(spans)), "us/span")
+}
+
+func BenchmarkOtelSpanRowTextStorage(b *testing.B) {
+ spans := benchmarkOtelSpans(512)
+ b.ReportAllocs()
+ for b.Loop() {
+ groups := OtelSpanGroups{}
+ for _, span := range spans {
+ row, err := NewOtelSpanRow(span, groups)
+ if err != nil {
+ b.Fatal(err)
+ }
+ row.ScalarValues(testOtelCodec)
+ if _, err := row.TextValues(); err != nil {
+ b.Fatal(err)
+ }
+ }
+ }
+ b.ReportMetric(float64(b.Elapsed().Microseconds())/float64(b.N*len(spans)), "us/span")
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_span_test.go b/backend/app/repositories/telemetry/shared/otel_span_test.go
new file mode 100644
index 00000000..7759fb4b
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_span_test.go
@@ -0,0 +1,179 @@
+package shared
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "testing"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+type graphReader struct {
+ topology func(context.Context, []SpanLookup, uint64) ([]models.OtelSpan, error)
+ attributes func(context.Context, SpanLookup, []string, OtelAttributeLimits) (map[string]OtelSpanAttributes, error)
+}
+
+func (r graphReader) FindTraceTopology(ctx context.Context, lookups []SpanLookup, from uint64) ([]models.OtelSpan, error) {
+ return r.topology(ctx, lookups, from)
+}
+func (r graphReader) FindSpanAttributes(ctx context.Context, lookup SpanLookup, ids []string, limits OtelAttributeLimits) (map[string]OtelSpanAttributes, error) {
+ if r.attributes == nil {
+ return nil, nil
+ }
+ return r.attributes(ctx, lookup, ids, limits)
+}
+func (r graphReader) IsReadLimitError(error) bool { return false }
+
+func (r graphReader) FindTraceOutline(ctx context.Context, lookups []SpanLookup) ([]models.OtelSpan, error) {
+ return r.topology(ctx, lookups, 0)
+}
+
+func TestOtelGraphsBatchQueries(t *testing.T) {
+ project := uuid.New()
+ lookups := make([]SpanLookup, SpanLookupBatchSize+1)
+ for i := range lookups {
+ lookups[i] = SpanLookup{ProjectId: project, TraceId: NormalizeTraceId(uuid.NewString()), SpanId: "0102030405060708"}
+ }
+ calls := 0
+ reader := graphReader{topology: func(ctx context.Context, batch []SpanLookup, from uint64) ([]models.OtelSpan, error) {
+ calls++
+ if len(batch) > SpanLookupBatchSize || from != 0 {
+ t.Fatalf("unexpected batch: %d, from %d", len(batch), from)
+ }
+ if _, ok := ctx.Deadline(); !ok {
+ t.Fatal("graph query has no deadline")
+ }
+ roots := make([]models.OtelSpan, len(batch))
+ for i, lookup := range batch {
+ roots[i] = models.OtelSpan{Span: models.Span{ProjectId: project, TraceId: lookup.TraceId, SpanId: lookup.SpanId}}
+ }
+ return roots, nil
+ }}
+ found, err := FindOtelGraphs(context.Background(), lookups, reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if calls != 2 || len(found) != len(lookups) {
+ t.Fatalf("got %d calls and %d roots", calls, len(found))
+ }
+}
+
+func graphFixture() (SpanLookup, []models.OtelSpan) {
+ lookup := SpanLookup{ProjectId: uuid.New(), TraceId: NormalizeTraceId(uuid.NewString()), SpanId: "0102030405060708"}
+ return lookup, []models.OtelSpan{
+ {Span: models.Span{ProjectId: lookup.ProjectId, TraceId: lookup.TraceId, SpanId: lookup.SpanId}},
+ {Span: models.Span{ProjectId: lookup.ProjectId, TraceId: lookup.TraceId, SpanId: "1112131415161718", ParentSpanId: lookup.SpanId}},
+ }
+}
+
+func TestOtelGraphsRetainPartialTopology(t *testing.T) {
+ lookup, topology := graphFixture()
+ previous := MaxOtelGraphRows
+ MaxOtelGraphRows = len(topology)
+ t.Cleanup(func() { MaxOtelGraphRows = previous })
+ reader := graphReader{topology: func(context.Context, []SpanLookup, uint64) ([]models.OtelSpan, error) {
+ return append(topology, models.OtelSpan{}), nil
+ }}
+ found, err := FindOtelGraphs(context.Background(), []SpanLookup{lookup}, reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ graph := found[lookup.Owner()]
+ if graph == nil || len(graph.Spans) != 1 || graph.Status.State != models.SpanGraphPartial || fmt.Sprint(graph.Status.Reasons) != "[row_limit]" {
+ t.Fatalf("partial graph: %+v", graph)
+ }
+}
+
+func TestOtelGraphsReadFailures(t *testing.T) {
+ lookup, _ := graphFixture()
+ for _, failure := range []error{context.DeadlineExceeded, errors.New("missing column")} {
+ reader := graphReader{topology: func(context.Context, []SpanLookup, uint64) ([]models.OtelSpan, error) { return nil, failure }}
+ found, err := FindOtelGraphs(context.Background(), []SpanLookup{lookup}, reader)
+ if errors.Is(failure, context.DeadlineExceeded) {
+ graph := found[lookup.Owner()]
+ if err != nil || graph == nil || graph.Status.State != models.SpanGraphUnavailable || len(graph.Spans) != 0 {
+ t.Fatalf("read limit: %+v, %v", graph, err)
+ }
+ } else if !errors.Is(err, failure) {
+ t.Fatalf("storage failure hidden: %v", err)
+ }
+ }
+}
+
+func TestOtelGraphsPreserveTopologyWhenAttributesAreOmitted(t *testing.T) {
+ lookup, topology := graphFixture()
+ for _, reason := range []string{models.SpanGraphAttributeSize, models.SpanGraphAttributeBudget, models.SpanGraphAttributesUnavailable} {
+ t.Run(reason, func(t *testing.T) {
+ reader := graphReader{
+ topology: func(context.Context, []SpanLookup, uint64) ([]models.OtelSpan, error) { return topology, nil },
+ attributes: func(_ context.Context, _ SpanLookup, ids []string, limits OtelAttributeLimits) (map[string]OtelSpanAttributes, error) {
+ if limits.PerSpanBytes != MaxOtelSpanAttributeBytes || limits.BudgetBytes != MaxOtelGraphBytes {
+ t.Fatalf("limits: %+v", limits)
+ }
+ if reason == models.SpanGraphAttributesUnavailable {
+ return nil, context.DeadlineExceeded
+ }
+ return map[string]OtelSpanAttributes{ids[0]: {Omitted: reason}}, nil
+ },
+ }
+ found, err := FindOtelGraphs(context.Background(), []SpanLookup{lookup}, reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ graph := found[lookup.Owner()]
+ if len(graph.Spans) != 1 || !graph.Spans[0].AttributesOmitted || graph.Status.State != models.SpanGraphPartial || graph.Status.OmittedAttributes != 1 || fmt.Sprint(graph.Status.Reasons) != "["+reason+"]" {
+ t.Fatalf("omitted attributes: %+v", graph)
+ }
+ })
+ }
+}
+
+func TestOtelGraphsSkipLookupsWithoutIds(t *testing.T) {
+ reader := graphReader{topology: func(context.Context, []SpanLookup, uint64) ([]models.OtelSpan, error) {
+ t.Fatal("a lookup that names no trace must not read spans")
+ return nil, nil
+ }}
+ found, err := FindOtelGraphs(context.Background(), []SpanLookup{{ProjectId: uuid.New()}, {ProjectId: uuid.New(), TraceId: "0102030405060708090a0b0c0d0e0f10"}}, reader)
+ if err != nil || len(found) != 0 {
+ t.Fatalf("lookups without ids: %v %v", found, err)
+ }
+}
+
+// A span whose parent never arrived shows under the trace's own root and nowhere else, and never crosses into another
+// project or trace read in the same batch.
+func TestOtelGraphOrphansBelongToTheirOwnRoot(t *testing.T) {
+ project, other := uuid.New(), uuid.New()
+ trace, otherTrace := "0102030405060708090a0b0c0d0e0f10", "1112131415161718191a1b1c1d1e1f20"
+ span := func(project uuid.UUID, trace, id, parent string) models.OtelSpan {
+ return models.OtelSpan{Span: models.Span{ProjectId: project, TraceId: trace, SpanId: id, ParentSpanId: parent}}
+ }
+ topology := []models.OtelSpan{
+ span(project, trace, "aaaaaaaaaaaaaaa1", ""),
+ span(project, trace, "aaaaaaaaaaaaaaa2", "aaaaaaaaaaaaaaa1"),
+ span(project, trace, "aaaaaaaaaaaaaaa3", "00000000000000ff"),
+ span(project, otherTrace, "bbbbbbbbbbbbbbb1", "00000000000000fe"),
+ span(other, trace, "ccccccccccccccc1", "00000000000000fd"),
+ }
+ reader := graphReader{topology: func(context.Context, []SpanLookup, uint64) ([]models.OtelSpan, error) { return topology, nil }}
+ root := SpanLookup{ProjectId: project, TraceId: trace, SpanId: "aaaaaaaaaaaaaaa1"}
+ child := SpanLookup{ProjectId: project, TraceId: trace, SpanId: "aaaaaaaaaaaaaaa2"}
+ found, err := FindOtelGraphs(context.Background(), []SpanLookup{root, child, {ProjectId: project, TraceId: otherTrace, SpanId: "bbbbbbbbbbbbbbb0"}, {ProjectId: other, TraceId: trace, SpanId: "ccccccccccccccc0"}}, reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ids := func(graph *SpanGraph) string {
+ out := []string{}
+ for _, span := range graph.Spans {
+ out = append(out, span.SpanId)
+ }
+ return fmt.Sprint(out)
+ }
+ if got := ids(found[root.Owner()]); got != "[aaaaaaaaaaaaaaa2 aaaaaaaaaaaaaaa3]" {
+ t.Fatalf("the root takes its own trace's orphan and nothing else: %s", got)
+ }
+ if got := ids(found[child.Owner()]); got != "[]" {
+ t.Fatalf("a span that is not the root takes no orphans: %s", got)
+ }
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_time.go b/backend/app/repositories/telemetry/shared/otel_time.go
new file mode 100644
index 00000000..075227b0
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_time.go
@@ -0,0 +1,42 @@
+package shared
+
+import (
+ "math"
+ "time"
+)
+
+var (
+ // ClickHouse partitions spans on a 32-bit DateTime and orders them on DateTime64(6).
+ otelPartitionTimeMax = time.Date(2106, 2, 7, 6, 28, 15, 0, time.UTC)
+ otelDisplayTimeMax = time.Date(2299, 12, 31, 23, 59, 59, 999999000, time.UTC)
+)
+
+func OtelNanosToTime(nanos uint64) time.Time {
+ return time.Unix(int64(nanos/uint64(time.Second)), int64(nanos%uint64(time.Second))).UTC()
+}
+
+// OtelStorageTimes derives the partition and ordering times from the exact source start, which is
+// stored separately. A missing or unrepresentable start falls back to the ingest time and reports it.
+func OtelStorageTimes(startUnixNano uint64, ingestedAt time.Time) (partition, display time.Time, fallback bool) {
+ start := OtelNanosToTime(startUnixNano)
+ if startUnixNano == 0 || start.After(otelPartitionTimeMax) {
+ partition, fallback = ingestedAt.UTC(), true
+ } else {
+ partition = start
+ }
+ if startUnixNano == 0 || start.After(otelDisplayTimeMax) {
+ return partition, partition, fallback
+ }
+ return partition, start, fallback
+}
+
+// OtelDuration is zero for a missing or reversed interval and saturates instead of overflowing.
+func OtelDuration(startUnixNano, endUnixNano uint64) time.Duration {
+ if endUnixNano <= startUnixNano {
+ return 0
+ }
+ if difference := endUnixNano - startUnixNano; difference < math.MaxInt64 {
+ return time.Duration(difference)
+ }
+ return time.Duration(math.MaxInt64)
+}
diff --git a/backend/app/repositories/telemetry/shared/otel_time_test.go b/backend/app/repositories/telemetry/shared/otel_time_test.go
new file mode 100644
index 00000000..4780b875
--- /dev/null
+++ b/backend/app/repositories/telemetry/shared/otel_time_test.go
@@ -0,0 +1,56 @@
+package shared
+
+import (
+ "math"
+ "testing"
+ "time"
+)
+
+func TestOtelStorageTimesPolicy(t *testing.T) {
+ ingested := time.Date(2026, 9, 18, 12, 0, 0, 0, time.UTC)
+ nanos := func(at time.Time) uint64 { return uint64(at.Unix())*uint64(time.Second) + uint64(at.Nanosecond()) }
+ historical := time.Date(2019, 3, 4, 5, 6, 7, 123456789, time.UTC)
+ afterPartitionRange := time.Date(2150, 1, 1, 0, 0, 0, 0, time.UTC)
+ afterDisplayRange := time.Date(2400, 1, 1, 0, 0, 0, 0, time.UTC)
+ for name, tc := range map[string]struct {
+ start uint64
+ partition, display time.Time
+ fallback bool
+ }{
+ "missing start uses the ingest time": {0, ingested, ingested, true},
+ "historical data keeps its own partition": {nanos(historical), historical, historical, false},
+ "the last representable partition second is kept": {nanos(otelPartitionTimeMax), otelPartitionTimeMax, otelPartitionTimeMax, false},
+ "past the partition range only the partition falls back": {nanos(afterPartitionRange), ingested, afterPartitionRange, true},
+ "past the ordering range both fall back": {nanos(afterDisplayRange), ingested, ingested, true},
+ "beyond signed nanoseconds does not wrap negative": {math.MaxInt64 + 1000, ingested, OtelNanosToTime(math.MaxInt64 + 1000), true},
+ "the largest unsigned value falls back": {math.MaxUint64, ingested, ingested, true},
+ } {
+ partition, display, fallback := OtelStorageTimes(tc.start, ingested)
+ if !partition.Equal(tc.partition) || !display.Equal(tc.display) || fallback != tc.fallback {
+ t.Errorf("%s: partition %s display %s fallback %v, want %s %s %v", name, partition, display, fallback, tc.partition, tc.display, tc.fallback)
+ }
+ if partition.Before(time.Unix(0, 0)) || partition.After(otelPartitionTimeMax) {
+ t.Errorf("%s: partition time %s cannot be stored in a 32-bit DateTime", name, partition)
+ }
+ }
+ if got := OtelNanosToTime(math.MaxInt64 + 1000); got.Year() != 2262 {
+ t.Errorf("values beyond signed nanoseconds must convert forward in time, got %s", got)
+ }
+}
+
+func TestOtelDurationPolicy(t *testing.T) {
+ for name, tc := range map[string]struct {
+ start, end uint64
+ want time.Duration
+ }{
+ "normal interval": {1_000, 4_500, 3_500},
+ "reversed interval": {4_500, 1_000, 0},
+ "missing end": {4_500, 0, 0},
+ "zero length": {7, 7, 0},
+ "saturates": {0, math.MaxUint64, time.Duration(math.MaxInt64)},
+ } {
+ if got := OtelDuration(tc.start, tc.end); got != tc.want {
+ t.Errorf("%s: %d, want %d", name, got, tc.want)
+ }
+ }
+}
diff --git a/backend/app/repositories/telemetry/shared/span_lookup.go b/backend/app/repositories/telemetry/shared/span_lookup.go
index 7c8958d5..f70b6258 100644
--- a/backend/app/repositories/telemetry/shared/span_lookup.go
+++ b/backend/app/repositories/telemetry/shared/span_lookup.go
@@ -1,7 +1,6 @@
package shared
import (
- "strings"
"time"
"github.com/google/uuid"
@@ -9,26 +8,44 @@ import (
const SpanLookupBatchSize = 100
+// SpanLookup names the spans to read: a trace in a project, and for a subtree the span it hangs under.
type SpanLookup struct {
- ProjectId uuid.UUID
- // TraceId is the owning occurrence ID, which may differ from the OTel trace ID.
- TraceId uuid.UUID
+ ProjectId uuid.UUID
+ TraceId string
+ SpanId string
RecordedAt *time.Time
+ // StartUnixNano is the start of the span named by SpanId, used to re-read a trace from that point when the row cap
+ // cut it off.
+ StartUnixNano uint64
}
-func SpanLookupQuery(lookups []SpanLookup, timeValue func(time.Time) any) (string, []any) {
- conditions := make([]string, 0, len(lookups))
- args := make([]any, 0, 4*len(lookups))
- for _, lookup := range lookups {
- condition := "project_id = ? AND trace_id = ?"
- args = append(args, lookup.ProjectId, lookup.TraceId)
+// NewSpanLookup names the subtree under an endpoint, a task or an AI trace. Each of them is recorded at its span's start.
+func NewSpanLookup(projectId uuid.UUID, traceId, spanId string, recordedAt time.Time) SpanLookup {
+ lookup := SpanLookup{ProjectId: projectId, TraceId: traceId, SpanId: spanId, RecordedAt: &recordedAt}
+ if recordedAt.Unix() > 0 {
+ lookup.StartUnixNano = uint64(recordedAt.Unix())*uint64(time.Second) + uint64(recordedAt.Nanosecond())
+ }
+ return lookup
+}
+
+// OtelLookupWindow is the recorded_at range every read of spans is held to: 24 hours either side of the rows looked up.
+// On ClickHouse that is what prunes the read to a few daily partitions. A lookup that names no time is anchored on now,
+// so no caller can reach an unbounded read.
+func OtelLookupWindow(lookups []SpanLookup) (from, to time.Time) {
+ for i, lookup := range lookups {
+ anchor := time.Now().UTC()
if lookup.RecordedAt != nil {
- from, to := TraceWindowBounds(*lookup.RecordedAt)
- condition += " AND recorded_at >= ? AND recorded_at <= ?"
- args = append(args, timeValue(from), timeValue(to))
+ anchor = *lookup.RecordedAt
+ } else if lookup.StartUnixNano != 0 {
+ anchor = OtelNanosToTime(lookup.StartUnixNano)
+ }
+ low, high := TraceWindowBounds(anchor)
+ if i == 0 || low.Before(from) {
+ from = low
+ }
+ if i == 0 || high.After(to) {
+ to = high
}
- conditions = append(conditions, "("+condition+")")
}
- return `SELECT id, trace_id, project_id, name, start_time, duration, recorded_at, parent_span_id, attributes
- FROM spans WHERE ` + strings.Join(conditions, " OR ") + ` ORDER BY start_time ASC`, args
+ return from, to
}
diff --git a/backend/app/repositories/telemetry/shared/sqlfilter.go b/backend/app/repositories/telemetry/shared/sqlfilter.go
index 1e17f571..56ec4a2c 100644
--- a/backend/app/repositories/telemetry/shared/sqlfilter.go
+++ b/backend/app/repositories/telemetry/shared/sqlfilter.go
@@ -42,3 +42,15 @@ func SortedKeys(m map[string]string) []string {
sort.Strings(keys)
return keys
}
+
+// TraceIdsFilter matches the supplied trace IDs and writes its named parameters into params.
+func TraceIdsFilter(traceIds []string, params map[string]any) string {
+ names := make([]string, len(traceIds))
+ for i, id := range traceIds {
+ key := fmt.Sprintf("trace_%d", i)
+ names[i] = ":" + key
+ params[key] = id
+ }
+ list := strings.Join(names, ",")
+ return "trace_id IN (" + list + ")"
+}
diff --git a/backend/app/repositories/telemetry/shared/types.go b/backend/app/repositories/telemetry/shared/types.go
index 50896fed..fa4dbc00 100644
--- a/backend/app/repositories/telemetry/shared/types.go
+++ b/backend/app/repositories/telemetry/shared/types.go
@@ -61,6 +61,7 @@ type LogAttributeFilter struct {
type LogSearchParams struct {
ProjectId uuid.UUID
+ ProjectIds []uuid.UUID
FromDate time.Time
ToDate time.Time
Search string
diff --git a/backend/app/repositories/telemetry/span_graph.go b/backend/app/repositories/telemetry/span_graph.go
new file mode 100644
index 00000000..78a526e8
--- /dev/null
+++ b/backend/app/repositories/telemetry/span_graph.go
@@ -0,0 +1,68 @@
+package telemetry
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+)
+
+type spanGraphRepository struct{}
+
+// InsertAsync stores spans that did not arrive as OTLP, the ones of the native protocol, next to the OTLP ones.
+func (r *spanGraphRepository) InsertAsync(ctx context.Context, spans []models.Span) error {
+ stored := make([]models.OtelSpan, len(spans))
+ for i, span := range spans {
+ stored[i] = models.OtelSpan{Span: span}
+ }
+ rejected, err := OtelSpanRepository.InsertAsync(ctx, stored)
+ if err == nil && rejected > 0 {
+ return fmt.Errorf("%d native spans could not be stored", rejected)
+ }
+ return err
+}
+
+// FindGraph returns the spans under one span of a trace: the waterfall of an endpoint, a task or an AI trace.
+func (r *spanGraphRepository) FindGraph(ctx context.Context, lookup shared.SpanLookup) (*shared.SpanGraph, error) {
+ graphs, err := r.FindGraphs(ctx, []shared.SpanLookup{lookup})
+ if err != nil {
+ return nil, err
+ }
+ if graph, ok := graphs[lookup.Owner()]; ok {
+ return graph, nil
+ }
+ return &shared.SpanGraph{Spans: []models.Span{}, Status: models.SpanGraphStatus{State: models.SpanGraphComplete}}, nil
+}
+
+func (r *spanGraphRepository) FindGraphs(ctx context.Context, lookups []shared.SpanLookup) (map[shared.SpanOwner]*shared.SpanGraph, error) {
+ return shared.FindOtelGraphs(ctx, lookups, OtelSpanRepository)
+}
+
+func traceLookups(projectIds []uuid.UUID, traceId string, at time.Time) []shared.SpanLookup {
+ lookups := make([]shared.SpanLookup, len(projectIds))
+ for i, projectId := range projectIds {
+ lookups[i] = shared.SpanLookup{ProjectId: projectId, TraceId: traceId, RecordedAt: &at}
+ }
+ return lookups
+}
+
+// FindTrace returns the stored spans of one trace from every project listed, for the span explorer. The caller decides
+// which projects the user may read. The read is anchored on at, like every other.
+func (r *spanGraphRepository) FindTrace(ctx context.Context, projectIds []uuid.UUID, traceId string, at time.Time) (*shared.SpanGraph, error) {
+ return shared.FindOtelTrace(ctx, traceLookups(projectIds, traceId, at), OtelSpanRepository)
+}
+
+// FindTraceParents maps each span of a trace to its parent span, read from every project listed.
+func (r *spanGraphRepository) FindTraceParents(ctx context.Context, projectIds []uuid.UUID, traceId string, at time.Time) (map[string]string, error) {
+ return shared.FindOtelTraceParents(ctx, traceLookups(projectIds, traceId, at), OtelSpanRepository)
+}
+
+// FindSpanAttributes loads one span's attributes for the explorer's popover. nil means the span is not stored near at.
+func (r *spanGraphRepository) FindSpanAttributes(ctx context.Context, projectId uuid.UUID, traceId, spanId string, at time.Time) (*shared.OtelSpanAttributes, error) {
+ return shared.FindOtelSpanAttributes(ctx, shared.SpanLookup{ProjectId: projectId, TraceId: traceId, RecordedAt: &at}, spanId, OtelSpanRepository)
+}
+
+var SpanRepository = &spanGraphRepository{}
diff --git a/backend/app/repositories/telemetry/span_repository_test.go b/backend/app/repositories/telemetry/span_repository_test.go
index 4a5bbd98..63019bd5 100644
--- a/backend/app/repositories/telemetry/span_repository_test.go
+++ b/backend/app/repositories/telemetry/span_repository_test.go
@@ -28,7 +28,7 @@ func TestSpanRepository_InsertAndFindByTraceId(t *testing.T) {
t.Fatalf("InsertAsync failed: %v", err)
}
- found, err := SpanRepository.FindByTraceId(ctx, projectId, traceId, nil)
+ found, err := findRunSpans(ctx, projectId, traceId, nil)
if err != nil {
t.Fatalf("FindByTraceId failed: %v", err)
}
@@ -71,7 +71,7 @@ func TestSpanRepository_AttributesRoundTrip(t *testing.T) {
t.Fatalf("InsertAsync failed: %v", err)
}
- found, err := SpanRepository.FindByTraceId(ctx, projectId, traceId, nil)
+ found, err := findRunSpans(ctx, projectId, traceId, nil)
if err != nil {
t.Fatalf("FindByTraceId failed: %v", err)
}
@@ -94,7 +94,7 @@ func TestSpanRepository_FindByTraceId_Empty(t *testing.T) {
setupTestDB(t)
ctx := context.Background()
- found, err := SpanRepository.FindByTraceId(ctx, uuid.New(), uuid.New(), nil)
+ found, err := findRunSpans(ctx, uuid.New(), uuid.New(), nil)
if err != nil {
t.Fatalf("FindByTraceId failed: %v", err)
}
@@ -129,7 +129,7 @@ func TestSpanRepository_ProjectIsolation(t *testing.T) {
t.Fatalf("InsertAsync failed: %v", err)
}
- found, err := SpanRepository.FindByTraceId(ctx, project1, traceId, nil)
+ found, err := findRunSpans(ctx, project1, traceId, nil)
if err != nil {
t.Fatalf("FindByTraceId failed: %v", err)
}
@@ -141,64 +141,52 @@ func TestSpanRepository_ProjectIsolation(t *testing.T) {
}
}
-func TestSpanRepository_FindByTraces(t *testing.T) {
+// One read serves many lookups, in batches. The same run id in another project, a run nobody asked for and a span
+// outside the 24 hour window must all stay out.
+func TestSpanRepository_FindGraphsBatchesAndIsolates(t *testing.T) {
setupTestDB(t)
ctx := context.Background()
project1, project2 := uuid.New(), uuid.New()
now := truncateMs(time.Now().UTC())
- later := now.Add(72 * time.Hour)
- var lookups []SpanLookup
+ var lookups []shared.SpanLookup
var spans []models.Span
- for i := 0; i < shared.SpanLookupBatchSize+5; i++ {
- traceID := uuid.New()
- lookups = append(lookups, SpanLookup{ProjectId: project1, TraceId: traceID, RecordedAt: &now})
- spans = append(spans, makeSpan(project1, traceID, fmt.Sprintf("span-%d", i), now.Add(-time.Duration(i)*time.Second), time.Millisecond))
+ runs := make([]uuid.UUID, shared.SpanLookupBatchSize+5)
+ for i := range runs {
+ runs[i] = uuid.New()
+ lookups = append(lookups, shared.SpanLookup{ProjectId: project1, TraceId: hexId(runs[i]), SpanId: hexId(runs[i]), RecordedAt: &now})
+ spans = append(spans, makeSpan(project1, runs[i], fmt.Sprintf("span-%d", i), now.Add(-time.Duration(i)*time.Second), time.Millisecond))
}
- parent := uuid.New()
- spans[0].ParentSpanId = &parent
spans[0].Attributes = map[string]string{"db.system": "postgresql"}
- // The same span ID in another project or occurrence must remain distinct.
- otherProject := spans[0]
- otherProject.ProjectId = project2
- otherProject.Name = "other-project"
- otherOccurrence := spans[0]
- otherOccurrence.StartTime, otherOccurrence.RecordedAt = later, later
- otherOccurrence.Name = "later-occurrence"
- lookups = append(lookups,
- lookups[0],
- SpanLookup{ProjectId: project2, TraceId: spans[0].TraceId, RecordedAt: &now},
- SpanLookup{ProjectId: project1, TraceId: spans[0].TraceId, RecordedAt: &later},
- )
- spans = append(spans, otherProject, otherOccurrence)
+ otherProject := makeSpan(project2, runs[0], "other-project", now, time.Millisecond)
+ lookups = append(lookups, lookups[0], shared.SpanLookup{ProjectId: project2, TraceId: hexId(runs[0]), SpanId: hexId(runs[0]), RecordedAt: &now})
excluded := []models.Span{
- makeSpan(uuid.New(), spans[0].TraceId, "inaccessible-project", now, time.Millisecond),
- makeSpan(project1, uuid.New(), "unrequested-owner", now, time.Millisecond),
- makeSpan(project1, spans[0].TraceId, "outside-window", now.Add(36*time.Hour), time.Millisecond),
+ makeSpan(uuid.New(), runs[0], "inaccessible-project", now, time.Millisecond),
+ makeSpan(project1, uuid.New(), "unrequested-run", now, time.Millisecond),
+ makeSpan(project1, runs[0], "outside-window", now.Add(36*time.Hour), time.Millisecond),
}
- if err := SpanRepository.InsertAsync(ctx, append(spans, excluded...)); err != nil {
+ if err := SpanRepository.InsertAsync(ctx, append(append(spans, otherProject), excluded...)); err != nil {
t.Fatal(err)
}
- found, err := SpanRepository.FindByTraces(ctx, lookups)
+ found, err := SpanRepository.FindGraphs(ctx, lookups)
if err != nil {
t.Fatal(err)
}
- if len(found) != len(spans) {
- t.Fatalf("got %d spans, want %d", len(found), len(spans))
+ if len(found) != len(runs)+1 {
+ t.Fatalf("got %d graphs, want one per distinct lookup (%d)", len(found), len(runs)+1)
}
- for i, span := range found {
- if i > 0 && span.StartTime.Before(found[i-1].StartTime) {
- t.Fatal("spans are not ordered across batches")
- }
- if span.Name == "span-0" && (span.ParentSpanId == nil || *span.ParentSpanId != parent || !reflect.DeepEqual(span.Attributes, spans[0].Attributes)) {
- t.Fatalf("span metadata did not round-trip: %+v", span)
- }
- for _, unwanted := range excluded {
- if span.Id == unwanted.Id {
- t.Fatalf("returned excluded span %s", span.Name)
- }
+ for i, lookup := range lookups[:len(runs)] {
+ graph := found[lookup.Owner()]
+ if graph == nil || len(graph.Spans) != 1 || graph.Spans[0].Name != fmt.Sprintf("span-%d", i) {
+ t.Fatalf("run %d: %+v", i, graph)
}
}
- empty, err := SpanRepository.FindByTraces(ctx, nil)
+ if first := found[lookups[0].Owner()].Spans[0]; first.ParentSpanId != hexId(runs[0]) || !reflect.DeepEqual(first.Attributes, spans[0].Attributes) {
+ t.Fatalf("span metadata did not round-trip: %+v", first)
+ }
+ if other := found[lookups[len(lookups)-1].Owner()]; len(other.Spans) != 1 || other.Spans[0].Name != "other-project" {
+ t.Fatalf("the same run id in another project is its own graph: %+v", other)
+ }
+ empty, err := SpanRepository.FindGraphs(ctx, nil)
if err != nil || len(empty) != 0 {
t.Fatalf("empty lookup: got %v, %v", empty, err)
}
diff --git a/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go b/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go
index 07e58a1f..03ed0b98 100644
--- a/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/ai_trace.repository.go
@@ -20,41 +20,43 @@ import (
type aiTraceRowNaming struct{ lit.DefaultDbNamingStrategy }
func (aiTraceRowNaming) GetTableNameFromStructName(string) string {
- return "ai_traces"
+ return "ai_traces_v2"
}
type aiTraceRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- Duration int64 `lit:"duration"`
- StatusCode uint8 `lit:"status_code"`
- Model string `lit:"model"`
- ResponseModel string `lit:"response_model"`
- Provider string `lit:"provider"`
- Operation string `lit:"operation"`
- InputTokens int64 `lit:"input_tokens"`
- OutputTokens int64 `lit:"output_tokens"`
- TotalTokens int64 `lit:"total_tokens"`
- CachedTokens int64 `lit:"cached_tokens"`
- ReasoningTokens int64 `lit:"reasoning_tokens"`
- InputCost float64 `lit:"input_cost"`
- OutputCost float64 `lit:"output_cost"`
- TotalCost float64 `lit:"total_cost"`
- TraceName string `lit:"trace_name"`
- UserId string `lit:"user_id"`
- FinishReason string `lit:"finish_reason"`
- ServerName string `lit:"server_name"`
- AppVersion string `lit:"app_version"`
- StorageKey string `lit:"storage_key"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- IsRoot bool `lit:"is_root"`
- ConversationId string `lit:"conversation_id"`
- ToolCallCount int64 `lit:"tool_call_count"`
- ToolNames string `lit:"tool_names"`
- Flagged bool `lit:"flagged"`
- FlaggedTerms string `lit:"flagged_terms"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ Duration int64 `lit:"duration"`
+ StatusCode uint8 `lit:"status_code"`
+ Model string `lit:"model"`
+ ResponseModel string `lit:"response_model"`
+ Provider string `lit:"provider"`
+ Operation string `lit:"operation"`
+ InputTokens int64 `lit:"input_tokens"`
+ OutputTokens int64 `lit:"output_tokens"`
+ TotalTokens int64 `lit:"total_tokens"`
+ CachedTokens int64 `lit:"cached_tokens"`
+ ReasoningTokens int64 `lit:"reasoning_tokens"`
+ InputCost float64 `lit:"input_cost"`
+ OutputCost float64 `lit:"output_cost"`
+ TotalCost float64 `lit:"total_cost"`
+ TraceName string `lit:"trace_name"`
+ UserId string `lit:"user_id"`
+ FinishReason string `lit:"finish_reason"`
+ ServerName string `lit:"server_name"`
+ AppVersion string `lit:"app_version"`
+ StorageKey string `lit:"storage_key"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ ParentSpanId string `lit:"parent_span_id"`
+ IsRoot bool `lit:"is_root"`
+ ConversationId string `lit:"conversation_id"`
+ ToolCallCount int64 `lit:"tool_call_count"`
+ ToolNames string `lit:"tool_names"`
+ Flagged bool `lit:"flagged"`
+ FlaggedTerms string `lit:"flagged_terms"`
}
type groupedAiTraceRow struct {
@@ -136,72 +138,76 @@ func init() {
func aiTraceToRow(t models.AiTrace) aiTraceRow {
return aiTraceRow{
- Id: t.Id,
- ProjectId: t.ProjectId,
- RecordedAt: sqlitetypes.NewSQLiteTime(t.RecordedAt),
- Duration: int64(t.Duration),
- StatusCode: t.StatusCode,
- Model: t.Model,
- ResponseModel: t.ResponseModel,
- Provider: t.Provider,
- Operation: t.Operation,
- InputTokens: t.InputTokens,
- OutputTokens: t.OutputTokens,
- TotalTokens: t.TotalTokens,
- CachedTokens: t.CachedTokens,
- ReasoningTokens: t.ReasoningTokens,
- InputCost: t.InputCost,
- OutputCost: t.OutputCost,
- TotalCost: t.TotalCost,
- TraceName: t.TraceName,
- UserId: t.UserId,
- FinishReason: t.FinishReason,
- ServerName: t.ServerName,
- AppVersion: t.AppVersion,
- StorageKey: t.StorageKey,
- Attributes: sqlitetypes.NewSQLiteJSONMap(t.Attributes),
- DistributedTraceId: t.DistributedTraceId,
- IsRoot: t.IsRoot,
- ConversationId: t.ConversationId,
- ToolCallCount: t.ToolCallCount,
- ToolNames: shared.JoinCSV(t.ToolNames),
- Flagged: t.Flagged,
- FlaggedTerms: shared.JoinCSV(t.FlaggedTerms),
+ Id: t.Id,
+ ProjectId: t.ProjectId,
+ RecordedAt: sqlitetypes.NewSQLiteTime(t.RecordedAt),
+ Duration: int64(t.Duration),
+ StatusCode: t.StatusCode,
+ Model: t.Model,
+ ResponseModel: t.ResponseModel,
+ Provider: t.Provider,
+ Operation: t.Operation,
+ InputTokens: t.InputTokens,
+ OutputTokens: t.OutputTokens,
+ TotalTokens: t.TotalTokens,
+ CachedTokens: t.CachedTokens,
+ ReasoningTokens: t.ReasoningTokens,
+ InputCost: t.InputCost,
+ OutputCost: t.OutputCost,
+ TotalCost: t.TotalCost,
+ TraceName: t.TraceName,
+ UserId: t.UserId,
+ FinishReason: t.FinishReason,
+ ServerName: t.ServerName,
+ AppVersion: t.AppVersion,
+ StorageKey: t.StorageKey,
+ Attributes: sqlitetypes.NewSQLiteJSONMap(t.Attributes),
+ TraceId: t.TraceId,
+ SpanId: t.SpanId,
+ ParentSpanId: t.ParentSpanId,
+ IsRoot: t.IsRoot,
+ ConversationId: t.ConversationId,
+ ToolCallCount: t.ToolCallCount,
+ ToolNames: shared.JoinCSV(t.ToolNames),
+ Flagged: t.Flagged,
+ FlaggedTerms: shared.JoinCSV(t.FlaggedTerms),
}
}
func (r *aiTraceRow) toModel() models.AiTrace {
t := models.AiTrace{
- Id: r.Id,
- ProjectId: r.ProjectId,
- RecordedAt: r.RecordedAt.Time,
- Duration: time.Duration(r.Duration),
- StatusCode: r.StatusCode,
- Model: r.Model,
- ResponseModel: r.ResponseModel,
- Provider: r.Provider,
- Operation: r.Operation,
- InputTokens: r.InputTokens,
- OutputTokens: r.OutputTokens,
- TotalTokens: r.TotalTokens,
- CachedTokens: r.CachedTokens,
- ReasoningTokens: r.ReasoningTokens,
- InputCost: r.InputCost,
- OutputCost: r.OutputCost,
- TotalCost: r.TotalCost,
- TraceName: r.TraceName,
- UserId: r.UserId,
- FinishReason: r.FinishReason,
- ServerName: r.ServerName,
- AppVersion: r.AppVersion,
- StorageKey: r.StorageKey,
- DistributedTraceId: r.DistributedTraceId,
- IsRoot: r.IsRoot,
- ConversationId: r.ConversationId,
- ToolCallCount: r.ToolCallCount,
- ToolNames: shared.SplitCSV(r.ToolNames),
- Flagged: r.Flagged,
- FlaggedTerms: shared.SplitCSV(r.FlaggedTerms),
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ RecordedAt: r.RecordedAt.Time,
+ Duration: time.Duration(r.Duration),
+ StatusCode: r.StatusCode,
+ Model: r.Model,
+ ResponseModel: r.ResponseModel,
+ Provider: r.Provider,
+ Operation: r.Operation,
+ InputTokens: r.InputTokens,
+ OutputTokens: r.OutputTokens,
+ TotalTokens: r.TotalTokens,
+ CachedTokens: r.CachedTokens,
+ ReasoningTokens: r.ReasoningTokens,
+ InputCost: r.InputCost,
+ OutputCost: r.OutputCost,
+ TotalCost: r.TotalCost,
+ TraceName: r.TraceName,
+ UserId: r.UserId,
+ FinishReason: r.FinishReason,
+ ServerName: r.ServerName,
+ AppVersion: r.AppVersion,
+ StorageKey: r.StorageKey,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ ParentSpanId: r.ParentSpanId,
+ IsRoot: r.IsRoot,
+ ConversationId: r.ConversationId,
+ ToolCallCount: r.ToolCallCount,
+ ToolNames: shared.SplitCSV(r.ToolNames),
+ Flagged: r.Flagged,
+ FlaggedTerms: shared.SplitCSV(r.FlaggedTerms),
}
if r.Attributes != nil {
t.Attributes = map[string]string(r.Attributes)
@@ -247,7 +253,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
whereClause := "project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to" + aiTraceFilterClause(params, search, rootFilter)
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(DISTINCT trace_name) AS count FROM ai_traces WHERE "+whereClause, params)
+ "SELECT COUNT(DISTINCT trace_name) AS count FROM ai_traces_v2 WHERE "+whereClause, params)
if err != nil {
return nil, 0, err
}
@@ -266,7 +272,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
MAX(recorded_at) AS last_seen,
MAX(is_root) AS has_root,
MAX(CASE WHEN is_root = 0 THEN 1 ELSE 0 END) AS has_non_root
- FROM ai_traces WHERE `+whereClause+`
+ FROM ai_traces_v2 WHERE `+whereClause+`
GROUP BY trace_name`, params)
if err != nil {
return nil, 0, err
@@ -278,7 +284,7 @@ func (r *aiTraceRepository) FindGroupedByTraceName(ctx context.Context, projectI
durationParams := lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate), "trace_name": row.TraceName}
durationFilter := aiTraceFilterClause(durationParams, search, rootFilter)
durationRows, err := lit.SelectNamed[aiTraceDurationRow](db.TelemetryDB,
- `SELECT CAST(duration AS REAL) AS duration FROM ai_traces
+ `SELECT CAST(duration AS REAL) AS duration FROM ai_traces_v2
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to`+durationFilter+`
ORDER BY duration ASC`, durationParams)
if err != nil {
@@ -348,7 +354,7 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.
params := lit.P{"project_id": projectId, "trace_name": traceName, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM ai_traces WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM ai_traces_v2 WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -378,9 +384,9 @@ func (r *aiTraceRepository) FindByTraceName(ctx context.Context, projectId uuid.
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
lit.P{"project_id": projectId, "trace_name": traceName, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate), "limit": pageSize, "offset": offset})
@@ -411,7 +417,7 @@ func (r *aiTraceRepository) GetTraceNameStats(ctx context.Context, projectId uui
COALESCE(SUM(total_cost), 0) AS total_cost,
COALESCE(AVG(input_tokens), 0) AS avg_input_tokens,
COALESCE(AVG(output_tokens), 0) AS avg_output_tokens
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
@@ -431,7 +437,7 @@ func (r *aiTraceRepository) GetTraceNameStats(ctx context.Context, projectId uui
// Compute median and p95 from raw durations
durationRows, err := lit.SelectNamed[aiTraceDurationRow](db.TelemetryDB,
- `SELECT CAST(duration AS REAL) / 1000000.0 AS duration FROM ai_traces
+ `SELECT CAST(duration AS REAL) / 1000000.0 AS duration FROM ai_traces_v2
WHERE project_id = :project_id AND trace_name = :trace_name AND recorded_at >= :from AND recorded_at <= :to
ORDER BY duration ASC`, params)
if err != nil {
@@ -454,9 +460,9 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": traceId}
if recordedAt != nil {
@@ -478,11 +484,12 @@ func (r *aiTraceRepository) FindById(ctx context.Context, projectId, traceId uui
return &result, nil
}
-func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.AiTrace, error) {
- if len(projectIds) == 0 {
+func (r *aiTraceRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.AiTrace, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
@@ -494,9 +501,9 @@ func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distri
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ FROM ai_traces_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -524,7 +531,7 @@ func (r *aiTraceRepository) FindByDistributedTraceId(ctx context.Context, distri
&row.InputTokens, &row.OutputTokens, &row.TotalTokens, &row.CachedTokens, &row.ReasoningTokens,
&row.InputCost, &row.OutputCost, &row.TotalCost,
&row.TraceName, &row.UserId, &row.FinishReason, &row.ServerName, &row.AppVersion,
- &row.StorageKey, &row.Attributes, &row.DistributedTraceId, &row.IsRoot,
+ &row.StorageKey, &row.Attributes, &row.TraceId, &row.SpanId, &row.ParentSpanId, &row.IsRoot,
&row.ConversationId, &row.ToolCallCount, &row.ToolNames, &row.Flagged, &row.FlaggedTerms,
); err != nil {
return nil, err
@@ -564,7 +571,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
whereClause := baseWhere
if len(rowPredicates) > 0 {
- whereClause += " AND conversation_id IN (SELECT DISTINCT conversation_id FROM ai_traces WHERE " +
+ whereClause += " AND conversation_id IN (SELECT DISTINCT conversation_id FROM ai_traces_v2 WHERE " +
baseWhere + " AND " + strings.Join(rowPredicates, " AND ") + ")"
}
@@ -586,7 +593,7 @@ func (r *aiTraceRepository) FindConversations(ctx context.Context, projectId uui
GROUP_CONCAT(DISTINCT flagged_terms) AS flagged_terms,
MIN(recorded_at) AS first_seen,
MAX(recorded_at) AS last_seen
- FROM ai_traces WHERE `+whereClause+`
+ FROM ai_traces_v2 WHERE `+whereClause+`
GROUP BY conversation_id`+havingClause, params)
if err != nil {
return nil, 0, nil, err
@@ -624,9 +631,9 @@ func (r *aiTraceRepository) FindByConversationId(ctx context.Context, projectId
input_tokens, output_tokens, total_tokens, cached_tokens, reasoning_tokens,
input_cost, output_cost, total_cost,
trace_name, user_id, finish_reason, server_name, app_version,
- storage_key, attributes, distributed_trace_id, is_root,
+ storage_key, attributes, trace_id, span_id, parent_span_id, is_root,
conversation_id, tool_call_count, tool_names, flagged, flagged_terms
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND conversation_id = :conversation_id`
params := lit.P{"project_id": projectId, "conversation_id": conversationId}
if !fromDate.IsZero() {
@@ -667,7 +674,7 @@ func (r *aiTraceRepository) FindUserStats(ctx context.Context, projectId uuid.UU
SUM(total_tokens) AS conv_tokens,
MAX(flagged) AS conv_flagged,
MAX(recorded_at) AS last_seen
- FROM ai_traces WHERE `+whereClause+`
+ FROM ai_traces_v2 WHERE `+whereClause+`
GROUP BY user_id, conversation_id`, params)
if err != nil {
return nil, 0, err
@@ -706,7 +713,7 @@ func (r *aiTraceRepository) GetConversationCosts(ctx context.Context, projectId
}
rows, err := lit.SelectNamed[conversationCostRow](db.TelemetryDB,
`SELECT conversation_id, SUM(total_cost) AS total_cost
- FROM ai_traces
+ FROM ai_traces_v2
WHERE project_id = :project_id AND recorded_at >= :since AND conversation_id IN (`+strings.Join(placeholders, ",")+`)
GROUP BY conversation_id`, params)
if err != nil {
@@ -720,7 +727,7 @@ func (r *aiTraceRepository) GetConversationCosts(ctx context.Context, projectId
func (r *aiTraceRepository) ListModels(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time) ([]string, error) {
rows, err := lit.SelectNamed[modelNameRow](db.TelemetryDB,
- `SELECT DISTINCT model FROM ai_traces
+ `SELECT DISTINCT model FROM ai_traces_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND model != ''
ORDER BY model LIMIT 200`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)})
@@ -736,7 +743,7 @@ func (r *aiTraceRepository) ListModels(ctx context.Context, projectId uuid.UUID,
func (r *aiTraceRepository) ListToolNames(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time) ([]string, error) {
rows, err := lit.SelectNamed[toolNamesRow](db.TelemetryDB,
- `SELECT DISTINCT tool_names FROM ai_traces
+ `SELECT DISTINCT tool_names FROM ai_traces_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND tool_names != ''
LIMIT 500`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)})
diff --git a/backend/app/repositories/telemetry/sqlite/endpoint.repository.go b/backend/app/repositories/telemetry/sqlite/endpoint.repository.go
index 45c4d47b..33e2445b 100644
--- a/backend/app/repositories/telemetry/sqlite/endpoint.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/endpoint.repository.go
@@ -20,21 +20,22 @@ import (
)
type endpoint struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- Endpoint string `lit:"endpoint"`
- Duration int64 `lit:"duration"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- StatusCode int16 `lit:"status_code"`
- BodySize int32 `lit:"body_size"`
- ClientIP string `lit:"client_ip"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- SpanId *uuid.UUID `lit:"span_id"`
- IsStream bool `lit:"is_stream"`
- IsRoot bool `lit:"is_root"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ Endpoint string `lit:"endpoint"`
+ Duration int64 `lit:"duration"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ StatusCode int16 `lit:"status_code"`
+ BodySize int32 `lit:"body_size"`
+ ClientIP string `lit:"client_ip"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ ParentSpanId string `lit:"parent_span_id"`
+ IsStream bool `lit:"is_stream"`
+ IsRoot bool `lit:"is_root"`
}
type groupedEndpointRow struct {
@@ -82,9 +83,15 @@ type isStreamFlagRow struct {
IsStream bool `lit:"is_stream"`
}
+type endpointRowNaming struct{ lit.DefaultDbNamingStrategy }
+
+func (endpointRowNaming) GetTableNameFromStructName(string) string {
+ return "endpoints_v2"
+}
+
func init() {
models.ExtensionModelRegistrations = append(models.ExtensionModelRegistrations, func(driver lit.Driver) {
- lit.RegisterModel[endpoint](driver)
+ lit.RegisterModelWithNaming[endpoint](driver, endpointRowNaming{})
lit.RegisterModel[groupedEndpointRow](driver)
lit.RegisterModel[endpointDurationRow](driver)
lit.RegisterModel[slowEndpointRow](driver)
@@ -97,40 +104,42 @@ func init() {
func endpointToRow(e models.Endpoint) endpoint {
return endpoint{
- Id: e.Id,
- ProjectId: e.ProjectId,
- Endpoint: e.Endpoint,
- Duration: int64(e.Duration),
- RecordedAt: sqlitetypes.NewSQLiteTime(e.RecordedAt),
- StatusCode: e.StatusCode,
- BodySize: e.BodySize,
- ClientIP: e.ClientIP,
- Attributes: sqlitetypes.NewSQLiteJSONMap(e.Attributes),
- AppVersion: e.AppVersion,
- ServerName: e.ServerName,
- DistributedTraceId: e.DistributedTraceId,
- SpanId: e.SpanId,
- IsStream: e.IsStream,
- IsRoot: e.IsRoot,
+ Id: e.Id,
+ ProjectId: e.ProjectId,
+ Endpoint: e.Endpoint,
+ Duration: int64(e.Duration),
+ RecordedAt: sqlitetypes.NewSQLiteTime(e.RecordedAt),
+ StatusCode: e.StatusCode,
+ BodySize: e.BodySize,
+ ClientIP: e.ClientIP,
+ Attributes: sqlitetypes.NewSQLiteJSONMap(e.Attributes),
+ AppVersion: e.AppVersion,
+ ServerName: e.ServerName,
+ TraceId: e.TraceId,
+ SpanId: e.SpanId,
+ ParentSpanId: e.ParentSpanId,
+ IsStream: e.IsStream,
+ IsRoot: e.IsRoot,
}
}
func (r *endpoint) toModel() models.Endpoint {
e := models.Endpoint{
- Id: r.Id,
- ProjectId: r.ProjectId,
- Endpoint: r.Endpoint,
- Duration: time.Duration(r.Duration),
- RecordedAt: r.RecordedAt.Time,
- StatusCode: r.StatusCode,
- BodySize: r.BodySize,
- ClientIP: r.ClientIP,
- AppVersion: r.AppVersion,
- ServerName: r.ServerName,
- DistributedTraceId: r.DistributedTraceId,
- SpanId: r.SpanId,
- IsStream: r.IsStream,
- IsRoot: r.IsRoot,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ Endpoint: r.Endpoint,
+ Duration: time.Duration(r.Duration),
+ RecordedAt: r.RecordedAt.Time,
+ StatusCode: r.StatusCode,
+ BodySize: r.BodySize,
+ ClientIP: r.ClientIP,
+ AppVersion: r.AppVersion,
+ ServerName: r.ServerName,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ ParentSpanId: r.ParentSpanId,
+ IsStream: r.IsStream,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
e.Attributes = map[string]string(r.Attributes)
@@ -163,7 +172,7 @@ func (e *endpointRepository) InsertAsync(ctx context.Context, lines []models.End
func (e *endpointRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
result, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
return 0, err
@@ -178,7 +187,7 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
params := lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -196,8 +205,8 @@ func (e *endpointRepository) FindAll(ctx context.Context, projectId uuid.UUID, f
}
rows, err := lit.SelectNamed[endpoint](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s DESC LIMIT :limit OFFSET :offset`, orderBy),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate), "limit": pageSize, "offset": offset})
if err != nil {
@@ -232,7 +241,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
whereClause := "e.project_id = :project_id AND e.recorded_at >= :from AND e.recorded_at <= :to"
whereClause += endpointFilterClause(params, "e.", search, rootFilter, methodFilter)
- countQuery := "SELECT COUNT(DISTINCT e.endpoint) AS count FROM endpoints e WHERE " + whereClause
+ countQuery := "SELECT COUNT(DISTINCT e.endpoint) AS count FROM endpoints_v2 e WHERE " + whereClause
totalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB, countQuery, params)
if err != nil {
return nil, 0, err
@@ -256,7 +265,7 @@ func (e *endpointRepository) FindGroupedByEndpoint(ctx context.Context, projectI
MAX(e.is_stream) as is_stream,
MAX(e.is_root) as has_root,
MAX(CASE WHEN e.is_root = 0 THEN 1 ELSE 0 END) as has_non_root
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE ` + whereClause + `
GROUP BY e.endpoint`
@@ -351,7 +360,7 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
params := lit.P{"project_id": projectId, "endpoint": endpointName, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -374,8 +383,8 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
}
rows, err := lit.SelectNamed[endpoint](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
lit.P{"project_id": projectId, "endpoint": endpointName, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate), "limit": pageSize, "offset": offset})
if err != nil {
@@ -391,8 +400,8 @@ func (e *endpointRepository) FindByEndpoint(ctx context.Context, projectId uuid.
}
func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId uuid.UUID, recordedAt *time.Time) (*models.Endpoint, error) {
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id
- FROM endpoints WHERE project_id = :project_id AND id = :id`
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root, is_stream
+ FROM endpoints_v2 WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": endpointId}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -416,7 +425,7 @@ func (e *endpointRepository) FindById(ctx context.Context, projectId, endpointId
func (e *endpointRepository) CountByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime('%Y-%m-%d %H:00:00', recorded_at) as bucket, CAST(COUNT(*) AS REAL) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -428,7 +437,7 @@ func (e *endpointRepository) CountByHour(ctx context.Context, projectId uuid.UUI
func (e *endpointRepository) AvgDurationByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime('%Y-%m-%d %H:00:00', recorded_at) as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -441,7 +450,7 @@ func (e *endpointRepository) ErrorRateByHour(ctx context.Context, projectId uuid
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime('%Y-%m-%d %H:00:00', recorded_at) as bucket,
SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -454,7 +463,7 @@ func (e *endpointRepository) CountByInterval(ctx context.Context, projectId uuid
secs := intervalMinutes * 60
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket, CAST(COUNT(*) AS REAL) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, secs, secs),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -467,7 +476,7 @@ func (e *endpointRepository) AvgDurationByInterval(ctx context.Context, projectI
secs := intervalMinutes * 60
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0
GROUP BY bucket ORDER BY bucket ASC`, secs, secs),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -481,7 +490,7 @@ func (e *endpointRepository) ErrorRateByInterval(ctx context.Context, projectId
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket,
SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) as agg_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, secs, secs),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -505,7 +514,7 @@ func (e *endpointRepository) FindWorstEndpoints(ctx context.Context, projectId u
SUM(CASE WHEN e.duration > (750000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.duration <= (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) AND e.status_code < 500 THEN 1 ELSE 0 END) as tolerating_count,
SUM(CASE WHEN e.duration > (1500000000 + COALESCE(s.offset_ms, 0) * 1000000) OR e.status_code >= 500 THEN 1 ELSE 0 END) as bad_count,
MAX(e.is_stream) as is_stream
- FROM endpoints e
+ FROM endpoints_v2 e
LEFT JOIN slow_endpoints s ON e.endpoint = s.endpoint AND e.project_id = s.project_id
WHERE e.project_id = :project_id AND e.recorded_at >= :from AND e.recorded_at <= :to AND e.is_stream = 0
GROUP BY e.endpoint`
@@ -587,7 +596,7 @@ func (e *endpointRepository) GetEndpointStats(ctx context.Context, projectId uui
CASE WHEN COUNT(*) > 0 THEN SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) ELSE 0 END as error_rate,
COALESCE(SUM(CASE WHEN duration <= 500000000 AND status_code < 500 THEN 1 ELSE 0 END) +
SUM(CASE WHEN duration > 500000000 AND duration <= 2000000000 AND status_code < 500 THEN 1 ELSE 0 END) * 0.5, 0) as satisfied_tolerating
- FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
+ FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
return nil, err
@@ -597,7 +606,7 @@ func (e *endpointRepository) GetEndpointStats(ctx context.Context, projectId uui
}
isStreamRow, err := lit.SelectSingleNamed[isStreamFlagRow](db.TelemetryDB,
- `SELECT COALESCE(MAX(is_stream), 0) as is_stream FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
+ `SELECT COALESCE(MAX(is_stream), 0) as is_stream FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to`,
params)
if err != nil {
return nil, err
@@ -673,7 +682,7 @@ func (e *endpointRepository) GetEndpointStackedChart(ctx context.Context, projec
datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket,
%s as endpoint_category,
%s as metric_value
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0%s
GROUP BY bucket, endpoint_category
ORDER BY bucket ASC, endpoint_category ASC`, secs, secs, caseExpr, metricExpr, filterClause)
@@ -738,7 +747,7 @@ func (e *endpointRepository) getStackedChartWithPercentiles(ctx context.Context,
datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket,
endpoint,
duration
- FROM endpoints
+ FROM endpoints_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0%s
ORDER BY bucket ASC, endpoint ASC, duration ASC`, bucketSecs, bucketSecs, filterClause)
@@ -843,7 +852,7 @@ func fetchSortedDurations(ctx context.Context, projectId uuid.UUID, endpoint str
filterClause := endpointFilterClause(params, "", search, rootFilter, methodFilter)
results, err := lit.SelectNamed[endpointDurationRow](db.TelemetryDB,
- "SELECT duration FROM endpoints WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0"+filterClause+" ORDER BY duration ASC",
+ "SELECT duration FROM endpoints_v2 WHERE project_id = :project_id AND endpoint = :endpoint AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0"+filterClause+" ORDER BY duration ASC",
params)
if err != nil {
return nil, err
@@ -888,7 +897,7 @@ func getTopEndpointsByMetric(ctx context.Context, projectId uuid.UUID, from, to
if metricType == "total_time" {
results, err := lit.SelectNamed[endpointMetricRow](db.TelemetryDB,
`SELECT endpoint, COUNT(*) * AVG(duration) / 1000000.0 as metric_value
- FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`+filterClause+`
+ FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`+filterClause+`
GROUP BY endpoint ORDER BY metric_value DESC LIMIT 5`,
params)
if err != nil {
@@ -911,7 +920,7 @@ func getTopEndpointsByMetric(ctx context.Context, projectId uuid.UUID, from, to
}
epRows, err := lit.SelectNamed[distinctEndpointRow](db.TelemetryDB,
- `SELECT DISTINCT endpoint FROM endpoints WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`+filterClause,
+ `SELECT DISTINCT endpoint FROM endpoints_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to AND is_stream = 0`+filterClause,
params)
if err != nil {
return nil, err
@@ -949,19 +958,20 @@ func getTopEndpointsByMetric(ctx context.Context, projectId uuid.UUID, from, to
return result, nil
}
-func (e *endpointRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Endpoint, error) {
- if len(projectIds) == 0 {
+func (e *endpointRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Endpoint, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
placeholders[i] = ":" + key
params[key] = pid
}
- query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM endpoints WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ query := `SELECT id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM endpoints_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -985,7 +995,7 @@ func (e *endpointRepository) FindByDistributedTraceId(ctx context.Context, distr
for sqlRows.Next() {
var row endpoint
if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.Endpoint, &row.Duration, &row.RecordedAt,
- &row.StatusCode, &row.BodySize, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.DistributedTraceId); err != nil {
+ &row.StatusCode, &row.BodySize, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.TraceId, &row.SpanId, &row.ParentSpanId, &row.IsRoot); err != nil {
return nil, err
}
endpoints = append(endpoints, row.toModel())
diff --git a/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go b/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go
index daa7ab71..d495e165 100644
--- a/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/exception_stack_trace.repository.go
@@ -17,25 +17,25 @@ import (
)
type exceptionRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- TraceId *uuid.UUID `lit:"trace_id"`
- TraceType string `lit:"trace_type"`
- ExceptionHash string `lit:"exception_hash"`
- StackTrace string `lit:"stack_trace"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- IsMessage bool `lit:"is_message"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- SessionId *uuid.UUID `lit:"session_id"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ TraceType string `lit:"trace_type"`
+ ExceptionHash string `lit:"exception_hash"`
+ StackTrace string `lit:"stack_trace"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ IsMessage bool `lit:"is_message"`
+ SessionId *uuid.UUID `lit:"session_id"`
}
type exceptionRowNaming struct{ lit.DefaultDbNamingStrategy }
func (exceptionRowNaming) GetTableNameFromStructName(string) string {
- return "exception_stack_traces"
+ return "exceptions_v2"
}
type exceptionGroupRow struct {
@@ -62,41 +62,37 @@ func init() {
}
func exceptionToRow(est models.ExceptionStackTrace) exceptionRow {
- traceType := est.TraceType
- if traceType == "" {
- traceType = "endpoint"
- }
return exceptionRow{
- Id: est.Id,
- ProjectId: est.ProjectId,
- TraceId: est.TraceId,
- TraceType: traceType,
- ExceptionHash: est.ExceptionHash,
- StackTrace: est.StackTrace,
- RecordedAt: sqlitetypes.NewSQLiteTime(est.RecordedAt),
- Attributes: sqlitetypes.NewSQLiteJSONMap(est.Attributes),
- AppVersion: est.AppVersion,
- ServerName: est.ServerName,
- IsMessage: est.IsMessage,
- DistributedTraceId: est.DistributedTraceId,
- SessionId: est.SessionId,
+ Id: est.Id,
+ ProjectId: est.ProjectId,
+ TraceId: est.TraceId,
+ SpanId: est.SpanId,
+ TraceType: est.TraceType,
+ ExceptionHash: est.ExceptionHash,
+ StackTrace: est.StackTrace,
+ RecordedAt: sqlitetypes.NewSQLiteTime(est.RecordedAt),
+ Attributes: sqlitetypes.NewSQLiteJSONMap(est.Attributes),
+ AppVersion: est.AppVersion,
+ ServerName: est.ServerName,
+ IsMessage: est.IsMessage,
+ SessionId: est.SessionId,
}
}
func (r *exceptionRow) toModel() models.ExceptionStackTrace {
est := models.ExceptionStackTrace{
- Id: r.Id,
- ProjectId: r.ProjectId,
- TraceId: r.TraceId,
- TraceType: r.TraceType,
- ExceptionHash: r.ExceptionHash,
- StackTrace: r.StackTrace,
- RecordedAt: r.RecordedAt.Time,
- AppVersion: r.AppVersion,
- ServerName: r.ServerName,
- IsMessage: r.IsMessage,
- DistributedTraceId: r.DistributedTraceId,
- SessionId: r.SessionId,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ TraceType: r.TraceType,
+ ExceptionHash: r.ExceptionHash,
+ StackTrace: r.StackTrace,
+ RecordedAt: r.RecordedAt.Time,
+ AppVersion: r.AppVersion,
+ ServerName: r.ServerName,
+ IsMessage: r.IsMessage,
+ SessionId: r.SessionId,
}
if r.Attributes != nil {
est.Attributes = map[string]string(r.Attributes)
@@ -129,7 +125,7 @@ func (e *exceptionStackTraceRepository) InsertAsync(ctx context.Context, lines [
func (e *exceptionStackTraceRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
result, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM exception_stack_traces WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM exceptions_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
return 0, err
@@ -181,7 +177,7 @@ func (e *exceptionStackTraceRepository) FindGrouped(ctx context.Context, project
countQuery := `SELECT COUNT(*) AS count FROM (
SELECT e.exception_hash, MAX(a.archived_at) as max_archived_at
- FROM exception_stack_traces e ` + archiveSubquery + `
+ FROM exceptions_v2 e ` + archiveSubquery + `
WHERE ` + whereClause + `
GROUP BY e.exception_hash` + havingClause + `)`
@@ -195,7 +191,7 @@ func (e *exceptionStackTraceRepository) FindGrouped(ctx context.Context, project
}
fullQuery := `SELECT e.exception_hash, e.stack_trace, MAX(e.recorded_at) as last_seen, MIN(e.recorded_at) as first_seen, COUNT(*) as count, MAX(a.archived_at) as max_archived_at
- FROM exception_stack_traces e ` + archiveSubquery + `
+ FROM exceptions_v2 e ` + archiveSubquery + `
WHERE ` + whereClause + `
GROUP BY e.exception_hash` + havingClause + `
ORDER BY ` + orderBy + ` ` + sortDirection + `, e.exception_hash LIMIT :limit OFFSET :offset`
@@ -229,7 +225,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
groupRow, err := lit.SelectSingleNamed[exceptionGroupRow](db.TelemetryDB,
`SELECT exception_hash, stack_trace, MAX(recorded_at) as last_seen, MIN(recorded_at) as first_seen, COUNT(*) as count, NULL as max_archived_at
- FROM exception_stack_traces WHERE project_id = :project_id AND exception_hash = :exception_hash
+ FROM exceptions_v2 WHERE project_id = :project_id AND exception_hash = :exception_hash
GROUP BY exception_hash`,
params)
if err != nil {
@@ -250,8 +246,8 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
}
rows, err := lit.SelectNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND exception_hash = :exception_hash
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE project_id = :project_id AND exception_hash = :exception_hash
ORDER BY recorded_at DESC LIMIT :limit OFFSET :offset`,
lit.P{"project_id": projectId, "exception_hash": exceptionHash, "limit": pageSize, "offset": offset})
if err != nil {
@@ -269,7 +265,7 @@ func (e *exceptionStackTraceRepository) FindByHash(ctx context.Context, projectI
func (e *exceptionStackTraceRepository) CountByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime('%Y-%m-%d %H:00:00', recorded_at) as bucket, CAST(COUNT(*) AS REAL) as agg_value
- FROM exception_stack_traces WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM exceptions_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -282,7 +278,7 @@ func (e *exceptionStackTraceRepository) CountByInterval(ctx context.Context, pro
intervalSeconds := intervalMinutes * 60
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket, CAST(COUNT(*) AS REAL) as agg_value
- FROM exception_stack_traces WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM exceptions_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, intervalSeconds, intervalSeconds),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -305,7 +301,7 @@ func (e *exceptionStackTraceRepository) GetHourlyTrendForHashes(ctx context.Cont
}
query := `SELECT exception_hash, strftime('%Y-%m-%d %H:00:00', recorded_at) as hour, COUNT(*) as count
- FROM exception_stack_traces
+ FROM exceptions_v2
WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
AND exception_hash IN (` + strings.Join(placeholders, ",") + `)
GROUP BY exception_hash, hour ORDER BY exception_hash, hour ASC`
@@ -384,24 +380,9 @@ func (e *exceptionStackTraceRepository) IsArchived(ctx context.Context, projectI
return result.Count > 0, nil
}
-func (e *exceptionStackTraceRepository) FindExceptionByTraceId(ctx context.Context, projectId uuid.UUID, traceId uuid.UUID) (*models.ExceptionStackTrace, error) {
- row, err := lit.SelectSingleNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND trace_id = :trace_id AND is_message = 0 LIMIT 1`,
- lit.P{"project_id": projectId, "trace_id": traceId})
- if err != nil {
- return nil, err
- }
- if row == nil {
- return nil, nil
- }
- est := row.toModel()
- return &est, nil
-}
-
-func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, projectId uuid.UUID, traceId uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND trace_id = :trace_id`
+func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, projectId uuid.UUID, traceId string, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE project_id = :project_id AND trace_id = :trace_id`
params := lit.P{"project_id": projectId, "trace_id": traceId}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -424,8 +405,8 @@ func (e *exceptionStackTraceRepository) FindAllByTraceId(ctx context.Context, pr
}
func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId uuid.UUID, id uuid.UUID, recordedAt *time.Time) (*models.ExceptionStackTrace, error) {
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE project_id = :project_id AND id = :id`
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": id}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -446,11 +427,12 @@ func (e *exceptionStackTraceRepository) FindById(ctx context.Context, projectId
return &est, nil
}
-func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
- if len(projectIds) == 0 {
+func (e *exceptionStackTraceRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.ExceptionStackTrace, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
@@ -458,8 +440,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
params[key] = pid
}
- query := `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `) AND is_message = 0`
+ query := `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `) AND is_message = 0`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -482,8 +464,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
var results []models.ExceptionStackTrace
for sqlRows.Next() {
var row exceptionRow
- if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TraceId, &row.TraceType, &row.ExceptionHash, &row.StackTrace,
- &row.RecordedAt, &row.Attributes, &row.AppVersion, &row.ServerName, &row.IsMessage, &row.DistributedTraceId, &row.SessionId); err != nil {
+ if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TraceId, &row.SpanId, &row.TraceType, &row.ExceptionHash, &row.StackTrace,
+ &row.RecordedAt, &row.Attributes, &row.AppVersion, &row.ServerName, &row.IsMessage, &row.SessionId); err != nil {
return nil, err
}
results = append(results, row.toModel())
@@ -496,8 +478,8 @@ func (e *exceptionStackTraceRepository) FindByDistributedTraceId(ctx context.Con
// (WHERE session_id IS NOT NULL); an IS, IN or COALESCE rewrite falls back to a whole-project scan.
func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context, projectId, sessionId uuid.UUID) ([]models.ExceptionStackTrace, error) {
rows, err := lit.SelectNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2
WHERE project_id = :project_id AND session_id = :session_id
ORDER BY recorded_at ASC`,
lit.P{"project_id": projectId, "session_id": sessionId})
@@ -514,8 +496,8 @@ func (e *exceptionStackTraceRepository) FindAllBySessionId(ctx context.Context,
// GetSessionIdForException returns the session_id for the given exception or nil when not linked.
func (e *exceptionStackTraceRepository) GetSessionIdForException(ctx context.Context, projectId, exceptionId uuid.UUID) (*uuid.UUID, error) {
row, err := lit.SelectSingleNamed[exceptionRow](db.TelemetryDB,
- `SELECT id, project_id, trace_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, distributed_trace_id, session_id
- FROM exception_stack_traces
+ `SELECT id, project_id, trace_id, span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes, app_version, server_name, is_message, session_id
+ FROM exceptions_v2
WHERE project_id = :project_id AND id = :id LIMIT 1`,
lit.P{"project_id": projectId, "id": exceptionId})
if err != nil || row == nil {
diff --git a/backend/app/repositories/telemetry/sqlite/legacy.repository.go b/backend/app/repositories/telemetry/sqlite/legacy.repository.go
new file mode 100644
index 00000000..0698919c
--- /dev/null
+++ b/backend/app/repositories/telemetry/sqlite/legacy.repository.go
@@ -0,0 +1,271 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package sqlite
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+// legacyRepository reads the tables V2 replaced, for the move-over. See shared/legacy.go for what the rows carry.
+type legacyRepository struct{}
+
+const legacyIdChunk = 500
+
+const (
+ legacyEndpointColumns = `id, project_id, endpoint, duration, recorded_at, status_code, body_size, client_ip, attributes, app_version, server_name,
+ COALESCE(distributed_trace_id, '') AS trace_id, COALESCE(span_id, '') AS span_id, '' AS parent_span_id, is_stream, is_root`
+ legacyTaskColumns = `id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name,
+ COALESCE(distributed_trace_id, '') AS trace_id, COALESCE(span_id, '') AS span_id, '' AS parent_span_id, is_root`
+ legacyAiTraceColumns = `id, project_id, recorded_at, duration, status_code, model, response_model, provider, operation, input_tokens, output_tokens,
+ total_tokens, cached_tokens, reasoning_tokens, input_cost, output_cost, total_cost, trace_name, user_id, finish_reason, server_name, app_version,
+ storage_key, attributes, COALESCE(distributed_trace_id, '') AS trace_id, '' AS span_id, '' AS parent_span_id, is_root,
+ conversation_id, tool_call_count, tool_names, flagged, flagged_terms`
+ legacyExceptionColumns = `id, project_id, COALESCE(trace_id, '') AS trace_id, '' AS span_id, trace_type, exception_hash, stack_trace, recorded_at, attributes,
+ app_version, server_name, is_message, COALESCE(distributed_trace_id, ''), session_id`
+)
+
+// legacyWindow sorts only overfull one-second slices; see shared.LegacyPageOrder.
+func legacyWindow(columns, table string, from, to time.Time, limit int, offset *int) (string, lit.P) {
+ if offset != nil {
+ query, params := legacyWindow(columns, table, from, to, limit, nil)
+ params["offset"] = *offset
+ return strings.Replace(query, " LIMIT :limit", " ORDER BY "+shared.LegacyPageOrder(table, false)+" LIMIT :limit OFFSET :offset", 1), params
+ }
+ return "SELECT " + columns + " FROM " + table + " WHERE recorded_at >= :from AND recorded_at < :to LIMIT :limit",
+ lit.P{"from": sqlitetypes.NewSQLiteTime(from), "to": sqlitetypes.NewSQLiteTime(to), "limit": limit}
+}
+
+func legacyByIds(columns, table string, ids []uuid.UUID, from, to time.Time) (string, lit.P) {
+ params := lit.P{"from": sqlitetypes.NewSQLiteTime(from), "to": sqlitetypes.NewSQLiteTime(to)}
+ names := make([]string, len(ids))
+ for i, id := range ids {
+ key := fmt.Sprintf("id_%d", i)
+ names[i], params[key] = ":"+key, id
+ }
+ return "SELECT " + columns + " FROM " + table + " WHERE id IN (" + strings.Join(names, ",") + ") AND recorded_at >= :from AND recorded_at < :to", params
+}
+
+func legacyEndpoints(ctx context.Context, query string, params lit.P) ([]models.Endpoint, error) {
+ rows, err := shared.SelectLegacy[endpoint](ctx, db.TelemetryDB, query, params)
+ if err != nil {
+ return nil, err
+ }
+ found := make([]models.Endpoint, len(rows))
+ for i, row := range rows {
+ found[i] = row.toModel()
+ }
+ return found, nil
+}
+
+func legacyTasks(ctx context.Context, query string, params lit.P) ([]models.Task, error) {
+ rows, err := shared.SelectLegacy[task](ctx, db.TelemetryDB, query, params)
+ if err != nil {
+ return nil, err
+ }
+ found := make([]models.Task, len(rows))
+ for i, row := range rows {
+ found[i] = row.toModel()
+ }
+ return found, nil
+}
+
+func legacyAiTraces(ctx context.Context, query string, params lit.P) ([]models.AiTrace, error) {
+ rows, err := shared.SelectLegacy[aiTraceRow](ctx, db.TelemetryDB, query, params)
+ if err != nil {
+ return nil, err
+ }
+ found := make([]models.AiTrace, len(rows))
+ for i, row := range rows {
+ found[i] = row.toModel()
+ }
+ return found, nil
+}
+
+func (legacyRepository) FindEndpoints(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Endpoint, error) {
+ query, params := legacyWindow(legacyEndpointColumns, "endpoints", from, to, limit, offset)
+ return legacyEndpoints(ctx, query, params)
+}
+
+func (legacyRepository) FindTasks(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Task, error) {
+ query, params := legacyWindow(legacyTaskColumns, "tasks", from, to, limit, offset)
+ return legacyTasks(ctx, query, params)
+}
+
+func (legacyRepository) FindAiTraces(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.AiTrace, error) {
+ query, params := legacyWindow(legacyAiTraceColumns, "ai_traces", from, to, limit, offset)
+ return legacyAiTraces(ctx, query, params)
+}
+
+func (legacyRepository) FindExceptions(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacyException, error) {
+ query, params := legacyWindow(legacyExceptionColumns, "exception_stack_traces", from, to, limit, offset)
+ query, args, err := lit.ParseNamedQuery(db.Driver, query, params)
+ if err != nil {
+ return nil, err
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var found []shared.LegacyException
+ for rows.Next() {
+ var row exceptionRow
+ var traceId string
+ if err := rows.Scan(&row.Id, &row.ProjectId, &row.TraceId, &row.SpanId, &row.TraceType, &row.ExceptionHash, &row.StackTrace, &row.RecordedAt, &row.Attributes, &row.AppVersion, &row.ServerName, &row.IsMessage, &traceId, &row.SessionId); err != nil {
+ return nil, err
+ }
+ found = append(found, shared.LegacyException{ExceptionStackTrace: row.toModel(), DistributedTraceId: traceId})
+ }
+ return found, rows.Err()
+}
+
+func (legacyRepository) FindSpans(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacySpan, error) {
+ query, params := legacyWindow("project_id, id, trace_id, COALESCE(parent_span_id, ''), name, start_time, duration, recorded_at, attributes", "spans", from, to, limit, offset)
+ query, args, err := lit.ParseNamedQuery(db.Driver, query, params)
+ if err != nil {
+ return nil, err
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var found []shared.LegacySpan
+ for rows.Next() {
+ var span shared.LegacySpan
+ var start, recorded sqlitetypes.SQLiteTime
+ var attributes sqlitetypes.SQLiteJSONMap
+ var duration int64
+ if err := rows.Scan(&span.ProjectId, &span.Id, &span.OwnerId, &span.ParentSpanId, &span.Name, &start, &duration, &recorded, &attributes); err != nil {
+ return nil, err
+ }
+ span.StartTime, span.RecordedAt, span.Duration, span.Attributes = start.Time, recorded.Time, time.Duration(duration), attributes
+ found = append(found, span)
+ }
+ return found, rows.Err()
+}
+
+// FindOwners looks the owners of spans and exceptions up by id, in all three tables an owner can live in.
+func (legacyRepository) FindOwners(ctx context.Context, ids []uuid.UUID, from, to time.Time) ([]models.Endpoint, []models.Task, []models.AiTrace, error) {
+ var endpoints []models.Endpoint
+ var tasks []models.Task
+ var aiTraces []models.AiTrace
+ for chunk := range slices.Chunk(ids, legacyIdChunk) {
+ query, params := legacyByIds(legacyEndpointColumns, "endpoints", chunk, from, to)
+ foundEndpoints, err := legacyEndpoints(ctx, query, params)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ query, params = legacyByIds(legacyTaskColumns, "tasks", chunk, from, to)
+ foundTasks, err := legacyTasks(ctx, query, params)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ query, params = legacyByIds(legacyAiTraceColumns, "ai_traces", chunk, from, to)
+ foundAiTraces, err := legacyAiTraces(ctx, query, params)
+ if err != nil {
+ return nil, nil, nil, err
+ }
+ endpoints, tasks, aiTraces = append(endpoints, foundEndpoints...), append(tasks, foundTasks...), append(aiTraces, foundAiTraces...)
+ }
+ return endpoints, tasks, aiTraces, nil
+}
+
+func (legacyRepository) FindBounds(ctx context.Context, table string) (oldest, newest time.Time, found bool, err error) {
+ if table, err = shared.LegacyTable(table); err != nil {
+ return oldest, newest, false, err
+ }
+ var low, high sqlitetypes.SQLiteTime
+ var count int64
+ if err = db.TelemetryDB.QueryRowContext(ctx, "SELECT COUNT(*), MIN(recorded_at), MAX(recorded_at) FROM "+table).Scan(&count, &low, &high); err != nil {
+ return oldest, newest, false, err
+ }
+ return low.Time, high.Time, count > 0, nil
+}
+
+// FindMovedIds returns full occurrence keys, never IDs shared by unrelated traces.
+func (legacyRepository) FindMovedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time) (map[shared.MovedOccurrence]bool, error) {
+ table, err := shared.LegacyTable(table)
+ if err != nil {
+ return nil, err
+ }
+ moved := map[shared.MovedOccurrence]bool{}
+ for chunk := range slices.Chunk(ids, legacyIdChunk) {
+ args := []any{sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to)}
+ for _, id := range chunk {
+ args = append(args, id)
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, "SELECT project_id, id, trace_id, span_id, recorded_at FROM "+table+" WHERE recorded_at >= ? AND recorded_at < ? AND id IN (?"+strings.Repeat(",?", len(chunk)-1)+")", args...)
+ if err != nil {
+ return nil, err
+ }
+ if err := scanMovedIds(rows, moved); err != nil {
+ return nil, err
+ }
+ }
+ return moved, nil
+}
+
+func scanMovedIds(rows *sql.Rows, moved map[shared.MovedOccurrence]bool) error {
+ defer rows.Close()
+ for rows.Next() {
+ var project, id uuid.UUID
+ var trace, span string
+ var at sqlitetypes.SQLiteTime
+ if err := rows.Scan(&project, &id, &trace, &span, &at); err != nil {
+ return err
+ }
+ moved[shared.MovedOccurrenceKey(project, id, trace, span, at.Time)] = true
+ }
+ return rows.Err()
+}
+
+func (legacyRepository) FindMovedSpans(ctx context.Context, traceIds []string, from, to time.Time) (map[string]bool, error) {
+ moved := map[string]bool{}
+ for chunk := range slices.Chunk(traceIds, legacyIdChunk) {
+ args := []any{sqlitetypes.NewSQLiteTime(from), sqlitetypes.NewSQLiteTime(to)}
+ for _, id := range chunk {
+ args = append(args, id)
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, "SELECT project_id, trace_id, span_id FROM "+shared.SpansTable+" WHERE recorded_at >= ? AND recorded_at < ? AND trace_id IN (?"+strings.Repeat(",?", len(chunk)-1)+")", args...)
+ if err != nil {
+ return nil, err
+ }
+ for rows.Next() {
+ var project uuid.UUID
+ var trace, span string
+ if err := rows.Scan(&project, &trace, &span); err != nil {
+ rows.Close()
+ return nil, err
+ }
+ moved[shared.MovedSpanKey(project, trace, span)] = true
+ }
+ if err := rows.Err(); err != nil {
+ rows.Close()
+ return nil, err
+ }
+ if err := rows.Close(); err != nil {
+ return nil, err
+ }
+ }
+ return moved, nil
+}
+
+// OccurrenceTime matches the destination column precision for recovery keys.
+func (legacyRepository) OccurrenceTime(table string, at time.Time) time.Time {
+ return at
+}
+
+var LegacyRepository = legacyRepository{}
diff --git a/backend/app/repositories/telemetry/sqlite/log_record.repository.go b/backend/app/repositories/telemetry/sqlite/log_record.repository.go
index ee752b49..aaf87a98 100644
--- a/backend/app/repositories/telemetry/sqlite/log_record.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/log_record.repository.go
@@ -225,6 +225,15 @@ func (r *logRecordRepository) buildWhere(params shared.LogSearchParams) (string,
"from": sqlitetypes.NewSQLiteTime(params.FromDate),
"to": sqlitetypes.NewSQLiteTime(params.ToDate),
}
+ if len(params.ProjectIds) > 0 {
+ placeholders := make([]string, len(params.ProjectIds))
+ for i, id := range params.ProjectIds {
+ key := fmt.Sprintf("project%d", i)
+ placeholders[i] = ":" + key
+ args[key] = id
+ }
+ clauses[0] = "project_id IN (" + strings.Join(placeholders, ", ") + ")"
+ }
if params.MinSeverity > 0 {
clauses = append(clauses, "severity_number >= :min_severity")
diff --git a/backend/app/repositories/telemetry/sqlite/otel_span.repository.go b/backend/app/repositories/telemetry/sqlite/otel_span.repository.go
new file mode 100644
index 00000000..b88bd2af
--- /dev/null
+++ b/backend/app/repositories/telemetry/sqlite/otel_span.repository.go
@@ -0,0 +1,204 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package sqlite
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "strconv"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
+)
+
+type otelSpanRepository struct{}
+
+var sqliteOtelCodec = shared.OtelValueCodec{
+ UUID: func(id uuid.UUID) any { return id.String() },
+ Time: func(t time.Time) any { return sqlitetypes.NewSQLiteTime(t) },
+ Nanos: func(n uint64) any { return strconv.FormatUint(n, 10) },
+ Uint32: func(n uint32) any { return int64(n) },
+}
+
+func sqliteOtelValues(span models.OtelSpan, groups shared.OtelSpanGroups) ([]any, error) {
+ row, err := shared.NewOtelSpanRow(span, groups)
+ if err != nil {
+ return nil, err
+ }
+ values := row.ScalarValues(sqliteOtelCodec)
+ text, err := row.TextValues()
+ if err != nil {
+ return nil, err
+ }
+ return append(values, text...), nil
+}
+
+func (r *otelSpanRepository) InsertAsync(ctx context.Context, spans []models.OtelSpan) (int, error) {
+ rejected, err := r.InsertWithRejections(ctx, spans)
+ return len(rejected), err
+}
+
+func (r *otelSpanRepository) InsertWithRejections(ctx context.Context, spans []models.OtelSpan) ([]shared.SpanOwner, error) {
+ if len(spans) == 0 {
+ return nil, nil
+ }
+ tx, err := db.TelemetryDB.BeginTx(ctx, nil)
+ if err != nil {
+ return nil, err
+ }
+ defer tx.Rollback()
+ stmt, err := tx.PrepareContext(ctx, shared.OtelTextInsertSQL)
+ if err != nil {
+ return nil, err
+ }
+ defer stmt.Close()
+ groups := shared.OtelSpanGroups{}
+ var rejected []shared.SpanOwner
+ for _, span := range spans {
+ values, err := sqliteOtelValues(span, groups)
+ if err != nil {
+ shared.RecordRejectedOtelSpan(err)
+ rejected = append(rejected, shared.SpanOwner{ProjectId: span.ProjectId, TraceId: span.TraceId, SpanId: span.SpanId})
+ continue
+ }
+ if _, err := stmt.ExecContext(ctx, values...); err != nil {
+ return nil, err
+ }
+ }
+ if err := tx.Commit(); err != nil {
+ return nil, err
+ }
+ return rejected, nil
+}
+
+var OtelSpanRepository = &otelSpanRepository{}
+
+func scanSqliteTopology(rows *sql.Rows, withStatement bool) ([]models.OtelSpan, error) {
+ result := make([]models.OtelSpan, 0)
+ for rows.Next() {
+ var span models.OtelSpan
+ var nanos string
+ var recordedAt sqlitetypes.SQLiteTime
+ columns := []any{&span.ProjectId, &span.TraceId, &span.SpanId, &span.ParentSpanId,
+ &span.Name, &span.Duration, &span.SpanKind, &span.StatusCode, &span.ServiceName, &span.ScopeName, &nanos, &recordedAt}
+ if withStatement {
+ columns = append(columns, &span.DbStatement)
+ }
+ if err := rows.Scan(columns...); err != nil {
+ return nil, err
+ }
+ nanoValue, err := strconv.ParseUint(nanos, 10, 64)
+ if err != nil {
+ return nil, err
+ }
+ span.StartTime = shared.OtelNanosToTime(nanoValue)
+ span.RecordedAt = recordedAt.Time
+ result = append(result, span)
+ }
+ return result, rows.Err()
+}
+
+func (r *otelSpanRepository) FindTraceTopology(ctx context.Context, lookups []shared.SpanLookup, fromUnixNano uint64) ([]models.OtelSpan, error) {
+ if len(lookups) == 0 {
+ return nil, nil
+ }
+ query, args := shared.OtelTopologyQuery(lookups, false, fromUnixNano, sqliteOtelCodec.Time)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return scanSqliteTopology(rows, false)
+}
+
+const sqliteStatementPreview = `substr(coalesce(json_extract(span_attributes, '$."db.query.text"'), json_extract(span_attributes, '$."db.statement"'), ''), 1, 240)`
+
+func (r *otelSpanRepository) FindTraceOutline(ctx context.Context, lookups []shared.SpanLookup) ([]models.OtelSpan, error) {
+ query, args := shared.OtelOutlineQuery(lookups, false, sqliteStatementPreview, sqliteOtelCodec.Time)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return scanSqliteTopology(rows, true)
+}
+
+func (r *otelSpanRepository) FindSpanAttributes(ctx context.Context, lookup shared.SpanLookup, spanIds []string, limits shared.OtelAttributeLimits) (map[string]shared.OtelSpanAttributes, error) {
+ args := []any{lookup.ProjectId.String(), lookup.TraceId}
+ for _, id := range spanIds {
+ args = append(args, id)
+ }
+ predicate := "project_id = ? AND trace_id = ? AND span_id IN (" + shared.OtelPlaceholders(len(spanIds)) + ")"
+ predicate, args = shared.OtelWindowPredicate(predicate, args, []shared.SpanLookup{lookup}, sqliteOtelCodec.Time)
+ rows, err := db.TelemetryDB.QueryContext(ctx, shared.OtelAttributeQuery("span_attributes", "length(CAST(span_attributes AS BLOB))", "CAST(start_time_unix_nano AS INTEGER)", predicate, shared.OtelTextWinnerOrder, limits), args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ return shared.ScanOtelAttributeRows(rows)
+}
+
+func (r *otelSpanRepository) IsReadLimitError(error) bool { return false }
+
+var sqliteSearchDialect = shared.OtelSearchDialect{
+ WinnerOrder: shared.OtelTextWinnerOrder,
+ Project: func(id uuid.UUID) any { return id.String() },
+ Time: sqliteOtelCodec.Time,
+ Trace: func(_ uuid.UUID, traceId string) (string, any) { return "trace_id = ?", traceId },
+ NameLike: "INSTR(LOWER(name), LOWER(?)) > 0",
+ Attribute: `json_extract(span_attributes, '$."' || ? || '"') = ?`,
+ StartOrder: "CAST(start_time_unix_nano AS INTEGER)",
+}
+
+func (r *otelSpanRepository) Search(ctx context.Context, search shared.OtelSpanSearch) ([]models.OtelSpan, uint64, error) {
+ page, pageArgs, count, countArgs := shared.OtelSearchQueries(search, sqliteSearchDialect)
+ var total uint64
+ if err := db.TelemetryDB.QueryRowContext(ctx, count, countArgs...).Scan(&total); err != nil {
+ return nil, 0, err
+ }
+ rows, err := db.TelemetryDB.QueryContext(ctx, page, pageArgs...)
+ if err != nil {
+ return nil, 0, err
+ }
+ defer rows.Close()
+ spans, err := scanSqliteTopology(rows, false)
+ return spans, total, err
+}
+
+func (r *otelSpanRepository) Services(ctx context.Context, project uuid.UUID, from, to time.Time) ([]string, error) {
+ query, args := shared.OtelServicesQuery(project, from, to, sqliteSearchDialect)
+ rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ services := make([]string, 0)
+ for rows.Next() {
+ var service string
+ if err := rows.Scan(&service); err != nil {
+ return nil, err
+ }
+ services = append(services, service)
+ }
+ return services, rows.Err()
+}
+
+func (r *otelSpanRepository) FindOTLP(ctx context.Context, project uuid.UUID, traceID, spanID string, at time.Time) ([]byte, error) {
+ ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+ defer cancel()
+ var value []byte
+ from, to := shared.TraceWindowBounds(at.UTC())
+ err := db.TelemetryDB.QueryRowContext(ctx, "SELECT otlp FROM "+shared.SpansTable+" WHERE project_id = ? AND trace_id = ? AND span_id = ? AND recorded_at >= ? AND recorded_at <= ? ORDER BY "+shared.OtelTextWinnerOrder+" LIMIT 1", project.String(), traceID, spanID, sqliteOtelCodec.Time(from), sqliteOtelCodec.Time(to)).Scan(&value)
+ if errors.Is(err, sql.ErrNoRows) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, err
+ }
+ return value, nil
+}
diff --git a/backend/app/repositories/telemetry/sqlite/profile.repository.go b/backend/app/repositories/telemetry/sqlite/profile.repository.go
index d72d3142..065f8849 100644
--- a/backend/app/repositories/telemetry/sqlite/profile.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/profile.repository.go
@@ -150,25 +150,24 @@ func (r *profileRepository) InsertProfilesAsync(ctx context.Context, profiles []
for _, p := range profiles {
attributesVal, _ := sqlitetypes.NewSQLiteJSONMap(p.Attributes).Value()
query, args, err := lit.ParseNamedQuery(db.Driver,
- "INSERT INTO profiles (id, project_id, recorded_at, duration, service_name, profile_type, unit, is_gauge, sample_count, total_value, server_name, app_version, attributes, storage_key, trace_id, span_id, distributed_trace_id) VALUES (:id, :project_id, :recorded_at, :duration, :service_name, :profile_type, :unit, :is_gauge, :sample_count, :total_value, :server_name, :app_version, :attributes, :storage_key, :trace_id, :span_id, :distributed_trace_id)",
+ "INSERT INTO profiles (id, project_id, recorded_at, duration, service_name, profile_type, unit, is_gauge, sample_count, total_value, server_name, app_version, attributes, storage_key, trace_id, span_id) VALUES (:id, :project_id, :recorded_at, :duration, :service_name, :profile_type, :unit, :is_gauge, :sample_count, :total_value, :server_name, :app_version, :attributes, :storage_key, :trace_id, :span_id)",
lit.P{
- "id": p.Id,
- "project_id": p.ProjectId,
- "recorded_at": sqlitetypes.NewSQLiteTime(p.RecordedAt),
- "duration": int64(p.Duration),
- "service_name": p.ServiceName,
- "profile_type": p.ProfileType,
- "unit": p.Unit,
- "is_gauge": boolToInt(p.IsGauge),
- "sample_count": int64(p.SampleCount),
- "total_value": p.TotalValue,
- "server_name": p.ServerName,
- "app_version": p.AppVersion,
- "attributes": attributesVal,
- "storage_key": p.StorageKey,
- "trace_id": p.TraceId,
- "span_id": p.SpanId,
- "distributed_trace_id": p.DistributedTraceId,
+ "id": p.Id,
+ "project_id": p.ProjectId,
+ "recorded_at": sqlitetypes.NewSQLiteTime(p.RecordedAt),
+ "duration": int64(p.Duration),
+ "service_name": p.ServiceName,
+ "profile_type": p.ProfileType,
+ "unit": p.Unit,
+ "is_gauge": boolToInt(p.IsGauge),
+ "sample_count": int64(p.SampleCount),
+ "total_value": p.TotalValue,
+ "server_name": p.ServerName,
+ "app_version": p.AppVersion,
+ "attributes": attributesVal,
+ "storage_key": p.StorageKey,
+ "trace_id": p.TraceId,
+ "span_id": p.SpanId,
})
if err != nil {
return err
diff --git a/backend/app/repositories/telemetry/sqlite/session.repository.go b/backend/app/repositories/telemetry/sqlite/session.repository.go
index f6ac894c..cb7b5c4e 100644
--- a/backend/app/repositories/telemetry/sqlite/session.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/session.repository.go
@@ -17,16 +17,16 @@ import (
)
type sessionRow struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- StartedAt sqlitetypes.SQLiteTime `lit:"started_at"`
- EndedAt *sqlitetypes.SQLiteTime `lit:"ended_at"`
- Duration int64 `lit:"duration"`
- ClientIP string `lit:"client_ip"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ StartedAt sqlitetypes.SQLiteTime `lit:"started_at"`
+ EndedAt *sqlitetypes.SQLiteTime `lit:"ended_at"`
+ Duration int64 `lit:"duration"`
+ ClientIP string `lit:"client_ip"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ TraceId string `lit:"trace_id"`
}
type sessionRowNaming struct{ lit.DefaultDbNamingStrategy }
@@ -43,15 +43,15 @@ func init() {
func sessionToRow(s models.Session) sessionRow {
row := sessionRow{
- Id: s.Id,
- ProjectId: s.ProjectId,
- StartedAt: sqlitetypes.NewSQLiteTime(s.StartedAt),
- Duration: s.Duration,
- ClientIP: s.ClientIP,
- Attributes: sqlitetypes.NewSQLiteJSONMap(s.Attributes),
- AppVersion: s.AppVersion,
- ServerName: s.ServerName,
- DistributedTraceId: s.DistributedTraceId,
+ Id: s.Id,
+ ProjectId: s.ProjectId,
+ StartedAt: sqlitetypes.NewSQLiteTime(s.StartedAt),
+ Duration: s.Duration,
+ ClientIP: s.ClientIP,
+ Attributes: sqlitetypes.NewSQLiteJSONMap(s.Attributes),
+ AppVersion: s.AppVersion,
+ ServerName: s.ServerName,
+ TraceId: s.TraceId,
}
if s.EndedAt != nil {
t := sqlitetypes.NewSQLiteTime(*s.EndedAt)
@@ -62,14 +62,14 @@ func sessionToRow(s models.Session) sessionRow {
func (row *sessionRow) toModel() models.Session {
s := models.Session{
- Id: row.Id,
- ProjectId: row.ProjectId,
- StartedAt: row.StartedAt.Time,
- Duration: row.Duration,
- ClientIP: row.ClientIP,
- AppVersion: row.AppVersion,
- ServerName: row.ServerName,
- DistributedTraceId: row.DistributedTraceId,
+ Id: row.Id,
+ ProjectId: row.ProjectId,
+ StartedAt: row.StartedAt.Time,
+ Duration: row.Duration,
+ ClientIP: row.ClientIP,
+ AppVersion: row.AppVersion,
+ ServerName: row.ServerName,
+ TraceId: row.TraceId,
}
if row.EndedAt != nil {
t := row.EndedAt.Time
@@ -88,7 +88,7 @@ func (r *sessionRepository) Upsert(ctx context.Context, sessions []models.Sessio
return nil
}
const stmt = `INSERT INTO sessions
- (id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id)
+ (id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
ended_at = COALESCE(excluded.ended_at, sessions.ended_at),
@@ -96,7 +96,7 @@ func (r *sessionRepository) Upsert(ctx context.Context, sessions []models.Sessio
attributes = excluded.attributes,
app_version = excluded.app_version,
server_name = excluded.server_name,
- distributed_trace_id = excluded.distributed_trace_id`
+ trace_id = excluded.trace_id`
tx, err := db.TelemetryDB.BeginTx(ctx, nil)
if err != nil {
@@ -124,7 +124,7 @@ func (r *sessionRepository) Upsert(ctx context.Context, sessions []models.Sessio
}
if _, err := tx.ExecContext(ctx, stmt,
row.Id, row.ProjectId, startedAt, endedAt, row.Duration,
- row.ClientIP, attrs, row.AppVersion, row.ServerName, row.DistributedTraceId,
+ row.ClientIP, attrs, row.AppVersion, row.ServerName, row.TraceId,
); err != nil {
return err
}
@@ -179,7 +179,7 @@ func (r *sessionRepository) FindAll(ctx context.Context, projectId uuid.UUID, fr
}
offset := (page - 1) * pageSize
- query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id FROM sessions WHERE project_id = :project_id AND started_at >= :start AND started_at <= :end" + whereExtra + " ORDER BY " + orderBy + " " + sortDir + " LIMIT :limit OFFSET :offset"
+ query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id FROM sessions WHERE project_id = :project_id AND started_at >= :start AND started_at <= :end" + whereExtra + " ORDER BY " + orderBy + " " + sortDir + " LIMIT :limit OFFSET :offset"
queryParams := lit.P{
"project_id": projectId,
@@ -233,7 +233,7 @@ func buildSessionFilterClauseSQLite(search string, filters []shared.SessionAttri
}
func (r *sessionRepository) FindById(ctx context.Context, projectId, sessionId uuid.UUID, startedAt *time.Time) (*models.Session, error) {
- query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, distributed_trace_id FROM sessions WHERE project_id = :project_id AND id = :id"
+ query := "SELECT id, project_id, started_at, ended_at, duration, client_ip, attributes, app_version, server_name, trace_id FROM sessions WHERE project_id = :project_id AND id = :id"
params := lit.P{"project_id": projectId, "id": sessionId}
if startedAt != nil {
from, to := shared.TraceWindowBounds(*startedAt)
diff --git a/backend/app/repositories/telemetry/sqlite/span.repository.go b/backend/app/repositories/telemetry/sqlite/span.repository.go
deleted file mode 100644
index b0ed3ed4..00000000
--- a/backend/app/repositories/telemetry/sqlite/span.repository.go
+++ /dev/null
@@ -1,140 +0,0 @@
-//go:build !telemetry_ch && !telemetry_duckdb
-
-package sqlite
-
-import (
- "context"
- "slices"
- "time"
-
- "github.com/google/uuid"
- "github.com/tracewayapp/lit/v2"
- "github.com/tracewayapp/traceway/backend/app/db"
- "github.com/tracewayapp/traceway/backend/app/models"
- "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/shared"
- "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
-)
-
-type span struct {
- Id uuid.UUID `lit:"id"`
- TraceId uuid.UUID `lit:"trace_id"`
- ProjectId uuid.UUID `lit:"project_id"`
- Name string `lit:"name"`
- StartTime sqlitetypes.SQLiteTime `lit:"start_time"`
- Duration int64 `lit:"duration"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- ParentSpanId *uuid.UUID `lit:"parent_span_id"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
-}
-
-func init() {
- models.ExtensionModelRegistrations = append(models.ExtensionModelRegistrations, func(driver lit.Driver) {
- lit.RegisterModel[span](driver)
- })
-}
-
-func spanToRow(s models.Span) span {
- return span{
- Id: s.Id,
- TraceId: s.TraceId,
- ProjectId: s.ProjectId,
- Name: s.Name,
- StartTime: sqlitetypes.NewSQLiteTime(s.StartTime),
- Duration: int64(s.Duration),
- RecordedAt: sqlitetypes.NewSQLiteTime(s.RecordedAt),
- ParentSpanId: s.ParentSpanId,
- Attributes: sqlitetypes.NewSQLiteJSONMap(s.Attributes),
- }
-}
-
-func (r *span) toModel() models.Span {
- s := models.Span{
- Id: r.Id,
- TraceId: r.TraceId,
- ProjectId: r.ProjectId,
- Name: r.Name,
- StartTime: r.StartTime.Time,
- Duration: time.Duration(r.Duration),
- RecordedAt: r.RecordedAt.Time,
- ParentSpanId: r.ParentSpanId,
- }
- if r.Attributes != nil {
- s.Attributes = map[string]string(r.Attributes)
- }
- return s
-}
-
-type spanRepository struct{}
-
-func (r *spanRepository) InsertAsync(ctx context.Context, spans []models.Span) error {
- if len(spans) == 0 {
- return nil
- }
-
- tx, err := db.TelemetryDB.BeginTx(ctx, nil)
- if err != nil {
- return err
- }
- defer tx.Rollback()
-
- for _, s := range spans {
- row := spanToRow(s)
- if err := lit.InsertExistingUuid(tx, &row); err != nil {
- return err
- }
- }
-
- return tx.Commit()
-}
-
-func (r *spanRepository) FindByTraceId(ctx context.Context, projectId, traceId uuid.UUID, recordedAt *time.Time) ([]models.Span, error) {
- query, args := shared.SpanLookupQuery([]shared.SpanLookup{{ProjectId: projectId, TraceId: traceId, RecordedAt: recordedAt}},
- func(t time.Time) any { return sqlitetypes.NewSQLiteTime(t) })
- return r.querySpans(ctx, query, args)
-}
-
-func (r *spanRepository) FindByTraces(ctx context.Context, lookups []shared.SpanLookup) ([]models.Span, error) {
- spans := make([]models.Span, 0)
- type spanKey struct {
- projectId, traceId, id uuid.UUID
- recordedAt time.Time
- }
- seen := make(map[spanKey]bool)
- for batch := range slices.Chunk(lookups, shared.SpanLookupBatchSize) {
- query, args := shared.SpanLookupQuery(batch, func(t time.Time) any { return sqlitetypes.NewSQLiteTime(t) })
- found, err := r.querySpans(ctx, query, args)
- if err != nil {
- return nil, err
- }
- for _, span := range found {
- key := spanKey{span.ProjectId, span.TraceId, span.Id, span.RecordedAt}
- if !seen[key] {
- seen[key] = true
- spans = append(spans, span)
- }
- }
- }
- slices.SortStableFunc(spans, func(a, b models.Span) int { return a.StartTime.Compare(b.StartTime) })
- return spans, nil
-}
-
-func (r *spanRepository) querySpans(ctx context.Context, query string, args []any) ([]models.Span, error) {
- rows, err := db.TelemetryDB.QueryContext(ctx, query, args...)
- if err != nil {
- return nil, err
- }
- defer rows.Close()
-
- spans := make([]models.Span, 0)
- for rows.Next() {
- var row span
- if err := rows.Scan(&row.Id, &row.TraceId, &row.ProjectId, &row.Name, &row.StartTime,
- &row.Duration, &row.RecordedAt, &row.ParentSpanId, &row.Attributes); err != nil {
- return nil, err
- }
- spans = append(spans, row.toModel())
- }
- return spans, rows.Err()
-}
-
-var SpanRepository = &spanRepository{}
diff --git a/backend/app/repositories/telemetry/sqlite/task.repository.go b/backend/app/repositories/telemetry/sqlite/task.repository.go
index de939d31..78a6b244 100644
--- a/backend/app/repositories/telemetry/sqlite/task.repository.go
+++ b/backend/app/repositories/telemetry/sqlite/task.repository.go
@@ -18,18 +18,19 @@ import (
)
type task struct {
- Id uuid.UUID `lit:"id"`
- ProjectId uuid.UUID `lit:"project_id"`
- TaskName string `lit:"task_name"`
- Duration int64 `lit:"duration"`
- RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
- ClientIP string `lit:"client_ip"`
- Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
- AppVersion string `lit:"app_version"`
- ServerName string `lit:"server_name"`
- DistributedTraceId *uuid.UUID `lit:"distributed_trace_id"`
- SpanId *uuid.UUID `lit:"span_id"`
- IsRoot bool `lit:"is_root"`
+ Id uuid.UUID `lit:"id"`
+ ProjectId uuid.UUID `lit:"project_id"`
+ TaskName string `lit:"task_name"`
+ Duration int64 `lit:"duration"`
+ RecordedAt sqlitetypes.SQLiteTime `lit:"recorded_at"`
+ ClientIP string `lit:"client_ip"`
+ Attributes sqlitetypes.SQLiteJSONMap `lit:"attributes"`
+ AppVersion string `lit:"app_version"`
+ ServerName string `lit:"server_name"`
+ TraceId string `lit:"trace_id"`
+ SpanId string `lit:"span_id"`
+ ParentSpanId string `lit:"parent_span_id"`
+ IsRoot bool `lit:"is_root"`
}
type taskGroupRow struct {
@@ -50,9 +51,15 @@ type durationValueRow struct {
Duration float64 `lit:"duration"`
}
+type taskRowNaming struct{ lit.DefaultDbNamingStrategy }
+
+func (taskRowNaming) GetTableNameFromStructName(string) string {
+ return "tasks_v2"
+}
+
func init() {
models.ExtensionModelRegistrations = append(models.ExtensionModelRegistrations, func(driver lit.Driver) {
- lit.RegisterModel[task](driver)
+ lit.RegisterModelWithNaming[task](driver, taskRowNaming{})
lit.RegisterModel[taskGroupRow](driver)
lit.RegisterModel[taskCountStatsRow](driver)
lit.RegisterModel[durationValueRow](driver)
@@ -61,34 +68,36 @@ func init() {
func taskToRow(t models.Task) task {
return task{
- Id: t.Id,
- ProjectId: t.ProjectId,
- TaskName: t.TaskName,
- Duration: int64(t.Duration),
- RecordedAt: sqlitetypes.NewSQLiteTime(t.RecordedAt),
- ClientIP: t.ClientIP,
- Attributes: sqlitetypes.NewSQLiteJSONMap(t.Attributes),
- AppVersion: t.AppVersion,
- ServerName: t.ServerName,
- DistributedTraceId: t.DistributedTraceId,
- SpanId: t.SpanId,
- IsRoot: t.IsRoot,
+ Id: t.Id,
+ ProjectId: t.ProjectId,
+ TaskName: t.TaskName,
+ Duration: int64(t.Duration),
+ RecordedAt: sqlitetypes.NewSQLiteTime(t.RecordedAt),
+ ClientIP: t.ClientIP,
+ Attributes: sqlitetypes.NewSQLiteJSONMap(t.Attributes),
+ AppVersion: t.AppVersion,
+ ServerName: t.ServerName,
+ TraceId: t.TraceId,
+ SpanId: t.SpanId,
+ ParentSpanId: t.ParentSpanId,
+ IsRoot: t.IsRoot,
}
}
func (r *task) toModel() models.Task {
t := models.Task{
- Id: r.Id,
- ProjectId: r.ProjectId,
- TaskName: r.TaskName,
- Duration: time.Duration(r.Duration),
- RecordedAt: r.RecordedAt.Time,
- ClientIP: r.ClientIP,
- AppVersion: r.AppVersion,
- ServerName: r.ServerName,
- DistributedTraceId: r.DistributedTraceId,
- SpanId: r.SpanId,
- IsRoot: r.IsRoot,
+ Id: r.Id,
+ ProjectId: r.ProjectId,
+ TaskName: r.TaskName,
+ Duration: time.Duration(r.Duration),
+ RecordedAt: r.RecordedAt.Time,
+ ClientIP: r.ClientIP,
+ AppVersion: r.AppVersion,
+ ServerName: r.ServerName,
+ TraceId: r.TraceId,
+ SpanId: r.SpanId,
+ ParentSpanId: r.ParentSpanId,
+ IsRoot: r.IsRoot,
}
if r.Attributes != nil {
t.Attributes = map[string]string(r.Attributes)
@@ -121,7 +130,7 @@ func (e *taskRepository) InsertAsync(ctx context.Context, lines []models.Task) e
func (e *taskRepository) CountBetween(ctx context.Context, projectId uuid.UUID, start, end time.Time) (int64, error) {
result, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
return 0, err
@@ -134,7 +143,7 @@ func (e *taskRepository) CountBetween(ctx context.Context, projectId uuid.UUID,
func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromDate, toDate time.Time, page, pageSize int, orderBy string) ([]models.Task, int64, error) {
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to",
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)})
if err != nil {
return nil, 0, err
@@ -152,8 +161,8 @@ func (e *taskRepository) FindAll(ctx context.Context, projectId uuid.UUID, fromD
}
rows, err := lit.SelectNamed[task](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s DESC LIMIT :limit OFFSET :offset`, orderBy),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate), "limit": pageSize, "offset": offset})
if err != nil {
@@ -183,7 +192,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
whereClause := "project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to" + taskFilterClause(params, search, rootFilter)
totalResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(DISTINCT task_name) AS count FROM tasks WHERE "+whereClause,
+ "SELECT COUNT(DISTINCT task_name) AS count FROM tasks_v2 WHERE "+whereClause,
params)
if err != nil {
return nil, 0, err
@@ -209,7 +218,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
if needsGoSort {
baseQuery = `SELECT ` + groupedCols + `
- FROM tasks WHERE ` + whereClause + `
+ FROM tasks_v2 WHERE ` + whereClause + `
GROUP BY task_name`
} else {
orderExpr := map[string]string{"count": "count", "last_seen": "last_seen"}
@@ -218,7 +227,7 @@ func (e *taskRepository) FindGroupedByTaskName(ctx context.Context, projectId uu
expr = "count"
}
baseQuery = fmt.Sprintf(`SELECT `+groupedCols+`
- FROM tasks WHERE `+whereClause+`
+ FROM tasks_v2 WHERE `+whereClause+`
GROUP BY task_name ORDER BY %s %s LIMIT :limit OFFSET :offset`, expr, sortDir)
params["limit"] = pageSize
params["offset"] = offset
@@ -294,7 +303,7 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
params := lit.P{"project_id": projectId, "task_name": taskName, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate)}
countResult, err := lit.SelectSingleNamed[models.CountResult](db.TelemetryDB,
- "SELECT COUNT(*) AS count FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, 0, err
@@ -317,8 +326,8 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
}
rows, err := lit.SelectNamed[task](db.TelemetryDB,
- fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to
+ fmt.Sprintf(`SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id
+ FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to
ORDER BY %s %s LIMIT :limit OFFSET :offset`, orderBy, sortDir),
lit.P{"project_id": projectId, "task_name": taskName, "from": sqlitetypes.NewSQLiteTime(fromDate), "to": sqlitetypes.NewSQLiteTime(toDate), "limit": pageSize, "offset": offset})
if err != nil {
@@ -334,8 +343,8 @@ func (e *taskRepository) FindByTaskName(ctx context.Context, projectId uuid.UUID
}
func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UUID, recordedAt *time.Time) (*models.Task, error) {
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id, span_id, is_root
- FROM tasks WHERE project_id = :project_id AND id = :id`
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM tasks_v2 WHERE project_id = :project_id AND id = :id`
params := lit.P{"project_id": projectId, "id": taskId}
if recordedAt != nil {
from, to := shared.TraceWindowBounds(*recordedAt)
@@ -359,7 +368,7 @@ func (e *taskRepository) FindById(ctx context.Context, projectId, taskId uuid.UU
func (e *taskRepository) CountByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime('%Y-%m-%d %H:00:00', recorded_at) as bucket, CAST(COUNT(*) AS REAL) as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -371,7 +380,7 @@ func (e *taskRepository) CountByHour(ctx context.Context, projectId uuid.UUID, s
func (e *taskRepository) AvgDurationByHour(ctx context.Context, projectId uuid.UUID, start, end time.Time) ([]models.TimeSeriesPoint, error) {
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
`SELECT strftime('%Y-%m-%d %H:00:00', recorded_at) as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`,
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -384,7 +393,7 @@ func (e *taskRepository) CountByInterval(ctx context.Context, projectId uuid.UUI
secs := intervalMinutes * 60
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket, CAST(COUNT(*) AS REAL) as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, secs, secs),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -397,7 +406,7 @@ func (e *taskRepository) AvgDurationByInterval(ctx context.Context, projectId uu
secs := intervalMinutes * 60
results, err := lit.SelectNamed[sqlitetypes.TimeSeriesResult](db.TelemetryDB,
fmt.Sprintf(`SELECT datetime((strftime('%%s', recorded_at) / %d) * %d, 'unixepoch') as bucket, AVG(duration) / 1000000.0 as agg_value
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY bucket ORDER BY bucket ASC`, secs, secs),
lit.P{"project_id": projectId, "from": sqlitetypes.NewSQLiteTime(start), "to": sqlitetypes.NewSQLiteTime(end)})
if err != nil {
@@ -411,7 +420,7 @@ func (e *taskRepository) FindWorstTasks(ctx context.Context, projectId uuid.UUID
groups, err := lit.SelectNamed[taskGroupRow](db.TelemetryDB,
`SELECT task_name, COUNT(*) as count, AVG(duration) as avg_duration, MAX(recorded_at) as last_seen
- FROM tasks WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
+ FROM tasks_v2 WHERE project_id = :project_id AND recorded_at >= :from AND recorded_at <= :to
GROUP BY task_name`,
params)
if err != nil {
@@ -457,7 +466,7 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
}
statsRow, err := lit.SelectSingleNamed[taskCountStatsRow](db.TelemetryDB,
- "SELECT COUNT(*) AS count, CASE WHEN COUNT(*) > 0 THEN AVG(duration) / 1000000.0 ELSE 0 END AS avg_dur_ms FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
+ "SELECT COUNT(*) AS count, CASE WHEN COUNT(*) > 0 THEN AVG(duration) / 1000000.0 ELSE 0 END AS avg_dur_ms FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to",
params)
if err != nil {
return nil, err
@@ -483,19 +492,20 @@ func (e *taskRepository) GetTaskStats(ctx context.Context, projectId uuid.UUID,
}, nil
}
-func (e *taskRepository) FindByDistributedTraceId(ctx context.Context, distributedTraceId uuid.UUID, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Task, error) {
- if len(projectIds) == 0 {
+func (e *taskRepository) FindByTraceIds(ctx context.Context, traceIds []string, projectIds []uuid.UUID, recordedAt *time.Time) ([]models.Task, error) {
+ if len(projectIds) == 0 || len(traceIds) == 0 {
return nil, nil
}
- params := lit.P{"trace_id": distributedTraceId}
+ params := lit.P{}
+ traceFilter := shared.TraceIdsFilter(traceIds, params)
placeholders := make([]string, len(projectIds))
for i, pid := range projectIds {
key := fmt.Sprintf("pid_%d", i)
placeholders[i] = ":" + key
params[key] = pid
}
- query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, distributed_trace_id
- FROM tasks WHERE distributed_trace_id = :trace_id AND project_id IN (` + strings.Join(placeholders, ",") + `)`
+ query := `SELECT id, project_id, task_name, duration, recorded_at, client_ip, attributes, app_version, server_name, trace_id, span_id, parent_span_id, is_root
+ FROM tasks_v2 WHERE ` + traceFilter + ` AND project_id IN (` + strings.Join(placeholders, ",") + `)`
if recordedAt != nil {
from, to := shared.DistributedTraceWindowBounds(*recordedAt)
query += ` AND recorded_at >= :from AND recorded_at <= :to`
@@ -518,7 +528,7 @@ func (e *taskRepository) FindByDistributedTraceId(ctx context.Context, distribut
var tasks []models.Task
for sqlRows.Next() {
var row task
- if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TaskName, &row.Duration, &row.RecordedAt, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.DistributedTraceId); err != nil {
+ if err := sqlRows.Scan(&row.Id, &row.ProjectId, &row.TaskName, &row.Duration, &row.RecordedAt, &row.ClientIP, &row.Attributes, &row.AppVersion, &row.ServerName, &row.TraceId, &row.SpanId, &row.ParentSpanId, &row.IsRoot); err != nil {
return nil, err
}
tasks = append(tasks, row.toModel())
@@ -531,7 +541,7 @@ func fetchSortedTaskDurations(ctx context.Context, projectId uuid.UUID, taskName
filterClause := taskFilterClause(params, search, rootFilter)
results, err := lit.SelectNamed[durationValueRow](db.TelemetryDB,
- "SELECT duration FROM tasks WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to"+filterClause+" ORDER BY duration ASC",
+ "SELECT duration FROM tasks_v2 WHERE project_id = :project_id AND task_name = :task_name AND recorded_at >= :from AND recorded_at <= :to"+filterClause+" ORDER BY duration ASC",
params)
if err != nil {
return nil, err
diff --git a/backend/app/repositories/telemetry/storage_errors.go b/backend/app/repositories/telemetry/storage_errors.go
new file mode 100644
index 00000000..6f5d133d
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors.go
@@ -0,0 +1,26 @@
+package telemetry
+
+import (
+ "context"
+ "errors"
+ "io"
+ "net"
+ "syscall"
+)
+
+// IsTransientStorageError reports failures worth retrying, so ingest can answer 503 instead of a terminal 500.
+func IsTransientStorageError(err error) bool {
+ if err == nil {
+ return false
+ }
+ if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) ||
+ errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, net.ErrClosed) ||
+ errors.Is(err, syscall.ECONNRESET) || errors.Is(err, syscall.ECONNREFUSED) || errors.Is(err, syscall.EPIPE) {
+ return true
+ }
+ var netErr net.Error
+ if errors.As(err, &netErr) && netErr.Timeout() {
+ return true
+ }
+ return isTransientBackendError(err)
+}
diff --git a/backend/app/repositories/telemetry/storage_errors_ch.go b/backend/app/repositories/telemetry/storage_errors_ch.go
new file mode 100644
index 00000000..c0ea28ac
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors_ch.go
@@ -0,0 +1,42 @@
+//go:build telemetry_ch
+
+package telemetry
+
+import (
+ "errors"
+ "strings"
+
+ "github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/tracewayapp/traceway/backend/app/chdb"
+)
+
+var transientClickHouseCodes = map[int32]bool{
+ 159: true, // TIMEOUT_EXCEEDED
+ 160: true, // TOO_SLOW
+ 164: true, // READONLY
+ 202: true, // TOO_MANY_SIMULTANEOUS_QUERIES
+ 203: true, // NO_FREE_CONNECTION
+ 209: true, // SOCKET_TIMEOUT
+ 210: true, // NETWORK_ERROR
+ 236: true, // ABORTED
+ 241: true, // MEMORY_LIMIT_EXCEEDED
+ 242: true, // TABLE_IS_READ_ONLY
+ 243: true, // NOT_ENOUGH_SPACE
+ 252: true, // TOO_MANY_PARTS
+ 319: true, // UNKNOWN_STATUS_OF_INSERT
+ 394: true, // QUERY_WAS_CANCELLED
+ 425: true, // SYSTEM_ERROR
+ 999: true, // KEEPER_EXCEPTION
+}
+
+func isTransientBackendError(err error) bool {
+ var exception *clickhouse.Exception
+ if errors.As(err, &exception) {
+ // Code 252 is also raised for one insert spanning too many partitions, which no retry can fix.
+ if exception.Code == 252 && strings.Contains(exception.Message, "partitions for single INSERT block") {
+ return false
+ }
+ return transientClickHouseCodes[exception.Code]
+ }
+ return chdb.IsConnError(err)
+}
diff --git a/backend/app/repositories/telemetry/storage_errors_ch_test.go b/backend/app/repositories/telemetry/storage_errors_ch_test.go
new file mode 100644
index 00000000..dc9b3679
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors_ch_test.go
@@ -0,0 +1,29 @@
+//go:build telemetry_ch
+
+package telemetry
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/ClickHouse/clickhouse-go/v2"
+)
+
+func TestClickHouseOverloadIsRetryableButPoisonIsNot(t *testing.T) {
+ for _, exception := range []*clickhouse.Exception{
+ {Code: 241, Message: "Memory limit (total) exceeded"}, {Code: 202, Message: "Too many simultaneous queries"},
+ {Code: 252, Message: "Too many parts (300). Merges are processing significantly slower than inserts"},
+ } {
+ if !IsTransientStorageError(fmt.Errorf("insert: %w", exception)) {
+ t.Errorf("code %d should be retryable", exception.Code)
+ }
+ }
+ for _, exception := range []*clickhouse.Exception{
+ {Code: 252, Message: "Too many partitions for single INSERT block (more than 100)"}, {Code: 62, Message: "Syntax error"},
+ {Code: 53, Message: "Type mismatch in IN or VALUES section"},
+ } {
+ if IsTransientStorageError(fmt.Errorf("insert: %w", exception)) {
+ t.Errorf("code %d (%s) is permanent and must not be retried", exception.Code, exception.Message)
+ }
+ }
+}
diff --git a/backend/app/repositories/telemetry/storage_errors_duckdb.go b/backend/app/repositories/telemetry/storage_errors_duckdb.go
new file mode 100644
index 00000000..5f3de8d7
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors_duckdb.go
@@ -0,0 +1,10 @@
+//go:build telemetry_duckdb
+
+package telemetry
+
+import "strings"
+
+func isTransientBackendError(err error) bool {
+ message := strings.ToLower(err.Error())
+ return strings.Contains(message, "out of memory") || strings.Contains(message, "could not set lock on file")
+}
diff --git a/backend/app/repositories/telemetry/storage_errors_sqlite.go b/backend/app/repositories/telemetry/storage_errors_sqlite.go
new file mode 100644
index 00000000..5f38620b
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors_sqlite.go
@@ -0,0 +1,10 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package telemetry
+
+import "strings"
+
+func isTransientBackendError(err error) bool {
+ message := strings.ToLower(err.Error())
+ return strings.Contains(message, "database is locked") || strings.Contains(message, "database table is locked") || strings.Contains(message, "sqlite_busy")
+}
diff --git a/backend/app/repositories/telemetry/storage_errors_sqlite_test.go b/backend/app/repositories/telemetry/storage_errors_sqlite_test.go
new file mode 100644
index 00000000..2b73dc65
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors_sqlite_test.go
@@ -0,0 +1,17 @@
+//go:build !telemetry_ch && !telemetry_duckdb
+
+package telemetry
+
+import (
+ "errors"
+ "testing"
+)
+
+func TestSQLiteBusyIsRetryable(t *testing.T) {
+ if !IsTransientStorageError(errors.New("database is locked (5) (SQLITE_BUSY)")) {
+ t.Fatal("a busy SQLite database should be retried")
+ }
+ if IsTransientStorageError(errors.New("UNIQUE constraint failed: sessions.id")) {
+ t.Fatal("a constraint violation is permanent")
+ }
+}
diff --git a/backend/app/repositories/telemetry/storage_errors_test.go b/backend/app/repositories/telemetry/storage_errors_test.go
new file mode 100644
index 00000000..a220f7ad
--- /dev/null
+++ b/backend/app/repositories/telemetry/storage_errors_test.go
@@ -0,0 +1,26 @@
+package telemetry
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io"
+ "syscall"
+ "testing"
+)
+
+func TestTransientStorageErrorsAreRetryable(t *testing.T) {
+ for _, err := range []error{
+ context.DeadlineExceeded, fmt.Errorf("insert: %w", context.DeadlineExceeded), fmt.Errorf("read: %w", io.ErrUnexpectedEOF),
+ fmt.Errorf("dial: %w", syscall.ECONNREFUSED), fmt.Errorf("write: %w", syscall.EPIPE),
+ } {
+ if !IsTransientStorageError(err) {
+ t.Errorf("%v should be retryable", err)
+ }
+ }
+ for _, err := range []error{nil, errors.New("invalid stored OTel span ID"), errors.New("syntax error near SELECT")} {
+ if IsTransientStorageError(err) {
+ t.Errorf("%v must not be retried forever", err)
+ }
+ }
+}
diff --git a/backend/app/repositories/telemetry/telemetry_ch.go b/backend/app/repositories/telemetry/telemetry_ch.go
index 4285a693..878131f7 100644
--- a/backend/app/repositories/telemetry/telemetry_ch.go
+++ b/backend/app/repositories/telemetry/telemetry_ch.go
@@ -15,6 +15,7 @@ var (
ProfileRepository = ch.ProfileRepository
SessionRecordingRepository = ch.SessionRecordingRepository
SessionRepository = ch.SessionRepository
- SpanRepository = ch.SpanRepository
+ OtelSpanRepository = ch.OtelSpanRepository
+ LegacyRepository = ch.LegacyRepository
TaskRepository = ch.TaskRepository
)
diff --git a/backend/app/repositories/telemetry/telemetry_duckdb.go b/backend/app/repositories/telemetry/telemetry_duckdb.go
index a77939ea..764c37d2 100644
--- a/backend/app/repositories/telemetry/telemetry_duckdb.go
+++ b/backend/app/repositories/telemetry/telemetry_duckdb.go
@@ -15,6 +15,7 @@ var (
ProfileRepository = duckdbrepo.ProfileRepository
SessionRecordingRepository = duckdbrepo.SessionRecordingRepository
SessionRepository = duckdbrepo.SessionRepository
- SpanRepository = duckdbrepo.SpanRepository
+ OtelSpanRepository = duckdbrepo.OtelSpanRepository
+ LegacyRepository = duckdbrepo.LegacyRepository
TaskRepository = duckdbrepo.TaskRepository
)
diff --git a/backend/app/repositories/telemetry/telemetry_sqlite.go b/backend/app/repositories/telemetry/telemetry_sqlite.go
index 94fa8ec9..4cd7d8ef 100644
--- a/backend/app/repositories/telemetry/telemetry_sqlite.go
+++ b/backend/app/repositories/telemetry/telemetry_sqlite.go
@@ -15,6 +15,7 @@ var (
ProfileRepository = sqliterepo.ProfileRepository
SessionRecordingRepository = sqliterepo.SessionRecordingRepository
SessionRepository = sqliterepo.SessionRepository
- SpanRepository = sqliterepo.SpanRepository
+ OtelSpanRepository = sqliterepo.OtelSpanRepository
+ LegacyRepository = sqliterepo.LegacyRepository
TaskRepository = sqliterepo.TaskRepository
)
diff --git a/backend/app/repositories/telemetry/testhelper_clickhouse_test.go b/backend/app/repositories/telemetry/testhelper_clickhouse_test.go
index a2c0885c..b2e28939 100644
--- a/backend/app/repositories/telemetry/testhelper_clickhouse_test.go
+++ b/backend/app/repositories/telemetry/testhelper_clickhouse_test.go
@@ -10,6 +10,7 @@ import (
"testing"
"github.com/ClickHouse/clickhouse-go/v2"
+ "github.com/google/uuid"
_ "github.com/lib/pq"
"github.com/tracewayapp/lit/v2"
"github.com/tracewayapp/traceway/backend/app/chdb"
@@ -100,8 +101,9 @@ func setupTestDB(t *testing.T) {
t.Cleanup(func() {
ctx := context.Background()
tables := []string{
- "endpoints", "tasks", "exception_stack_traces",
- "spans", "metric_points", "session_recordings",
+ "endpoints_v2", "tasks_v2", "exceptions_v2", "spans_v2", "ai_traces_v2",
+ "endpoints", "tasks", "exception_stack_traces", "spans", "ai_traces",
+ "metric_points", "session_recordings",
"archived_exceptions", "slow_endpoints", "fired_notifications",
}
for _, table := range tables {
@@ -110,3 +112,42 @@ func setupTestDB(t *testing.T) {
pgDB.Close()
})
}
+
+// legacyExec writes to the tables V2 replaced, which no repository writes any more.
+func legacyExec(t *testing.T, query string, args ...any) {
+ t.Helper()
+ for i, arg := range args {
+ switch value := arg.(type) {
+ case bool:
+ args[i] = uint8(0)
+ if value {
+ args[i] = uint8(1)
+ }
+ case *uuid.UUID:
+ args[i] = nil
+ if value != nil {
+ args[i] = *value
+ }
+ }
+ }
+ if err := chdb.Conn.Exec(context.Background(), query, args...); err != nil {
+ t.Fatalf("%s: %v", query, err)
+ }
+}
+
+func legacyReset(t *testing.T) {
+ t.Helper()
+ for _, statement := range []string{"TRUNCATE TABLE IF EXISTS endpoints", "TRUNCATE TABLE IF EXISTS tasks", "TRUNCATE TABLE IF EXISTS ai_traces",
+ "TRUNCATE TABLE IF EXISTS exception_stack_traces", "TRUNCATE TABLE IF EXISTS spans"} {
+ legacyExec(t, statement)
+ }
+}
+
+func moveOverRowCount(t *testing.T, table string) int64 {
+ t.Helper()
+ var count uint64
+ if err := chdb.Conn.QueryRow(context.Background(), "SELECT count() FROM "+table).Scan(&count); err != nil {
+ t.Fatal(err)
+ }
+ return int64(count)
+}
diff --git a/backend/app/repositories/telemetry/testhelper_duckdb_test.go b/backend/app/repositories/telemetry/testhelper_duckdb_test.go
index 272c905e..161ae809 100644
--- a/backend/app/repositories/telemetry/testhelper_duckdb_test.go
+++ b/backend/app/repositories/telemetry/testhelper_duckdb_test.go
@@ -4,6 +4,9 @@ package telemetry
import (
"database/sql"
+ "time"
+
+ "github.com/google/uuid"
"os"
"path/filepath"
"sort"
@@ -90,3 +93,41 @@ func runDuckDBTelemetryMigrations(telemetryDB *sql.DB) error {
}
return nil
}
+
+// legacyExec writes to the tables V2 replaced, which no repository writes any more.
+func legacyExec(t *testing.T, query string, args ...any) {
+ t.Helper()
+ for i, arg := range args {
+ switch value := arg.(type) {
+ case uuid.UUID:
+ args[i] = value.String()
+ case *uuid.UUID:
+ args[i] = nil
+ if value != nil {
+ args[i] = value.String()
+ }
+ case time.Time:
+ args[i] = value.UTC()
+ }
+ }
+ if _, err := db.TelemetryDB.Exec(query, args...); err != nil {
+ t.Fatalf("%s: %v", query, err)
+ }
+}
+
+func legacyReset(t *testing.T) {
+ t.Helper()
+ for _, statement := range []string{"DELETE FROM endpoints", "DELETE FROM tasks", "DELETE FROM ai_traces", "DELETE FROM exception_stack_traces",
+ "DELETE FROM spans"} {
+ legacyExec(t, statement)
+ }
+}
+
+func moveOverRowCount(t *testing.T, table string) int64 {
+ t.Helper()
+ var count int64
+ if err := db.TelemetryDB.QueryRow("SELECT COUNT(*) FROM " + table).Scan(&count); err != nil {
+ t.Fatal(err)
+ }
+ return count
+}
diff --git a/backend/app/repositories/telemetry/testhelper_sqlite_test.go b/backend/app/repositories/telemetry/testhelper_sqlite_test.go
index eafd9005..d68242a7 100644
--- a/backend/app/repositories/telemetry/testhelper_sqlite_test.go
+++ b/backend/app/repositories/telemetry/testhelper_sqlite_test.go
@@ -3,6 +3,11 @@
package telemetry
import (
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/db"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry/sqlitetypes"
"testing"
"github.com/tracewayapp/traceway/backend/app/dbtest"
@@ -12,3 +17,41 @@ func setupTestDB(t *testing.T) {
t.Helper()
dbtest.SetupSQLite(t)
}
+
+// legacyExec writes to the tables V2 replaced, which no repository writes any more.
+func legacyExec(t *testing.T, query string, args ...any) {
+ t.Helper()
+ for i, arg := range args {
+ switch value := arg.(type) {
+ case uuid.UUID:
+ args[i] = value.String()
+ case *uuid.UUID:
+ args[i] = nil
+ if value != nil {
+ args[i] = value.String()
+ }
+ case time.Time:
+ args[i] = sqlitetypes.NewSQLiteTime(value)
+ }
+ }
+ if _, err := db.TelemetryDB.Exec(query, args...); err != nil {
+ t.Fatalf("%s: %v", query, err)
+ }
+}
+
+func legacyReset(t *testing.T) {
+ t.Helper()
+ for _, statement := range []string{"DELETE FROM endpoints", "DELETE FROM tasks", "DELETE FROM ai_traces", "DELETE FROM exception_stack_traces",
+ "DELETE FROM spans"} {
+ legacyExec(t, statement)
+ }
+}
+
+func moveOverRowCount(t *testing.T, table string) int64 {
+ t.Helper()
+ var count int64
+ if err := db.TelemetryDB.QueryRow("SELECT COUNT(*) FROM " + table).Scan(&count); err != nil {
+ t.Fatal(err)
+ }
+ return count
+}
diff --git a/backend/app/repositories/telemetry/trace_ids_lookup_test.go b/backend/app/repositories/telemetry/trace_ids_lookup_test.go
new file mode 100644
index 00000000..9059f817
--- /dev/null
+++ b/backend/app/repositories/telemetry/trace_ids_lookup_test.go
@@ -0,0 +1,234 @@
+package telemetry
+
+import (
+ "context"
+ "sort"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ "github.com/tracewayapp/traceway/backend/app/models"
+)
+
+const (
+ backendTrace = "0102030405060708090a0b0c0d0e0f10"
+ browserTrace = "a1a2a3a4a5a6a7a8a9aaabacadaeaf00"
+ otherTrace = "f1f2f3f4f5f6f7f8f9fafbfcfdfeff00"
+)
+
+// One request seen from a browser and three services, stored the way ingest stores it: every row keeps the trace id
+// it arrived with. Retired browser correlation metadata must not affect lookups.
+func seedTraceWithLegacyMetadata(t *testing.T, gateway, payments, stranger uuid.UUID, at time.Time) {
+ t.Helper()
+ ctx := context.Background()
+ endpoint := func(project uuid.UUID, name, trace, span, parent, linked string, recordedAt time.Time) models.Endpoint {
+ row := makeEndpoint(project, name, time.Millisecond, 200, recordedAt)
+ row.TraceId, row.SpanId, row.ParentSpanId, row.IsRoot = trace, span, parent, parent == ""
+ row.Attributes = map[string]string{"traceway.distributed_trace_id": linked}
+ return row
+ }
+ if err := EndpointRepository.InsertAsync(ctx, []models.Endpoint{
+ endpoint(gateway, "GET /checkout", backendTrace, "e100000000000001", "", browserTrace, at),
+ endpoint(payments, "POST /charge", backendTrace, "e200000000000002", "c100000000000001", "", at.Add(time.Millisecond)),
+ endpoint(gateway, "GET /unrelated", otherTrace, "e300000000000003", "", "", at),
+ endpoint(stranger, "GET /not-yours", backendTrace, "e400000000000004", "", "", at),
+ endpoint(gateway, "GET /long-ago", backendTrace, "e500000000000005", "", "", at.Add(-72*time.Hour)),
+ }); err != nil {
+ t.Fatal(err)
+ }
+ task := makeTask(payments, "settle", time.Millisecond, at.Add(2*time.Millisecond))
+ task.TraceId, task.SpanId, task.ParentSpanId = backendTrace, "7a00000000000001", "e200000000000002"
+ if err := TaskRepository.InsertAsync(ctx, []models.Task{task}); err != nil {
+ t.Fatal(err)
+ }
+ ai := makeAiTrace(payments, "fraud-check", time.Millisecond, 10, 0.1, at.Add(3*time.Millisecond))
+ ai.TraceId, ai.SpanId, ai.ParentSpanId, ai.IsRoot = backendTrace, "a100000000000001", "e200000000000002", false
+ if err := AiTraceRepository.InsertAsync(ctx, []models.AiTrace{ai}); err != nil {
+ t.Fatal(err)
+ }
+ exception := func(project uuid.UUID, hash, trace, span, kind, linked string) models.ExceptionStackTrace {
+ row := makeException(project, hash, "Error: "+hash, at.Add(4*time.Millisecond))
+ row.TraceId, row.SpanId, row.TraceType = trace, span, kind
+ row.Attributes = map[string]string{"traceway.distributed_trace_id": linked}
+ return row
+ }
+ message := exception(payments, "message", backendTrace, "e200000000000002", "endpoint", "")
+ message.IsMessage = true
+ if err := ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{
+ exception(payments, "on-the-endpoint", backendTrace, "e200000000000002", "endpoint", ""),
+ exception(payments, "below-the-endpoint", backendTrace, "5b00000000000001", "", ""),
+ exception(gateway, "in-the-browser", browserTrace, "b000000000000001", "", ""),
+ exception(gateway, "unrelated", otherTrace, "e300000000000003", "endpoint", ""),
+ message,
+ }); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestFindByTraceIdsNeverFollowsLegacyLinks(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ gateway, payments, stranger := uuid.New(), uuid.New(), uuid.New()
+ at := truncateMs(time.Now().UTC())
+ seedTraceWithLegacyMetadata(t, gateway, payments, stranger, at)
+ readable := []uuid.UUID{gateway, payments}
+
+ names := func(endpoints []models.Endpoint) string {
+ out := make([]string, len(endpoints))
+ for i, endpoint := range endpoints {
+ out[i] = endpoint.Endpoint
+ }
+ sort.Strings(out)
+ return strings.Join(out, ", ")
+ }
+
+ byTrace, err := EndpointRepository.FindByTraceIds(ctx, []string{backendTrace}, readable, &at)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := names(byTrace); got != "GET /checkout, POST /charge" {
+ t.Fatalf("the trace id finds every service's endpoint in the readable projects, inside the window: %s", got)
+ }
+ for _, endpoint := range byTrace {
+ if endpoint.Endpoint == "GET /checkout" && (endpoint.SpanId != "e100000000000001" || endpoint.ParentSpanId != "" || !endpoint.IsRoot) {
+ t.Fatalf("ids did not round trip: %+v", endpoint)
+ }
+ if endpoint.Endpoint == "POST /charge" && (endpoint.ParentSpanId != "c100000000000001" || endpoint.IsRoot) {
+ t.Fatalf("ids did not round trip: %+v", endpoint)
+ }
+ }
+
+ byLink, err := EndpointRepository.FindByTraceIds(ctx, []string{browserTrace}, readable, &at)
+ if err != nil || len(byLink) != 0 {
+ t.Fatalf("legacy metadata must not join separate traces: %s %v", names(byLink), err)
+ }
+ both, err := EndpointRepository.FindByTraceIds(ctx, []string{browserTrace, otherTrace}, readable, &at)
+ if err != nil || names(both) != "GET /unrelated" {
+ t.Fatalf("several ids in one read: %s %v", names(both), err)
+ }
+ unbounded, err := EndpointRepository.FindByTraceIds(ctx, []string{backendTrace}, readable, nil)
+ if err != nil || names(unbounded) != "GET /checkout, GET /long-ago, POST /charge" {
+ t.Fatalf("without a time the window is off: %s %v", names(unbounded), err)
+ }
+ for name, empty := range map[string]func() (int, error){
+ "no trace ids": func() (int, error) {
+ found, err := EndpointRepository.FindByTraceIds(ctx, nil, readable, &at)
+ return len(found), err
+ },
+ "no projects": func() (int, error) {
+ found, err := EndpointRepository.FindByTraceIds(ctx, []string{backendTrace}, nil, &at)
+ return len(found), err
+ },
+ } {
+ if count, err := empty(); err != nil || count != 0 {
+ t.Fatalf("%s: %d %v", name, count, err)
+ }
+ }
+
+ tasks, err := TaskRepository.FindByTraceIds(ctx, []string{backendTrace}, readable, &at)
+ if err != nil || len(tasks) != 1 || tasks[0].SpanId != "7a00000000000001" || tasks[0].ParentSpanId != "e200000000000002" || tasks[0].TraceId != backendTrace {
+ t.Fatalf("tasks: %+v %v", tasks, err)
+ }
+ aiTraces, err := AiTraceRepository.FindByTraceIds(ctx, []string{backendTrace}, readable, &at)
+ if err != nil || len(aiTraces) != 1 || aiTraces[0].SpanId != "a100000000000001" || aiTraces[0].ParentSpanId != "e200000000000002" || aiTraces[0].TraceId != backendTrace {
+ t.Fatalf("ai traces: %+v %v", aiTraces, err)
+ }
+ if none, err := TaskRepository.FindByTraceIds(ctx, nil, readable, &at); err != nil || len(none) != 0 {
+ t.Fatalf("tasks without ids: %v %v", none, err)
+ }
+ if none, err := AiTraceRepository.FindByTraceIds(ctx, nil, readable, &at); err != nil || len(none) != 0 {
+ t.Fatalf("ai traces without ids: %v %v", none, err)
+ }
+}
+
+func TestExceptionsAreFoundByTraceAndKeepTheirSpan(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ gateway, payments, stranger := uuid.New(), uuid.New(), uuid.New()
+ at := truncateMs(time.Now().UTC())
+ seedTraceWithLegacyMetadata(t, gateway, payments, stranger, at)
+
+ hashes := func(exceptions []models.ExceptionStackTrace) string {
+ out := make([]string, len(exceptions))
+ for i, exception := range exceptions {
+ out[i] = exception.ExceptionHash
+ }
+ sort.Strings(out)
+ return strings.Join(out, ", ")
+ }
+
+ card, err := ExceptionStackTraceRepository.FindByTraceIds(ctx, []string{backendTrace, browserTrace}, []uuid.UUID{gateway, payments}, &at)
+ if err != nil || hashes(card) != "below-the-endpoint, in-the-browser, on-the-endpoint" {
+ t.Fatalf("the card reads the exceptions of every id it knows, and no messages: %s %v", hashes(card), err)
+ }
+ if none, err := ExceptionStackTraceRepository.FindByTraceIds(ctx, nil, []uuid.UUID{gateway}, &at); err != nil || len(none) != 0 {
+ t.Fatalf("no ids: %v %v", none, err)
+ }
+
+ page, err := ExceptionStackTraceRepository.FindAllByTraceId(ctx, payments, backendTrace, &at)
+ if err != nil || hashes(page) != "below-the-endpoint, message, on-the-endpoint" {
+ t.Fatalf("a detail page reads its own project's exceptions of the trace, messages included: %s %v", hashes(page), err)
+ }
+ for _, exception := range page {
+ switch exception.ExceptionHash {
+ case "on-the-endpoint":
+ if exception.SpanId != "e200000000000002" || exception.TraceType != "endpoint" {
+ t.Fatalf("span and kind did not round trip: %+v", exception)
+ }
+ case "below-the-endpoint":
+ if exception.SpanId != "5b00000000000001" || exception.TraceType != "" {
+ t.Fatalf("an exception below a promoted span keeps no kind: %+v", exception)
+ }
+ }
+ }
+ if elsewhere, err := ExceptionStackTraceRepository.FindAllByTraceId(ctx, gateway, backendTrace, &at); err != nil || len(elsewhere) != 0 {
+ t.Fatalf("another project of the same trace holds none: %v %v", elsewhere, err)
+ }
+}
+
+func TestFindExceptionOwner(t *testing.T) {
+ setupTestDB(t)
+ ctx := context.Background()
+ gateway, payments, stranger := uuid.New(), uuid.New(), uuid.New()
+ at := truncateMs(time.Now().UTC())
+ seedTraceWithLegacyMetadata(t, gateway, payments, stranger, at)
+ // POST /charge (e2) -> db span (5b) -> driver span (5c). The task hangs under e2 as well.
+ span := func(id, parent string) models.Span {
+ return models.Span{ProjectId: payments, TraceId: backendTrace, SpanId: id, ParentSpanId: parent, Name: id, StartTime: at, RecordedAt: at, Duration: time.Millisecond}
+ }
+ if err := SpanRepository.InsertAsync(ctx, []models.Span{span("e200000000000002", "c100000000000001"), span("5b00000000000001", "e200000000000002"),
+ span("5c00000000000001", "5b00000000000001"), span("7a00000000000001", "e200000000000002"), span("7b00000000000001", "7a00000000000001")}); err != nil {
+ t.Fatal(err)
+ }
+ exceptionOn := func(project uuid.UUID, trace, span string) models.ExceptionStackTrace {
+ return models.ExceptionStackTrace{Id: uuid.New(), ProjectId: project, TraceId: trace, SpanId: span, RecordedAt: at}
+ }
+ for name, test := range map[string]struct {
+ exception models.ExceptionStackTrace
+ kind string
+ owner string
+ }{
+ "on the endpoint's own span": {exceptionOn(payments, backendTrace, "e200000000000002"), "endpoint", "POST /charge"},
+ "two spans below the endpoint": {exceptionOn(payments, backendTrace, "5c00000000000001"), "endpoint", "POST /charge"},
+ "below the task, the nearest owner": {exceptionOn(payments, backendTrace, "7b00000000000001"), "task", "settle"},
+ "on a span that was never stored": {exceptionOn(payments, backendTrace, "ffffffffffffffff"), "", ""},
+ "in a project with no such trace": {exceptionOn(stranger, otherTrace, "e300000000000003"), "", ""},
+ "in the browser, linked but no span": {exceptionOn(gateway, browserTrace, "b000000000000001"), "", ""},
+ "without a trace": {exceptionOn(payments, "", ""), "", ""},
+ } {
+ owner, err := FindExceptionOwner(ctx, test.exception)
+ if err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ if test.kind == "" {
+ if owner != nil {
+ t.Fatalf("%s: expected no owner, got %+v", name, owner)
+ }
+ continue
+ }
+ if owner == nil || owner.TraceType != test.kind || owner.Name != test.owner {
+ t.Fatalf("%s: got %+v", name, owner)
+ }
+ }
+}
diff --git a/backend/app/repositories/transactional/move_over.repository.go b/backend/app/repositories/transactional/move_over.repository.go
new file mode 100644
index 00000000..a577ebb2
--- /dev/null
+++ b/backend/app/repositories/transactional/move_over.repository.go
@@ -0,0 +1,59 @@
+package transactional
+
+import (
+ "context"
+ "database/sql"
+ "time"
+
+ "github.com/tracewayapp/lit/v2"
+ "github.com/tracewayapp/traceway/backend/app/db"
+)
+
+const (
+ MoveOverStarted = "started"
+ MoveOverDone = "done"
+)
+
+type MoveOverDay struct {
+ Table string
+ Day string
+ State string
+ Rows int64
+}
+
+type moveOverRepository struct{}
+
+func (moveOverRepository) FindProgress(ctx context.Context, tx *sql.Tx) ([]MoveOverDay, error) {
+ rows, err := tx.QueryContext(ctx, "SELECT source_table, day, state, moved_rows FROM v2_move_over_days ORDER BY day DESC, source_table")
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+ var days []MoveOverDay
+ for rows.Next() {
+ var day MoveOverDay
+ if err := rows.Scan(&day.Table, &day.Day, &day.State, &day.Rows); err != nil {
+ return nil, err
+ }
+ days = append(days, day)
+ }
+ return days, rows.Err()
+}
+
+func (moveOverRepository) SaveProgress(ctx context.Context, tx *sql.Tx, day MoveOverDay) error {
+ query, args, err := lit.ParseNamedQuery(db.Driver, `
+ INSERT INTO v2_move_over_days (source_table, day, state, moved_rows, updated_at)
+ VALUES (:table, :day, :state, :rows, :updated_at)
+ ON CONFLICT (source_table, day) DO UPDATE
+ SET state = excluded.state, moved_rows = excluded.moved_rows, updated_at = excluded.updated_at`, lit.P{
+ "table": day.Table, "day": day.Day, "state": day.State, "rows": day.Rows,
+ "updated_at": time.Now().UTC().Format(time.RFC3339Nano),
+ })
+ if err != nil {
+ return err
+ }
+ _, err = tx.ExecContext(ctx, query, args...)
+ return err
+}
+
+var MoveOverRepository = moveOverRepository{}
diff --git a/backend/app/retention/move_over.go b/backend/app/retention/move_over.go
new file mode 100644
index 00000000..010ceda4
--- /dev/null
+++ b/backend/app/retention/move_over.go
@@ -0,0 +1,66 @@
+package retention
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "log"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/config"
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+ traceway "go.tracewayapp.com"
+)
+
+const moveOverInterval = time.Hour
+
+// Opt in on one instance: copying history competes with live ingest for telemetry I/O.
+func startMoveOver(ctx context.Context, telemetryRetentionDays int) {
+ cfg := config.Config
+ enabled, err := strconv.ParseBool(strings.TrimSpace(cfg.V2MoveOver))
+ if err != nil || !enabled {
+ return
+ }
+ options := telemetry.MoveOverOptions{}
+ if oldest := strings.TrimSpace(cfg.V2MoveOverOldest); oldest != "" {
+ if options.Oldest, err = time.Parse(time.DateOnly, oldest); err != nil {
+ log.Printf("[tracewaybackend] move-over: V2_MOVE_OVER_OLDEST %q is not a YYYY-MM-DD date, not starting", oldest)
+ return
+ }
+ }
+ go func() {
+ defer traceway.Recover()
+ runMoveOverSchedule(ctx, moveOverInterval, func() {
+ current := moveOverOptionsForRun(options, telemetryRetentionDays, time.Now())
+ if err := telemetry.RunMoveOver(ctx, current); err != nil && !errors.Is(err, context.Canceled) {
+ log.Printf("[tracewaybackend] move-over: stopped, retrying in %s: %v", moveOverInterval, err)
+ traceway.CaptureException(fmt.Errorf("V2 move-over stopped: %w", err))
+ }
+ })
+ }()
+}
+
+func moveOverOptionsForRun(options telemetry.MoveOverOptions, retentionDays int, now time.Time) telemetry.MoveOverOptions {
+ // Recompute for every pass: a failed migration can outlive the retention window it started with.
+ if retentionDays > 0 {
+ if kept := now.UTC().AddDate(0, 0, -retentionDays); kept.After(options.Oldest) {
+ options.Oldest = kept
+ }
+ }
+ return options
+}
+
+func runMoveOverSchedule(ctx context.Context, interval time.Duration, run func()) {
+ for ctx.Err() == nil {
+ run()
+ timer := time.NewTimer(interval)
+ select {
+ case <-ctx.Done():
+ timer.Stop()
+ return
+ case <-timer.C:
+ }
+ }
+}
diff --git a/backend/app/retention/move_over_test.go b/backend/app/retention/move_over_test.go
new file mode 100644
index 00000000..1ec2753e
--- /dev/null
+++ b/backend/app/retention/move_over_test.go
@@ -0,0 +1,73 @@
+package retention
+
+import (
+ "context"
+ "testing"
+ "testing/synctest"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
+)
+
+func TestMoveOverScheduleWaitsAfterPassAndStopsOnCancellation(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ starts := make(chan time.Time, 8)
+ start := time.Now()
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ runMoveOverSchedule(ctx, time.Hour, func() {
+ starts <- time.Now()
+ time.Sleep(2 * time.Hour)
+ })
+ }()
+ synctest.Wait()
+ if len(starts) != 1 {
+ t.Fatalf("first pass must start immediately: %d passes", len(starts))
+ }
+ if got := <-starts; !got.Equal(start) {
+ t.Fatalf("first pass started at %v, want %v", got, start)
+ }
+ time.Sleep(3 * time.Hour)
+ synctest.Wait()
+ if len(starts) != 1 {
+ t.Fatalf("passes must not overlap or queue while the previous pass runs: %d new passes", len(starts))
+ }
+ if got := <-starts; !got.Equal(start.Add(3 * time.Hour)) {
+ t.Fatalf("second pass started at %v, want %v", got, start.Add(3*time.Hour))
+ }
+ cancel()
+ <-done
+ time.Sleep(2 * time.Hour)
+ if len(starts) != 0 {
+ t.Fatalf("cancelled worker ran again: %d passes", len(starts))
+ }
+ })
+}
+
+func TestMoveOverScheduleDoesNotStartWhenCancelled(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ runMoveOverSchedule(ctx, time.Hour, func() { t.Fatal("cancelled worker started") })
+}
+
+func TestMoveOverRecomputesRetentionForEachPass(t *testing.T) {
+ now := time.Date(2026, 9, 20, 1, 0, 0, 0, time.FixedZone("UTC+2", 2*60*60))
+ options := telemetry.MoveOverOptions{Oldest: now.AddDate(0, 0, -90), PageSize: 25}
+ for _, elapsed := range []int{0, 10} {
+ current := now.AddDate(0, 0, elapsed)
+ got := moveOverOptionsForRun(options, 30, current)
+ if !got.Oldest.Equal(current.UTC().AddDate(0, 0, -30)) || got.Oldest.Location() != time.UTC || got.PageSize != 25 {
+ t.Fatalf("retention window at day %d: %+v", elapsed, got)
+ }
+ }
+ if got := moveOverOptionsForRun(options, 0, now); !got.Oldest.Equal(options.Oldest) {
+ t.Fatal("disabled retention must preserve the operator's lower bound")
+ }
+ options.Oldest = now.AddDate(0, 0, -2)
+ if got := moveOverOptionsForRun(options, 30, now); !got.Oldest.Equal(options.Oldest) {
+ t.Fatal("a newer operator lower bound must win")
+ }
+}
diff --git a/backend/app/retention/retention.go b/backend/app/retention/retention.go
index c361f732..2056e671 100644
--- a/backend/app/retention/retention.go
+++ b/backend/app/retention/retention.go
@@ -27,6 +27,11 @@ func Start(ctx context.Context) {
startOAuthSessionsPrune(ctx)
startAuthTokensPrune(ctx)
startOutboxPrune(ctx)
+ moveOverWindow := 0
+ if db.IsSQLite() {
+ moveOverWindow = telemetryDays
+ }
+ startMoveOver(ctx, moveOverWindow)
}
func parseRetentionDays(value string) int {
diff --git a/backend/app/retention/sqlite.go b/backend/app/retention/sqlite.go
index 2935fbee..67c0a0c7 100644
--- a/backend/app/retention/sqlite.go
+++ b/backend/app/retention/sqlite.go
@@ -15,6 +15,12 @@ var telemetryRetentionTargets = []struct {
table string
column string
}{
+ {"endpoints_v2", "recorded_at"},
+ {"tasks_v2", "recorded_at"},
+ {"exceptions_v2", "recorded_at"},
+ {"spans_v2", "recorded_at"},
+ {"ai_traces_v2", "recorded_at"},
+ // The tables V2 replaced. They hold history until the move-over script has run and they are dropped.
{"endpoints", "recorded_at"},
{"tasks", "recorded_at"},
{"exception_stack_traces", "recorded_at"},
diff --git a/backend/app/services/healthcheck.service.go b/backend/app/services/healthcheck.service.go
index 631798f9..19c4c746 100644
--- a/backend/app/services/healthcheck.service.go
+++ b/backend/app/services/healthcheck.service.go
@@ -3,7 +3,6 @@ package services
import (
"strings"
- "github.com/google/uuid"
"github.com/tracewayapp/traceway/backend/app/models"
)
@@ -83,41 +82,66 @@ func matchesCustomPath(path, pattern string) bool {
return path == pattern
}
-func FilterHealthchecks(project *models.Project, endpoints []models.Endpoint, spans []models.Span, exceptions []models.ExceptionStackTrace) ([]models.Endpoint, []models.Span, int) {
+// SpanKey names one span of one trace inside a payload.
+func SpanKey(traceId, spanId string) string { return traceId + ":" + spanId }
+
+// FilterHealthchecks drops healthy healthcheck endpoints and returns the spans they were promoted from, keyed by
+// SpanKey. A healthcheck whose trace carries an exception in the same payload is kept.
+func FilterHealthchecks(project *models.Project, endpoints []models.Endpoint, exceptions []models.ExceptionStackTrace) ([]models.Endpoint, map[string]bool) {
if project == nil || !project.DropHealthyHealthchecks || len(endpoints) == 0 {
- return endpoints, spans, 0
+ return endpoints, nil
}
-
- dropped := map[uuid.UUID]bool{}
- for _, e := range endpoints {
- if ShouldDropHealthcheck(project, e.Endpoint, e.StatusCode) {
- dropped[e.Id] = true
+ failing := map[string]bool{}
+ for _, exception := range exceptions {
+ if exception.TraceId != "" {
+ failing[exception.TraceId] = true
}
}
- if len(dropped) == 0 {
- return endpoints, spans, 0
- }
-
- for _, exc := range exceptions {
- if exc.TraceId != nil {
- delete(dropped, *exc.TraceId)
+ dropped := map[string]bool{}
+ kept := endpoints[:0]
+ for _, endpoint := range endpoints {
+ if ShouldDropHealthcheck(project, endpoint.Endpoint, endpoint.StatusCode) && !failing[endpoint.TraceId] {
+ dropped[SpanKey(endpoint.TraceId, endpoint.SpanId)] = true
+ continue
}
+ kept = append(kept, endpoint)
}
+ return kept, dropped
+}
+
+// DropSpanSubtrees removes the dropped spans and everything under them, stopping at a retained span: a task or an AI
+// trace started by a healthcheck keeps its own spans.
+func DropSpanSubtrees[T any](spans []T, ids func(T) (traceId, spanId, parentSpanId string), dropped, retained map[string]bool) []T {
if len(dropped) == 0 {
- return endpoints, spans, 0
+ return spans
+ }
+ children := make(map[string][]int)
+ pending := make([]int, 0)
+ for i, span := range spans {
+ traceId, spanId, parentSpanId := ids(span)
+ if dropped[SpanKey(traceId, spanId)] {
+ pending = append(pending, i)
+ }
+ if parentSpanId != "" {
+ children[SpanKey(traceId, parentSpanId)] = append(children[SpanKey(traceId, parentSpanId)], i)
+ }
}
-
- keptEndpoints := endpoints[:0]
- for _, e := range endpoints {
- if !dropped[e.Id] {
- keptEndpoints = append(keptEndpoints, e)
+ removed := make(map[int]bool)
+ for len(pending) > 0 {
+ i := pending[len(pending)-1]
+ pending = pending[:len(pending)-1]
+ traceId, spanId, _ := ids(spans[i])
+ if removed[i] || retained[SpanKey(traceId, spanId)] {
+ continue
}
+ removed[i] = true
+ pending = append(pending, children[SpanKey(traceId, spanId)]...)
}
- keptSpans := spans[:0]
- for _, s := range spans {
- if !dropped[s.TraceId] {
- keptSpans = append(keptSpans, s)
+ kept := spans[:0]
+ for i, span := range spans {
+ if !removed[i] {
+ kept = append(kept, span)
}
}
- return keptEndpoints, keptSpans, len(dropped)
+ return kept
}
diff --git a/backend/app/services/healthcheck.service_test.go b/backend/app/services/healthcheck.service_test.go
index 49b7496d..111fe74b 100644
--- a/backend/app/services/healthcheck.service_test.go
+++ b/backend/app/services/healthcheck.service_test.go
@@ -75,46 +75,50 @@ func TestShouldDropHealthcheck(t *testing.T) {
func TestFilterHealthchecks(t *testing.T) {
project := &models.Project{DropHealthyHealthchecks: true}
- healthyId := uuid.New()
- failingId := uuid.New()
- regularId := uuid.New()
- excId := uuid.New()
-
endpoints := []models.Endpoint{
- {Id: healthyId, Endpoint: "GET /health", StatusCode: 200},
- {Id: failingId, Endpoint: "GET /health", StatusCode: 503},
- {Id: regularId, Endpoint: "GET /api/users", StatusCode: 200},
- {Id: excId, Endpoint: "GET /healthz", StatusCode: 200},
- }
- spans := []models.Span{
- {Id: uuid.New(), TraceId: healthyId},
- {Id: uuid.New(), TraceId: regularId},
- {Id: uuid.New(), TraceId: excId},
+ {Id: uuid.New(), TraceId: "t-healthy", SpanId: "e1", Endpoint: "GET /health", StatusCode: 200},
+ {Id: uuid.New(), TraceId: "t-failing", SpanId: "e2", Endpoint: "GET /health", StatusCode: 503},
+ {Id: uuid.New(), TraceId: "t-regular", SpanId: "e3", Endpoint: "GET /api/users", StatusCode: 200},
+ {Id: uuid.New(), TraceId: "t-exception", SpanId: "e4", Endpoint: "GET /healthz", StatusCode: 200},
}
exceptions := []models.ExceptionStackTrace{
- {Id: uuid.New(), TraceId: &excId},
+ {Id: uuid.New(), TraceId: "t-exception", SpanId: "e4-child"},
}
- keptEndpoints, keptSpans, dropped := FilterHealthchecks(project, endpoints, spans, exceptions)
+ kept, dropped := FilterHealthchecks(project, endpoints, exceptions)
- if dropped != 1 {
- t.Errorf("dropped = %d, expected 1", dropped)
+ if len(dropped) != 1 || !dropped[SpanKey("t-healthy", "e1")] {
+ t.Errorf("dropped = %v, expected only the healthy healthcheck", dropped)
}
- if len(keptEndpoints) != 3 {
- t.Fatalf("len(keptEndpoints) = %d, expected 3", len(keptEndpoints))
+ if len(kept) != 3 {
+ t.Fatalf("len(kept) = %d, expected 3", len(kept))
}
- for _, e := range keptEndpoints {
- if e.Id == healthyId {
+ for _, e := range kept {
+ if e.TraceId == "t-healthy" {
t.Errorf("healthy healthcheck endpoint was not dropped")
}
}
- if len(keptSpans) != 2 {
- t.Fatalf("len(keptSpans) = %d, expected 2", len(keptSpans))
+}
+
+func TestDropSpanSubtrees(t *testing.T) {
+ spans := []models.Span{
+ {TraceId: "t-healthy", SpanId: "e1"},
+ {TraceId: "t-healthy", SpanId: "db", ParentSpanId: "e1"},
+ {TraceId: "t-healthy", SpanId: "row", ParentSpanId: "db"},
+ {TraceId: "t-healthy", SpanId: "job", ParentSpanId: "e1"},
+ {TraceId: "t-healthy", SpanId: "job-child", ParentSpanId: "job"},
+ {TraceId: "t-regular", SpanId: "e1"},
+ }
+ ids := func(span models.Span) (string, string, string) { return span.TraceId, span.SpanId, span.ParentSpanId }
+
+ kept := DropSpanSubtrees(spans, ids, map[string]bool{SpanKey("t-healthy", "e1"): true}, map[string]bool{SpanKey("t-healthy", "job"): true})
+
+ got := ""
+ for _, span := range kept {
+ got += span.TraceId + "/" + span.SpanId + " "
}
- for _, s := range keptSpans {
- if s.TraceId == healthyId {
- t.Errorf("span of dropped healthcheck was not dropped")
- }
+ if got != "t-healthy/job t-healthy/job-child t-regular/e1 " {
+ t.Fatalf("a dropped healthcheck takes its own spans and stops at a task it started, in its own trace only: %s", got)
}
}
@@ -124,9 +128,9 @@ func TestFilterHealthchecksDisabled(t *testing.T) {
{Id: uuid.New(), Endpoint: "GET /health", StatusCode: 200},
}
- keptEndpoints, _, dropped := FilterHealthchecks(project, endpoints, nil, nil)
+ kept, dropped := FilterHealthchecks(project, endpoints, nil)
- if dropped != 0 || len(keptEndpoints) != 1 {
- t.Errorf("disabled filter dropped endpoints: kept=%d dropped=%d", len(keptEndpoints), dropped)
+ if len(dropped) != 0 || len(kept) != 1 {
+ t.Errorf("disabled filter dropped endpoints: kept=%d dropped=%d", len(kept), len(dropped))
}
}
diff --git a/cli/.claude/skills/integration-test/SKILL.md b/cli/.claude/skills/integration-test/SKILL.md
index 78a1acd9..977836dd 100644
--- a/cli/.claude/skills/integration-test/SKILL.md
+++ b/cli/.claude/skills/integration-test/SKILL.md
@@ -122,8 +122,8 @@ These all **require** a timestamp flag; capture the id and its `recordedAt` toge
OCC=$(./bin/traceway exceptions show "$HASH" --output json | jq -c '.occurrences[0]')
OID=$(jq -r '.id' <<<"$OCC")
OTS=$(jq -r '.recordedAt' <<<"$OCC")
- DT=$(jq -r '.distributedTraceId // empty' <<<"$OCC")
- SID=$(jq -r '.sessionId // empty' <<<"$OCC")
+ DT=$(jq -r '.traceId // empty' <<<"$OCC")
+ SID=$(jq -r '.sessionId // empty' <<<"$OCC")
```
2. `exceptions occurrence $OID --recorded-at $OTS` — assert exit 0 and `.exception.id == $OID`.
3. **Required-flag enforcement** (no live data needed): `exceptions occurrence $OID` with no `--recorded-at` → exit 2 `usage_error`; `--recorded-at notadate` → exit 2 `invalid_timestamp`; `endpoints show not-a-uuid --recorded-at $OTS` → exit 2 `usage_error`.
diff --git a/cli/.claude/skills/traceway-cli.md b/cli/.claude/skills/traceway-cli.md
index eefa6c27..c97b884a 100644
--- a/cli/.claude/skills/traceway-cli.md
+++ b/cli/.claude/skills/traceway-cli.md
@@ -115,7 +115,7 @@ Result shape (JSON):
traceway exceptions show <exceptionHash> [--page 1] [--page-size 20]
```
-Returns the exception group plus an array of recent `occurrences`. Each occurrence has at least `recordedAt`, `attributes`, and optional `distributedTraceId` / `sessionId` (verify the exact shape against your build — schema is still settling). A bogus hash exits with code `5` and `not_found`.
+Returns the exception group plus an array of recent `occurrences`. Each occurrence has at least `recordedAt`, `attributes`, and optional `traceId` / `spanId` / `sessionId`. `traceId` is the OpenTelemetry trace id as 32 hex characters, the same id the request's logs carry. The response also carries `relatedEntity` (`{traceType, id, name, statusCode, duration, recordedAt, traceId}`): the endpoint, task or AI trace the newest occurrence happened in. A bogus hash exits with code `5` and `not_found`.
```
traceway exceptions occurrence <exceptionId> --recorded-at <RFC3339>
@@ -160,7 +160,7 @@ Result shape: `{data: [...] | null, pagination: {page, pageSize, total, totalPag
Recipe — pull logs around an exception:
```bash
-traceway exceptions show $HASH --output json | jq -r '.occurrences[0].distributedTraceId' \
+traceway exceptions show $HASH --output json | jq -r '.occurrences[0].traceId' \
| xargs -I{} traceway logs query --trace-id {} --page 1 --output json
```
@@ -238,12 +238,12 @@ These resources currently expose only a by-id `show` (no `list` verb). The id co
```
traceway tasks show <taskId> --recorded-at <RFC3339> # {task, spans, hasSpans, exception?, messages}
-traceway ai-traces show <traceId> --recorded-at <RFC3339> # {aiTrace, conversation?} (token/cost stats + stored conversation)
+traceway ai-traces show <aiTraceId> --recorded-at <RFC3339> # {aiTrace, spans, conversation?} (token/cost stats + stored conversation)
traceway sessions show <sessionId> --started-at <RFC3339> # {session, exceptions} (uses --started-at, not --recorded-at)
-traceway traces show <distributedTraceId> --recorded-at <RFC3339> # {distributedTraceId, nodes:[...]}
+traceway traces show <traceId> --recorded-at <RFC3339> # {traceId, nodes:[...]}
```
-`traces show` is the cross-service waterfall: every endpoint/task/ai-trace/exception node sharing a `distributedTraceId`, across all projects you can see (the route scopes by your JWT, so it ignores the active project). It's the highest-value RCA call — feed it an occurrence's `distributedTraceId` plus that occurrence's `recordedAt`.
+`traces show` is the cross-service waterfall: every endpoint/task/ai-trace/exception node sharing a trace id, across all projects you can see (the route scopes by your JWT, so it ignores the active project). It's the highest-value RCA call: feed it an occurrence's `traceId` (32 hex characters; a dashed UUID works too) plus that occurrence's `recordedAt`. Each node carries `traceId`, `spanId` and `parentEntitySpanId`, the `spanId` of the node it sits under.
`sessions show` takes `--started-at` because the sessions table is partitioned on `started_at`. The session URL carries no `t=`; use the session's start, the URL's `from=`, or a linked occurrence's `recordedAt` (it falls inside the ±24h window).
@@ -295,7 +295,7 @@ traceway logs query --since 15m --service api --min-severity 13 \
--page 1 --output json | jq '.data[]? | {timestamp, body, traceId}'
```
-Once you have a `distributedTraceId` from an occurrence or endpoint/task, pull the cross-service waterfall directly: `traceway traces show <distributedTraceId> --recorded-at <that node's recordedAt>` (see the by-id detail section above). It stitches every endpoint/task/ai-trace/exception node sharing the id into one timeline — usually the highest-value RCA call.
+Once you have a `traceId` from an occurrence or endpoint/task, pull the cross-service waterfall directly: `traceway traces show <traceId> --recorded-at <that node's recordedAt>` (see the by-id detail section above). It stitches every endpoint/task/ai-trace/exception node sharing the id into one timeline, usually the highest-value RCA call.
### "Show me errors for service X in the last hour"
diff --git a/cli/cmd/traceway/aitraces.go b/cli/cmd/traceway/aitraces.go
index a74b5601..4170ee61 100644
--- a/cli/cmd/traceway/aitraces.go
+++ b/cli/cmd/traceway/aitraces.go
@@ -20,10 +20,11 @@ func newAiTracesCmd() *cobra.Command {
func newAiTracesShowCmd() *cobra.Command {
cmd := &cobra.Command{
- Use: "show <traceId>",
+ Use: "show <aiTraceId>",
Short: "Show one AI trace by id with its conversation",
Long: `Show a single AI/LLM trace by its UUID, including token/cost stats and the
-stored conversation. This is the detail behind /ai-traces/<name>/<traceId>.
+stored conversation. This is the detail behind /ai-traces/<name>/<id>. The UUID
+is the AI trace's own id, not the 32 hex traceId that "traces show" takes.
--recorded-at is REQUIRED. The ai_traces table is daily-partitioned; the
timestamp bounds the lookup to a window around it so ClickHouse prunes
@@ -45,7 +46,7 @@ func runAiTracesShow(cmd *cobra.Command, args []string) error {
if err != nil {
return renderSessionError(cmd.ErrOrStderr(), mode, err)
}
- if err := validateUUIDArg(cmd, mode, args[0], "trace id"); err != nil {
+ if err := validateUUIDArg(cmd, mode, args[0], "ai trace id"); err != nil {
return err
}
recordedAt, err := resolveTimestamp(cmd, "recorded-at")
@@ -76,9 +77,7 @@ func runAiTracesShow(cmd *cobra.Command, args []string) error {
a.InputTokens, a.OutputTokens, a.TotalTokens,
a.TotalCost, formatDuration(a.Duration),
)
- if a.DistributedTraceId != nil {
- _, _ = fmt.Fprintf(out, "TRACE ID: %s\n", a.DistributedTraceId.String())
- }
+ renderTraceId(out, a.TraceId)
if a.ConversationId != "" {
_, _ = fmt.Fprintf(out, "CONVERSATION: %s\n", a.ConversationId)
}
diff --git a/cli/cmd/traceway/detail_test.go b/cli/cmd/traceway/detail_test.go
index 6e68b3d1..61a9de92 100644
--- a/cli/cmd/traceway/detail_test.go
+++ b/cli/cmd/traceway/detail_test.go
@@ -115,17 +115,41 @@ func TestExceptionsOccurrence_postsRecordedAt(t *testing.T) {
}
}
+func TestExceptionsOccurrence_rendersTraceAndRelatedEntity(t *testing.T) {
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ _, _ = w.Write([]byte(`{"exception":{"id":"00000000-0000-0000-0000-000000000001","exceptionHash":"abc","stackTrace":"boom","recordedAt":"2026-06-23T14:30:00Z","traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"00f067aa0ba902b7"},"relatedEntity":{"traceType":"endpoint","id":"00000000-0000-0000-0000-0000000000d1","name":"GET /x","statusCode":500,"duration":1000000,"recordedAt":"2026-06-23T14:29:59.5Z","traceId":"0af7651916cd43dd8448eb211c80319c"}}`))
+ }))
+ defer srv.Close()
+ seedSessionFor(t, srv.URL)
+
+ stdout, _, err := runCmd(t, "", "exceptions", "occurrence",
+ "00000000-0000-0000-0000-000000000001",
+ "--recorded-at", "2026-06-23T14:30:00Z", "--output", "table")
+ if err != nil {
+ t.Fatalf("exceptions occurrence: %v", err)
+ }
+ out := stdout.String()
+ for _, want := range []string{
+ "TRACE ID: 0af7651916cd43dd8448eb211c80319c",
+ "RELATED: endpoint GET /x (00000000-0000-0000-0000-0000000000d1, recorded 2026-06-23T14:29:59.5Z)",
+ } {
+ if !strings.Contains(out, want) {
+ t.Errorf("missing %q in:\n%s", want, out)
+ }
+ }
+}
+
func TestTracesShow_rendersNodesAndNoProjectId(t *testing.T) {
var gotRawQuery string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotRawQuery = r.URL.RawQuery
- _, _ = w.Write([]byte(`{"distributedTraceId":"00000000-0000-0000-0000-000000000006","nodes":[{"projectName":"api","traceType":"endpoint","endpoint":{"id":"00000000-0000-0000-0000-0000000000d1","endpoint":"GET /x"},"spans":[]}]}`))
+ _, _ = w.Write([]byte(`{"traceId":"0af7651916cd43dd8448eb211c80319c","nodes":[{"projectName":"api","traceType":"endpoint","traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"b7ad6b7169203331","endpoint":{"id":"00000000-0000-0000-0000-0000000000d1","endpoint":"GET /x","traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"b7ad6b7169203331"},"spans":[],"spanGraphStatus":{"state":"partial","reasons":["row_limit"]}}]}`))
}))
defer srv.Close()
seedSessionFor(t, srv.URL)
stdout, _, err := runCmd(t, "", "traces", "show",
- "00000000-0000-0000-0000-000000000006",
+ "0af7651916cd43dd8448eb211c80319c",
"--recorded-at", "2026-06-23T14:30:00Z", "--output", "table")
if err != nil {
t.Fatalf("traces show: %v", err)
@@ -134,8 +158,27 @@ func TestTracesShow_rendersNodesAndNoProjectId(t *testing.T) {
t.Errorf("traces show must not send a query string, got %q", gotRawQuery)
}
out := stdout.String()
- if !strings.Contains(out, "GET /x") || !strings.Contains(out, "endpoint") {
- t.Errorf("expected node row in table: %s", out)
+ if !strings.Contains(out, "GRAPH") || !strings.Contains(out, "partial") || !strings.Contains(out, "GET /x") || !strings.Contains(out, "endpoint") || !strings.Contains(out, "0af7651916cd43dd8448eb211c80319c") {
+ t.Errorf("expected trace id and node row in table: %s", out)
+ }
+}
+
+func TestTracesShow_acceptsHexOrUUIDTraceId(t *testing.T) {
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ _, _ = w.Write([]byte(`{"traceId":"0af7651916cd43dd8448eb211c80319c","nodes":[]}`))
+ }))
+ defer srv.Close()
+ seedSessionFor(t, srv.URL)
+
+ for _, id := range []string{"0af7651916cd43dd8448eb211c80319c", "0af76519-16cd-43dd-8448-eb211c80319c"} {
+ if _, _, err := runCmd(t, "", "traces", "show", id, "--recorded-at", "2026-06-23T14:30:00Z"); err != nil {
+ t.Errorf("trace id %q must be accepted: %v", id, err)
+ }
+ }
+ for _, id := range []string{"b7ad6b7169203331", "not-a-trace-id"} {
+ if _, _, err := runCmd(t, "", "traces", "show", id, "--recorded-at", "2026-06-23T14:30:00Z"); err == nil {
+ t.Errorf("trace id %q must be rejected", id)
+ }
}
}
diff --git a/cli/cmd/traceway/endpoints.go b/cli/cmd/traceway/endpoints.go
index ee23a638..d641b6d7 100644
--- a/cli/cmd/traceway/endpoints.go
+++ b/cli/cmd/traceway/endpoints.go
@@ -322,9 +322,7 @@ func runEndpointsShow(cmd *cobra.Command, args []string) error {
formatDuration(e.Duration), e.StatusCode,
pickStr(e.ServerName, "-"), pickStr(e.AppVersion, "-"),
)
- if e.DistributedTraceId != nil {
- _, _ = fmt.Fprintf(out, "TRACE ID: %s\n", e.DistributedTraceId.String())
- }
+ renderTraceId(out, e.TraceId)
}
renderSpansTable(out, resp.Spans)
renderLinkedErrors(out, resp.Exception, resp.Messages)
@@ -332,6 +330,12 @@ func runEndpointsShow(cmd *cobra.Command, args []string) error {
}
}
+func renderTraceId(out io.Writer, traceId string) {
+ if traceId != "" {
+ _, _ = fmt.Fprintf(out, "TRACE ID: %s\n", traceId)
+ }
+}
+
// renderSpansTable prints a span waterfall (name + duration), or a count line
// when there are none. Shared by endpoints show and tasks show.
func renderSpansTable(out io.Writer, spans []client.Span) {
diff --git a/cli/cmd/traceway/exceptions.go b/cli/cmd/traceway/exceptions.go
index 98bad9ea..bb616235 100644
--- a/cli/cmd/traceway/exceptions.go
+++ b/cli/cmd/traceway/exceptions.go
@@ -3,6 +3,7 @@ package main
import (
"context"
"fmt"
+ "time"
"github.com/spf13/cobra"
@@ -152,10 +153,17 @@ func runExceptionsShow(cmd *cobra.Command, args []string) error {
// Group header, then occurrences table.
if resp.Group != nil {
_, _ = fmt.Fprintf(cmd.OutOrStdout(),
- "HASH: %s\nCOUNT: %d\nFIRST SEEN: %s\nLAST SEEN: %s\n\nSTACK TRACE:\n%s\n\nOCCURRENCES (%d):\n",
+ "HASH: %s\nCOUNT: %d\nFIRST SEEN: %s\nLAST SEEN: %s\n",
resp.Group.ExceptionHash, resp.Group.Count,
resp.Group.FirstSeen.Format("2006-01-02 15:04:05"),
resp.Group.LastSeen.Format("2006-01-02 15:04:05"),
+ )
+ if related := resp.RelatedEntity; related != nil {
+ _, _ = fmt.Fprintf(cmd.OutOrStdout(), "RELATED: %s %s (%s, recorded %s)\n",
+ related.TraceType, related.Name, related.Id,
+ related.RecordedAt.UTC().Format(time.RFC3339Nano))
+ }
+ _, _ = fmt.Fprintf(cmd.OutOrStdout(), "\nSTACK TRACE:\n%s\n\nOCCURRENCES (%d):\n",
resp.Group.StackTrace,
len(resp.Occurrences),
)
@@ -243,8 +251,11 @@ func runExceptionsOccurrence(cmd *cobra.Command, args []string) error {
pickStr(occ.AppVersion, "-"),
pickStr(occ.TraceType, "-"),
)
- if occ.DistributedTraceId != nil {
- _, _ = fmt.Fprintf(out, "TRACE ID: %s\n", occ.DistributedTraceId.String())
+ renderTraceId(out, occ.TraceId)
+ if related := resp.RelatedEntity; related != nil {
+ _, _ = fmt.Fprintf(out, "RELATED: %s %s (%s, recorded %s)\n",
+ related.TraceType, related.Name, related.Id,
+ related.RecordedAt.UTC().Format(time.RFC3339Nano))
}
sid := resp.SessionId
if sid == nil {
diff --git a/cli/cmd/traceway/querycommon.go b/cli/cmd/traceway/querycommon.go
index 96551b54..9078355c 100644
--- a/cli/cmd/traceway/querycommon.go
+++ b/cli/cmd/traceway/querycommon.go
@@ -2,6 +2,7 @@ package main
import (
"bufio"
+ "encoding/hex"
"errors"
"fmt"
"io"
@@ -95,6 +96,17 @@ func validateUUIDArg(cmd *cobra.Command, mode output.Mode, id, label string) err
return nil
}
+// validateTraceIDArg checks that id is a trace id: 32 hex characters, which is
+// how OpenTelemetry writes one, or the same bytes as a dashed UUID.
+func validateTraceIDArg(cmd *cobra.Command, mode output.Mode, id string) error {
+ decoded, err := hex.DecodeString(strings.ReplaceAll(id, "-", ""))
+ if err != nil || len(decoded) != 16 {
+ return renderUsageError(cmd.ErrOrStderr(), mode,
+ fmt.Sprintf("invalid trace id %q: must be 32 hex characters or a UUID", id), "")
+ }
+ return nil
+}
+
// renderUsageError writes a usage_error envelope and returns a *cliError
// with exitcode.Usage. Used when a command receives invalid flag combinations
// or values that aren't covered by other validators.
diff --git a/cli/cmd/traceway/tasks.go b/cli/cmd/traceway/tasks.go
index ec3dd777..eb0bb4da 100644
--- a/cli/cmd/traceway/tasks.go
+++ b/cli/cmd/traceway/tasks.go
@@ -73,9 +73,7 @@ func runTasksShow(cmd *cobra.Command, args []string) error {
formatDuration(t.Duration),
pickStr(t.ServerName, "-"), pickStr(t.AppVersion, "-"),
)
- if t.DistributedTraceId != nil {
- _, _ = fmt.Fprintf(out, "TRACE ID: %s\n", t.DistributedTraceId.String())
- }
+ renderTraceId(out, t.TraceId)
}
renderSpansTable(out, resp.Spans)
renderLinkedErrors(out, resp.Exception, resp.Messages)
diff --git a/cli/cmd/traceway/traces.go b/cli/cmd/traceway/traces.go
index 754371d9..32551d8e 100644
--- a/cli/cmd/traceway/traces.go
+++ b/cli/cmd/traceway/traces.go
@@ -19,16 +19,16 @@ func newTracesCmd() *cobra.Command {
func newTracesShowCmd() *cobra.Command {
cmd := &cobra.Command{
- Use: "show <distributedTraceId>",
- Short: "Show a distributed trace: every service node that shares the trace id",
- Long: `Show the full cross-service timeline for a distributed trace id — every
-endpoint, task, ai-trace, and exception node that shares it, across all projects
-you can see. This is the highest-value root-cause view: it stitches one logical
-request together end to end.
+ Use: "show <traceId>",
+ Short: "Show the available service nodes for a trace id",
+ Long: `Show the available cross-service timeline for a trace id: endpoint, task,
+ai-trace, and exception nodes across the projects you can see. Time windows,
+retention and read limits can make the view incomplete. The GRAPH column reports
+partial or unavailable span graphs; exceptions on missing spans may be absent.
-The distributed trace id comes from an occurrence's distributedTraceId (in
-"exceptions show" / "exceptions occurrence" output) or an endpoint/task's
-distributedTraceId.
+The trace id is the traceId of an occurrence (in "exceptions show" /
+"exceptions occurrence" output) or of an endpoint, task or ai-trace: 32 hex
+characters, the OpenTelemetry trace id shared by all participating services.
--recorded-at is REQUIRED. The lookup is bounded to a window around it so
ClickHouse prunes partitions instead of scanning all of them. Use the recordedAt
@@ -48,7 +48,7 @@ func runTracesShow(cmd *cobra.Command, args []string) error {
if err != nil {
return renderSessionError(cmd.ErrOrStderr(), mode, err)
}
- if err := validateUUIDArg(cmd, mode, args[0], "distributed trace id"); err != nil {
+ if err := validateTraceIDArg(cmd, mode, args[0]); err != nil {
return err
}
recordedAt, err := resolveTimestamp(cmd, "recorded-at")
@@ -69,9 +69,9 @@ func runTracesShow(cmd *cobra.Command, args []string) error {
return output.RenderYAML(cmd.OutOrStdout(), resp, output.ParseFieldsFlag(flagFields))
default:
out := cmd.OutOrStdout()
- _, _ = fmt.Fprintf(out, "DISTRIBUTED TRACE: %s\nNODES (%d):\n", resp.DistributedTraceId, len(resp.Nodes))
+ _, _ = fmt.Fprintf(out, "DISTRIBUTED TRACE: %s\nNODES (%d):\n", resp.TraceId, len(resp.Nodes))
tw := output.NewTabWriter(out)
- _, _ = fmt.Fprintln(tw, "PROJECT\tTYPE\tNAME\tERROR")
+ _, _ = fmt.Fprintln(tw, "PROJECT\tTYPE\tNAME\tERROR\tGRAPH")
for _, n := range resp.Nodes {
name := "-"
switch {
@@ -86,7 +86,11 @@ func runTracesShow(cmd *cobra.Command, args []string) error {
if n.Exception != nil {
errCol = truncateHash(n.Exception.ExceptionHash, 12)
}
- _, _ = fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n", pickStr(n.ProjectName, "-"), n.TraceType, name, errCol)
+ graph := "-"
+ if n.SpanGraphStatus != nil {
+ graph = n.SpanGraphStatus.State
+ }
+ _, _ = fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\n", pickStr(n.ProjectName, "-"), n.TraceType, name, errCol, graph)
}
return tw.Flush()
}
diff --git a/cli/pkg/client/detail.go b/cli/pkg/client/detail.go
index 705e7693..b80a75f0 100644
--- a/cli/pkg/client/detail.go
+++ b/cli/pkg/client/detail.go
@@ -24,82 +24,102 @@ type startedAtBody struct {
}
// Span mirrors models.Span. Returned inside endpoint, task, and distributed
-// trace detail responses.
+// trace detail responses. TraceId, SpanId and ParentSpanId are the ids the span
+// arrived with, as lowercase hex: 32 characters for a trace, 16 for an
+// OpenTelemetry span, 32 for a span of the native protocol.
type Span struct {
+ ProjectId uuid.UUID `json:"projectId"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ ParentSpanId string `json:"parentSpanId,omitempty"`
+ Name string `json:"name"`
+ StartTime time.Time `json:"startTime"`
+ Duration time.Duration `json:"duration"`
+ RecordedAt time.Time `json:"recordedAt"`
+ SpanKind int32 `json:"spanKind,omitempty"`
+ StatusCode int32 `json:"statusCode,omitempty"`
+ ServiceName string `json:"serviceName,omitempty"`
+ ScopeName string `json:"scopeName,omitempty"`
+ Attributes map[string]string `json:"attributes,omitempty"`
+ AttributesOmitted bool `json:"attributesOmitted,omitempty"`
+ DbStatement string `json:"dbStatement,omitempty"`
+}
+
+// SpanGraphStatus says whether the spans beside it are the whole graph. State
+// is complete, partial or unavailable; Reasons names the limit that was hit.
+type SpanGraphStatus struct {
+ State string `json:"state"`
+ Reasons []string `json:"reasons,omitempty"`
+ OmittedAttributes int `json:"omittedAttributes,omitempty"`
+}
+
+// Task mirrors models.Task: one run of a background task. TraceId and SpanId
+// are the IDs of the span it was promoted from.
+type Task struct {
Id uuid.UUID `json:"id"`
- TraceId uuid.UUID `json:"traceId"`
ProjectId uuid.UUID `json:"projectId"`
- Name string `json:"name"`
- StartTime time.Time `json:"startTime"`
+ TaskName string `json:"taskName"`
Duration time.Duration `json:"duration"`
RecordedAt time.Time `json:"recordedAt"`
- ParentSpanId *uuid.UUID `json:"parentSpanId,omitempty"`
- Attributes map[string]string `json:"attributes,omitempty"`
-}
-
-// Task mirrors models.Task — one run of a background task.
-type Task struct {
- Id uuid.UUID `json:"id"`
- ProjectId uuid.UUID `json:"projectId"`
- TaskName string `json:"taskName"`
- Duration time.Duration `json:"duration"`
- RecordedAt time.Time `json:"recordedAt"`
- ClientIP string `json:"clientIP"`
- Attributes map[string]string `json:"attributes"`
- AppVersion string `json:"appVersion"`
- ServerName string `json:"serverName"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty"`
- SpanId *uuid.UUID `json:"spanId,omitempty"`
- IsRoot bool `json:"isRoot"`
+ ClientIP string `json:"clientIP"`
+ Attributes map[string]string `json:"attributes"`
+ AppVersion string `json:"appVersion"`
+ ServerName string `json:"serverName"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ ParentSpanId string `json:"parentSpanId,omitempty"`
+ IsRoot bool `json:"isRoot"`
}
// AiTrace mirrors models.AiTrace — one LLM call/operation.
type AiTrace struct {
- Id uuid.UUID `json:"id"`
- ProjectId uuid.UUID `json:"projectId"`
- RecordedAt time.Time `json:"recordedAt"`
- Duration time.Duration `json:"duration"`
- StatusCode uint8 `json:"statusCode"`
- Model string `json:"model"`
- ResponseModel string `json:"responseModel"`
- Provider string `json:"provider"`
- Operation string `json:"operation"`
- InputTokens int64 `json:"inputTokens"`
- OutputTokens int64 `json:"outputTokens"`
- TotalTokens int64 `json:"totalTokens"`
- CachedTokens int64 `json:"cachedTokens"`
- ReasoningTokens int64 `json:"reasoningTokens"`
- InputCost float64 `json:"inputCost"`
- OutputCost float64 `json:"outputCost"`
- TotalCost float64 `json:"totalCost"`
- TraceName string `json:"traceName"`
- UserId string `json:"userId"`
- FinishReason string `json:"finishReason"`
- ServerName string `json:"serverName"`
- AppVersion string `json:"appVersion"`
- StorageKey string `json:"storageKey"`
- Attributes map[string]string `json:"attributes"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty"`
- IsRoot bool `json:"isRoot"`
- ConversationId string `json:"conversationId"`
- ToolCallCount int64 `json:"toolCallCount"`
- ToolNames []string `json:"toolNames"`
- Flagged bool `json:"flagged"`
- FlaggedTerms []string `json:"flaggedTerms"`
+ Id uuid.UUID `json:"id"`
+ ProjectId uuid.UUID `json:"projectId"`
+ RecordedAt time.Time `json:"recordedAt"`
+ Duration time.Duration `json:"duration"`
+ StatusCode uint8 `json:"statusCode"`
+ Model string `json:"model"`
+ ResponseModel string `json:"responseModel"`
+ Provider string `json:"provider"`
+ Operation string `json:"operation"`
+ InputTokens int64 `json:"inputTokens"`
+ OutputTokens int64 `json:"outputTokens"`
+ TotalTokens int64 `json:"totalTokens"`
+ CachedTokens int64 `json:"cachedTokens"`
+ ReasoningTokens int64 `json:"reasoningTokens"`
+ InputCost float64 `json:"inputCost"`
+ OutputCost float64 `json:"outputCost"`
+ TotalCost float64 `json:"totalCost"`
+ TraceName string `json:"traceName"`
+ UserId string `json:"userId"`
+ FinishReason string `json:"finishReason"`
+ ServerName string `json:"serverName"`
+ AppVersion string `json:"appVersion"`
+ StorageKey string `json:"storageKey"`
+ Attributes map[string]string `json:"attributes"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ ParentSpanId string `json:"parentSpanId,omitempty"`
+ IsRoot bool `json:"isRoot"`
+ ConversationId string `json:"conversationId"`
+ ToolCallCount int64 `json:"toolCallCount"`
+ ToolNames []string `json:"toolNames"`
+ Flagged bool `json:"flagged"`
+ FlaggedTerms []string `json:"flaggedTerms"`
}
// Session mirrors models.Session — one user session that can be replayed.
type Session struct {
- Id uuid.UUID `json:"id"`
- ProjectId uuid.UUID `json:"projectId"`
- StartedAt time.Time `json:"startedAt"`
- EndedAt *time.Time `json:"endedAt,omitempty"`
- Duration int64 `json:"duration"`
- ClientIP string `json:"clientIP"`
- Attributes map[string]string `json:"attributes"`
- AppVersion string `json:"appVersion"`
- ServerName string `json:"serverName"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty"`
+ Id uuid.UUID `json:"id"`
+ ProjectId uuid.UUID `json:"projectId"`
+ StartedAt time.Time `json:"startedAt"`
+ EndedAt *time.Time `json:"endedAt,omitempty"`
+ Duration int64 `json:"duration"`
+ ClientIP string `json:"clientIP"`
+ Attributes map[string]string `json:"attributes"`
+ AppVersion string `json:"appVersion"`
+ ServerName string `json:"serverName"`
+ TraceId string `json:"traceId,omitempty"`
}
// LinkedException is the exception/message summary attached to endpoint, task,
@@ -123,11 +143,12 @@ type LinkedMessage struct {
// TaskDetailResponse is the body of POST /api/tasks/:taskId.
type TaskDetailResponse struct {
- Task *Task `json:"task"`
- Spans []Span `json:"spans"`
- HasSpans bool `json:"hasSpans"`
- Exception *LinkedException `json:"exception,omitempty"`
- Messages []LinkedMessage `json:"messages"`
+ Task *Task `json:"task"`
+ SpanGraphStatus *SpanGraphStatus `json:"spanGraphStatus,omitempty"`
+ Spans []Span `json:"spans"`
+ HasSpans bool `json:"hasSpans"`
+ Exception *LinkedException `json:"exception,omitempty"`
+ Messages []LinkedMessage `json:"messages"`
}
// GetTask returns one task run plus its spans and linked exceptions/messages.
@@ -145,8 +166,10 @@ func (c *Client) GetTask(ctx context.Context, projectID, id string, recordedAt t
// AiTraceDetailResponse is the body of POST /api/ai-traces/:traceId. The
// conversation blob is passed through verbatim (server stores it opaquely).
type AiTraceDetailResponse struct {
- AiTrace *AiTrace `json:"aiTrace"`
- Conversation json.RawMessage `json:"conversation,omitempty"`
+ AiTrace *AiTrace `json:"aiTrace"`
+ SpanGraphStatus *SpanGraphStatus `json:"spanGraphStatus,omitempty"`
+ Spans []Span `json:"spans"`
+ Conversation json.RawMessage `json:"conversation,omitempty"`
}
// GetAiTrace returns one AI trace plus its stored conversation. recordedAt is
@@ -191,27 +214,36 @@ func (c *Client) GetSession(ctx context.Context, projectID, id string, startedAt
// DistributedTraceNode is one resource (endpoint/task/ai-trace/exception) that
// participated in a distributed trace, scoped to its originating project.
+// TraceId and SpanId name the span the node was promoted from, and
+// ParentEntitySpanId is the SpanId of the nearest node above it, which is how
+// the nodes nest across services and projects.
type DistributedTraceNode struct {
- ProjectId uuid.UUID `json:"projectId"`
- ProjectName string `json:"projectName"`
- TraceType string `json:"traceType"`
- Endpoint *Endpoint `json:"endpoint,omitempty"`
- Task *Task `json:"task,omitempty"`
- AiTrace *AiTrace `json:"aiTrace,omitempty"`
- Spans []Span `json:"spans"`
- Exception *LinkedException `json:"exception,omitempty"`
-}
-
-// DistributedTraceResponse is the body of POST /api/distributed-traces/:id —
+ ProjectId uuid.UUID `json:"projectId"`
+ ProjectName string `json:"projectName"`
+ TraceType string `json:"traceType"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ Endpoint *Endpoint `json:"endpoint,omitempty"`
+ Task *Task `json:"task,omitempty"`
+ AiTrace *AiTrace `json:"aiTrace,omitempty"`
+ SpanGraphStatus *SpanGraphStatus `json:"spanGraphStatus,omitempty"`
+ Spans []Span `json:"spans"`
+ Exception *LinkedException `json:"exception,omitempty"`
+ ParentEntitySpanId string `json:"parentEntitySpanId,omitempty"`
+}
+
+// DistributedTraceResponse is the body of POST /api/distributed-traces/:traceId,
// the full cross-service request timeline, the highest-value RCA view.
type DistributedTraceResponse struct {
- DistributedTraceId string `json:"distributedTraceId"`
- Nodes []DistributedTraceNode `json:"nodes"`
+ TraceId string `json:"traceId"`
+ Nodes []DistributedTraceNode `json:"nodes"`
}
-// GetDistributedTrace returns every node sharing a distributed trace id across
-// all projects the user can see. The route resolves projects from the JWT, so
-// no projectId query param is sent. recordedAt bounds the lookup to ±48h.
+// GetDistributedTrace returns every node of a trace across all projects the
+// user can see. id is the trace id as 32 hex characters (a dashed UUID is
+// accepted too), and a trace linked to it, such as the browser's, comes back
+// with it. The route resolves projects from the JWT, so no projectId query
+// param is sent. recordedAt bounds the lookup to ±48h.
func (c *Client) GetDistributedTrace(ctx context.Context, id string, recordedAt time.Time) (*DistributedTraceResponse, error) {
path := "/api/distributed-traces/" + url.PathEscape(id)
var resp DistributedTraceResponse
diff --git a/cli/pkg/client/detail_test.go b/cli/pkg/client/detail_test.go
index 394da5a4..6d2d69f5 100644
--- a/cli/pkg/client/detail_test.go
+++ b/cli/pkg/client/detail_test.go
@@ -169,16 +169,16 @@ func TestGetDistributedTrace_noProjectIdQueryParam(t *testing.T) {
gotPath = r.URL.Path
gotRawQuery = r.URL.RawQuery
_ = json.NewDecoder(r.Body).Decode(&gotBody)
- _, _ = w.Write([]byte(`{"distributedTraceId":"00000000-0000-0000-0000-000000000006","nodes":[{"projectName":"api","traceType":"endpoint","endpoint":{"id":"00000000-0000-0000-0000-0000000000d1","endpoint":"GET /x"},"spans":[]}]}`))
+ _, _ = w.Write([]byte(`{"traceId":"0af7651916cd43dd8448eb211c80319c","nodes":[{"projectName":"api","traceType":"endpoint","traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"b7ad6b7169203331","parentEntitySpanId":"53995c3f42cd8ad8","endpoint":{"id":"00000000-0000-0000-0000-0000000000d1","endpoint":"GET /x","traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"b7ad6b7169203331"},"spans":[{"traceId":"0af7651916cd43dd8448eb211c80319c","spanId":"00f067aa0ba902b7","parentSpanId":"b7ad6b7169203331","name":"db"}]}]}`))
}))
defer srv.Close()
c := New(srv.URL, WithJWT("tok"))
- resp, err := c.GetDistributedTrace(context.Background(), "00000000-0000-0000-0000-000000000006", ts)
+ resp, err := c.GetDistributedTrace(context.Background(), "0af7651916cd43dd8448eb211c80319c", ts)
if err != nil {
t.Fatalf("GetDistributedTrace: %v", err)
}
- if gotPath != "/api/distributed-traces/00000000-0000-0000-0000-000000000006" {
+ if gotPath != "/api/distributed-traces/0af7651916cd43dd8448eb211c80319c" {
t.Errorf("path = %q", gotPath)
}
if gotRawQuery != "" {
@@ -188,6 +188,16 @@ func TestGetDistributedTrace_noProjectIdQueryParam(t *testing.T) {
t.Errorf("recordedAt = %v", gotBody["recordedAt"])
}
if len(resp.Nodes) != 1 || resp.Nodes[0].TraceType != "endpoint" {
- t.Errorf("nodes wrong: %+v", resp.Nodes)
+ t.Fatalf("nodes wrong: %+v", resp.Nodes)
+ }
+ node := resp.Nodes[0]
+ if resp.TraceId != "0af7651916cd43dd8448eb211c80319c" || node.SpanId != "b7ad6b7169203331" || node.ParentEntitySpanId != "53995c3f42cd8ad8" {
+ t.Errorf("trace identity wrong: %q %+v", resp.TraceId, node)
+ }
+ if node.Endpoint.TraceId != "0af7651916cd43dd8448eb211c80319c" {
+ t.Errorf("endpoint ids wrong: %+v", node.Endpoint)
+ }
+ if len(node.Spans) != 1 || node.Spans[0].ParentSpanId != "b7ad6b7169203331" {
+ t.Errorf("span ids wrong: %+v", node.Spans)
}
}
diff --git a/cli/pkg/client/endpoints.go b/cli/pkg/client/endpoints.go
index 59189539..372fbb98 100644
--- a/cli/pkg/client/endpoints.go
+++ b/cli/pkg/client/endpoints.go
@@ -65,30 +65,32 @@ func (c *Client) ListEndpoints(ctx context.Context, projectID string, req ListEn
// the by-id detail endpoint. EndpointStats (above) is the grouped/list shape;
// this is the single-row shape keyed by the transaction's id.
type Endpoint struct {
- Id uuid.UUID `json:"id"`
- ProjectId uuid.UUID `json:"projectId"`
- Endpoint string `json:"endpoint"`
- Duration time.Duration `json:"duration"`
- RecordedAt time.Time `json:"recordedAt"`
- StatusCode int16 `json:"statusCode"`
- BodySize int32 `json:"bodySize"`
- ClientIP string `json:"clientIP"`
- Attributes map[string]string `json:"attributes"`
- AppVersion string `json:"appVersion"`
- ServerName string `json:"serverName"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty"`
- SpanId *uuid.UUID `json:"spanId,omitempty"`
- IsStream bool `json:"isStream"`
- IsRoot bool `json:"isRoot"`
+ Id uuid.UUID `json:"id"`
+ ProjectId uuid.UUID `json:"projectId"`
+ Endpoint string `json:"endpoint"`
+ Duration time.Duration `json:"duration"`
+ RecordedAt time.Time `json:"recordedAt"`
+ StatusCode int16 `json:"statusCode"`
+ BodySize int32 `json:"bodySize"`
+ ClientIP string `json:"clientIP"`
+ Attributes map[string]string `json:"attributes"`
+ AppVersion string `json:"appVersion"`
+ ServerName string `json:"serverName"`
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ ParentSpanId string `json:"parentSpanId,omitempty"`
+ IsStream bool `json:"isStream"`
+ IsRoot bool `json:"isRoot"`
}
// EndpointDetailResponse is the body of POST /api/endpoints/:endpointId.
type EndpointDetailResponse struct {
- Endpoint *Endpoint `json:"endpoint"`
- Spans []Span `json:"spans"`
- HasSpans bool `json:"hasSpans"`
- Exception *LinkedException `json:"exception,omitempty"`
- Messages []LinkedMessage `json:"messages"`
+ Endpoint *Endpoint `json:"endpoint"`
+ SpanGraphStatus *SpanGraphStatus `json:"spanGraphStatus,omitempty"`
+ Spans []Span `json:"spans"`
+ HasSpans bool `json:"hasSpans"`
+ Exception *LinkedException `json:"exception,omitempty"`
+ Messages []LinkedMessage `json:"messages"`
}
// GetEndpoint returns one request (transaction) by id plus its spans and any
diff --git a/cli/pkg/client/exceptions.go b/cli/pkg/client/exceptions.go
index d844e0d2..663f415e 100644
--- a/cli/pkg/client/exceptions.go
+++ b/cli/pkg/client/exceptions.go
@@ -72,20 +72,37 @@ func (c *Client) ListExceptions(ctx context.Context, projectID string, req ListE
return &resp, nil
}
-// ExceptionStackTrace is one occurrence of a grouped exception.
+// ExceptionStackTrace is one occurrence of a grouped exception. TraceId and
+// SpanId name the span it was recorded on (empty when it arrived outside any
+// trace), and TraceType is set only when that span is itself an endpoint, a
+// task or an AI trace. RelatedEntity on the responses names the owner.
type ExceptionStackTrace struct {
- Id uuid.UUID `json:"id"`
- ExceptionHash string `json:"exceptionHash"`
- StackTrace string `json:"stackTrace"`
- RecordedAt time.Time `json:"recordedAt"`
- TraceId *uuid.UUID `json:"traceId,omitempty"`
- TraceType string `json:"traceType,omitempty"`
- ServerName string `json:"serverName,omitempty"`
- AppVersion string `json:"appVersion,omitempty"`
- IsMessage bool `json:"isMessage,omitempty"`
- Attributes map[string]string `json:"attributes,omitempty"`
- DistributedTraceId *uuid.UUID `json:"distributedTraceId,omitempty"`
- SessionId *uuid.UUID `json:"sessionId,omitempty"`
+ Id uuid.UUID `json:"id"`
+ ExceptionHash string `json:"exceptionHash"`
+ StackTrace string `json:"stackTrace"`
+ RecordedAt time.Time `json:"recordedAt"`
+ TraceId string `json:"traceId,omitempty"`
+ SpanId string `json:"spanId,omitempty"`
+ TraceType string `json:"traceType,omitempty"`
+ ServerName string `json:"serverName,omitempty"`
+ AppVersion string `json:"appVersion,omitempty"`
+ IsMessage bool `json:"isMessage,omitempty"`
+ Attributes map[string]string `json:"attributes,omitempty"`
+ SessionId *uuid.UUID `json:"sessionId,omitempty"`
+}
+
+// RelatedEntity is the endpoint, task or AI trace an exception happened in:
+// the one whose span it was recorded on, or the nearest one above that span.
+// Id and RecordedAt feed straight into endpoints/tasks/ai-traces show, and
+// TraceId into traces show.
+type RelatedEntity struct {
+ TraceType string `json:"traceType"`
+ Id uuid.UUID `json:"id"`
+ Name string `json:"name"`
+ StatusCode int16 `json:"statusCode"`
+ Duration time.Duration `json:"duration"`
+ RecordedAt time.Time `json:"recordedAt"`
+ TraceId string `json:"traceId"`
}
// getExceptionRequest is the body for POST /api/exception-stack-traces/:hash.
@@ -99,6 +116,8 @@ type GetExceptionResponse struct {
Group *ExceptionGroup `json:"group"`
Occurrences []ExceptionStackTrace `json:"occurrences"`
Pagination Pagination `json:"pagination"`
+ // RelatedEntity belongs to the newest occurrence, the first of page 1.
+ RelatedEntity *RelatedEntity `json:"relatedEntity,omitempty"`
}
// GetException returns the group + paginated occurrences for the given hash.
@@ -118,6 +137,7 @@ func (c *Client) GetException(ctx context.Context, projectID, hash string, page
// without walking pages.
type ExceptionByIdResponse struct {
Exception *ExceptionStackTrace `json:"exception"`
+ RelatedEntity *RelatedEntity `json:"relatedEntity,omitempty"`
SessionId *uuid.UUID `json:"sessionId,omitempty"`
SessionRecording json.RawMessage `json:"sessionRecording,omitempty"`
}
diff --git a/cli/pkg/mcpserver/common.go b/cli/pkg/mcpserver/common.go
index 8c9ac361..956b59dd 100644
--- a/cli/pkg/mcpserver/common.go
+++ b/cli/pkg/mcpserver/common.go
@@ -1,6 +1,7 @@
package mcpserver
import (
+ "encoding/hex"
"errors"
"fmt"
"regexp"
@@ -146,6 +147,14 @@ func validateUUID(param, value string) error {
return nil
}
+func validateTraceID(param, value string) error {
+ decoded, err := hex.DecodeString(strings.ReplaceAll(value, "-", ""))
+ if err != nil || len(decoded) != 16 {
+ return usageErrf("invalid %s %q: must be 32 hex characters or a UUID", param, value)
+ }
+ return nil
+}
+
var exceptionHashPattern = regexp.MustCompile(`^[0-9a-f]{16}$`)
func readOnly() *mcp.ToolAnnotations {
diff --git a/cli/pkg/mcpserver/knowledge/debug-flow.md b/cli/pkg/mcpserver/knowledge/debug-flow.md
index 36437c16..6b7a59a2 100644
--- a/cli/pkg/mcpserver/knowledge/debug-flow.md
+++ b/cli/pkg/mcpserver/knowledge/debug-flow.md
@@ -26,7 +26,7 @@ traceway exceptions list --since 7d --search "checkout" --output json \
traceway exceptions show <hash>
```
-This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `distributedTraceId` / `sessionId` per occurrence. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.
+This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `traceId` / `spanId` / `sessionId` per occurrence. `traceId` is the OpenTelemetry trace id (32 hex characters), the same id the request's logs and spans carry. The response also has `relatedEntity`: the endpoint, task or AI trace the newest occurrence happened in, with the `id` and `recordedAt` that `endpoints show` / `tasks show` / `ai-traces show` take. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.
**When the user gave an issue URL (or hash), fix the LAST occurrence — not "the group".** A single hash can bundle *several distinct errors*: the hash is computed from a normalized stack trace with the message stripped, so two unrelated failures that share their top frames (e.g. both captured at the same middleware/recovery frame) collapse into one group. The group's representative stack trace and `firstSeen` may belong to a different, now-dormant error than the one the user is looking at. Anchor on the most recent occurrence and fix that specific failure path:
@@ -34,7 +34,7 @@ This is the high-value call: full stack trace, occurrence list with `recordedAt`
# The occurrence the user actually wants: the latest one. Pin its exact message + attributes + trace.
traceway exceptions show <hash> --output json \
| jq '.occurrences | sort_by(.recordedAt) | last
- | {recordedAt, message: (.stackTrace | split("\n")[0]), attributes, traceId, distributedTraceId}'
+ | {recordedAt, message: (.stackTrace | split("\n")[0]), attributes, traceId, spanId}'
# Then confirm whether the group is homogeneous or mixed — distinct first lines = distinct bugs:
traceway exceptions show <hash> --output json \
@@ -61,7 +61,7 @@ Severity is an OTel number, not a name: 1 TRACE, 5 DEBUG, 9 INFO, 13 WARN, 17 ER
**Correlate by trace**: when an occurrence or log line carries a trace ID, pull the whole request timeline; this is usually the fastest route to a root cause:
```bash
-traceway exceptions show $HASH --output json | jq -r '.occurrences[0].distributedTraceId' \
+traceway exceptions show $HASH --output json | jq -r '.occurrences[0].traceId' \
| xargs -I{} traceway logs query --trace-id {} --output json
```
@@ -70,13 +70,13 @@ Pull the whole cross-service trace and the user's session, reusing the occurrenc
```bash
OCC=$(traceway exceptions show $HASH --output json | jq -c '.occurrences[0]')
TS=$(jq -r '.recordedAt' <<<"$OCC")
-DT=$(jq -r '.distributedTraceId // empty' <<<"$OCC")
+DT=$(jq -r '.traceId // empty' <<<"$OCC")
SID=$(jq -r '.sessionId // empty' <<<"$OCC")
-[ -n "$DT" ] && traceway traces show "$DT" --recorded-at "$TS" # every endpoint/task/ai-trace/exception node across services
+[ -n "$DT" ] && traceway traces show "$DT" --recorded-at "$TS" # available endpoint/task/ai-trace/exception nodes across services
[ -n "$SID" ] && traceway sessions show "$SID" --started-at "$TS" # the session + the exceptions that fired in it
```
-`traces show` is usually the single highest-value RCA call: it stitches one logical request together end to end across services.
+`traces show` connects available nodes across services. Inspect each node's `spanGraphStatus` (the `GRAPH` column in table output): time windows, retention and read limits can leave spans and their related exceptions absent.
**Check metrics for systemic causes** (spikes lining up with `firstSeen` suggest saturation rather than a code bug):
diff --git a/cli/pkg/mcpserver/knowledge/notifications.md b/cli/pkg/mcpserver/knowledge/notifications.md
index 049bdc6e..29fde2b3 100644
--- a/cli/pkg/mcpserver/knowledge/notifications.md
+++ b/cli/pkg/mcpserver/knowledge/notifications.md
@@ -15,7 +15,7 @@ So from a notification, go straight to the occurrence (fast), then pivot reusing
```bash
traceway exceptions occurrence <Exception ID> --recorded-at <Occurred at → RFC3339> --output json
-# the result carries distributedTraceId and sessionId → traces show / sessions show below
+# the result carries traceId, relatedEntity and sessionId → traces show / endpoints show / sessions show below
```
### Performance notification (an endpoint became slow or critical)
diff --git a/cli/pkg/mcpserver/knowledge/performance-flow.md b/cli/pkg/mcpserver/knowledge/performance-flow.md
index fd2bf0ae..0585323e 100644
--- a/cli/pkg/mcpserver/knowledge/performance-flow.md
+++ b/cli/pkg/mcpserver/knowledge/performance-flow.md
@@ -9,7 +9,7 @@ The loop, using the read commands documented in this skill:
1. **Quantify and localize.** `traceway endpoints list --since 24h --order-by p95` (or `impact`). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note that `impact`/`impactReason` are offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not.
2. **Check the accepted baseline.** Before calling an endpoint slow, run `traceway endpoints slow "<endpoint>"`. A non-zero `offsetMs` means an operator accepted that much latency (with a `reason`), so only latency *beyond* the offset is a real regression; `offsetMs 0` means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown.
3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".
-4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <distributedTraceId> --recorded-at <t>` for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.
+4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <traceId> --recorded-at <t>` (the `traceId` that `endpoints show` prints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.
5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.
6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --aggregation max`, then `mem.used`, `go.gc_pause`. A spike that lines up with the latency onset points at saturation, not a code bug.
7. **Correlate with code and deploys.** With the onset time from step 3, check what shipped then: `git log --since '<onset - 30m>' --until '<onset + 30m>'` or the deploy history. A jump at a release is a regression; a gradual ramp with no deploy is data growth (N+1, missing index, no pagination).
diff --git a/cli/pkg/mcpserver/knowledge/performance.md b/cli/pkg/mcpserver/knowledge/performance.md
index ad3a61de..116515c7 100644
--- a/cli/pkg/mcpserver/knowledge/performance.md
+++ b/cli/pkg/mcpserver/knowledge/performance.md
@@ -16,7 +16,7 @@ All commands are read-only. Never archive or mutate anything during a performanc
2. **Get a representative slow trace.** Drill into one slow request to see where the time goes. Capture an id and its `recordedAt` together, then:
```bash
traceway endpoints show <endpointId> --recorded-at <t> # span waterfall for one request
- traceway traces show <distributedTraceId> --recorded-at <t> # cross-service timeline
+ traceway traces show <traceId> --recorded-at <t> # cross-service timeline (traceId from endpoints show)
```
`traces show` is the highest-value call when more than one service is involved: it stitches the whole logical request together end to end.
diff --git a/cli/pkg/mcpserver/knowledge/url-resolution.md b/cli/pkg/mcpserver/knowledge/url-resolution.md
index 3ea8dc16..f5d5d44b 100644
--- a/cli/pkg/mcpserver/knowledge/url-resolution.md
+++ b/cli/pkg/mcpserver/knowledge/url-resolution.md
@@ -12,7 +12,7 @@ Users paste dashboard URLs (`https://<instance>/<route>`) as references in any f
| `/tasks/<task>/<taskId>` | Single task run | `traceway tasks show <taskId> --recorded-at <t>` (`t` = the URL's `?t=` param) |
| `/sessions/<sessionId>` | Session (the exceptions that fired during it; replay stays dashboard-only) | `traceway sessions show <sessionId> --started-at <t>`. The URL has no `?t=`; use the session's start, the URL's `from=`, or a linked occurrence's `recordedAt` (it falls inside the window). Occurrences reference sessions via their `sessionId` |
| `/ai-traces/<traceName>` | AI trace group | No CLI for the group; for one trace use the next row |
-| `/ai-traces/<traceName>/<traceId>` | Single AI trace | `traceway ai-traces show <traceId> --recorded-at <t>` (`t` = the URL's `?t=` param); returns token/cost stats + the conversation |
+| `/ai-traces/<traceName>/<traceId>` | Single AI trace | `traceway ai-traces show <aiTraceId> --recorded-at <t>` (the UUID in the URL, not a 32 hex trace id) (`t` = the URL's `?t=` param); returns token/cost stats + the conversation |
| `/logs` | Logs page (its filters are not stored in the URL) | `traceway logs query` with flags taken from the user's description |
| `/issues`, `/endpoints`, `/metrics`, `/` | List and dashboard pages | The matching `list` / `query` command |
diff --git a/cli/pkg/mcpserver/schema_test.go b/cli/pkg/mcpserver/schema_test.go
index 5b50c5b6..c2b91a5b 100644
--- a/cli/pkg/mcpserver/schema_test.go
+++ b/cli/pkg/mcpserver/schema_test.go
@@ -10,24 +10,36 @@ import (
)
func TestOutSchema_matchesMarshaledShape(t *testing.T) {
- traceID := uuid.New()
+ traceID := "0af7651916cd43dd8448eb211c80319c"
resp := client.EndpointDetailResponse{
Endpoint: &client.Endpoint{
- Id: uuid.New(),
- ProjectId: uuid.New(),
- Endpoint: "GET /api/users",
- DistributedTraceId: &traceID,
+ Id: uuid.New(),
+ ProjectId: uuid.New(),
+ Endpoint: "GET /api/users",
+ TraceId: traceID,
+ SpanId: "b7ad6b7169203331",
},
- Spans: []client.Span{{Id: uuid.New(), TraceId: traceID, ProjectId: uuid.New(), Name: "db"}},
+ SpanGraphStatus: &client.SpanGraphStatus{State: "partial", Reasons: []string{"row_limit"}},
+ Spans: []client.Span{{ProjectId: uuid.New(), TraceId: traceID, SpanId: "00f067aa0ba902b7", ParentSpanId: "b7ad6b7169203331", Name: "db"}},
}
validateAgainstSchema(t, resp)
byID := client.ExceptionByIdResponse{
- Exception: &client.ExceptionStackTrace{Id: uuid.New(), ExceptionHash: "abc123def4567890"},
+ Exception: &client.ExceptionStackTrace{Id: uuid.New(), ExceptionHash: "abc123def4567890", TraceId: traceID, SpanId: "00f067aa0ba902b7"},
+ RelatedEntity: &client.RelatedEntity{TraceType: "endpoint", Id: uuid.New(), Name: "GET /api/users", TraceId: traceID},
SessionRecording: json.RawMessage(`{"segments": [1, 2]}`),
}
validateAgainstSchema(t, byID)
+ trace := client.DistributedTraceResponse{
+ TraceId: traceID,
+ Nodes: []client.DistributedTraceNode{{
+ ProjectId: uuid.New(), ProjectName: "api", TraceType: "endpoint", TraceId: traceID, SpanId: "b7ad6b7169203331",
+ Endpoint: resp.Endpoint, Spans: resp.Spans, ParentEntitySpanId: "53995c3f42cd8ad8",
+ }},
+ }
+ validateAgainstSchema(t, trace)
+
metrics := client.QueryMetricsResponse{
Results: []client.MetricQueryResult{{Name: "cpu", Series: map[string][]client.TimeSeriesPoint{"all": {{Value: 1.5}}}}},
}
diff --git a/cli/pkg/mcpserver/tools.go b/cli/pkg/mcpserver/tools.go
index f0b972ae..198e8317 100644
--- a/cli/pkg/mcpserver/tools.go
+++ b/cli/pkg/mcpserver/tools.go
@@ -27,14 +27,14 @@ func (s *server) addTools(srv *mcp.Server) {
mcp.AddTool(srv, &mcp.Tool{
Name: "get_exception",
OutputSchema: outSchema[client.GetExceptionResponse](),
- Description: "Get one exception group by its 16-hex hash: the representative stack trace plus paginated occurrences, each with recordedAt, attributes, and optional distributedTraceId/sessionId. A hash can bundle several distinct errors that share top frames: when debugging, anchor on the most recent occurrence, not the group; distinct first stack lines mean distinct bugs. Capture each occurrence's id together with its recordedAt for the by-id drill-in tools.",
+ Description: "Get one exception group by its 16-hex hash: the representative stack trace plus paginated occurrences, each with recordedAt, attributes, and optional traceId/spanId/sessionId, plus relatedEntity: the endpoint, task or AI trace the newest occurrence happened in (its id and recordedAt feed the by-id tools, its traceId feeds get_trace). A hash can bundle several distinct errors that share top frames: when debugging, anchor on the most recent occurrence, not the group; distinct first stack lines mean distinct bugs. Capture each occurrence's id together with its recordedAt for the by-id drill-in tools.",
Annotations: readOnly(),
}, s.getException)
mcp.AddTool(srv, &mcp.Tool{
Name: "get_exception_occurrence",
OutputSchema: outSchema[client.ExceptionByIdResponse](),
- Description: "Get a single exception occurrence by UUID, including its sessionId and session recording reference. Prefer this over get_exception when you already hold the occurrence id and its timestamp (dashboard URL path + ?t= param, or a notification's Exception ID + Occurred at).",
+ Description: "Get a single exception occurrence by UUID, including its traceId, its relatedEntity (the endpoint, task or AI trace it happened in), its sessionId and session recording reference. Prefer this over get_exception when you already hold the occurrence id and its timestamp (dashboard URL path + ?t= param, or a notification's Exception ID + Occurred at).",
Annotations: readOnly(),
}, s.getExceptionOccurrence)
@@ -104,7 +104,7 @@ func (s *server) addTools(srv *mcp.Server) {
mcp.AddTool(srv, &mcp.Tool{
Name: "get_trace",
OutputSchema: outSchema[client.DistributedTraceResponse](),
- Description: "Get a distributed trace by id: every endpoint/task/AI-trace/exception node sharing the trace id, across services and across all projects the user can see (no project_id param). Usually the single highest-value root-cause call: it stitches one logical request together end to end. The lookup window is 48h around recorded_at.",
+ Description: "Get a distributed trace by trace id (32 hex characters, the traceId of any occurrence or request): every endpoint/task/AI-trace/exception node sharing the trace id, across services and across all projects the user can see (no project_id param). Each node carries its spanId and parentEntitySpanId, the spanId of the node it sits under, so the call chain can be rebuilt. Usually the single highest-value root-cause call: it stitches one logical request together end to end. The lookup window is 48h around recorded_at.",
Annotations: readOnly(),
}, s.getTrace)
diff --git a/cli/pkg/mcpserver/tools_detail.go b/cli/pkg/mcpserver/tools_detail.go
index 82b02c21..1c7159e7 100644
--- a/cli/pkg/mcpserver/tools_detail.go
+++ b/cli/pkg/mcpserver/tools_detail.go
@@ -82,12 +82,12 @@ func (s *server) getSession(ctx context.Context, req *mcp.CallToolRequest, in ge
}
type getTraceIn struct {
- ID string `json:"id" jsonschema:"The distributed trace UUID, from an occurrence's or request's distributedTraceId."`
+ ID string `json:"id" jsonschema:"The trace id: 32 hex characters, from an occurrence's or request's traceId. A dashed UUID is accepted too."`
RecordedAt string `json:"recorded_at" jsonschema:"REQUIRED for a fast lookup: any participating record's timestamp, RFC3339. Approximate is fine (within 48h for traces). Reuse the occurrence's recordedAt that gave you the trace id. Never pass the current time for an old trace."`
}
func (s *server) getTrace(ctx context.Context, req *mcp.CallToolRequest, in getTraceIn) (*mcp.CallToolResult, any, error) {
- if err := validateUUID("id", in.ID); err != nil {
+ if err := validateTraceID("id", in.ID); err != nil {
return nil, nil, err
}
recordedAt, err := parseTimestamp("recorded_at", in.RecordedAt)
diff --git a/cli/test/contract/contract_test.go b/cli/test/contract/contract_test.go
index cd962c1a..17554136 100644
--- a/cli/test/contract/contract_test.go
+++ b/cli/test/contract/contract_test.go
@@ -56,9 +56,17 @@ func TestContract_endpointsGrouped(t *testing.T) {
func TestContract_endpointDetail(t *testing.T) {
c, rt := capturedClient()
- if _, err := c.GetEndpoint(context.Background(), projectID, seedEndpointID.String(), seedAt); err != nil {
+ resp, err := c.GetEndpoint(context.Background(), projectID, seedEndpointID.String(), seedAt)
+ if err != nil {
t.Fatalf("endpoint detail: %v", err)
}
+ // The waterfall is everything under the endpoint's span, the task's spans included.
+ if resp.Endpoint == nil || resp.Endpoint.TraceId != seedTraceID || resp.Endpoint.SpanId != seedEndpointSpanID || len(resp.Spans) != 4 {
+ t.Errorf("endpoint ids or waterfall wrong: %s", rt.body)
+ }
+ if resp.Exception == nil || resp.Exception.ExceptionHash != seedHash {
+ t.Errorf("the exception recorded below the endpoint's span is missing: %s", rt.body)
+ }
goldenAssert(t, "endpoint-detail", rt.body)
}
@@ -106,20 +114,36 @@ func TestContract_exceptionsGrouped(t *testing.T) {
func TestContract_exceptionByHash(t *testing.T) {
c, rt := capturedClient()
- if _, err := c.GetException(context.Background(), projectID, seedHash, client.PaginationParams{Page: 1, PageSize: 20}); err != nil {
+ resp, err := c.GetException(context.Background(), projectID, seedHash, client.PaginationParams{Page: 1, PageSize: 20})
+ if err != nil {
t.Fatalf("exception by hash: %v", err)
}
+ assertRelatedEndpoint(t, resp.RelatedEntity, rt.body)
goldenAssert(t, "exception-by-hash", rt.body)
}
func TestContract_exceptionById(t *testing.T) {
c, rt := capturedClient()
- if _, err := c.GetExceptionById(context.Background(), projectID, seedExceptionID.String(), seedAt); err != nil {
+ resp, err := c.GetExceptionById(context.Background(), projectID, seedExceptionID.String(), seedAt)
+ if err != nil {
t.Fatalf("exception by id: %v", err)
}
+ if resp.Exception == nil || resp.Exception.TraceId != seedTraceID || resp.Exception.SpanId != seedChildSpanID {
+ t.Errorf("occurrence lost its trace ids: %s", rt.body)
+ }
+ assertRelatedEndpoint(t, resp.RelatedEntity, rt.body)
goldenAssert(t, "exception-by-id", rt.body)
}
+// The seeded exception sits on a span under the endpoint's, so the server has
+// to walk up the trace to name the endpoint.
+func assertRelatedEndpoint(t *testing.T, related *client.RelatedEntity, body []byte) {
+ t.Helper()
+ if related == nil || related.TraceType != "endpoint" || related.Id != seedEndpointID || related.Name != "GET /api/contract" || related.TraceId != seedTraceID {
+ t.Errorf("relatedEntity should name the seeded endpoint: %s", body)
+ }
+}
+
func TestContract_exceptionArchiveRoundTrip(t *testing.T) {
c := client.New(baseURL, client.WithJWT(jwtToken))
ctx := context.Background()
@@ -167,9 +191,14 @@ func TestContract_metricsQuery(t *testing.T) {
func TestContract_taskDetail(t *testing.T) {
c, rt := capturedClient()
- if _, err := c.GetTask(context.Background(), projectID, seedTaskID.String(), seedAt); err != nil {
+ resp, err := c.GetTask(context.Background(), projectID, seedTaskID.String(), seedAt)
+ if err != nil {
t.Fatalf("task detail: %v", err)
}
+ // The exception sits above the task's span, so it is not the task's.
+ if resp.Task == nil || resp.Task.SpanId != seedTaskSpanID || resp.Task.ParentSpanId != seedChildSpanID || len(resp.Spans) != 1 || resp.Exception != nil {
+ t.Errorf("task ids, waterfall or exception wrong: %s", rt.body)
+ }
goldenAssert(t, "task-detail", rt.body)
}
@@ -191,14 +220,31 @@ func TestContract_sessionDetail(t *testing.T) {
func TestContract_distributedTrace(t *testing.T) {
c, rt := capturedClient()
- resp, err := c.GetDistributedTrace(context.Background(), seedTraceID.String(), seedAt)
+ resp, err := c.GetDistributedTrace(context.Background(), seedTraceID, seedAt)
if err != nil {
t.Fatalf("distributed trace: %v", err)
}
- if len(resp.Nodes) == 0 {
- t.Fatal("expected seeded nodes in distributed trace")
+ wire := rt.body
+ // The exception was recorded under the endpoint, so it rides on that node.
+ parents := map[string]string{}
+ for _, node := range resp.Nodes {
+ parents[node.TraceType] = node.ParentEntitySpanId
+ if node.TraceType == "endpoint" && node.Exception == nil {
+ t.Error("the endpoint node lost the exception recorded below its span")
+ }
+ }
+ if len(resp.Nodes) != 3 || parents["task"] != seedEndpointSpanID || parents["ai_trace"] != seedEndpointSpanID || parents["endpoint"] != "" {
+ t.Fatalf("expected the task and the AI trace nested under the endpoint: %s", wire)
+ }
+ // A different trace ID must not expand through legacy metadata.
+ linked, err := c.GetDistributedTrace(context.Background(), seedLinkedTraceID, seedAt)
+ if err != nil {
+ t.Fatalf("distributed trace by linked id: %v", err)
+ }
+ if len(linked.Nodes) != 0 {
+ t.Errorf("the linked trace id returned %d nodes, the trace id %d", len(linked.Nodes), len(resp.Nodes))
}
- goldenAssert(t, "distributed-trace", rt.body)
+ goldenAssert(t, "distributed-trace", wire)
}
// TestContract_cliMetricsValueRoundTrip drives the actual binary end to end and
diff --git a/cli/test/contract/mcp_test.go b/cli/test/contract/mcp_test.go
index 9bbeec6b..620ced11 100644
--- a/cli/test/contract/mcp_test.go
+++ b/cli/test/contract/mcp_test.go
@@ -72,7 +72,7 @@ func TestContract_mcpToolSurface(t *testing.T) {
call("get_task", map[string]any{"id": seedTaskID.String(), "recorded_at": at}, seedTaskID.String())
call("get_ai_trace", map[string]any{"id": seedAiTraceID.String(), "recorded_at": at}, seedAiTraceID.String())
call("get_session", map[string]any{"id": seedSessionID.String(), "started_at": at}, seedSessionID.String())
- call("get_trace", map[string]any{"id": seedTraceID.String(), "recorded_at": at}, seedTraceID.String())
+ call("get_trace", map[string]any{"id": seedTraceID, "recorded_at": at}, seedTraceID)
// Archive round trip last, restoring state for any test that follows.
call("archive_exceptions", map[string]any{"hashes": []string{seedHash}}, "archived")
diff --git a/cli/test/contract/otel_graph_test.go b/cli/test/contract/otel_graph_test.go
new file mode 100644
index 00000000..682cd84c
--- /dev/null
+++ b/cli/test/contract/otel_graph_test.go
@@ -0,0 +1,159 @@
+package contract
+
+import (
+ "bytes"
+ "context"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "net/url"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/google/uuid"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+func TestContract_otelGraphLateParent(t *testing.T) {
+ trace := uuid.New()
+ rootBytes := []byte{1, 2, 3, 4, 5, 6, 7, 8}
+ sourceRoot := hex.EncodeToString(rootBytes)
+ now := time.Now().UTC()
+ traceHex := hex.EncodeToString(trace[:])
+ makeSpan := func(id, parent string, kind int) map[string]any {
+ return map[string]any{"traceId": traceHex, "spanId": id, "parentSpanId": parent, "name": "otel-graph-contract", "kind": kind,
+ "startTimeUnixNano": fmt.Sprint(now.UnixNano()), "endTimeUnixNano": fmt.Sprint(now.Add(time.Millisecond).UnixNano())}
+ }
+ // Export the descendants before their product root in separate HTTP requests.
+ for _, spans := range [][]map[string]any{
+ {makeSpan("1112131415161718", sourceRoot, 1), makeSpan("2122232425262728", "1112131415161718", 1)},
+ {makeSpan(sourceRoot, "", 5)},
+ } {
+ body, err := json.Marshal(map[string]any{"resourceSpans": []any{map[string]any{"scopeSpans": []any{map[string]any{"spans": spans}}}}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ request, err := http.NewRequestWithContext(context.Background(), http.MethodPost, baseURL+"/api/otel/v1/traces", bytes.NewReader(body))
+ if err != nil {
+ t.Fatal(err)
+ }
+ request.Header.Set("Content-Type", "application/json")
+ request.Header.Set("Authorization", "Bearer "+seedToken)
+ response, err := http.DefaultClient.Do(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ payload, err := io.ReadAll(response.Body)
+ response.Body.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if response.StatusCode != http.StatusOK {
+ t.Fatalf("OTLP export: %d %s", response.StatusCode, payload)
+ }
+ }
+ owner := uuid.NewSHA1(uuid.MustParse(projectID), append(trace[:], rootBytes...))
+ c, rt := capturedClient()
+ if _, err := c.GetTask(context.Background(), projectID, owner.String(), now); err != nil {
+ t.Fatal(err)
+ }
+ var detail struct {
+ Task struct {
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ Attributes map[string]string `json:"attributes"`
+ } `json:"task"`
+ Spans []struct {
+ TraceId string `json:"traceId"`
+ SpanId string `json:"spanId"`
+ ParentSpanId string `json:"parentSpanId"`
+ } `json:"spans"`
+ }
+ if err := json.Unmarshal(rt.body, &detail); err != nil {
+ t.Fatal(err)
+ }
+ if len(detail.Spans) != 2 || detail.Task.TraceId != traceHex || detail.Task.SpanId != sourceRoot {
+ t.Fatalf("task detail lost the ids its span arrived with: %s", rt.body)
+ }
+ for key := range detail.Task.Attributes {
+ if strings.HasPrefix(key, "traceway.otel.") {
+ t.Fatalf("identity belongs in traceId and spanId, not in attributes: %s", rt.body)
+ }
+ }
+ for _, span := range detail.Spans {
+ want := sourceRoot
+ if span.SpanId == "2122232425262728" {
+ want = "1112131415161718"
+ }
+ if span.TraceId != traceHex || span.ParentSpanId != want {
+ t.Fatalf("span lost its source ids: %+v", span)
+ }
+ }
+ // The dashed spelling of the same trace id is accepted and answered in hex.
+ for _, id := range []string{traceHex, trace.String()} {
+ resp, err := c.GetDistributedTrace(context.Background(), id, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if resp.TraceId != traceHex || len(resp.Nodes) != 1 || len(resp.Nodes[0].Spans) != 2 || resp.Nodes[0].SpanId != sourceRoot {
+ t.Fatalf("distributed graph contract for %s: %s", id, rt.body)
+ }
+ }
+}
+
+func TestContract_otelSpanPayloadAuthorization(t *testing.T) {
+ trace := uuid.New()
+ traceHex := hex.EncodeToString(trace[:])
+ spanID := "f123456789abcdef"
+ body := fmt.Sprintf(`{"resourceSpans":[{"schemaUrl":"resource-schema","scopeSpans":[{"scope":{"name":"test","version":"1.0"},"spans":[{"traceId":%q,"spanId":%q,"name":"full payload","events":[{"name":"event","attributes":[{"key":"value","value":{"intValue":"9223372036854775807"}}]}]}]}]}]}`, traceHex, spanID)
+ request, _ := http.NewRequest(http.MethodPost, baseURL+"/api/otel/v1/traces", strings.NewReader(body))
+ request.Header.Set("Authorization", "Bearer "+seedToken)
+ request.Header.Set("Content-Type", "application/json")
+ response, err := http.DefaultClient.Do(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ response.Body.Close()
+ if response.StatusCode != 200 {
+ t.Fatalf("ingest: %d", response.StatusCode)
+ }
+ // Every span read is anchored on a time and held to 24 hours either side of it.
+ endpoint := baseURL + "/api/otel/spans/" + traceHex + "/" + spanID + "?projectId=" + projectID + "&at=" + url.QueryEscape(time.Now().UTC().Format(time.RFC3339Nano))
+ for _, authorized := range []bool{false, true} {
+ request, _ := http.NewRequest(http.MethodGet, endpoint, nil)
+ if authorized {
+ request.Header.Set("Authorization", "Bearer "+jwtToken)
+ }
+ response, err := http.DefaultClient.Do(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ payload, err := io.ReadAll(response.Body)
+ response.Body.Close()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !authorized {
+ if response.StatusCode != 401 {
+ t.Fatalf("anonymous payload access: %d", response.StatusCode)
+ }
+ continue
+ }
+ if response.StatusCode != 200 {
+ t.Fatalf("payload: %d %s", response.StatusCode, payload)
+ }
+ var decoded coltracepb.ExportTraceServiceRequest
+ if err := proto.Unmarshal(payload, &decoded); err != nil {
+ t.Fatal(err)
+ }
+ resource := decoded.ResourceSpans[0]
+ span := resource.ScopeSpans[0].Spans[0]
+ if resource.SchemaUrl != "resource-schema" || resource.ScopeSpans[0].Scope.Version != "1.0" || span.Events[0].Attributes[0].Value.GetIntValue() != 9223372036854775807 {
+ t.Fatalf("lost OTLP metadata: %v", &decoded)
+ }
+ }
+}
diff --git a/cli/test/contract/seed_test.go b/cli/test/contract/seed_test.go
index 0c4039ab..774097d4 100644
--- a/cli/test/contract/seed_test.go
+++ b/cli/test/contract/seed_test.go
@@ -2,6 +2,7 @@ package contract
import (
"context"
+ "encoding/hex"
"fmt"
"time"
@@ -13,27 +14,36 @@ import (
)
// Seeded ids, set by seedTelemetry and used to address the by-id detail
-// endpoints. They are shared by every node of one distributed trace so the
-// distributed-trace endpoint returns a populated graph.
+// endpoints. Every node belongs to one trace, and its spans hang together the
+// way a real request's do, so the distributed-trace endpoint returns a nested
+// graph and the exception finds its endpoint through the span above it.
var (
seedEndpointID uuid.UUID
seedTaskID uuid.UUID
seedExceptionID uuid.UUID
- seedSpanID uuid.UUID
seedSessionID uuid.UUID
seedAiTraceID uuid.UUID
seedLogID uuid.UUID
- seedTraceID uuid.UUID // the distributed_trace_id shared across nodes
- seedHash string // exception grouping hash (computed, not assigned)
+ seedTraceID string // the trace id shared across nodes, 32 hex
+ seedHash string // exception grouping hash (computed, not assigned)
seedMetricName = "contract.metric"
seedMetricVal = 42.5
)
+const (
+ seedLinkedTraceID = "4bf92f3577b34da6a3ce929d0e0e4736"
+ seedEndpointSpanID = "e1e1e1e1e1e1e1e1"
+ seedChildSpanID = "c2c2c2c2c2c2c2c2"
+ seedTaskSpanID = "a3a3a3a3a3a3a3a3"
+ seedTaskChildID = "b4b4b4b4b4b4b4b4"
+ seedAiSpanID = "d5d5d5d5d5d5d5d5"
+)
+
// seedTelemetry inserts exactly one row of every telemetry type the CLI reads,
// directly via the backend repositories (which, in the default build, write to
// the SQLite telemetry DB synchronously). Everything is timestamped at `at` and
-// linked by a shared distributed trace id.
+// linked by a shared trace id.
func seedTelemetry(ctx context.Context, projectIDStr string, at time.Time) error {
pid, err := uuid.Parse(projectIDStr)
if err != nil {
@@ -43,82 +53,88 @@ func seedTelemetry(ctx context.Context, projectIDStr string, at time.Time) error
seedEndpointID = uuid.New()
seedTaskID = uuid.New()
seedExceptionID = uuid.New()
- seedSpanID = uuid.New()
seedSessionID = uuid.New()
seedAiTraceID = uuid.New()
seedLogID = uuid.New()
- seedTraceID = uuid.New()
+ traceUUID := uuid.New()
+ seedTraceID = hex.EncodeToString(traceUUID[:])
stack := "ContractError: seeded failure\n\tat contract.go:42"
seedHash = clientcontrollers.ComputeExceptionHash(stack, false)
endpoint := models.Endpoint{
- Id: seedEndpointID,
- ProjectId: pid,
- Endpoint: "GET /api/contract",
- Duration: 123 * time.Millisecond,
- RecordedAt: at,
- StatusCode: 200,
- BodySize: 1024,
- ClientIP: "127.0.0.1",
- Attributes: map[string]string{},
- AppVersion: "1.0.0",
- ServerName: "contract-host",
- DistributedTraceId: &seedTraceID,
- SpanId: &seedSpanID,
- IsRoot: true,
+ Id: seedEndpointID,
+ ProjectId: pid,
+ Endpoint: "GET /api/contract",
+ Duration: 123 * time.Millisecond,
+ RecordedAt: at,
+ StatusCode: 200,
+ BodySize: 1024,
+ ClientIP: "127.0.0.1",
+ Attributes: map[string]string{},
+ AppVersion: "1.0.0",
+ ServerName: "contract-host",
+ TraceId: seedTraceID,
+ SpanId: seedEndpointSpanID,
+ IsRoot: true,
}
if err := telemetry.EndpointRepository.InsertAsync(ctx, []models.Endpoint{endpoint}); err != nil {
return fmt.Errorf("endpoint: %w", err)
}
task := models.Task{
- Id: seedTaskID,
- ProjectId: pid,
- TaskName: "contract-task",
- Duration: 250 * time.Millisecond,
- RecordedAt: at,
- ClientIP: "127.0.0.1",
- Attributes: map[string]string{},
- AppVersion: "1.0.0",
- ServerName: "contract-host",
- DistributedTraceId: &seedTraceID,
- IsRoot: true,
+ Id: seedTaskID,
+ ProjectId: pid,
+ TaskName: "contract-task",
+ Duration: 250 * time.Millisecond,
+ RecordedAt: at,
+ ClientIP: "127.0.0.1",
+ Attributes: map[string]string{},
+ AppVersion: "1.0.0",
+ ServerName: "contract-host",
+ TraceId: seedTraceID,
+ SpanId: seedTaskSpanID,
+ ParentSpanId: seedChildSpanID,
}
if err := telemetry.TaskRepository.InsertAsync(ctx, []models.Task{task}); err != nil {
return fmt.Errorf("task: %w", err)
}
+ // Recorded on the span under the endpoint, with no trace type: the way an
+ // exception arrives from a span that is not itself an endpoint or a task.
exception := models.ExceptionStackTrace{
- Id: seedExceptionID,
- ProjectId: pid,
- TraceId: &seedEndpointID,
- TraceType: "endpoint",
- ExceptionHash: seedHash,
- StackTrace: stack,
- RecordedAt: at,
- Attributes: map[string]string{},
- AppVersion: "1.0.0",
- ServerName: "contract-host",
- IsMessage: false,
- DistributedTraceId: &seedTraceID,
- SessionId: &seedSessionID,
+ Id: seedExceptionID,
+ ProjectId: pid,
+ TraceId: seedTraceID,
+ SpanId: seedChildSpanID,
+ ExceptionHash: seedHash,
+ StackTrace: stack,
+ RecordedAt: at,
+ Attributes: map[string]string{},
+ AppVersion: "1.0.0",
+ ServerName: "contract-host",
+ IsMessage: false,
+ SessionId: &seedSessionID,
}
if err := telemetry.ExceptionStackTraceRepository.InsertAsync(ctx, []models.ExceptionStackTrace{exception}); err != nil {
return fmt.Errorf("exception: %w", err)
}
- span := models.Span{
- Id: seedSpanID,
- TraceId: seedTraceID,
- ProjectId: pid,
- Name: "contract-span",
- StartTime: at,
- Duration: 100 * time.Millisecond,
- RecordedAt: at,
- Attributes: map[string]string{},
+ span := func(id, parent, name string, kind int32) models.Span {
+ return models.Span{
+ ProjectId: pid, TraceId: seedTraceID, SpanId: id, ParentSpanId: parent,
+ Name: name, StartTime: at, Duration: 100 * time.Millisecond, RecordedAt: at,
+ SpanKind: kind, ServiceName: "contract-svc", Attributes: map[string]string{"contract.key": "value"},
+ }
}
- if err := telemetry.SpanRepository.InsertAsync(ctx, []models.Span{span}); err != nil {
+ spans := []models.Span{
+ span(seedEndpointSpanID, "", "GET /api/contract", 2),
+ span(seedChildSpanID, seedEndpointSpanID, "contract-span", 1),
+ span(seedTaskSpanID, seedChildSpanID, "contract-task", 5),
+ span(seedTaskChildID, seedTaskSpanID, "contract-task-span", 1),
+ span(seedAiSpanID, seedEndpointSpanID, "chat gpt-4", 3),
+ }
+ if err := telemetry.SpanRepository.InsertAsync(ctx, spans); err != nil {
return fmt.Errorf("span: %w", err)
}
@@ -134,43 +150,44 @@ func seedTelemetry(ctx context.Context, projectIDStr string, at time.Time) error
}
session := models.Session{
- Id: seedSessionID,
- ProjectId: pid,
- StartedAt: at,
- Duration: 60000,
- ClientIP: "127.0.0.1",
- Attributes: map[string]string{},
- AppVersion: "1.0.0",
- ServerName: "contract-host",
- DistributedTraceId: &seedTraceID,
+ Id: seedSessionID,
+ ProjectId: pid,
+ StartedAt: at,
+ Duration: 60000,
+ ClientIP: "127.0.0.1",
+ Attributes: map[string]string{},
+ AppVersion: "1.0.0",
+ ServerName: "contract-host",
+ TraceId: seedTraceID,
}
if err := telemetry.SessionRepository.Upsert(ctx, []models.Session{session}); err != nil {
return fmt.Errorf("session: %w", err)
}
aiTrace := models.AiTrace{
- Id: seedAiTraceID,
- ProjectId: pid,
- RecordedAt: at,
- Duration: 500 * time.Millisecond,
- StatusCode: 200,
- Model: "gpt-4",
- ResponseModel: "gpt-4",
- Provider: "openai",
- Operation: "chat",
- InputTokens: 10,
- OutputTokens: 20,
- TotalTokens: 30,
- TotalCost: 0.0012,
- TraceName: "contract-ai",
- Attributes: map[string]string{},
- DistributedTraceId: &seedTraceID,
- IsRoot: true,
- ConversationId: "contract-conversation",
- ToolCallCount: 1,
- ToolNames: []string{"contract_tool"},
- Flagged: true,
- FlaggedTerms: []string{"contract-term"},
+ Id: seedAiTraceID,
+ ProjectId: pid,
+ RecordedAt: at,
+ Duration: 500 * time.Millisecond,
+ StatusCode: 200,
+ Model: "gpt-4",
+ ResponseModel: "gpt-4",
+ Provider: "openai",
+ Operation: "chat",
+ InputTokens: 10,
+ OutputTokens: 20,
+ TotalTokens: 30,
+ TotalCost: 0.0012,
+ TraceName: "contract-ai",
+ Attributes: map[string]string{},
+ TraceId: seedTraceID,
+ SpanId: seedAiSpanID,
+ ParentSpanId: seedEndpointSpanID,
+ ConversationId: "contract-conversation",
+ ToolCallCount: 1,
+ ToolNames: []string{"contract_tool"},
+ Flagged: true,
+ FlaggedTerms: []string{"contract-term"},
}
if err := telemetry.AiTraceRepository.InsertAsync(ctx, []models.AiTrace{aiTrace}); err != nil {
return fmt.Errorf("ai trace: %w", err)
diff --git a/cli/test/contract/testdata/aitrace-detail.schema.txt b/cli/test/contract/testdata/aitrace-detail.schema.txt
index 81a4cf26..882d5131 100644
--- a/cli/test/contract/testdata/aitrace-detail.schema.txt
+++ b/cli/test/contract/testdata/aitrace-detail.schema.txt
@@ -2,7 +2,6 @@ aiTrace.appVersion: string
aiTrace.attributes: null
aiTrace.cachedTokens: number
aiTrace.conversationId: string
-aiTrace.distributedTraceId: string
aiTrace.duration: number
aiTrace.finishReason: string
aiTrace.flagged: bool
@@ -15,17 +14,21 @@ aiTrace.model: string
aiTrace.operation: string
aiTrace.outputCost: number
aiTrace.outputTokens: number
+aiTrace.parentSpanId: string
aiTrace.projectId: string
aiTrace.provider: string
aiTrace.reasoningTokens: number
aiTrace.recordedAt: string
aiTrace.responseModel: string
aiTrace.serverName: string
+aiTrace.spanId: string
aiTrace.statusCode: number
aiTrace.storageKey: string
aiTrace.toolCallCount: number
aiTrace.toolNames[]: string
aiTrace.totalCost: number
aiTrace.totalTokens: number
+aiTrace.traceId: string
aiTrace.traceName: string
aiTrace.userId: string
+spanGraphStatus.state: string
diff --git a/cli/test/contract/testdata/distributed-trace.schema.txt b/cli/test/contract/testdata/distributed-trace.schema.txt
index c5a7755e..276f0a7b 100644
--- a/cli/test/contract/testdata/distributed-trace.schema.txt
+++ b/cli/test/contract/testdata/distributed-trace.schema.txt
@@ -1,9 +1,7 @@
-distributedTraceId: string
nodes[].aiTrace.appVersion: string
nodes[].aiTrace.attributes: null
nodes[].aiTrace.cachedTokens: number
nodes[].aiTrace.conversationId: string
-nodes[].aiTrace.distributedTraceId: string
nodes[].aiTrace.duration: number
nodes[].aiTrace.finishReason: string
nodes[].aiTrace.flagged: bool
@@ -16,25 +14,27 @@ nodes[].aiTrace.model: string
nodes[].aiTrace.operation: string
nodes[].aiTrace.outputCost: number
nodes[].aiTrace.outputTokens: number
+nodes[].aiTrace.parentSpanId: string
nodes[].aiTrace.projectId: string
nodes[].aiTrace.provider: string
nodes[].aiTrace.reasoningTokens: number
nodes[].aiTrace.recordedAt: string
nodes[].aiTrace.responseModel: string
nodes[].aiTrace.serverName: string
+nodes[].aiTrace.spanId: string
nodes[].aiTrace.statusCode: number
nodes[].aiTrace.storageKey: string
nodes[].aiTrace.toolCallCount: number
nodes[].aiTrace.toolNames[]: string
nodes[].aiTrace.totalCost: number
nodes[].aiTrace.totalTokens: number
+nodes[].aiTrace.traceId: string
nodes[].aiTrace.traceName: string
nodes[].aiTrace.userId: string
nodes[].endpoint.appVersion: string
nodes[].endpoint.attributes: null
nodes[].endpoint.bodySize: number
nodes[].endpoint.clientIP: string
-nodes[].endpoint.distributedTraceId: string
nodes[].endpoint.duration: number
nodes[].endpoint.endpoint: string
nodes[].endpoint.id: string
@@ -43,22 +43,42 @@ nodes[].endpoint.isStream: bool
nodes[].endpoint.projectId: string
nodes[].endpoint.recordedAt: string
nodes[].endpoint.serverName: string
+nodes[].endpoint.spanId: string
nodes[].endpoint.statusCode: number
+nodes[].endpoint.traceId: string
nodes[].exception.exceptionHash: string
nodes[].exception.recordedAt: string
nodes[].exception.stackTrace: string
+nodes[].parentEntitySpanId: string
nodes[].projectId: string
nodes[].projectName: string
+nodes[].spanGraphStatus.state: string
+nodes[].spanId: string
+nodes[].spans[].attributes.contract.key: string
+nodes[].spans[].duration: number
+nodes[].spans[].name: string
+nodes[].spans[].parentSpanId: string
+nodes[].spans[].projectId: string
+nodes[].spans[].recordedAt: string
+nodes[].spans[].serviceName: string
+nodes[].spans[].spanId: string
+nodes[].spans[].spanKind: number
+nodes[].spans[].startTime: string
+nodes[].spans[].traceId: string
nodes[].spans[]: empty
nodes[].task.appVersion: string
nodes[].task.attributes: null
nodes[].task.clientIP: string
-nodes[].task.distributedTraceId: string
nodes[].task.duration: number
nodes[].task.id: string
nodes[].task.isRoot: bool
+nodes[].task.parentSpanId: string
nodes[].task.projectId: string
nodes[].task.recordedAt: string
nodes[].task.serverName: string
+nodes[].task.spanId: string
nodes[].task.taskName: string
+nodes[].task.traceId: string
+nodes[].traceId: string
nodes[].traceType: string
+traceId: string
diff --git a/cli/test/contract/testdata/endpoint-detail.schema.txt b/cli/test/contract/testdata/endpoint-detail.schema.txt
index 8a5bab38..0e7dd2ef 100644
--- a/cli/test/contract/testdata/endpoint-detail.schema.txt
+++ b/cli/test/contract/testdata/endpoint-detail.schema.txt
@@ -2,7 +2,6 @@ endpoint.appVersion: string
endpoint.attributes: null
endpoint.bodySize: number
endpoint.clientIP: string
-endpoint.distributedTraceId: string
endpoint.duration: number
endpoint.endpoint: string
endpoint.id: string
@@ -13,9 +12,21 @@ endpoint.recordedAt: string
endpoint.serverName: string
endpoint.spanId: string
endpoint.statusCode: number
+endpoint.traceId: string
exception.exceptionHash: string
exception.recordedAt: string
exception.stackTrace: string
hasSpans: bool
messages[]: empty
-spans[]: empty
+spanGraphStatus.state: string
+spans[].attributes.contract.key: string
+spans[].duration: number
+spans[].name: string
+spans[].parentSpanId: string
+spans[].projectId: string
+spans[].recordedAt: string
+spans[].serviceName: string
+spans[].spanId: string
+spans[].spanKind: number
+spans[].startTime: string
+spans[].traceId: string
diff --git a/cli/test/contract/testdata/exception-by-hash.schema.txt b/cli/test/contract/testdata/exception-by-hash.schema.txt
index d3eaec2f..21b78630 100644
--- a/cli/test/contract/testdata/exception-by-hash.schema.txt
+++ b/cli/test/contract/testdata/exception-by-hash.schema.txt
@@ -5,7 +5,6 @@ group.lastSeen: string
group.stackTrace: string
occurrences[].appVersion: string
occurrences[].attributes: null
-occurrences[].distributedTraceId: string
occurrences[].exceptionHash: string
occurrences[].id: string
occurrences[].isMessage: bool
@@ -13,6 +12,7 @@ occurrences[].projectId: string
occurrences[].recordedAt: string
occurrences[].serverName: string
occurrences[].sessionId: string
+occurrences[].spanId: string
occurrences[].stackTrace: string
occurrences[].traceId: string
occurrences[].traceType: string
@@ -20,4 +20,11 @@ pagination.page: number
pagination.pageSize: number
pagination.total: number
pagination.totalPages: number
+relatedEntity.duration: number
+relatedEntity.id: string
+relatedEntity.name: string
+relatedEntity.recordedAt: string
+relatedEntity.statusCode: number
+relatedEntity.traceId: string
+relatedEntity.traceType: string
sessionId: string
diff --git a/cli/test/contract/testdata/exception-by-id.schema.txt b/cli/test/contract/testdata/exception-by-id.schema.txt
index 69edca7c..95d34c84 100644
--- a/cli/test/contract/testdata/exception-by-id.schema.txt
+++ b/cli/test/contract/testdata/exception-by-id.schema.txt
@@ -1,6 +1,5 @@
exception.appVersion: string
exception.attributes: null
-exception.distributedTraceId: string
exception.exceptionHash: string
exception.id: string
exception.isMessage: bool
@@ -8,7 +7,15 @@ exception.projectId: string
exception.recordedAt: string
exception.serverName: string
exception.sessionId: string
+exception.spanId: string
exception.stackTrace: string
exception.traceId: string
exception.traceType: string
+relatedEntity.duration: number
+relatedEntity.id: string
+relatedEntity.name: string
+relatedEntity.recordedAt: string
+relatedEntity.statusCode: number
+relatedEntity.traceId: string
+relatedEntity.traceType: string
sessionId: string
diff --git a/cli/test/contract/testdata/session-detail.schema.txt b/cli/test/contract/testdata/session-detail.schema.txt
index 7c039cf9..33484d3e 100644
--- a/cli/test/contract/testdata/session-detail.schema.txt
+++ b/cli/test/contract/testdata/session-detail.schema.txt
@@ -6,9 +6,9 @@ exceptions[].stackTrace: string
session.appVersion: string
session.attributes: null
session.clientIP: string
-session.distributedTraceId: string
session.duration: number
session.id: string
session.projectId: string
session.serverName: string
session.startedAt: string
+session.traceId: string
diff --git a/cli/test/contract/testdata/task-detail.schema.txt b/cli/test/contract/testdata/task-detail.schema.txt
index 6d1318cb..58cd42e7 100644
--- a/cli/test/contract/testdata/task-detail.schema.txt
+++ b/cli/test/contract/testdata/task-detail.schema.txt
@@ -1,14 +1,27 @@
hasSpans: bool
messages[]: empty
-spans[]: empty
+spanGraphStatus.state: string
+spans[].attributes.contract.key: string
+spans[].duration: number
+spans[].name: string
+spans[].parentSpanId: string
+spans[].projectId: string
+spans[].recordedAt: string
+spans[].serviceName: string
+spans[].spanId: string
+spans[].spanKind: number
+spans[].startTime: string
+spans[].traceId: string
task.appVersion: string
task.attributes: null
task.clientIP: string
-task.distributedTraceId: string
task.duration: number
task.id: string
task.isRoot: bool
+task.parentSpanId: string
task.projectId: string
task.recordedAt: string
task.serverName: string
+task.spanId: string
task.taskName: string
+task.traceId: string
diff --git a/diagrams/architecture-diagram.drawio b/diagrams/architecture-diagram.drawio
index 449365ca..f84383d5 100644
--- a/diagrams/architecture-diagram.drawio
+++ b/diagrams/architecture-diagram.drawio
@@ -4,7 +4,7 @@
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
- <mxCell id="dt-1" parent="1" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;verticalAlign=top;align=center;spacingTop=2;fontSize=11;" value="&lt;b&gt;Distributed Trace&lt;/b&gt; &amp;nbsp;&lt;i&gt;traces from any service that share a distributedTraceId are grouped together&lt;/i&gt;" vertex="1">
+ <mxCell id="dt-1" parent="1" style="rounded=0;whiteSpace=wrap;html=1;dashed=1;verticalAlign=top;align=center;spacingTop=2;fontSize=11;" value="&lt;b&gt;Distributed Trace&lt;/b&gt; &amp;nbsp;&lt;i&gt;traces from any service that share a traceId are grouped together&lt;/i&gt;" vertex="1">
<mxGeometry height="150" width="640" x="105" y="40" as="geometry" />
</mxCell>
<mxCell id="trace-1" parent="1" style="swimlane;whiteSpace=wrap;html=1;expand=1;collapsible=0;" value="Trace" vertex="1">
diff --git a/docs/pages/_meta.json b/docs/pages/_meta.json
index 0b6d1c50..56599257 100644
--- a/docs/pages/_meta.json
+++ b/docs/pages/_meta.json
@@ -3,10 +3,5 @@
"client": { "title": "Docs", "type": "page", "withIndexPage": true },
"learn": { "title": "Learn", "type": "page" },
"symbolicator": { "title": "Symbolicator", "type": "page" },
- "protocol": {
- "title": "Protocol",
- "type": "page",
- "theme": { "sidebar": false }
- },
"server": { "title": "Self Host", "type": "page" }
}
diff --git a/docs/pages/client/js-sdk/distributed-tracing.mdx b/docs/pages/client/js-sdk/distributed-tracing.mdx
index b5e23c1e..5ccff380 100644
--- a/docs/pages/client/js-sdk/distributed-tracing.mdx
+++ b/docs/pages/client/js-sdk/distributed-tracing.mdx
@@ -7,7 +7,7 @@ Correlate frontend errors with the backend requests that caused them. The Tracew
1. The frontend SDK generates a unique trace ID for each outgoing HTTP request
2. The ID is sent as a `traceway-trace-id` header to your backend
3. Your backend sets that ID on its server span as the `traceway.distributed_trace_id` attribute
-4. Traceway uses that attribute as the row's distributed trace ID, so both sides land on one trace
+4. Traceway stores that ID on the backend row as its linked trace, so the browser's trace and the backend trace are shown together
5. In the Traceway dashboard, you can jump between the frontend error and the backend request
Step 3 is the one thing you have to add yourself. See [Backend Setup](#backend-setup) below.
@@ -52,7 +52,7 @@ const res = await api.post("/orders", order);
Your backend closes the loop in one middleware, in whatever language it uses:
1. Read the `traceway-trace-id` request header.
-2. Set it on the active server span as the attribute `traceway.distributed_trace_id`. Traceway reads that attribute and uses it as the row's distributed trace ID, overriding the one derived from the OTel trace ID. This is what puts the browser exception and the backend endpoint on the same distributed trace. The value must be a bare UUID; anything else is ignored silently.
+2. Set it on the active server span as the attribute `traceway.distributed_trace_id`. Traceway keeps the span's own OpenTelemetry trace ID and stores this value next to it as the linked trace. This is what puts the browser exception and the backend endpoint on the same distributed trace. The value must be a bare UUID or 32 hex characters. Anything else is ignored silently.
3. Echo the header back on the response so manual `captureException` calls after `await fetch` can read it. This step is optional. Cross-origin, also add `traceway-trace-id` to `Access-Control-Expose-Headers`.
The [Symfony bundle](/client/otel/symfony) does step 2 for you; add step 3 yourself if you want it. Node.js, NestJS, Next.js, Hono, Cloudflare Workers, Laravel, and Django install vanilla OpenTelemetry, which knows nothing about the header, so do all three:
@@ -74,6 +74,8 @@ Register it after your OpenTelemetry instrumentation, so there is an active serv
To verify, open the backend endpoint's detail page and select **View distributed trace**. The browser exception appears as a second node.
+Only the service that receives the browser's request needs this middleware. Services further down the call chain never see the `traceway-trace-id` header, only `traceparent`, and they do not need to: every backend row carries the same OpenTelemetry trace ID. The first service's row links that trace to the browser's, so Traceway shows the browser exception and every backend service as one distributed trace, whichever of them you open it from.
+
## Manual Capture After an Await
The SDK holds the active trace ID only while the request is in flight and clears it as soon as the request settles. A `captureException` call that runs after `await fetch` therefore sees `null` and stores the exception unlinked, even though the request header was sent correctly.
@@ -124,3 +126,5 @@ The SDK keeps a module-level variable that tracks which request is currently in
3. When the request settles (resolves, rejects, or errors), the active ID is cleared
If an exception is captured (via `captureException`, `window.onerror`, or `window.onunhandledrejection`) while a request is in flight, the SDK automatically attaches the active trace ID to the exception. This is how frontend errors get linked to their triggering backend request without any manual work.
+
+Trace views are limited to projects you can access, the lookup time window and the graph row/attribute limits. Missing or sampled spans cannot be reconstructed.
diff --git a/docs/pages/client/otel/traces.mdx b/docs/pages/client/otel/traces.mdx
index f42c3e19..42693cab 100644
--- a/docs/pages/client/otel/traces.mdx
+++ b/docs/pages/client/otel/traces.mdx
@@ -29,7 +29,7 @@ Rule 1 exists because browser SDKs stamp page context like `url.path` onto their
Child spans that carry `db.query.text` (or the legacy `db.statement`) are displayed under the query text instead of the span name, so a database waterfall reads as the statements that ran rather than a column of identical `pg.query` rows.
-Each Endpoint / Task / AI Trace row carries an `is_root` flag. Rows triggered by another trace surface a **Non Root** chip in the dashboard list, or a **Mixed** chip when a grouped row holds both root and non-root runs. The detail page renders a **Distributed Trace** card with the full waterfall (endpoint → producer → consumer, request → child LLM call) whenever the trace touches more than one run. The `distributed_trace_id` that links the rows is derived from the OTel `trace_id`, so cross-service lookup works with no vendor attribute. If you need to stitch runs that genuinely carry different trace ids, set the span attribute `traceway.distributed_trace_id` to a shared UUID and it overrides the derived value.
+Each Endpoint / Task / AI Trace row carries an `is_root` flag derived from whether its span has a parent. The detail page groups the entities sharing the span's actual `traceId`, including cross-service requests. Standard W3C context propagation connects browser, server and downstream spans. OTel span links are retained on spans and can reference other traces; they do not merge those traces.
> **Sub-handler spans are not duplicated, as long as parent and child ship together.** An in-process child SERVER/INTERNAL+HTTP span (a framework's "handler /path" sub-span) stays in the Spans waterfall when its parent arrives in the same `ResourceSpans` block. Parents are matched inside one block, not across the whole export, though spans from one service normally share a Resource and land in the same block. Traceway cannot see process boundaries, so it treats "parent missing from this block" as a cross-service hop and promotes the child to its own Endpoint. If you see the same route listed twice with a **Mixed** chip, a batch processor split the parent and the child across two exports. Raise the exporter's `maxExportBatchSize` and `scheduledDelayMillis` so a whole trace ships together, or stop emitting the redundant sub-handler span.
diff --git a/docs/pages/learn/ai-tracing.mdx b/docs/pages/learn/ai-tracing.mdx
index 5871a222..af579707 100644
--- a/docs/pages/learn/ai-tracing.mdx
+++ b/docs/pages/learn/ai-tracing.mdx
@@ -47,7 +47,7 @@ The data flow:
1. **OTLP span arrives.** From an AI provider broadcast (e.g. [OpenRouter](/client/openrouter)) or your own SDK instrumentation. The span contains `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.total_cost`, and other standardized attributes.
2. **Traceway promotes the span by attributes, not by position.** If a span has `gen_ai.*` attributes, it lands in the AI Traces table. Root spans use the OTel trace_id as their id; non-root child spans use their own span_id, so a request handler and the LLM call it made each get their own row without collision.
3. **Conversation content stored separately.** The full prompt and completion are stored in object storage (S3 or local filesystem) to keep the database lean.
-4. **Linked across the trace.** `distributed_trace_id` is auto-derived from the OTel trace_id, so a child AI trace and its parent endpoint show up together in the distributed-trace view with no extra configuration.
+4. **Linked across the trace.** An AI trace keeps the OTel trace id of its span, so a child AI trace and its parent endpoint show up together in the distributed-trace view with no extra configuration.
5. **Dashboard shows aggregated view.** AI traces are grouped by trace name (your agent or workflow identifier), with drill-down to individual calls. Groups containing non-root calls are flagged with a **Non-root** chip.
## Three-Level Drill-Down
@@ -80,7 +80,7 @@ Each AI call gets a conversation id at ingest, resolved in this order:
1. `gen_ai.conversation.id` span attribute. Set this in your app for reliable multi-turn grouping.
2. `session.id` span or resource attribute.
-3. The distributed trace id. A single agent run inside one request still groups its calls, even without an explicit id.
+3. The trace id. A single agent run inside one request still groups its calls, even without an explicit id.
Calls ingested before this feature (or with none of the above) have no conversation id and are excluded from conversation analytics.
diff --git a/docs/pages/learn/contributing.mdx b/docs/pages/learn/contributing.mdx
index a8feb70d..15199802 100644
--- a/docs/pages/learn/contributing.mdx
+++ b/docs/pages/learn/contributing.mdx
@@ -181,7 +181,7 @@ npm run build # Production build
### Data Flow
```
-Application → [SDK] → POST /api/report → Backend → ClickHouse/SQLite
+Application → [OpenTelemetry] → POST /api/otel/v1/* → Backend → ClickHouse/SQLite/DuckDB
↓
Dashboard ← [SvelteKit Frontend] ← JSON API ← Gin Controllers
```
diff --git a/docs/pages/learn/data-flow.mdx b/docs/pages/learn/data-flow.mdx
index 2a20c8c6..fcf06a73 100644
--- a/docs/pages/learn/data-flow.mdx
+++ b/docs/pages/learn/data-flow.mdx
@@ -1,36 +1,38 @@
# Data Flow
-## Endpoint Flow
+Traceway is OpenTelemetry first. Your services are instrumented with the OpenTelemetry SDK for their language and export over OTLP/HTTP. See [OpenTelemetry](/client/otel) for the setup.
+
+## Request Flow
Here's how the concepts connect in a typical request:
-1. **Request arrives** → Middleware creates a new **Trace** (Automatically by the client SDK for your framework)
-2. **Auth middleware runs** → Adds `user_id` to **Attributes** (Manually done by you - Traceway does not know about your authentication)
-3. **Handler executes** → Creates **Spans** for sub-operations (DB queries, API calls, your own spans...)
-4. **Error occurs** → **Issue** captured with full attributes context, linked to the trace (Automatically by the client SDK for your framework)
-5. **Request completes** → Trace recorded with duration, status code, and all spans (Automatically by the client SDK for your framework)
+1. **Request arrives** → The HTTP instrumentation starts a server **Span**, which begins a **Trace** or joins the one the caller started (automatic)
+2. **Auth middleware runs** → You add `user_id` to the active span's **Attributes** (done by you, Traceway does not know about your authentication)
+3. **Handler executes** → Child **Spans** are created for sub-operations (DB queries, outgoing HTTP calls, your own spans)
+4. **Error occurs** → The exception is recorded on the span it happened in and becomes an **Issue**, linked to that trace and span (automatic)
+5. **Request completes** → The exporter sends the spans. Traceway stores accepted spans and shows the server span as an **Endpoint** with its duration, status code and waterfall
## Task Flow
-Here's how the concepts connect in a typical scheduled/async task:
+Here's how the concepts connect in a typical scheduled or queued job:
-1. **Task starts** → Client SDK provides a wrapper function for task execution
-2. **Your task code executes** → Creates **Spans** for sub-operations (DB queries, API calls, your own spans...)
-3. **Error occurs** → **Issue** captured with full attributes context, linked to the task trace (Automatically by the client SDK for your framework)
-4. **Task completes** → Trace recorded with duration, status code, and all spans (Automatically by the client SDK for your framework)
+1. **Job starts** → The queue instrumentation starts a consumer **Span**. Depending on the queue instrumentation, it either joins the producer's trace or starts a new trace with an OTel link
+2. **Your job code executes** → Child **Spans** are created for sub-operations
+3. **Error occurs** → The exception is recorded on its span and becomes an **Issue**, linked to the job's trace
+4. **Job completes** → Traceway stores the spans and shows the consumer span as a **Task**, alongside the producer endpoint when both share a trace ID
## Metrics Flow
-1. **System metrics collected** → CPU, memory, goroutines, and GC stats are captured automatically every 30 seconds (Automatically by the client SDK)
-2. **Custom metrics captured** → Your code calls `CaptureMetric` or `CaptureMetricWithTags` to record application-specific measurements with optional tags
-3. **Batched and uploaded** → All metrics are batched, compressed, and sent to Traceway with the next collection frame
+1. **System metrics collected** → The OpenTelemetry Collector's host metrics receiver, or your runtime's metrics instrumentation, reports CPU, memory, disk and network
+2. **Custom metrics recorded** → Your code records counters, gauges and histograms with the OpenTelemetry metrics API, with attributes as tags
+3. **Exported** → Metrics are batched and sent over OTLP
4. **Displayed on dashboards** → Metrics appear on configurable, org-shareable dashboards with line charts, area charts, bar charts, single values, gauges, and tables, filterable and groupable by tags
-## Ingestion pipelines
+## Ingestion
-On the server side all of this arrives through one of two paths: Traceway SDKs (frontend, mobile, and Go) post gzipped collection frames to `/api/report`, while OpenTelemetry-instrumented backends export OTLP to `/api/otel/v1/*`. Both authenticate with the project token, symbolicate stack traces at ingest, and land in the same telemetry store:
+Traces, metrics and logs arrive over OTLP/HTTP at `/api/otel/v1/traces`, `/api/otel/v1/metrics` and `/api/otel/v1/logs`. Each request authenticates with the project token. Stack traces are symbolicated at ingest, and everything lands in the same telemetry store.
-<img src="/ingestion-flows.png" alt="Traceway ingestion pipelines" style={{ maxWidth: '100%', height: 'auto' }} />
+Browser and mobile apps use the Traceway SDK for their platform, which sends the native JSON collection-frame format to `/api/report` and adds session replay and on-device crash capture. Traceway builds canonical spans from those frames and keeps the API’s native UUID identities; OTLP export maps span and parent IDs to eight bytes and includes the available attributes and service metadata. Pick yours under [Docs](/client).
## After the data collection
@@ -40,3 +42,5 @@ All this data flows to your Traceway dashboard where you can:
- Get notified when errors happen
- Track task durations over time
- Monitor system health through metrics
+
+Sampling, project permissions, healthcheck filtering and row rejection can remove telemetry before storage. Trace views are also bounded by access, time, row and attribute limits.
diff --git a/docs/pages/learn/healthchecks.mdx b/docs/pages/learn/healthchecks.mdx
index 2e66478a..19326f55 100644
--- a/docs/pages/learn/healthchecks.mdx
+++ b/docs/pages/learn/healthchecks.mdx
@@ -4,7 +4,7 @@ Healthcheck endpoints get hit constantly: Kubernetes probes every few seconds, l
## How It Works
-When a request arrives on either ingestion path (the native SDK `/api/report` endpoint or OTLP traces), Traceway checks three things:
+When a request arrives, Traceway checks three things:
1. The method is `GET` or `HEAD`
2. The path matches a known healthcheck pattern (built-in or custom)
diff --git a/docs/pages/learn/index.mdx b/docs/pages/learn/index.mdx
index 9bd4689b..bcd4f1bd 100644
--- a/docs/pages/learn/index.mdx
+++ b/docs/pages/learn/index.mdx
@@ -23,7 +23,7 @@ Each table is indexed and shaped for one job: answering "what's slow?", "what's
## Root and non-root entities
-Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root** when the span that produced it was triggered by another trace: a queue worker started by an HTTP request, an LLM call made inside that same handler, a downstream service hop. Non-root rows link back to the originating trace via the auto-derived `distributed_trace_id`, so the distributed-trace view shows the full picture across all the entities that share one OTel trace id.
+Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root** when the span that produced it was a child of another span: a queue worker started by an HTTP request, an LLM call made inside that same handler, a downstream service hop. Non-root rows link back to the originating trace via their `traceId`, so the distributed-trace view shows the full picture across all the entities that share one OTel trace id.
[More on root vs non-root →](/learn/traces#root-and-non-root-entities)
diff --git a/docs/pages/learn/logs.mdx b/docs/pages/learn/logs.mdx
index 9e096241..571d5456 100644
--- a/docs/pages/learn/logs.mdx
+++ b/docs/pages/learn/logs.mdx
@@ -74,7 +74,7 @@ The **Logs** page supports filtering and search across all three attribute layer
- **Severity threshold**: keep only records at or above a minimum level (e.g. `WARN+` hides INFO and below).
- **Service**: exact match on `service.name`.
-- **Trace**: exact match on the hex trace ID, or on a Traceway distributed trace ID (returns logs from every linked trace).
+- **Trace**: exact match on the hex trace ID.
- **Body search**: full-text search over the log message. Single-token queries use a token-bloom index for speed; multi-word or punctuation-heavy queries fall back to substring matching.
- **Attribute filters**: match on any `resource.*`, `scope.*`, or log attribute key, either as equals or not equals.
- **Time range**: pick from presets (24h, 7d, 30d) or a custom window.
@@ -110,7 +110,7 @@ A few details worth knowing:
When your SDK emits a log inside an active OTel span context, the log record automatically carries that span's `trace_id` and `span_id`. This means:
- Every endpoint and task detail page has a **Logs** panel showing the log lines for that trace.
-- When a trace crosses service boundaries, a **Distributed Traces** tab shows logs from every service that participated.
+- Logs from services reporting to the selected project are shown together when they share the trace ID. The trace log panel does not cross project boundaries.
- Jumping from a log to the trace that produced it is a single click in the dashboard.
No extra configuration is required. The standard OTel context propagation carries the IDs to your logger.
@@ -126,3 +126,5 @@ See the SDK-specific guides for setup instructions:
- [OpenTelemetry](/client/otel/logs): direct SDK export or OTel Collector
- [Symfony](/client/otel/symfony/logs): Monolog handler and the OTel PHP Logs API
- [Cloudflare Workers](/client/otel/cloudflare): Worker-side OTel logs exporter
+
+For API callers, `POST /api/logs` accepts `traceId` and optional `spanId`, scoped to the selected project and time window. The retired `distributedTraceId` and `excludeTraceId` selectors return 400, so stale clients cannot silently widen a query. Invalid trace IDs also return 400.
diff --git a/docs/pages/learn/sessions.mdx b/docs/pages/learn/sessions.mdx
index 56ca7819..8b37a9fb 100644
--- a/docs/pages/learn/sessions.mdx
+++ b/docs/pages/learn/sessions.mdx
@@ -90,7 +90,7 @@ clearAttributes();
Then add `userId=u_42` to find that user's sessions, or combine `tenant=acme` with `email~=example.com`. Attribute values must be strings. Changes replace the current session's context; `clearAttributes()` removes app-defined values while retaining browser defaults.
-Older SDKs and `/api/report` payloads that omit session attributes remain supported.
+Older SDKs that omit session attributes remain supported.
## Sessions and Issues
diff --git a/docs/pages/learn/spans.mdx b/docs/pages/learn/spans.mdx
index 33063934..2e5e92ed 100644
--- a/docs/pages/learn/spans.mdx
+++ b/docs/pages/learn/spans.mdx
@@ -6,23 +6,28 @@ Spans are the underlying primitive Traceway stores. Everything else (Endpoints,
| Field | Description |
| --- | --- |
-| `span_id` | 8-byte OTel span id, stored as the low 8 bytes of a UUID |
-| `trace_id` | The id of the owning entity (endpoint, task, or ai_trace); see "Owning entity" below |
-| `parent_span_id` | The span's direct parent within the trace, if any |
+| `trace_id` | The OTel trace id the span arrived with, 32 hex characters |
+| `span_id` | 16 hex characters for OTLP input; native and migrated UUID IDs can be 32 in the Traceway API |
+| `parent_span_id` | The span's direct parent within the trace, empty for a root span |
| `name` | Descriptive label (`db.users.find`, `POST https://api.stripe.com/v1/charges`, etc.) |
| `kind` | One of OTel's `SERVER`, `CLIENT`, `INTERNAL`, `PRODUCER`, `CONSUMER` |
| `start_time` / `duration` | When it started and how long it ran |
| `attributes` | Arbitrary key-value metadata (HTTP, DB, AI, custom) |
| `events` | Timestamped events on the span; `"exception"` events become [Issues](/learn/issues) |
-## Owning entity, not OTel trace id
+## A trace is a trace, a span is a span
-Each row in the `spans` table carries a `trace_id` column, but in Traceway it points to the **owning entity** (an endpoint, task, or ai_trace id), not the raw OTel trace id. This is what makes the waterfall view cheap:
+Traceway stores accepted spans after sampling, permission and healthcheck filtering, with the trace id, span id and parent span id it arrived with. Nothing is renamed or re-rooted at ingestion.
-- "Show me the children of this endpoint" is a single indexed read on `spans.trace_id = <endpoint.id>`.
-- The full cross-process trace is still reachable via the entity's `distributed_trace_id`, which equals the OTel trace id.
+An endpoint, a task and an AI trace are each a copy of one span, the one that started the request, the job or the LLM call. The copy carries the same three ids as its span. That is all the pages need:
-Re-rooting happens at ingestion. When a non-root entity (e.g. a CONSUMER task) is promoted, its descendant spans get the task's id as their `trace_id`. Sibling spans under the originating endpoint keep the endpoint's id. So a single OTel trace that produced an endpoint + a task ends up with two clean waterfalls (one rooted at the endpoint, one at the task), joinable through `distributed_trace_id`.
+- The waterfall of an endpoint is the spans of its trace that sit below its span.
+- Everything connected to it is whatever else shares its trace id, in any project you can open.
+- An exception records the trace and the span it happened on. The endpoint or task it belongs to is the nearest one above that span.
+
+Canonical span storage preserves IDs across export batches. Entity classification can depend on visible ancestry: a nested HTTP server span is suppressed only when a local HTTP server ancestor is present in its resource block. Missing parents are treated as unknown, so separately exported nested server spans can produce extra endpoint projections. A child can join its parent once both are stored and included in the query window and graph limits.
+
+Span reads look 24 hours either side of the endpoint, task or AI trace you opened. A span recorded further away than that is not shown on its page.
## What spans are good for
diff --git a/docs/pages/learn/traces.mdx b/docs/pages/learn/traces.mdx
index f394b2f9..550016e5 100644
--- a/docs/pages/learn/traces.mdx
+++ b/docs/pages/learn/traces.mdx
@@ -19,7 +19,7 @@ OTLP span batch
├──► ai_traces ─┤ indexed for fast aggregation
├──► exception_… ─┘
│
- └──► spans (everything else, the raw waterfall)
+ └──► spans_v2 (all accepted spans, including promoted ones)
```
You can think of `endpoints`, `tasks`, and `ai_traces` as **cached projections** of the spans table. They exist for one reason: aggregations like "P95 of every endpoint over 7 days" or "total LLM cost grouped by trace_name" need to be cheap. Scanning the full spans table to compute those every dashboard load would not be cheap.
@@ -40,7 +40,7 @@ A span lands in a top-level table when its kind or attributes match a rule. The
| Root `INTERNAL` span with a `console.command` attribute (artisan / artisan-style runners) | **Task** |
| Any span carrying `gen_ai.*` attributes (root or child) | **AI Trace** |
| `"exception"` event on any span | **Issue** (attributed to the owning entity above) |
-| Anything else with a parent | **Span** in the waterfall, re-rooted to its nearest enclosing entity |
+| Anything else with a parent | **Span** in the waterfall, with its original trace/span/parent IDs |
In-process sub-handler spans (e.g. a framework's "handler /path" INTERNAL+HTTP child) stay in the spans table. Endpoints are only promoted for actual entry points, so frameworks don't silently double-emit.
@@ -56,19 +56,17 @@ Each Endpoint / Task / AI Trace row carries an `is_root` flag.
Non-root rows surface a **Non-root** chip in the dashboard list. On the row's detail page, a **View distributed trace** link jumps to the full picture across all the entities sharing that trace id.
-The id rules avoid collisions when a root and a non-root entity share the same trace id:
-- Root entity id = OTel trace id (so the request and its lineage are easy to look up by trace id).
-- Non-root entity id = the entity's own span id (distinct from the root's id even within the same trace).
+Every OTLP projection keeps the ids of the span it was made from: `trace_id`, `span_id` and `parent_span_id`. Its own row id is derived from the project, the trace and the span, so a retried export produces the same id. Native `/api/report` occurrences keep their SDK UUIDs instead; the API exposes UUID-width span IDs and OTLP export maps them to eight bytes.
## Distributed traces
-Every Endpoint / Task / AI Trace stores a **`distributed_trace_id`** equal to the OTel trace id, auto-derived on ingestion. So:
+Every Endpoint / Task / AI Trace stores the **`trace_id`** of its span. Rows that share a trace id belong together, whichever service or project reported them. So:
-- A request that dispatches a queue job and the worker that processes it share the same `distributed_trace_id`. They show up as two rows (the endpoint and the task), and a single page (`/distributed-traces/<id>`) renders them side by side with their spans.
-- A request handler that calls an LLM produces an endpoint row and an ai_trace row with the same `distributed_trace_id`. Same cross-trace page.
-- Cross-service HTTP hops (each service exports its own SERVER span) all share the trace id. Each hop appears as its own endpoint row.
+- A request and its queue worker share a trace id when the instrumentation propagates a parent context. Some queue integrations start a new trace with a link instead; those runs are not grouped by shared trace ID. With parent-context propagation the card shows the endpoint and task together, nested by their span parentage. A task started as a new trace with a span link has its own trace view.
+- A request handler that calls an LLM produces an endpoint row and an ai_trace row with the same trace id. Same card.
+- Cross-service HTTP hops (each service exports its own SERVER span) all share the trace id. Each hop appears as its own endpoint row, nested by the spans in between.
-You don't need to set any vendor attribute for this to work, OTLP's native trace_id is enough. If you do need to stitch runs that genuinely carry different trace ids, the span attribute `traceway.distributed_trace_id` overrides the derived value.
+Standard W3C propagation also makes browser requests share the backend trace ID. OTel span links can reference another trace, but those traces remain separate. See [Distributed Tracing](/client/js-sdk/distributed-tracing).
## What each entity stores
@@ -86,9 +84,7 @@ On the OTel ingestion path, a 404 from a matched route keeps its route identity.
- the span carries no valid `http.route` attribute, meaning the framework matched no route, or
- `http.route` is a catch-all made only of slashes and wildcards, such as `/`, `/*`, `/**`, or `*/*`. Some instrumentations report these for unmatched requests: Express reports `/` when only app-level middleware ran, and Spring reports `/**` for its static resource handler.
-Scoped wildcards like `GET /api/*` are real routes and keep their name even on 404.
-
-This route check only exists for OTel ingestion. The legacy `/api/report` path has no route information on the wire, so it groups every 404 under `UNMATCHED`.
+Scoped wildcards like `GET /api/*` are real routes and keep their name even on 404. The native `/api/report` path retains its existing behavior of grouping all 404 responses as `UNMATCHED`; it cannot use the same OTLP route-attribute distinction.
### Tasks
Identified by a task name (job class, scheduled command, agent operation). Ranked by count × (P95 − P50) as a rough impact proxy. CONSUMER spans from any queue driver (database, Redis, SQS, Beanstalk, …) land here, as do root `console.command` spans.
@@ -101,7 +97,7 @@ Identified by `trace.name` (your agent or workflow). Carries model, provider, in
1. A span is created in your app (by a framework's auto-instrumentation or by manual OTel calls).
2. Child spans, attributes, and events accumulate against it.
3. The trace is exported via OTLP.
-4. Traceway classifies each span, inserts into the matching entity table, and re-roots the remaining children into `spans` under their nearest enclosing entity.
+4. Traceway stores accepted spans in `spans_v2`, then writes matching entity projections. Parent IDs remain unchanged. Sampling, healthcheck filtering, permissions and rejected rows can limit what is stored.
5. Exceptions recorded as `"exception"` events become Issues attributed to the owning entity.
Sampling:
diff --git a/docs/pages/protocol/_meta.json b/docs/pages/protocol/_meta.json
deleted file mode 100644
index ca0a2a6c..00000000
--- a/docs/pages/protocol/_meta.json
+++ /dev/null
@@ -1,3 +0,0 @@
-{
- "index": "Overview"
-}
diff --git a/docs/pages/protocol/index.md b/docs/pages/protocol/index.md
deleted file mode 100644
index 0668dfaf..00000000
--- a/docs/pages/protocol/index.md
+++ /dev/null
@@ -1,598 +0,0 @@
-# Traceway Protocol Specification
-
-This document is the complete protocol specification for the Traceway `/api/report` endpoint. Use it as a reference when building a new Traceway client SDK in any language.
-
-## Endpoint
-
-```
-POST /api/report
-```
-
-A single endpoint accepts all telemetry data: **traces** (endpoints and tasks), **exceptions** (errors and messages), **metrics**, and (for browser SDKs) **sessions** with **session recordings**.
-
-## Authentication
-
-Every request must include a bearer token in the `Authorization` header. The token is a project token (UUID v4) obtained from the Traceway dashboard.
-
-```
-Authorization: Bearer {token}
-```
-
-The backend validates the token against a project cache. Invalid or missing tokens result in a `401 Unauthorized` response.
-
-## Transport
-
-### Connection String
-
-SDKs accept a connection string in the format:
-
-```
-{token}@{apiUrl}
-```
-
-**Example:**
-
-```
-550e8400-e29b-41d4-a716-446655440000@https://traceway.example.com/api/report
-```
-
-Parse by splitting on the first `@`:
-- Left side → bearer token
-- Right side → full URL to POST to
-
-### Required Headers
-
-| Header | Value |
-|--------|-------|
-| `Content-Type` | `application/json` |
-| `Content-Encoding` | `gzip` |
-| `Authorization` | `Bearer {token}` |
-
-### Gzip Requirement
-
-The request body **should** be gzip-compressed JSON for the regular sync path. The backend's gzip middleware decompresses when `Content-Encoding: gzip` is present and passes the body through unchanged when it is absent. Browser SDKs use the uncompressed path on the page-unload (`pagehide`) flush so the request can dispatch synchronously inside the unload handler; `CompressionStream` is async and would yield control before the request leaves. Servers MUST therefore accept both compressed and uncompressed JSON on `/api/report`.
-
-The body is capped **after decompression**, so gzip does not raise the ceiling. The default is 64 MB, tunable with `REPORT_MAX_BODY_MB`. A body over the cap answers `413` and is not ingested. See [Response Handling](#response-handling).
-
-## Request Payload
-
-### Top-Level Structure
-
-```json
-{
- "collectionFrames": [ ... ],
- "appVersion": "1.2.3",
- "serverName": "web-01"
-}
-```
-
-| Field | Type | Required | Description |
-|-------|------|----------|-------------|
-| `collectionFrames` | `CollectionFrame[]` | Yes | Array of collection frames containing telemetry data |
-| `appVersion` | `string` | Yes | Application version string (can be empty `""`) |
-| `serverName` | `string` | Yes | Hostname or server identifier (can be empty `""`) |
-
-### CollectionFrame
-
-Each frame is a batch of data collected during one collection interval.
-
-```json
-{
- "stackTraces": [ ... ],
- "metrics": [ ... ],
- "traces": [ ... ],
- "sessions": [ ... ],
- "sessionRecordings": [ ... ]
-}
-```
-
-| Field | Type | Required | Description |
-|-------|------|----------|-------------|
-| `stackTraces` | `ExceptionStackTrace[]` | Yes | Array of exception/message records |
-| `metrics` | `MetricRecord[]` | Yes | Array of metric data points |
-| `traces` | `Trace[]` | Yes | Array of endpoint and task traces |
-| `sessions` | `Session[]` | No | Browser SDKs only: session-lifecycle markers (open / refresh / close). Omit for non-browser SDKs |
-| `sessionRecordings` | `SessionRecording[]` | No | Browser SDKs only: rrweb segment uploads. Omit when there is nothing to ship |
-
-All five fields are accepted. Use empty arrays `[]` or `null` (or omit entirely) when there is no data of that type.
-
-## Traces
-
-A trace represents either an **endpoint** (HTTP request) or a **task** (background job). The `isTask` field distinguishes them.
-
-### Trace Object
-
-```json
-{
- "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
- "endpoint": "GET /api/users/:id",
- "duration": 15234000,
- "recordedAt": "2025-01-15T10:30:00.123Z",
- "statusCode": 200,
- "bodySize": 1024,
- "clientIP": "192.168.1.1",
- "attributes": { "user_id": "42" },
- "spans": [ ... ],
- "isTask": false
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| Id | `id` | `string` | Yes | UUID v4 identifying this trace |
-| Endpoint | `endpoint` | `string` | Yes | Endpoint name or task name |
-| Duration | `duration` | `integer` | Yes | Duration in **nanoseconds** (int64) |
-| RecordedAt | `recordedAt` | `string` | Yes | Timestamp in **RFC 3339** format |
-| StatusCode | `statusCode` | `integer` | Yes | HTTP status code (0 for tasks) |
-| BodySize | `bodySize` | `integer` | Yes | Response body size in bytes (0 for tasks) |
-| ClientIP | `clientIP` | `string` | Yes | Client IP address (empty `""` for tasks) |
-| Attributes | `attributes` | `object` | No | Key-value string pairs. Omit or `null` when empty. |
-| Spans | `spans` | `Span[]` | No | Sub-operation spans. Omit or `null` when empty. |
-| IsTask | `isTask` | `boolean` | No | `true` for tasks, omit or `false` for endpoints. |
-
-### Important Notes on Trace Fields
-
-**Duration is nanoseconds.** Go's `time.Duration` serializes as an int64 representing nanoseconds. A duration of 15.234ms is sent as `15234000`. Other languages must convert their duration representation to nanoseconds before serializing.
-
-**Timestamps are RFC 3339.** Go's `time.Time` serializes as RFC 3339 with optional nanosecond precision (e.g., `"2025-01-15T10:30:00.123456789Z"`).
-
-**UUIDs** are v4, formatted as `"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"`.
-
-**`attributes` and `spans`** use `omitempty` in Go, which means they are omitted from JSON when null/empty. Your SDK may omit them or send `null`/`{}` and `[]`.
-
-**`isTask`** uses `omitempty`, so it is omitted when `false`. The backend treats a missing `isTask` as `false`.
-
-### Endpoints vs. Tasks
-
-| | Endpoint | Task |
-|---|----------|------|
-| `isTask` | `false` (or omitted) | `true` |
-| `endpoint` | `"METHOD /path"` (e.g., `"GET /api/users/:id"`) | Descriptive name (e.g., `"report.monthly"`) |
-| `statusCode` | HTTP status code | `0` |
-| `bodySize` | Response body size | `0` |
-| `clientIP` | Client IP | `""` |
-
-### Span Object
-
-Spans represent sub-operations within a trace.
-
-```json
-{
- "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
- "name": "db.query",
- "startTime": "2025-01-15T10:30:00.100Z",
- "duration": 5000000
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| Id | `id` | `string` | Yes | UUID v4 identifying this span |
-| Name | `name` | `string` | Yes | Descriptive name of the operation |
-| StartTime | `startTime` | `string` | Yes | Start timestamp in RFC 3339 format |
-| Duration | `duration` | `integer` | Yes | Duration in **nanoseconds** (int64) |
-
-## Exceptions
-
-An exception record represents either an **error/panic** or an explicitly **captured message**. The `isMessage` field distinguishes them.
-
-### ExceptionStackTrace Object
-
-```json
-{
- "traceId": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
- "isTask": false,
- "stackTrace": "*errors.errorString: connection refused\nhandleRequest()\n handler.go:42\n",
- "recordedAt": "2025-01-15T10:30:01.500Z",
- "attributes": { "user_id": "42" },
- "isMessage": false
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| TraceId | `traceId` | `string \| null` | No | UUID of the linked trace, or `null`/omitted for standalone exceptions |
-| IsTask | `isTask` | `boolean` | No | `true` if the linked trace is a task. Omit or `false` for endpoints. |
-| StackTrace | `stackTrace` | `string` | Yes | Stack trace string or message text |
-| RecordedAt | `recordedAt` | `string` | Yes | Timestamp in RFC 3339 format |
-| Attributes | `attributes` | `object` | No | Key-value string pairs. Omit or `null` when empty. |
-| IsMessage | `isMessage` | `boolean` | Yes | `false` for errors/panics, `true` for captured messages |
-
-### Notes
-
-- **`traceId`** is a nullable string pointer. When there is no linked trace, send `null` or omit the field entirely.
-- **`isTask`** uses `omitempty` (omitted when `false`).
-- **`isMessage=false`** → error or panic with a stack trace. The backend normalizes and hashes the stack trace for grouping.
-- **`isMessage=true`** → an explicitly captured message (e.g., via `CaptureMessage`). The full `stackTrace` string is hashed as-is for grouping.
-
-## Stack Trace Format
-
-The Go client produces stack traces in this format:
-
-```
-{ErrorType}: {message}
-{FuncName}()
- {file}:{line}
-{FuncName}()
- {file}:{line}
-...
-```
-
-**Example:**
-
-```
-*errors.errorString: connection refused
-handleRequest()
- handler.go:42
-processConnection()
- server.go:128
-main()
- main.go:15
-```
-
-- The first line contains the Go type name (via `reflect.TypeOf(err).String()`) and the error message.
-- Each subsequent frame has the function name (short form, without full package path) on one line, then the file path and line number indented with 4 spaces on the next.
-- Function names are shortened: the full path is trimmed to everything after the last `/`, then everything after the first `.` (e.g., `github.com/user/pkg.Handler` → `Handler`).
-
-### Backend Normalization for Grouping
-
-The backend computes a SHA-256 hash (truncated to 16 hex characters) of a normalized form of the stack trace. Normalization (for errors, not messages) includes:
-
-- Removing the error message content (keeping only the error type)
-- Replacing absolute file paths with `filename:line`
-- Replacing hex addresses, UUIDs, large numbers, email addresses, IP addresses, and goroutine numbers with placeholders
-- Removing module version strings (e.g., `@v1.2.3`)
-- Normalizing whitespace
-
-This means the same logical error is grouped together even when runtime values differ.
-
-## Metrics
-
-### MetricRecord Object
-
-```json
-{
- "name": "cpu.used_pcnt",
- "value": 45.2,
- "recordedAt": "2025-01-15T10:30:00Z"
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| Name | `name` | `string` | Yes | Metric name |
-| Value | `value` | `number` | Yes | Metric value (float64) |
-| RecordedAt | `recordedAt` | `string` | Yes | Timestamp in RFC 3339 format |
-
-### Predefined Metric Names
-
-The Go client collects these system metrics automatically:
-
-| Name | Description | Unit |
-|------|-------------|------|
-| `mem.used` | Allocated memory | MB |
-| `mem.total` | Total system memory | MB |
-| `cpu.used_pcnt` | CPU usage percentage | % (0–100) |
-| `go.go_routines` | Active goroutine count | count |
-| `go.heap_objects` | Heap object count | count |
-| `go.num_gc` | Total GC cycles | count |
-| `go.gc_pause` | Total GC pause time | nanoseconds |
-
-### Custom Metrics
-
-SDKs can send any `name`/`value` pair. There is no registry: any string name is accepted.
-
-```json
-{ "name": "queue.length", "value": 42.0, "recordedAt": "2025-01-15T10:30:00Z" }
-```
-
-## Sessions
-
-A **session** represents one user's continuous interaction with a browser app, typically the lifetime of an open tab. Sessions are emitted only by browser SDKs that have **always-on session recording** enabled (`recordAllSessions: true`); back-end and mobile SDKs omit the `sessions` and `sessionRecordings` fields entirely.
-
-The session row is upserted on the backend (keyed by `id`), so the same session can be emitted multiple times during its lifetime; each emission overwrites mutable fields (`endedAt`, `duration`, `attributes`). The shape is identical for the open, mid-session refresh, and close payloads; only the populated fields differ.
-
-### Session Object
-
-```json
-{
- "id": "8a1f0b3c-2e7a-4f1d-9bd4-71e6c8a2b5d0",
- "startedAt": "2026-05-06T14:02:00.000Z",
- "endedAt": "2026-05-06T14:23:17.412Z",
- "clientIP": "",
- "attributes": {
- "url": "https://app.example.com/dashboard",
- "userAgent": "Mozilla/5.0 ...",
- "viewport": "1920x1080",
- "userId": "u_42",
- "tenant": "acme"
- },
- "distributedTraceId": ""
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| Id | `id` | `string` | Yes | UUID v4. Generated client-side at session start; persistent for the page lifecycle. The backend uses this directly as the session row id |
-| StartedAt | `startedAt` | `string` | Yes | RFC 3339 timestamp of session start |
-| EndedAt | `endedAt` | `string \| null` | No | RFC 3339 timestamp of session end. Omitted on the opening payload; populated on the close payload |
-| ClientIP | `clientIP` | `string` | No | Optional client-side hint. The backend stamps the request's IP into `attributes["client.ip"]` regardless |
-| Attributes | `attributes` | `object` | No | Key-value pairs (auto-collected browser context + caller-provided global scope) |
-| DistributedTraceId | `distributedTraceId` | `string` | No | Optional UUID linking the session to a distributed trace |
-
-### Session Lifecycle (browser SDKs)
-
-A new session begins on:
-
-1. SDK init when `recordAllSessions: true` and a browser context is available.
-2. Page restore from bfcache (`pageshow` with `event.persisted === true`). A fresh `id` is generated and the previous session row is left as-is.
-
-A session ends on:
-
-1. **Inactivity timeout**: 15 min since the last DOM event observed by rrweb (or the last `markActivity()` call).
-2. **Maximum duration**: 60 min from `startedAt`.
-3. **Page unload**: `pagehide` triggers a final flush and end.
-
-The closing payload is sent via `fetch(..., { keepalive: true })` with raw JSON (no gzip) so it dispatches synchronously inside the `pagehide` handler.
-
-### Auto-collected Attributes
-
-Browser SDKs stamp these on every session at open and close (caller-provided keys via `setAttribute*` win on collision):
-
-| Key | Source |
-|---|---|
-| `url` | `window.location.href` |
-| `path` | `window.location.pathname` |
-| `referrer` | `document.referrer` |
-| `userAgent` | `navigator.userAgent` |
-| `language` | `navigator.language` |
-| `platform` | `navigator.platform` |
-| `viewport` | `${window.innerWidth}x${window.innerHeight}` |
-| `screen` | `${screen.width}x${screen.height}` |
-| `timezone` | `Intl.DateTimeFormat().resolvedOptions().timeZone` |
-| `client.ip` | Stamped server-side from the request's `RemoteAddr` |
-
-## Session Recordings
-
-Session recordings carry the actual replay data. Each recording is **one segment**, typically ~30 s of rrweb events. A session is reassembled at view time by ordering all of its recordings by `segmentIndex`.
-
-A recording can be **session-bound** (always-on path), **exception-bound** (legacy, when an exception is captured), or both at once when an exception fires inside an always-on segment.
-
-### SessionRecording Object
-
-```json
-{
- "exceptionId": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
- "sessionId": "8a1f0b3c-2e7a-4f1d-9bd4-71e6c8a2b5d0",
- "segmentIndex": 4,
- "events": [ /* rrweb event objects */ ],
- "logs": [ /* console output entries */ ],
- "actions": [ /* fetch / navigation / custom breadcrumbs */ ],
- "startedAt": "2026-05-06T14:14:00.000Z",
- "endedAt": "2026-05-06T14:14:30.000Z"
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| ExceptionId | `exceptionId` | `string` | No | UUID linking the recording to a specific exception. Set on exception-bound clips; omitted for pure always-on segments |
-| SessionId | `sessionId` | `string` | No | UUID linking the recording to a parent session. Set on always-on segments; omitted for legacy exception-bound clips |
-| SegmentIndex | `segmentIndex` | `integer` | No | Monotonically increasing per-session counter starting at 0. Omitted (defaults to 0) for exception-bound clips |
-| Events | `events` | `array` | Yes | rrweb event array (DOM mutations, input events, mouse moves) |
-| Logs | `logs` | `array` | No | Console output entries captured in the segment window |
-| Actions | `actions` | `array` | No | Network / navigation / custom breadcrumbs captured in the segment window |
-| StartedAt | `startedAt` | `string` | No | RFC 3339 timestamp of the first event in the segment |
-| EndedAt | `endedAt` | `string` | No | RFC 3339 timestamp of the last event in the segment |
-
-The backend stores each recording as a row in `session_recordings` with the events bytes written to S3 (or a configured object store). At least one of `exceptionId` or `sessionId` must be set; recordings with neither linkage are dropped.
-
-## Exception ↔ Session Linking
-
-`ExceptionStackTrace` gains an optional `sessionId` field. Browser SDKs running with `recordAllSessions: true` stamp every exception with the current session id, so the dashboard can render a "View full session" link from the issue page even when the per-exception clip is the inline replay.
-
-```json
-{
- "stackTrace": "...",
- "recordedAt": "2026-05-06T14:14:23.000Z",
- "sessionId": "8a1f0b3c-2e7a-4f1d-9bd4-71e6c8a2b5d0",
- "isMessage": false
-}
-```
-
-| Field | JSON Key | Type | Required | Description |
-|-------|----------|------|----------|-------------|
-| SessionId | `sessionId` | `string \| null` | No | UUID of the parent session. Omitted when no session is active or when `recordAllSessions` is off |
-
-The legacy `sessionRecordingId` field still works for exception-bound clips. Both can be set on a single exception captured inside an always-on session.
-
-## Batching and Collection Strategy
-
-This section describes the recommended client-side implementation for anyone writing a client against this protocol directly. Most applications should use OpenTelemetry instead, which handles all of it. See [OpenTelemetry](/client/otel).
-
-### Collection Interval
-
-Collect telemetry data into a "current frame" (`CollectionFrame`). Every **5 seconds** (default), rotate:
-
-1. Push the current frame into a **send queue** (ring buffer).
-2. Create a new empty frame for incoming data.
-
-### Ring Buffer
-
-Use a fixed-capacity ring buffer (default: **12 frames**) for the send queue. When the buffer is full, the oldest frame is overwritten. This prevents unbounded memory growth if uploads fail repeatedly.
-
-### Upload Flow
-
-On each rotation tick:
-
-1. Check if the current frame has data and enough time has passed since it was created.
-2. Rotate: push current frame to send queue, set current to `nil`.
-3. Trigger upload: read all frames from the send queue, gzip-compress the JSON, POST to the API.
-4. On `200 OK`: remove the successfully sent frames from the send queue.
-5. On failure: frames remain in the send queue for retry on the next interval.
-
-### Upload Throttling
-
-Enforce a minimum gap between upload attempts (default: **2 seconds**). This prevents rapid-fire uploads if the collection interval is very short.
-
-### Metrics Collection
-
-System metrics (CPU, memory, goroutines, etc.) are collected on a separate interval (default: **30 seconds**) and written into the current collection frame like any other data.
-
-## Sampling
-
-SDKs should support two sampling rates:
-
-| Parameter | Range | Default | Description |
-|-----------|-------|---------|-------------|
-| `sampleRate` | 0.0–1.0 | 1.0 | Probability of recording a normal trace |
-| `errorSampleRate` | 0.0–1.0 | 1.0 | Probability of recording an error trace |
-
-### Sampling Logic
-
-A trace is considered an **error** if:
-- A panic was recovered during execution, OR
-- The HTTP status code is ≥ 500
-
-The sampling decision should be made **after** execution completes (so you know whether it's an error) but **before** recording:
-
-```
-isError = (panicRecovered || statusCode >= 500)
-rate = isError ? errorSampleRate : sampleRate
-
-if rate >= 1.0 → always record
-if rate <= 0.0 → never record
-else → record if random() < rate
-```
-
-When a trace is not sampled, do not send the trace or any associated exceptions.
-
-## Response Handling
-
-| Status | Meaning | Action |
-|--------|---------|--------|
-| `200` | Success | Remove sent frames from the send queue |
-| `400` | Malformed JSON, or a body that claims `Content-Encoding: gzip` but is not gzip | Fix the request; do not retry as-is |
-| `401` | Invalid or expired token | Check the project token configuration |
-| `413` | Payload too large after decompression | Drop or split the batch. Do **not** retry it as-is: the same body fails every time, and re-queueing it wedges the client permanently |
-| `429` | Rate limited | Back off and retry on the next interval |
-| Any other | Server error or network issue | Retry on the next interval |
-
-The `200 OK` response body is `{}` (empty JSON object).
-
-## Complete Example Payload
-
-Below is a full JSON payload (before gzip compression) demonstrating all three data types:
-
-```json
-{
- "collectionFrames": [
- {
- "stackTraces": [
- {
- "traceId": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
- "stackTrace": "*errors.errorString: connection refused\nhandleRequest()\n handler.go:42\nprocessConnection()\n server.go:128\n",
- "recordedAt": "2025-01-15T10:30:01.500Z",
- "attributes": {
- "user_id": "1234",
- "endpoint": "/api/users"
- },
- "isMessage": false
- },
- {
- "traceId": null,
- "stackTrace": "Deployment completed successfully for version 1.2.3",
- "recordedAt": "2025-01-15T10:30:02.000Z",
- "isMessage": true
- }
- ],
- "metrics": [
- {
- "name": "cpu.used_pcnt",
- "value": 45.2,
- "recordedAt": "2025-01-15T10:30:00Z"
- },
- {
- "name": "mem.used",
- "value": 256.5,
- "recordedAt": "2025-01-15T10:30:00Z"
- },
- {
- "name": "mem.total",
- "value": 8192.0,
- "recordedAt": "2025-01-15T10:30:00Z"
- },
- {
- "name": "go.go_routines",
- "value": 47.0,
- "recordedAt": "2025-01-15T10:30:00Z"
- },
- {
- "name": "queue.length",
- "value": 12.0,
- "recordedAt": "2025-01-15T10:30:00Z"
- }
- ],
- "traces": [
- {
- "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
- "endpoint": "GET /api/users/:id",
- "duration": 15234000,
- "recordedAt": "2025-01-15T10:30:00.123Z",
- "statusCode": 200,
- "bodySize": 1024,
- "clientIP": "192.168.1.100",
- "attributes": {
- "user_id": "1234"
- },
- "spans": [
- {
- "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
- "name": "db.query.find_user",
- "startTime": "2025-01-15T10:30:00.125Z",
- "duration": 5200000
- },
- {
- "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
- "name": "cache.set",
- "startTime": "2025-01-15T10:30:00.131Z",
- "duration": 800000
- }
- ]
- },
- {
- "id": "c3d4e5f6-a7b8-9012-cdef-123456789012",
- "endpoint": "POST /api/orders",
- "duration": 45000000,
- "recordedAt": "2025-01-15T10:30:00.200Z",
- "statusCode": 500,
- "bodySize": 256,
- "clientIP": "10.0.0.50"
- },
- {
- "id": "d4e5f6a7-b8c9-0123-defa-234567890123",
- "endpoint": "report.monthly",
- "duration": 3200000000,
- "recordedAt": "2025-01-15T10:30:00.300Z",
- "statusCode": 0,
- "bodySize": 0,
- "clientIP": "",
- "isTask": true,
- "attributes": {
- "report_type": "revenue"
- }
- }
- ]
- }
- ],
- "appVersion": "1.2.3",
- "serverName": "web-01"
-}
-```
-
-### Payload Notes
-
-- The first trace is a successful endpoint with two spans and attributes.
-- The second trace is a failed endpoint (status 500) with no spans or attributes (fields omitted).
-- The third trace is a task (`isTask: true`) with a 3.2-second duration.
-- The first exception links to the first trace via `traceId`. The second is a standalone message with no trace link.
-- Metrics include both predefined system metrics and a custom metric (`queue.length`).
diff --git a/docs/pages/server/_meta.json b/docs/pages/server/_meta.json
index e0596ec7..80a00ad7 100644
--- a/docs/pages/server/_meta.json
+++ b/docs/pages/server/_meta.json
@@ -12,5 +12,6 @@
"haloy": "Haloy",
"capacity-monitoring": "Capacity & Self-Monitoring",
"upgrade-1-8": "Upgrading to 1.8",
- "upgrade-request-limits": "Upgrade Notes: Request Limits"
+ "upgrade-request-limits": "Upgrade Notes: Request Limits",
+ "upgrade-v2-tables": "V2 Telemetry Migration"
}
diff --git a/docs/pages/server/capacity-monitoring.mdx b/docs/pages/server/capacity-monitoring.mdx
index d44575ed..f9c68625 100644
--- a/docs/pages/server/capacity-monitoring.mdx
+++ b/docs/pages/server/capacity-monitoring.mdx
@@ -15,7 +15,7 @@ MONITORING_TRACEWAY_URL="<project_token>@https://your-traceway-host/api/report"
That single variable turns everything on:
-- Every HTTP request to the backend is captured as a transaction (so endpoint latency/error-rate alert rules work on `POST /api/report` out of the box).
+- Every HTTP request to the backend is captured as a transaction (so endpoint latency/error-rate alert rules work on `POST /api/otel/v1/traces` out of the box).
- Runtime metrics are reported every 30s: `cpu.used_pcnt`, `mem.used`, `mem.total`, `go.go_routines`, `go.heap_objects`, `go.num_gc`, `go.gc_pause`.
- The backend reporters described below start emitting `traceway.*` metrics.
@@ -25,7 +25,7 @@ Note: `mem.used` is Go **heap** memory only. The real OOM-relevant number is `tr
### Ingestion (every batch)
-Emitted once per `/api/report` and OTLP request, tagged with `signal` and `table`:
+Emitted once per ingest request, tagged with `signal` and `table`:
| Metric | Meaning |
|--------|---------|
@@ -121,7 +121,7 @@ Create rules in the monitoring project under **Notifications** using the **Metri
Two more rule types complete the picture:
-- **Endpoint P95 threshold** on `POST /api/report`: client-visible ingest latency (the backend's own HTTP requests are transactions in the monitoring project).
+- **Endpoint P95 threshold** on `POST /api/otel/v1/traces`: client-visible ingest latency (the backend's own HTTP requests are transactions in the monitoring project).
- **No data** with data type `metrics`, which fires when the backend stops self-reporting entirely (crash, network partition).
## Recommended starting alerts
@@ -144,11 +144,11 @@ Two more rule types complete the picture:
| `traceway.proc.rss_mb` | avg | gt | 80% of host RAM | 10 | warning | Memory pressure / OOM risk (e.g. 800 on a 1 GB box) |
| `traceway.sourcemap.bytes` | avg | gt | ~half of `SOURCEMAP_CACHE_MAX_BYTES_MB`×1MB | 30 | info | Source map cache growing, revisit sizing |
-Plus the two non-metric rules: **Endpoint P95 threshold** on `POST /api/report` (e.g. 1000 ms over 10 minutes) and **No data** on `metrics` with 10 minutes of silence.
+Plus the two non-metric rules: **Endpoint P95 threshold** on `POST /api/otel/v1/traces` (e.g. 1000 ms over 10 minutes) and **No data** on `metrics` with 10 minutes of silence.
## Scale ClickHouse, scale the backend, or add Kafka?
-Ingestion in Traceway is synchronous: each `/api/report` request converts the payload and writes it to ClickHouse before responding. That makes the diagnosis straightforward: the metrics tell you which side of that pipeline is saturated, and whether the problem is sustained or bursty.
+Ingestion in Traceway is synchronous: each ingest request converts the payload and writes it to ClickHouse before responding. That makes the diagnosis straightforward: the metrics tell you which side of that pipeline is saturated, and whether the problem is sustained or bursty.
### Scale ClickHouse when the pressure is sustained and CH-side
@@ -171,12 +171,12 @@ Fix: a bigger box, or multiple backend instances behind a load balancer (the bac
### Add Kafka when the load is bursty and ClickHouse averages look healthy
-Because inserts happen inside the request, every ClickHouse hiccup or burst becomes client-visible `/api/report` latency. The tell-tale combination:
+Because inserts happen inside the request, every ClickHouse hiccup or burst becomes client-visible ingest latency. The tell-tale combination:
- `traceway.ingest.batch.insert_ms` p95 spiking while ClickHouse CPU, disk, and merges look calm on average
- `traceway.ch.pool.open` pinned at 15 (pool saturated) during peaks
- `traceway.recordings.dropped.delta` > 0; the one buffered path is already overflowing during bursts
-- P95 on `POST /api/report` rising even though total `traceway.ch.inserted_rows.delta` is well within what ClickHouse handles at steady state
+- P95 on `POST /api/otel/v1/traces` rising even though total `traceway.ch.inserted_rows.delta` is well within what ClickHouse handles at steady state
That pattern means ClickHouse can absorb your *average* write rate but not your *peaks*, and the synchronous path turns peaks into client errors and data loss. A queue (Kafka) between the HTTP handlers and ClickHouse decouples client latency from insert latency, absorbs spikes, and enables larger, better-shaped batches.
diff --git a/docs/pages/server/upgrade-v2-tables.mdx b/docs/pages/server/upgrade-v2-tables.mdx
new file mode 100644
index 00000000..384cd2d3
--- /dev/null
+++ b/docs/pages/server/upgrade-v2-tables.mdx
@@ -0,0 +1,217 @@
+# Migrating to V2 telemetry tables
+
+The V2 move-over supports **ClickHouse, SQLite and DuckDB**. It copies existing trace history into the new tables in the background while the backend accepts new telemetry. The same conversion and recovery logic runs on all three storage backends.
+
+Schema creation is automatic when the new backend starts. **Copying historical telemetry is opt-in:** set `V2_MOVE_OVER=true` if you want existing history to appear in the new dashboard. Without it, the five telemetry families below show only newly ingested data. Their old rows remain in the legacy tables, subject to existing retention.
+
+## Supported deployments
+
+| Telemetry backend | Where history is copied | Where progress is stored | History cutoff |
+|-------------------|-------------------------|--------------------------|----------------|
+| ClickHouse | Old and V2 tables in the configured ClickHouse database | PostgreSQL main database | `V2_MOVE_OVER_OLDEST`, if set; otherwise all available history |
+| SQLite | Old and V2 tables in the existing telemetry SQLite file | Main SQLite file, separate from telemetry | Configured oldest date and `SQLITE_RETENTION_DAYS` |
+| DuckDB | Old and V2 tables in the existing telemetry DuckDB file | Main SQLite file, not DuckDB | Configured oldest date and `DUCKDB_RETENTION_DAYS`, falling back to `SQLITE_RETENTION_DAYS` when unset |
+
+The backend selects the correct reader for its [storage build](/learn/build-tags). This is a schema migration within the same storage engine; it does not transfer data between SQLite, DuckDB and ClickHouse. Keep your existing database connections, database names and persistent volumes when upgrading.
+
+The worker runs inside the backend and reuses its database connections. There is no separate migration binary. In particular, do not open the live DuckDB file from another process to run the backfill.
+
+## What changes
+
+| Legacy table | V2 destination | Conversion |
+|--------------|----------------|------------|
+| `endpoints` | `endpoints_v2` and `spans_v2` | Preserve the endpoint occurrence and create its corresponding span |
+| `tasks` | `tasks_v2` and `spans_v2` | Preserve the task occurrence and create its corresponding span |
+| `ai_traces` | `ai_traces_v2` and `spans_v2` | Preserve the AI occurrence and create its corresponding span |
+| `exception_stack_traces` | `exceptions_v2` | Recover trace/span identity from the old owner where possible |
+| `spans` | `spans_v2` | Convert child-span identities and recover their owning trace |
+
+From the first start of the new release, new writes and dashboard/API reads for these five telemetry families use V2 only. Their notification rules also read V2, so evaluations can see incomplete historical windows until backfill reaches them. New and old backend versions write different tables; finish replacing old writers before copying history.
+
+Users, organizations, projects, tokens and configuration are retained. Logs, metrics, sessions, profiles and synthetic-check results are not copied by this worker. Blob storage is not moved: keep the existing local storage volume or S3 configuration, including AI conversation payloads and session recordings.
+
+### Trace identity schema cleanup
+
+Startup migrations also clean the active session, profile and V2 entity schemas, independently of `V2_MOVE_OVER`:
+
+- `sessions.trace_id` replaces `distributed_trace_id`. It stores lowercase 32-character hex, with an empty string for no association. Existing nonzero UUIDs keep the same 128 bits; null and zero IDs become empty strings. New protocol-v2 reports supply the actual W3C trace ID. A migration cannot reconstruct an OTel trace that an old client never recorded.
+- `endpoints_v2`, `tasks_v2`, `ai_traces_v2` and `exceptions_v2` lose `linked_trace_id` and its indexes. Actual span links remain in `spans_v2` and the OTLP payload.
+- `profiles` keeps its existing `trace_id` and `span_id`; the unused `distributed_trace_id` column and its ClickHouse index are dropped. Profiles stay in the same table.
+
+These are forward migrations (`ch/0091`–`0099`, `sqlite_telemetry/0026`, `duckdb_telemetry/0006`–`0007`), so databases that already ran the V2 preview are cleaned too. ClickHouse materializes the converted session IDs synchronously before removing the source column; this can add startup I/O and time proportional to session history. SQLite and DuckDB commit each migration together with its tracking record. DuckDB sets the final session non-null constraint in a separate migration because it cannot rebuild that constraint with outstanding updates in the same transaction.
+
+Stop all old backend instances before starting the first upgraded instance: their session/profile SQL refers to columns this release removes. Complete the database migrations before starting the other new instances, then deploy the updated SDK. This is a coordinated upgrade; switching back to the old binary alone is no longer sufficient for rollback.
+
+New accepted OTLP spans retain their original trace, span and parent IDs and lossless protobuf payloads. Migrated history can only recover what the old schema stored; see [historical-data differences](#what-migrated-history-looks-like).
+
+## Before upgrading
+
+1. Back up both the main database and telemetry database, and preserve blob storage and deployment configuration. For embedded deployments, stop the backend before taking a filesystem snapshot of its data directory, or use the storage engine's supported backup procedure.
+2. Allow disk space for both legacy and V2 data. V2 adds a span for every migrated endpoint, task and AI trace, so destination row counts and disk usage will not match the old tables one-to-one.
+3. Choose the oldest history you need and review retention before the first upgraded start. On SQLite and DuckDB, retention runs at startup and continues pruning both old and new tables. Backfill cannot restore already expired rows. Increase retention or set the applicable retention variable to `0` before startup if you need older history; `V2_MOVE_OVER_OLDEST` does not override retention.
+4. Upgrade the backend and dashboard together, and use a CLI/MCP release compatible with the changed API. Retain the previous image or binary and the backups for rollback.
+5. Stop every old backend writer before the first upgraded start, then let one upgraded instance complete startup migrations. With multiple replicas, enable the backfill worker on exactly one upgraded instance. The progress table does not provide a distributed lock.
+
+Fresh installations have no legacy history to copy and do not need `V2_MOVE_OVER`. Existing installations that do not need historical traces can also leave it unset; normal retention still applies to their old data.
+
+## Enable the backfill
+
+Add these settings to your existing backend configuration and restart or recreate that instance:
+
+```dotenv
+V2_MOVE_OVER=true
+# Optional: copy this UTC source date and newer dates.
+V2_MOVE_OVER_OLDEST=2026-09-01
+```
+
+The date is an example; replace it with your desired oldest day or omit it. For Docker Compose, add the settings to the backend service's existing `environment` block and recreate that service. A plain container restart does not change its environment. Preserve the service's other settings and volume mounts.
+
+| Variable | Default | Behavior |
+|----------|---------|----------|
+| `V2_MOVE_OVER` | unset; disabled | `true` enables one background worker in this backend process |
+| `V2_MOVE_OVER_OLDEST` | unset | Inclusive oldest UTC source day, formatted `YYYY-MM-DD`; older days are left in the old tables |
+
+An invalid oldest date is logged and the worker does not start. An unset, false or unparseable `V2_MOVE_OVER` leaves the worker disabled.
+
+A pass starts immediately when the backend starts. After a pass completes or returns an error, the worker waits one hour before the next pass. Restarting starts another pass immediately. A long pass never overlaps the next pass in the same process.
+
+On SQLite and DuckDB, the oldest source day is constrained by both the configured date and the retention window; the later cutoff wins. The retention cutoff is recalculated before each pass and rounded to a UTC day for copying. Normal retention can still prune individual rows within that boundary day. ClickHouse has no automatic backfill retention cutoff.
+
+## How copying works
+
+The worker processes the newest UTC source day first, then walks backward. Within each day it copies:
+
+```text
+endpoints → tasks → ai_traces → exception_stack_traces → spans
+```
+
+History becomes visible progressively as batches are inserted. An occurrence can appear before its child spans and exceptions have been copied; a day is not published atomically.
+
+Each primary source read returns at most 20,000 rows. It starts with a one-hour window. A full result may be truncated, so the worker halves the window and reads again before writing. Once a window is only one second wide, a full result is paged using a stable order over all copied source values. Sparse windows grow again.
+
+This bounds the source page size, not total memory or I/O. Large attributes, owner lookups, sorting and destination inserts still compete with live traffic. There is no configurable throughput limiter. Keep legacy source rows unchanged while copying: do not import, manually delete or mutate them during a pass. Account for automatic retention when selecting the history to preserve.
+
+## Progress and verification
+
+Progress is stored in `v2_move_over_days` in the **main database** shown in the deployment table. Query PostgreSQL for ClickHouse deployments, or the main SQLite file for SQLite/DuckDB deployments:
+
+```sql
+SELECT source_table, day, state, moved_rows, updated_at
+FROM v2_move_over_days
+ORDER BY day DESC, source_table;
+```
+
+| State | Meaning |
+|-------|---------|
+| No row | This table/day has not started, has no work in the scanned bounds, or is outside the requested range |
+| `started` | The table/day is in progress or was interrupted; inspect backend logs to distinguish them |
+| `done` | The pass completed this table/day without a reported error; later passes skip it |
+
+`updated_at` changes when a table/day is started or completed; it is not a per-batch heartbeat. `moved_rows` is recorded on completion and counts processed source rows, including rows skipped during recovery. It is not the number of newly inserted V2 rows or a live progress counter.
+
+Backend logs use the `move-over:` prefix. For example:
+
+```text
+move-over: 2026-09-19 spans: 23677 rows in 1.0s
+move-over: done, every day from 2026-09-19 back to 2026-09-01 is in the V2 tables
+```
+
+After completion, open historical endpoints, tasks, AI traces and issues in the dashboard using dates inside the copied range. Check their child spans, exception associations and AI conversation payloads. Also verify new telemetry is arriving. Counts alone are insufficient because occurrences generate additional span rows and recovery can coalesce identical legacy retries.
+
+Leave the setting enabled to retry remaining work hourly, or disable it and restart/recreate the worker instance after verification. Completed days remain recorded either way. Progress is currently exposed through logs and this table; there is no dedicated dashboard migration-progress page.
+
+## Stopping, failures and recovery
+
+Set `V2_MOVE_OVER=false` or remove it, then restart/recreate the worker instance to stop background copying. Already copied rows remain available. Re-enable it later to resume; retain the main database and its progress records along with the telemetry database.
+
+Before writing a table/day, the worker commits a `started` marker. It writes spans before their corresponding endpoint, task or AI rows, and records `done` after the day completes. A read, write or progress error stops the pass, logs the failure and leaves unfinished work resumable. A rejected canonical span also prevents that day from being marked complete. Persistent errors require correcting the cause; retrying does not bypass invalid data or insufficient disk space.
+
+Recovery rescans the unfinished day rather than resuming a saved page offset. It skips destination rows already found using these identities:
+
+- Occurrences: project, occurrence ID, trace ID, span ID and recorded time at the destination's precision.
+- Spans: project, trace ID and span ID.
+
+For example, if the process stops after writing an endpoint's span but before writing its endpoint row, recovery skips the existing span and inserts the missing endpoint. No transaction covers the main database and telemetry store together; the progress markers and identity checks bridge that gap. Identical legacy retries can be coalesced during recovery, so physical duplicate counts are not guaranteed to remain identical.
+
+### Rechecking a completed day
+
+A completed day is never rescanned automatically, even if an old writer or importer later adds rows. Earlier previews with ID-only paging or recovery may also have marked incomplete days as done.
+
+To recheck an affected day, stop the worker, back up the main and telemetry databases, and change the existing marker to `started` in the main database:
+
+```sql
+UPDATE v2_move_over_days
+SET state = 'started'
+WHERE source_table = 'endpoints' AND day = '2026-09-19';
+```
+
+Use the actual source table/date and verify that the update matched its progress row. Do not delete progress records: a missing marker selects the first-pass path without recovery checks. Re-enable the worker with a cutoff that includes the day. Only legacy data still present can be recovered. This checks for missing identities; it does not overwrite already migrated rows or repair changed payloads with the same identity.
+
+## What migrated history looks like
+
+Endpoint, task and AI occurrence IDs are preserved. Trace and span IDs are normalized to the V2 representation. The converter first uses an original OTel trace ID saved in legacy attributes, then an old distributed trace ID, then the occurrence's own ID. If the original OTel and browser correlation IDs differ, the original OTel trace ID wins; separate traces are not merged.
+
+An old OTel span ID stored as a zero-padded UUID becomes its original 16 hexadecimal characters. Native UUID span IDs retain 32 hexadecimal characters. Legacy OTel tasks move from their old end timestamp to their start timestamp by subtracting duration; native task timestamps stay unchanged.
+
+Old child spans and exceptions identify an owning occurrence rather than always carrying the original trace/span relationship. The converter searches nearby legacy endpoint, task and AI rows in the same project, extending the source window by 24 hours on either side. When an owner ID occurs more than once, it chooses the nearest timestamp. Missing owners, ambiguous IDs and long gaps can limit reconstruction; exceptions can remain without a recovered owner.
+
+Legacy endpoint/task/AI rows did not preserve their parent span IDs, so their converted spans have no recovered parent. Related occurrences may appear together on a distributed trace without nesting as new data does. Parent links that were retained on old child spans are converted where available.
+
+The original OTLP envelope was not retained in V1. Downloading a migrated span as OTLP returns a payload synthesized from the fields that survived: it cannot restore missing events, links, resource metadata or unknown protobuf fields. Existing AI conversation storage keys are preserved; their external payloads must still exist in blob storage.
+
+## Schema and storage details
+
+Startup applies the existing migration history followed by the new migrations. Do not reset the database's migration ledger to perform this upgrade.
+
+| Database | Added migrations |
+|----------|------------------|
+| ClickHouse telemetry | `0085`–`0089` create the five V2 tables; `0090` adds `span_version` |
+| SQLite telemetry | `0025` creates the V2 tables and indexes |
+| DuckDB telemetry | `0005` creates the V2 tables and indexes |
+| PostgreSQL main | `0143` creates `v2_move_over_days` |
+| SQLite main | `0077` creates `v2_move_over_days` |
+
+Table creation does not copy historical rows. Startup still depends on database availability and schema locks. The move-over neither rewrites nor drops the legacy tables, and it does not require switching storage engines or upgrading ClickHouse for these tables.
+
+ClickHouse V2 tables use ordinary `MergeTree` storage. Go converts and inserts each batch; no materialized view automatically copies old rows. `span_version` is a materialized 32-byte SHA-256 fingerprint computed on new inserts so consistent retry selection can avoid reading full protobuf payloads.
+
+If an earlier V2 preview already wrote parts before `0090`, ClickHouse can compute the missing fingerprint while reading them. Those reads still incur payload cost until the column is stored through a merge that materializes it or an explicit mutation. Operators may schedule this after considering I/O and disk load:
+
+```sql
+ALTER TABLE spans_v2 MATERIALIZE COLUMN span_version;
+```
+
+This is an optional ClickHouse operation, not the historical backfill, and the upgrade does not run it automatically. It is unnecessary for newly backfilled rows written after `0090`. Equal-duration conflicting retries can select a different version than an earlier preview; current ClickHouse readers use the same winner order.
+
+### Retention and cleanup
+
+Copied history occupies space in both old and new tables until retention or an operator removes the old copy. V2 stores additional spans and synthesized payloads, so plan capacity from your workload rather than assuming exactly twice the old size. ClickHouse per-table part and row metrics distinguish legacy and V2 tables by the `table` tag.
+
+The shipped ClickHouse V2 tables have no TTL. Any TTL or cleanup policy you added to legacy tables must be reviewed and applied deliberately to the new tables. SQLite/DuckDB retention covers both legacy and V2 tables; their old tables are not permanent backups.
+
+Verify the copied history and retain a recoverable backup before considering legacy cleanup. This release does not automatically drop legacy tables. Do not drop them while the worker is enabled: subsequent passes still query their bounds. Dropping legacy data also removes the previous release's ability to display that history; further schema cleanup should follow an explicit upgrade procedure.
+
+## Rolling back
+
+Stop all backend instances and restore the pre-upgrade databases before restarting the previous release. The previous binary references removed session/profile columns; preserving the old trace tables does not make it compatible with the cleaned schema. There is no reverse backfill from V2 to the legacy trace tables.
+
+Running the old backend again writes to old tables. If you later return to V2, affected source days already marked `done` need explicit rechecking after old writers have stopped. Do not run both versions as a steady-state deployment: they read and write different trace datasets.
+
+A database restore returns to the snapshot's state and loses later writes unless separately preserved. Check the rest of the target release's upgrade notes too; preservation of these telemetry tables is not a guarantee that every other schema or application change supports an arbitrary downgrade.
+
+## API and SDK compatibility
+
+Existing OTLP and native `/api/report` ingestion formats remain supported. HTTP API consumers and the CLI/MCP need to account for these response changes:
+
+- Endpoints, tasks and AI traces carry `traceId`, `spanId` and `parentSpanId` as hex strings. Their `distributedTraceId` field is gone.
+- Spans carry `traceId`, `spanId` and `parentSpanId`; `id`, `otelTraceId` and `distributedTraceId` are gone.
+- An exception's `traceId` is the trace ID, and `spanId` identifies its span. The recovered endpoint, task or AI owner is returned as `relatedEntity` on exception responses.
+- `POST /api/distributed-traces/:traceId` accepts a hex trace ID or dashed UUID. Pass an RFC 3339 `recordedAt` for historical traces; omitting it anchors entity lookup on now, 48 hours either side. Malformed timestamps return 400. The response uses `traceId` instead of `distributedTraceId`.
+- `POST /api/logs` uses `traceId` and optional `spanId`, within the selected project. Retired `distributedTraceId` and `excludeTraceId` filters return 400.
+- The retired span attribute `traceway.distributed_trace_id` no longer joins traces. Use W3C context propagation to share a trace ID, or actual OTLP span links to express cross-trace relationships.
+
+Native UUID span IDs remain unchanged in Traceway's API. OTLP export maps them to stable eight-byte span/parent IDs and includes the originals in `traceway.native.span_id` and `traceway.native.parent_span_id`. Actual OTLP input retains its original payload.
+
+Span search, topology, attributes and export choose one deterministic retry version: longest duration first, with the stored fingerprint breaking ties on ClickHouse and stored protobuf bytes breaking ties on SQLite/DuckDB. This does not make all occurrence counts exactly-once. Graph reads also apply row, time and attribute-byte limits. Detail responses expose `spanGraphStatus`; a partial graph can omit exceptions on missing descendant spans, so an empty exception list does not establish that the occurrence succeeded.
+
+Active session, profile and V2 entity tables have no `distributed_trace_id` or `linked_trace_id` columns after startup migrations complete. Historical SQL migrations and the old trace tables still contain the old names because the move-over reads that history. Legacy native-report fields remain only at the decoder boundary. `/api/report` retains the existing SDK wire format; these storage migrations do not introduce a new report protocol. True OTel span links remain intact.
diff --git a/docs/pages/symbolicator/index.mdx b/docs/pages/symbolicator/index.mdx
index 3828ac32..35e55f9d 100644
--- a/docs/pages/symbolicator/index.mdx
+++ b/docs/pages/symbolicator/index.mdx
@@ -4,7 +4,7 @@ Symbolication turns a minified or stripped production stack trace back into the
The same engine runs three ways:
-- **At ingest**, inside Traceway's `/api/report` and OTLP endpoints, so issues are stored already symbolicated.
+- **At ingest**, inside Traceway, so issues are stored already symbolicated.
- **As an [OpenTelemetry Collector processor](/symbolicator/opentelemetry)**, so traces are rewritten before they reach any backend.
- **As an importable [Go library](/symbolicator/library)**, so you can call it from your own tooling.
diff --git a/docs/public/_redirects b/docs/public/_redirects
index 4a5afd4c..9d16700a 100644
--- a/docs/public/_redirects
+++ b/docs/public/_redirects
@@ -42,3 +42,5 @@
/client/fasthttp-middleware/* /client/otel 301
/client/http-middleware /client/otel 301
/client/http-middleware/* /client/otel 301
+/protocol /client/otel 301
+/protocol/* /client/otel 301
diff --git a/examples/devtesting-embedded/README.md b/examples/devtesting-embedded/README.md
index d0a1daf7..2aef6ee1 100644
--- a/examples/devtesting-embedded/README.md
+++ b/examples/devtesting-embedded/README.md
@@ -64,7 +64,7 @@ Five endpoints exercise different parts of the logs feature end-to-end:
| `GET /api/test-success` | DEBUG + INFO logs on a simple successful request |
| `GET /api/test-log-levels` | One log at each severity (TRACE/DEBUG/INFO/WARN/ERROR/FATAL) — validates `SeverityBadge` rendering |
| `GET /api/test-spans-with-logs` | Nested child spans (`auth.verify` → `db.query` → `cache.lookup`) with logs emitted inside each and attributes on every child span. Exercises `parent_span_id` capture, the span tree + attribute popover on the trace detail page, and span-chip linking |
-| `GET /api/test-distributed-logs` | Two services (`backend-service` + `worker-service`) emit logs under a shared `traceway.distributed_trace_id`. Powers the "Load logs from other traces" button on the trace detail page |
+| `GET /api/test-distributed-logs` | Two services (`backend-service` + `worker-service`) emit logs under a shared W3C trace ID. Both services appear in the trace log panel |
Trigger them all at once:
diff --git a/examples/devtesting-embedded/main.go b/examples/devtesting-embedded/main.go
index 92828a82..09705457 100644
--- a/examples/devtesting-embedded/main.go
+++ b/examples/devtesting-embedded/main.go
@@ -21,6 +21,7 @@ import (
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
otellog "go.opentelemetry.io/otel/log"
+ "go.opentelemetry.io/otel/propagation"
sdklog "go.opentelemetry.io/otel/sdk/log"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
@@ -201,7 +202,7 @@ func main() {
c.Next()
})
- router.Use(otelgin.Middleware(backendServiceName, otelgin.WithTracerProvider(backendSvc.tp)))
+ router.Use(otelgin.Middleware(backendServiceName, otelgin.WithTracerProvider(backendSvc.tp), otelgin.WithPropagators(propagation.TraceContext{})))
router.GET("/", func(c *gin.Context) {
c.Data(http.StatusOK, "text/html; charset=utf-8", indexHTML)
@@ -289,20 +290,16 @@ func main() {
router.GET("/api/test-distributed-logs", func(c *gin.Context) {
ctx := c.Request.Context()
- dtid := uuid.New().String()
-
- rootSpan := trace.SpanFromContext(ctx)
- rootSpan.SetAttributes(attribute.String("traceway.distributed_trace_id", dtid))
+ traceId := trace.SpanContextFromContext(ctx).TraceID().String()
backendSvc.log(ctx, otellog.SeverityInfo, "INFO", "backend: received request, about to call worker",
- otellog.String("distributed_trace_id", dtid))
+ otellog.String("trace_id", traceId))
- workerCtx, workerSpan := workerSvc.tr.Start(context.Background(), "worker.process-job",
+ workerCtx, workerSpan := workerSvc.tr.Start(ctx, "worker.process-job",
trace.WithSpanKind(trace.SpanKindConsumer))
- workerSpan.SetAttributes(attribute.String("traceway.distributed_trace_id", dtid))
workerSvc.log(workerCtx, otellog.SeverityInfo, "INFO", "worker: starting job",
- otellog.String("distributed_trace_id", dtid))
+ otellog.String("trace_id", traceId))
time.Sleep(15 * time.Millisecond)
workerSvc.log(workerCtx, otellog.SeverityDebug, "DEBUG", "worker: step 1 complete")
time.Sleep(15 * time.Millisecond)
@@ -313,8 +310,8 @@ func main() {
backendSvc.log(ctx, otellog.SeverityInfo, "INFO", "backend: worker reported success, returning 200")
c.JSON(http.StatusOK, gin.H{
- "status": "ok",
- "distributedTraceId": dtid,
+ "status": "ok",
+ "traceId": traceId,
})
})
diff --git a/examples/devtesting-embedded/static/app.js b/examples/devtesting-embedded/static/app.js
index 3e6f2fdb..a3f06566 100644
--- a/examples/devtesting-embedded/static/app.js
+++ b/examples/devtesting-embedded/static/app.js
@@ -1,63 +1,139 @@
-var jf=Object.defineProperty;var Hf=(ce,ge,J)=>ge in ce?jf(ce,ge,{enumerable:!0,configurable:!0,writable:!0,value:J}):ce[ge]=J;var v=(ce,ge,J)=>Hf(ce,typeof ge!="symbol"?ge+"":ge,J);(function(){"use strict";function ce(){return typeof crypto<"u"&&typeof crypto.randomUUID=="function"?crypto.randomUUID():"xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g,s=>{const e=Math.random()*16|0;return(s==="x"?e:e&3|8).toString(16)})}function ge(s){const e=s.indexOf("@");if(e===-1)throw new Error("Invalid connection string: must be in format {token}@{apiUrl}");const t=s.slice(0,e),r=s.slice(e+1);if(!t||!r)throw new Error("Invalid connection string: token and apiUrl must not be empty");return{token:t,apiUrl:r}}function J(){return new Date().toISOString()}var Vs=class{constructor(s={}){v(this,"windowMs");v(this,"maxSize");v(this,"q",[]);this.windowMs=s.windowMs??1e4,this.maxSize=s.maxSize??200}add(s){this.q.push(s),this.prune()}snapshot(){return this.prune(),this.q.slice()}clear(){this.q.length=0}get length(){return this.q.length}prune(){const s=Date.now()-this.windowMs;for(;this.q.length>0&&Date.parse(this.q[0].timestamp)<s;)this.q.shift();for(;this.q.length>this.maxSize;)this.q.shift()}},Xo=Object.defineProperty,Qo=(s,e,t)=>e in s?Xo(s,e,{enumerable:!0,configurable:!0,writable:!0,value:t}):s[e]=t,w=(s,e,t)=>Qo(s,typeof e!="symbol"?e+"":e,t),js,qo=Object.defineProperty,ea=(s,e,t)=>e in s?qo(s,e,{enumerable:!0,configurable:!0,writable:!0,value:t}):s[e]=t,Hs=(s,e,t)=>ea(s,typeof e!="symbol"?e+"":e,t),j=(s=>(s[s.Document=0]="Document",s[s.DocumentType=1]="DocumentType",s[s.Element=2]="Element",s[s.Text=3]="Text",s[s.CDATA=4]="CDATA",s[s.Comment=5]="Comment",s))(j||{});const Ys={Node:["childNodes","parentNode","parentElement","textContent","ownerDocument"],ShadowRoot:["host","styleSheets"],Element:["shadowRoot","querySelector","querySelectorAll"],MutationObserver:[]},Zs={Node:["contains","getRootNode"],ShadowRoot:["getSelection"],Element:[],MutationObserver:["constructor"]},gt={},ta=()=>!!globalThis.Zone;function Cr(s){if(gt[s])return gt[s];const e=globalThis[s],t=e.prototype,r=s in Ys?Ys[s]:void 0,i=!!(r&&r.every(l=>{var a,u;return!!((u=(a=Object.getOwnPropertyDescriptor(t,l))==null?void 0:a.get)!=null&&u.toString().includes("[native code]"))})),n=s in Zs?Zs[s]:void 0,o=!!(n&&n.every(l=>{var a;return typeof t[l]=="function"&&((a=t[l])==null?void 0:a.toString().includes("[native code]"))}));if(i&&o&&!ta())return gt[s]=e.prototype,e.prototype;try{const l=document.createElement("iframe");document.body.appendChild(l);const a=l.contentWindow;if(!a)return e.prototype;const u=a[s].prototype;return document.body.removeChild(l),u?gt[s]=u:t}catch{return t}}const Ir={};function ye(s,e,t){var r;const i=`${s}.${String(t)}`;if(Ir[i])return Ir[i].call(e);const n=Cr(s),o=(r=Object.getOwnPropertyDescriptor(n,t))==null?void 0:r.get;return o?(Ir[i]=o,o.call(e)):e[t]}const xr={};function Js(s,e,t){const r=`${s}.${String(t)}`;if(xr[r])return xr[r].bind(e);const n=Cr(s)[t];return typeof n!="function"?e[t]:(xr[r]=n,n.bind(e))}function ra(s){return ye("Node",s,"ownerDocument")}function sa(s){return ye("Node",s,"childNodes")}function ia(s){return ye("Node",s,"parentNode")}function na(s){return ye("Node",s,"parentElement")}function oa(s){return ye("Node",s,"textContent")}function aa(s,e){return Js("Node",s,"contains")(e)}function la(s){return Js("Node",s,"getRootNode")()}function ua(s){return!s||!("host"in s)?null:ye("ShadowRoot",s,"host")}function ca(s){return s.styleSheets}function ha(s){return!s||!("shadowRoot"in s)?null:ye("Element",s,"shadowRoot")}function fa(s,e){return ye("Element",s,"querySelector")(e)}function da(s,e){return ye("Element",s,"querySelectorAll")(e)}function pa(){return Cr("MutationObserver").constructor}function ma(s,e,t){try{if(!(e in s))return()=>{};const r=s[e],i=t(r);return typeof i=="function"&&(i.prototype=i.prototype||{},Object.defineProperties(i,{__rrweb_original__:{enumerable:!1,value:r}})),s[e]=i,()=>{s[e]=r}}catch{return()=>{}}}const K={ownerDocument:ra,childNodes:sa,parentNode:ia,parentElement:na,textContent:oa,contains:aa,getRootNode:la,host:ua,styleSheets:ca,shadowRoot:ha,querySelector:fa,querySelectorAll:da,mutationObserver:pa,patch:ma};function Ks(s){return s.nodeType===s.ELEMENT_NODE}function He(s){const e=s&&"host"in s&&"mode"in s&&K.host(s)||null;return!!(e&&"shadowRoot"in e&&K.shadowRoot(e)===s)}function Ye(s){return Object.prototype.toString.call(s)==="[object ShadowRoot]"}function ga(s){return s.includes(" background-clip: text;")&&!s.includes(" -webkit-background-clip: text;")&&(s=s.replace(/\sbackground-clip:\s*text;/g," -webkit-background-clip: text; background-clip: text;")),s}function ya(s){const{cssText:e}=s;if(e.split('"').length<3)return e;const t=["@import",`url(${JSON.stringify(s.href)})`];return s.layerName===""?t.push("layer"):s.layerName&&t.push(`layer(${s.layerName})`),s.supportsText&&t.push(`supports(${s.supportsText})`),s.media.length&&t.push(s.media.mediaText),t.join(" ")+";"}function Ar(s){try{const e=s.rules||s.cssRules;if(!e)return null;let t=s.href;!t&&s.ownerNode&&(t=s.ownerNode.baseURI);const r=Array.from(e,i=>Xs(i,t)).join("");return ga(r)}catch{return null}}function Xs(s,e){if(Sa(s)){let t;try{t=Ar(s.styleSheet)||ya(s)}catch{t=s.cssText}return s.styleSheet.href?St(t,s.styleSheet.href):t}else{let t=s.cssText;return ba(s)&&s.selectorText.includes(":")&&(t=wa(t)),e?St(t,e):t}}function wa(s){const e=/(\[(?:[\w-]+)[^\\])(:(?:[\w-]+)\])/gm;return s.replace(e,"$1\\$2")}function Sa(s){return"styleSheet"in s}function ba(s){return"selectorText"in s}class Qs{constructor(){Hs(this,"idNodeMap",new Map),Hs(this,"nodeMetaMap",new WeakMap)}getId(e){var t;return e?((t=this.getMeta(e))==null?void 0:t.id)??-1:-1}getNode(e){return this.idNodeMap.get(e)||null}getIds(){return Array.from(this.idNodeMap.keys())}getMeta(e){return this.nodeMetaMap.get(e)||null}removeNodeFromMap(e){const t=this.getId(e);this.idNodeMap.delete(t),e.childNodes&&e.childNodes.forEach(r=>this.removeNodeFromMap(r))}has(e){return this.idNodeMap.has(e)}hasNode(e){return this.nodeMetaMap.has(e)}add(e,t){const r=t.id;this.idNodeMap.set(r,e),this.nodeMetaMap.set(e,t)}replace(e,t){const r=this.getNode(e);if(r){const i=this.nodeMetaMap.get(r);i&&this.nodeMetaMap.set(t,i)}this.idNodeMap.set(e,t)}reset(){this.idNodeMap=new Map,this.nodeMetaMap=new WeakMap}}function va(){return new Qs}function yt({element:s,maskInputOptions:e,tagName:t,type:r,value:i,maskInputFn:n}){let o=i||"";const l=r&&Ie(r);return(e[t.toLowerCase()]||l&&e[l])&&(n?o=n(o,s):o="*".repeat(o.length)),o}function Ie(s){return s.toLowerCase()}const qs="__rrweb_original__";function Ca(s){const e=s.getContext("2d");if(!e)return!0;const t=50;for(let r=0;r<s.width;r+=t)for(let i=0;i<s.height;i+=t){const n=e.getImageData,o=qs in n?n[qs]:n;if(new Uint32Array(o.call(e,r,i,Math.min(t,s.width-r),Math.min(t,s.height-i)).data.buffer).some(a=>a!==0))return!1}return!0}function wt(s){const e=s.type;return s.hasAttribute("data-rr-is-password")?"password":e?Ie(e):null}function ei(s,e){let t;try{t=new URL(s,e??window.location.href)}catch{return null}const r=/\.([0-9a-z]+)(?:$)/i,i=t.pathname.match(r);return(i==null?void 0:i[1])??null}function Ia(s){let e="";return s.indexOf("//")>-1?e=s.split("/").slice(0,3).join("/"):e=s.split("/")[0],e=e.split("?")[0],e}const xa=/url\((?:(')([^']*)'|(")(.*?)"|([^)]*))\)/gm,Aa=/^(?:[a-z+]+:)?\/\//i,Ra=/^www\..*/i,Oa=/^(data:)([^,]*),(.*)/i;function St(s,e){return(s||"").replace(xa,(t,r,i,n,o,l)=>{const a=i||o||l,u=r||n||"";if(!a)return t;if(Aa.test(a)||Ra.test(a))return`url(${u}${a}${u})`;if(Oa.test(a))return`url(${u}${a}${u})`;if(a[0]==="/")return`url(${u}${Ia(e)+a}${u})`;const c=e.split("/"),h=a.split("/");c.pop();for(const p of h)p!=="."&&(p===".."?c.pop():c.push(p));return`url(${u}${c.join("/")}${u})`})}function bt(s,e=!1){return e?s.replace(/(\/\*[^*]*\*\/)|[\s;]/g,""):s.replace(/(\/\*[^*]*\*\/)|[\s;]/g,"").replace(/0px/g,"0")}function Ea(s,e,t=!1){const r=Array.from(e.childNodes),i=[];let n=0;if(r.length>1&&s&&typeof s=="string"){let o=bt(s,t);const l=o.length/s.length;for(let a=1;a<r.length;a++)if(r[a].textContent&&typeof r[a].textContent=="string"){const u=bt(r[a].textContent,t),c=100;let h=3;for(;h<u.length&&(u[h].match(/[a-zA-Z0-9]/)||u.indexOf(u.substring(0,h),1)!==-1);h++);for(;h<u.length;h++){let p=u.substring(0,h),m=o.split(p),g=-1;if(m.length===2)g=m[0].length;else if(m.length>2&&m[0]===""&&r[a-1].textContent!=="")g=o.indexOf(p,1);else if(m.length===1){if(p=p.substring(0,p.length-1),m=o.split(p),m.length<=1)return i.push(s),i;h=c+1}else h===u.length-1&&(g=o.indexOf(p));if(m.length>=2&&h>c){const d=r[a-1].textContent;if(d&&typeof d=="string"){const f=bt(d).length;g=o.indexOf(p,f)}g===-1&&(g=m[0].length)}if(g!==-1){let d=Math.floor(g/l);for(;d>0&&d<s.length;){if(n+=1,n>50*r.length)return i.push(s),i;const f=bt(s.substring(0,d),t);if(f.length===g){i.push(s.substring(0,d)),s=s.substring(d),o=o.substring(g);break}else f.length<g?d+=Math.max(1,Math.floor((g-f.length)/l)):d-=Math.max(1,Math.floor((f.length-g)*l))}break}}}}return i.push(s),i}function Ma(s,e){return Ea(s,e).join("/* rr_split */")}let $a=1;const Na=new RegExp("[^a-z0-9-_:]"),Ze=-2;function ti(){return $a++}function ka(s){if(s instanceof HTMLFormElement)return"form";const e=Ie(s.tagName);return Na.test(e)?"div":e}let ke,ri;const Ta=/^[^ \t\n\r\u000c]+/,Pa=/^[, \t\n\r\u000c]+/;function Da(s,e){if(e.trim()==="")return e;let t=0;function r(n){let o;const l=n.exec(e.substring(t));return l?(o=l[0],t+=o.length,o):""}const i=[];for(;r(Pa),!(t>=e.length);){let n=r(Ta);if(n.slice(-1)===",")n=Te(s,n.substring(0,n.length-1)),i.push(n);else{let o="";n=Te(s,n);let l=!1;for(;;){const a=e.charAt(t);if(a===""){i.push((n+o).trim());break}else if(l)a===")"&&(l=!1);else if(a===","){t+=1,i.push((n+o).trim());break}else a==="("&&(l=!0);o+=a,t+=1}}}return i.join(", ")}const si=new WeakMap;function Te(s,e){return!e||e.trim()===""?e:Rr(s,e)}function La(s){return!!(s.tagName==="svg"||s.ownerSVGElement)}function Rr(s,e){let t=si.get(s);if(t||(t=s.createElement("a"),si.set(s,t)),!e)e="";else if(e.startsWith("blob:")||e.startsWith("data:"))return e;return t.setAttribute("href",e),t.href}function ii(s,e,t,r){return r&&(t==="src"||t==="href"&&!(e==="use"&&r[0]==="#")||t==="xlink:href"&&r[0]!=="#"||t==="background"&&["table","td","th"].includes(e)?Te(s,r):t==="srcset"?Da(s,r):t==="style"?St(r,Rr(s)):e==="object"&&t==="data"?Te(s,r):r)}function ni(s,e,t){return["video","audio"].includes(s)&&e==="autoplay"}function _a(s,e,t){try{if(typeof e=="string"){if(s.classList.contains(e))return!0}else for(let r=s.classList.length;r--;){const i=s.classList[r];if(e.test(i))return!0}if(t)return s.matches(t)}catch{}return!1}function vt(s,e,t){if(!s)return!1;if(s.nodeType!==s.ELEMENT_NODE)return t?vt(K.parentNode(s),e,t):!1;for(let r=s.classList.length;r--;){const i=s.classList[r];if(e.test(i))return!0}return t?vt(K.parentNode(s),e,t):!1}function oi(s,e,t,r){let i;if(Ks(s)){if(i=s,!K.childNodes(i).length)return!1}else{if(K.parentElement(s)===null)return!1;i=K.parentElement(s)}try{if(typeof e=="string"){if(r){if(i.closest(`.${e}`))return!0}else if(i.classList.contains(e))return!0}else if(vt(i,e,r))return!0;if(t){if(r){if(i.closest(t))return!0}else if(i.matches(t))return!0}}catch{}return!1}function Fa(s,e,t){const r=s.contentWindow;if(!r)return;let i=!1,n;try{n=r.document.readyState}catch{return}if(n!=="complete"){const l=setTimeout(()=>{i||(e(),i=!0)},t);s.addEventListener("load",()=>{clearTimeout(l),i=!0,e()});return}const o="about:blank";if(r.location.href!==o||s.src===o||s.src==="")return setTimeout(e,0),s.addEventListener("load",e);s.addEventListener("load",e)}function Ua(s,e,t){let r=!1,i;try{i=s.sheet}catch{return}if(i)return;const n=setTimeout(()=>{r||(e(),r=!0)},t);s.addEventListener("load",()=>{clearTimeout(n),r=!0,e()})}function Ba(s,e){const{doc:t,mirror:r,blockClass:i,blockSelector:n,needsMask:o,inlineStylesheet:l,maskInputOptions:a={},maskTextFn:u,maskInputFn:c,dataURLOptions:h={},inlineImages:p,recordCanvas:m,keepIframeSrcFn:g,newlyAddedElement:d=!1,cssCaptured:f=!1}=e,S=za(t,r);switch(s.nodeType){case s.DOCUMENT_NODE:return s.compatMode!=="CSS1Compat"?{type:j.Document,childNodes:[],compatMode:s.compatMode}:{type:j.Document,childNodes:[]};case s.DOCUMENT_TYPE_NODE:return{type:j.DocumentType,name:s.name,publicId:s.publicId,systemId:s.systemId,rootId:S};case s.ELEMENT_NODE:return Ga(s,{doc:t,blockClass:i,blockSelector:n,inlineStylesheet:l,maskInputOptions:a,maskInputFn:c,dataURLOptions:h,inlineImages:p,recordCanvas:m,keepIframeSrcFn:g,newlyAddedElement:d,rootId:S});case s.TEXT_NODE:return Wa(s,{doc:t,needsMask:o,maskTextFn:u,rootId:S,cssCaptured:f});case s.CDATA_SECTION_NODE:return{type:j.CDATA,textContent:"",rootId:S};case s.COMMENT_NODE:return{type:j.Comment,textContent:K.textContent(s)||"",rootId:S};default:return!1}}function za(s,e){if(!e.hasNode(s))return;const t=e.getId(s);return t===1?void 0:t}function Wa(s,e){const{needsMask:t,maskTextFn:r,rootId:i,cssCaptured:n}=e,o=K.parentNode(s),l=o&&o.tagName;let a="";const u=l==="STYLE"?!0:void 0,c=l==="SCRIPT"?!0:void 0;return c?a="SCRIPT_PLACEHOLDER":n||(a=K.textContent(s),u&&a&&(a=St(a,Rr(e.doc)))),!u&&!c&&a&&t&&(a=r?r(a,K.parentElement(s)):a.replace(/[\S]/g,"*")),{type:j.Text,textContent:a||"",rootId:i}}function Ga(s,e){const{doc:t,blockClass:r,blockSelector:i,inlineStylesheet:n,maskInputOptions:o={},maskInputFn:l,dataURLOptions:a={},inlineImages:u,recordCanvas:c,keepIframeSrcFn:h,newlyAddedElement:p=!1,rootId:m}=e,g=_a(s,r,i),d=ka(s);let f={};const S=s.attributes.length;for(let y=0;y<S;y++){const C=s.attributes[y];ni(d,C.name,C.value)||(f[C.name]=ii(t,d,Ie(C.name),C.value))}if(d==="link"&&n){const y=Array.from(t.styleSheets).find(k=>k.href===s.href);let C=null;y&&(C=Ar(y)),C&&(delete f.rel,delete f.href,f._cssText=C)}if(d==="style"&&s.sheet){let y=Ar(s.sheet);y&&(s.childNodes.length>1&&(y=Ma(y,s)),f._cssText=y)}if(["input","textarea","select"].includes(d)){const y=s.value,C=s.checked;f.type!=="radio"&&f.type!=="checkbox"&&f.type!=="submit"&&f.type!=="button"&&y?f.value=yt({element:s,type:wt(s),tagName:d,value:y,maskInputOptions:o,maskInputFn:l}):C&&(f.checked=C)}if(d==="option"&&(s.selected&&!o.select?f.selected=!0:delete f.selected),d==="dialog"&&s.open&&(f.rr_open_mode=s.matches("dialog:modal")?"modal":"non-modal"),d==="canvas"&&c){if(s.__context==="2d")Ca(s)||(f.rr_dataURL=s.toDataURL(a.type,a.quality));else if(!("__context"in s)){const y=s.toDataURL(a.type,a.quality),C=t.createElement("canvas");C.width=s.width,C.height=s.height;const k=C.toDataURL(a.type,a.quality);y!==k&&(f.rr_dataURL=y)}}if(d==="img"&&u){ke||(ke=t.createElement("canvas"),ri=ke.getContext("2d"));const y=s,C=y.currentSrc||y.getAttribute("src")||"<unknown-src>",k=y.crossOrigin,D=()=>{y.removeEventListener("load",D);try{ke.width=y.naturalWidth,ke.height=y.naturalHeight,ri.drawImage(y,0,0),f.rr_dataURL=ke.toDataURL(a.type,a.quality)}catch(H){if(y.crossOrigin!=="anonymous"){y.crossOrigin="anonymous",y.complete&&y.naturalWidth!==0?D():y.addEventListener("load",D);return}else console.warn(`Cannot inline img src=${C}! Error: ${H}`)}y.crossOrigin==="anonymous"&&(k?f.crossOrigin=k:y.removeAttribute("crossorigin"))};y.complete&&y.naturalWidth!==0?D():y.addEventListener("load",D)}if(["audio","video"].includes(d)){const y=f;y.rr_mediaState=s.paused?"paused":"played",y.rr_mediaCurrentTime=s.currentTime,y.rr_mediaPlaybackRate=s.playbackRate,y.rr_mediaMuted=s.muted,y.rr_mediaLoop=s.loop,y.rr_mediaVolume=s.volume}if(p||(s.scrollLeft&&(f.rr_scrollLeft=s.scrollLeft),s.scrollTop&&(f.rr_scrollTop=s.scrollTop)),g){const{width:y,height:C}=s.getBoundingClientRect();f={class:f.class,rr_width:`${y}px`,rr_height:`${C}px`}}d==="iframe"&&!h(f.src)&&(s.contentDocument||(f.rr_src=f.src),delete f.src);let b;try{customElements.get(d)&&(b=!0)}catch{}return{type:j.Element,tagName:d,attributes:f,childNodes:[],isSVG:La(s)||void 0,needBlock:g,rootId:m,isCustom:b}}function P(s){return s==null?"":s.toLowerCase()}function ai(s){return s===!0||s==="all"?{script:!0,comment:!0,headFavicon:!0,headWhitespace:!0,headMetaSocial:!0,headMetaRobots:!0,headMetaHttpEquiv:!0,headMetaVerification:!0,headMetaAuthorship:s==="all",headMetaDescKeywords:s==="all",headTitleMutations:s==="all"}:s||{}}function Va(s,e){if(e.comment&&s.type===j.Comment)return!0;if(s.type===j.Element){if(e.script&&(s.tagName==="script"||s.tagName==="link"&&(s.attributes.rel==="preload"&&s.attributes.as==="script"||s.attributes.rel==="modulepreload")||s.tagName==="link"&&s.attributes.rel==="prefetch"&&typeof s.attributes.href=="string"&&ei(s.attributes.href)==="js"))return!0;if(e.headFavicon&&(s.tagName==="link"&&s.attributes.rel==="shortcut icon"||s.tagName==="meta"&&(P(s.attributes.name).match(/^msapplication-tile(image|color)$/)||P(s.attributes.name)==="application-name"||P(s.attributes.rel)==="icon"||P(s.attributes.rel)==="apple-touch-icon"||P(s.attributes.rel)==="shortcut icon")))return!0;if(s.tagName==="meta"){if(e.headMetaDescKeywords&&P(s.attributes.name).match(/^description|keywords$/))return!0;if(e.headMetaSocial&&(P(s.attributes.property).match(/^(og|twitter|fb):/)||P(s.attributes.name).match(/^(og|twitter):/)||P(s.attributes.name)==="pinterest"))return!0;if(e.headMetaRobots&&(P(s.attributes.name)==="robots"||P(s.attributes.name)==="googlebot"||P(s.attributes.name)==="bingbot"))return!0;if(e.headMetaHttpEquiv&&s.attributes["http-equiv"]!==void 0)return!0;if(e.headMetaAuthorship&&(P(s.attributes.name)==="author"||P(s.attributes.name)==="generator"||P(s.attributes.name)==="framework"||P(s.attributes.name)==="publisher"||P(s.attributes.name)==="progid"||P(s.attributes.property).match(/^article:/)||P(s.attributes.property).match(/^product:/)))return!0;if(e.headMetaVerification&&(P(s.attributes.name)==="google-site-verification"||P(s.attributes.name)==="yandex-verification"||P(s.attributes.name)==="csrf-token"||P(s.attributes.name)==="p:domain_verify"||P(s.attributes.name)==="verify-v1"||P(s.attributes.name)==="verification"||P(s.attributes.name)==="shopify-checkout-api-token"))return!0}}return!1}function Pe(s,e){const{doc:t,mirror:r,blockClass:i,blockSelector:n,maskTextClass:o,maskTextSelector:l,skipChild:a=!1,inlineStylesheet:u=!0,maskInputOptions:c={},maskTextFn:h,maskInputFn:p,slimDOMOptions:m,dataURLOptions:g={},inlineImages:d=!1,recordCanvas:f=!1,onSerialize:S,onIframeLoad:b,iframeLoadTimeout:y=5e3,onStylesheetLoad:C,stylesheetLoadTimeout:k=5e3,keepIframeSrcFn:D=()=>!1,newlyAddedElement:H=!1,cssCaptured:B=!1}=e;let{needsMask:V}=e,{preserveWhiteSpace:Z=!0}=e;V||(V=oi(s,o,l,V===void 0));const ie=Ba(s,{doc:t,mirror:r,blockClass:i,blockSelector:n,needsMask:V,inlineStylesheet:u,maskInputOptions:c,maskTextFn:h,maskInputFn:p,dataURLOptions:g,inlineImages:d,recordCanvas:f,keepIframeSrcFn:D,newlyAddedElement:H,cssCaptured:B});if(!ie)return console.warn(s,"not serialized"),null;let pe;r.hasNode(s)?pe=r.getId(s):Va(ie,m)||!Z&&ie.type===j.Text&&!ie.textContent.replace(/^\s+|\s+$/gm,"").length?pe=Ze:pe=ti();const O=Object.assign(ie,{id:pe});if(r.add(s,O),pe===Ze)return null;S&&S(s);let pt=!a;if(O.type===j.Element){pt=pt&&!O.needBlock,delete O.needBlock;const Y=K.shadowRoot(s);Y&&Ye(Y)&&(O.isShadowHost=!0)}if((O.type===j.Document||O.type===j.Element)&&pt){m.headWhitespace&&O.type===j.Element&&O.tagName==="head"&&(Z=!1);const Y={doc:t,mirror:r,blockClass:i,blockSelector:n,needsMask:V,maskTextClass:o,maskTextSelector:l,skipChild:a,inlineStylesheet:u,maskInputOptions:c,maskTextFn:h,maskInputFn:p,slimDOMOptions:m,dataURLOptions:g,inlineImages:d,recordCanvas:f,preserveWhiteSpace:Z,onSerialize:S,onIframeLoad:b,iframeLoadTimeout:y,onStylesheetLoad:C,stylesheetLoadTimeout:k,keepIframeSrcFn:D,cssCaptured:!1};if(!(O.type===j.Element&&O.tagName==="textarea"&&O.attributes.value!==void 0)){O.type===j.Element&&O.attributes._cssText!==void 0&&typeof O.attributes._cssText=="string"&&(Y.cssCaptured=!0);for(const Me of Array.from(K.childNodes(s))){const me=Pe(Me,Y);me&&O.childNodes.push(me)}}let ne=null;if(Ks(s)&&(ne=K.shadowRoot(s)))for(const Me of Array.from(K.childNodes(ne))){const me=Pe(Me,Y);me&&(Ye(ne)&&(me.isShadow=!0),O.childNodes.push(me))}}const mt=K.parentNode(s);return mt&&He(mt)&&Ye(mt)&&(O.isShadow=!0),O.type===j.Element&&O.tagName==="iframe"&&Fa(s,()=>{const Y=s.contentDocument;if(Y&&b){const ne=Pe(Y,{doc:Y,mirror:r,blockClass:i,blockSelector:n,needsMask:V,maskTextClass:o,maskTextSelector:l,skipChild:!1,inlineStylesheet:u,maskInputOptions:c,maskTextFn:h,maskInputFn:p,slimDOMOptions:m,dataURLOptions:g,inlineImages:d,recordCanvas:f,preserveWhiteSpace:Z,onSerialize:S,onIframeLoad:b,iframeLoadTimeout:y,onStylesheetLoad:C,stylesheetLoadTimeout:k,keepIframeSrcFn:D});ne&&b(s,ne)}},y),O.type===j.Element&&O.tagName==="link"&&typeof O.attributes.rel=="string"&&(O.attributes.rel==="stylesheet"||O.attributes.rel==="preload"&&typeof O.attributes.href=="string"&&ei(O.attributes.href)==="css")&&Ua(s,()=>{if(C){const Y=Pe(s,{doc:t,mirror:r,blockClass:i,blockSelector:n,needsMask:V,maskTextClass:o,maskTextSelector:l,skipChild:!1,inlineStylesheet:u,maskInputOptions:c,maskTextFn:h,maskInputFn:p,slimDOMOptions:m,dataURLOptions:g,inlineImages:d,recordCanvas:f,preserveWhiteSpace:Z,onSerialize:S,onIframeLoad:b,iframeLoadTimeout:y,onStylesheetLoad:C,stylesheetLoadTimeout:k,keepIframeSrcFn:D});Y&&C(s,Y)}},k),O}function ja(s,e){const{mirror:t=new Qs,blockClass:r="rr-block",blockSelector:i=null,maskTextClass:n="rr-mask",maskTextSelector:o=null,inlineStylesheet:l=!0,inlineImages:a=!1,recordCanvas:u=!1,maskAllInputs:c=!1,maskTextFn:h,maskInputFn:p,slimDOM:m=!1,dataURLOptions:g,preserveWhiteSpace:d,onSerialize:f,onIframeLoad:S,iframeLoadTimeout:b,onStylesheetLoad:y,stylesheetLoadTimeout:C,keepIframeSrcFn:k=()=>!1}=e||{},D=c===!0?{color:!0,date:!0,"datetime-local":!0,email:!0,month:!0,number:!0,range:!0,search:!0,tel:!0,text:!0,time:!0,url:!0,week:!0,textarea:!0,select:!0,password:!0}:c===!1?{password:!0}:c,H=ai(m);return Pe(s,{doc:s,mirror:t,blockClass:r,blockSelector:i,maskTextClass:n,maskTextSelector:o,skipChild:!1,inlineStylesheet:l,maskInputOptions:D,maskTextFn:h,maskInputFn:p,slimDOMOptions:H,dataURLOptions:g,inlineImages:a,recordCanvas:u,preserveWhiteSpace:d,onSerialize:f,onIframeLoad:S,iframeLoadTimeout:b,onStylesheetLoad:y,stylesheetLoadTimeout:C,keepIframeSrcFn:k,newlyAddedElement:!1})}function Ha(s){return s&&s.__esModule&&Object.prototype.hasOwnProperty.call(s,"default")?s.default:s}function Ya(s){if(s.__esModule)return s;var e=s.default;if(typeof e=="function"){var t=function r(){return this instanceof r?Reflect.construct(e,arguments,this.constructor):e.apply(this,arguments)};t.prototype=e.prototype}else t={};return Object.defineProperty(t,"__esModule",{value:!0}),Object.keys(s).forEach(function(r){var i=Object.getOwnPropertyDescriptor(s,r);Object.defineProperty(t,r,i.get?i:{enumerable:!0,get:function(){return s[r]}})}),t}var Or={exports:{}},L=String,li=function(){return{isColorSupported:!1,reset:L,bold:L,dim:L,italic:L,underline:L,inverse:L,hidden:L,strikethrough:L,black:L,red:L,green:L,yellow:L,blue:L,magenta:L,cyan:L,white:L,gray:L,bgBlack:L,bgRed:L,bgGreen:L,bgYellow:L,bgBlue:L,bgMagenta:L,bgCyan:L,bgWhite:L}};Or.exports=li(),Or.exports.createColors=li;var Za=Or.exports;const oe=Ya(Object.freeze(Object.defineProperty({__proto__:null,default:{}},Symbol.toStringTag,{value:"Module"})));let ui=Za,ci=oe,Er=class jo extends Error{constructor(e,t,r,i,n,o){super(e),this.name="CssSyntaxError",this.reason=e,n&&(this.file=n),i&&(this.source=i),o&&(this.plugin=o),typeof t<"u"&&typeof r<"u"&&(typeof t=="number"?(this.line=t,this.column=r):(this.line=t.line,this.column=t.column,this.endLine=r.line,this.endColumn=r.column)),this.setMessage(),Error.captureStackTrace&&Error.captureStackTrace(this,jo)}setMessage(){this.message=this.plugin?this.plugin+": ":"",this.message+=this.file?this.file:"<css input>",typeof this.line<"u"&&(this.message+=":"+this.line+":"+this.column),this.message+=": "+this.reason}showSourceCode(e){if(!this.source)return"";let t=this.source;e==null&&(e=ui.isColorSupported),ci&&e&&(t=ci(t));let r=t.split(/\r?\n/),i=Math.max(this.line-3,0),n=Math.min(this.line+2,r.length),o=String(n).length,l,a;if(e){let{bold:u,gray:c,red:h}=ui.createColors(!0);l=p=>u(h(p)),a=p=>c(p)}else l=a=u=>u;return r.slice(i,n).map((u,c)=>{let h=i+1+c,p=" "+(" "+h).slice(-o)+" | ";if(h===this.line){let m=a(p.replace(/\d/g," "))+u.slice(0,this.column-1).replace(/[^\t]/g," ");return l(">")+a(p)+u+`
- `+m+l("^")}return" "+a(p)+u}).join(`
-`)}toString(){let e=this.showSourceCode();return e&&(e=`
+var Nl=Object.defineProperty;var kl=(be,Ce,X)=>Ce in be?Nl(be,Ce,{enumerable:!0,configurable:!0,writable:!0,value:X}):be[Ce]=X;var k=(be,Ce,X)=>kl(be,typeof Ce!="symbol"?Ce+"":Ce,X);(function(){"use strict";function be(){return typeof crypto<"u"&&typeof crypto.randomUUID=="function"?crypto.randomUUID():"xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g,e=>{const t=Math.random()*16|0;return(e==="x"?t:t&3|8).toString(16)})}function Ce(e){const t=e.indexOf("@");if(t===-1)throw new Error("Invalid connection string: must be in format {token}@{apiUrl}");const r=e.slice(0,t),o=e.slice(t+1);if(!r||!o)throw new Error("Invalid connection string: token and apiUrl must not be empty");return{token:r,apiUrl:o}}function X(){return new Date().toISOString()}var as=class{constructor(e={}){k(this,"windowMs");k(this,"maxSize");k(this,"q",[]);this.windowMs=e.windowMs??1e4,this.maxSize=e.maxSize??200}add(e){this.q.push(e),this.prune()}snapshot(){return this.prune(),this.q.slice()}clear(){this.q.length=0}get length(){return this.q.length}prune(){const e=Date.now()-this.windowMs;for(;this.q.length>0&&Date.parse(this.q[0].timestamp)<e;)this.q.shift();for(;this.q.length>this.maxSize;)this.q.shift()}},An=Object.defineProperty,Nn=(e,t,r)=>t in e?An(e,t,{enumerable:!0,configurable:!0,writable:!0,value:r}):e[t]=r,A=(e,t,r)=>Nn(e,typeof t!="symbol"?t+"":t,r),ls,kn=Object.defineProperty,Pn=(e,t,r)=>t in e?kn(e,t,{enumerable:!0,configurable:!0,writable:!0,value:r}):e[t]=r,us=(e,t,r)=>Pn(e,typeof t!="symbol"?t+"":t,r),Z=(e=>(e[e.Document=0]="Document",e[e.DocumentType=1]="DocumentType",e[e.Element=2]="Element",e[e.Text=3]="Text",e[e.CDATA=4]="CDATA",e[e.Comment=5]="Comment",e))(Z||{});const cs={Node:["childNodes","parentNode","parentElement","textContent","ownerDocument","firstChild","lastChild","nextSibling","previousSibling"],ShadowRoot:["host","styleSheets"],Element:["shadowRoot","querySelector","querySelectorAll"],MutationObserver:[]},hs={Node:["contains","getRootNode"],ShadowRoot:["getSelection"],Element:[],MutationObserver:["constructor"]},Ze={},fs={},Tn=()=>!!globalThis.Zone;function At(e){if(Ze[e])return Ze[e];const t=globalThis[e],r=t.prototype,o=e in cs?cs[e]:void 0,s=!!(o&&o.every(p=>{var l,d;return!!((d=(l=Object.getOwnPropertyDescriptor(r,p))==null?void 0:l.get)!=null&&d.toString().includes("[native code]"))})),u=e in hs?hs[e]:void 0,c=!!(u&&u.every(p=>{var l;return typeof r[p]=="function"&&((l=r[p])==null?void 0:l.toString().includes("[native code]"))}));if(s&&c&&!Tn())return Ze[e]=t.prototype,t.prototype;try{const p=document.createElement("iframe");p.style.display="none",document.body.appendChild(p);const l=p.contentWindow;if(!l)return t.prototype;const d=l[e].prototype;if(!d)return p.remove(),r;const i=navigator.userAgent;return i.includes("Safari")&&!i.includes("Chrome")?(p.classList.add("rr-block"),p.setAttribute("__rrwebUntaintedMutationObserver",""),fs[e]=()=>p.remove()):p.remove(),Ze[e]=d}catch{return r}}const Nt={};function ae(e,t,r){var o,s;const u=(o=Nt[e])==null?void 0:o[r];if(u)return u.call(t);const c=At(e),p=(s=Object.getOwnPropertyDescriptor(c,r))==null?void 0:s.get;return p?((Nt[e]||(Nt[e]={}))[r]=p,p.call(t)):t[r]}const kt={};function ds(e,t,r){const o=`${e}.${String(r)}`;if(kt[o])return kt[o].bind(t);const u=At(e)[r];return typeof u!="function"?t[r]:(kt[o]=u,u.bind(t))}function Dn(e){return ae("Node",e,"ownerDocument")}function Ln(e){return ae("Node",e,"childNodes")}function _n(e){return ae("Node",e,"parentNode")}function $n(e){return ae("Node",e,"parentElement")}function Fn(e){return ae("Node",e,"textContent")}function Un(e){return ae("Node",e,"firstChild")}function Bn(e){return ae("Node",e,"lastChild")}function zn(e){return ae("Node",e,"nextSibling")}function Wn(e){return ae("Node",e,"previousSibling")}function qn(e,t){return ds("Node",e,"contains")(t)}function jn(e){return ds("Node",e,"getRootNode")()}function Gn(e){return!e||!("host"in e)?null:ae("ShadowRoot",e,"host")}function Vn(e){return e.styleSheets}function Hn(e){return!e||!("shadowRoot"in e)?null:ae("Element",e,"shadowRoot")}function Jn(e,t){return ae("Element",e,"querySelector")(t)}function Yn(e,t){return ae("Element",e,"querySelectorAll")(t)}function Qn(){return[At("MutationObserver").constructor,fs.MutationObserver??(()=>{})]}let ps=Date.now;/[1-9][0-9]{12}/.test(Date.now().toString())||(ps=()=>new Date().getTime());function Zn(e,t,r){try{if(!(t in e))return()=>{};const o=e[t],s=r(o);return typeof s=="function"&&(s.prototype=s.prototype||{},Object.defineProperties(s,{__rrweb_original__:{enumerable:!1,value:o}})),e[t]=s,()=>{e[t]=o}}catch{return()=>{}}}const ee={ownerDocument:Dn,childNodes:Ln,parentNode:_n,parentElement:$n,textContent:Fn,firstChild:Un,lastChild:Bn,nextSibling:zn,previousSibling:Wn,contains:qn,getRootNode:jn,host:Gn,styleSheets:Vn,shadowRoot:Hn,querySelector:Jn,querySelectorAll:Yn,nowTimestamp:ps,mutationObserverCtor:Qn,patch:Zn};function ms(e){return e.nodeType===e.ELEMENT_NODE}function We(e){const t=e&&"host"in e&&"mode"in e&&ee.host(e)||null;return!!(t&&"shadowRoot"in t&&ee.shadowRoot(t)===e)}function qe(e){return Object.prototype.toString.call(e)==="[object ShadowRoot]"}function Kn(e){return e.includes(" background-clip: text;")&&!e.includes(" -webkit-background-clip: text;")&&(e=e.replace(/\sbackground-clip:\s*text;/g," -webkit-background-clip: text; background-clip: text;")),e}function Xn(e){const{cssText:t}=e;if(t.split('"').length<3)return t;const r=["@import",`url(${JSON.stringify(e.href)})`];return e.layerName===""?r.push("layer"):e.layerName&&r.push(`layer(${e.layerName})`),e.supportsText&&r.push(`supports(${e.supportsText})`),e.media.length&&r.push(e.media.mediaText),r.join(" ")+";"}function Pt(e){try{const t=e.rules||e.cssRules;if(!t)return null;let r=e.href;!r&&e.ownerNode&&(r=e.ownerNode.baseURI);const o=Array.from(t,s=>gs(s,r)).join("");return Kn(o)}catch{return null}}function gs(e,t){if(to(e)){let r;try{r=Pt(e.styleSheet)||Xn(e)}catch{r=e.cssText}return e.styleSheet.href?et(r,e.styleSheet.href):r}else{let r=e.cssText;return ro(e)&&e.selectorText.includes(":")&&(r=eo(r)),t?et(r,t):r}}function eo(e){const t=/(\[(?:[\w-]+)[^\\])(:(?:[\w-]+)\])/gm;return e.replace(t,"$1\\$2")}function to(e){return"styleSheet"in e}function ro(e){return"selectorText"in e}class ys{constructor(){us(this,"idNodeMap",new Map),us(this,"nodeMetaMap",new WeakMap)}getId(t){var r;return t?((r=this.getMeta(t))==null?void 0:r.id)??-1:-1}getNode(t){return this.idNodeMap.get(t)||null}getIds(){return Array.from(this.idNodeMap.keys())}getMeta(t){return this.nodeMetaMap.get(t)||null}removeNodeFromMap(t){const r=this.getId(t);this.idNodeMap.delete(r),t.childNodes&&t.childNodes.forEach(o=>this.removeNodeFromMap(o))}has(t){return this.idNodeMap.has(t)}hasNode(t){return this.nodeMetaMap.has(t)}add(t,r){const o=r.id;this.idNodeMap.set(o,t),this.nodeMetaMap.set(t,r)}replace(t,r){const o=this.getNode(t);if(o){const s=this.nodeMetaMap.get(o);s&&this.nodeMetaMap.set(r,s)}this.idNodeMap.set(t,r)}reset(){this.idNodeMap=new Map,this.nodeMetaMap=new WeakMap}}function so(){return new ys}function Ke({element:e,maskInputOptions:t,tagName:r,type:o,value:s,maskInputFn:u}){let c=s||"";const p=o&&Re(o);return(t[r.toLowerCase()]||p&&t[p])&&(u?c=u(c,e):c="*".repeat(c.length)),c}function Re(e){return e.toLowerCase()}const ws="__rrweb_original__";function io(e){const t=e.getContext("2d");if(!t)return!0;const r=50;for(let o=0;o<e.width;o+=r)for(let s=0;s<e.height;s+=r){const u=t.getImageData,c=ws in u?u[ws]:u;if(new Uint32Array(c.call(t,o,s,Math.min(r,e.width-o),Math.min(r,e.height-s)).data.buffer).some(l=>l!==0))return!1}return!0}function Xe(e){const t=e.type;return e.hasAttribute("data-rr-is-password")?"password":t?Re(t):null}function bs(e,t){let r;try{r=new URL(e,t??window.location.href)}catch{return null}const o=/\.([0-9a-z]+)(?:$)/i,s=r.pathname.match(o);return(s==null?void 0:s[1])??null}function no(e){let t="";return e.indexOf("//")>-1?t=e.split("/").slice(0,3).join("/"):t=e.split("/")[0],t=t.split("?")[0],t}const oo=/url\((?:(')([^']*)'|(")(.*?)"|([^)]*))\)/gm,ao=/^(?:[a-z+]+:)?\/\//i,lo=/^www\..*/i,uo=/^(data:)([^,]*),(.*)/i;function et(e,t){return(e||"").replace(oo,(r,o,s,u,c,p)=>{const l=s||c||p,d=o||u||"";if(!l)return r;if(ao.test(l)||lo.test(l))return`url(${d}${l}${d})`;if(uo.test(l))return`url(${d}${l}${d})`;if(l[0]==="/")return`url(${d}${no(t)+l}${d})`;const i=l.split("#")[0],f=l.substring(i.length),a=t.split("#")[0].split("/"),n=i.split("/");a.pop();for(const m of n)m!=="."&&(m===".."?a.pop():a.push(m));return`url(${d}${a.join("/")}${f}${d})`})}function tt(e,t=!1){return t?e.replace(/(\/\*[^*]*\*\/)|[\s;]/g,""):e.replace(/(\/\*[^*]*\*\/)|[\s;]/g,"").replace(/0px/g,"0")}function co(e,t,r=!1){const o=Array.from(t.childNodes),s=[];let u=0;if(o.length>1&&e&&typeof e=="string"){let c=tt(e,r);const p=c.length/e.length;for(let l=1;l<o.length;l++)if(o[l].textContent&&typeof o[l].textContent=="string"){const d=tt(o[l].textContent,r),i=100;let f=3;for(;f<d.length&&(d[f].match(/[a-zA-Z0-9]/)||d.indexOf(d.substring(0,f),1)!==-1);f++);for(;f<d.length;f++){let a=d.substring(0,f),n=c.split(a),m=-1;if(n.length===2)m=n[0].length;else if(n.length>2&&n[0]===""&&o[l-1].textContent!=="")m=c.indexOf(a,1);else if(n.length===1){if(a=a.substring(0,a.length-1),n=c.split(a),n.length<=1)return s.push(e),s;f=i+1}else f===d.length-1&&(m=c.indexOf(a));if(n.length>=2&&f>i){const h=o[l-1].textContent;if(h&&typeof h=="string"){const g=tt(h).length;m=c.indexOf(a,g)}m===-1&&(m=n[0].length)}if(m!==-1){let h=Math.floor(m/p);for(;h>0&&h<e.length;){if(u+=1,u>50*o.length)return s.push(e),s;const g=tt(e.substring(0,h),r);if(g.length===m){s.push(e.substring(0,h)),e=e.substring(h),c=c.substring(m);break}else g.length<m?h+=Math.max(1,Math.floor((m-g.length)/p)):h-=Math.max(1,Math.floor((g.length-m)*p))}break}}}}return s.push(e),s}function ho(e,t){return co(e,t).join("/* rr_split */")}let fo=1;const po=new RegExp("[^a-z0-9-_:]"),rt=-2;function Ss(){return fo++}function mo(e){if(e instanceof HTMLFormElement)return"form";const t=Re(e.tagName);return po.test(t)?"div":t}let De,vs;const go=/^[^ \t\n\r\u000c]+/,yo=/^[, \t\n\r\u000c]+/;function wo(e,t){if(t.trim()==="")return t;let r=0;function o(u){let c;const p=u.exec(t.substring(r));return p?(c=p[0],r+=c.length,c):""}const s=[];for(;o(yo),!(r>=t.length);){let u=o(go);if(u.slice(-1)===",")u=Le(e,u.substring(0,u.length-1)),s.push(u);else{let c="";u=Le(e,u);let p=!1;for(;;){const l=t.charAt(r);if(l===""){s.push((u+c).trim());break}else if(p)l===")"&&(p=!1);else if(l===","){r+=1,s.push((u+c).trim());break}else l==="("&&(p=!0);c+=l,r+=1}}}return s.join(", ")}const Cs=new WeakMap;function Le(e,t){return!t||t.trim()===""?t:Tt(e,t)}function bo(e){return!!(e.tagName==="svg"||e.ownerSVGElement)}function Tt(e,t){let r=Cs.get(e);if(r||(r=e.createElement("a"),Cs.set(e,r)),!t)t="";else if(t.startsWith("blob:")||t.startsWith("data:"))return t;return r.setAttribute("href",t),r.href}function xs(e,t,r,o){return o&&(r==="src"||r==="href"&&!(t==="use"&&o[0]==="#")||r==="xlink:href"&&o[0]!=="#"||r==="background"&&["table","td","th"].includes(t)?Le(e,o):r==="srcset"?wo(e,o):r==="style"?et(o,Tt(e)):t==="object"&&r==="data"?Le(e,o):o)}function Rs(e,t,r){return["video","audio"].includes(e)&&Re(t)==="autoplay"}function So(e,t,r){try{if(typeof t=="string"){if(e.classList.contains(t))return!0}else for(let o=e.classList.length;o--;){const s=e.classList[o];if(t.test(s))return!0}if(r)return e.matches(r)}catch{}return!1}function st(e,t,r){if(!e)return!1;if(e.nodeType!==e.ELEMENT_NODE)return r?st(ee.parentNode(e),t,r):!1;for(let o=e.classList.length;o--;){const s=e.classList[o];if(t.test(s))return!0}return r?st(ee.parentNode(e),t,r):!1}function Os(e,t,r,o){let s;if(ms(e)){if(s=e,!ee.childNodes(s).length)return!1}else{if(ee.parentElement(e)===null)return!1;s=ee.parentElement(e)}try{if(typeof t=="string"){if(o){if(s.closest(`.${t}`))return!0}else if(s.classList.contains(t))return!0}else if(st(s,t,o))return!0;if(r){if(o){if(s.closest(r))return!0}else if(s.matches(r))return!0}}catch{}return!1}function vo(e,t,r){const o=e.contentWindow;if(!o)return;let s=!1,u;try{u=o.document.readyState}catch{return}if(u!=="complete"){const p=setTimeout(()=>{s||(t(),s=!0)},r);e.addEventListener("load",()=>{clearTimeout(p),s=!0,t()});return}const c="about:blank";if(o.location.href!==c||e.src===c||e.src==="")return setTimeout(t,0),e.addEventListener("load",t);e.addEventListener("load",t)}function Co(e,t,r){let o=!1,s;try{s=e.sheet}catch{return}if(s)return;const u=setTimeout(()=>{o||(t(),o=!0)},r);e.addEventListener("load",()=>{clearTimeout(u),o=!0,t()})}function xo(e,t){const{doc:r,mirror:o,blockClass:s,blockSelector:u,needsMask:c,inlineStylesheet:p,maskInputOptions:l={},maskTextFn:d,maskInputFn:i,dataURLOptions:f={},inlineImages:a,recordCanvas:n,keepIframeSrcFn:m,newlyAddedElement:h=!1,cssCaptured:g=!1}=t,v=Ro(r,o);switch(e.nodeType){case e.DOCUMENT_NODE:return e.compatMode!=="CSS1Compat"?{type:Z.Document,childNodes:[],compatMode:e.compatMode}:{type:Z.Document,childNodes:[]};case e.DOCUMENT_TYPE_NODE:return{type:Z.DocumentType,name:e.name,publicId:e.publicId,systemId:e.systemId,rootId:v};case e.ELEMENT_NODE:return Mo(e,{doc:r,blockClass:s,blockSelector:u,inlineStylesheet:p,maskInputOptions:l,maskInputFn:i,dataURLOptions:f,inlineImages:a,recordCanvas:n,keepIframeSrcFn:m,newlyAddedElement:h,rootId:v});case e.TEXT_NODE:return Oo(e,{doc:r,needsMask:c,maskTextFn:d,rootId:v,cssCaptured:g});case e.CDATA_SECTION_NODE:return{type:Z.CDATA,textContent:"",rootId:v};case e.COMMENT_NODE:return{type:Z.Comment,textContent:ee.textContent(e)||"",rootId:v};default:return!1}}function Ro(e,t){if(!t.hasNode(e))return;const r=t.getId(e);return r===1?void 0:r}function Oo(e,t){const{needsMask:r,maskTextFn:o,rootId:s,cssCaptured:u}=t,c=ee.parentNode(e),p=c&&c.tagName;let l="";const d=p==="STYLE"?!0:void 0,i=p==="SCRIPT"?!0:void 0;return i?l="SCRIPT_PLACEHOLDER":u||(l=ee.textContent(e),d&&l&&(l=et(l,Tt(t.doc)))),!d&&!i&&l&&r&&(l=o?o(l,ee.parentElement(e)):l.replace(/[\S]/g,"*")),{type:Z.Text,textContent:l||"",rootId:s}}function Mo(e,t){const{doc:r,blockClass:o,blockSelector:s,inlineStylesheet:u,maskInputOptions:c={},maskInputFn:p,dataURLOptions:l={},inlineImages:d,recordCanvas:i,keepIframeSrcFn:f,newlyAddedElement:a=!1,rootId:n}=t,m=So(e,o,s),h=mo(e);let g={};const v=e.attributes.length;for(let S=0;S<v;S++){const x=e.attributes[S];Rs(h,x.name,x.value)||(g[x.name]=xs(r,h,Re(x.name),x.value))}if(h==="link"&&u){const S=Array.from(r.styleSheets).find(w=>w.href===e.href);let x=null;S&&(x=Pt(S)),x&&(delete g.rel,delete g.href,g._cssText=x)}if(h==="style"&&e.sheet){let S=Pt(e.sheet);S&&(e.childNodes.length>1&&(S=ho(S,e)),g._cssText=S)}if(["input","textarea","select"].includes(h)){const S=e.value,x=e.checked;g.type!=="radio"&&g.type!=="checkbox"&&g.type!=="submit"&&g.type!=="button"&&S?g.value=Ke({element:e,type:Xe(e),tagName:h,value:S,maskInputOptions:c,maskInputFn:p}):x&&(g.checked=x)}if(h==="option"&&(e.selected&&!c.select?g.selected=!0:delete g.selected),h==="dialog"&&e.open&&(g.rr_open_mode=e.matches("dialog:modal")?"modal":"non-modal"),h==="canvas"&&i){if(e.__context==="2d")io(e)||(g.rr_dataURL=e.toDataURL(l.type,l.quality));else if(!("__context"in e)){const S=e.toDataURL(l.type,l.quality),x=r.createElement("canvas");x.width=e.width,x.height=e.height;const w=x.toDataURL(l.type,l.quality);S!==w&&(g.rr_dataURL=S)}}if(h==="img"&&d){De||(De=r.createElement("canvas"),vs=De.getContext("2d"));const S=e,x=S.currentSrc||S.getAttribute("src")||"<unknown-src>",w=S.crossOrigin,y=()=>{S.removeEventListener("load",y);try{De.width=S.naturalWidth,De.height=S.naturalHeight,vs.drawImage(S,0,0),g.rr_dataURL=De.toDataURL(l.type,l.quality)}catch(C){if(S.crossOrigin!=="anonymous"){S.crossOrigin="anonymous",S.complete&&S.naturalWidth!==0?y():S.addEventListener("load",y);return}else console.warn(`Cannot inline img src=${x}! Error: ${C}`)}S.crossOrigin==="anonymous"&&(w?g.crossOrigin=w:S.removeAttribute("crossorigin"))};S.complete&&S.naturalWidth!==0?y():S.addEventListener("load",y)}if(["audio","video"].includes(h)){const S=g;S.rr_mediaState=e.paused?"paused":"played",S.rr_mediaCurrentTime=e.currentTime,S.rr_mediaPlaybackRate=e.playbackRate,S.rr_mediaMuted=e.muted,S.rr_mediaLoop=e.loop,S.rr_mediaVolume=e.volume}if(a||(e.scrollLeft&&(g.rr_scrollLeft=e.scrollLeft),e.scrollTop&&(g.rr_scrollTop=e.scrollTop)),m){const{width:S,height:x}=e.getBoundingClientRect();g={class:g.class,rr_width:`${S}px`,rr_height:`${x}px`}}h==="iframe"&&!f(g.src)&&(e.contentDocument||(g.rr_src=g.src),delete g.src);let b;try{customElements.get(h)&&(b=!0)}catch{}return{type:Z.Element,tagName:h,attributes:g,childNodes:[],isSVG:bo(e)||void 0,needBlock:m,rootId:n,isCustom:b}}function j(e){return e==null?"":e.toLowerCase()}function Ms(e){return e===!0||e==="all"?{script:!0,comment:!0,headFavicon:!0,headWhitespace:!0,headMetaSocial:!0,headMetaRobots:!0,headMetaHttpEquiv:!0,headMetaVerification:!0,headMetaAuthorship:e==="all",headMetaDescKeywords:e==="all",headTitleMutations:e==="all"}:e||{}}function Eo(e,t){if(t.comment&&e.type===Z.Comment)return!0;if(e.type===Z.Element){if(t.script&&(e.tagName==="script"||e.tagName==="link"&&(e.attributes.rel==="preload"&&e.attributes.as==="script"||e.attributes.rel==="modulepreload")||e.tagName==="link"&&e.attributes.rel==="prefetch"&&typeof e.attributes.href=="string"&&bs(e.attributes.href)==="js"))return!0;if(t.headFavicon&&(e.tagName==="link"&&e.attributes.rel==="shortcut icon"||e.tagName==="meta"&&(j(e.attributes.name).match(/^msapplication-tile(image|color)$/)||j(e.attributes.name)==="application-name"||j(e.attributes.rel)==="icon"||j(e.attributes.rel)==="apple-touch-icon"||j(e.attributes.rel)==="shortcut icon")))return!0;if(e.tagName==="meta"){if(t.headMetaDescKeywords&&j(e.attributes.name).match(/^description|keywords$/))return!0;if(t.headMetaSocial&&(j(e.attributes.property).match(/^(og|twitter|fb):/)||j(e.attributes.name).match(/^(og|twitter):/)||j(e.attributes.name)==="pinterest"))return!0;if(t.headMetaRobots&&(j(e.attributes.name)==="robots"||j(e.attributes.name)==="googlebot"||j(e.attributes.name)==="bingbot"))return!0;if(t.headMetaHttpEquiv&&e.attributes["http-equiv"]!==void 0)return!0;if(t.headMetaAuthorship&&(j(e.attributes.name)==="author"||j(e.attributes.name)==="generator"||j(e.attributes.name)==="framework"||j(e.attributes.name)==="publisher"||j(e.attributes.name)==="progid"||j(e.attributes.property).match(/^article:/)||j(e.attributes.property).match(/^product:/)))return!0;if(t.headMetaVerification&&(j(e.attributes.name)==="google-site-verification"||j(e.attributes.name)==="yandex-verification"||j(e.attributes.name)==="csrf-token"||j(e.attributes.name)==="p:domain_verify"||j(e.attributes.name)==="verify-v1"||j(e.attributes.name)==="verification"||j(e.attributes.name)==="shopify-checkout-api-token"))return!0}}return!1}function _e(e,t){const{doc:r,mirror:o,blockClass:s,blockSelector:u,maskTextClass:c,maskTextSelector:p,skipChild:l=!1,inlineStylesheet:d=!0,maskInputOptions:i={},maskTextFn:f,maskInputFn:a,slimDOMOptions:n,dataURLOptions:m={},inlineImages:h=!1,recordCanvas:g=!1,onSerialize:v,onIframeLoad:b,iframeLoadTimeout:S=5e3,onStylesheetLoad:x,stylesheetLoadTimeout:w=5e3,keepIframeSrcFn:y=()=>!1,newlyAddedElement:C=!1,cssCaptured:O=!1}=t;let{needsMask:I}=t,{preserveWhiteSpace:M=!0}=t;I||(I=Os(e,c,p,I===void 0));const P=xo(e,{doc:r,mirror:o,blockClass:s,blockSelector:u,needsMask:I,inlineStylesheet:d,maskInputOptions:i,maskTextFn:f,maskInputFn:a,dataURLOptions:m,inlineImages:h,recordCanvas:g,keepIframeSrcFn:y,newlyAddedElement:C,cssCaptured:O});if(!P)return console.warn(e,"not serialized"),null;let N;o.hasNode(e)?N=o.getId(e):Eo(P,n)||!M&&P.type===Z.Text&&!P.textContent.replace(/^\s+|\s+$/gm,"").length?N=rt:N=Ss();const R=Object.assign(P,{id:N});if(o.add(e,R),N===rt)return null;v&&v(e);let ye=!l;if(R.type===Z.Element){ye=ye&&!R.needBlock,delete R.needBlock;const B=ee.shadowRoot(e);B&&qe(B)&&(R.isShadowHost=!0)}if((R.type===Z.Document||R.type===Z.Element)&&ye){n.headWhitespace&&R.type===Z.Element&&R.tagName==="head"&&(M=!1);const B={doc:r,mirror:o,blockClass:s,blockSelector:u,needsMask:I,maskTextClass:c,maskTextSelector:p,skipChild:l,inlineStylesheet:d,maskInputOptions:i,maskTextFn:f,maskInputFn:a,slimDOMOptions:n,dataURLOptions:m,inlineImages:h,recordCanvas:g,preserveWhiteSpace:M,onSerialize:v,onIframeLoad:b,iframeLoadTimeout:S,onStylesheetLoad:x,stylesheetLoadTimeout:w,keepIframeSrcFn:y,cssCaptured:!1};if(!(R.type===Z.Element&&R.tagName==="textarea"&&R.attributes.value!==void 0)){R.type===Z.Element&&R.attributes._cssText!==void 0&&typeof R.attributes._cssText=="string"&&(B.cssCaptured=!0);for(const Y of Array.from(ee.childNodes(e))){const te=_e(Y,B);te&&R.childNodes.push(te)}}let U=null;if(ms(e)&&(U=ee.shadowRoot(e)))for(const Y of Array.from(ee.childNodes(U))){const te=_e(Y,B);te&&(qe(U)&&(te.isShadow=!0),R.childNodes.push(te))}}const ue=ee.parentNode(e);return ue&&We(ue)&&qe(ue)&&(R.isShadow=!0),R.type===Z.Element&&R.tagName==="iframe"&&vo(e,()=>{const B=e.contentDocument;if(B&&b){const U=_e(B,{doc:B,mirror:o,blockClass:s,blockSelector:u,needsMask:I,maskTextClass:c,maskTextSelector:p,skipChild:!1,inlineStylesheet:d,maskInputOptions:i,maskTextFn:f,maskInputFn:a,slimDOMOptions:n,dataURLOptions:m,inlineImages:h,recordCanvas:g,preserveWhiteSpace:M,onSerialize:v,onIframeLoad:b,iframeLoadTimeout:S,onStylesheetLoad:x,stylesheetLoadTimeout:w,keepIframeSrcFn:y});U&&b(e,U)}},S),R.type===Z.Element&&R.tagName==="link"&&typeof R.attributes.rel=="string"&&(R.attributes.rel==="stylesheet"||R.attributes.rel==="preload"&&typeof R.attributes.href=="string"&&bs(R.attributes.href)==="css")&&Co(e,()=>{if(x){const B=_e(e,{doc:r,mirror:o,blockClass:s,blockSelector:u,needsMask:I,maskTextClass:c,maskTextSelector:p,skipChild:!1,inlineStylesheet:d,maskInputOptions:i,maskTextFn:f,maskInputFn:a,slimDOMOptions:n,dataURLOptions:m,inlineImages:h,recordCanvas:g,preserveWhiteSpace:M,onSerialize:v,onIframeLoad:b,iframeLoadTimeout:S,onStylesheetLoad:x,stylesheetLoadTimeout:w,keepIframeSrcFn:y});B&&x(e,B)}},w),R}function Io(e,t){const{mirror:r=new ys,blockClass:o="rr-block",blockSelector:s=null,maskTextClass:u="rr-mask",maskTextSelector:c=null,inlineStylesheet:p=!0,inlineImages:l=!1,recordCanvas:d=!1,maskAllInputs:i=!1,maskTextFn:f,maskInputFn:a,slimDOM:n=!1,dataURLOptions:m,preserveWhiteSpace:h,onSerialize:g,onIframeLoad:v,iframeLoadTimeout:b,onStylesheetLoad:S,stylesheetLoadTimeout:x,keepIframeSrcFn:w=()=>!1}=t,y=i===!0?{color:!0,date:!0,"datetime-local":!0,email:!0,month:!0,number:!0,range:!0,search:!0,tel:!0,text:!0,time:!0,url:!0,week:!0,textarea:!0,select:!0,password:!0}:i===!1?{password:!0}:i,C=Ms(n);return _e(e,{doc:e,mirror:r,blockClass:o,blockSelector:s,maskTextClass:u,maskTextSelector:c,skipChild:!1,inlineStylesheet:p,maskInputOptions:y,maskTextFn:f,maskInputFn:a,slimDOMOptions:C,dataURLOptions:m,inlineImages:l,recordCanvas:d,preserveWhiteSpace:h,onSerialize:g,onIframeLoad:v,iframeLoadTimeout:b,onStylesheetLoad:S,stylesheetLoadTimeout:x,keepIframeSrcFn:w,newlyAddedElement:!1})}function Ao(e){return e&&e.__esModule&&Object.prototype.hasOwnProperty.call(e,"default")?e.default:e}function No(e){if(e.__esModule)return e;var t=e.default;if(typeof t=="function"){var r=function o(){return this instanceof o?Reflect.construct(t,arguments,this.constructor):t.apply(this,arguments)};r.prototype=t.prototype}else r={};return Object.defineProperty(r,"__esModule",{value:!0}),Object.keys(e).forEach(function(o){var s=Object.getOwnPropertyDescriptor(e,o);Object.defineProperty(r,o,s.get?s:{enumerable:!0,get:function(){return e[o]}})}),r}var it={exports:{}},Es;function ko(){if(Es)return it.exports;Es=1;var e=String,t=function(){return{isColorSupported:!1,reset:e,bold:e,dim:e,italic:e,underline:e,inverse:e,hidden:e,strikethrough:e,black:e,red:e,green:e,yellow:e,blue:e,magenta:e,cyan:e,white:e,gray:e,bgBlack:e,bgRed:e,bgGreen:e,bgYellow:e,bgBlue:e,bgMagenta:e,bgCyan:e,bgWhite:e}};return it.exports=t(),it.exports.createColors=t,it.exports}const pe=No(Object.freeze(Object.defineProperty({__proto__:null,default:{}},Symbol.toStringTag,{value:"Module"})));var Dt,Is;function Lt(){if(Is)return Dt;Is=1;let e=ko(),t=pe;class r extends Error{constructor(s,u,c,p,l,d){super(s),this.name="CssSyntaxError",this.reason=s,l&&(this.file=l),p&&(this.source=p),d&&(this.plugin=d),typeof u<"u"&&typeof c<"u"&&(typeof u=="number"?(this.line=u,this.column=c):(this.line=u.line,this.column=u.column,this.endLine=c.line,this.endColumn=c.column)),this.setMessage(),Error.captureStackTrace&&Error.captureStackTrace(this,r)}setMessage(){this.message=this.plugin?this.plugin+": ":"",this.message+=this.file?this.file:"<css input>",typeof this.line<"u"&&(this.message+=":"+this.line+":"+this.column),this.message+=": "+this.reason}showSourceCode(s){if(!this.source)return"";let u=this.source;s==null&&(s=e.isColorSupported),t&&s&&(u=t(u));let c=u.split(/\r?\n/),p=Math.max(this.line-3,0),l=Math.min(this.line+2,c.length),d=String(l).length,i,f;if(s){let{bold:a,gray:n,red:m}=e.createColors(!0);i=h=>a(m(h)),f=h=>n(h)}else i=f=a=>a;return c.slice(p,l).map((a,n)=>{let m=p+1+n,h=" "+(" "+m).slice(-d)+" | ";if(m===this.line){let g=f(h.replace(/\d/g," "))+a.slice(0,this.column-1).replace(/[^\t]/g," ");return i(">")+f(h)+a+`
+ `+g+i("^")}return" "+f(h)+a}).join(`
+`)}toString(){let s=this.showSourceCode();return s&&(s=`
-`+e+`
-`),this.name+": "+this.message+e}};var Mr=Er;Er.default=Er;var Je={};Je.isClean=Symbol("isClean"),Je.my=Symbol("my");const hi={after:`
+`+s+`
+`),this.name+": "+this.message+s}}return Dt=r,r.default=r,Dt}var nt={},As;function _t(){return As||(As=1,nt.isClean=Symbol("isClean"),nt.my=Symbol("my")),nt}var $t,Ns;function ks(){if(Ns)return $t;Ns=1;const e={after:`
`,beforeClose:`
`,beforeComment:`
`,beforeDecl:`
`,beforeOpen:" ",beforeRule:`
-`,colon:": ",commentLeft:" ",commentRight:" ",emptyBody:"",indent:" ",semicolon:!1};function Ja(s){return s[0].toUpperCase()+s.slice(1)}let $r=class{constructor(e){this.builder=e}atrule(e,t){let r="@"+e.name,i=e.params?this.rawValue(e,"params"):"";if(typeof e.raws.afterName<"u"?r+=e.raws.afterName:i&&(r+=" "),e.nodes)this.block(e,r+i);else{let n=(e.raws.between||"")+(t?";":"");this.builder(r+i+n,e)}}beforeAfter(e,t){let r;e.type==="decl"?r=this.raw(e,null,"beforeDecl"):e.type==="comment"?r=this.raw(e,null,"beforeComment"):t==="before"?r=this.raw(e,null,"beforeRule"):r=this.raw(e,null,"beforeClose");let i=e.parent,n=0;for(;i&&i.type!=="root";)n+=1,i=i.parent;if(r.includes(`
-`)){let o=this.raw(e,null,"indent");if(o.length)for(let l=0;l<n;l++)r+=o}return r}block(e,t){let r=this.raw(e,"between","beforeOpen");this.builder(t+r+"{",e,"start");let i;e.nodes&&e.nodes.length?(this.body(e),i=this.raw(e,"after")):i=this.raw(e,"after","emptyBody"),i&&this.builder(i),this.builder("}",e,"end")}body(e){let t=e.nodes.length-1;for(;t>0&&e.nodes[t].type==="comment";)t-=1;let r=this.raw(e,"semicolon");for(let i=0;i<e.nodes.length;i++){let n=e.nodes[i],o=this.raw(n,"before");o&&this.builder(o),this.stringify(n,t!==i||r)}}comment(e){let t=this.raw(e,"left","commentLeft"),r=this.raw(e,"right","commentRight");this.builder("/*"+t+e.text+r+"*/",e)}decl(e,t){let r=this.raw(e,"between","colon"),i=e.prop+r+this.rawValue(e,"value");e.important&&(i+=e.raws.important||" !important"),t&&(i+=";"),this.builder(i,e)}document(e){this.body(e)}raw(e,t,r){let i;if(r||(r=t),t&&(i=e.raws[t],typeof i<"u"))return i;let n=e.parent;if(r==="before"&&(!n||n.type==="root"&&n.first===e||n&&n.type==="document"))return"";if(!n)return hi[r];let o=e.root();if(o.rawCache||(o.rawCache={}),typeof o.rawCache[r]<"u")return o.rawCache[r];if(r==="before"||r==="after")return this.beforeAfter(e,r);{let l="raw"+Ja(r);this[l]?i=this[l](o,e):o.walk(a=>{if(i=a.raws[t],typeof i<"u")return!1})}return typeof i>"u"&&(i=hi[r]),o.rawCache[r]=i,i}rawBeforeClose(e){let t;return e.walk(r=>{if(r.nodes&&r.nodes.length>0&&typeof r.raws.after<"u")return t=r.raws.after,t.includes(`
-`)&&(t=t.replace(/[^\n]+$/,"")),!1}),t&&(t=t.replace(/\S/g,"")),t}rawBeforeComment(e,t){let r;return e.walkComments(i=>{if(typeof i.raws.before<"u")return r=i.raws.before,r.includes(`
-`)&&(r=r.replace(/[^\n]+$/,"")),!1}),typeof r>"u"?r=this.raw(t,null,"beforeDecl"):r&&(r=r.replace(/\S/g,"")),r}rawBeforeDecl(e,t){let r;return e.walkDecls(i=>{if(typeof i.raws.before<"u")return r=i.raws.before,r.includes(`
-`)&&(r=r.replace(/[^\n]+$/,"")),!1}),typeof r>"u"?r=this.raw(t,null,"beforeRule"):r&&(r=r.replace(/\S/g,"")),r}rawBeforeOpen(e){let t;return e.walk(r=>{if(r.type!=="decl"&&(t=r.raws.between,typeof t<"u"))return!1}),t}rawBeforeRule(e){let t;return e.walk(r=>{if(r.nodes&&(r.parent!==e||e.first!==r)&&typeof r.raws.before<"u")return t=r.raws.before,t.includes(`
-`)&&(t=t.replace(/[^\n]+$/,"")),!1}),t&&(t=t.replace(/\S/g,"")),t}rawColon(e){let t;return e.walkDecls(r=>{if(typeof r.raws.between<"u")return t=r.raws.between.replace(/[^\s:]/g,""),!1}),t}rawEmptyBody(e){let t;return e.walk(r=>{if(r.nodes&&r.nodes.length===0&&(t=r.raws.after,typeof t<"u"))return!1}),t}rawIndent(e){if(e.raws.indent)return e.raws.indent;let t;return e.walk(r=>{let i=r.parent;if(i&&i!==e&&i.parent&&i.parent===e&&typeof r.raws.before<"u"){let n=r.raws.before.split(`
-`);return t=n[n.length-1],t=t.replace(/\S/g,""),!1}}),t}rawSemicolon(e){let t;return e.walk(r=>{if(r.nodes&&r.nodes.length&&r.last.type==="decl"&&(t=r.raws.semicolon,typeof t<"u"))return!1}),t}rawValue(e,t){let r=e[t],i=e.raws[t];return i&&i.value===r?i.raw:r}root(e){this.body(e),e.raws.after&&this.builder(e.raws.after)}rule(e){this.block(e,this.rawValue(e,"selector")),e.raws.ownSemicolon&&this.builder(e.raws.ownSemicolon,e,"end")}stringify(e,t){if(!this[e.type])throw new Error("Unknown AST node type "+e.type+". Maybe you need to change PostCSS stringifier.");this[e.type](e,t)}};var fi=$r;$r.default=$r;let Ka=fi;function Nr(s,e){new Ka(e).stringify(s)}var Ct=Nr;Nr.default=Nr;let{isClean:It,my:Xa}=Je,Qa=Mr,qa=fi,el=Ct;function kr(s,e){let t=new s.constructor;for(let r in s){if(!Object.prototype.hasOwnProperty.call(s,r)||r==="proxyCache")continue;let i=s[r],n=typeof i;r==="parent"&&n==="object"?e&&(t[r]=e):r==="source"?t[r]=i:Array.isArray(i)?t[r]=i.map(o=>kr(o,t)):(n==="object"&&i!==null&&(i=kr(i)),t[r]=i)}return t}let Tr=class{constructor(e={}){this.raws={},this[It]=!1,this[Xa]=!0;for(let t in e)if(t==="nodes"){this.nodes=[];for(let r of e[t])typeof r.clone=="function"?this.append(r.clone()):this.append(r)}else this[t]=e[t]}addToError(e){if(e.postcssNode=this,e.stack&&this.source&&/\n\s{4}at /.test(e.stack)){let t=this.source;e.stack=e.stack.replace(/\n\s{4}at /,`$&${t.input.from}:${t.start.line}:${t.start.column}$&`)}return e}after(e){return this.parent.insertAfter(this,e),this}assign(e={}){for(let t in e)this[t]=e[t];return this}before(e){return this.parent.insertBefore(this,e),this}cleanRaws(e){delete this.raws.before,delete this.raws.after,e||delete this.raws.between}clone(e={}){let t=kr(this);for(let r in e)t[r]=e[r];return t}cloneAfter(e={}){let t=this.clone(e);return this.parent.insertAfter(this,t),t}cloneBefore(e={}){let t=this.clone(e);return this.parent.insertBefore(this,t),t}error(e,t={}){if(this.source){let{end:r,start:i}=this.rangeBy(t);return this.source.input.error(e,{column:i.column,line:i.line},{column:r.column,line:r.line},t)}return new Qa(e)}getProxyProcessor(){return{get(e,t){return t==="proxyOf"?e:t==="root"?()=>e.root().toProxy():e[t]},set(e,t,r){return e[t]===r||(e[t]=r,(t==="prop"||t==="value"||t==="name"||t==="params"||t==="important"||t==="text")&&e.markDirty()),!0}}}markDirty(){if(this[It]){this[It]=!1;let e=this;for(;e=e.parent;)e[It]=!1}}next(){if(!this.parent)return;let e=this.parent.index(this);return this.parent.nodes[e+1]}positionBy(e,t){let r=this.source.start;if(e.index)r=this.positionInside(e.index,t);else if(e.word){t=this.toString();let i=t.indexOf(e.word);i!==-1&&(r=this.positionInside(i,t))}return r}positionInside(e,t){let r=t||this.toString(),i=this.source.start.column,n=this.source.start.line;for(let o=0;o<e;o++)r[o]===`
-`?(i=1,n+=1):i+=1;return{column:i,line:n}}prev(){if(!this.parent)return;let e=this.parent.index(this);return this.parent.nodes[e-1]}rangeBy(e){let t={column:this.source.start.column,line:this.source.start.line},r=this.source.end?{column:this.source.end.column+1,line:this.source.end.line}:{column:t.column+1,line:t.line};if(e.word){let i=this.toString(),n=i.indexOf(e.word);n!==-1&&(t=this.positionInside(n,i),r=this.positionInside(n+e.word.length,i))}else e.start?t={column:e.start.column,line:e.start.line}:e.index&&(t=this.positionInside(e.index)),e.end?r={column:e.end.column,line:e.end.line}:typeof e.endIndex=="number"?r=this.positionInside(e.endIndex):e.index&&(r=this.positionInside(e.index+1));return(r.line<t.line||r.line===t.line&&r.column<=t.column)&&(r={column:t.column+1,line:t.line}),{end:r,start:t}}raw(e,t){return new qa().raw(this,e,t)}remove(){return this.parent&&this.parent.removeChild(this),this.parent=void 0,this}replaceWith(...e){if(this.parent){let t=this,r=!1;for(let i of e)i===this?r=!0:r?(this.parent.insertAfter(t,i),t=i):this.parent.insertBefore(t,i);r||this.remove()}return this}root(){let e=this;for(;e.parent&&e.parent.type!=="document";)e=e.parent;return e}toJSON(e,t){let r={},i=t==null;t=t||new Map;let n=0;for(let o in this){if(!Object.prototype.hasOwnProperty.call(this,o)||o==="parent"||o==="proxyCache")continue;let l=this[o];if(Array.isArray(l))r[o]=l.map(a=>typeof a=="object"&&a.toJSON?a.toJSON(null,t):a);else if(typeof l=="object"&&l.toJSON)r[o]=l.toJSON(null,t);else if(o==="source"){let a=t.get(l.input);a==null&&(a=n,t.set(l.input,n),n++),r[o]={end:l.end,inputId:a,start:l.start}}else r[o]=l}return i&&(r.inputs=[...t.keys()].map(o=>o.toJSON())),r}toProxy(){return this.proxyCache||(this.proxyCache=new Proxy(this,this.getProxyProcessor())),this.proxyCache}toString(e=el){e.stringify&&(e=e.stringify);let t="";return e(this,r=>{t+=r}),t}warn(e,t,r){let i={node:this};for(let n in r)i[n]=r[n];return e.warn(t,i)}get proxyOf(){return this}};var xt=Tr;Tr.default=Tr;let tl=xt,Pr=class extends tl{constructor(e){e&&typeof e.value<"u"&&typeof e.value!="string"&&(e={...e,value:String(e.value)}),super(e),this.type="decl"}get variable(){return this.prop.startsWith("--")||this.prop[0]==="$"}};var At=Pr;Pr.default=Pr;let rl="useandom-26T198340PX75pxJACKVERYMINDBUSHWOLF_GQZbfghjklqvwyzrict";var sl={nanoid:(s=21)=>{let e="",t=s;for(;t--;)e+=rl[Math.random()*64|0];return e}};let{SourceMapConsumer:di,SourceMapGenerator:pi}=oe,{existsSync:il,readFileSync:nl}=oe,{dirname:Dr,join:ol}=oe;function al(s){return Buffer?Buffer.from(s,"base64").toString():window.atob(s)}let Lr=class{constructor(e,t){if(t.map===!1)return;this.loadAnnotation(e),this.inline=this.startWith(this.annotation,"data:");let r=t.map?t.map.prev:void 0,i=this.loadMap(t.from,r);!this.mapFile&&t.from&&(this.mapFile=t.from),this.mapFile&&(this.root=Dr(this.mapFile)),i&&(this.text=i)}consumer(){return this.consumerCache||(this.consumerCache=new di(this.text)),this.consumerCache}decodeInline(e){let t=/^data:application\/json;charset=utf-?8;base64,/,r=/^data:application\/json;base64,/,i=/^data:application\/json;charset=utf-?8,/,n=/^data:application\/json,/;if(i.test(e)||n.test(e))return decodeURIComponent(e.substr(RegExp.lastMatch.length));if(t.test(e)||r.test(e))return al(e.substr(RegExp.lastMatch.length));let o=e.match(/data:application\/json;([^,]+),/)[1];throw new Error("Unsupported source map encoding "+o)}getAnnotationURL(e){return e.replace(/^\/\*\s*# sourceMappingURL=/,"").trim()}isMap(e){return typeof e!="object"?!1:typeof e.mappings=="string"||typeof e._mappings=="string"||Array.isArray(e.sections)}loadAnnotation(e){let t=e.match(/\/\*\s*# sourceMappingURL=/gm);if(!t)return;let r=e.lastIndexOf(t.pop()),i=e.indexOf("*/",r);r>-1&&i>-1&&(this.annotation=this.getAnnotationURL(e.substring(r,i)))}loadFile(e){if(this.root=Dr(e),il(e))return this.mapFile=e,nl(e,"utf-8").toString().trim()}loadMap(e,t){if(t===!1)return!1;if(t){if(typeof t=="string")return t;if(typeof t=="function"){let r=t(e);if(r){let i=this.loadFile(r);if(!i)throw new Error("Unable to load previous source map: "+r.toString());return i}}else{if(t instanceof di)return pi.fromSourceMap(t).toString();if(t instanceof pi)return t.toString();if(this.isMap(t))return JSON.stringify(t);throw new Error("Unsupported previous source map format: "+t.toString())}}else{if(this.inline)return this.decodeInline(this.annotation);if(this.annotation){let r=this.annotation;return e&&(r=ol(Dr(e),r)),this.loadFile(r)}}}startWith(e,t){return e?e.substr(0,t.length)===t:!1}withContent(){return!!(this.consumer().sourcesContent&&this.consumer().sourcesContent.length>0)}};var mi=Lr;Lr.default=Lr;let{SourceMapConsumer:ll,SourceMapGenerator:ul}=oe,{fileURLToPath:gi,pathToFileURL:Rt}=oe,{isAbsolute:_r,resolve:Fr}=oe,{nanoid:cl}=sl,Ur=oe,yi=Mr,hl=mi,Br=Symbol("fromOffsetCache"),fl=!!(ll&&ul),wi=!!(Fr&&_r),Ot=class{constructor(e,t={}){if(e===null||typeof e>"u"||typeof e=="object"&&!e.toString)throw new Error(`PostCSS received ${e} instead of CSS string`);if(this.css=e.toString(),this.css[0]==="\uFEFF"||this.css[0]==="￾"?(this.hasBOM=!0,this.css=this.css.slice(1)):this.hasBOM=!1,t.from&&(!wi||/^\w+:\/\//.test(t.from)||_r(t.from)?this.file=t.from:this.file=Fr(t.from)),wi&&fl){let r=new hl(this.css,t);if(r.text){this.map=r;let i=r.consumer().file;!this.file&&i&&(this.file=this.mapResolve(i))}}this.file||(this.id="<input css "+cl(6)+">"),this.map&&(this.map.file=this.from)}error(e,t,r,i={}){let n,o,l;if(t&&typeof t=="object"){let u=t,c=r;if(typeof u.offset=="number"){let h=this.fromOffset(u.offset);t=h.line,r=h.col}else t=u.line,r=u.column;if(typeof c.offset=="number"){let h=this.fromOffset(c.offset);o=h.line,l=h.col}else o=c.line,l=c.column}else if(!r){let u=this.fromOffset(t);t=u.line,r=u.col}let a=this.origin(t,r,o,l);return a?n=new yi(e,a.endLine===void 0?a.line:{column:a.column,line:a.line},a.endLine===void 0?a.column:{column:a.endColumn,line:a.endLine},a.source,a.file,i.plugin):n=new yi(e,o===void 0?t:{column:r,line:t},o===void 0?r:{column:l,line:o},this.css,this.file,i.plugin),n.input={column:r,endColumn:l,endLine:o,line:t,source:this.css},this.file&&(Rt&&(n.input.url=Rt(this.file).toString()),n.input.file=this.file),n}fromOffset(e){let t,r;if(this[Br])r=this[Br];else{let n=this.css.split(`
-`);r=new Array(n.length);let o=0;for(let l=0,a=n.length;l<a;l++)r[l]=o,o+=n[l].length+1;this[Br]=r}t=r[r.length-1];let i=0;if(e>=t)i=r.length-1;else{let n=r.length-2,o;for(;i<n;)if(o=i+(n-i>>1),e<r[o])n=o-1;else if(e>=r[o+1])i=o+1;else{i=o;break}}return{col:e-r[i]+1,line:i+1}}mapResolve(e){return/^\w+:\/\//.test(e)?e:Fr(this.map.consumer().sourceRoot||this.map.root||".",e)}origin(e,t,r,i){if(!this.map)return!1;let n=this.map.consumer(),o=n.originalPositionFor({column:t,line:e});if(!o.source)return!1;let l;typeof r=="number"&&(l=n.originalPositionFor({column:i,line:r}));let a;_r(o.source)?a=Rt(o.source):a=new URL(o.source,this.map.consumer().sourceRoot||Rt(this.map.mapFile));let u={column:o.column,endColumn:l&&l.column,endLine:l&&l.line,line:o.line,url:a.toString()};if(a.protocol==="file:")if(gi)u.file=gi(a);else throw new Error("file: protocol is not available in this PostCSS build");let c=n.sourceContentFor(o.source);return c&&(u.source=c),u}toJSON(){let e={};for(let t of["hasBOM","css","file","id"])this[t]!=null&&(e[t]=this[t]);return this.map&&(e.map={...this.map},e.map.consumerCache&&(e.map.consumerCache=void 0)),e}get from(){return this.file||this.id}};var Et=Ot;Ot.default=Ot,Ur&&Ur.registerInput&&Ur.registerInput(Ot);let{SourceMapConsumer:Si,SourceMapGenerator:Mt}=oe,{dirname:$t,relative:bi,resolve:vi,sep:Ci}=oe,{pathToFileURL:Ii}=oe,dl=Et,pl=!!(Si&&Mt),ml=!!($t&&vi&&bi&&Ci);var xi=class{constructor(e,t,r,i){this.stringify=e,this.mapOpts=r.map||{},this.root=t,this.opts=r,this.css=i,this.originalCSS=i,this.usesFileUrls=!this.mapOpts.from&&this.mapOpts.absolute,this.memoizedFileURLs=new Map,this.memoizedPaths=new Map,this.memoizedURLs=new Map}addAnnotation(){let e;this.isInline()?e="data:application/json;base64,"+this.toBase64(this.map.toString()):typeof this.mapOpts.annotation=="string"?e=this.mapOpts.annotation:typeof this.mapOpts.annotation=="function"?e=this.mapOpts.annotation(this.opts.to,this.root):e=this.outputFile()+".map";let t=`
+`,colon:": ",commentLeft:" ",commentRight:" ",emptyBody:"",indent:" ",semicolon:!1};function t(o){return o[0].toUpperCase()+o.slice(1)}class r{constructor(s){this.builder=s}atrule(s,u){let c="@"+s.name,p=s.params?this.rawValue(s,"params"):"";if(typeof s.raws.afterName<"u"?c+=s.raws.afterName:p&&(c+=" "),s.nodes)this.block(s,c+p);else{let l=(s.raws.between||"")+(u?";":"");this.builder(c+p+l,s)}}beforeAfter(s,u){let c;s.type==="decl"?c=this.raw(s,null,"beforeDecl"):s.type==="comment"?c=this.raw(s,null,"beforeComment"):u==="before"?c=this.raw(s,null,"beforeRule"):c=this.raw(s,null,"beforeClose");let p=s.parent,l=0;for(;p&&p.type!=="root";)l+=1,p=p.parent;if(c.includes(`
+`)){let d=this.raw(s,null,"indent");if(d.length)for(let i=0;i<l;i++)c+=d}return c}block(s,u){let c=this.raw(s,"between","beforeOpen");this.builder(u+c+"{",s,"start");let p;s.nodes&&s.nodes.length?(this.body(s),p=this.raw(s,"after")):p=this.raw(s,"after","emptyBody"),p&&this.builder(p),this.builder("}",s,"end")}body(s){let u=s.nodes.length-1;for(;u>0&&s.nodes[u].type==="comment";)u-=1;let c=this.raw(s,"semicolon");for(let p=0;p<s.nodes.length;p++){let l=s.nodes[p],d=this.raw(l,"before");d&&this.builder(d),this.stringify(l,u!==p||c)}}comment(s){let u=this.raw(s,"left","commentLeft"),c=this.raw(s,"right","commentRight");this.builder("/*"+u+s.text+c+"*/",s)}decl(s,u){let c=this.raw(s,"between","colon"),p=s.prop+c+this.rawValue(s,"value");s.important&&(p+=s.raws.important||" !important"),u&&(p+=";"),this.builder(p,s)}document(s){this.body(s)}raw(s,u,c){let p;if(c||(c=u),u&&(p=s.raws[u],typeof p<"u"))return p;let l=s.parent;if(c==="before"&&(!l||l.type==="root"&&l.first===s||l&&l.type==="document"))return"";if(!l)return e[c];let d=s.root();if(d.rawCache||(d.rawCache={}),typeof d.rawCache[c]<"u")return d.rawCache[c];if(c==="before"||c==="after")return this.beforeAfter(s,c);{let i="raw"+t(c);this[i]?p=this[i](d,s):d.walk(f=>{if(p=f.raws[u],typeof p<"u")return!1})}return typeof p>"u"&&(p=e[c]),d.rawCache[c]=p,p}rawBeforeClose(s){let u;return s.walk(c=>{if(c.nodes&&c.nodes.length>0&&typeof c.raws.after<"u")return u=c.raws.after,u.includes(`
+`)&&(u=u.replace(/[^\n]+$/,"")),!1}),u&&(u=u.replace(/\S/g,"")),u}rawBeforeComment(s,u){let c;return s.walkComments(p=>{if(typeof p.raws.before<"u")return c=p.raws.before,c.includes(`
+`)&&(c=c.replace(/[^\n]+$/,"")),!1}),typeof c>"u"?c=this.raw(u,null,"beforeDecl"):c&&(c=c.replace(/\S/g,"")),c}rawBeforeDecl(s,u){let c;return s.walkDecls(p=>{if(typeof p.raws.before<"u")return c=p.raws.before,c.includes(`
+`)&&(c=c.replace(/[^\n]+$/,"")),!1}),typeof c>"u"?c=this.raw(u,null,"beforeRule"):c&&(c=c.replace(/\S/g,"")),c}rawBeforeOpen(s){let u;return s.walk(c=>{if(c.type!=="decl"&&(u=c.raws.between,typeof u<"u"))return!1}),u}rawBeforeRule(s){let u;return s.walk(c=>{if(c.nodes&&(c.parent!==s||s.first!==c)&&typeof c.raws.before<"u")return u=c.raws.before,u.includes(`
+`)&&(u=u.replace(/[^\n]+$/,"")),!1}),u&&(u=u.replace(/\S/g,"")),u}rawColon(s){let u;return s.walkDecls(c=>{if(typeof c.raws.between<"u")return u=c.raws.between.replace(/[^\s:]/g,""),!1}),u}rawEmptyBody(s){let u;return s.walk(c=>{if(c.nodes&&c.nodes.length===0&&(u=c.raws.after,typeof u<"u"))return!1}),u}rawIndent(s){if(s.raws.indent)return s.raws.indent;let u;return s.walk(c=>{let p=c.parent;if(p&&p!==s&&p.parent&&p.parent===s&&typeof c.raws.before<"u"){let l=c.raws.before.split(`
+`);return u=l[l.length-1],u=u.replace(/\S/g,""),!1}}),u}rawSemicolon(s){let u;return s.walk(c=>{if(c.nodes&&c.nodes.length&&c.last.type==="decl"&&(u=c.raws.semicolon,typeof u<"u"))return!1}),u}rawValue(s,u){let c=s[u],p=s.raws[u];return p&&p.value===c?p.raw:c}root(s){this.body(s),s.raws.after&&this.builder(s.raws.after)}rule(s){this.block(s,this.rawValue(s,"selector")),s.raws.ownSemicolon&&this.builder(s.raws.ownSemicolon,s,"end")}stringify(s,u){if(!this[s.type])throw new Error("Unknown AST node type "+s.type+". Maybe you need to change PostCSS stringifier.");this[s.type](s,u)}}return $t=r,r.default=r,$t}var Ft,Ps;function ot(){if(Ps)return Ft;Ps=1;let e=ks();function t(r,o){new e(o).stringify(r)}return Ft=t,t.default=t,Ft}var Ut,Ts;function at(){if(Ts)return Ut;Ts=1;let{isClean:e,my:t}=_t(),r=Lt(),o=ks(),s=ot();function u(p,l){let d=new p.constructor;for(let i in p){if(!Object.prototype.hasOwnProperty.call(p,i)||i==="proxyCache")continue;let f=p[i],a=typeof f;i==="parent"&&a==="object"?l&&(d[i]=l):i==="source"?d[i]=f:Array.isArray(f)?d[i]=f.map(n=>u(n,d)):(a==="object"&&f!==null&&(f=u(f)),d[i]=f)}return d}class c{constructor(l={}){this.raws={},this[e]=!1,this[t]=!0;for(let d in l)if(d==="nodes"){this.nodes=[];for(let i of l[d])typeof i.clone=="function"?this.append(i.clone()):this.append(i)}else this[d]=l[d]}addToError(l){if(l.postcssNode=this,l.stack&&this.source&&/\n\s{4}at /.test(l.stack)){let d=this.source;l.stack=l.stack.replace(/\n\s{4}at /,`$&${d.input.from}:${d.start.line}:${d.start.column}$&`)}return l}after(l){return this.parent.insertAfter(this,l),this}assign(l={}){for(let d in l)this[d]=l[d];return this}before(l){return this.parent.insertBefore(this,l),this}cleanRaws(l){delete this.raws.before,delete this.raws.after,l||delete this.raws.between}clone(l={}){let d=u(this);for(let i in l)d[i]=l[i];return d}cloneAfter(l={}){let d=this.clone(l);return this.parent.insertAfter(this,d),d}cloneBefore(l={}){let d=this.clone(l);return this.parent.insertBefore(this,d),d}error(l,d={}){if(this.source){let{end:i,start:f}=this.rangeBy(d);return this.source.input.error(l,{column:f.column,line:f.line},{column:i.column,line:i.line},d)}return new r(l)}getProxyProcessor(){return{get(l,d){return d==="proxyOf"?l:d==="root"?()=>l.root().toProxy():l[d]},set(l,d,i){return l[d]===i||(l[d]=i,(d==="prop"||d==="value"||d==="name"||d==="params"||d==="important"||d==="text")&&l.markDirty()),!0}}}markDirty(){if(this[e]){this[e]=!1;let l=this;for(;l=l.parent;)l[e]=!1}}next(){if(!this.parent)return;let l=this.parent.index(this);return this.parent.nodes[l+1]}positionBy(l,d){let i=this.source.start;if(l.index)i=this.positionInside(l.index,d);else if(l.word){d=this.toString();let f=d.indexOf(l.word);f!==-1&&(i=this.positionInside(f,d))}return i}positionInside(l,d){let i=d||this.toString(),f=this.source.start.column,a=this.source.start.line;for(let n=0;n<l;n++)i[n]===`
+`?(f=1,a+=1):f+=1;return{column:f,line:a}}prev(){if(!this.parent)return;let l=this.parent.index(this);return this.parent.nodes[l-1]}rangeBy(l){let d={column:this.source.start.column,line:this.source.start.line},i=this.source.end?{column:this.source.end.column+1,line:this.source.end.line}:{column:d.column+1,line:d.line};if(l.word){let f=this.toString(),a=f.indexOf(l.word);a!==-1&&(d=this.positionInside(a,f),i=this.positionInside(a+l.word.length,f))}else l.start?d={column:l.start.column,line:l.start.line}:l.index&&(d=this.positionInside(l.index)),l.end?i={column:l.end.column,line:l.end.line}:typeof l.endIndex=="number"?i=this.positionInside(l.endIndex):l.index&&(i=this.positionInside(l.index+1));return(i.line<d.line||i.line===d.line&&i.column<=d.column)&&(i={column:d.column+1,line:d.line}),{end:i,start:d}}raw(l,d){return new o().raw(this,l,d)}remove(){return this.parent&&this.parent.removeChild(this),this.parent=void 0,this}replaceWith(...l){if(this.parent){let d=this,i=!1;for(let f of l)f===this?i=!0:i?(this.parent.insertAfter(d,f),d=f):this.parent.insertBefore(d,f);i||this.remove()}return this}root(){let l=this;for(;l.parent&&l.parent.type!=="document";)l=l.parent;return l}toJSON(l,d){let i={},f=d==null;d=d||new Map;let a=0;for(let n in this){if(!Object.prototype.hasOwnProperty.call(this,n)||n==="parent"||n==="proxyCache")continue;let m=this[n];if(Array.isArray(m))i[n]=m.map(h=>typeof h=="object"&&h.toJSON?h.toJSON(null,d):h);else if(typeof m=="object"&&m.toJSON)i[n]=m.toJSON(null,d);else if(n==="source"){let h=d.get(m.input);h==null&&(h=a,d.set(m.input,a),a++),i[n]={end:m.end,inputId:h,start:m.start}}else i[n]=m}return f&&(i.inputs=[...d.keys()].map(n=>n.toJSON())),i}toProxy(){return this.proxyCache||(this.proxyCache=new Proxy(this,this.getProxyProcessor())),this.proxyCache}toString(l=s){l.stringify&&(l=l.stringify);let d="";return l(this,i=>{d+=i}),d}warn(l,d,i){let f={node:this};for(let a in i)f[a]=i[a];return l.warn(d,f)}get proxyOf(){return this}}return Ut=c,c.default=c,Ut}var Bt,Ds;function lt(){if(Ds)return Bt;Ds=1;let e=at();class t extends e{constructor(o){o&&typeof o.value<"u"&&typeof o.value!="string"&&(o={...o,value:String(o.value)}),super(o),this.type="decl"}get variable(){return this.prop.startsWith("--")||this.prop[0]==="$"}}return Bt=t,t.default=t,Bt}var zt,Ls;function Po(){if(Ls)return zt;Ls=1;let e="useandom-26T198340PX75pxJACKVERYMINDBUSHWOLF_GQZbfghjklqvwyzrict";return zt={nanoid:(o=21)=>{let s="",u=o;for(;u--;)s+=e[Math.random()*64|0];return s},customAlphabet:(o,s=21)=>(u=s)=>{let c="",p=u;for(;p--;)c+=o[Math.random()*o.length|0];return c}},zt}var Wt,_s;function $s(){if(_s)return Wt;_s=1;let{SourceMapConsumer:e,SourceMapGenerator:t}=pe,{existsSync:r,readFileSync:o}=pe,{dirname:s,join:u}=pe;function c(l){return Buffer?Buffer.from(l,"base64").toString():window.atob(l)}class p{constructor(d,i){if(i.map===!1)return;this.loadAnnotation(d),this.inline=this.startWith(this.annotation,"data:");let f=i.map?i.map.prev:void 0,a=this.loadMap(i.from,f);!this.mapFile&&i.from&&(this.mapFile=i.from),this.mapFile&&(this.root=s(this.mapFile)),a&&(this.text=a)}consumer(){return this.consumerCache||(this.consumerCache=new e(this.text)),this.consumerCache}decodeInline(d){let i=/^data:application\/json;charset=utf-?8;base64,/,f=/^data:application\/json;base64,/,a=/^data:application\/json;charset=utf-?8,/,n=/^data:application\/json,/;if(a.test(d)||n.test(d))return decodeURIComponent(d.substr(RegExp.lastMatch.length));if(i.test(d)||f.test(d))return c(d.substr(RegExp.lastMatch.length));let m=d.match(/data:application\/json;([^,]+),/)[1];throw new Error("Unsupported source map encoding "+m)}getAnnotationURL(d){return d.replace(/^\/\*\s*# sourceMappingURL=/,"").trim()}isMap(d){return typeof d!="object"?!1:typeof d.mappings=="string"||typeof d._mappings=="string"||Array.isArray(d.sections)}loadAnnotation(d){let i=d.match(/\/\*\s*# sourceMappingURL=/gm);if(!i)return;let f=d.lastIndexOf(i.pop()),a=d.indexOf("*/",f);f>-1&&a>-1&&(this.annotation=this.getAnnotationURL(d.substring(f,a)))}loadFile(d){if(this.root=s(d),r(d))return this.mapFile=d,o(d,"utf-8").toString().trim()}loadMap(d,i){if(i===!1)return!1;if(i){if(typeof i=="string")return i;if(typeof i=="function"){let f=i(d);if(f){let a=this.loadFile(f);if(!a)throw new Error("Unable to load previous source map: "+f.toString());return a}}else{if(i instanceof e)return t.fromSourceMap(i).toString();if(i instanceof t)return i.toString();if(this.isMap(i))return JSON.stringify(i);throw new Error("Unsupported previous source map format: "+i.toString())}}else{if(this.inline)return this.decodeInline(this.annotation);if(this.annotation){let f=this.annotation;return d&&(f=u(s(d),f)),this.loadFile(f)}}}startWith(d,i){return d?d.substr(0,i.length)===i:!1}withContent(){return!!(this.consumer().sourcesContent&&this.consumer().sourcesContent.length>0)}}return Wt=p,p.default=p,Wt}var qt,Fs;function ut(){if(Fs)return qt;Fs=1;let{SourceMapConsumer:e,SourceMapGenerator:t}=pe,{fileURLToPath:r,pathToFileURL:o}=pe,{isAbsolute:s,resolve:u}=pe,{nanoid:c}=Po(),p=pe,l=Lt(),d=$s(),i=Symbol("fromOffsetCache"),f=!!(e&&t),a=!!(u&&s);class n{constructor(h,g={}){if(h===null||typeof h>"u"||typeof h=="object"&&!h.toString)throw new Error(`PostCSS received ${h} instead of CSS string`);if(this.css=h.toString(),this.css[0]==="\uFEFF"||this.css[0]==="￾"?(this.hasBOM=!0,this.css=this.css.slice(1)):this.hasBOM=!1,g.from&&(!a||/^\w+:\/\//.test(g.from)||s(g.from)?this.file=g.from:this.file=u(g.from)),a&&f){let v=new d(this.css,g);if(v.text){this.map=v;let b=v.consumer().file;!this.file&&b&&(this.file=this.mapResolve(b))}}this.file||(this.id="<input css "+c(6)+">"),this.map&&(this.map.file=this.from)}error(h,g,v,b={}){let S,x,w;if(g&&typeof g=="object"){let C=g,O=v;if(typeof C.offset=="number"){let I=this.fromOffset(C.offset);g=I.line,v=I.col}else g=C.line,v=C.column;if(typeof O.offset=="number"){let I=this.fromOffset(O.offset);x=I.line,w=I.col}else x=O.line,w=O.column}else if(!v){let C=this.fromOffset(g);g=C.line,v=C.col}let y=this.origin(g,v,x,w);return y?S=new l(h,y.endLine===void 0?y.line:{column:y.column,line:y.line},y.endLine===void 0?y.column:{column:y.endColumn,line:y.endLine},y.source,y.file,b.plugin):S=new l(h,x===void 0?g:{column:v,line:g},x===void 0?v:{column:w,line:x},this.css,this.file,b.plugin),S.input={column:v,endColumn:w,endLine:x,line:g,source:this.css},this.file&&(o&&(S.input.url=o(this.file).toString()),S.input.file=this.file),S}fromOffset(h){let g,v;if(this[i])v=this[i];else{let S=this.css.split(`
+`);v=new Array(S.length);let x=0;for(let w=0,y=S.length;w<y;w++)v[w]=x,x+=S[w].length+1;this[i]=v}g=v[v.length-1];let b=0;if(h>=g)b=v.length-1;else{let S=v.length-2,x;for(;b<S;)if(x=b+(S-b>>1),h<v[x])S=x-1;else if(h>=v[x+1])b=x+1;else{b=x;break}}return{col:h-v[b]+1,line:b+1}}mapResolve(h){return/^\w+:\/\//.test(h)?h:u(this.map.consumer().sourceRoot||this.map.root||".",h)}origin(h,g,v,b){if(!this.map)return!1;let S=this.map.consumer(),x=S.originalPositionFor({column:g,line:h});if(!x.source)return!1;let w;typeof v=="number"&&(w=S.originalPositionFor({column:b,line:v}));let y;s(x.source)?y=o(x.source):y=new URL(x.source,this.map.consumer().sourceRoot||o(this.map.mapFile));let C={column:x.column,endColumn:w&&w.column,endLine:w&&w.line,line:x.line,url:y.toString()};if(y.protocol==="file:")if(r)C.file=r(y);else throw new Error("file: protocol is not available in this PostCSS build");let O=S.sourceContentFor(x.source);return O&&(C.source=O),C}toJSON(){let h={};for(let g of["hasBOM","css","file","id"])this[g]!=null&&(h[g]=this[g]);return this.map&&(h.map={...this.map},h.map.consumerCache&&(h.map.consumerCache=void 0)),h}get from(){return this.file||this.id}}return qt=n,n.default=n,p&&p.registerInput&&p.registerInput(n),qt}var jt,Us;function Bs(){if(Us)return jt;Us=1;let{SourceMapConsumer:e,SourceMapGenerator:t}=pe,{dirname:r,relative:o,resolve:s,sep:u}=pe,{pathToFileURL:c}=pe,p=ut(),l=!!(e&&t),d=!!(r&&s&&o&&u);class i{constructor(a,n,m,h){this.stringify=a,this.mapOpts=m.map||{},this.root=n,this.opts=m,this.css=h,this.originalCSS=h,this.usesFileUrls=!this.mapOpts.from&&this.mapOpts.absolute,this.memoizedFileURLs=new Map,this.memoizedPaths=new Map,this.memoizedURLs=new Map}addAnnotation(){let a;this.isInline()?a="data:application/json;base64,"+this.toBase64(this.map.toString()):typeof this.mapOpts.annotation=="string"?a=this.mapOpts.annotation:typeof this.mapOpts.annotation=="function"?a=this.mapOpts.annotation(this.opts.to,this.root):a=this.outputFile()+".map";let n=`
`;this.css.includes(`\r
-`)&&(t=`\r
-`),this.css+=t+"/*# sourceMappingURL="+e+" */"}applyPrevMaps(){for(let e of this.previous()){let t=this.toUrl(this.path(e.file)),r=e.root||$t(e.file),i;this.mapOpts.sourcesContent===!1?(i=new Si(e.text),i.sourcesContent&&(i.sourcesContent=null)):i=e.consumer(),this.map.applySourceMap(i,t,this.toUrl(this.path(r)))}}clearAnnotation(){if(this.mapOpts.annotation!==!1)if(this.root){let e;for(let t=this.root.nodes.length-1;t>=0;t--)e=this.root.nodes[t],e.type==="comment"&&e.text.indexOf("# sourceMappingURL=")===0&&this.root.removeChild(t)}else this.css&&(this.css=this.css.replace(/\n*?\/\*#[\S\s]*?\*\/$/gm,""))}generate(){if(this.clearAnnotation(),ml&&pl&&this.isMap())return this.generateMap();{let e="";return this.stringify(this.root,t=>{e+=t}),[e]}}generateMap(){if(this.root)this.generateString();else if(this.previous().length===1){let e=this.previous()[0].consumer();e.file=this.outputFile(),this.map=Mt.fromSourceMap(e,{ignoreInvalidMapping:!0})}else this.map=new Mt({file:this.outputFile(),ignoreInvalidMapping:!0}),this.map.addMapping({generated:{column:0,line:1},original:{column:0,line:1},source:this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>"});return this.isSourcesContent()&&this.setSourcesContent(),this.root&&this.previous().length>0&&this.applyPrevMaps(),this.isAnnotation()&&this.addAnnotation(),this.isInline()?[this.css]:[this.css,this.map]}generateString(){this.css="",this.map=new Mt({file:this.outputFile(),ignoreInvalidMapping:!0});let e=1,t=1,r="<no source>",i={generated:{column:0,line:0},original:{column:0,line:0},source:""},n,o;this.stringify(this.root,(l,a,u)=>{if(this.css+=l,a&&u!=="end"&&(i.generated.line=e,i.generated.column=t-1,a.source&&a.source.start?(i.source=this.sourcePath(a),i.original.line=a.source.start.line,i.original.column=a.source.start.column-1,this.map.addMapping(i)):(i.source=r,i.original.line=1,i.original.column=0,this.map.addMapping(i))),n=l.match(/\n/g),n?(e+=n.length,o=l.lastIndexOf(`
-`),t=l.length-o):t+=l.length,a&&u!=="start"){let c=a.parent||{raws:{}};(!(a.type==="decl"||a.type==="atrule"&&!a.nodes)||a!==c.last||c.raws.semicolon)&&(a.source&&a.source.end?(i.source=this.sourcePath(a),i.original.line=a.source.end.line,i.original.column=a.source.end.column-1,i.generated.line=e,i.generated.column=t-2,this.map.addMapping(i)):(i.source=r,i.original.line=1,i.original.column=0,i.generated.line=e,i.generated.column=t-1,this.map.addMapping(i)))}})}isAnnotation(){return this.isInline()?!0:typeof this.mapOpts.annotation<"u"?this.mapOpts.annotation:this.previous().length?this.previous().some(e=>e.annotation):!0}isInline(){if(typeof this.mapOpts.inline<"u")return this.mapOpts.inline;let e=this.mapOpts.annotation;return typeof e<"u"&&e!==!0?!1:this.previous().length?this.previous().some(t=>t.inline):!0}isMap(){return typeof this.opts.map<"u"?!!this.opts.map:this.previous().length>0}isSourcesContent(){return typeof this.mapOpts.sourcesContent<"u"?this.mapOpts.sourcesContent:this.previous().length?this.previous().some(e=>e.withContent()):!0}outputFile(){return this.opts.to?this.path(this.opts.to):this.opts.from?this.path(this.opts.from):"to.css"}path(e){if(this.mapOpts.absolute||e.charCodeAt(0)===60||/^\w+:\/\//.test(e))return e;let t=this.memoizedPaths.get(e);if(t)return t;let r=this.opts.to?$t(this.opts.to):".";typeof this.mapOpts.annotation=="string"&&(r=$t(vi(r,this.mapOpts.annotation)));let i=bi(r,e);return this.memoizedPaths.set(e,i),i}previous(){if(!this.previousMaps)if(this.previousMaps=[],this.root)this.root.walk(e=>{if(e.source&&e.source.input.map){let t=e.source.input.map;this.previousMaps.includes(t)||this.previousMaps.push(t)}});else{let e=new dl(this.originalCSS,this.opts);e.map&&this.previousMaps.push(e.map)}return this.previousMaps}setSourcesContent(){let e={};if(this.root)this.root.walk(t=>{if(t.source){let r=t.source.input.from;if(r&&!e[r]){e[r]=!0;let i=this.usesFileUrls?this.toFileUrl(r):this.toUrl(this.path(r));this.map.setSourceContent(i,t.source.input.css)}}});else if(this.css){let t=this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>";this.map.setSourceContent(t,this.css)}}sourcePath(e){return this.mapOpts.from?this.toUrl(this.mapOpts.from):this.usesFileUrls?this.toFileUrl(e.source.input.from):this.toUrl(this.path(e.source.input.from))}toBase64(e){return Buffer?Buffer.from(e).toString("base64"):window.btoa(unescape(encodeURIComponent(e)))}toFileUrl(e){let t=this.memoizedFileURLs.get(e);if(t)return t;if(Ii){let r=Ii(e).toString();return this.memoizedFileURLs.set(e,r),r}else throw new Error("`map.absolute` option is not available in this PostCSS build")}toUrl(e){let t=this.memoizedURLs.get(e);if(t)return t;Ci==="\\"&&(e=e.replace(/\\/g,"/"));let r=encodeURI(e).replace(/[#?]/g,encodeURIComponent);return this.memoizedURLs.set(e,r),r}};let gl=xt,zr=class extends gl{constructor(e){super(e),this.type="comment"}};var Nt=zr;zr.default=zr;let{isClean:Ai,my:Ri}=Je,Oi=At,Ei=Nt,yl=xt,Mi,Wr,Gr,$i;function Ni(s){return s.map(e=>(e.nodes&&(e.nodes=Ni(e.nodes)),delete e.source,e))}function ki(s){if(s[Ai]=!1,s.proxyOf.nodes)for(let e of s.proxyOf.nodes)ki(e)}let we=class Ho extends yl{append(...e){for(let t of e){let r=this.normalize(t,this.last);for(let i of r)this.proxyOf.nodes.push(i)}return this.markDirty(),this}cleanRaws(e){if(super.cleanRaws(e),this.nodes)for(let t of this.nodes)t.cleanRaws(e)}each(e){if(!this.proxyOf.nodes)return;let t=this.getIterator(),r,i;for(;this.indexes[t]<this.proxyOf.nodes.length&&(r=this.indexes[t],i=e(this.proxyOf.nodes[r],r),i!==!1);)this.indexes[t]+=1;return delete this.indexes[t],i}every(e){return this.nodes.every(e)}getIterator(){this.lastEach||(this.lastEach=0),this.indexes||(this.indexes={}),this.lastEach+=1;let e=this.lastEach;return this.indexes[e]=0,e}getProxyProcessor(){return{get(e,t){return t==="proxyOf"?e:e[t]?t==="each"||typeof t=="string"&&t.startsWith("walk")?(...r)=>e[t](...r.map(i=>typeof i=="function"?(n,o)=>i(n.toProxy(),o):i)):t==="every"||t==="some"?r=>e[t]((i,...n)=>r(i.toProxy(),...n)):t==="root"?()=>e.root().toProxy():t==="nodes"?e.nodes.map(r=>r.toProxy()):t==="first"||t==="last"?e[t].toProxy():e[t]:e[t]},set(e,t,r){return e[t]===r||(e[t]=r,(t==="name"||t==="params"||t==="selector")&&e.markDirty()),!0}}}index(e){return typeof e=="number"?e:(e.proxyOf&&(e=e.proxyOf),this.proxyOf.nodes.indexOf(e))}insertAfter(e,t){let r=this.index(e),i=this.normalize(t,this.proxyOf.nodes[r]).reverse();r=this.index(e);for(let o of i)this.proxyOf.nodes.splice(r+1,0,o);let n;for(let o in this.indexes)n=this.indexes[o],r<n&&(this.indexes[o]=n+i.length);return this.markDirty(),this}insertBefore(e,t){let r=this.index(e),i=r===0?"prepend":!1,n=this.normalize(t,this.proxyOf.nodes[r],i).reverse();r=this.index(e);for(let l of n)this.proxyOf.nodes.splice(r,0,l);let o;for(let l in this.indexes)o=this.indexes[l],r<=o&&(this.indexes[l]=o+n.length);return this.markDirty(),this}normalize(e,t){if(typeof e=="string")e=Ni(Mi(e).nodes);else if(typeof e>"u")e=[];else if(Array.isArray(e)){e=e.slice(0);for(let i of e)i.parent&&i.parent.removeChild(i,"ignore")}else if(e.type==="root"&&this.type!=="document"){e=e.nodes.slice(0);for(let i of e)i.parent&&i.parent.removeChild(i,"ignore")}else if(e.type)e=[e];else if(e.prop){if(typeof e.value>"u")throw new Error("Value field is missed in node creation");typeof e.value!="string"&&(e.value=String(e.value)),e=[new Oi(e)]}else if(e.selector)e=[new Wr(e)];else if(e.name)e=[new Gr(e)];else if(e.text)e=[new Ei(e)];else throw new Error("Unknown node type in node creation");return e.map(i=>(i[Ri]||Ho.rebuild(i),i=i.proxyOf,i.parent&&i.parent.removeChild(i),i[Ai]&&ki(i),typeof i.raws.before>"u"&&t&&typeof t.raws.before<"u"&&(i.raws.before=t.raws.before.replace(/\S/g,"")),i.parent=this.proxyOf,i))}prepend(...e){e=e.reverse();for(let t of e){let r=this.normalize(t,this.first,"prepend").reverse();for(let i of r)this.proxyOf.nodes.unshift(i);for(let i in this.indexes)this.indexes[i]=this.indexes[i]+r.length}return this.markDirty(),this}push(e){return e.parent=this,this.proxyOf.nodes.push(e),this}removeAll(){for(let e of this.proxyOf.nodes)e.parent=void 0;return this.proxyOf.nodes=[],this.markDirty(),this}removeChild(e){e=this.index(e),this.proxyOf.nodes[e].parent=void 0,this.proxyOf.nodes.splice(e,1);let t;for(let r in this.indexes)t=this.indexes[r],t>=e&&(this.indexes[r]=t-1);return this.markDirty(),this}replaceValues(e,t,r){return r||(r=t,t={}),this.walkDecls(i=>{t.props&&!t.props.includes(i.prop)||t.fast&&!i.value.includes(t.fast)||(i.value=i.value.replace(e,r))}),this.markDirty(),this}some(e){return this.nodes.some(e)}walk(e){return this.each((t,r)=>{let i;try{i=e(t,r)}catch(n){throw t.addToError(n)}return i!==!1&&t.walk&&(i=t.walk(e)),i})}walkAtRules(e,t){return t?e instanceof RegExp?this.walk((r,i)=>{if(r.type==="atrule"&&e.test(r.name))return t(r,i)}):this.walk((r,i)=>{if(r.type==="atrule"&&r.name===e)return t(r,i)}):(t=e,this.walk((r,i)=>{if(r.type==="atrule")return t(r,i)}))}walkComments(e){return this.walk((t,r)=>{if(t.type==="comment")return e(t,r)})}walkDecls(e,t){return t?e instanceof RegExp?this.walk((r,i)=>{if(r.type==="decl"&&e.test(r.prop))return t(r,i)}):this.walk((r,i)=>{if(r.type==="decl"&&r.prop===e)return t(r,i)}):(t=e,this.walk((r,i)=>{if(r.type==="decl")return t(r,i)}))}walkRules(e,t){return t?e instanceof RegExp?this.walk((r,i)=>{if(r.type==="rule"&&e.test(r.selector))return t(r,i)}):this.walk((r,i)=>{if(r.type==="rule"&&r.selector===e)return t(r,i)}):(t=e,this.walk((r,i)=>{if(r.type==="rule")return t(r,i)}))}get first(){if(this.proxyOf.nodes)return this.proxyOf.nodes[0]}get last(){if(this.proxyOf.nodes)return this.proxyOf.nodes[this.proxyOf.nodes.length-1]}};we.registerParse=s=>{Mi=s},we.registerRule=s=>{Wr=s},we.registerAtRule=s=>{Gr=s},we.registerRoot=s=>{$i=s};var xe=we;we.default=we,we.rebuild=s=>{s.type==="atrule"?Object.setPrototypeOf(s,Gr.prototype):s.type==="rule"?Object.setPrototypeOf(s,Wr.prototype):s.type==="decl"?Object.setPrototypeOf(s,Oi.prototype):s.type==="comment"?Object.setPrototypeOf(s,Ei.prototype):s.type==="root"&&Object.setPrototypeOf(s,$i.prototype),s[Ri]=!0,s.nodes&&s.nodes.forEach(e=>{we.rebuild(e)})};let wl=xe,Ti,Pi,Ke=class extends wl{constructor(e){super({type:"document",...e}),this.nodes||(this.nodes=[])}toResult(e={}){return new Ti(new Pi,this,e).stringify()}};Ke.registerLazyResult=s=>{Ti=s},Ke.registerProcessor=s=>{Pi=s};var Vr=Ke;Ke.default=Ke;let Di={};var Li=function(e){Di[e]||(Di[e]=!0,typeof console<"u"&&console.warn&&console.warn(e))};let jr=class{constructor(e,t={}){if(this.type="warning",this.text=e,t.node&&t.node.source){let r=t.node.rangeBy(t);this.line=r.start.line,this.column=r.start.column,this.endLine=r.end.line,this.endColumn=r.end.column}for(let r in t)this[r]=t[r]}toString(){return this.node?this.node.error(this.text,{index:this.index,plugin:this.plugin,word:this.word}).message:this.plugin?this.plugin+": "+this.text:this.text}};var _i=jr;jr.default=jr;let Sl=_i,Hr=class{constructor(e,t,r){this.processor=e,this.messages=[],this.root=t,this.opts=r,this.css=void 0,this.map=void 0}toString(){return this.css}warn(e,t={}){t.plugin||this.lastPlugin&&this.lastPlugin.postcssPlugin&&(t.plugin=this.lastPlugin.postcssPlugin);let r=new Sl(e,t);return this.messages.push(r),r}warnings(){return this.messages.filter(e=>e.type==="warning")}get content(){return this.css}};var Yr=Hr;Hr.default=Hr;const Zr=39,Fi=34,kt=92,Ui=47,Tt=10,Xe=32,Pt=12,Dt=9,Lt=13,bl=91,vl=93,Cl=40,Il=41,xl=123,Al=125,Rl=59,Ol=42,El=58,Ml=64,_t=/[\t\n\f\r "#'()/;[\\\]{}]/g,Ft=/[\t\n\f\r !"#'():;@[\\\]{}]|\/(?=\*)/g,$l=/.[\r\n"'(/\\]/,Bi=/[\da-f]/i;var Nl=function(e,t={}){let r=e.css.valueOf(),i=t.ignoreErrors,n,o,l,a,u,c,h,p,m,g,d=r.length,f=0,S=[],b=[];function y(){return f}function C(B){throw e.error("Unclosed "+B,f)}function k(){return b.length===0&&f>=d}function D(B){if(b.length)return b.pop();if(f>=d)return;let V=B?B.ignoreUnclosed:!1;switch(n=r.charCodeAt(f),n){case Tt:case Xe:case Dt:case Lt:case Pt:{o=f;do o+=1,n=r.charCodeAt(o);while(n===Xe||n===Tt||n===Dt||n===Lt||n===Pt);g=["space",r.slice(f,o)],f=o-1;break}case bl:case vl:case xl:case Al:case El:case Rl:case Il:{let Z=String.fromCharCode(n);g=[Z,Z,f];break}case Cl:{if(p=S.length?S.pop()[1]:"",m=r.charCodeAt(f+1),p==="url"&&m!==Zr&&m!==Fi&&m!==Xe&&m!==Tt&&m!==Dt&&m!==Pt&&m!==Lt){o=f;do{if(c=!1,o=r.indexOf(")",o+1),o===-1)if(i||V){o=f;break}else C("bracket");for(h=o;r.charCodeAt(h-1)===kt;)h-=1,c=!c}while(c);g=["brackets",r.slice(f,o+1),f,o],f=o}else o=r.indexOf(")",f+1),a=r.slice(f,o+1),o===-1||$l.test(a)?g=["(","(",f]:(g=["brackets",a,f,o],f=o);break}case Zr:case Fi:{l=n===Zr?"'":'"',o=f;do{if(c=!1,o=r.indexOf(l,o+1),o===-1)if(i||V){o=f+1;break}else C("string");for(h=o;r.charCodeAt(h-1)===kt;)h-=1,c=!c}while(c);g=["string",r.slice(f,o+1),f,o],f=o;break}case Ml:{_t.lastIndex=f+1,_t.test(r),_t.lastIndex===0?o=r.length-1:o=_t.lastIndex-2,g=["at-word",r.slice(f,o+1),f,o],f=o;break}case kt:{for(o=f,u=!0;r.charCodeAt(o+1)===kt;)o+=1,u=!u;if(n=r.charCodeAt(o+1),u&&n!==Ui&&n!==Xe&&n!==Tt&&n!==Dt&&n!==Lt&&n!==Pt&&(o+=1,Bi.test(r.charAt(o)))){for(;Bi.test(r.charAt(o+1));)o+=1;r.charCodeAt(o+1)===Xe&&(o+=1)}g=["word",r.slice(f,o+1),f,o],f=o;break}default:{n===Ui&&r.charCodeAt(f+1)===Ol?(o=r.indexOf("*/",f+2)+1,o===0&&(i||V?o=r.length:C("comment")),g=["comment",r.slice(f,o+1),f,o],f=o):(Ft.lastIndex=f+1,Ft.test(r),Ft.lastIndex===0?o=r.length-1:o=Ft.lastIndex-2,g=["word",r.slice(f,o+1),f,o],S.push(g),f=o);break}}return f++,g}function H(B){b.push(B)}return{back:H,endOfFile:k,nextToken:D,position:y}};let zi=xe,Ut=class extends zi{constructor(e){super(e),this.type="atrule"}append(...e){return this.proxyOf.nodes||(this.nodes=[]),super.append(...e)}prepend(...e){return this.proxyOf.nodes||(this.nodes=[]),super.prepend(...e)}};var Jr=Ut;Ut.default=Ut,zi.registerAtRule(Ut);let Wi=xe,Gi,Vi,De=class extends Wi{constructor(e){super(e),this.type="root",this.nodes||(this.nodes=[])}normalize(e,t,r){let i=super.normalize(e);if(t){if(r==="prepend")this.nodes.length>1?t.raws.before=this.nodes[1].raws.before:delete t.raws.before;else if(this.first!==t)for(let n of i)n.raws.before=t.raws.before}return i}removeChild(e,t){let r=this.index(e);return!t&&r===0&&this.nodes.length>1&&(this.nodes[1].raws.before=this.nodes[r].raws.before),super.removeChild(e)}toResult(e={}){return new Gi(new Vi,this,e).stringify()}};De.registerLazyResult=s=>{Gi=s},De.registerProcessor=s=>{Vi=s};var Qe=De;De.default=De,Wi.registerRoot(De);let qe={comma(s){return qe.split(s,[","],!0)},space(s){let e=[" ",`
-`," "];return qe.split(s,e)},split(s,e,t){let r=[],i="",n=!1,o=0,l=!1,a="",u=!1;for(let c of s)u?u=!1:c==="\\"?u=!0:l?c===a&&(l=!1):c==='"'||c==="'"?(l=!0,a=c):c==="("?o+=1:c===")"?o>0&&(o-=1):o===0&&e.includes(c)&&(n=!0),n?(i!==""&&r.push(i.trim()),i="",n=!1):i+=c;return(t||i!=="")&&r.push(i.trim()),r}};var ji=qe;qe.default=qe;let Hi=xe,kl=ji,Bt=class extends Hi{constructor(e){super(e),this.type="rule",this.nodes||(this.nodes=[])}get selectors(){return kl.comma(this.selector)}set selectors(e){let t=this.selector?this.selector.match(/,\s*/):null,r=t?t[0]:","+this.raw("between","beforeOpen");this.selector=e.join(r)}};var Kr=Bt;Bt.default=Bt,Hi.registerRule(Bt);let Tl=At,Pl=Nl,Dl=Nt,Ll=Jr,_l=Qe,Yi=Kr;const Zi={empty:!0,space:!0};function Fl(s){for(let e=s.length-1;e>=0;e--){let t=s[e],r=t[3]||t[2];if(r)return r}}var Ul=class{constructor(e){this.input=e,this.root=new _l,this.current=this.root,this.spaces="",this.semicolon=!1,this.createTokenizer(),this.root.source={input:e,start:{column:1,line:1,offset:0}}}atrule(e){let t=new Ll;t.name=e[1].slice(1),t.name===""&&this.unnamedAtrule(t,e),this.init(t,e[2]);let r,i,n,o=!1,l=!1,a=[],u=[];for(;!this.tokenizer.endOfFile();){if(e=this.tokenizer.nextToken(),r=e[0],r==="("||r==="["?u.push(r==="("?")":"]"):r==="{"&&u.length>0?u.push("}"):r===u[u.length-1]&&u.pop(),u.length===0)if(r===";"){t.source.end=this.getPosition(e[2]),t.source.end.offset++,this.semicolon=!0;break}else if(r==="{"){l=!0;break}else if(r==="}"){if(a.length>0){for(n=a.length-1,i=a[n];i&&i[0]==="space";)i=a[--n];i&&(t.source.end=this.getPosition(i[3]||i[2]),t.source.end.offset++)}this.end(e);break}else a.push(e);else a.push(e);if(this.tokenizer.endOfFile()){o=!0;break}}t.raws.between=this.spacesAndCommentsFromEnd(a),a.length?(t.raws.afterName=this.spacesAndCommentsFromStart(a),this.raw(t,"params",a),o&&(e=a[a.length-1],t.source.end=this.getPosition(e[3]||e[2]),t.source.end.offset++,this.spaces=t.raws.between,t.raws.between="")):(t.raws.afterName="",t.params=""),l&&(t.nodes=[],this.current=t)}checkMissedSemicolon(e){let t=this.colon(e);if(t===!1)return;let r=0,i;for(let n=t-1;n>=0&&(i=e[n],!(i[0]!=="space"&&(r+=1,r===2)));n--);throw this.input.error("Missed semicolon",i[0]==="word"?i[3]+1:i[2])}colon(e){let t=0,r,i,n;for(let[o,l]of e.entries()){if(r=l,i=r[0],i==="("&&(t+=1),i===")"&&(t-=1),t===0&&i===":")if(!n)this.doubleColon(r);else{if(n[0]==="word"&&n[1]==="progid")continue;return o}n=r}return!1}comment(e){let t=new Dl;this.init(t,e[2]),t.source.end=this.getPosition(e[3]||e[2]),t.source.end.offset++;let r=e[1].slice(2,-2);if(/^\s*$/.test(r))t.text="",t.raws.left=r,t.raws.right="";else{let i=r.match(/^(\s*)([^]*\S)(\s*)$/);t.text=i[2],t.raws.left=i[1],t.raws.right=i[3]}}createTokenizer(){this.tokenizer=Pl(this.input)}decl(e,t){let r=new Tl;this.init(r,e[0][2]);let i=e[e.length-1];for(i[0]===";"&&(this.semicolon=!0,e.pop()),r.source.end=this.getPosition(i[3]||i[2]||Fl(e)),r.source.end.offset++;e[0][0]!=="word";)e.length===1&&this.unknownWord(e),r.raws.before+=e.shift()[1];for(r.source.start=this.getPosition(e[0][2]),r.prop="";e.length;){let u=e[0][0];if(u===":"||u==="space"||u==="comment")break;r.prop+=e.shift()[1]}r.raws.between="";let n;for(;e.length;)if(n=e.shift(),n[0]===":"){r.raws.between+=n[1];break}else n[0]==="word"&&/\w/.test(n[1])&&this.unknownWord([n]),r.raws.between+=n[1];(r.prop[0]==="_"||r.prop[0]==="*")&&(r.raws.before+=r.prop[0],r.prop=r.prop.slice(1));let o=[],l;for(;e.length&&(l=e[0][0],!(l!=="space"&&l!=="comment"));)o.push(e.shift());this.precheckMissedSemicolon(e);for(let u=e.length-1;u>=0;u--){if(n=e[u],n[1].toLowerCase()==="!important"){r.important=!0;let c=this.stringFrom(e,u);c=this.spacesFromEnd(e)+c,c!==" !important"&&(r.raws.important=c);break}else if(n[1].toLowerCase()==="important"){let c=e.slice(0),h="";for(let p=u;p>0;p--){let m=c[p][0];if(h.trim().indexOf("!")===0&&m!=="space")break;h=c.pop()[1]+h}h.trim().indexOf("!")===0&&(r.important=!0,r.raws.important=h,e=c)}if(n[0]!=="space"&&n[0]!=="comment")break}e.some(u=>u[0]!=="space"&&u[0]!=="comment")&&(r.raws.between+=o.map(u=>u[1]).join(""),o=[]),this.raw(r,"value",o.concat(e),t),r.value.includes(":")&&!t&&this.checkMissedSemicolon(e)}doubleColon(e){throw this.input.error("Double colon",{offset:e[2]},{offset:e[2]+e[1].length})}emptyRule(e){let t=new Yi;this.init(t,e[2]),t.selector="",t.raws.between="",this.current=t}end(e){this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.semicolon=!1,this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.spaces="",this.current.parent?(this.current.source.end=this.getPosition(e[2]),this.current.source.end.offset++,this.current=this.current.parent):this.unexpectedClose(e)}endFile(){this.current.parent&&this.unclosedBlock(),this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.root.source.end=this.getPosition(this.tokenizer.position())}freeSemicolon(e){if(this.spaces+=e[1],this.current.nodes){let t=this.current.nodes[this.current.nodes.length-1];t&&t.type==="rule"&&!t.raws.ownSemicolon&&(t.raws.ownSemicolon=this.spaces,this.spaces="")}}getPosition(e){let t=this.input.fromOffset(e);return{column:t.col,line:t.line,offset:e}}init(e,t){this.current.push(e),e.source={input:this.input,start:this.getPosition(t)},e.raws.before=this.spaces,this.spaces="",e.type!=="comment"&&(this.semicolon=!1)}other(e){let t=!1,r=null,i=!1,n=null,o=[],l=e[1].startsWith("--"),a=[],u=e;for(;u;){if(r=u[0],a.push(u),r==="("||r==="[")n||(n=u),o.push(r==="("?")":"]");else if(l&&i&&r==="{")n||(n=u),o.push("}");else if(o.length===0)if(r===";")if(i){this.decl(a,l);return}else break;else if(r==="{"){this.rule(a);return}else if(r==="}"){this.tokenizer.back(a.pop()),t=!0;break}else r===":"&&(i=!0);else r===o[o.length-1]&&(o.pop(),o.length===0&&(n=null));u=this.tokenizer.nextToken()}if(this.tokenizer.endOfFile()&&(t=!0),o.length>0&&this.unclosedBracket(n),t&&i){if(!l)for(;a.length&&(u=a[a.length-1][0],!(u!=="space"&&u!=="comment"));)this.tokenizer.back(a.pop());this.decl(a,l)}else this.unknownWord(a)}parse(){let e;for(;!this.tokenizer.endOfFile();)switch(e=this.tokenizer.nextToken(),e[0]){case"space":this.spaces+=e[1];break;case";":this.freeSemicolon(e);break;case"}":this.end(e);break;case"comment":this.comment(e);break;case"at-word":this.atrule(e);break;case"{":this.emptyRule(e);break;default:this.other(e);break}this.endFile()}precheckMissedSemicolon(){}raw(e,t,r,i){let n,o,l=r.length,a="",u=!0,c,h;for(let p=0;p<l;p+=1)n=r[p],o=n[0],o==="space"&&p===l-1&&!i?u=!1:o==="comment"?(h=r[p-1]?r[p-1][0]:"empty",c=r[p+1]?r[p+1][0]:"empty",!Zi[h]&&!Zi[c]?a.slice(-1)===","?u=!1:a+=n[1]:u=!1):a+=n[1];if(!u){let p=r.reduce((m,g)=>m+g[1],"");e.raws[t]={raw:p,value:a}}e[t]=a}rule(e){e.pop();let t=new Yi;this.init(t,e[0][2]),t.raws.between=this.spacesAndCommentsFromEnd(e),this.raw(t,"selector",e),this.current=t}spacesAndCommentsFromEnd(e){let t,r="";for(;e.length&&(t=e[e.length-1][0],!(t!=="space"&&t!=="comment"));)r=e.pop()[1]+r;return r}spacesAndCommentsFromStart(e){let t,r="";for(;e.length&&(t=e[0][0],!(t!=="space"&&t!=="comment"));)r+=e.shift()[1];return r}spacesFromEnd(e){let t,r="";for(;e.length&&(t=e[e.length-1][0],t==="space");)r=e.pop()[1]+r;return r}stringFrom(e,t){let r="";for(let i=t;i<e.length;i++)r+=e[i][1];return e.splice(t,e.length-t),r}unclosedBlock(){let e=this.current.source.start;throw this.input.error("Unclosed block",e.line,e.column)}unclosedBracket(e){throw this.input.error("Unclosed bracket",{offset:e[2]},{offset:e[2]+1})}unexpectedClose(e){throw this.input.error("Unexpected }",{offset:e[2]},{offset:e[2]+1})}unknownWord(e){throw this.input.error("Unknown word",{offset:e[0][2]},{offset:e[0][2]+e[0][1].length})}unnamedAtrule(e,t){throw this.input.error("At-rule without name",{offset:t[2]},{offset:t[2]+t[1].length})}};let Bl=xe,zl=Ul,Wl=Et;function zt(s,e){let t=new Wl(s,e),r=new zl(t);try{r.parse()}catch(i){throw process.env.NODE_ENV!=="production"&&i.name==="CssSyntaxError"&&e&&e.from&&(/\.scss$/i.test(e.from)?i.message+=`
-You tried to parse SCSS with the standard CSS parser; try again with the postcss-scss parser`:/\.sass/i.test(e.from)?i.message+=`
-You tried to parse Sass with the standard CSS parser; try again with the postcss-sass parser`:/\.less$/i.test(e.from)&&(i.message+=`
-You tried to parse Less with the standard CSS parser; try again with the postcss-less parser`)),i}return r.root}var Xr=zt;zt.default=zt,Bl.registerParse(zt);let{isClean:he,my:Gl}=Je,Vl=xi,jl=Ct,Hl=xe,Yl=Vr,Zl=Li,Ji=Yr,Jl=Xr,Kl=Qe;const Xl={atrule:"AtRule",comment:"Comment",decl:"Declaration",document:"Document",root:"Root",rule:"Rule"},Ql={AtRule:!0,AtRuleExit:!0,Comment:!0,CommentExit:!0,Declaration:!0,DeclarationExit:!0,Document:!0,DocumentExit:!0,Once:!0,OnceExit:!0,postcssPlugin:!0,prepare:!0,Root:!0,RootExit:!0,Rule:!0,RuleExit:!0},ql={Once:!0,postcssPlugin:!0,prepare:!0},Le=0;function et(s){return typeof s=="object"&&typeof s.then=="function"}function Ki(s){let e=!1,t=Xl[s.type];return s.type==="decl"?e=s.prop.toLowerCase():s.type==="atrule"&&(e=s.name.toLowerCase()),e&&s.append?[t,t+"-"+e,Le,t+"Exit",t+"Exit-"+e]:e?[t,t+"-"+e,t+"Exit",t+"Exit-"+e]:s.append?[t,Le,t+"Exit"]:[t,t+"Exit"]}function Xi(s){let e;return s.type==="document"?e=["Document",Le,"DocumentExit"]:s.type==="root"?e=["Root",Le,"RootExit"]:e=Ki(s),{eventIndex:0,events:e,iterator:0,node:s,visitorIndex:0,visitors:[]}}function Qr(s){return s[he]=!1,s.nodes&&s.nodes.forEach(e=>Qr(e)),s}let qr={},_e=class Yo{constructor(e,t,r){this.stringified=!1,this.processed=!1;let i;if(typeof t=="object"&&t!==null&&(t.type==="root"||t.type==="document"))i=Qr(t);else if(t instanceof Yo||t instanceof Ji)i=Qr(t.root),t.map&&(typeof r.map>"u"&&(r.map={}),r.map.inline||(r.map.inline=!1),r.map.prev=t.map);else{let n=Jl;r.syntax&&(n=r.syntax.parse),r.parser&&(n=r.parser),n.parse&&(n=n.parse);try{i=n(t,r)}catch(o){this.processed=!0,this.error=o}i&&!i[Gl]&&Hl.rebuild(i)}this.result=new Ji(e,i,r),this.helpers={...qr,postcss:qr,result:this.result},this.plugins=this.processor.plugins.map(n=>typeof n=="object"&&n.prepare?{...n,...n.prepare(this.result)}:n)}async(){return this.error?Promise.reject(this.error):this.processed?Promise.resolve(this.result):(this.processing||(this.processing=this.runAsync()),this.processing)}catch(e){return this.async().catch(e)}finally(e){return this.async().then(e,e)}getAsyncError(){throw new Error("Use process(css).then(cb) to work with async plugins")}handleError(e,t){let r=this.result.lastPlugin;try{if(t&&t.addToError(e),this.error=e,e.name==="CssSyntaxError"&&!e.plugin)e.plugin=r.postcssPlugin,e.setMessage();else if(r.postcssVersion&&process.env.NODE_ENV!=="production"){let i=r.postcssPlugin,n=r.postcssVersion,o=this.result.processor.version,l=n.split("."),a=o.split(".");(l[0]!==a[0]||parseInt(l[1])>parseInt(a[1]))&&console.error("Unknown error from PostCSS plugin. Your current PostCSS version is "+o+", but "+i+" uses "+n+". Perhaps this is the source of the error below.")}}catch(i){console&&console.error&&console.error(i)}return e}prepareVisitors(){this.listeners={};let e=(t,r,i)=>{this.listeners[r]||(this.listeners[r]=[]),this.listeners[r].push([t,i])};for(let t of this.plugins)if(typeof t=="object")for(let r in t){if(!Ql[r]&&/^[A-Z]/.test(r))throw new Error(`Unknown event ${r} in ${t.postcssPlugin}. Try to update PostCSS (${this.processor.version} now).`);if(!ql[r])if(typeof t[r]=="object")for(let i in t[r])i==="*"?e(t,r,t[r][i]):e(t,r+"-"+i.toLowerCase(),t[r][i]);else typeof t[r]=="function"&&e(t,r,t[r])}this.hasListener=Object.keys(this.listeners).length>0}async runAsync(){this.plugin=0;for(let e=0;e<this.plugins.length;e++){let t=this.plugins[e],r=this.runOnRoot(t);if(et(r))try{await r}catch(i){throw this.handleError(i)}}if(this.prepareVisitors(),this.hasListener){let e=this.result.root;for(;!e[he];){e[he]=!0;let t=[Xi(e)];for(;t.length>0;){let r=this.visitTick(t);if(et(r))try{await r}catch(i){let n=t[t.length-1].node;throw this.handleError(i,n)}}}if(this.listeners.OnceExit)for(let[t,r]of this.listeners.OnceExit){this.result.lastPlugin=t;try{if(e.type==="document"){let i=e.nodes.map(n=>r(n,this.helpers));await Promise.all(i)}else await r(e,this.helpers)}catch(i){throw this.handleError(i)}}}return this.processed=!0,this.stringify()}runOnRoot(e){this.result.lastPlugin=e;try{if(typeof e=="object"&&e.Once){if(this.result.root.type==="document"){let t=this.result.root.nodes.map(r=>e.Once(r,this.helpers));return et(t[0])?Promise.all(t):t}return e.Once(this.result.root,this.helpers)}else if(typeof e=="function")return e(this.result.root,this.result)}catch(t){throw this.handleError(t)}}stringify(){if(this.error)throw this.error;if(this.stringified)return this.result;this.stringified=!0,this.sync();let e=this.result.opts,t=jl;e.syntax&&(t=e.syntax.stringify),e.stringifier&&(t=e.stringifier),t.stringify&&(t=t.stringify);let i=new Vl(t,this.result.root,this.result.opts).generate();return this.result.css=i[0],this.result.map=i[1],this.result}sync(){if(this.error)throw this.error;if(this.processed)return this.result;if(this.processed=!0,this.processing)throw this.getAsyncError();for(let e of this.plugins){let t=this.runOnRoot(e);if(et(t))throw this.getAsyncError()}if(this.prepareVisitors(),this.hasListener){let e=this.result.root;for(;!e[he];)e[he]=!0,this.walkSync(e);if(this.listeners.OnceExit)if(e.type==="document")for(let t of e.nodes)this.visitSync(this.listeners.OnceExit,t);else this.visitSync(this.listeners.OnceExit,e)}return this.result}then(e,t){return process.env.NODE_ENV!=="production"&&("from"in this.opts||Zl("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(e,t)}toString(){return this.css}visitSync(e,t){for(let[r,i]of e){this.result.lastPlugin=r;let n;try{n=i(t,this.helpers)}catch(o){throw this.handleError(o,t.proxyOf)}if(t.type!=="root"&&t.type!=="document"&&!t.parent)return!0;if(et(n))throw this.getAsyncError()}}visitTick(e){let t=e[e.length-1],{node:r,visitors:i}=t;if(r.type!=="root"&&r.type!=="document"&&!r.parent){e.pop();return}if(i.length>0&&t.visitorIndex<i.length){let[o,l]=i[t.visitorIndex];t.visitorIndex+=1,t.visitorIndex===i.length&&(t.visitors=[],t.visitorIndex=0),this.result.lastPlugin=o;try{return l(r.toProxy(),this.helpers)}catch(a){throw this.handleError(a,r)}}if(t.iterator!==0){let o=t.iterator,l;for(;l=r.nodes[r.indexes[o]];)if(r.indexes[o]+=1,!l[he]){l[he]=!0,e.push(Xi(l));return}t.iterator=0,delete r.indexes[o]}let n=t.events;for(;t.eventIndex<n.length;){let o=n[t.eventIndex];if(t.eventIndex+=1,o===Le){r.nodes&&r.nodes.length&&(r[he]=!0,t.iterator=r.getIterator());return}else if(this.listeners[o]){t.visitors=this.listeners[o];return}}e.pop()}walkSync(e){e[he]=!0;let t=Ki(e);for(let r of t)if(r===Le)e.nodes&&e.each(i=>{i[he]||this.walkSync(i)});else{let i=this.listeners[r];if(i&&this.visitSync(i,e.toProxy()))return}}warnings(){return this.sync().warnings()}get content(){return this.stringify().content}get css(){return this.stringify().css}get map(){return this.stringify().map}get messages(){return this.sync().messages}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){return this.sync().root}get[Symbol.toStringTag](){return"LazyResult"}};_e.registerPostcss=s=>{qr=s};var Qi=_e;_e.default=_e,Kl.registerLazyResult(_e),Yl.registerLazyResult(_e);let eu=xi,tu=Ct,ru=Li,su=Xr;const iu=Yr;let es=class{constructor(e,t,r){t=t.toString(),this.stringified=!1,this._processor=e,this._css=t,this._opts=r,this._map=void 0;let i,n=tu;this.result=new iu(this._processor,i,this._opts),this.result.css=t;let o=this;Object.defineProperty(this.result,"root",{get(){return o.root}});let l=new eu(n,i,this._opts,t);if(l.isMap()){let[a,u]=l.generate();a&&(this.result.css=a),u&&(this.result.map=u)}else l.clearAnnotation(),this.result.css=l.css}async(){return this.error?Promise.reject(this.error):Promise.resolve(this.result)}catch(e){return this.async().catch(e)}finally(e){return this.async().then(e,e)}sync(){if(this.error)throw this.error;return this.result}then(e,t){return process.env.NODE_ENV!=="production"&&("from"in this._opts||ru("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(e,t)}toString(){return this._css}warnings(){return[]}get content(){return this.result.css}get css(){return this.result.css}get map(){return this.result.map}get messages(){return[]}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){if(this._root)return this._root;let e,t=su;try{e=t(this._css,this._opts)}catch(r){this.error=r}if(this.error)throw this.error;return this._root=e,e}get[Symbol.toStringTag](){return"NoWorkResult"}};var nu=es;es.default=es;let ou=nu,au=Qi,lu=Vr,uu=Qe,tt=class{constructor(e=[]){this.version="8.4.38",this.plugins=this.normalize(e)}normalize(e){let t=[];for(let r of e)if(r.postcss===!0?r=r():r.postcss&&(r=r.postcss),typeof r=="object"&&Array.isArray(r.plugins))t=t.concat(r.plugins);else if(typeof r=="object"&&r.postcssPlugin)t.push(r);else if(typeof r=="function")t.push(r);else if(typeof r=="object"&&(r.parse||r.stringify)){if(process.env.NODE_ENV!=="production")throw new Error("PostCSS syntaxes cannot be used as plugins. Instead, please use one of the syntax/parser/stringifier options as outlined in your PostCSS runner documentation.")}else throw new Error(r+" is not a PostCSS plugin");return t}process(e,t={}){return!this.plugins.length&&!t.parser&&!t.stringifier&&!t.syntax?new ou(this,e,t):new au(this,e,t)}use(e){return this.plugins=this.plugins.concat(this.normalize([e])),this}};var cu=tt;tt.default=tt,uu.registerProcessor(tt),lu.registerProcessor(tt);let hu=At,fu=mi,du=Nt,pu=Jr,mu=Et,gu=Qe,yu=Kr;function rt(s,e){if(Array.isArray(s))return s.map(i=>rt(i));let{inputs:t,...r}=s;if(t){e=[];for(let i of t){let n={...i,__proto__:mu.prototype};n.map&&(n.map={...n.map,__proto__:fu.prototype}),e.push(n)}}if(r.nodes&&(r.nodes=s.nodes.map(i=>rt(i,e))),r.source){let{inputId:i,...n}=r.source;r.source=n,i!=null&&(r.source.input=e[i])}if(r.type==="root")return new gu(r);if(r.type==="decl")return new hu(r);if(r.type==="rule")return new yu(r);if(r.type==="comment")return new du(r);if(r.type==="atrule")return new pu(r);throw new Error("Unknown node type: "+s.type)}var wu=rt;rt.default=rt;let Su=Mr,qi=At,bu=Qi,vu=xe,ts=cu,Cu=Ct,Iu=wu,en=Vr,xu=_i,tn=Nt,rn=Jr,Au=Yr,Ru=Et,Ou=Xr,Eu=ji,sn=Kr,nn=Qe,Mu=xt;function M(...s){return s.length===1&&Array.isArray(s[0])&&(s=s[0]),new ts(s)}M.plugin=function(e,t){let r=!1;function i(...o){console&&console.warn&&!r&&(r=!0,console.warn(e+`: postcss.plugin was deprecated. Migration guide:
-https://evilmartians.com/chronicles/postcss-8-plugin-migration`),process.env.LANG&&process.env.LANG.startsWith("cn")&&console.warn(e+`: 里面 postcss.plugin 被弃用. 迁移指南:
-https://www.w3ctech.com/topic/2226`));let l=t(...o);return l.postcssPlugin=e,l.postcssVersion=new ts().version,l}let n;return Object.defineProperty(i,"postcss",{get(){return n||(n=i()),n}}),i.process=function(o,l,a){return M([i(a)]).process(o,l)},i},M.stringify=Cu,M.parse=Ou,M.fromJSON=Iu,M.list=Eu,M.comment=s=>new tn(s),M.atRule=s=>new rn(s),M.decl=s=>new qi(s),M.rule=s=>new sn(s),M.root=s=>new nn(s),M.document=s=>new en(s),M.CssSyntaxError=Su,M.Declaration=qi,M.Container=vu,M.Processor=ts,M.Document=en,M.Comment=tn,M.Warning=xu,M.AtRule=rn,M.Result=Au,M.Input=Ru,M.Rule=sn,M.Root=nn,M.Node=Mu,bu.registerPostcss(M);var $u=M;M.default=M;const F=Ha($u);F.stringify,F.fromJSON,F.plugin,F.parse,F.list,F.document,F.comment,F.atRule,F.rule,F.decl,F.root,F.CssSyntaxError,F.Declaration,F.Container,F.Processor,F.Document,F.Comment,F.Warning,F.AtRule,F.Result,F.Input,F.Rule,F.Root,F.Node;var Nu=Object.defineProperty,ku=(s,e,t)=>e in s?Nu(s,e,{enumerable:!0,configurable:!0,writable:!0,value:t}):s[e]=t,re=(s,e,t)=>ku(s,typeof e!="symbol"?e+"":e,t);function Tu(s){return s&&s.__esModule&&Object.prototype.hasOwnProperty.call(s,"default")?s.default:s}function Pu(s){if(s.__esModule)return s;var e=s.default;if(typeof e=="function"){var t=function r(){return this instanceof r?Reflect.construct(e,arguments,this.constructor):e.apply(this,arguments)};t.prototype=e.prototype}else t={};return Object.defineProperty(t,"__esModule",{value:!0}),Object.keys(s).forEach(function(r){var i=Object.getOwnPropertyDescriptor(s,r);Object.defineProperty(t,r,i.get?i:{enumerable:!0,get:function(){return s[r]}})}),t}var rs={exports:{}},_=String,on=function(){return{isColorSupported:!1,reset:_,bold:_,dim:_,italic:_,underline:_,inverse:_,hidden:_,strikethrough:_,black:_,red:_,green:_,yellow:_,blue:_,magenta:_,cyan:_,white:_,gray:_,bgBlack:_,bgRed:_,bgGreen:_,bgYellow:_,bgBlue:_,bgMagenta:_,bgCyan:_,bgWhite:_}};rs.exports=on(),rs.exports.createColors=on;var Du=rs.exports;const ae=Pu(Object.freeze(Object.defineProperty({__proto__:null,default:{}},Symbol.toStringTag,{value:"Module"})));let an=Du,ln=ae,ss=class Zo extends Error{constructor(e,t,r,i,n,o){super(e),this.name="CssSyntaxError",this.reason=e,n&&(this.file=n),i&&(this.source=i),o&&(this.plugin=o),typeof t<"u"&&typeof r<"u"&&(typeof t=="number"?(this.line=t,this.column=r):(this.line=t.line,this.column=t.column,this.endLine=r.line,this.endColumn=r.column)),this.setMessage(),Error.captureStackTrace&&Error.captureStackTrace(this,Zo)}setMessage(){this.message=this.plugin?this.plugin+": ":"",this.message+=this.file?this.file:"<css input>",typeof this.line<"u"&&(this.message+=":"+this.line+":"+this.column),this.message+=": "+this.reason}showSourceCode(e){if(!this.source)return"";let t=this.source;e==null&&(e=an.isColorSupported),ln&&e&&(t=ln(t));let r=t.split(/\r?\n/),i=Math.max(this.line-3,0),n=Math.min(this.line+2,r.length),o=String(n).length,l,a;if(e){let{bold:u,gray:c,red:h}=an.createColors(!0);l=p=>u(h(p)),a=p=>c(p)}else l=a=u=>u;return r.slice(i,n).map((u,c)=>{let h=i+1+c,p=" "+(" "+h).slice(-o)+" | ";if(h===this.line){let m=a(p.replace(/\d/g," "))+u.slice(0,this.column-1).replace(/[^\t]/g," ");return l(">")+a(p)+u+`
- `+m+l("^")}return" "+a(p)+u}).join(`
-`)}toString(){let e=this.showSourceCode();return e&&(e=`
+`)&&(n=`\r
+`),this.css+=n+"/*# sourceMappingURL="+a+" */"}applyPrevMaps(){for(let a of this.previous()){let n=this.toUrl(this.path(a.file)),m=a.root||r(a.file),h;this.mapOpts.sourcesContent===!1?(h=new e(a.text),h.sourcesContent&&(h.sourcesContent=null)):h=a.consumer(),this.map.applySourceMap(h,n,this.toUrl(this.path(m)))}}clearAnnotation(){if(this.mapOpts.annotation!==!1)if(this.root){let a;for(let n=this.root.nodes.length-1;n>=0;n--)a=this.root.nodes[n],a.type==="comment"&&a.text.indexOf("# sourceMappingURL=")===0&&this.root.removeChild(n)}else this.css&&(this.css=this.css.replace(/\n*?\/\*#[\S\s]*?\*\/$/gm,""))}generate(){if(this.clearAnnotation(),d&&l&&this.isMap())return this.generateMap();{let a="";return this.stringify(this.root,n=>{a+=n}),[a]}}generateMap(){if(this.root)this.generateString();else if(this.previous().length===1){let a=this.previous()[0].consumer();a.file=this.outputFile(),this.map=t.fromSourceMap(a,{ignoreInvalidMapping:!0})}else this.map=new t({file:this.outputFile(),ignoreInvalidMapping:!0}),this.map.addMapping({generated:{column:0,line:1},original:{column:0,line:1},source:this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>"});return this.isSourcesContent()&&this.setSourcesContent(),this.root&&this.previous().length>0&&this.applyPrevMaps(),this.isAnnotation()&&this.addAnnotation(),this.isInline()?[this.css]:[this.css,this.map]}generateString(){this.css="",this.map=new t({file:this.outputFile(),ignoreInvalidMapping:!0});let a=1,n=1,m="<no source>",h={generated:{column:0,line:0},original:{column:0,line:0},source:""},g,v;this.stringify(this.root,(b,S,x)=>{if(this.css+=b,S&&x!=="end"&&(h.generated.line=a,h.generated.column=n-1,S.source&&S.source.start?(h.source=this.sourcePath(S),h.original.line=S.source.start.line,h.original.column=S.source.start.column-1,this.map.addMapping(h)):(h.source=m,h.original.line=1,h.original.column=0,this.map.addMapping(h))),g=b.match(/\n/g),g?(a+=g.length,v=b.lastIndexOf(`
+`),n=b.length-v):n+=b.length,S&&x!=="start"){let w=S.parent||{raws:{}};(!(S.type==="decl"||S.type==="atrule"&&!S.nodes)||S!==w.last||w.raws.semicolon)&&(S.source&&S.source.end?(h.source=this.sourcePath(S),h.original.line=S.source.end.line,h.original.column=S.source.end.column-1,h.generated.line=a,h.generated.column=n-2,this.map.addMapping(h)):(h.source=m,h.original.line=1,h.original.column=0,h.generated.line=a,h.generated.column=n-1,this.map.addMapping(h)))}})}isAnnotation(){return this.isInline()?!0:typeof this.mapOpts.annotation<"u"?this.mapOpts.annotation:this.previous().length?this.previous().some(a=>a.annotation):!0}isInline(){if(typeof this.mapOpts.inline<"u")return this.mapOpts.inline;let a=this.mapOpts.annotation;return typeof a<"u"&&a!==!0?!1:this.previous().length?this.previous().some(n=>n.inline):!0}isMap(){return typeof this.opts.map<"u"?!!this.opts.map:this.previous().length>0}isSourcesContent(){return typeof this.mapOpts.sourcesContent<"u"?this.mapOpts.sourcesContent:this.previous().length?this.previous().some(a=>a.withContent()):!0}outputFile(){return this.opts.to?this.path(this.opts.to):this.opts.from?this.path(this.opts.from):"to.css"}path(a){if(this.mapOpts.absolute||a.charCodeAt(0)===60||/^\w+:\/\//.test(a))return a;let n=this.memoizedPaths.get(a);if(n)return n;let m=this.opts.to?r(this.opts.to):".";typeof this.mapOpts.annotation=="string"&&(m=r(s(m,this.mapOpts.annotation)));let h=o(m,a);return this.memoizedPaths.set(a,h),h}previous(){if(!this.previousMaps)if(this.previousMaps=[],this.root)this.root.walk(a=>{if(a.source&&a.source.input.map){let n=a.source.input.map;this.previousMaps.includes(n)||this.previousMaps.push(n)}});else{let a=new p(this.originalCSS,this.opts);a.map&&this.previousMaps.push(a.map)}return this.previousMaps}setSourcesContent(){let a={};if(this.root)this.root.walk(n=>{if(n.source){let m=n.source.input.from;if(m&&!a[m]){a[m]=!0;let h=this.usesFileUrls?this.toFileUrl(m):this.toUrl(this.path(m));this.map.setSourceContent(h,n.source.input.css)}}});else if(this.css){let n=this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>";this.map.setSourceContent(n,this.css)}}sourcePath(a){return this.mapOpts.from?this.toUrl(this.mapOpts.from):this.usesFileUrls?this.toFileUrl(a.source.input.from):this.toUrl(this.path(a.source.input.from))}toBase64(a){return Buffer?Buffer.from(a).toString("base64"):window.btoa(unescape(encodeURIComponent(a)))}toFileUrl(a){let n=this.memoizedFileURLs.get(a);if(n)return n;if(c){let m=c(a).toString();return this.memoizedFileURLs.set(a,m),m}else throw new Error("`map.absolute` option is not available in this PostCSS build")}toUrl(a){let n=this.memoizedURLs.get(a);if(n)return n;u==="\\"&&(a=a.replace(/\\/g,"/"));let m=encodeURI(a).replace(/[#?]/g,encodeURIComponent);return this.memoizedURLs.set(a,m),m}}return jt=i,jt}var Gt,zs;function ct(){if(zs)return Gt;zs=1;let e=at();class t extends e{constructor(o){super(o),this.type="comment"}}return Gt=t,t.default=t,Gt}var Vt,Ws;function Ie(){if(Ws)return Vt;Ws=1;let{isClean:e,my:t}=_t(),r=lt(),o=ct(),s=at(),u,c,p,l;function d(a){return a.map(n=>(n.nodes&&(n.nodes=d(n.nodes)),delete n.source,n))}function i(a){if(a[e]=!1,a.proxyOf.nodes)for(let n of a.proxyOf.nodes)i(n)}class f extends s{append(...n){for(let m of n){let h=this.normalize(m,this.last);for(let g of h)this.proxyOf.nodes.push(g)}return this.markDirty(),this}cleanRaws(n){if(super.cleanRaws(n),this.nodes)for(let m of this.nodes)m.cleanRaws(n)}each(n){if(!this.proxyOf.nodes)return;let m=this.getIterator(),h,g;for(;this.indexes[m]<this.proxyOf.nodes.length&&(h=this.indexes[m],g=n(this.proxyOf.nodes[h],h),g!==!1);)this.indexes[m]+=1;return delete this.indexes[m],g}every(n){return this.nodes.every(n)}getIterator(){this.lastEach||(this.lastEach=0),this.indexes||(this.indexes={}),this.lastEach+=1;let n=this.lastEach;return this.indexes[n]=0,n}getProxyProcessor(){return{get(n,m){return m==="proxyOf"?n:n[m]?m==="each"||typeof m=="string"&&m.startsWith("walk")?(...h)=>n[m](...h.map(g=>typeof g=="function"?(v,b)=>g(v.toProxy(),b):g)):m==="every"||m==="some"?h=>n[m]((g,...v)=>h(g.toProxy(),...v)):m==="root"?()=>n.root().toProxy():m==="nodes"?n.nodes.map(h=>h.toProxy()):m==="first"||m==="last"?n[m].toProxy():n[m]:n[m]},set(n,m,h){return n[m]===h||(n[m]=h,(m==="name"||m==="params"||m==="selector")&&n.markDirty()),!0}}}index(n){return typeof n=="number"?n:(n.proxyOf&&(n=n.proxyOf),this.proxyOf.nodes.indexOf(n))}insertAfter(n,m){let h=this.index(n),g=this.normalize(m,this.proxyOf.nodes[h]).reverse();h=this.index(n);for(let b of g)this.proxyOf.nodes.splice(h+1,0,b);let v;for(let b in this.indexes)v=this.indexes[b],h<v&&(this.indexes[b]=v+g.length);return this.markDirty(),this}insertBefore(n,m){let h=this.index(n),g=h===0?"prepend":!1,v=this.normalize(m,this.proxyOf.nodes[h],g).reverse();h=this.index(n);for(let S of v)this.proxyOf.nodes.splice(h,0,S);let b;for(let S in this.indexes)b=this.indexes[S],h<=b&&(this.indexes[S]=b+v.length);return this.markDirty(),this}normalize(n,m){if(typeof n=="string")n=d(u(n).nodes);else if(typeof n>"u")n=[];else if(Array.isArray(n)){n=n.slice(0);for(let g of n)g.parent&&g.parent.removeChild(g,"ignore")}else if(n.type==="root"&&this.type!=="document"){n=n.nodes.slice(0);for(let g of n)g.parent&&g.parent.removeChild(g,"ignore")}else if(n.type)n=[n];else if(n.prop){if(typeof n.value>"u")throw new Error("Value field is missed in node creation");typeof n.value!="string"&&(n.value=String(n.value)),n=[new r(n)]}else if(n.selector)n=[new c(n)];else if(n.name)n=[new p(n)];else if(n.text)n=[new o(n)];else throw new Error("Unknown node type in node creation");return n.map(g=>(g[t]||f.rebuild(g),g=g.proxyOf,g.parent&&g.parent.removeChild(g),g[e]&&i(g),typeof g.raws.before>"u"&&m&&typeof m.raws.before<"u"&&(g.raws.before=m.raws.before.replace(/\S/g,"")),g.parent=this.proxyOf,g))}prepend(...n){n=n.reverse();for(let m of n){let h=this.normalize(m,this.first,"prepend").reverse();for(let g of h)this.proxyOf.nodes.unshift(g);for(let g in this.indexes)this.indexes[g]=this.indexes[g]+h.length}return this.markDirty(),this}push(n){return n.parent=this,this.proxyOf.nodes.push(n),this}removeAll(){for(let n of this.proxyOf.nodes)n.parent=void 0;return this.proxyOf.nodes=[],this.markDirty(),this}removeChild(n){n=this.index(n),this.proxyOf.nodes[n].parent=void 0,this.proxyOf.nodes.splice(n,1);let m;for(let h in this.indexes)m=this.indexes[h],m>=n&&(this.indexes[h]=m-1);return this.markDirty(),this}replaceValues(n,m,h){return h||(h=m,m={}),this.walkDecls(g=>{m.props&&!m.props.includes(g.prop)||m.fast&&!g.value.includes(m.fast)||(g.value=g.value.replace(n,h))}),this.markDirty(),this}some(n){return this.nodes.some(n)}walk(n){return this.each((m,h)=>{let g;try{g=n(m,h)}catch(v){throw m.addToError(v)}return g!==!1&&m.walk&&(g=m.walk(n)),g})}walkAtRules(n,m){return m?n instanceof RegExp?this.walk((h,g)=>{if(h.type==="atrule"&&n.test(h.name))return m(h,g)}):this.walk((h,g)=>{if(h.type==="atrule"&&h.name===n)return m(h,g)}):(m=n,this.walk((h,g)=>{if(h.type==="atrule")return m(h,g)}))}walkComments(n){return this.walk((m,h)=>{if(m.type==="comment")return n(m,h)})}walkDecls(n,m){return m?n instanceof RegExp?this.walk((h,g)=>{if(h.type==="decl"&&n.test(h.prop))return m(h,g)}):this.walk((h,g)=>{if(h.type==="decl"&&h.prop===n)return m(h,g)}):(m=n,this.walk((h,g)=>{if(h.type==="decl")return m(h,g)}))}walkRules(n,m){return m?n instanceof RegExp?this.walk((h,g)=>{if(h.type==="rule"&&n.test(h.selector))return m(h,g)}):this.walk((h,g)=>{if(h.type==="rule"&&h.selector===n)return m(h,g)}):(m=n,this.walk((h,g)=>{if(h.type==="rule")return m(h,g)}))}get first(){if(this.proxyOf.nodes)return this.proxyOf.nodes[0]}get last(){if(this.proxyOf.nodes)return this.proxyOf.nodes[this.proxyOf.nodes.length-1]}}return f.registerParse=a=>{u=a},f.registerRule=a=>{c=a},f.registerAtRule=a=>{p=a},f.registerRoot=a=>{l=a},Vt=f,f.default=f,f.rebuild=a=>{a.type==="atrule"?Object.setPrototypeOf(a,p.prototype):a.type==="rule"?Object.setPrototypeOf(a,c.prototype):a.type==="decl"?Object.setPrototypeOf(a,r.prototype):a.type==="comment"?Object.setPrototypeOf(a,o.prototype):a.type==="root"&&Object.setPrototypeOf(a,l.prototype),a[t]=!0,a.nodes&&a.nodes.forEach(n=>{f.rebuild(n)})},Vt}var Ht,qs;function Jt(){if(qs)return Ht;qs=1;let e=Ie(),t,r;class o extends e{constructor(u){super({type:"document",...u}),this.nodes||(this.nodes=[])}toResult(u={}){return new t(new r,this,u).stringify()}}return o.registerLazyResult=s=>{t=s},o.registerProcessor=s=>{r=s},Ht=o,o.default=o,Ht}var Yt,js;function Gs(){if(js)return Yt;js=1;let e={};return Yt=function(r){e[r]||(e[r]=!0,typeof console<"u"&&console.warn&&console.warn(r))},Yt}var Qt,Vs;function Hs(){if(Vs)return Qt;Vs=1;class e{constructor(r,o={}){if(this.type="warning",this.text=r,o.node&&o.node.source){let s=o.node.rangeBy(o);this.line=s.start.line,this.column=s.start.column,this.endLine=s.end.line,this.endColumn=s.end.column}for(let s in o)this[s]=o[s]}toString(){return this.node?this.node.error(this.text,{index:this.index,plugin:this.plugin,word:this.word}).message:this.plugin?this.plugin+": "+this.text:this.text}}return Qt=e,e.default=e,Qt}var Zt,Js;function Kt(){if(Js)return Zt;Js=1;let e=Hs();class t{constructor(o,s,u){this.processor=o,this.messages=[],this.root=s,this.opts=u,this.css=void 0,this.map=void 0}toString(){return this.css}warn(o,s={}){s.plugin||this.lastPlugin&&this.lastPlugin.postcssPlugin&&(s.plugin=this.lastPlugin.postcssPlugin);let u=new e(o,s);return this.messages.push(u),u}warnings(){return this.messages.filter(o=>o.type==="warning")}get content(){return this.css}}return Zt=t,t.default=t,Zt}var Xt,Ys;function To(){if(Ys)return Xt;Ys=1;const e=39,t=34,r=92,o=47,s=10,u=32,c=12,p=9,l=13,d=91,i=93,f=40,a=41,n=123,m=125,h=59,g=42,v=58,b=64,S=/[\t\n\f\r "#'()/;[\\\]{}]/g,x=/[\t\n\f\r !"#'():;@[\\\]{}]|\/(?=\*)/g,w=/.[\r\n"'(/\\]/,y=/[\da-f]/i;return Xt=function(O,I={}){let M=O.css.valueOf(),P=I.ignoreErrors,N,R,ye,ue,B,U,Y,te,Q,z,Me=M.length,E=0,ve=[],de=[];function Qe(){return E}function ie(T){throw O.error("Unclosed "+T,E)}function we(){return de.length===0&&E>=Me}function Te(T){if(de.length)return de.pop();if(E>=Me)return;let K=T?T.ignoreUnclosed:!1;switch(N=M.charCodeAt(E),N){case s:case u:case p:case l:case c:{R=E;do R+=1,N=M.charCodeAt(R);while(N===u||N===s||N===p||N===l||N===c);z=["space",M.slice(E,R)],E=R-1;break}case d:case i:case n:case m:case v:case h:case a:{let W=String.fromCharCode(N);z=[W,W,E];break}case f:{if(te=ve.length?ve.pop()[1]:"",Q=M.charCodeAt(E+1),te==="url"&&Q!==e&&Q!==t&&Q!==u&&Q!==s&&Q!==p&&Q!==c&&Q!==l){R=E;do{if(U=!1,R=M.indexOf(")",R+1),R===-1)if(P||K){R=E;break}else ie("bracket");for(Y=R;M.charCodeAt(Y-1)===r;)Y-=1,U=!U}while(U);z=["brackets",M.slice(E,R+1),E,R],E=R}else R=M.indexOf(")",E+1),ue=M.slice(E,R+1),R===-1||w.test(ue)?z=["(","(",E]:(z=["brackets",ue,E,R],E=R);break}case e:case t:{ye=N===e?"'":'"',R=E;do{if(U=!1,R=M.indexOf(ye,R+1),R===-1)if(P||K){R=E+1;break}else ie("string");for(Y=R;M.charCodeAt(Y-1)===r;)Y-=1,U=!U}while(U);z=["string",M.slice(E,R+1),E,R],E=R;break}case b:{S.lastIndex=E+1,S.test(M),S.lastIndex===0?R=M.length-1:R=S.lastIndex-2,z=["at-word",M.slice(E,R+1),E,R],E=R;break}case r:{for(R=E,B=!0;M.charCodeAt(R+1)===r;)R+=1,B=!B;if(N=M.charCodeAt(R+1),B&&N!==o&&N!==u&&N!==s&&N!==p&&N!==l&&N!==c&&(R+=1,y.test(M.charAt(R)))){for(;y.test(M.charAt(R+1));)R+=1;M.charCodeAt(R+1)===u&&(R+=1)}z=["word",M.slice(E,R+1),E,R],E=R;break}default:{N===o&&M.charCodeAt(E+1)===g?(R=M.indexOf("*/",E+2)+1,R===0&&(P||K?R=M.length:ie("comment")),z=["comment",M.slice(E,R+1),E,R],E=R):(x.lastIndex=E+1,x.test(M),x.lastIndex===0?R=M.length-1:R=x.lastIndex-2,z=["word",M.slice(E,R+1),E,R],ve.push(z),E=R);break}}return E++,z}function Ee(T){de.push(T)}return{back:Ee,endOfFile:we,nextToken:Te,position:Qe}},Xt}var er,Qs;function tr(){if(Qs)return er;Qs=1;let e=Ie();class t extends e{constructor(o){super(o),this.type="atrule"}append(...o){return this.proxyOf.nodes||(this.nodes=[]),super.append(...o)}prepend(...o){return this.proxyOf.nodes||(this.nodes=[]),super.prepend(...o)}}return er=t,t.default=t,e.registerAtRule(t),er}var rr,Zs;function je(){if(Zs)return rr;Zs=1;let e=Ie(),t,r;class o extends e{constructor(u){super(u),this.type="root",this.nodes||(this.nodes=[])}normalize(u,c,p){let l=super.normalize(u);if(c){if(p==="prepend")this.nodes.length>1?c.raws.before=this.nodes[1].raws.before:delete c.raws.before;else if(this.first!==c)for(let d of l)d.raws.before=c.raws.before}return l}removeChild(u,c){let p=this.index(u);return!c&&p===0&&this.nodes.length>1&&(this.nodes[1].raws.before=this.nodes[p].raws.before),super.removeChild(u)}toResult(u={}){return new t(new r,this,u).stringify()}}return o.registerLazyResult=s=>{t=s},o.registerProcessor=s=>{r=s},rr=o,o.default=o,e.registerRoot(o),rr}var sr,Ks;function Xs(){if(Ks)return sr;Ks=1;let e={comma(t){return e.split(t,[","],!0)},space(t){let r=[" ",`
+`," "];return e.split(t,r)},split(t,r,o){let s=[],u="",c=!1,p=0,l=!1,d="",i=!1;for(let f of t)i?i=!1:f==="\\"?i=!0:l?f===d&&(l=!1):f==='"'||f==="'"?(l=!0,d=f):f==="("?p+=1:f===")"?p>0&&(p-=1):p===0&&r.includes(f)&&(c=!0),c?(u!==""&&s.push(u.trim()),u="",c=!1):u+=f;return(o||u!=="")&&s.push(u.trim()),s}};return sr=e,e.default=e,sr}var ir,ei;function nr(){if(ei)return ir;ei=1;let e=Ie(),t=Xs();class r extends e{constructor(s){super(s),this.type="rule",this.nodes||(this.nodes=[])}get selectors(){return t.comma(this.selector)}set selectors(s){let u=this.selector?this.selector.match(/,\s*/):null,c=u?u[0]:","+this.raw("between","beforeOpen");this.selector=s.join(c)}}return ir=r,r.default=r,e.registerRule(r),ir}var or,ti;function Do(){if(ti)return or;ti=1;let e=lt(),t=To(),r=ct(),o=tr(),s=je(),u=nr();const c={empty:!0,space:!0};function p(d){for(let i=d.length-1;i>=0;i--){let f=d[i],a=f[3]||f[2];if(a)return a}}class l{constructor(i){this.input=i,this.root=new s,this.current=this.root,this.spaces="",this.semicolon=!1,this.createTokenizer(),this.root.source={input:i,start:{column:1,line:1,offset:0}}}atrule(i){let f=new o;f.name=i[1].slice(1),f.name===""&&this.unnamedAtrule(f,i),this.init(f,i[2]);let a,n,m,h=!1,g=!1,v=[],b=[];for(;!this.tokenizer.endOfFile();){if(i=this.tokenizer.nextToken(),a=i[0],a==="("||a==="["?b.push(a==="("?")":"]"):a==="{"&&b.length>0?b.push("}"):a===b[b.length-1]&&b.pop(),b.length===0)if(a===";"){f.source.end=this.getPosition(i[2]),f.source.end.offset++,this.semicolon=!0;break}else if(a==="{"){g=!0;break}else if(a==="}"){if(v.length>0){for(m=v.length-1,n=v[m];n&&n[0]==="space";)n=v[--m];n&&(f.source.end=this.getPosition(n[3]||n[2]),f.source.end.offset++)}this.end(i);break}else v.push(i);else v.push(i);if(this.tokenizer.endOfFile()){h=!0;break}}f.raws.between=this.spacesAndCommentsFromEnd(v),v.length?(f.raws.afterName=this.spacesAndCommentsFromStart(v),this.raw(f,"params",v),h&&(i=v[v.length-1],f.source.end=this.getPosition(i[3]||i[2]),f.source.end.offset++,this.spaces=f.raws.between,f.raws.between="")):(f.raws.afterName="",f.params=""),g&&(f.nodes=[],this.current=f)}checkMissedSemicolon(i){let f=this.colon(i);if(f===!1)return;let a=0,n;for(let m=f-1;m>=0&&(n=i[m],!(n[0]!=="space"&&(a+=1,a===2)));m--);throw this.input.error("Missed semicolon",n[0]==="word"?n[3]+1:n[2])}colon(i){let f=0,a,n,m;for(let[h,g]of i.entries()){if(a=g,n=a[0],n==="("&&(f+=1),n===")"&&(f-=1),f===0&&n===":")if(!m)this.doubleColon(a);else{if(m[0]==="word"&&m[1]==="progid")continue;return h}m=a}return!1}comment(i){let f=new r;this.init(f,i[2]),f.source.end=this.getPosition(i[3]||i[2]),f.source.end.offset++;let a=i[1].slice(2,-2);if(/^\s*$/.test(a))f.text="",f.raws.left=a,f.raws.right="";else{let n=a.match(/^(\s*)([^]*\S)(\s*)$/);f.text=n[2],f.raws.left=n[1],f.raws.right=n[3]}}createTokenizer(){this.tokenizer=t(this.input)}decl(i,f){let a=new e;this.init(a,i[0][2]);let n=i[i.length-1];for(n[0]===";"&&(this.semicolon=!0,i.pop()),a.source.end=this.getPosition(n[3]||n[2]||p(i)),a.source.end.offset++;i[0][0]!=="word";)i.length===1&&this.unknownWord(i),a.raws.before+=i.shift()[1];for(a.source.start=this.getPosition(i[0][2]),a.prop="";i.length;){let b=i[0][0];if(b===":"||b==="space"||b==="comment")break;a.prop+=i.shift()[1]}a.raws.between="";let m;for(;i.length;)if(m=i.shift(),m[0]===":"){a.raws.between+=m[1];break}else m[0]==="word"&&/\w/.test(m[1])&&this.unknownWord([m]),a.raws.between+=m[1];(a.prop[0]==="_"||a.prop[0]==="*")&&(a.raws.before+=a.prop[0],a.prop=a.prop.slice(1));let h=[],g;for(;i.length&&(g=i[0][0],!(g!=="space"&&g!=="comment"));)h.push(i.shift());this.precheckMissedSemicolon(i);for(let b=i.length-1;b>=0;b--){if(m=i[b],m[1].toLowerCase()==="!important"){a.important=!0;let S=this.stringFrom(i,b);S=this.spacesFromEnd(i)+S,S!==" !important"&&(a.raws.important=S);break}else if(m[1].toLowerCase()==="important"){let S=i.slice(0),x="";for(let w=b;w>0;w--){let y=S[w][0];if(x.trim().indexOf("!")===0&&y!=="space")break;x=S.pop()[1]+x}x.trim().indexOf("!")===0&&(a.important=!0,a.raws.important=x,i=S)}if(m[0]!=="space"&&m[0]!=="comment")break}i.some(b=>b[0]!=="space"&&b[0]!=="comment")&&(a.raws.between+=h.map(b=>b[1]).join(""),h=[]),this.raw(a,"value",h.concat(i),f),a.value.includes(":")&&!f&&this.checkMissedSemicolon(i)}doubleColon(i){throw this.input.error("Double colon",{offset:i[2]},{offset:i[2]+i[1].length})}emptyRule(i){let f=new u;this.init(f,i[2]),f.selector="",f.raws.between="",this.current=f}end(i){this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.semicolon=!1,this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.spaces="",this.current.parent?(this.current.source.end=this.getPosition(i[2]),this.current.source.end.offset++,this.current=this.current.parent):this.unexpectedClose(i)}endFile(){this.current.parent&&this.unclosedBlock(),this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.root.source.end=this.getPosition(this.tokenizer.position())}freeSemicolon(i){if(this.spaces+=i[1],this.current.nodes){let f=this.current.nodes[this.current.nodes.length-1];f&&f.type==="rule"&&!f.raws.ownSemicolon&&(f.raws.ownSemicolon=this.spaces,this.spaces="")}}getPosition(i){let f=this.input.fromOffset(i);return{column:f.col,line:f.line,offset:i}}init(i,f){this.current.push(i),i.source={input:this.input,start:this.getPosition(f)},i.raws.before=this.spaces,this.spaces="",i.type!=="comment"&&(this.semicolon=!1)}other(i){let f=!1,a=null,n=!1,m=null,h=[],g=i[1].startsWith("--"),v=[],b=i;for(;b;){if(a=b[0],v.push(b),a==="("||a==="[")m||(m=b),h.push(a==="("?")":"]");else if(g&&n&&a==="{")m||(m=b),h.push("}");else if(h.length===0)if(a===";")if(n){this.decl(v,g);return}else break;else if(a==="{"){this.rule(v);return}else if(a==="}"){this.tokenizer.back(v.pop()),f=!0;break}else a===":"&&(n=!0);else a===h[h.length-1]&&(h.pop(),h.length===0&&(m=null));b=this.tokenizer.nextToken()}if(this.tokenizer.endOfFile()&&(f=!0),h.length>0&&this.unclosedBracket(m),f&&n){if(!g)for(;v.length&&(b=v[v.length-1][0],!(b!=="space"&&b!=="comment"));)this.tokenizer.back(v.pop());this.decl(v,g)}else this.unknownWord(v)}parse(){let i;for(;!this.tokenizer.endOfFile();)switch(i=this.tokenizer.nextToken(),i[0]){case"space":this.spaces+=i[1];break;case";":this.freeSemicolon(i);break;case"}":this.end(i);break;case"comment":this.comment(i);break;case"at-word":this.atrule(i);break;case"{":this.emptyRule(i);break;default:this.other(i);break}this.endFile()}precheckMissedSemicolon(){}raw(i,f,a,n){let m,h,g=a.length,v="",b=!0,S,x;for(let w=0;w<g;w+=1)m=a[w],h=m[0],h==="space"&&w===g-1&&!n?b=!1:h==="comment"?(x=a[w-1]?a[w-1][0]:"empty",S=a[w+1]?a[w+1][0]:"empty",!c[x]&&!c[S]?v.slice(-1)===","?b=!1:v+=m[1]:b=!1):v+=m[1];if(!b){let w=a.reduce((y,C)=>y+C[1],"");i.raws[f]={raw:w,value:v}}i[f]=v}rule(i){i.pop();let f=new u;this.init(f,i[0][2]),f.raws.between=this.spacesAndCommentsFromEnd(i),this.raw(f,"selector",i),this.current=f}spacesAndCommentsFromEnd(i){let f,a="";for(;i.length&&(f=i[i.length-1][0],!(f!=="space"&&f!=="comment"));)a=i.pop()[1]+a;return a}spacesAndCommentsFromStart(i){let f,a="";for(;i.length&&(f=i[0][0],!(f!=="space"&&f!=="comment"));)a+=i.shift()[1];return a}spacesFromEnd(i){let f,a="";for(;i.length&&(f=i[i.length-1][0],f==="space");)a=i.pop()[1]+a;return a}stringFrom(i,f){let a="";for(let n=f;n<i.length;n++)a+=i[n][1];return i.splice(f,i.length-f),a}unclosedBlock(){let i=this.current.source.start;throw this.input.error("Unclosed block",i.line,i.column)}unclosedBracket(i){throw this.input.error("Unclosed bracket",{offset:i[2]},{offset:i[2]+1})}unexpectedClose(i){throw this.input.error("Unexpected }",{offset:i[2]},{offset:i[2]+1})}unknownWord(i){throw this.input.error("Unknown word",{offset:i[0][2]},{offset:i[0][2]+i[0][1].length})}unnamedAtrule(i,f){throw this.input.error("At-rule without name",{offset:f[2]},{offset:f[2]+f[1].length})}}return or=l,or}var ar,ri;function lr(){if(ri)return ar;ri=1;let e=Ie(),t=Do(),r=ut();function o(s,u){let c=new r(s,u),p=new t(c);try{p.parse()}catch(l){throw process.env.NODE_ENV!=="production"&&l.name==="CssSyntaxError"&&u&&u.from&&(/\.scss$/i.test(u.from)?l.message+=`
+You tried to parse SCSS with the standard CSS parser; try again with the postcss-scss parser`:/\.sass/i.test(u.from)?l.message+=`
+You tried to parse Sass with the standard CSS parser; try again with the postcss-sass parser`:/\.less$/i.test(u.from)&&(l.message+=`
+You tried to parse Less with the standard CSS parser; try again with the postcss-less parser`)),l}return p.root}return ar=o,o.default=o,e.registerParse(o),ar}var ur,si;function ii(){if(si)return ur;si=1;let{isClean:e,my:t}=_t(),r=Bs(),o=ot(),s=Ie(),u=Jt(),c=Gs(),p=Kt(),l=lr(),d=je();const i={atrule:"AtRule",comment:"Comment",decl:"Declaration",document:"Document",root:"Root",rule:"Rule"},f={AtRule:!0,AtRuleExit:!0,Comment:!0,CommentExit:!0,Declaration:!0,DeclarationExit:!0,Document:!0,DocumentExit:!0,Once:!0,OnceExit:!0,postcssPlugin:!0,prepare:!0,Root:!0,RootExit:!0,Rule:!0,RuleExit:!0},a={Once:!0,postcssPlugin:!0,prepare:!0},n=0;function m(x){return typeof x=="object"&&typeof x.then=="function"}function h(x){let w=!1,y=i[x.type];return x.type==="decl"?w=x.prop.toLowerCase():x.type==="atrule"&&(w=x.name.toLowerCase()),w&&x.append?[y,y+"-"+w,n,y+"Exit",y+"Exit-"+w]:w?[y,y+"-"+w,y+"Exit",y+"Exit-"+w]:x.append?[y,n,y+"Exit"]:[y,y+"Exit"]}function g(x){let w;return x.type==="document"?w=["Document",n,"DocumentExit"]:x.type==="root"?w=["Root",n,"RootExit"]:w=h(x),{eventIndex:0,events:w,iterator:0,node:x,visitorIndex:0,visitors:[]}}function v(x){return x[e]=!1,x.nodes&&x.nodes.forEach(w=>v(w)),x}let b={};class S{constructor(w,y,C){this.stringified=!1,this.processed=!1;let O;if(typeof y=="object"&&y!==null&&(y.type==="root"||y.type==="document"))O=v(y);else if(y instanceof S||y instanceof p)O=v(y.root),y.map&&(typeof C.map>"u"&&(C.map={}),C.map.inline||(C.map.inline=!1),C.map.prev=y.map);else{let I=l;C.syntax&&(I=C.syntax.parse),C.parser&&(I=C.parser),I.parse&&(I=I.parse);try{O=I(y,C)}catch(M){this.processed=!0,this.error=M}O&&!O[t]&&s.rebuild(O)}this.result=new p(w,O,C),this.helpers={...b,postcss:b,result:this.result},this.plugins=this.processor.plugins.map(I=>typeof I=="object"&&I.prepare?{...I,...I.prepare(this.result)}:I)}async(){return this.error?Promise.reject(this.error):this.processed?Promise.resolve(this.result):(this.processing||(this.processing=this.runAsync()),this.processing)}catch(w){return this.async().catch(w)}finally(w){return this.async().then(w,w)}getAsyncError(){throw new Error("Use process(css).then(cb) to work with async plugins")}handleError(w,y){let C=this.result.lastPlugin;try{if(y&&y.addToError(w),this.error=w,w.name==="CssSyntaxError"&&!w.plugin)w.plugin=C.postcssPlugin,w.setMessage();else if(C.postcssVersion&&process.env.NODE_ENV!=="production"){let O=C.postcssPlugin,I=C.postcssVersion,M=this.result.processor.version,P=I.split("."),N=M.split(".");(P[0]!==N[0]||parseInt(P[1])>parseInt(N[1]))&&console.error("Unknown error from PostCSS plugin. Your current PostCSS version is "+M+", but "+O+" uses "+I+". Perhaps this is the source of the error below.")}}catch(O){console&&console.error&&console.error(O)}return w}prepareVisitors(){this.listeners={};let w=(y,C,O)=>{this.listeners[C]||(this.listeners[C]=[]),this.listeners[C].push([y,O])};for(let y of this.plugins)if(typeof y=="object")for(let C in y){if(!f[C]&&/^[A-Z]/.test(C))throw new Error(`Unknown event ${C} in ${y.postcssPlugin}. Try to update PostCSS (${this.processor.version} now).`);if(!a[C])if(typeof y[C]=="object")for(let O in y[C])O==="*"?w(y,C,y[C][O]):w(y,C+"-"+O.toLowerCase(),y[C][O]);else typeof y[C]=="function"&&w(y,C,y[C])}this.hasListener=Object.keys(this.listeners).length>0}async runAsync(){this.plugin=0;for(let w=0;w<this.plugins.length;w++){let y=this.plugins[w],C=this.runOnRoot(y);if(m(C))try{await C}catch(O){throw this.handleError(O)}}if(this.prepareVisitors(),this.hasListener){let w=this.result.root;for(;!w[e];){w[e]=!0;let y=[g(w)];for(;y.length>0;){let C=this.visitTick(y);if(m(C))try{await C}catch(O){let I=y[y.length-1].node;throw this.handleError(O,I)}}}if(this.listeners.OnceExit)for(let[y,C]of this.listeners.OnceExit){this.result.lastPlugin=y;try{if(w.type==="document"){let O=w.nodes.map(I=>C(I,this.helpers));await Promise.all(O)}else await C(w,this.helpers)}catch(O){throw this.handleError(O)}}}return this.processed=!0,this.stringify()}runOnRoot(w){this.result.lastPlugin=w;try{if(typeof w=="object"&&w.Once){if(this.result.root.type==="document"){let y=this.result.root.nodes.map(C=>w.Once(C,this.helpers));return m(y[0])?Promise.all(y):y}return w.Once(this.result.root,this.helpers)}else if(typeof w=="function")return w(this.result.root,this.result)}catch(y){throw this.handleError(y)}}stringify(){if(this.error)throw this.error;if(this.stringified)return this.result;this.stringified=!0,this.sync();let w=this.result.opts,y=o;w.syntax&&(y=w.syntax.stringify),w.stringifier&&(y=w.stringifier),y.stringify&&(y=y.stringify);let O=new r(y,this.result.root,this.result.opts).generate();return this.result.css=O[0],this.result.map=O[1],this.result}sync(){if(this.error)throw this.error;if(this.processed)return this.result;if(this.processed=!0,this.processing)throw this.getAsyncError();for(let w of this.plugins){let y=this.runOnRoot(w);if(m(y))throw this.getAsyncError()}if(this.prepareVisitors(),this.hasListener){let w=this.result.root;for(;!w[e];)w[e]=!0,this.walkSync(w);if(this.listeners.OnceExit)if(w.type==="document")for(let y of w.nodes)this.visitSync(this.listeners.OnceExit,y);else this.visitSync(this.listeners.OnceExit,w)}return this.result}then(w,y){return process.env.NODE_ENV!=="production"&&("from"in this.opts||c("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(w,y)}toString(){return this.css}visitSync(w,y){for(let[C,O]of w){this.result.lastPlugin=C;let I;try{I=O(y,this.helpers)}catch(M){throw this.handleError(M,y.proxyOf)}if(y.type!=="root"&&y.type!=="document"&&!y.parent)return!0;if(m(I))throw this.getAsyncError()}}visitTick(w){let y=w[w.length-1],{node:C,visitors:O}=y;if(C.type!=="root"&&C.type!=="document"&&!C.parent){w.pop();return}if(O.length>0&&y.visitorIndex<O.length){let[M,P]=O[y.visitorIndex];y.visitorIndex+=1,y.visitorIndex===O.length&&(y.visitors=[],y.visitorIndex=0),this.result.lastPlugin=M;try{return P(C.toProxy(),this.helpers)}catch(N){throw this.handleError(N,C)}}if(y.iterator!==0){let M=y.iterator,P;for(;P=C.nodes[C.indexes[M]];)if(C.indexes[M]+=1,!P[e]){P[e]=!0,w.push(g(P));return}y.iterator=0,delete C.indexes[M]}let I=y.events;for(;y.eventIndex<I.length;){let M=I[y.eventIndex];if(y.eventIndex+=1,M===n){C.nodes&&C.nodes.length&&(C[e]=!0,y.iterator=C.getIterator());return}else if(this.listeners[M]){y.visitors=this.listeners[M];return}}w.pop()}walkSync(w){w[e]=!0;let y=h(w);for(let C of y)if(C===n)w.nodes&&w.each(O=>{O[e]||this.walkSync(O)});else{let O=this.listeners[C];if(O&&this.visitSync(O,w.toProxy()))return}}warnings(){return this.sync().warnings()}get content(){return this.stringify().content}get css(){return this.stringify().css}get map(){return this.stringify().map}get messages(){return this.sync().messages}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){return this.sync().root}get[Symbol.toStringTag](){return"LazyResult"}}return S.registerPostcss=x=>{b=x},ur=S,S.default=S,d.registerLazyResult(S),u.registerLazyResult(S),ur}var cr,ni;function Lo(){if(ni)return cr;ni=1;let e=Bs(),t=ot(),r=Gs(),o=lr();const s=Kt();class u{constructor(p,l,d){l=l.toString(),this.stringified=!1,this._processor=p,this._css=l,this._opts=d,this._map=void 0;let i,f=t;this.result=new s(this._processor,i,this._opts),this.result.css=l;let a=this;Object.defineProperty(this.result,"root",{get(){return a.root}});let n=new e(f,i,this._opts,l);if(n.isMap()){let[m,h]=n.generate();m&&(this.result.css=m),h&&(this.result.map=h)}else n.clearAnnotation(),this.result.css=n.css}async(){return this.error?Promise.reject(this.error):Promise.resolve(this.result)}catch(p){return this.async().catch(p)}finally(p){return this.async().then(p,p)}sync(){if(this.error)throw this.error;return this.result}then(p,l){return process.env.NODE_ENV!=="production"&&("from"in this._opts||r("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(p,l)}toString(){return this._css}warnings(){return[]}get content(){return this.result.css}get css(){return this.result.css}get map(){return this.result.map}get messages(){return[]}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){if(this._root)return this._root;let p,l=o;try{p=l(this._css,this._opts)}catch(d){this.error=d}if(this.error)throw this.error;return this._root=p,p}get[Symbol.toStringTag](){return"NoWorkResult"}}return cr=u,u.default=u,cr}var hr,oi;function _o(){if(oi)return hr;oi=1;let e=Lo(),t=ii(),r=Jt(),o=je();class s{constructor(c=[]){this.version="8.4.38",this.plugins=this.normalize(c)}normalize(c){let p=[];for(let l of c)if(l.postcss===!0?l=l():l.postcss&&(l=l.postcss),typeof l=="object"&&Array.isArray(l.plugins))p=p.concat(l.plugins);else if(typeof l=="object"&&l.postcssPlugin)p.push(l);else if(typeof l=="function")p.push(l);else if(typeof l=="object"&&(l.parse||l.stringify)){if(process.env.NODE_ENV!=="production")throw new Error("PostCSS syntaxes cannot be used as plugins. Instead, please use one of the syntax/parser/stringifier options as outlined in your PostCSS runner documentation.")}else throw new Error(l+" is not a PostCSS plugin");return p}process(c,p={}){return!this.plugins.length&&!p.parser&&!p.stringifier&&!p.syntax?new e(this,c,p):new t(this,c,p)}use(c){return this.plugins=this.plugins.concat(this.normalize([c])),this}}return hr=s,s.default=s,o.registerProcessor(s),r.registerProcessor(s),hr}var fr,ai;function $o(){if(ai)return fr;ai=1;let e=lt(),t=$s(),r=ct(),o=tr(),s=ut(),u=je(),c=nr();function p(l,d){if(Array.isArray(l))return l.map(a=>p(a));let{inputs:i,...f}=l;if(i){d=[];for(let a of i){let n={...a,__proto__:s.prototype};n.map&&(n.map={...n.map,__proto__:t.prototype}),d.push(n)}}if(f.nodes&&(f.nodes=l.nodes.map(a=>p(a,d))),f.source){let{inputId:a,...n}=f.source;f.source=n,a!=null&&(f.source.input=d[a])}if(f.type==="root")return new u(f);if(f.type==="decl")return new e(f);if(f.type==="rule")return new c(f);if(f.type==="comment")return new r(f);if(f.type==="atrule")return new o(f);throw new Error("Unknown node type: "+l.type)}return fr=p,p.default=p,fr}var dr,li;function Fo(){if(li)return dr;li=1;let e=Lt(),t=lt(),r=ii(),o=Ie(),s=_o(),u=ot(),c=$o(),p=Jt(),l=Hs(),d=ct(),i=tr(),f=Kt(),a=ut(),n=lr(),m=Xs(),h=nr(),g=je(),v=at();function b(...S){return S.length===1&&Array.isArray(S[0])&&(S=S[0]),new s(S)}return b.plugin=function(x,w){let y=!1;function C(...I){console&&console.warn&&!y&&(y=!0,console.warn(x+`: postcss.plugin was deprecated. Migration guide:
+https://evilmartians.com/chronicles/postcss-8-plugin-migration`),process.env.LANG&&process.env.LANG.startsWith("cn")&&console.warn(x+`: 里面 postcss.plugin 被弃用. 迁移指南:
+https://www.w3ctech.com/topic/2226`));let M=w(...I);return M.postcssPlugin=x,M.postcssVersion=new s().version,M}let O;return Object.defineProperty(C,"postcss",{get(){return O||(O=C()),O}}),C.process=function(I,M,P){return b([C(P)]).process(I,M)},C},b.stringify=u,b.parse=n,b.fromJSON=c,b.list=m,b.comment=S=>new d(S),b.atRule=S=>new i(S),b.decl=S=>new t(S),b.rule=S=>new h(S),b.root=S=>new g(S),b.document=S=>new p(S),b.CssSyntaxError=e,b.Declaration=t,b.Container=o,b.Processor=s,b.Document=p,b.Comment=d,b.Warning=l,b.AtRule=i,b.Result=f,b.Input=a,b.Rule=h,b.Root=g,b.Node=v,r.registerPostcss(b),dr=b,b.default=b,dr}var Uo=Fo();const G=Ao(Uo);G.stringify,G.fromJSON,G.plugin,G.parse,G.list,G.document,G.comment,G.atRule,G.rule,G.decl,G.root,G.CssSyntaxError,G.Declaration,G.Container,G.Processor,G.Document,G.Comment,G.Warning,G.AtRule,G.Result,G.Input,G.Rule,G.Root,G.Node;var Bo=Object.defineProperty,zo=(e,t,r)=>t in e?Bo(e,t,{enumerable:!0,configurable:!0,writable:!0,value:r}):e[t]=r,he=(e,t,r)=>zo(e,typeof t!="symbol"?t+"":t,r);Date.now().toString();function Wo(e){return e&&e.__esModule&&Object.prototype.hasOwnProperty.call(e,"default")?e.default:e}function qo(e){if(e.__esModule)return e;var t=e.default;if(typeof t=="function"){var r=function o(){return this instanceof o?Reflect.construct(t,arguments,this.constructor):t.apply(this,arguments)};r.prototype=t.prototype}else r={};return Object.defineProperty(r,"__esModule",{value:!0}),Object.keys(e).forEach(function(o){var s=Object.getOwnPropertyDescriptor(e,o);Object.defineProperty(r,o,s.get?s:{enumerable:!0,get:function(){return e[o]}})}),r}var ht={exports:{}},ui;function jo(){if(ui)return ht.exports;ui=1;var e=String,t=function(){return{isColorSupported:!1,reset:e,bold:e,dim:e,italic:e,underline:e,inverse:e,hidden:e,strikethrough:e,black:e,red:e,green:e,yellow:e,blue:e,magenta:e,cyan:e,white:e,gray:e,bgBlack:e,bgRed:e,bgGreen:e,bgYellow:e,bgBlue:e,bgMagenta:e,bgCyan:e,bgWhite:e}};return ht.exports=t(),ht.exports.createColors=t,ht.exports}const me=qo(Object.freeze(Object.defineProperty({__proto__:null,default:{}},Symbol.toStringTag,{value:"Module"})));var pr,ci;function mr(){if(ci)return pr;ci=1;let e=jo(),t=me;class r extends Error{constructor(s,u,c,p,l,d){super(s),this.name="CssSyntaxError",this.reason=s,l&&(this.file=l),p&&(this.source=p),d&&(this.plugin=d),typeof u<"u"&&typeof c<"u"&&(typeof u=="number"?(this.line=u,this.column=c):(this.line=u.line,this.column=u.column,this.endLine=c.line,this.endColumn=c.column)),this.setMessage(),Error.captureStackTrace&&Error.captureStackTrace(this,r)}setMessage(){this.message=this.plugin?this.plugin+": ":"",this.message+=this.file?this.file:"<css input>",typeof this.line<"u"&&(this.message+=":"+this.line+":"+this.column),this.message+=": "+this.reason}showSourceCode(s){if(!this.source)return"";let u=this.source;s==null&&(s=e.isColorSupported),t&&s&&(u=t(u));let c=u.split(/\r?\n/),p=Math.max(this.line-3,0),l=Math.min(this.line+2,c.length),d=String(l).length,i,f;if(s){let{bold:a,gray:n,red:m}=e.createColors(!0);i=h=>a(m(h)),f=h=>n(h)}else i=f=a=>a;return c.slice(p,l).map((a,n)=>{let m=p+1+n,h=" "+(" "+m).slice(-d)+" | ";if(m===this.line){let g=f(h.replace(/\d/g," "))+a.slice(0,this.column-1).replace(/[^\t]/g," ");return i(">")+f(h)+a+`
+ `+g+i("^")}return" "+f(h)+a}).join(`
+`)}toString(){let s=this.showSourceCode();return s&&(s=`
-`+e+`
-`),this.name+": "+this.message+e}};var is=ss;ss.default=ss;var st={};st.isClean=Symbol("isClean"),st.my=Symbol("my");const un={after:`
+`+s+`
+`),this.name+": "+this.message+s}}return pr=r,r.default=r,pr}var ft={},hi;function gr(){return hi||(hi=1,ft.isClean=Symbol("isClean"),ft.my=Symbol("my")),ft}var yr,fi;function di(){if(fi)return yr;fi=1;const e={after:`
`,beforeClose:`
`,beforeComment:`
`,beforeDecl:`
`,beforeOpen:" ",beforeRule:`
-`,colon:": ",commentLeft:" ",commentRight:" ",emptyBody:"",indent:" ",semicolon:!1};function Lu(s){return s[0].toUpperCase()+s.slice(1)}let ns=class{constructor(e){this.builder=e}atrule(e,t){let r="@"+e.name,i=e.params?this.rawValue(e,"params"):"";if(typeof e.raws.afterName<"u"?r+=e.raws.afterName:i&&(r+=" "),e.nodes)this.block(e,r+i);else{let n=(e.raws.between||"")+(t?";":"");this.builder(r+i+n,e)}}beforeAfter(e,t){let r;e.type==="decl"?r=this.raw(e,null,"beforeDecl"):e.type==="comment"?r=this.raw(e,null,"beforeComment"):t==="before"?r=this.raw(e,null,"beforeRule"):r=this.raw(e,null,"beforeClose");let i=e.parent,n=0;for(;i&&i.type!=="root";)n+=1,i=i.parent;if(r.includes(`
-`)){let o=this.raw(e,null,"indent");if(o.length)for(let l=0;l<n;l++)r+=o}return r}block(e,t){let r=this.raw(e,"between","beforeOpen");this.builder(t+r+"{",e,"start");let i;e.nodes&&e.nodes.length?(this.body(e),i=this.raw(e,"after")):i=this.raw(e,"after","emptyBody"),i&&this.builder(i),this.builder("}",e,"end")}body(e){let t=e.nodes.length-1;for(;t>0&&e.nodes[t].type==="comment";)t-=1;let r=this.raw(e,"semicolon");for(let i=0;i<e.nodes.length;i++){let n=e.nodes[i],o=this.raw(n,"before");o&&this.builder(o),this.stringify(n,t!==i||r)}}comment(e){let t=this.raw(e,"left","commentLeft"),r=this.raw(e,"right","commentRight");this.builder("/*"+t+e.text+r+"*/",e)}decl(e,t){let r=this.raw(e,"between","colon"),i=e.prop+r+this.rawValue(e,"value");e.important&&(i+=e.raws.important||" !important"),t&&(i+=";"),this.builder(i,e)}document(e){this.body(e)}raw(e,t,r){let i;if(r||(r=t),t&&(i=e.raws[t],typeof i<"u"))return i;let n=e.parent;if(r==="before"&&(!n||n.type==="root"&&n.first===e||n&&n.type==="document"))return"";if(!n)return un[r];let o=e.root();if(o.rawCache||(o.rawCache={}),typeof o.rawCache[r]<"u")return o.rawCache[r];if(r==="before"||r==="after")return this.beforeAfter(e,r);{let l="raw"+Lu(r);this[l]?i=this[l](o,e):o.walk(a=>{if(i=a.raws[t],typeof i<"u")return!1})}return typeof i>"u"&&(i=un[r]),o.rawCache[r]=i,i}rawBeforeClose(e){let t;return e.walk(r=>{if(r.nodes&&r.nodes.length>0&&typeof r.raws.after<"u")return t=r.raws.after,t.includes(`
-`)&&(t=t.replace(/[^\n]+$/,"")),!1}),t&&(t=t.replace(/\S/g,"")),t}rawBeforeComment(e,t){let r;return e.walkComments(i=>{if(typeof i.raws.before<"u")return r=i.raws.before,r.includes(`
-`)&&(r=r.replace(/[^\n]+$/,"")),!1}),typeof r>"u"?r=this.raw(t,null,"beforeDecl"):r&&(r=r.replace(/\S/g,"")),r}rawBeforeDecl(e,t){let r;return e.walkDecls(i=>{if(typeof i.raws.before<"u")return r=i.raws.before,r.includes(`
-`)&&(r=r.replace(/[^\n]+$/,"")),!1}),typeof r>"u"?r=this.raw(t,null,"beforeRule"):r&&(r=r.replace(/\S/g,"")),r}rawBeforeOpen(e){let t;return e.walk(r=>{if(r.type!=="decl"&&(t=r.raws.between,typeof t<"u"))return!1}),t}rawBeforeRule(e){let t;return e.walk(r=>{if(r.nodes&&(r.parent!==e||e.first!==r)&&typeof r.raws.before<"u")return t=r.raws.before,t.includes(`
-`)&&(t=t.replace(/[^\n]+$/,"")),!1}),t&&(t=t.replace(/\S/g,"")),t}rawColon(e){let t;return e.walkDecls(r=>{if(typeof r.raws.between<"u")return t=r.raws.between.replace(/[^\s:]/g,""),!1}),t}rawEmptyBody(e){let t;return e.walk(r=>{if(r.nodes&&r.nodes.length===0&&(t=r.raws.after,typeof t<"u"))return!1}),t}rawIndent(e){if(e.raws.indent)return e.raws.indent;let t;return e.walk(r=>{let i=r.parent;if(i&&i!==e&&i.parent&&i.parent===e&&typeof r.raws.before<"u"){let n=r.raws.before.split(`
-`);return t=n[n.length-1],t=t.replace(/\S/g,""),!1}}),t}rawSemicolon(e){let t;return e.walk(r=>{if(r.nodes&&r.nodes.length&&r.last.type==="decl"&&(t=r.raws.semicolon,typeof t<"u"))return!1}),t}rawValue(e,t){let r=e[t],i=e.raws[t];return i&&i.value===r?i.raw:r}root(e){this.body(e),e.raws.after&&this.builder(e.raws.after)}rule(e){this.block(e,this.rawValue(e,"selector")),e.raws.ownSemicolon&&this.builder(e.raws.ownSemicolon,e,"end")}stringify(e,t){if(!this[e.type])throw new Error("Unknown AST node type "+e.type+". Maybe you need to change PostCSS stringifier.");this[e.type](e,t)}};var cn=ns;ns.default=ns;let _u=cn;function os(s,e){new _u(e).stringify(s)}var Wt=os;os.default=os;let{isClean:Gt,my:Fu}=st,Uu=is,Bu=cn,zu=Wt;function as(s,e){let t=new s.constructor;for(let r in s){if(!Object.prototype.hasOwnProperty.call(s,r)||r==="proxyCache")continue;let i=s[r],n=typeof i;r==="parent"&&n==="object"?e&&(t[r]=e):r==="source"?t[r]=i:Array.isArray(i)?t[r]=i.map(o=>as(o,t)):(n==="object"&&i!==null&&(i=as(i)),t[r]=i)}return t}let ls=class{constructor(e={}){this.raws={},this[Gt]=!1,this[Fu]=!0;for(let t in e)if(t==="nodes"){this.nodes=[];for(let r of e[t])typeof r.clone=="function"?this.append(r.clone()):this.append(r)}else this[t]=e[t]}addToError(e){if(e.postcssNode=this,e.stack&&this.source&&/\n\s{4}at /.test(e.stack)){let t=this.source;e.stack=e.stack.replace(/\n\s{4}at /,`$&${t.input.from}:${t.start.line}:${t.start.column}$&`)}return e}after(e){return this.parent.insertAfter(this,e),this}assign(e={}){for(let t in e)this[t]=e[t];return this}before(e){return this.parent.insertBefore(this,e),this}cleanRaws(e){delete this.raws.before,delete this.raws.after,e||delete this.raws.between}clone(e={}){let t=as(this);for(let r in e)t[r]=e[r];return t}cloneAfter(e={}){let t=this.clone(e);return this.parent.insertAfter(this,t),t}cloneBefore(e={}){let t=this.clone(e);return this.parent.insertBefore(this,t),t}error(e,t={}){if(this.source){let{end:r,start:i}=this.rangeBy(t);return this.source.input.error(e,{column:i.column,line:i.line},{column:r.column,line:r.line},t)}return new Uu(e)}getProxyProcessor(){return{get(e,t){return t==="proxyOf"?e:t==="root"?()=>e.root().toProxy():e[t]},set(e,t,r){return e[t]===r||(e[t]=r,(t==="prop"||t==="value"||t==="name"||t==="params"||t==="important"||t==="text")&&e.markDirty()),!0}}}markDirty(){if(this[Gt]){this[Gt]=!1;let e=this;for(;e=e.parent;)e[Gt]=!1}}next(){if(!this.parent)return;let e=this.parent.index(this);return this.parent.nodes[e+1]}positionBy(e,t){let r=this.source.start;if(e.index)r=this.positionInside(e.index,t);else if(e.word){t=this.toString();let i=t.indexOf(e.word);i!==-1&&(r=this.positionInside(i,t))}return r}positionInside(e,t){let r=t||this.toString(),i=this.source.start.column,n=this.source.start.line;for(let o=0;o<e;o++)r[o]===`
-`?(i=1,n+=1):i+=1;return{column:i,line:n}}prev(){if(!this.parent)return;let e=this.parent.index(this);return this.parent.nodes[e-1]}rangeBy(e){let t={column:this.source.start.column,line:this.source.start.line},r=this.source.end?{column:this.source.end.column+1,line:this.source.end.line}:{column:t.column+1,line:t.line};if(e.word){let i=this.toString(),n=i.indexOf(e.word);n!==-1&&(t=this.positionInside(n,i),r=this.positionInside(n+e.word.length,i))}else e.start?t={column:e.start.column,line:e.start.line}:e.index&&(t=this.positionInside(e.index)),e.end?r={column:e.end.column,line:e.end.line}:typeof e.endIndex=="number"?r=this.positionInside(e.endIndex):e.index&&(r=this.positionInside(e.index+1));return(r.line<t.line||r.line===t.line&&r.column<=t.column)&&(r={column:t.column+1,line:t.line}),{end:r,start:t}}raw(e,t){return new Bu().raw(this,e,t)}remove(){return this.parent&&this.parent.removeChild(this),this.parent=void 0,this}replaceWith(...e){if(this.parent){let t=this,r=!1;for(let i of e)i===this?r=!0:r?(this.parent.insertAfter(t,i),t=i):this.parent.insertBefore(t,i);r||this.remove()}return this}root(){let e=this;for(;e.parent&&e.parent.type!=="document";)e=e.parent;return e}toJSON(e,t){let r={},i=t==null;t=t||new Map;let n=0;for(let o in this){if(!Object.prototype.hasOwnProperty.call(this,o)||o==="parent"||o==="proxyCache")continue;let l=this[o];if(Array.isArray(l))r[o]=l.map(a=>typeof a=="object"&&a.toJSON?a.toJSON(null,t):a);else if(typeof l=="object"&&l.toJSON)r[o]=l.toJSON(null,t);else if(o==="source"){let a=t.get(l.input);a==null&&(a=n,t.set(l.input,n),n++),r[o]={end:l.end,inputId:a,start:l.start}}else r[o]=l}return i&&(r.inputs=[...t.keys()].map(o=>o.toJSON())),r}toProxy(){return this.proxyCache||(this.proxyCache=new Proxy(this,this.getProxyProcessor())),this.proxyCache}toString(e=zu){e.stringify&&(e=e.stringify);let t="";return e(this,r=>{t+=r}),t}warn(e,t,r){let i={node:this};for(let n in r)i[n]=r[n];return e.warn(t,i)}get proxyOf(){return this}};var Vt=ls;ls.default=ls;let Wu=Vt,us=class extends Wu{constructor(e){e&&typeof e.value<"u"&&typeof e.value!="string"&&(e={...e,value:String(e.value)}),super(e),this.type="decl"}get variable(){return this.prop.startsWith("--")||this.prop[0]==="$"}};var jt=us;us.default=us;let Gu="useandom-26T198340PX75pxJACKVERYMINDBUSHWOLF_GQZbfghjklqvwyzrict";var Vu={nanoid:(s=21)=>{let e="",t=s;for(;t--;)e+=Gu[Math.random()*64|0];return e}};let{SourceMapConsumer:hn,SourceMapGenerator:fn}=ae,{existsSync:ju,readFileSync:Hu}=ae,{dirname:cs,join:Yu}=ae;function Zu(s){return Buffer?Buffer.from(s,"base64").toString():window.atob(s)}let hs=class{constructor(e,t){if(t.map===!1)return;this.loadAnnotation(e),this.inline=this.startWith(this.annotation,"data:");let r=t.map?t.map.prev:void 0,i=this.loadMap(t.from,r);!this.mapFile&&t.from&&(this.mapFile=t.from),this.mapFile&&(this.root=cs(this.mapFile)),i&&(this.text=i)}consumer(){return this.consumerCache||(this.consumerCache=new hn(this.text)),this.consumerCache}decodeInline(e){let t=/^data:application\/json;charset=utf-?8;base64,/,r=/^data:application\/json;base64,/,i=/^data:application\/json;charset=utf-?8,/,n=/^data:application\/json,/;if(i.test(e)||n.test(e))return decodeURIComponent(e.substr(RegExp.lastMatch.length));if(t.test(e)||r.test(e))return Zu(e.substr(RegExp.lastMatch.length));let o=e.match(/data:application\/json;([^,]+),/)[1];throw new Error("Unsupported source map encoding "+o)}getAnnotationURL(e){return e.replace(/^\/\*\s*# sourceMappingURL=/,"").trim()}isMap(e){return typeof e!="object"?!1:typeof e.mappings=="string"||typeof e._mappings=="string"||Array.isArray(e.sections)}loadAnnotation(e){let t=e.match(/\/\*\s*# sourceMappingURL=/gm);if(!t)return;let r=e.lastIndexOf(t.pop()),i=e.indexOf("*/",r);r>-1&&i>-1&&(this.annotation=this.getAnnotationURL(e.substring(r,i)))}loadFile(e){if(this.root=cs(e),ju(e))return this.mapFile=e,Hu(e,"utf-8").toString().trim()}loadMap(e,t){if(t===!1)return!1;if(t){if(typeof t=="string")return t;if(typeof t=="function"){let r=t(e);if(r){let i=this.loadFile(r);if(!i)throw new Error("Unable to load previous source map: "+r.toString());return i}}else{if(t instanceof hn)return fn.fromSourceMap(t).toString();if(t instanceof fn)return t.toString();if(this.isMap(t))return JSON.stringify(t);throw new Error("Unsupported previous source map format: "+t.toString())}}else{if(this.inline)return this.decodeInline(this.annotation);if(this.annotation){let r=this.annotation;return e&&(r=Yu(cs(e),r)),this.loadFile(r)}}}startWith(e,t){return e?e.substr(0,t.length)===t:!1}withContent(){return!!(this.consumer().sourcesContent&&this.consumer().sourcesContent.length>0)}};var dn=hs;hs.default=hs;let{SourceMapConsumer:Ju,SourceMapGenerator:Ku}=ae,{fileURLToPath:pn,pathToFileURL:Ht}=ae,{isAbsolute:fs,resolve:ds}=ae,{nanoid:Xu}=Vu,ps=ae,mn=is,Qu=dn,ms=Symbol("fromOffsetCache"),qu=!!(Ju&&Ku),gn=!!(ds&&fs),Yt=class{constructor(e,t={}){if(e===null||typeof e>"u"||typeof e=="object"&&!e.toString)throw new Error(`PostCSS received ${e} instead of CSS string`);if(this.css=e.toString(),this.css[0]==="\uFEFF"||this.css[0]==="￾"?(this.hasBOM=!0,this.css=this.css.slice(1)):this.hasBOM=!1,t.from&&(!gn||/^\w+:\/\//.test(t.from)||fs(t.from)?this.file=t.from:this.file=ds(t.from)),gn&&qu){let r=new Qu(this.css,t);if(r.text){this.map=r;let i=r.consumer().file;!this.file&&i&&(this.file=this.mapResolve(i))}}this.file||(this.id="<input css "+Xu(6)+">"),this.map&&(this.map.file=this.from)}error(e,t,r,i={}){let n,o,l;if(t&&typeof t=="object"){let u=t,c=r;if(typeof u.offset=="number"){let h=this.fromOffset(u.offset);t=h.line,r=h.col}else t=u.line,r=u.column;if(typeof c.offset=="number"){let h=this.fromOffset(c.offset);o=h.line,l=h.col}else o=c.line,l=c.column}else if(!r){let u=this.fromOffset(t);t=u.line,r=u.col}let a=this.origin(t,r,o,l);return a?n=new mn(e,a.endLine===void 0?a.line:{column:a.column,line:a.line},a.endLine===void 0?a.column:{column:a.endColumn,line:a.endLine},a.source,a.file,i.plugin):n=new mn(e,o===void 0?t:{column:r,line:t},o===void 0?r:{column:l,line:o},this.css,this.file,i.plugin),n.input={column:r,endColumn:l,endLine:o,line:t,source:this.css},this.file&&(Ht&&(n.input.url=Ht(this.file).toString()),n.input.file=this.file),n}fromOffset(e){let t,r;if(this[ms])r=this[ms];else{let n=this.css.split(`
-`);r=new Array(n.length);let o=0;for(let l=0,a=n.length;l<a;l++)r[l]=o,o+=n[l].length+1;this[ms]=r}t=r[r.length-1];let i=0;if(e>=t)i=r.length-1;else{let n=r.length-2,o;for(;i<n;)if(o=i+(n-i>>1),e<r[o])n=o-1;else if(e>=r[o+1])i=o+1;else{i=o;break}}return{col:e-r[i]+1,line:i+1}}mapResolve(e){return/^\w+:\/\//.test(e)?e:ds(this.map.consumer().sourceRoot||this.map.root||".",e)}origin(e,t,r,i){if(!this.map)return!1;let n=this.map.consumer(),o=n.originalPositionFor({column:t,line:e});if(!o.source)return!1;let l;typeof r=="number"&&(l=n.originalPositionFor({column:i,line:r}));let a;fs(o.source)?a=Ht(o.source):a=new URL(o.source,this.map.consumer().sourceRoot||Ht(this.map.mapFile));let u={column:o.column,endColumn:l&&l.column,endLine:l&&l.line,line:o.line,url:a.toString()};if(a.protocol==="file:")if(pn)u.file=pn(a);else throw new Error("file: protocol is not available in this PostCSS build");let c=n.sourceContentFor(o.source);return c&&(u.source=c),u}toJSON(){let e={};for(let t of["hasBOM","css","file","id"])this[t]!=null&&(e[t]=this[t]);return this.map&&(e.map={...this.map},e.map.consumerCache&&(e.map.consumerCache=void 0)),e}get from(){return this.file||this.id}};var Zt=Yt;Yt.default=Yt,ps&&ps.registerInput&&ps.registerInput(Yt);let{SourceMapConsumer:yn,SourceMapGenerator:Jt}=ae,{dirname:Kt,relative:wn,resolve:Sn,sep:bn}=ae,{pathToFileURL:vn}=ae,ec=Zt,tc=!!(yn&&Jt),rc=!!(Kt&&Sn&&wn&&bn);var Cn=class{constructor(e,t,r,i){this.stringify=e,this.mapOpts=r.map||{},this.root=t,this.opts=r,this.css=i,this.originalCSS=i,this.usesFileUrls=!this.mapOpts.from&&this.mapOpts.absolute,this.memoizedFileURLs=new Map,this.memoizedPaths=new Map,this.memoizedURLs=new Map}addAnnotation(){let e;this.isInline()?e="data:application/json;base64,"+this.toBase64(this.map.toString()):typeof this.mapOpts.annotation=="string"?e=this.mapOpts.annotation:typeof this.mapOpts.annotation=="function"?e=this.mapOpts.annotation(this.opts.to,this.root):e=this.outputFile()+".map";let t=`
+`,colon:": ",commentLeft:" ",commentRight:" ",emptyBody:"",indent:" ",semicolon:!1};function t(o){return o[0].toUpperCase()+o.slice(1)}class r{constructor(s){this.builder=s}atrule(s,u){let c="@"+s.name,p=s.params?this.rawValue(s,"params"):"";if(typeof s.raws.afterName<"u"?c+=s.raws.afterName:p&&(c+=" "),s.nodes)this.block(s,c+p);else{let l=(s.raws.between||"")+(u?";":"");this.builder(c+p+l,s)}}beforeAfter(s,u){let c;s.type==="decl"?c=this.raw(s,null,"beforeDecl"):s.type==="comment"?c=this.raw(s,null,"beforeComment"):u==="before"?c=this.raw(s,null,"beforeRule"):c=this.raw(s,null,"beforeClose");let p=s.parent,l=0;for(;p&&p.type!=="root";)l+=1,p=p.parent;if(c.includes(`
+`)){let d=this.raw(s,null,"indent");if(d.length)for(let i=0;i<l;i++)c+=d}return c}block(s,u){let c=this.raw(s,"between","beforeOpen");this.builder(u+c+"{",s,"start");let p;s.nodes&&s.nodes.length?(this.body(s),p=this.raw(s,"after")):p=this.raw(s,"after","emptyBody"),p&&this.builder(p),this.builder("}",s,"end")}body(s){let u=s.nodes.length-1;for(;u>0&&s.nodes[u].type==="comment";)u-=1;let c=this.raw(s,"semicolon");for(let p=0;p<s.nodes.length;p++){let l=s.nodes[p],d=this.raw(l,"before");d&&this.builder(d),this.stringify(l,u!==p||c)}}comment(s){let u=this.raw(s,"left","commentLeft"),c=this.raw(s,"right","commentRight");this.builder("/*"+u+s.text+c+"*/",s)}decl(s,u){let c=this.raw(s,"between","colon"),p=s.prop+c+this.rawValue(s,"value");s.important&&(p+=s.raws.important||" !important"),u&&(p+=";"),this.builder(p,s)}document(s){this.body(s)}raw(s,u,c){let p;if(c||(c=u),u&&(p=s.raws[u],typeof p<"u"))return p;let l=s.parent;if(c==="before"&&(!l||l.type==="root"&&l.first===s||l&&l.type==="document"))return"";if(!l)return e[c];let d=s.root();if(d.rawCache||(d.rawCache={}),typeof d.rawCache[c]<"u")return d.rawCache[c];if(c==="before"||c==="after")return this.beforeAfter(s,c);{let i="raw"+t(c);this[i]?p=this[i](d,s):d.walk(f=>{if(p=f.raws[u],typeof p<"u")return!1})}return typeof p>"u"&&(p=e[c]),d.rawCache[c]=p,p}rawBeforeClose(s){let u;return s.walk(c=>{if(c.nodes&&c.nodes.length>0&&typeof c.raws.after<"u")return u=c.raws.after,u.includes(`
+`)&&(u=u.replace(/[^\n]+$/,"")),!1}),u&&(u=u.replace(/\S/g,"")),u}rawBeforeComment(s,u){let c;return s.walkComments(p=>{if(typeof p.raws.before<"u")return c=p.raws.before,c.includes(`
+`)&&(c=c.replace(/[^\n]+$/,"")),!1}),typeof c>"u"?c=this.raw(u,null,"beforeDecl"):c&&(c=c.replace(/\S/g,"")),c}rawBeforeDecl(s,u){let c;return s.walkDecls(p=>{if(typeof p.raws.before<"u")return c=p.raws.before,c.includes(`
+`)&&(c=c.replace(/[^\n]+$/,"")),!1}),typeof c>"u"?c=this.raw(u,null,"beforeRule"):c&&(c=c.replace(/\S/g,"")),c}rawBeforeOpen(s){let u;return s.walk(c=>{if(c.type!=="decl"&&(u=c.raws.between,typeof u<"u"))return!1}),u}rawBeforeRule(s){let u;return s.walk(c=>{if(c.nodes&&(c.parent!==s||s.first!==c)&&typeof c.raws.before<"u")return u=c.raws.before,u.includes(`
+`)&&(u=u.replace(/[^\n]+$/,"")),!1}),u&&(u=u.replace(/\S/g,"")),u}rawColon(s){let u;return s.walkDecls(c=>{if(typeof c.raws.between<"u")return u=c.raws.between.replace(/[^\s:]/g,""),!1}),u}rawEmptyBody(s){let u;return s.walk(c=>{if(c.nodes&&c.nodes.length===0&&(u=c.raws.after,typeof u<"u"))return!1}),u}rawIndent(s){if(s.raws.indent)return s.raws.indent;let u;return s.walk(c=>{let p=c.parent;if(p&&p!==s&&p.parent&&p.parent===s&&typeof c.raws.before<"u"){let l=c.raws.before.split(`
+`);return u=l[l.length-1],u=u.replace(/\S/g,""),!1}}),u}rawSemicolon(s){let u;return s.walk(c=>{if(c.nodes&&c.nodes.length&&c.last.type==="decl"&&(u=c.raws.semicolon,typeof u<"u"))return!1}),u}rawValue(s,u){let c=s[u],p=s.raws[u];return p&&p.value===c?p.raw:c}root(s){this.body(s),s.raws.after&&this.builder(s.raws.after)}rule(s){this.block(s,this.rawValue(s,"selector")),s.raws.ownSemicolon&&this.builder(s.raws.ownSemicolon,s,"end")}stringify(s,u){if(!this[s.type])throw new Error("Unknown AST node type "+s.type+". Maybe you need to change PostCSS stringifier.");this[s.type](s,u)}}return yr=r,r.default=r,yr}var wr,pi;function dt(){if(pi)return wr;pi=1;let e=di();function t(r,o){new e(o).stringify(r)}return wr=t,t.default=t,wr}var br,mi;function pt(){if(mi)return br;mi=1;let{isClean:e,my:t}=gr(),r=mr(),o=di(),s=dt();function u(p,l){let d=new p.constructor;for(let i in p){if(!Object.prototype.hasOwnProperty.call(p,i)||i==="proxyCache")continue;let f=p[i],a=typeof f;i==="parent"&&a==="object"?l&&(d[i]=l):i==="source"?d[i]=f:Array.isArray(f)?d[i]=f.map(n=>u(n,d)):(a==="object"&&f!==null&&(f=u(f)),d[i]=f)}return d}class c{constructor(l={}){this.raws={},this[e]=!1,this[t]=!0;for(let d in l)if(d==="nodes"){this.nodes=[];for(let i of l[d])typeof i.clone=="function"?this.append(i.clone()):this.append(i)}else this[d]=l[d]}addToError(l){if(l.postcssNode=this,l.stack&&this.source&&/\n\s{4}at /.test(l.stack)){let d=this.source;l.stack=l.stack.replace(/\n\s{4}at /,`$&${d.input.from}:${d.start.line}:${d.start.column}$&`)}return l}after(l){return this.parent.insertAfter(this,l),this}assign(l={}){for(let d in l)this[d]=l[d];return this}before(l){return this.parent.insertBefore(this,l),this}cleanRaws(l){delete this.raws.before,delete this.raws.after,l||delete this.raws.between}clone(l={}){let d=u(this);for(let i in l)d[i]=l[i];return d}cloneAfter(l={}){let d=this.clone(l);return this.parent.insertAfter(this,d),d}cloneBefore(l={}){let d=this.clone(l);return this.parent.insertBefore(this,d),d}error(l,d={}){if(this.source){let{end:i,start:f}=this.rangeBy(d);return this.source.input.error(l,{column:f.column,line:f.line},{column:i.column,line:i.line},d)}return new r(l)}getProxyProcessor(){return{get(l,d){return d==="proxyOf"?l:d==="root"?()=>l.root().toProxy():l[d]},set(l,d,i){return l[d]===i||(l[d]=i,(d==="prop"||d==="value"||d==="name"||d==="params"||d==="important"||d==="text")&&l.markDirty()),!0}}}markDirty(){if(this[e]){this[e]=!1;let l=this;for(;l=l.parent;)l[e]=!1}}next(){if(!this.parent)return;let l=this.parent.index(this);return this.parent.nodes[l+1]}positionBy(l,d){let i=this.source.start;if(l.index)i=this.positionInside(l.index,d);else if(l.word){d=this.toString();let f=d.indexOf(l.word);f!==-1&&(i=this.positionInside(f,d))}return i}positionInside(l,d){let i=d||this.toString(),f=this.source.start.column,a=this.source.start.line;for(let n=0;n<l;n++)i[n]===`
+`?(f=1,a+=1):f+=1;return{column:f,line:a}}prev(){if(!this.parent)return;let l=this.parent.index(this);return this.parent.nodes[l-1]}rangeBy(l){let d={column:this.source.start.column,line:this.source.start.line},i=this.source.end?{column:this.source.end.column+1,line:this.source.end.line}:{column:d.column+1,line:d.line};if(l.word){let f=this.toString(),a=f.indexOf(l.word);a!==-1&&(d=this.positionInside(a,f),i=this.positionInside(a+l.word.length,f))}else l.start?d={column:l.start.column,line:l.start.line}:l.index&&(d=this.positionInside(l.index)),l.end?i={column:l.end.column,line:l.end.line}:typeof l.endIndex=="number"?i=this.positionInside(l.endIndex):l.index&&(i=this.positionInside(l.index+1));return(i.line<d.line||i.line===d.line&&i.column<=d.column)&&(i={column:d.column+1,line:d.line}),{end:i,start:d}}raw(l,d){return new o().raw(this,l,d)}remove(){return this.parent&&this.parent.removeChild(this),this.parent=void 0,this}replaceWith(...l){if(this.parent){let d=this,i=!1;for(let f of l)f===this?i=!0:i?(this.parent.insertAfter(d,f),d=f):this.parent.insertBefore(d,f);i||this.remove()}return this}root(){let l=this;for(;l.parent&&l.parent.type!=="document";)l=l.parent;return l}toJSON(l,d){let i={},f=d==null;d=d||new Map;let a=0;for(let n in this){if(!Object.prototype.hasOwnProperty.call(this,n)||n==="parent"||n==="proxyCache")continue;let m=this[n];if(Array.isArray(m))i[n]=m.map(h=>typeof h=="object"&&h.toJSON?h.toJSON(null,d):h);else if(typeof m=="object"&&m.toJSON)i[n]=m.toJSON(null,d);else if(n==="source"){let h=d.get(m.input);h==null&&(h=a,d.set(m.input,a),a++),i[n]={end:m.end,inputId:h,start:m.start}}else i[n]=m}return f&&(i.inputs=[...d.keys()].map(n=>n.toJSON())),i}toProxy(){return this.proxyCache||(this.proxyCache=new Proxy(this,this.getProxyProcessor())),this.proxyCache}toString(l=s){l.stringify&&(l=l.stringify);let d="";return l(this,i=>{d+=i}),d}warn(l,d,i){let f={node:this};for(let a in i)f[a]=i[a];return l.warn(d,f)}get proxyOf(){return this}}return br=c,c.default=c,br}var Sr,gi;function mt(){if(gi)return Sr;gi=1;let e=pt();class t extends e{constructor(o){o&&typeof o.value<"u"&&typeof o.value!="string"&&(o={...o,value:String(o.value)}),super(o),this.type="decl"}get variable(){return this.prop.startsWith("--")||this.prop[0]==="$"}}return Sr=t,t.default=t,Sr}var vr,yi;function Go(){if(yi)return vr;yi=1;let e="useandom-26T198340PX75pxJACKVERYMINDBUSHWOLF_GQZbfghjklqvwyzrict";return vr={nanoid:(o=21)=>{let s="",u=o;for(;u--;)s+=e[Math.random()*64|0];return s},customAlphabet:(o,s=21)=>(u=s)=>{let c="",p=u;for(;p--;)c+=o[Math.random()*o.length|0];return c}},vr}var Cr,wi;function bi(){if(wi)return Cr;wi=1;let{SourceMapConsumer:e,SourceMapGenerator:t}=me,{existsSync:r,readFileSync:o}=me,{dirname:s,join:u}=me;function c(l){return Buffer?Buffer.from(l,"base64").toString():window.atob(l)}class p{constructor(d,i){if(i.map===!1)return;this.loadAnnotation(d),this.inline=this.startWith(this.annotation,"data:");let f=i.map?i.map.prev:void 0,a=this.loadMap(i.from,f);!this.mapFile&&i.from&&(this.mapFile=i.from),this.mapFile&&(this.root=s(this.mapFile)),a&&(this.text=a)}consumer(){return this.consumerCache||(this.consumerCache=new e(this.text)),this.consumerCache}decodeInline(d){let i=/^data:application\/json;charset=utf-?8;base64,/,f=/^data:application\/json;base64,/,a=/^data:application\/json;charset=utf-?8,/,n=/^data:application\/json,/;if(a.test(d)||n.test(d))return decodeURIComponent(d.substr(RegExp.lastMatch.length));if(i.test(d)||f.test(d))return c(d.substr(RegExp.lastMatch.length));let m=d.match(/data:application\/json;([^,]+),/)[1];throw new Error("Unsupported source map encoding "+m)}getAnnotationURL(d){return d.replace(/^\/\*\s*# sourceMappingURL=/,"").trim()}isMap(d){return typeof d!="object"?!1:typeof d.mappings=="string"||typeof d._mappings=="string"||Array.isArray(d.sections)}loadAnnotation(d){let i=d.match(/\/\*\s*# sourceMappingURL=/gm);if(!i)return;let f=d.lastIndexOf(i.pop()),a=d.indexOf("*/",f);f>-1&&a>-1&&(this.annotation=this.getAnnotationURL(d.substring(f,a)))}loadFile(d){if(this.root=s(d),r(d))return this.mapFile=d,o(d,"utf-8").toString().trim()}loadMap(d,i){if(i===!1)return!1;if(i){if(typeof i=="string")return i;if(typeof i=="function"){let f=i(d);if(f){let a=this.loadFile(f);if(!a)throw new Error("Unable to load previous source map: "+f.toString());return a}}else{if(i instanceof e)return t.fromSourceMap(i).toString();if(i instanceof t)return i.toString();if(this.isMap(i))return JSON.stringify(i);throw new Error("Unsupported previous source map format: "+i.toString())}}else{if(this.inline)return this.decodeInline(this.annotation);if(this.annotation){let f=this.annotation;return d&&(f=u(s(d),f)),this.loadFile(f)}}}startWith(d,i){return d?d.substr(0,i.length)===i:!1}withContent(){return!!(this.consumer().sourcesContent&&this.consumer().sourcesContent.length>0)}}return Cr=p,p.default=p,Cr}var xr,Si;function gt(){if(Si)return xr;Si=1;let{SourceMapConsumer:e,SourceMapGenerator:t}=me,{fileURLToPath:r,pathToFileURL:o}=me,{isAbsolute:s,resolve:u}=me,{nanoid:c}=Go(),p=me,l=mr(),d=bi(),i=Symbol("fromOffsetCache"),f=!!(e&&t),a=!!(u&&s);class n{constructor(h,g={}){if(h===null||typeof h>"u"||typeof h=="object"&&!h.toString)throw new Error(`PostCSS received ${h} instead of CSS string`);if(this.css=h.toString(),this.css[0]==="\uFEFF"||this.css[0]==="￾"?(this.hasBOM=!0,this.css=this.css.slice(1)):this.hasBOM=!1,g.from&&(!a||/^\w+:\/\//.test(g.from)||s(g.from)?this.file=g.from:this.file=u(g.from)),a&&f){let v=new d(this.css,g);if(v.text){this.map=v;let b=v.consumer().file;!this.file&&b&&(this.file=this.mapResolve(b))}}this.file||(this.id="<input css "+c(6)+">"),this.map&&(this.map.file=this.from)}error(h,g,v,b={}){let S,x,w;if(g&&typeof g=="object"){let C=g,O=v;if(typeof C.offset=="number"){let I=this.fromOffset(C.offset);g=I.line,v=I.col}else g=C.line,v=C.column;if(typeof O.offset=="number"){let I=this.fromOffset(O.offset);x=I.line,w=I.col}else x=O.line,w=O.column}else if(!v){let C=this.fromOffset(g);g=C.line,v=C.col}let y=this.origin(g,v,x,w);return y?S=new l(h,y.endLine===void 0?y.line:{column:y.column,line:y.line},y.endLine===void 0?y.column:{column:y.endColumn,line:y.endLine},y.source,y.file,b.plugin):S=new l(h,x===void 0?g:{column:v,line:g},x===void 0?v:{column:w,line:x},this.css,this.file,b.plugin),S.input={column:v,endColumn:w,endLine:x,line:g,source:this.css},this.file&&(o&&(S.input.url=o(this.file).toString()),S.input.file=this.file),S}fromOffset(h){let g,v;if(this[i])v=this[i];else{let S=this.css.split(`
+`);v=new Array(S.length);let x=0;for(let w=0,y=S.length;w<y;w++)v[w]=x,x+=S[w].length+1;this[i]=v}g=v[v.length-1];let b=0;if(h>=g)b=v.length-1;else{let S=v.length-2,x;for(;b<S;)if(x=b+(S-b>>1),h<v[x])S=x-1;else if(h>=v[x+1])b=x+1;else{b=x;break}}return{col:h-v[b]+1,line:b+1}}mapResolve(h){return/^\w+:\/\//.test(h)?h:u(this.map.consumer().sourceRoot||this.map.root||".",h)}origin(h,g,v,b){if(!this.map)return!1;let S=this.map.consumer(),x=S.originalPositionFor({column:g,line:h});if(!x.source)return!1;let w;typeof v=="number"&&(w=S.originalPositionFor({column:b,line:v}));let y;s(x.source)?y=o(x.source):y=new URL(x.source,this.map.consumer().sourceRoot||o(this.map.mapFile));let C={column:x.column,endColumn:w&&w.column,endLine:w&&w.line,line:x.line,url:y.toString()};if(y.protocol==="file:")if(r)C.file=r(y);else throw new Error("file: protocol is not available in this PostCSS build");let O=S.sourceContentFor(x.source);return O&&(C.source=O),C}toJSON(){let h={};for(let g of["hasBOM","css","file","id"])this[g]!=null&&(h[g]=this[g]);return this.map&&(h.map={...this.map},h.map.consumerCache&&(h.map.consumerCache=void 0)),h}get from(){return this.file||this.id}}return xr=n,n.default=n,p&&p.registerInput&&p.registerInput(n),xr}var Rr,vi;function Ci(){if(vi)return Rr;vi=1;let{SourceMapConsumer:e,SourceMapGenerator:t}=me,{dirname:r,relative:o,resolve:s,sep:u}=me,{pathToFileURL:c}=me,p=gt(),l=!!(e&&t),d=!!(r&&s&&o&&u);class i{constructor(a,n,m,h){this.stringify=a,this.mapOpts=m.map||{},this.root=n,this.opts=m,this.css=h,this.originalCSS=h,this.usesFileUrls=!this.mapOpts.from&&this.mapOpts.absolute,this.memoizedFileURLs=new Map,this.memoizedPaths=new Map,this.memoizedURLs=new Map}addAnnotation(){let a;this.isInline()?a="data:application/json;base64,"+this.toBase64(this.map.toString()):typeof this.mapOpts.annotation=="string"?a=this.mapOpts.annotation:typeof this.mapOpts.annotation=="function"?a=this.mapOpts.annotation(this.opts.to,this.root):a=this.outputFile()+".map";let n=`
`;this.css.includes(`\r
-`)&&(t=`\r
-`),this.css+=t+"/*# sourceMappingURL="+e+" */"}applyPrevMaps(){for(let e of this.previous()){let t=this.toUrl(this.path(e.file)),r=e.root||Kt(e.file),i;this.mapOpts.sourcesContent===!1?(i=new yn(e.text),i.sourcesContent&&(i.sourcesContent=null)):i=e.consumer(),this.map.applySourceMap(i,t,this.toUrl(this.path(r)))}}clearAnnotation(){if(this.mapOpts.annotation!==!1)if(this.root){let e;for(let t=this.root.nodes.length-1;t>=0;t--)e=this.root.nodes[t],e.type==="comment"&&e.text.indexOf("# sourceMappingURL=")===0&&this.root.removeChild(t)}else this.css&&(this.css=this.css.replace(/\n*?\/\*#[\S\s]*?\*\/$/gm,""))}generate(){if(this.clearAnnotation(),rc&&tc&&this.isMap())return this.generateMap();{let e="";return this.stringify(this.root,t=>{e+=t}),[e]}}generateMap(){if(this.root)this.generateString();else if(this.previous().length===1){let e=this.previous()[0].consumer();e.file=this.outputFile(),this.map=Jt.fromSourceMap(e,{ignoreInvalidMapping:!0})}else this.map=new Jt({file:this.outputFile(),ignoreInvalidMapping:!0}),this.map.addMapping({generated:{column:0,line:1},original:{column:0,line:1},source:this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>"});return this.isSourcesContent()&&this.setSourcesContent(),this.root&&this.previous().length>0&&this.applyPrevMaps(),this.isAnnotation()&&this.addAnnotation(),this.isInline()?[this.css]:[this.css,this.map]}generateString(){this.css="",this.map=new Jt({file:this.outputFile(),ignoreInvalidMapping:!0});let e=1,t=1,r="<no source>",i={generated:{column:0,line:0},original:{column:0,line:0},source:""},n,o;this.stringify(this.root,(l,a,u)=>{if(this.css+=l,a&&u!=="end"&&(i.generated.line=e,i.generated.column=t-1,a.source&&a.source.start?(i.source=this.sourcePath(a),i.original.line=a.source.start.line,i.original.column=a.source.start.column-1,this.map.addMapping(i)):(i.source=r,i.original.line=1,i.original.column=0,this.map.addMapping(i))),n=l.match(/\n/g),n?(e+=n.length,o=l.lastIndexOf(`
-`),t=l.length-o):t+=l.length,a&&u!=="start"){let c=a.parent||{raws:{}};(!(a.type==="decl"||a.type==="atrule"&&!a.nodes)||a!==c.last||c.raws.semicolon)&&(a.source&&a.source.end?(i.source=this.sourcePath(a),i.original.line=a.source.end.line,i.original.column=a.source.end.column-1,i.generated.line=e,i.generated.column=t-2,this.map.addMapping(i)):(i.source=r,i.original.line=1,i.original.column=0,i.generated.line=e,i.generated.column=t-1,this.map.addMapping(i)))}})}isAnnotation(){return this.isInline()?!0:typeof this.mapOpts.annotation<"u"?this.mapOpts.annotation:this.previous().length?this.previous().some(e=>e.annotation):!0}isInline(){if(typeof this.mapOpts.inline<"u")return this.mapOpts.inline;let e=this.mapOpts.annotation;return typeof e<"u"&&e!==!0?!1:this.previous().length?this.previous().some(t=>t.inline):!0}isMap(){return typeof this.opts.map<"u"?!!this.opts.map:this.previous().length>0}isSourcesContent(){return typeof this.mapOpts.sourcesContent<"u"?this.mapOpts.sourcesContent:this.previous().length?this.previous().some(e=>e.withContent()):!0}outputFile(){return this.opts.to?this.path(this.opts.to):this.opts.from?this.path(this.opts.from):"to.css"}path(e){if(this.mapOpts.absolute||e.charCodeAt(0)===60||/^\w+:\/\//.test(e))return e;let t=this.memoizedPaths.get(e);if(t)return t;let r=this.opts.to?Kt(this.opts.to):".";typeof this.mapOpts.annotation=="string"&&(r=Kt(Sn(r,this.mapOpts.annotation)));let i=wn(r,e);return this.memoizedPaths.set(e,i),i}previous(){if(!this.previousMaps)if(this.previousMaps=[],this.root)this.root.walk(e=>{if(e.source&&e.source.input.map){let t=e.source.input.map;this.previousMaps.includes(t)||this.previousMaps.push(t)}});else{let e=new ec(this.originalCSS,this.opts);e.map&&this.previousMaps.push(e.map)}return this.previousMaps}setSourcesContent(){let e={};if(this.root)this.root.walk(t=>{if(t.source){let r=t.source.input.from;if(r&&!e[r]){e[r]=!0;let i=this.usesFileUrls?this.toFileUrl(r):this.toUrl(this.path(r));this.map.setSourceContent(i,t.source.input.css)}}});else if(this.css){let t=this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>";this.map.setSourceContent(t,this.css)}}sourcePath(e){return this.mapOpts.from?this.toUrl(this.mapOpts.from):this.usesFileUrls?this.toFileUrl(e.source.input.from):this.toUrl(this.path(e.source.input.from))}toBase64(e){return Buffer?Buffer.from(e).toString("base64"):window.btoa(unescape(encodeURIComponent(e)))}toFileUrl(e){let t=this.memoizedFileURLs.get(e);if(t)return t;if(vn){let r=vn(e).toString();return this.memoizedFileURLs.set(e,r),r}else throw new Error("`map.absolute` option is not available in this PostCSS build")}toUrl(e){let t=this.memoizedURLs.get(e);if(t)return t;bn==="\\"&&(e=e.replace(/\\/g,"/"));let r=encodeURI(e).replace(/[#?]/g,encodeURIComponent);return this.memoizedURLs.set(e,r),r}};let sc=Vt,gs=class extends sc{constructor(e){super(e),this.type="comment"}};var Xt=gs;gs.default=gs;let{isClean:In,my:xn}=st,An=jt,Rn=Xt,ic=Vt,On,ys,ws,En;function Mn(s){return s.map(e=>(e.nodes&&(e.nodes=Mn(e.nodes)),delete e.source,e))}function $n(s){if(s[In]=!1,s.proxyOf.nodes)for(let e of s.proxyOf.nodes)$n(e)}let Se=class Jo extends ic{append(...e){for(let t of e){let r=this.normalize(t,this.last);for(let i of r)this.proxyOf.nodes.push(i)}return this.markDirty(),this}cleanRaws(e){if(super.cleanRaws(e),this.nodes)for(let t of this.nodes)t.cleanRaws(e)}each(e){if(!this.proxyOf.nodes)return;let t=this.getIterator(),r,i;for(;this.indexes[t]<this.proxyOf.nodes.length&&(r=this.indexes[t],i=e(this.proxyOf.nodes[r],r),i!==!1);)this.indexes[t]+=1;return delete this.indexes[t],i}every(e){return this.nodes.every(e)}getIterator(){this.lastEach||(this.lastEach=0),this.indexes||(this.indexes={}),this.lastEach+=1;let e=this.lastEach;return this.indexes[e]=0,e}getProxyProcessor(){return{get(e,t){return t==="proxyOf"?e:e[t]?t==="each"||typeof t=="string"&&t.startsWith("walk")?(...r)=>e[t](...r.map(i=>typeof i=="function"?(n,o)=>i(n.toProxy(),o):i)):t==="every"||t==="some"?r=>e[t]((i,...n)=>r(i.toProxy(),...n)):t==="root"?()=>e.root().toProxy():t==="nodes"?e.nodes.map(r=>r.toProxy()):t==="first"||t==="last"?e[t].toProxy():e[t]:e[t]},set(e,t,r){return e[t]===r||(e[t]=r,(t==="name"||t==="params"||t==="selector")&&e.markDirty()),!0}}}index(e){return typeof e=="number"?e:(e.proxyOf&&(e=e.proxyOf),this.proxyOf.nodes.indexOf(e))}insertAfter(e,t){let r=this.index(e),i=this.normalize(t,this.proxyOf.nodes[r]).reverse();r=this.index(e);for(let o of i)this.proxyOf.nodes.splice(r+1,0,o);let n;for(let o in this.indexes)n=this.indexes[o],r<n&&(this.indexes[o]=n+i.length);return this.markDirty(),this}insertBefore(e,t){let r=this.index(e),i=r===0?"prepend":!1,n=this.normalize(t,this.proxyOf.nodes[r],i).reverse();r=this.index(e);for(let l of n)this.proxyOf.nodes.splice(r,0,l);let o;for(let l in this.indexes)o=this.indexes[l],r<=o&&(this.indexes[l]=o+n.length);return this.markDirty(),this}normalize(e,t){if(typeof e=="string")e=Mn(On(e).nodes);else if(typeof e>"u")e=[];else if(Array.isArray(e)){e=e.slice(0);for(let i of e)i.parent&&i.parent.removeChild(i,"ignore")}else if(e.type==="root"&&this.type!=="document"){e=e.nodes.slice(0);for(let i of e)i.parent&&i.parent.removeChild(i,"ignore")}else if(e.type)e=[e];else if(e.prop){if(typeof e.value>"u")throw new Error("Value field is missed in node creation");typeof e.value!="string"&&(e.value=String(e.value)),e=[new An(e)]}else if(e.selector)e=[new ys(e)];else if(e.name)e=[new ws(e)];else if(e.text)e=[new Rn(e)];else throw new Error("Unknown node type in node creation");return e.map(i=>(i[xn]||Jo.rebuild(i),i=i.proxyOf,i.parent&&i.parent.removeChild(i),i[In]&&$n(i),typeof i.raws.before>"u"&&t&&typeof t.raws.before<"u"&&(i.raws.before=t.raws.before.replace(/\S/g,"")),i.parent=this.proxyOf,i))}prepend(...e){e=e.reverse();for(let t of e){let r=this.normalize(t,this.first,"prepend").reverse();for(let i of r)this.proxyOf.nodes.unshift(i);for(let i in this.indexes)this.indexes[i]=this.indexes[i]+r.length}return this.markDirty(),this}push(e){return e.parent=this,this.proxyOf.nodes.push(e),this}removeAll(){for(let e of this.proxyOf.nodes)e.parent=void 0;return this.proxyOf.nodes=[],this.markDirty(),this}removeChild(e){e=this.index(e),this.proxyOf.nodes[e].parent=void 0,this.proxyOf.nodes.splice(e,1);let t;for(let r in this.indexes)t=this.indexes[r],t>=e&&(this.indexes[r]=t-1);return this.markDirty(),this}replaceValues(e,t,r){return r||(r=t,t={}),this.walkDecls(i=>{t.props&&!t.props.includes(i.prop)||t.fast&&!i.value.includes(t.fast)||(i.value=i.value.replace(e,r))}),this.markDirty(),this}some(e){return this.nodes.some(e)}walk(e){return this.each((t,r)=>{let i;try{i=e(t,r)}catch(n){throw t.addToError(n)}return i!==!1&&t.walk&&(i=t.walk(e)),i})}walkAtRules(e,t){return t?e instanceof RegExp?this.walk((r,i)=>{if(r.type==="atrule"&&e.test(r.name))return t(r,i)}):this.walk((r,i)=>{if(r.type==="atrule"&&r.name===e)return t(r,i)}):(t=e,this.walk((r,i)=>{if(r.type==="atrule")return t(r,i)}))}walkComments(e){return this.walk((t,r)=>{if(t.type==="comment")return e(t,r)})}walkDecls(e,t){return t?e instanceof RegExp?this.walk((r,i)=>{if(r.type==="decl"&&e.test(r.prop))return t(r,i)}):this.walk((r,i)=>{if(r.type==="decl"&&r.prop===e)return t(r,i)}):(t=e,this.walk((r,i)=>{if(r.type==="decl")return t(r,i)}))}walkRules(e,t){return t?e instanceof RegExp?this.walk((r,i)=>{if(r.type==="rule"&&e.test(r.selector))return t(r,i)}):this.walk((r,i)=>{if(r.type==="rule"&&r.selector===e)return t(r,i)}):(t=e,this.walk((r,i)=>{if(r.type==="rule")return t(r,i)}))}get first(){if(this.proxyOf.nodes)return this.proxyOf.nodes[0]}get last(){if(this.proxyOf.nodes)return this.proxyOf.nodes[this.proxyOf.nodes.length-1]}};Se.registerParse=s=>{On=s},Se.registerRule=s=>{ys=s},Se.registerAtRule=s=>{ws=s},Se.registerRoot=s=>{En=s};var Ae=Se;Se.default=Se,Se.rebuild=s=>{s.type==="atrule"?Object.setPrototypeOf(s,ws.prototype):s.type==="rule"?Object.setPrototypeOf(s,ys.prototype):s.type==="decl"?Object.setPrototypeOf(s,An.prototype):s.type==="comment"?Object.setPrototypeOf(s,Rn.prototype):s.type==="root"&&Object.setPrototypeOf(s,En.prototype),s[xn]=!0,s.nodes&&s.nodes.forEach(e=>{Se.rebuild(e)})};let nc=Ae,Nn,kn,it=class extends nc{constructor(e){super({type:"document",...e}),this.nodes||(this.nodes=[])}toResult(e={}){return new Nn(new kn,this,e).stringify()}};it.registerLazyResult=s=>{Nn=s},it.registerProcessor=s=>{kn=s};var Ss=it;it.default=it;let Tn={};var Pn=function(e){Tn[e]||(Tn[e]=!0,typeof console<"u"&&console.warn&&console.warn(e))};let bs=class{constructor(e,t={}){if(this.type="warning",this.text=e,t.node&&t.node.source){let r=t.node.rangeBy(t);this.line=r.start.line,this.column=r.start.column,this.endLine=r.end.line,this.endColumn=r.end.column}for(let r in t)this[r]=t[r]}toString(){return this.node?this.node.error(this.text,{index:this.index,plugin:this.plugin,word:this.word}).message:this.plugin?this.plugin+": "+this.text:this.text}};var Dn=bs;bs.default=bs;let oc=Dn,vs=class{constructor(e,t,r){this.processor=e,this.messages=[],this.root=t,this.opts=r,this.css=void 0,this.map=void 0}toString(){return this.css}warn(e,t={}){t.plugin||this.lastPlugin&&this.lastPlugin.postcssPlugin&&(t.plugin=this.lastPlugin.postcssPlugin);let r=new oc(e,t);return this.messages.push(r),r}warnings(){return this.messages.filter(e=>e.type==="warning")}get content(){return this.css}};var Cs=vs;vs.default=vs;const Is=39,Ln=34,Qt=92,_n=47,qt=10,nt=32,er=12,tr=9,rr=13,ac=91,lc=93,uc=40,cc=41,hc=123,fc=125,dc=59,pc=42,mc=58,gc=64,sr=/[\t\n\f\r "#'()/;[\\\]{}]/g,ir=/[\t\n\f\r !"#'():;@[\\\]{}]|\/(?=\*)/g,yc=/.[\r\n"'(/\\]/,Fn=/[\da-f]/i;var wc=function(e,t={}){let r=e.css.valueOf(),i=t.ignoreErrors,n,o,l,a,u,c,h,p,m,g,d=r.length,f=0,S=[],b=[];function y(){return f}function C(B){throw e.error("Unclosed "+B,f)}function k(){return b.length===0&&f>=d}function D(B){if(b.length)return b.pop();if(f>=d)return;let V=B?B.ignoreUnclosed:!1;switch(n=r.charCodeAt(f),n){case qt:case nt:case tr:case rr:case er:{o=f;do o+=1,n=r.charCodeAt(o);while(n===nt||n===qt||n===tr||n===rr||n===er);g=["space",r.slice(f,o)],f=o-1;break}case ac:case lc:case hc:case fc:case mc:case dc:case cc:{let Z=String.fromCharCode(n);g=[Z,Z,f];break}case uc:{if(p=S.length?S.pop()[1]:"",m=r.charCodeAt(f+1),p==="url"&&m!==Is&&m!==Ln&&m!==nt&&m!==qt&&m!==tr&&m!==er&&m!==rr){o=f;do{if(c=!1,o=r.indexOf(")",o+1),o===-1)if(i||V){o=f;break}else C("bracket");for(h=o;r.charCodeAt(h-1)===Qt;)h-=1,c=!c}while(c);g=["brackets",r.slice(f,o+1),f,o],f=o}else o=r.indexOf(")",f+1),a=r.slice(f,o+1),o===-1||yc.test(a)?g=["(","(",f]:(g=["brackets",a,f,o],f=o);break}case Is:case Ln:{l=n===Is?"'":'"',o=f;do{if(c=!1,o=r.indexOf(l,o+1),o===-1)if(i||V){o=f+1;break}else C("string");for(h=o;r.charCodeAt(h-1)===Qt;)h-=1,c=!c}while(c);g=["string",r.slice(f,o+1),f,o],f=o;break}case gc:{sr.lastIndex=f+1,sr.test(r),sr.lastIndex===0?o=r.length-1:o=sr.lastIndex-2,g=["at-word",r.slice(f,o+1),f,o],f=o;break}case Qt:{for(o=f,u=!0;r.charCodeAt(o+1)===Qt;)o+=1,u=!u;if(n=r.charCodeAt(o+1),u&&n!==_n&&n!==nt&&n!==qt&&n!==tr&&n!==rr&&n!==er&&(o+=1,Fn.test(r.charAt(o)))){for(;Fn.test(r.charAt(o+1));)o+=1;r.charCodeAt(o+1)===nt&&(o+=1)}g=["word",r.slice(f,o+1),f,o],f=o;break}default:{n===_n&&r.charCodeAt(f+1)===pc?(o=r.indexOf("*/",f+2)+1,o===0&&(i||V?o=r.length:C("comment")),g=["comment",r.slice(f,o+1),f,o],f=o):(ir.lastIndex=f+1,ir.test(r),ir.lastIndex===0?o=r.length-1:o=ir.lastIndex-2,g=["word",r.slice(f,o+1),f,o],S.push(g),f=o);break}}return f++,g}function H(B){b.push(B)}return{back:H,endOfFile:k,nextToken:D,position:y}};let Un=Ae,nr=class extends Un{constructor(e){super(e),this.type="atrule"}append(...e){return this.proxyOf.nodes||(this.nodes=[]),super.append(...e)}prepend(...e){return this.proxyOf.nodes||(this.nodes=[]),super.prepend(...e)}};var xs=nr;nr.default=nr,Un.registerAtRule(nr);let Bn=Ae,zn,Wn,Fe=class extends Bn{constructor(e){super(e),this.type="root",this.nodes||(this.nodes=[])}normalize(e,t,r){let i=super.normalize(e);if(t){if(r==="prepend")this.nodes.length>1?t.raws.before=this.nodes[1].raws.before:delete t.raws.before;else if(this.first!==t)for(let n of i)n.raws.before=t.raws.before}return i}removeChild(e,t){let r=this.index(e);return!t&&r===0&&this.nodes.length>1&&(this.nodes[1].raws.before=this.nodes[r].raws.before),super.removeChild(e)}toResult(e={}){return new zn(new Wn,this,e).stringify()}};Fe.registerLazyResult=s=>{zn=s},Fe.registerProcessor=s=>{Wn=s};var ot=Fe;Fe.default=Fe,Bn.registerRoot(Fe);let at={comma(s){return at.split(s,[","],!0)},space(s){let e=[" ",`
-`," "];return at.split(s,e)},split(s,e,t){let r=[],i="",n=!1,o=0,l=!1,a="",u=!1;for(let c of s)u?u=!1:c==="\\"?u=!0:l?c===a&&(l=!1):c==='"'||c==="'"?(l=!0,a=c):c==="("?o+=1:c===")"?o>0&&(o-=1):o===0&&e.includes(c)&&(n=!0),n?(i!==""&&r.push(i.trim()),i="",n=!1):i+=c;return(t||i!=="")&&r.push(i.trim()),r}};var Gn=at;at.default=at;let Vn=Ae,Sc=Gn,or=class extends Vn{constructor(e){super(e),this.type="rule",this.nodes||(this.nodes=[])}get selectors(){return Sc.comma(this.selector)}set selectors(e){let t=this.selector?this.selector.match(/,\s*/):null,r=t?t[0]:","+this.raw("between","beforeOpen");this.selector=e.join(r)}};var As=or;or.default=or,Vn.registerRule(or);let bc=jt,vc=wc,Cc=Xt,Ic=xs,xc=ot,jn=As;const Hn={empty:!0,space:!0};function Ac(s){for(let e=s.length-1;e>=0;e--){let t=s[e],r=t[3]||t[2];if(r)return r}}var Rc=class{constructor(e){this.input=e,this.root=new xc,this.current=this.root,this.spaces="",this.semicolon=!1,this.createTokenizer(),this.root.source={input:e,start:{column:1,line:1,offset:0}}}atrule(e){let t=new Ic;t.name=e[1].slice(1),t.name===""&&this.unnamedAtrule(t,e),this.init(t,e[2]);let r,i,n,o=!1,l=!1,a=[],u=[];for(;!this.tokenizer.endOfFile();){if(e=this.tokenizer.nextToken(),r=e[0],r==="("||r==="["?u.push(r==="("?")":"]"):r==="{"&&u.length>0?u.push("}"):r===u[u.length-1]&&u.pop(),u.length===0)if(r===";"){t.source.end=this.getPosition(e[2]),t.source.end.offset++,this.semicolon=!0;break}else if(r==="{"){l=!0;break}else if(r==="}"){if(a.length>0){for(n=a.length-1,i=a[n];i&&i[0]==="space";)i=a[--n];i&&(t.source.end=this.getPosition(i[3]||i[2]),t.source.end.offset++)}this.end(e);break}else a.push(e);else a.push(e);if(this.tokenizer.endOfFile()){o=!0;break}}t.raws.between=this.spacesAndCommentsFromEnd(a),a.length?(t.raws.afterName=this.spacesAndCommentsFromStart(a),this.raw(t,"params",a),o&&(e=a[a.length-1],t.source.end=this.getPosition(e[3]||e[2]),t.source.end.offset++,this.spaces=t.raws.between,t.raws.between="")):(t.raws.afterName="",t.params=""),l&&(t.nodes=[],this.current=t)}checkMissedSemicolon(e){let t=this.colon(e);if(t===!1)return;let r=0,i;for(let n=t-1;n>=0&&(i=e[n],!(i[0]!=="space"&&(r+=1,r===2)));n--);throw this.input.error("Missed semicolon",i[0]==="word"?i[3]+1:i[2])}colon(e){let t=0,r,i,n;for(let[o,l]of e.entries()){if(r=l,i=r[0],i==="("&&(t+=1),i===")"&&(t-=1),t===0&&i===":")if(!n)this.doubleColon(r);else{if(n[0]==="word"&&n[1]==="progid")continue;return o}n=r}return!1}comment(e){let t=new Cc;this.init(t,e[2]),t.source.end=this.getPosition(e[3]||e[2]),t.source.end.offset++;let r=e[1].slice(2,-2);if(/^\s*$/.test(r))t.text="",t.raws.left=r,t.raws.right="";else{let i=r.match(/^(\s*)([^]*\S)(\s*)$/);t.text=i[2],t.raws.left=i[1],t.raws.right=i[3]}}createTokenizer(){this.tokenizer=vc(this.input)}decl(e,t){let r=new bc;this.init(r,e[0][2]);let i=e[e.length-1];for(i[0]===";"&&(this.semicolon=!0,e.pop()),r.source.end=this.getPosition(i[3]||i[2]||Ac(e)),r.source.end.offset++;e[0][0]!=="word";)e.length===1&&this.unknownWord(e),r.raws.before+=e.shift()[1];for(r.source.start=this.getPosition(e[0][2]),r.prop="";e.length;){let u=e[0][0];if(u===":"||u==="space"||u==="comment")break;r.prop+=e.shift()[1]}r.raws.between="";let n;for(;e.length;)if(n=e.shift(),n[0]===":"){r.raws.between+=n[1];break}else n[0]==="word"&&/\w/.test(n[1])&&this.unknownWord([n]),r.raws.between+=n[1];(r.prop[0]==="_"||r.prop[0]==="*")&&(r.raws.before+=r.prop[0],r.prop=r.prop.slice(1));let o=[],l;for(;e.length&&(l=e[0][0],!(l!=="space"&&l!=="comment"));)o.push(e.shift());this.precheckMissedSemicolon(e);for(let u=e.length-1;u>=0;u--){if(n=e[u],n[1].toLowerCase()==="!important"){r.important=!0;let c=this.stringFrom(e,u);c=this.spacesFromEnd(e)+c,c!==" !important"&&(r.raws.important=c);break}else if(n[1].toLowerCase()==="important"){let c=e.slice(0),h="";for(let p=u;p>0;p--){let m=c[p][0];if(h.trim().indexOf("!")===0&&m!=="space")break;h=c.pop()[1]+h}h.trim().indexOf("!")===0&&(r.important=!0,r.raws.important=h,e=c)}if(n[0]!=="space"&&n[0]!=="comment")break}e.some(u=>u[0]!=="space"&&u[0]!=="comment")&&(r.raws.between+=o.map(u=>u[1]).join(""),o=[]),this.raw(r,"value",o.concat(e),t),r.value.includes(":")&&!t&&this.checkMissedSemicolon(e)}doubleColon(e){throw this.input.error("Double colon",{offset:e[2]},{offset:e[2]+e[1].length})}emptyRule(e){let t=new jn;this.init(t,e[2]),t.selector="",t.raws.between="",this.current=t}end(e){this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.semicolon=!1,this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.spaces="",this.current.parent?(this.current.source.end=this.getPosition(e[2]),this.current.source.end.offset++,this.current=this.current.parent):this.unexpectedClose(e)}endFile(){this.current.parent&&this.unclosedBlock(),this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.root.source.end=this.getPosition(this.tokenizer.position())}freeSemicolon(e){if(this.spaces+=e[1],this.current.nodes){let t=this.current.nodes[this.current.nodes.length-1];t&&t.type==="rule"&&!t.raws.ownSemicolon&&(t.raws.ownSemicolon=this.spaces,this.spaces="")}}getPosition(e){let t=this.input.fromOffset(e);return{column:t.col,line:t.line,offset:e}}init(e,t){this.current.push(e),e.source={input:this.input,start:this.getPosition(t)},e.raws.before=this.spaces,this.spaces="",e.type!=="comment"&&(this.semicolon=!1)}other(e){let t=!1,r=null,i=!1,n=null,o=[],l=e[1].startsWith("--"),a=[],u=e;for(;u;){if(r=u[0],a.push(u),r==="("||r==="[")n||(n=u),o.push(r==="("?")":"]");else if(l&&i&&r==="{")n||(n=u),o.push("}");else if(o.length===0)if(r===";")if(i){this.decl(a,l);return}else break;else if(r==="{"){this.rule(a);return}else if(r==="}"){this.tokenizer.back(a.pop()),t=!0;break}else r===":"&&(i=!0);else r===o[o.length-1]&&(o.pop(),o.length===0&&(n=null));u=this.tokenizer.nextToken()}if(this.tokenizer.endOfFile()&&(t=!0),o.length>0&&this.unclosedBracket(n),t&&i){if(!l)for(;a.length&&(u=a[a.length-1][0],!(u!=="space"&&u!=="comment"));)this.tokenizer.back(a.pop());this.decl(a,l)}else this.unknownWord(a)}parse(){let e;for(;!this.tokenizer.endOfFile();)switch(e=this.tokenizer.nextToken(),e[0]){case"space":this.spaces+=e[1];break;case";":this.freeSemicolon(e);break;case"}":this.end(e);break;case"comment":this.comment(e);break;case"at-word":this.atrule(e);break;case"{":this.emptyRule(e);break;default:this.other(e);break}this.endFile()}precheckMissedSemicolon(){}raw(e,t,r,i){let n,o,l=r.length,a="",u=!0,c,h;for(let p=0;p<l;p+=1)n=r[p],o=n[0],o==="space"&&p===l-1&&!i?u=!1:o==="comment"?(h=r[p-1]?r[p-1][0]:"empty",c=r[p+1]?r[p+1][0]:"empty",!Hn[h]&&!Hn[c]?a.slice(-1)===","?u=!1:a+=n[1]:u=!1):a+=n[1];if(!u){let p=r.reduce((m,g)=>m+g[1],"");e.raws[t]={raw:p,value:a}}e[t]=a}rule(e){e.pop();let t=new jn;this.init(t,e[0][2]),t.raws.between=this.spacesAndCommentsFromEnd(e),this.raw(t,"selector",e),this.current=t}spacesAndCommentsFromEnd(e){let t,r="";for(;e.length&&(t=e[e.length-1][0],!(t!=="space"&&t!=="comment"));)r=e.pop()[1]+r;return r}spacesAndCommentsFromStart(e){let t,r="";for(;e.length&&(t=e[0][0],!(t!=="space"&&t!=="comment"));)r+=e.shift()[1];return r}spacesFromEnd(e){let t,r="";for(;e.length&&(t=e[e.length-1][0],t==="space");)r=e.pop()[1]+r;return r}stringFrom(e,t){let r="";for(let i=t;i<e.length;i++)r+=e[i][1];return e.splice(t,e.length-t),r}unclosedBlock(){let e=this.current.source.start;throw this.input.error("Unclosed block",e.line,e.column)}unclosedBracket(e){throw this.input.error("Unclosed bracket",{offset:e[2]},{offset:e[2]+1})}unexpectedClose(e){throw this.input.error("Unexpected }",{offset:e[2]},{offset:e[2]+1})}unknownWord(e){throw this.input.error("Unknown word",{offset:e[0][2]},{offset:e[0][2]+e[0][1].length})}unnamedAtrule(e,t){throw this.input.error("At-rule without name",{offset:t[2]},{offset:t[2]+t[1].length})}};let Oc=Ae,Ec=Rc,Mc=Zt;function ar(s,e){let t=new Mc(s,e),r=new Ec(t);try{r.parse()}catch(i){throw process.env.NODE_ENV!=="production"&&i.name==="CssSyntaxError"&&e&&e.from&&(/\.scss$/i.test(e.from)?i.message+=`
-You tried to parse SCSS with the standard CSS parser; try again with the postcss-scss parser`:/\.sass/i.test(e.from)?i.message+=`
-You tried to parse Sass with the standard CSS parser; try again with the postcss-sass parser`:/\.less$/i.test(e.from)&&(i.message+=`
-You tried to parse Less with the standard CSS parser; try again with the postcss-less parser`)),i}return r.root}var Rs=ar;ar.default=ar,Oc.registerParse(ar);let{isClean:fe,my:$c}=st,Nc=Cn,kc=Wt,Tc=Ae,Pc=Ss,Dc=Pn,Yn=Cs,Lc=Rs,_c=ot;const Fc={atrule:"AtRule",comment:"Comment",decl:"Declaration",document:"Document",root:"Root",rule:"Rule"},Uc={AtRule:!0,AtRuleExit:!0,Comment:!0,CommentExit:!0,Declaration:!0,DeclarationExit:!0,Document:!0,DocumentExit:!0,Once:!0,OnceExit:!0,postcssPlugin:!0,prepare:!0,Root:!0,RootExit:!0,Rule:!0,RuleExit:!0},Bc={Once:!0,postcssPlugin:!0,prepare:!0},Ue=0;function lt(s){return typeof s=="object"&&typeof s.then=="function"}function Zn(s){let e=!1,t=Fc[s.type];return s.type==="decl"?e=s.prop.toLowerCase():s.type==="atrule"&&(e=s.name.toLowerCase()),e&&s.append?[t,t+"-"+e,Ue,t+"Exit",t+"Exit-"+e]:e?[t,t+"-"+e,t+"Exit",t+"Exit-"+e]:s.append?[t,Ue,t+"Exit"]:[t,t+"Exit"]}function Jn(s){let e;return s.type==="document"?e=["Document",Ue,"DocumentExit"]:s.type==="root"?e=["Root",Ue,"RootExit"]:e=Zn(s),{eventIndex:0,events:e,iterator:0,node:s,visitorIndex:0,visitors:[]}}function Os(s){return s[fe]=!1,s.nodes&&s.nodes.forEach(e=>Os(e)),s}let Es={},Be=class Ko{constructor(e,t,r){this.stringified=!1,this.processed=!1;let i;if(typeof t=="object"&&t!==null&&(t.type==="root"||t.type==="document"))i=Os(t);else if(t instanceof Ko||t instanceof Yn)i=Os(t.root),t.map&&(typeof r.map>"u"&&(r.map={}),r.map.inline||(r.map.inline=!1),r.map.prev=t.map);else{let n=Lc;r.syntax&&(n=r.syntax.parse),r.parser&&(n=r.parser),n.parse&&(n=n.parse);try{i=n(t,r)}catch(o){this.processed=!0,this.error=o}i&&!i[$c]&&Tc.rebuild(i)}this.result=new Yn(e,i,r),this.helpers={...Es,postcss:Es,result:this.result},this.plugins=this.processor.plugins.map(n=>typeof n=="object"&&n.prepare?{...n,...n.prepare(this.result)}:n)}async(){return this.error?Promise.reject(this.error):this.processed?Promise.resolve(this.result):(this.processing||(this.processing=this.runAsync()),this.processing)}catch(e){return this.async().catch(e)}finally(e){return this.async().then(e,e)}getAsyncError(){throw new Error("Use process(css).then(cb) to work with async plugins")}handleError(e,t){let r=this.result.lastPlugin;try{if(t&&t.addToError(e),this.error=e,e.name==="CssSyntaxError"&&!e.plugin)e.plugin=r.postcssPlugin,e.setMessage();else if(r.postcssVersion&&process.env.NODE_ENV!=="production"){let i=r.postcssPlugin,n=r.postcssVersion,o=this.result.processor.version,l=n.split("."),a=o.split(".");(l[0]!==a[0]||parseInt(l[1])>parseInt(a[1]))&&console.error("Unknown error from PostCSS plugin. Your current PostCSS version is "+o+", but "+i+" uses "+n+". Perhaps this is the source of the error below.")}}catch(i){console&&console.error&&console.error(i)}return e}prepareVisitors(){this.listeners={};let e=(t,r,i)=>{this.listeners[r]||(this.listeners[r]=[]),this.listeners[r].push([t,i])};for(let t of this.plugins)if(typeof t=="object")for(let r in t){if(!Uc[r]&&/^[A-Z]/.test(r))throw new Error(`Unknown event ${r} in ${t.postcssPlugin}. Try to update PostCSS (${this.processor.version} now).`);if(!Bc[r])if(typeof t[r]=="object")for(let i in t[r])i==="*"?e(t,r,t[r][i]):e(t,r+"-"+i.toLowerCase(),t[r][i]);else typeof t[r]=="function"&&e(t,r,t[r])}this.hasListener=Object.keys(this.listeners).length>0}async runAsync(){this.plugin=0;for(let e=0;e<this.plugins.length;e++){let t=this.plugins[e],r=this.runOnRoot(t);if(lt(r))try{await r}catch(i){throw this.handleError(i)}}if(this.prepareVisitors(),this.hasListener){let e=this.result.root;for(;!e[fe];){e[fe]=!0;let t=[Jn(e)];for(;t.length>0;){let r=this.visitTick(t);if(lt(r))try{await r}catch(i){let n=t[t.length-1].node;throw this.handleError(i,n)}}}if(this.listeners.OnceExit)for(let[t,r]of this.listeners.OnceExit){this.result.lastPlugin=t;try{if(e.type==="document"){let i=e.nodes.map(n=>r(n,this.helpers));await Promise.all(i)}else await r(e,this.helpers)}catch(i){throw this.handleError(i)}}}return this.processed=!0,this.stringify()}runOnRoot(e){this.result.lastPlugin=e;try{if(typeof e=="object"&&e.Once){if(this.result.root.type==="document"){let t=this.result.root.nodes.map(r=>e.Once(r,this.helpers));return lt(t[0])?Promise.all(t):t}return e.Once(this.result.root,this.helpers)}else if(typeof e=="function")return e(this.result.root,this.result)}catch(t){throw this.handleError(t)}}stringify(){if(this.error)throw this.error;if(this.stringified)return this.result;this.stringified=!0,this.sync();let e=this.result.opts,t=kc;e.syntax&&(t=e.syntax.stringify),e.stringifier&&(t=e.stringifier),t.stringify&&(t=t.stringify);let i=new Nc(t,this.result.root,this.result.opts).generate();return this.result.css=i[0],this.result.map=i[1],this.result}sync(){if(this.error)throw this.error;if(this.processed)return this.result;if(this.processed=!0,this.processing)throw this.getAsyncError();for(let e of this.plugins){let t=this.runOnRoot(e);if(lt(t))throw this.getAsyncError()}if(this.prepareVisitors(),this.hasListener){let e=this.result.root;for(;!e[fe];)e[fe]=!0,this.walkSync(e);if(this.listeners.OnceExit)if(e.type==="document")for(let t of e.nodes)this.visitSync(this.listeners.OnceExit,t);else this.visitSync(this.listeners.OnceExit,e)}return this.result}then(e,t){return process.env.NODE_ENV!=="production"&&("from"in this.opts||Dc("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(e,t)}toString(){return this.css}visitSync(e,t){for(let[r,i]of e){this.result.lastPlugin=r;let n;try{n=i(t,this.helpers)}catch(o){throw this.handleError(o,t.proxyOf)}if(t.type!=="root"&&t.type!=="document"&&!t.parent)return!0;if(lt(n))throw this.getAsyncError()}}visitTick(e){let t=e[e.length-1],{node:r,visitors:i}=t;if(r.type!=="root"&&r.type!=="document"&&!r.parent){e.pop();return}if(i.length>0&&t.visitorIndex<i.length){let[o,l]=i[t.visitorIndex];t.visitorIndex+=1,t.visitorIndex===i.length&&(t.visitors=[],t.visitorIndex=0),this.result.lastPlugin=o;try{return l(r.toProxy(),this.helpers)}catch(a){throw this.handleError(a,r)}}if(t.iterator!==0){let o=t.iterator,l;for(;l=r.nodes[r.indexes[o]];)if(r.indexes[o]+=1,!l[fe]){l[fe]=!0,e.push(Jn(l));return}t.iterator=0,delete r.indexes[o]}let n=t.events;for(;t.eventIndex<n.length;){let o=n[t.eventIndex];if(t.eventIndex+=1,o===Ue){r.nodes&&r.nodes.length&&(r[fe]=!0,t.iterator=r.getIterator());return}else if(this.listeners[o]){t.visitors=this.listeners[o];return}}e.pop()}walkSync(e){e[fe]=!0;let t=Zn(e);for(let r of t)if(r===Ue)e.nodes&&e.each(i=>{i[fe]||this.walkSync(i)});else{let i=this.listeners[r];if(i&&this.visitSync(i,e.toProxy()))return}}warnings(){return this.sync().warnings()}get content(){return this.stringify().content}get css(){return this.stringify().css}get map(){return this.stringify().map}get messages(){return this.sync().messages}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){return this.sync().root}get[Symbol.toStringTag](){return"LazyResult"}};Be.registerPostcss=s=>{Es=s};var Kn=Be;Be.default=Be,_c.registerLazyResult(Be),Pc.registerLazyResult(Be);let zc=Cn,Wc=Wt,Gc=Pn,Vc=Rs;const jc=Cs;let Ms=class{constructor(e,t,r){t=t.toString(),this.stringified=!1,this._processor=e,this._css=t,this._opts=r,this._map=void 0;let i,n=Wc;this.result=new jc(this._processor,i,this._opts),this.result.css=t;let o=this;Object.defineProperty(this.result,"root",{get(){return o.root}});let l=new zc(n,i,this._opts,t);if(l.isMap()){let[a,u]=l.generate();a&&(this.result.css=a),u&&(this.result.map=u)}else l.clearAnnotation(),this.result.css=l.css}async(){return this.error?Promise.reject(this.error):Promise.resolve(this.result)}catch(e){return this.async().catch(e)}finally(e){return this.async().then(e,e)}sync(){if(this.error)throw this.error;return this.result}then(e,t){return process.env.NODE_ENV!=="production"&&("from"in this._opts||Gc("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(e,t)}toString(){return this._css}warnings(){return[]}get content(){return this.result.css}get css(){return this.result.css}get map(){return this.result.map}get messages(){return[]}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){if(this._root)return this._root;let e,t=Vc;try{e=t(this._css,this._opts)}catch(r){this.error=r}if(this.error)throw this.error;return this._root=e,e}get[Symbol.toStringTag](){return"NoWorkResult"}};var Hc=Ms;Ms.default=Ms;let Yc=Hc,Zc=Kn,Jc=Ss,Kc=ot,ut=class{constructor(e=[]){this.version="8.4.38",this.plugins=this.normalize(e)}normalize(e){let t=[];for(let r of e)if(r.postcss===!0?r=r():r.postcss&&(r=r.postcss),typeof r=="object"&&Array.isArray(r.plugins))t=t.concat(r.plugins);else if(typeof r=="object"&&r.postcssPlugin)t.push(r);else if(typeof r=="function")t.push(r);else if(typeof r=="object"&&(r.parse||r.stringify)){if(process.env.NODE_ENV!=="production")throw new Error("PostCSS syntaxes cannot be used as plugins. Instead, please use one of the syntax/parser/stringifier options as outlined in your PostCSS runner documentation.")}else throw new Error(r+" is not a PostCSS plugin");return t}process(e,t={}){return!this.plugins.length&&!t.parser&&!t.stringifier&&!t.syntax?new Yc(this,e,t):new Zc(this,e,t)}use(e){return this.plugins=this.plugins.concat(this.normalize([e])),this}};var Xc=ut;ut.default=ut,Kc.registerProcessor(ut),Jc.registerProcessor(ut);let Qc=jt,qc=dn,eh=Xt,th=xs,rh=Zt,sh=ot,ih=As;function ct(s,e){if(Array.isArray(s))return s.map(i=>ct(i));let{inputs:t,...r}=s;if(t){e=[];for(let i of t){let n={...i,__proto__:rh.prototype};n.map&&(n.map={...n.map,__proto__:qc.prototype}),e.push(n)}}if(r.nodes&&(r.nodes=s.nodes.map(i=>ct(i,e))),r.source){let{inputId:i,...n}=r.source;r.source=n,i!=null&&(r.source.input=e[i])}if(r.type==="root")return new sh(r);if(r.type==="decl")return new Qc(r);if(r.type==="rule")return new ih(r);if(r.type==="comment")return new eh(r);if(r.type==="atrule")return new th(r);throw new Error("Unknown node type: "+s.type)}var nh=ct;ct.default=ct;let oh=is,Xn=jt,ah=Kn,lh=Ae,$s=Xc,uh=Wt,ch=nh,Qn=Ss,hh=Dn,qn=Xt,eo=xs,fh=Cs,dh=Zt,ph=Rs,mh=Gn,to=As,ro=ot,gh=Vt;function N(...s){return s.length===1&&Array.isArray(s[0])&&(s=s[0]),new $s(s)}N.plugin=function(e,t){let r=!1;function i(...o){console&&console.warn&&!r&&(r=!0,console.warn(e+`: postcss.plugin was deprecated. Migration guide:
-https://evilmartians.com/chronicles/postcss-8-plugin-migration`),process.env.LANG&&process.env.LANG.startsWith("cn")&&console.warn(e+`: 里面 postcss.plugin 被弃用. 迁移指南:
-https://www.w3ctech.com/topic/2226`));let l=t(...o);return l.postcssPlugin=e,l.postcssVersion=new $s().version,l}let n;return Object.defineProperty(i,"postcss",{get(){return n||(n=i()),n}}),i.process=function(o,l,a){return N([i(a)]).process(o,l)},i},N.stringify=uh,N.parse=ph,N.fromJSON=ch,N.list=mh,N.comment=s=>new qn(s),N.atRule=s=>new eo(s),N.decl=s=>new Xn(s),N.rule=s=>new to(s),N.root=s=>new ro(s),N.document=s=>new Qn(s),N.CssSyntaxError=oh,N.Declaration=Xn,N.Container=lh,N.Processor=$s,N.Document=Qn,N.Comment=qn,N.Warning=hh,N.AtRule=eo,N.Result=fh,N.Input=dh,N.Rule=to,N.Root=ro,N.Node=gh,ah.registerPostcss(N);var yh=N;N.default=N;const U=Tu(yh);U.stringify,U.fromJSON,U.plugin,U.parse,U.list,U.document,U.comment,U.atRule,U.rule,U.decl,U.root,U.CssSyntaxError,U.Declaration,U.Container,U.Processor,U.Document,U.Comment,U.Warning,U.AtRule,U.Result,U.Input,U.Rule,U.Root,U.Node;class Ns{constructor(...e){re(this,"parentElement",null),re(this,"parentNode",null),re(this,"ownerDocument"),re(this,"firstChild",null),re(this,"lastChild",null),re(this,"previousSibling",null),re(this,"nextSibling",null),re(this,"ELEMENT_NODE",1),re(this,"TEXT_NODE",3),re(this,"nodeType"),re(this,"nodeName"),re(this,"RRNodeType")}get childNodes(){const e=[];let t=this.firstChild;for(;t;)e.push(t),t=t.nextSibling;return e}contains(e){if(e instanceof Ns){if(e.ownerDocument!==this.ownerDocument)return!1;if(e===this)return!0}else return!1;for(;e.parentNode;){if(e.parentNode===this)return!0;e=e.parentNode}return!1}appendChild(e){throw new Error("RRDomException: Failed to execute 'appendChild' on 'RRNode': This RRNode type does not support this method.")}insertBefore(e,t){throw new Error("RRDomException: Failed to execute 'insertBefore' on 'RRNode': This RRNode type does not support this method.")}removeChild(e){throw new Error("RRDomException: Failed to execute 'removeChild' on 'RRNode': This RRNode type does not support this method.")}toString(){return"RRNode"}}const so={Node:["childNodes","parentNode","parentElement","textContent","ownerDocument"],ShadowRoot:["host","styleSheets"],Element:["shadowRoot","querySelector","querySelectorAll"],MutationObserver:[]},io={Node:["contains","getRootNode"],ShadowRoot:["getSelection"],Element:[],MutationObserver:["constructor"]},lr={},wh=()=>!!globalThis.Zone;function ks(s){if(lr[s])return lr[s];const e=globalThis[s],t=e.prototype,r=s in so?so[s]:void 0,i=!!(r&&r.every(l=>{var a,u;return!!((u=(a=Object.getOwnPropertyDescriptor(t,l))==null?void 0:a.get)!=null&&u.toString().includes("[native code]"))})),n=s in io?io[s]:void 0,o=!!(n&&n.every(l=>{var a;return typeof t[l]=="function"&&((a=t[l])==null?void 0:a.toString().includes("[native code]"))}));if(i&&o&&!wh())return lr[s]=e.prototype,e.prototype;try{const l=document.createElement("iframe");document.body.appendChild(l);const a=l.contentWindow;if(!a)return e.prototype;const u=a[s].prototype;return document.body.removeChild(l),u?lr[s]=u:t}catch{return t}}const Ts={};function be(s,e,t){var r;const i=`${s}.${String(t)}`;if(Ts[i])return Ts[i].call(e);const n=ks(s),o=(r=Object.getOwnPropertyDescriptor(n,t))==null?void 0:r.get;return o?(Ts[i]=o,o.call(e)):e[t]}const Ps={};function no(s,e,t){const r=`${s}.${String(t)}`;if(Ps[r])return Ps[r].bind(e);const n=ks(s)[t];return typeof n!="function"?e[t]:(Ps[r]=n,n.bind(e))}function Sh(s){return be("Node",s,"ownerDocument")}function bh(s){return be("Node",s,"childNodes")}function vh(s){return be("Node",s,"parentNode")}function Ch(s){return be("Node",s,"parentElement")}function Ih(s){return be("Node",s,"textContent")}function xh(s,e){return no("Node",s,"contains")(e)}function Ah(s){return no("Node",s,"getRootNode")()}function Rh(s){return!s||!("host"in s)?null:be("ShadowRoot",s,"host")}function Oh(s){return s.styleSheets}function Eh(s){return!s||!("shadowRoot"in s)?null:be("Element",s,"shadowRoot")}function Mh(s,e){return be("Element",s,"querySelector")(e)}function $h(s,e){return be("Element",s,"querySelectorAll")(e)}function oo(){return ks("MutationObserver").constructor}function Re(s,e,t){try{if(!(e in s))return()=>{};const r=s[e],i=t(r);return typeof i=="function"&&(i.prototype=i.prototype||{},Object.defineProperties(i,{__rrweb_original__:{enumerable:!1,value:r}})),s[e]=i,()=>{s[e]=r}}catch{return()=>{}}}const I={ownerDocument:Sh,childNodes:bh,parentNode:vh,parentElement:Ch,textContent:Ih,contains:xh,getRootNode:Ah,host:Rh,styleSheets:Oh,shadowRoot:Eh,querySelector:Mh,querySelectorAll:$h,mutationObserver:oo,patch:Re};function X(s,e,t=document){const r={capture:!0,passive:!0};return t.addEventListener(s,e,r),()=>t.removeEventListener(s,e,r)}const ze=`Please stop import mirror directly. Instead of that,\r
+`)&&(n=`\r
+`),this.css+=n+"/*# sourceMappingURL="+a+" */"}applyPrevMaps(){for(let a of this.previous()){let n=this.toUrl(this.path(a.file)),m=a.root||r(a.file),h;this.mapOpts.sourcesContent===!1?(h=new e(a.text),h.sourcesContent&&(h.sourcesContent=null)):h=a.consumer(),this.map.applySourceMap(h,n,this.toUrl(this.path(m)))}}clearAnnotation(){if(this.mapOpts.annotation!==!1)if(this.root){let a;for(let n=this.root.nodes.length-1;n>=0;n--)a=this.root.nodes[n],a.type==="comment"&&a.text.indexOf("# sourceMappingURL=")===0&&this.root.removeChild(n)}else this.css&&(this.css=this.css.replace(/\n*?\/\*#[\S\s]*?\*\/$/gm,""))}generate(){if(this.clearAnnotation(),d&&l&&this.isMap())return this.generateMap();{let a="";return this.stringify(this.root,n=>{a+=n}),[a]}}generateMap(){if(this.root)this.generateString();else if(this.previous().length===1){let a=this.previous()[0].consumer();a.file=this.outputFile(),this.map=t.fromSourceMap(a,{ignoreInvalidMapping:!0})}else this.map=new t({file:this.outputFile(),ignoreInvalidMapping:!0}),this.map.addMapping({generated:{column:0,line:1},original:{column:0,line:1},source:this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>"});return this.isSourcesContent()&&this.setSourcesContent(),this.root&&this.previous().length>0&&this.applyPrevMaps(),this.isAnnotation()&&this.addAnnotation(),this.isInline()?[this.css]:[this.css,this.map]}generateString(){this.css="",this.map=new t({file:this.outputFile(),ignoreInvalidMapping:!0});let a=1,n=1,m="<no source>",h={generated:{column:0,line:0},original:{column:0,line:0},source:""},g,v;this.stringify(this.root,(b,S,x)=>{if(this.css+=b,S&&x!=="end"&&(h.generated.line=a,h.generated.column=n-1,S.source&&S.source.start?(h.source=this.sourcePath(S),h.original.line=S.source.start.line,h.original.column=S.source.start.column-1,this.map.addMapping(h)):(h.source=m,h.original.line=1,h.original.column=0,this.map.addMapping(h))),g=b.match(/\n/g),g?(a+=g.length,v=b.lastIndexOf(`
+`),n=b.length-v):n+=b.length,S&&x!=="start"){let w=S.parent||{raws:{}};(!(S.type==="decl"||S.type==="atrule"&&!S.nodes)||S!==w.last||w.raws.semicolon)&&(S.source&&S.source.end?(h.source=this.sourcePath(S),h.original.line=S.source.end.line,h.original.column=S.source.end.column-1,h.generated.line=a,h.generated.column=n-2,this.map.addMapping(h)):(h.source=m,h.original.line=1,h.original.column=0,h.generated.line=a,h.generated.column=n-1,this.map.addMapping(h)))}})}isAnnotation(){return this.isInline()?!0:typeof this.mapOpts.annotation<"u"?this.mapOpts.annotation:this.previous().length?this.previous().some(a=>a.annotation):!0}isInline(){if(typeof this.mapOpts.inline<"u")return this.mapOpts.inline;let a=this.mapOpts.annotation;return typeof a<"u"&&a!==!0?!1:this.previous().length?this.previous().some(n=>n.inline):!0}isMap(){return typeof this.opts.map<"u"?!!this.opts.map:this.previous().length>0}isSourcesContent(){return typeof this.mapOpts.sourcesContent<"u"?this.mapOpts.sourcesContent:this.previous().length?this.previous().some(a=>a.withContent()):!0}outputFile(){return this.opts.to?this.path(this.opts.to):this.opts.from?this.path(this.opts.from):"to.css"}path(a){if(this.mapOpts.absolute||a.charCodeAt(0)===60||/^\w+:\/\//.test(a))return a;let n=this.memoizedPaths.get(a);if(n)return n;let m=this.opts.to?r(this.opts.to):".";typeof this.mapOpts.annotation=="string"&&(m=r(s(m,this.mapOpts.annotation)));let h=o(m,a);return this.memoizedPaths.set(a,h),h}previous(){if(!this.previousMaps)if(this.previousMaps=[],this.root)this.root.walk(a=>{if(a.source&&a.source.input.map){let n=a.source.input.map;this.previousMaps.includes(n)||this.previousMaps.push(n)}});else{let a=new p(this.originalCSS,this.opts);a.map&&this.previousMaps.push(a.map)}return this.previousMaps}setSourcesContent(){let a={};if(this.root)this.root.walk(n=>{if(n.source){let m=n.source.input.from;if(m&&!a[m]){a[m]=!0;let h=this.usesFileUrls?this.toFileUrl(m):this.toUrl(this.path(m));this.map.setSourceContent(h,n.source.input.css)}}});else if(this.css){let n=this.opts.from?this.toUrl(this.path(this.opts.from)):"<no source>";this.map.setSourceContent(n,this.css)}}sourcePath(a){return this.mapOpts.from?this.toUrl(this.mapOpts.from):this.usesFileUrls?this.toFileUrl(a.source.input.from):this.toUrl(this.path(a.source.input.from))}toBase64(a){return Buffer?Buffer.from(a).toString("base64"):window.btoa(unescape(encodeURIComponent(a)))}toFileUrl(a){let n=this.memoizedFileURLs.get(a);if(n)return n;if(c){let m=c(a).toString();return this.memoizedFileURLs.set(a,m),m}else throw new Error("`map.absolute` option is not available in this PostCSS build")}toUrl(a){let n=this.memoizedURLs.get(a);if(n)return n;u==="\\"&&(a=a.replace(/\\/g,"/"));let m=encodeURI(a).replace(/[#?]/g,encodeURIComponent);return this.memoizedURLs.set(a,m),m}}return Rr=i,Rr}var Or,xi;function yt(){if(xi)return Or;xi=1;let e=pt();class t extends e{constructor(o){super(o),this.type="comment"}}return Or=t,t.default=t,Or}var Mr,Ri;function Ae(){if(Ri)return Mr;Ri=1;let{isClean:e,my:t}=gr(),r=mt(),o=yt(),s=pt(),u,c,p,l;function d(a){return a.map(n=>(n.nodes&&(n.nodes=d(n.nodes)),delete n.source,n))}function i(a){if(a[e]=!1,a.proxyOf.nodes)for(let n of a.proxyOf.nodes)i(n)}class f extends s{append(...n){for(let m of n){let h=this.normalize(m,this.last);for(let g of h)this.proxyOf.nodes.push(g)}return this.markDirty(),this}cleanRaws(n){if(super.cleanRaws(n),this.nodes)for(let m of this.nodes)m.cleanRaws(n)}each(n){if(!this.proxyOf.nodes)return;let m=this.getIterator(),h,g;for(;this.indexes[m]<this.proxyOf.nodes.length&&(h=this.indexes[m],g=n(this.proxyOf.nodes[h],h),g!==!1);)this.indexes[m]+=1;return delete this.indexes[m],g}every(n){return this.nodes.every(n)}getIterator(){this.lastEach||(this.lastEach=0),this.indexes||(this.indexes={}),this.lastEach+=1;let n=this.lastEach;return this.indexes[n]=0,n}getProxyProcessor(){return{get(n,m){return m==="proxyOf"?n:n[m]?m==="each"||typeof m=="string"&&m.startsWith("walk")?(...h)=>n[m](...h.map(g=>typeof g=="function"?(v,b)=>g(v.toProxy(),b):g)):m==="every"||m==="some"?h=>n[m]((g,...v)=>h(g.toProxy(),...v)):m==="root"?()=>n.root().toProxy():m==="nodes"?n.nodes.map(h=>h.toProxy()):m==="first"||m==="last"?n[m].toProxy():n[m]:n[m]},set(n,m,h){return n[m]===h||(n[m]=h,(m==="name"||m==="params"||m==="selector")&&n.markDirty()),!0}}}index(n){return typeof n=="number"?n:(n.proxyOf&&(n=n.proxyOf),this.proxyOf.nodes.indexOf(n))}insertAfter(n,m){let h=this.index(n),g=this.normalize(m,this.proxyOf.nodes[h]).reverse();h=this.index(n);for(let b of g)this.proxyOf.nodes.splice(h+1,0,b);let v;for(let b in this.indexes)v=this.indexes[b],h<v&&(this.indexes[b]=v+g.length);return this.markDirty(),this}insertBefore(n,m){let h=this.index(n),g=h===0?"prepend":!1,v=this.normalize(m,this.proxyOf.nodes[h],g).reverse();h=this.index(n);for(let S of v)this.proxyOf.nodes.splice(h,0,S);let b;for(let S in this.indexes)b=this.indexes[S],h<=b&&(this.indexes[S]=b+v.length);return this.markDirty(),this}normalize(n,m){if(typeof n=="string")n=d(u(n).nodes);else if(typeof n>"u")n=[];else if(Array.isArray(n)){n=n.slice(0);for(let g of n)g.parent&&g.parent.removeChild(g,"ignore")}else if(n.type==="root"&&this.type!=="document"){n=n.nodes.slice(0);for(let g of n)g.parent&&g.parent.removeChild(g,"ignore")}else if(n.type)n=[n];else if(n.prop){if(typeof n.value>"u")throw new Error("Value field is missed in node creation");typeof n.value!="string"&&(n.value=String(n.value)),n=[new r(n)]}else if(n.selector)n=[new c(n)];else if(n.name)n=[new p(n)];else if(n.text)n=[new o(n)];else throw new Error("Unknown node type in node creation");return n.map(g=>(g[t]||f.rebuild(g),g=g.proxyOf,g.parent&&g.parent.removeChild(g),g[e]&&i(g),typeof g.raws.before>"u"&&m&&typeof m.raws.before<"u"&&(g.raws.before=m.raws.before.replace(/\S/g,"")),g.parent=this.proxyOf,g))}prepend(...n){n=n.reverse();for(let m of n){let h=this.normalize(m,this.first,"prepend").reverse();for(let g of h)this.proxyOf.nodes.unshift(g);for(let g in this.indexes)this.indexes[g]=this.indexes[g]+h.length}return this.markDirty(),this}push(n){return n.parent=this,this.proxyOf.nodes.push(n),this}removeAll(){for(let n of this.proxyOf.nodes)n.parent=void 0;return this.proxyOf.nodes=[],this.markDirty(),this}removeChild(n){n=this.index(n),this.proxyOf.nodes[n].parent=void 0,this.proxyOf.nodes.splice(n,1);let m;for(let h in this.indexes)m=this.indexes[h],m>=n&&(this.indexes[h]=m-1);return this.markDirty(),this}replaceValues(n,m,h){return h||(h=m,m={}),this.walkDecls(g=>{m.props&&!m.props.includes(g.prop)||m.fast&&!g.value.includes(m.fast)||(g.value=g.value.replace(n,h))}),this.markDirty(),this}some(n){return this.nodes.some(n)}walk(n){return this.each((m,h)=>{let g;try{g=n(m,h)}catch(v){throw m.addToError(v)}return g!==!1&&m.walk&&(g=m.walk(n)),g})}walkAtRules(n,m){return m?n instanceof RegExp?this.walk((h,g)=>{if(h.type==="atrule"&&n.test(h.name))return m(h,g)}):this.walk((h,g)=>{if(h.type==="atrule"&&h.name===n)return m(h,g)}):(m=n,this.walk((h,g)=>{if(h.type==="atrule")return m(h,g)}))}walkComments(n){return this.walk((m,h)=>{if(m.type==="comment")return n(m,h)})}walkDecls(n,m){return m?n instanceof RegExp?this.walk((h,g)=>{if(h.type==="decl"&&n.test(h.prop))return m(h,g)}):this.walk((h,g)=>{if(h.type==="decl"&&h.prop===n)return m(h,g)}):(m=n,this.walk((h,g)=>{if(h.type==="decl")return m(h,g)}))}walkRules(n,m){return m?n instanceof RegExp?this.walk((h,g)=>{if(h.type==="rule"&&n.test(h.selector))return m(h,g)}):this.walk((h,g)=>{if(h.type==="rule"&&h.selector===n)return m(h,g)}):(m=n,this.walk((h,g)=>{if(h.type==="rule")return m(h,g)}))}get first(){if(this.proxyOf.nodes)return this.proxyOf.nodes[0]}get last(){if(this.proxyOf.nodes)return this.proxyOf.nodes[this.proxyOf.nodes.length-1]}}return f.registerParse=a=>{u=a},f.registerRule=a=>{c=a},f.registerAtRule=a=>{p=a},f.registerRoot=a=>{l=a},Mr=f,f.default=f,f.rebuild=a=>{a.type==="atrule"?Object.setPrototypeOf(a,p.prototype):a.type==="rule"?Object.setPrototypeOf(a,c.prototype):a.type==="decl"?Object.setPrototypeOf(a,r.prototype):a.type==="comment"?Object.setPrototypeOf(a,o.prototype):a.type==="root"&&Object.setPrototypeOf(a,l.prototype),a[t]=!0,a.nodes&&a.nodes.forEach(n=>{f.rebuild(n)})},Mr}var Er,Oi;function Ir(){if(Oi)return Er;Oi=1;let e=Ae(),t,r;class o extends e{constructor(u){super({type:"document",...u}),this.nodes||(this.nodes=[])}toResult(u={}){return new t(new r,this,u).stringify()}}return o.registerLazyResult=s=>{t=s},o.registerProcessor=s=>{r=s},Er=o,o.default=o,Er}var Ar,Mi;function Ei(){if(Mi)return Ar;Mi=1;let e={};return Ar=function(r){e[r]||(e[r]=!0,typeof console<"u"&&console.warn&&console.warn(r))},Ar}var Nr,Ii;function Ai(){if(Ii)return Nr;Ii=1;class e{constructor(r,o={}){if(this.type="warning",this.text=r,o.node&&o.node.source){let s=o.node.rangeBy(o);this.line=s.start.line,this.column=s.start.column,this.endLine=s.end.line,this.endColumn=s.end.column}for(let s in o)this[s]=o[s]}toString(){return this.node?this.node.error(this.text,{index:this.index,plugin:this.plugin,word:this.word}).message:this.plugin?this.plugin+": "+this.text:this.text}}return Nr=e,e.default=e,Nr}var kr,Ni;function Pr(){if(Ni)return kr;Ni=1;let e=Ai();class t{constructor(o,s,u){this.processor=o,this.messages=[],this.root=s,this.opts=u,this.css=void 0,this.map=void 0}toString(){return this.css}warn(o,s={}){s.plugin||this.lastPlugin&&this.lastPlugin.postcssPlugin&&(s.plugin=this.lastPlugin.postcssPlugin);let u=new e(o,s);return this.messages.push(u),u}warnings(){return this.messages.filter(o=>o.type==="warning")}get content(){return this.css}}return kr=t,t.default=t,kr}var Tr,ki;function Vo(){if(ki)return Tr;ki=1;const e=39,t=34,r=92,o=47,s=10,u=32,c=12,p=9,l=13,d=91,i=93,f=40,a=41,n=123,m=125,h=59,g=42,v=58,b=64,S=/[\t\n\f\r "#'()/;[\\\]{}]/g,x=/[\t\n\f\r !"#'():;@[\\\]{}]|\/(?=\*)/g,w=/.[\r\n"'(/\\]/,y=/[\da-f]/i;return Tr=function(O,I={}){let M=O.css.valueOf(),P=I.ignoreErrors,N,R,ye,ue,B,U,Y,te,Q,z,Me=M.length,E=0,ve=[],de=[];function Qe(){return E}function ie(T){throw O.error("Unclosed "+T,E)}function we(){return de.length===0&&E>=Me}function Te(T){if(de.length)return de.pop();if(E>=Me)return;let K=T?T.ignoreUnclosed:!1;switch(N=M.charCodeAt(E),N){case s:case u:case p:case l:case c:{R=E;do R+=1,N=M.charCodeAt(R);while(N===u||N===s||N===p||N===l||N===c);z=["space",M.slice(E,R)],E=R-1;break}case d:case i:case n:case m:case v:case h:case a:{let W=String.fromCharCode(N);z=[W,W,E];break}case f:{if(te=ve.length?ve.pop()[1]:"",Q=M.charCodeAt(E+1),te==="url"&&Q!==e&&Q!==t&&Q!==u&&Q!==s&&Q!==p&&Q!==c&&Q!==l){R=E;do{if(U=!1,R=M.indexOf(")",R+1),R===-1)if(P||K){R=E;break}else ie("bracket");for(Y=R;M.charCodeAt(Y-1)===r;)Y-=1,U=!U}while(U);z=["brackets",M.slice(E,R+1),E,R],E=R}else R=M.indexOf(")",E+1),ue=M.slice(E,R+1),R===-1||w.test(ue)?z=["(","(",E]:(z=["brackets",ue,E,R],E=R);break}case e:case t:{ye=N===e?"'":'"',R=E;do{if(U=!1,R=M.indexOf(ye,R+1),R===-1)if(P||K){R=E+1;break}else ie("string");for(Y=R;M.charCodeAt(Y-1)===r;)Y-=1,U=!U}while(U);z=["string",M.slice(E,R+1),E,R],E=R;break}case b:{S.lastIndex=E+1,S.test(M),S.lastIndex===0?R=M.length-1:R=S.lastIndex-2,z=["at-word",M.slice(E,R+1),E,R],E=R;break}case r:{for(R=E,B=!0;M.charCodeAt(R+1)===r;)R+=1,B=!B;if(N=M.charCodeAt(R+1),B&&N!==o&&N!==u&&N!==s&&N!==p&&N!==l&&N!==c&&(R+=1,y.test(M.charAt(R)))){for(;y.test(M.charAt(R+1));)R+=1;M.charCodeAt(R+1)===u&&(R+=1)}z=["word",M.slice(E,R+1),E,R],E=R;break}default:{N===o&&M.charCodeAt(E+1)===g?(R=M.indexOf("*/",E+2)+1,R===0&&(P||K?R=M.length:ie("comment")),z=["comment",M.slice(E,R+1),E,R],E=R):(x.lastIndex=E+1,x.test(M),x.lastIndex===0?R=M.length-1:R=x.lastIndex-2,z=["word",M.slice(E,R+1),E,R],ve.push(z),E=R);break}}return E++,z}function Ee(T){de.push(T)}return{back:Ee,endOfFile:we,nextToken:Te,position:Qe}},Tr}var Dr,Pi;function Lr(){if(Pi)return Dr;Pi=1;let e=Ae();class t extends e{constructor(o){super(o),this.type="atrule"}append(...o){return this.proxyOf.nodes||(this.nodes=[]),super.append(...o)}prepend(...o){return this.proxyOf.nodes||(this.nodes=[]),super.prepend(...o)}}return Dr=t,t.default=t,e.registerAtRule(t),Dr}var _r,Ti;function Ge(){if(Ti)return _r;Ti=1;let e=Ae(),t,r;class o extends e{constructor(u){super(u),this.type="root",this.nodes||(this.nodes=[])}normalize(u,c,p){let l=super.normalize(u);if(c){if(p==="prepend")this.nodes.length>1?c.raws.before=this.nodes[1].raws.before:delete c.raws.before;else if(this.first!==c)for(let d of l)d.raws.before=c.raws.before}return l}removeChild(u,c){let p=this.index(u);return!c&&p===0&&this.nodes.length>1&&(this.nodes[1].raws.before=this.nodes[p].raws.before),super.removeChild(u)}toResult(u={}){return new t(new r,this,u).stringify()}}return o.registerLazyResult=s=>{t=s},o.registerProcessor=s=>{r=s},_r=o,o.default=o,e.registerRoot(o),_r}var $r,Di;function Li(){if(Di)return $r;Di=1;let e={comma(t){return e.split(t,[","],!0)},space(t){let r=[" ",`
+`," "];return e.split(t,r)},split(t,r,o){let s=[],u="",c=!1,p=0,l=!1,d="",i=!1;for(let f of t)i?i=!1:f==="\\"?i=!0:l?f===d&&(l=!1):f==='"'||f==="'"?(l=!0,d=f):f==="("?p+=1:f===")"?p>0&&(p-=1):p===0&&r.includes(f)&&(c=!0),c?(u!==""&&s.push(u.trim()),u="",c=!1):u+=f;return(o||u!=="")&&s.push(u.trim()),s}};return $r=e,e.default=e,$r}var Fr,_i;function Ur(){if(_i)return Fr;_i=1;let e=Ae(),t=Li();class r extends e{constructor(s){super(s),this.type="rule",this.nodes||(this.nodes=[])}get selectors(){return t.comma(this.selector)}set selectors(s){let u=this.selector?this.selector.match(/,\s*/):null,c=u?u[0]:","+this.raw("between","beforeOpen");this.selector=s.join(c)}}return Fr=r,r.default=r,e.registerRule(r),Fr}var Br,$i;function Ho(){if($i)return Br;$i=1;let e=mt(),t=Vo(),r=yt(),o=Lr(),s=Ge(),u=Ur();const c={empty:!0,space:!0};function p(d){for(let i=d.length-1;i>=0;i--){let f=d[i],a=f[3]||f[2];if(a)return a}}class l{constructor(i){this.input=i,this.root=new s,this.current=this.root,this.spaces="",this.semicolon=!1,this.createTokenizer(),this.root.source={input:i,start:{column:1,line:1,offset:0}}}atrule(i){let f=new o;f.name=i[1].slice(1),f.name===""&&this.unnamedAtrule(f,i),this.init(f,i[2]);let a,n,m,h=!1,g=!1,v=[],b=[];for(;!this.tokenizer.endOfFile();){if(i=this.tokenizer.nextToken(),a=i[0],a==="("||a==="["?b.push(a==="("?")":"]"):a==="{"&&b.length>0?b.push("}"):a===b[b.length-1]&&b.pop(),b.length===0)if(a===";"){f.source.end=this.getPosition(i[2]),f.source.end.offset++,this.semicolon=!0;break}else if(a==="{"){g=!0;break}else if(a==="}"){if(v.length>0){for(m=v.length-1,n=v[m];n&&n[0]==="space";)n=v[--m];n&&(f.source.end=this.getPosition(n[3]||n[2]),f.source.end.offset++)}this.end(i);break}else v.push(i);else v.push(i);if(this.tokenizer.endOfFile()){h=!0;break}}f.raws.between=this.spacesAndCommentsFromEnd(v),v.length?(f.raws.afterName=this.spacesAndCommentsFromStart(v),this.raw(f,"params",v),h&&(i=v[v.length-1],f.source.end=this.getPosition(i[3]||i[2]),f.source.end.offset++,this.spaces=f.raws.between,f.raws.between="")):(f.raws.afterName="",f.params=""),g&&(f.nodes=[],this.current=f)}checkMissedSemicolon(i){let f=this.colon(i);if(f===!1)return;let a=0,n;for(let m=f-1;m>=0&&(n=i[m],!(n[0]!=="space"&&(a+=1,a===2)));m--);throw this.input.error("Missed semicolon",n[0]==="word"?n[3]+1:n[2])}colon(i){let f=0,a,n,m;for(let[h,g]of i.entries()){if(a=g,n=a[0],n==="("&&(f+=1),n===")"&&(f-=1),f===0&&n===":")if(!m)this.doubleColon(a);else{if(m[0]==="word"&&m[1]==="progid")continue;return h}m=a}return!1}comment(i){let f=new r;this.init(f,i[2]),f.source.end=this.getPosition(i[3]||i[2]),f.source.end.offset++;let a=i[1].slice(2,-2);if(/^\s*$/.test(a))f.text="",f.raws.left=a,f.raws.right="";else{let n=a.match(/^(\s*)([^]*\S)(\s*)$/);f.text=n[2],f.raws.left=n[1],f.raws.right=n[3]}}createTokenizer(){this.tokenizer=t(this.input)}decl(i,f){let a=new e;this.init(a,i[0][2]);let n=i[i.length-1];for(n[0]===";"&&(this.semicolon=!0,i.pop()),a.source.end=this.getPosition(n[3]||n[2]||p(i)),a.source.end.offset++;i[0][0]!=="word";)i.length===1&&this.unknownWord(i),a.raws.before+=i.shift()[1];for(a.source.start=this.getPosition(i[0][2]),a.prop="";i.length;){let b=i[0][0];if(b===":"||b==="space"||b==="comment")break;a.prop+=i.shift()[1]}a.raws.between="";let m;for(;i.length;)if(m=i.shift(),m[0]===":"){a.raws.between+=m[1];break}else m[0]==="word"&&/\w/.test(m[1])&&this.unknownWord([m]),a.raws.between+=m[1];(a.prop[0]==="_"||a.prop[0]==="*")&&(a.raws.before+=a.prop[0],a.prop=a.prop.slice(1));let h=[],g;for(;i.length&&(g=i[0][0],!(g!=="space"&&g!=="comment"));)h.push(i.shift());this.precheckMissedSemicolon(i);for(let b=i.length-1;b>=0;b--){if(m=i[b],m[1].toLowerCase()==="!important"){a.important=!0;let S=this.stringFrom(i,b);S=this.spacesFromEnd(i)+S,S!==" !important"&&(a.raws.important=S);break}else if(m[1].toLowerCase()==="important"){let S=i.slice(0),x="";for(let w=b;w>0;w--){let y=S[w][0];if(x.trim().indexOf("!")===0&&y!=="space")break;x=S.pop()[1]+x}x.trim().indexOf("!")===0&&(a.important=!0,a.raws.important=x,i=S)}if(m[0]!=="space"&&m[0]!=="comment")break}i.some(b=>b[0]!=="space"&&b[0]!=="comment")&&(a.raws.between+=h.map(b=>b[1]).join(""),h=[]),this.raw(a,"value",h.concat(i),f),a.value.includes(":")&&!f&&this.checkMissedSemicolon(i)}doubleColon(i){throw this.input.error("Double colon",{offset:i[2]},{offset:i[2]+i[1].length})}emptyRule(i){let f=new u;this.init(f,i[2]),f.selector="",f.raws.between="",this.current=f}end(i){this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.semicolon=!1,this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.spaces="",this.current.parent?(this.current.source.end=this.getPosition(i[2]),this.current.source.end.offset++,this.current=this.current.parent):this.unexpectedClose(i)}endFile(){this.current.parent&&this.unclosedBlock(),this.current.nodes&&this.current.nodes.length&&(this.current.raws.semicolon=this.semicolon),this.current.raws.after=(this.current.raws.after||"")+this.spaces,this.root.source.end=this.getPosition(this.tokenizer.position())}freeSemicolon(i){if(this.spaces+=i[1],this.current.nodes){let f=this.current.nodes[this.current.nodes.length-1];f&&f.type==="rule"&&!f.raws.ownSemicolon&&(f.raws.ownSemicolon=this.spaces,this.spaces="")}}getPosition(i){let f=this.input.fromOffset(i);return{column:f.col,line:f.line,offset:i}}init(i,f){this.current.push(i),i.source={input:this.input,start:this.getPosition(f)},i.raws.before=this.spaces,this.spaces="",i.type!=="comment"&&(this.semicolon=!1)}other(i){let f=!1,a=null,n=!1,m=null,h=[],g=i[1].startsWith("--"),v=[],b=i;for(;b;){if(a=b[0],v.push(b),a==="("||a==="[")m||(m=b),h.push(a==="("?")":"]");else if(g&&n&&a==="{")m||(m=b),h.push("}");else if(h.length===0)if(a===";")if(n){this.decl(v,g);return}else break;else if(a==="{"){this.rule(v);return}else if(a==="}"){this.tokenizer.back(v.pop()),f=!0;break}else a===":"&&(n=!0);else a===h[h.length-1]&&(h.pop(),h.length===0&&(m=null));b=this.tokenizer.nextToken()}if(this.tokenizer.endOfFile()&&(f=!0),h.length>0&&this.unclosedBracket(m),f&&n){if(!g)for(;v.length&&(b=v[v.length-1][0],!(b!=="space"&&b!=="comment"));)this.tokenizer.back(v.pop());this.decl(v,g)}else this.unknownWord(v)}parse(){let i;for(;!this.tokenizer.endOfFile();)switch(i=this.tokenizer.nextToken(),i[0]){case"space":this.spaces+=i[1];break;case";":this.freeSemicolon(i);break;case"}":this.end(i);break;case"comment":this.comment(i);break;case"at-word":this.atrule(i);break;case"{":this.emptyRule(i);break;default:this.other(i);break}this.endFile()}precheckMissedSemicolon(){}raw(i,f,a,n){let m,h,g=a.length,v="",b=!0,S,x;for(let w=0;w<g;w+=1)m=a[w],h=m[0],h==="space"&&w===g-1&&!n?b=!1:h==="comment"?(x=a[w-1]?a[w-1][0]:"empty",S=a[w+1]?a[w+1][0]:"empty",!c[x]&&!c[S]?v.slice(-1)===","?b=!1:v+=m[1]:b=!1):v+=m[1];if(!b){let w=a.reduce((y,C)=>y+C[1],"");i.raws[f]={raw:w,value:v}}i[f]=v}rule(i){i.pop();let f=new u;this.init(f,i[0][2]),f.raws.between=this.spacesAndCommentsFromEnd(i),this.raw(f,"selector",i),this.current=f}spacesAndCommentsFromEnd(i){let f,a="";for(;i.length&&(f=i[i.length-1][0],!(f!=="space"&&f!=="comment"));)a=i.pop()[1]+a;return a}spacesAndCommentsFromStart(i){let f,a="";for(;i.length&&(f=i[0][0],!(f!=="space"&&f!=="comment"));)a+=i.shift()[1];return a}spacesFromEnd(i){let f,a="";for(;i.length&&(f=i[i.length-1][0],f==="space");)a=i.pop()[1]+a;return a}stringFrom(i,f){let a="";for(let n=f;n<i.length;n++)a+=i[n][1];return i.splice(f,i.length-f),a}unclosedBlock(){let i=this.current.source.start;throw this.input.error("Unclosed block",i.line,i.column)}unclosedBracket(i){throw this.input.error("Unclosed bracket",{offset:i[2]},{offset:i[2]+1})}unexpectedClose(i){throw this.input.error("Unexpected }",{offset:i[2]},{offset:i[2]+1})}unknownWord(i){throw this.input.error("Unknown word",{offset:i[0][2]},{offset:i[0][2]+i[0][1].length})}unnamedAtrule(i,f){throw this.input.error("At-rule without name",{offset:f[2]},{offset:f[2]+f[1].length})}}return Br=l,Br}var zr,Fi;function Wr(){if(Fi)return zr;Fi=1;let e=Ae(),t=Ho(),r=gt();function o(s,u){let c=new r(s,u),p=new t(c);try{p.parse()}catch(l){throw process.env.NODE_ENV!=="production"&&l.name==="CssSyntaxError"&&u&&u.from&&(/\.scss$/i.test(u.from)?l.message+=`
+You tried to parse SCSS with the standard CSS parser; try again with the postcss-scss parser`:/\.sass/i.test(u.from)?l.message+=`
+You tried to parse Sass with the standard CSS parser; try again with the postcss-sass parser`:/\.less$/i.test(u.from)&&(l.message+=`
+You tried to parse Less with the standard CSS parser; try again with the postcss-less parser`)),l}return p.root}return zr=o,o.default=o,e.registerParse(o),zr}var qr,Ui;function Bi(){if(Ui)return qr;Ui=1;let{isClean:e,my:t}=gr(),r=Ci(),o=dt(),s=Ae(),u=Ir(),c=Ei(),p=Pr(),l=Wr(),d=Ge();const i={atrule:"AtRule",comment:"Comment",decl:"Declaration",document:"Document",root:"Root",rule:"Rule"},f={AtRule:!0,AtRuleExit:!0,Comment:!0,CommentExit:!0,Declaration:!0,DeclarationExit:!0,Document:!0,DocumentExit:!0,Once:!0,OnceExit:!0,postcssPlugin:!0,prepare:!0,Root:!0,RootExit:!0,Rule:!0,RuleExit:!0},a={Once:!0,postcssPlugin:!0,prepare:!0},n=0;function m(x){return typeof x=="object"&&typeof x.then=="function"}function h(x){let w=!1,y=i[x.type];return x.type==="decl"?w=x.prop.toLowerCase():x.type==="atrule"&&(w=x.name.toLowerCase()),w&&x.append?[y,y+"-"+w,n,y+"Exit",y+"Exit-"+w]:w?[y,y+"-"+w,y+"Exit",y+"Exit-"+w]:x.append?[y,n,y+"Exit"]:[y,y+"Exit"]}function g(x){let w;return x.type==="document"?w=["Document",n,"DocumentExit"]:x.type==="root"?w=["Root",n,"RootExit"]:w=h(x),{eventIndex:0,events:w,iterator:0,node:x,visitorIndex:0,visitors:[]}}function v(x){return x[e]=!1,x.nodes&&x.nodes.forEach(w=>v(w)),x}let b={};class S{constructor(w,y,C){this.stringified=!1,this.processed=!1;let O;if(typeof y=="object"&&y!==null&&(y.type==="root"||y.type==="document"))O=v(y);else if(y instanceof S||y instanceof p)O=v(y.root),y.map&&(typeof C.map>"u"&&(C.map={}),C.map.inline||(C.map.inline=!1),C.map.prev=y.map);else{let I=l;C.syntax&&(I=C.syntax.parse),C.parser&&(I=C.parser),I.parse&&(I=I.parse);try{O=I(y,C)}catch(M){this.processed=!0,this.error=M}O&&!O[t]&&s.rebuild(O)}this.result=new p(w,O,C),this.helpers={...b,postcss:b,result:this.result},this.plugins=this.processor.plugins.map(I=>typeof I=="object"&&I.prepare?{...I,...I.prepare(this.result)}:I)}async(){return this.error?Promise.reject(this.error):this.processed?Promise.resolve(this.result):(this.processing||(this.processing=this.runAsync()),this.processing)}catch(w){return this.async().catch(w)}finally(w){return this.async().then(w,w)}getAsyncError(){throw new Error("Use process(css).then(cb) to work with async plugins")}handleError(w,y){let C=this.result.lastPlugin;try{if(y&&y.addToError(w),this.error=w,w.name==="CssSyntaxError"&&!w.plugin)w.plugin=C.postcssPlugin,w.setMessage();else if(C.postcssVersion&&process.env.NODE_ENV!=="production"){let O=C.postcssPlugin,I=C.postcssVersion,M=this.result.processor.version,P=I.split("."),N=M.split(".");(P[0]!==N[0]||parseInt(P[1])>parseInt(N[1]))&&console.error("Unknown error from PostCSS plugin. Your current PostCSS version is "+M+", but "+O+" uses "+I+". Perhaps this is the source of the error below.")}}catch(O){console&&console.error&&console.error(O)}return w}prepareVisitors(){this.listeners={};let w=(y,C,O)=>{this.listeners[C]||(this.listeners[C]=[]),this.listeners[C].push([y,O])};for(let y of this.plugins)if(typeof y=="object")for(let C in y){if(!f[C]&&/^[A-Z]/.test(C))throw new Error(`Unknown event ${C} in ${y.postcssPlugin}. Try to update PostCSS (${this.processor.version} now).`);if(!a[C])if(typeof y[C]=="object")for(let O in y[C])O==="*"?w(y,C,y[C][O]):w(y,C+"-"+O.toLowerCase(),y[C][O]);else typeof y[C]=="function"&&w(y,C,y[C])}this.hasListener=Object.keys(this.listeners).length>0}async runAsync(){this.plugin=0;for(let w=0;w<this.plugins.length;w++){let y=this.plugins[w],C=this.runOnRoot(y);if(m(C))try{await C}catch(O){throw this.handleError(O)}}if(this.prepareVisitors(),this.hasListener){let w=this.result.root;for(;!w[e];){w[e]=!0;let y=[g(w)];for(;y.length>0;){let C=this.visitTick(y);if(m(C))try{await C}catch(O){let I=y[y.length-1].node;throw this.handleError(O,I)}}}if(this.listeners.OnceExit)for(let[y,C]of this.listeners.OnceExit){this.result.lastPlugin=y;try{if(w.type==="document"){let O=w.nodes.map(I=>C(I,this.helpers));await Promise.all(O)}else await C(w,this.helpers)}catch(O){throw this.handleError(O)}}}return this.processed=!0,this.stringify()}runOnRoot(w){this.result.lastPlugin=w;try{if(typeof w=="object"&&w.Once){if(this.result.root.type==="document"){let y=this.result.root.nodes.map(C=>w.Once(C,this.helpers));return m(y[0])?Promise.all(y):y}return w.Once(this.result.root,this.helpers)}else if(typeof w=="function")return w(this.result.root,this.result)}catch(y){throw this.handleError(y)}}stringify(){if(this.error)throw this.error;if(this.stringified)return this.result;this.stringified=!0,this.sync();let w=this.result.opts,y=o;w.syntax&&(y=w.syntax.stringify),w.stringifier&&(y=w.stringifier),y.stringify&&(y=y.stringify);let O=new r(y,this.result.root,this.result.opts).generate();return this.result.css=O[0],this.result.map=O[1],this.result}sync(){if(this.error)throw this.error;if(this.processed)return this.result;if(this.processed=!0,this.processing)throw this.getAsyncError();for(let w of this.plugins){let y=this.runOnRoot(w);if(m(y))throw this.getAsyncError()}if(this.prepareVisitors(),this.hasListener){let w=this.result.root;for(;!w[e];)w[e]=!0,this.walkSync(w);if(this.listeners.OnceExit)if(w.type==="document")for(let y of w.nodes)this.visitSync(this.listeners.OnceExit,y);else this.visitSync(this.listeners.OnceExit,w)}return this.result}then(w,y){return process.env.NODE_ENV!=="production"&&("from"in this.opts||c("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(w,y)}toString(){return this.css}visitSync(w,y){for(let[C,O]of w){this.result.lastPlugin=C;let I;try{I=O(y,this.helpers)}catch(M){throw this.handleError(M,y.proxyOf)}if(y.type!=="root"&&y.type!=="document"&&!y.parent)return!0;if(m(I))throw this.getAsyncError()}}visitTick(w){let y=w[w.length-1],{node:C,visitors:O}=y;if(C.type!=="root"&&C.type!=="document"&&!C.parent){w.pop();return}if(O.length>0&&y.visitorIndex<O.length){let[M,P]=O[y.visitorIndex];y.visitorIndex+=1,y.visitorIndex===O.length&&(y.visitors=[],y.visitorIndex=0),this.result.lastPlugin=M;try{return P(C.toProxy(),this.helpers)}catch(N){throw this.handleError(N,C)}}if(y.iterator!==0){let M=y.iterator,P;for(;P=C.nodes[C.indexes[M]];)if(C.indexes[M]+=1,!P[e]){P[e]=!0,w.push(g(P));return}y.iterator=0,delete C.indexes[M]}let I=y.events;for(;y.eventIndex<I.length;){let M=I[y.eventIndex];if(y.eventIndex+=1,M===n){C.nodes&&C.nodes.length&&(C[e]=!0,y.iterator=C.getIterator());return}else if(this.listeners[M]){y.visitors=this.listeners[M];return}}w.pop()}walkSync(w){w[e]=!0;let y=h(w);for(let C of y)if(C===n)w.nodes&&w.each(O=>{O[e]||this.walkSync(O)});else{let O=this.listeners[C];if(O&&this.visitSync(O,w.toProxy()))return}}warnings(){return this.sync().warnings()}get content(){return this.stringify().content}get css(){return this.stringify().css}get map(){return this.stringify().map}get messages(){return this.sync().messages}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){return this.sync().root}get[Symbol.toStringTag](){return"LazyResult"}}return S.registerPostcss=x=>{b=x},qr=S,S.default=S,d.registerLazyResult(S),u.registerLazyResult(S),qr}var jr,zi;function Jo(){if(zi)return jr;zi=1;let e=Ci(),t=dt(),r=Ei(),o=Wr();const s=Pr();class u{constructor(p,l,d){l=l.toString(),this.stringified=!1,this._processor=p,this._css=l,this._opts=d,this._map=void 0;let i,f=t;this.result=new s(this._processor,i,this._opts),this.result.css=l;let a=this;Object.defineProperty(this.result,"root",{get(){return a.root}});let n=new e(f,i,this._opts,l);if(n.isMap()){let[m,h]=n.generate();m&&(this.result.css=m),h&&(this.result.map=h)}else n.clearAnnotation(),this.result.css=n.css}async(){return this.error?Promise.reject(this.error):Promise.resolve(this.result)}catch(p){return this.async().catch(p)}finally(p){return this.async().then(p,p)}sync(){if(this.error)throw this.error;return this.result}then(p,l){return process.env.NODE_ENV!=="production"&&("from"in this._opts||r("Without `from` option PostCSS could generate wrong source map and will not find Browserslist config. Set it to CSS file path or to `undefined` to prevent this warning.")),this.async().then(p,l)}toString(){return this._css}warnings(){return[]}get content(){return this.result.css}get css(){return this.result.css}get map(){return this.result.map}get messages(){return[]}get opts(){return this.result.opts}get processor(){return this.result.processor}get root(){if(this._root)return this._root;let p,l=o;try{p=l(this._css,this._opts)}catch(d){this.error=d}if(this.error)throw this.error;return this._root=p,p}get[Symbol.toStringTag](){return"NoWorkResult"}}return jr=u,u.default=u,jr}var Gr,Wi;function Yo(){if(Wi)return Gr;Wi=1;let e=Jo(),t=Bi(),r=Ir(),o=Ge();class s{constructor(c=[]){this.version="8.4.38",this.plugins=this.normalize(c)}normalize(c){let p=[];for(let l of c)if(l.postcss===!0?l=l():l.postcss&&(l=l.postcss),typeof l=="object"&&Array.isArray(l.plugins))p=p.concat(l.plugins);else if(typeof l=="object"&&l.postcssPlugin)p.push(l);else if(typeof l=="function")p.push(l);else if(typeof l=="object"&&(l.parse||l.stringify)){if(process.env.NODE_ENV!=="production")throw new Error("PostCSS syntaxes cannot be used as plugins. Instead, please use one of the syntax/parser/stringifier options as outlined in your PostCSS runner documentation.")}else throw new Error(l+" is not a PostCSS plugin");return p}process(c,p={}){return!this.plugins.length&&!p.parser&&!p.stringifier&&!p.syntax?new e(this,c,p):new t(this,c,p)}use(c){return this.plugins=this.plugins.concat(this.normalize([c])),this}}return Gr=s,s.default=s,o.registerProcessor(s),r.registerProcessor(s),Gr}var Vr,qi;function Qo(){if(qi)return Vr;qi=1;let e=mt(),t=bi(),r=yt(),o=Lr(),s=gt(),u=Ge(),c=Ur();function p(l,d){if(Array.isArray(l))return l.map(a=>p(a));let{inputs:i,...f}=l;if(i){d=[];for(let a of i){let n={...a,__proto__:s.prototype};n.map&&(n.map={...n.map,__proto__:t.prototype}),d.push(n)}}if(f.nodes&&(f.nodes=l.nodes.map(a=>p(a,d))),f.source){let{inputId:a,...n}=f.source;f.source=n,a!=null&&(f.source.input=d[a])}if(f.type==="root")return new u(f);if(f.type==="decl")return new e(f);if(f.type==="rule")return new c(f);if(f.type==="comment")return new r(f);if(f.type==="atrule")return new o(f);throw new Error("Unknown node type: "+l.type)}return Vr=p,p.default=p,Vr}var Hr,ji;function Zo(){if(ji)return Hr;ji=1;let e=mr(),t=mt(),r=Bi(),o=Ae(),s=Yo(),u=dt(),c=Qo(),p=Ir(),l=Ai(),d=yt(),i=Lr(),f=Pr(),a=gt(),n=Wr(),m=Li(),h=Ur(),g=Ge(),v=pt();function b(...S){return S.length===1&&Array.isArray(S[0])&&(S=S[0]),new s(S)}return b.plugin=function(x,w){let y=!1;function C(...I){console&&console.warn&&!y&&(y=!0,console.warn(x+`: postcss.plugin was deprecated. Migration guide:
+https://evilmartians.com/chronicles/postcss-8-plugin-migration`),process.env.LANG&&process.env.LANG.startsWith("cn")&&console.warn(x+`: 里面 postcss.plugin 被弃用. 迁移指南:
+https://www.w3ctech.com/topic/2226`));let M=w(...I);return M.postcssPlugin=x,M.postcssVersion=new s().version,M}let O;return Object.defineProperty(C,"postcss",{get(){return O||(O=C()),O}}),C.process=function(I,M,P){return b([C(P)]).process(I,M)},C},b.stringify=u,b.parse=n,b.fromJSON=c,b.list=m,b.comment=S=>new d(S),b.atRule=S=>new i(S),b.decl=S=>new t(S),b.rule=S=>new h(S),b.root=S=>new g(S),b.document=S=>new p(S),b.CssSyntaxError=e,b.Declaration=t,b.Container=o,b.Processor=s,b.Document=p,b.Comment=d,b.Warning=l,b.AtRule=i,b.Result=f,b.Input=a,b.Rule=h,b.Root=g,b.Node=v,r.registerPostcss(b),Hr=b,b.default=b,Hr}var Ko=Zo();const V=Wo(Ko);V.stringify,V.fromJSON,V.plugin,V.parse,V.list,V.document,V.comment,V.atRule,V.rule,V.decl,V.root,V.CssSyntaxError,V.Declaration,V.Container,V.Processor,V.Document,V.Comment,V.Warning,V.AtRule,V.Result,V.Input,V.Rule,V.Root,V.Node;class Jr{constructor(...t){he(this,"parentElement",null),he(this,"parentNode",null),he(this,"ownerDocument"),he(this,"firstChild",null),he(this,"lastChild",null),he(this,"previousSibling",null),he(this,"nextSibling",null),he(this,"ELEMENT_NODE",1),he(this,"TEXT_NODE",3),he(this,"nodeType"),he(this,"nodeName"),he(this,"RRNodeType")}get childNodes(){const t=[];let r=this.firstChild;for(;r;)t.push(r),r=r.nextSibling;return t}contains(t){if(t instanceof Jr){if(t.ownerDocument!==this.ownerDocument)return!1;if(t===this)return!0}else return!1;for(;t.parentNode;){if(t.parentNode===this)return!0;t=t.parentNode}return!1}appendChild(t){throw new Error("RRDomException: Failed to execute 'appendChild' on 'RRNode': This RRNode type does not support this method.")}insertBefore(t,r){throw new Error("RRDomException: Failed to execute 'insertBefore' on 'RRNode': This RRNode type does not support this method.")}removeChild(t){throw new Error("RRDomException: Failed to execute 'removeChild' on 'RRNode': This RRNode type does not support this method.")}toString(){return"RRNode"}}const Gi={Node:["childNodes","parentNode","parentElement","textContent","ownerDocument","firstChild","lastChild","nextSibling","previousSibling"],ShadowRoot:["host","styleSheets"],Element:["shadowRoot","querySelector","querySelectorAll"],MutationObserver:[]},Vi={Node:["contains","getRootNode"],ShadowRoot:["getSelection"],Element:[],MutationObserver:["constructor"]},wt={},Hi={},Xo=()=>!!globalThis.Zone;function Yr(e){if(wt[e])return wt[e];const t=globalThis[e],r=t.prototype,o=e in Gi?Gi[e]:void 0,s=!!(o&&o.every(p=>{var l,d;return!!((d=(l=Object.getOwnPropertyDescriptor(r,p))==null?void 0:l.get)!=null&&d.toString().includes("[native code]"))})),u=e in Vi?Vi[e]:void 0,c=!!(u&&u.every(p=>{var l;return typeof r[p]=="function"&&((l=r[p])==null?void 0:l.toString().includes("[native code]"))}));if(s&&c&&!Xo())return wt[e]=t.prototype,t.prototype;try{const p=document.createElement("iframe");p.style.display="none",document.body.appendChild(p);const l=p.contentWindow;if(!l)return t.prototype;const d=l[e].prototype;if(!d)return p.remove(),r;const i=navigator.userAgent;return i.includes("Safari")&&!i.includes("Chrome")?(p.classList.add("rr-block"),p.setAttribute("__rrwebUntaintedMutationObserver",""),Hi[e]=()=>p.remove()):p.remove(),wt[e]=d}catch{return r}}const Qr={};function le(e,t,r){var o,s;const u=(o=Qr[e])==null?void 0:o[r];if(u)return u.call(t);const c=Yr(e),p=(s=Object.getOwnPropertyDescriptor(c,r))==null?void 0:s.get;return p?((Qr[e]||(Qr[e]={}))[r]=p,p.call(t)):t[r]}const Zr={};function Ji(e,t,r){const o=`${e}.${String(r)}`;if(Zr[o])return Zr[o].bind(t);const u=Yr(e)[r];return typeof u!="function"?t[r]:(Zr[o]=u,u.bind(t))}function ea(e){return le("Node",e,"ownerDocument")}function ta(e){return le("Node",e,"childNodes")}function ra(e){return le("Node",e,"parentNode")}function sa(e){return le("Node",e,"parentElement")}function ia(e){return le("Node",e,"textContent")}function na(e){return le("Node",e,"firstChild")}function oa(e){return le("Node",e,"lastChild")}function aa(e){return le("Node",e,"nextSibling")}function la(e){return le("Node",e,"previousSibling")}function ua(e,t){return Ji("Node",e,"contains")(t)}function ca(e){return Ji("Node",e,"getRootNode")()}function ha(e){return!e||!("host"in e)?null:le("ShadowRoot",e,"host")}function fa(e){return e.styleSheets}function da(e){return!e||!("shadowRoot"in e)?null:le("Element",e,"shadowRoot")}function pa(e,t){return le("Element",e,"querySelector")(t)}function ma(e,t){return le("Element",e,"querySelectorAll")(t)}function Yi(){return[Yr("MutationObserver").constructor,Hi.MutationObserver??(()=>{})]}let Ve=Date.now;/[1-9][0-9]{12}/.test(Date.now().toString())||(Ve=()=>new Date().getTime());function Ne(e,t,r){try{if(!(t in e))return()=>{};const o=e[t],s=r(o);return typeof s=="function"&&(s.prototype=s.prototype||{},Object.defineProperties(s,{__rrweb_original__:{enumerable:!1,value:o}})),e[t]=s,()=>{e[t]=o}}catch{return()=>{}}}const D={ownerDocument:ea,childNodes:ta,parentNode:ra,parentElement:sa,textContent:ia,firstChild:na,lastChild:oa,nextSibling:aa,previousSibling:la,contains:ua,getRootNode:ca,host:ha,styleSheets:fa,shadowRoot:da,querySelector:pa,querySelectorAll:ma,nowTimestamp:Ve,mutationObserverCtor:Yi,patch:Ne};function re(e,t,r=document){const o={capture:!0,passive:!0};return r.addEventListener(e,t,o),()=>r.removeEventListener(e,t,o)}const $e=`Please stop import mirror directly. Instead of that,\r
now you can use replayer.getMirror() to access the mirror instance of a replayer,\r
-or you can use record.mirror to access the mirror instance during recording.`;let ao={map:{},getId(){return console.error(ze),-1},getNode(){return console.error(ze),null},removeNodeFromMap(){console.error(ze)},has(){return console.error(ze),!1},reset(){console.error(ze)}};typeof window<"u"&&window.Proxy&&window.Reflect&&(ao=new Proxy(ao,{get(s,e,t){return e==="map"&&console.error(ze),Reflect.get(s,e,t)}}));function ht(s,e,t={}){let r=null,i=0;return function(...n){const o=Date.now();!i&&t.leading===!1&&(i=o);const l=e-(o-i),a=this;l<=0||l>e?(r&&(clearTimeout(r),r=null),i=o,s.apply(a,n)):!r&&t.trailing!==!1&&(r=setTimeout(()=>{i=t.leading===!1?0:Date.now(),r=null,s.apply(a,n)},l))}}function ur(s,e,t,r,i=window){const n=i.Object.getOwnPropertyDescriptor(s,e);return i.Object.defineProperty(s,e,r?t:{set(o){setTimeout(()=>{t.set.call(this,o)},0),n&&n.set&&n.set.call(this,o)}}),()=>ur(s,e,n||{},!0)}let cr=Date.now;/[1-9][0-9]{12}/.test(Date.now().toString())||(cr=()=>new Date().getTime());function lo(s){var e,t,r,i;const n=s.document;return{left:n.scrollingElement?n.scrollingElement.scrollLeft:s.pageXOffset!==void 0?s.pageXOffset:n.documentElement.scrollLeft||(n==null?void 0:n.body)&&((e=I.parentElement(n.body))==null?void 0:e.scrollLeft)||((t=n==null?void 0:n.body)==null?void 0:t.scrollLeft)||0,top:n.scrollingElement?n.scrollingElement.scrollTop:s.pageYOffset!==void 0?s.pageYOffset:(n==null?void 0:n.documentElement.scrollTop)||(n==null?void 0:n.body)&&((r=I.parentElement(n.body))==null?void 0:r.scrollTop)||((i=n==null?void 0:n.body)==null?void 0:i.scrollTop)||0}}function uo(){return window.innerHeight||document.documentElement&&document.documentElement.clientHeight||document.body&&document.body.clientHeight}function co(){return window.innerWidth||document.documentElement&&document.documentElement.clientWidth||document.body&&document.body.clientWidth}function ho(s){return s?s.nodeType===s.ELEMENT_NODE?s:I.parentElement(s):null}function Q(s,e,t,r){if(!s)return!1;const i=ho(s);if(!i)return!1;try{if(typeof e=="string"){if(i.classList.contains(e)||r&&i.closest("."+e)!==null)return!0}else if(vt(i,e,r))return!0}catch{}return!!(t&&(i.matches(t)||r&&i.closest(t)!==null))}function Nh(s,e){return e.getId(s)!==-1}function Ds(s,e,t){return s.tagName==="TITLE"&&t.headTitleMutations?!0:e.getId(s)===Ze}function fo(s,e){if(He(s))return!1;const t=e.getId(s);if(!e.has(t))return!0;const r=I.parentNode(s);return r&&r.nodeType===s.DOCUMENT_NODE?!1:r?fo(r,e):!0}function Ls(s){return!!s.changedTouches}function kh(s=window){"NodeList"in s&&!s.NodeList.prototype.forEach&&(s.NodeList.prototype.forEach=Array.prototype.forEach),"DOMTokenList"in s&&!s.DOMTokenList.prototype.forEach&&(s.DOMTokenList.prototype.forEach=Array.prototype.forEach)}function po(s,e){return!!(s.nodeName==="IFRAME"&&e.getMeta(s))}function mo(s,e){return!!(s.nodeName==="LINK"&&s.nodeType===s.ELEMENT_NODE&&s.getAttribute&&s.getAttribute("rel")==="stylesheet"&&e.getMeta(s))}function _s(s){return s?s instanceof Ns&&"shadowRoot"in s?!!s.shadowRoot:!!I.shadowRoot(s):!1}class Th{constructor(){w(this,"id",1),w(this,"styleIDMap",new WeakMap),w(this,"idStyleMap",new Map)}getId(e){return this.styleIDMap.get(e)??-1}has(e){return this.styleIDMap.has(e)}add(e,t){if(this.has(e))return this.getId(e);let r;return t===void 0?r=this.id++:r=t,this.styleIDMap.set(e,r),this.idStyleMap.set(r,e),r}getStyle(e){return this.idStyleMap.get(e)||null}reset(){this.styleIDMap=new WeakMap,this.idStyleMap=new Map,this.id=1}generateId(){return this.id++}}function go(s){var e;let t=null;return"getRootNode"in s&&((e=I.getRootNode(s))==null?void 0:e.nodeType)===Node.DOCUMENT_FRAGMENT_NODE&&I.host(I.getRootNode(s))&&(t=I.host(I.getRootNode(s))),t}function Ph(s){let e=s,t;for(;t=go(e);)e=t;return e}function Dh(s){const e=I.ownerDocument(s);if(!e)return!1;const t=Ph(s);return I.contains(e,t)}function yo(s){const e=I.ownerDocument(s);return e?I.contains(e,s)||Dh(s):!1}var R=(s=>(s[s.DomContentLoaded=0]="DomContentLoaded",s[s.Load=1]="Load",s[s.FullSnapshot=2]="FullSnapshot",s[s.IncrementalSnapshot=3]="IncrementalSnapshot",s[s.Meta=4]="Meta",s[s.Custom=5]="Custom",s[s.Plugin=6]="Plugin",s))(R||{}),x=(s=>(s[s.Mutation=0]="Mutation",s[s.MouseMove=1]="MouseMove",s[s.MouseInteraction=2]="MouseInteraction",s[s.Scroll=3]="Scroll",s[s.ViewportResize=4]="ViewportResize",s[s.Input=5]="Input",s[s.TouchMove=6]="TouchMove",s[s.MediaInteraction=7]="MediaInteraction",s[s.StyleSheetRule=8]="StyleSheetRule",s[s.CanvasMutation=9]="CanvasMutation",s[s.Font=10]="Font",s[s.Log=11]="Log",s[s.Drag=12]="Drag",s[s.StyleDeclaration=13]="StyleDeclaration",s[s.Selection=14]="Selection",s[s.AdoptedStyleSheet=15]="AdoptedStyleSheet",s[s.CustomElement=16]="CustomElement",s))(x||{}),q=(s=>(s[s.MouseUp=0]="MouseUp",s[s.MouseDown=1]="MouseDown",s[s.Click=2]="Click",s[s.ContextMenu=3]="ContextMenu",s[s.DblClick=4]="DblClick",s[s.Focus=5]="Focus",s[s.Blur=6]="Blur",s[s.TouchStart=7]="TouchStart",s[s.TouchMove_Departed=8]="TouchMove_Departed",s[s.TouchEnd=9]="TouchEnd",s[s.TouchCancel=10]="TouchCancel",s))(q||{}),ve=(s=>(s[s.Mouse=0]="Mouse",s[s.Pen=1]="Pen",s[s.Touch=2]="Touch",s))(ve||{}),We=(s=>(s[s["2D"]=0]="2D",s[s.WebGL=1]="WebGL",s[s.WebGL2=2]="WebGL2",s))(We||{}),Ge=(s=>(s[s.Play=0]="Play",s[s.Pause=1]="Pause",s[s.Seeked=2]="Seeked",s[s.VolumeChange=3]="VolumeChange",s[s.RateChange=4]="RateChange",s))(Ge||{}),wo=(s=>(s[s.Document=0]="Document",s[s.DocumentType=1]="DocumentType",s[s.Element=2]="Element",s[s.Text=3]="Text",s[s.CDATA=4]="CDATA",s[s.Comment=5]="Comment",s))(wo||{});function So(s){return"__ln"in s}class Lh{constructor(){w(this,"length",0),w(this,"head",null),w(this,"tail",null)}get(e){if(e>=this.length)throw new Error("Position outside of list range");let t=this.head;for(let r=0;r<e;r++)t=(t==null?void 0:t.next)||null;return t}addNode(e){const t={value:e,previous:null,next:null};if(e.__ln=t,e.previousSibling&&So(e.previousSibling)){const r=e.previousSibling.__ln.next;t.next=r,t.previous=e.previousSibling.__ln,e.previousSibling.__ln.next=t,r&&(r.previous=t)}else if(e.nextSibling&&So(e.nextSibling)&&e.nextSibling.__ln.previous){const r=e.nextSibling.__ln.previous;t.previous=r,t.next=e.nextSibling.__ln,e.nextSibling.__ln.previous=t,r&&(r.next=t)}else this.head&&(this.head.previous=t),t.next=this.head,this.head=t;t.next===null&&(this.tail=t),this.length++}removeNode(e){const t=e.__ln;this.head&&(t.previous?(t.previous.next=t.next,t.next?t.next.previous=t.previous:this.tail=t.previous):(this.head=t.next,this.head?this.head.previous=null:this.tail=null),e.__ln&&delete e.__ln,this.length--)}}const bo=(s,e)=>`${s}@${e}`;class _h{constructor(){w(this,"frozen",!1),w(this,"locked",!1),w(this,"texts",[]),w(this,"attributes",[]),w(this,"attributeMap",new WeakMap),w(this,"removes",[]),w(this,"mapRemoves",[]),w(this,"movedMap",{}),w(this,"addedSet",new Set),w(this,"movedSet",new Set),w(this,"droppedSet",new Set),w(this,"removesSubTreeCache",new Set),w(this,"mutationCb"),w(this,"blockClass"),w(this,"blockSelector"),w(this,"maskTextClass"),w(this,"maskTextSelector"),w(this,"inlineStylesheet"),w(this,"maskInputOptions"),w(this,"maskTextFn"),w(this,"maskInputFn"),w(this,"keepIframeSrcFn"),w(this,"recordCanvas"),w(this,"inlineImages"),w(this,"slimDOMOptions"),w(this,"dataURLOptions"),w(this,"doc"),w(this,"mirror"),w(this,"iframeManager"),w(this,"stylesheetManager"),w(this,"shadowDomManager"),w(this,"canvasManager"),w(this,"processedNodeManager"),w(this,"unattachedDoc"),w(this,"processMutations",e=>{e.forEach(this.processMutation),this.emit()}),w(this,"emit",()=>{if(this.frozen||this.locked)return;const e=[],t=new Set,r=new Lh,i=a=>{let u=a,c=Ze;for(;c===Ze;)u=u&&u.nextSibling,c=u&&this.mirror.getId(u);return c},n=a=>{const u=I.parentNode(a);if(!u||!yo(a))return;let c=!1;if(a.nodeType===Node.TEXT_NODE){const g=u.tagName;if(g==="TEXTAREA")return;g==="STYLE"&&this.addedSet.has(u)&&(c=!0)}const h=He(u)?this.mirror.getId(go(a)):this.mirror.getId(u),p=i(a);if(h===-1||p===-1)return r.addNode(a);const m=Pe(a,{doc:this.doc,mirror:this.mirror,blockClass:this.blockClass,blockSelector:this.blockSelector,maskTextClass:this.maskTextClass,maskTextSelector:this.maskTextSelector,skipChild:!0,newlyAddedElement:!0,inlineStylesheet:this.inlineStylesheet,maskInputOptions:this.maskInputOptions,maskTextFn:this.maskTextFn,maskInputFn:this.maskInputFn,slimDOMOptions:this.slimDOMOptions,dataURLOptions:this.dataURLOptions,recordCanvas:this.recordCanvas,inlineImages:this.inlineImages,onSerialize:g=>{po(g,this.mirror)&&this.iframeManager.addIframe(g),mo(g,this.mirror)&&this.stylesheetManager.trackLinkElement(g),_s(a)&&this.shadowDomManager.addShadowRoot(I.shadowRoot(a),this.doc)},onIframeLoad:(g,d)=>{this.iframeManager.attachIframe(g,d),this.shadowDomManager.observeAttachShadow(g)},onStylesheetLoad:(g,d)=>{this.stylesheetManager.attachLinkElement(g,d)},cssCaptured:c});m&&(e.push({parentId:h,nextId:p,node:m}),t.add(m.id))};for(;this.mapRemoves.length;)this.mirror.removeNodeFromMap(this.mapRemoves.shift());for(const a of this.movedSet)vo(this.removesSubTreeCache,a,this.mirror)&&!this.movedSet.has(I.parentNode(a))||n(a);for(const a of this.addedSet)!Co(this.droppedSet,a)&&!vo(this.removesSubTreeCache,a,this.mirror)||Co(this.movedSet,a)?n(a):this.droppedSet.add(a);let o=null;for(;r.length;){let a=null;if(o){const u=this.mirror.getId(I.parentNode(o.value)),c=i(o.value);u!==-1&&c!==-1&&(a=o)}if(!a){let u=r.tail;for(;u;){const c=u;if(u=u.previous,c){const h=this.mirror.getId(I.parentNode(c.value));if(i(c.value)===-1)continue;if(h!==-1){a=c;break}else{const m=c.value,g=I.parentNode(m);if(g&&g.nodeType===Node.DOCUMENT_FRAGMENT_NODE){const d=I.host(g);if(this.mirror.getId(d)!==-1){a=c;break}}}}}}if(!a){for(;r.head;)r.removeNode(r.head.value);break}o=a.previous,r.removeNode(a.value),n(a.value)}const l={texts:this.texts.map(a=>{const u=a.node,c=I.parentNode(u);return c&&c.tagName==="TEXTAREA"&&this.genTextAreaValueMutation(c),{id:this.mirror.getId(u),value:a.value}}).filter(a=>!t.has(a.id)).filter(a=>this.mirror.has(a.id)),attributes:this.attributes.map(a=>{const{attributes:u}=a;if(typeof u.style=="string"){const c=JSON.stringify(a.styleDiff),h=JSON.stringify(a._unchangedStyles);c.length<u.style.length&&(c+h).split("var(").length===u.style.split("var(").length&&(u.style=a.styleDiff)}return{id:this.mirror.getId(a.node),attributes:u}}).filter(a=>!t.has(a.id)).filter(a=>this.mirror.has(a.id)),removes:this.removes,adds:e};!l.texts.length&&!l.attributes.length&&!l.removes.length&&!l.adds.length||(this.texts=[],this.attributes=[],this.attributeMap=new WeakMap,this.removes=[],this.addedSet=new Set,this.movedSet=new Set,this.droppedSet=new Set,this.removesSubTreeCache=new Set,this.movedMap={},this.mutationCb(l))}),w(this,"genTextAreaValueMutation",e=>{let t=this.attributeMap.get(e);t||(t={node:e,attributes:{},styleDiff:{},_unchangedStyles:{}},this.attributes.push(t),this.attributeMap.set(e,t));const r=Array.from(I.childNodes(e),i=>I.textContent(i)||"").join("");t.attributes.value=yt({element:e,maskInputOptions:this.maskInputOptions,tagName:e.tagName,type:wt(e),value:r,maskInputFn:this.maskInputFn})}),w(this,"processMutation",e=>{if(!Ds(e.target,this.mirror,this.slimDOMOptions))switch(e.type){case"characterData":{const t=I.textContent(e.target);!Q(e.target,this.blockClass,this.blockSelector,!1)&&t!==e.oldValue&&this.texts.push({value:oi(e.target,this.maskTextClass,this.maskTextSelector,!0)&&t?this.maskTextFn?this.maskTextFn(t,ho(e.target)):t.replace(/[\S]/g,"*"):t,node:e.target});break}case"attributes":{const t=e.target;let r=e.attributeName,i=e.target.getAttribute(r);if(r==="value"){const o=wt(t);i=yt({element:t,maskInputOptions:this.maskInputOptions,tagName:t.tagName,type:o,value:i,maskInputFn:this.maskInputFn})}if(Q(e.target,this.blockClass,this.blockSelector,!1)||i===e.oldValue)return;let n=this.attributeMap.get(e.target);if(t.tagName==="IFRAME"&&r==="src"&&!this.keepIframeSrcFn(i))if(!t.contentDocument)r="rr_src";else return;if(n||(n={node:e.target,attributes:{},styleDiff:{},_unchangedStyles:{}},this.attributes.push(n),this.attributeMap.set(e.target,n)),r==="type"&&t.tagName==="INPUT"&&(e.oldValue||"").toLowerCase()==="password"&&t.setAttribute("data-rr-is-password","true"),!ni(t.tagName,r))if(n.attributes[r]=ii(this.doc,Ie(t.tagName),Ie(r),i),r==="style"){if(!this.unattachedDoc)try{this.unattachedDoc=document.implementation.createHTMLDocument()}catch{this.unattachedDoc=this.doc}const o=this.unattachedDoc.createElement("span");e.oldValue&&o.setAttribute("style",e.oldValue);for(const l of Array.from(t.style)){const a=t.style.getPropertyValue(l),u=t.style.getPropertyPriority(l);a!==o.style.getPropertyValue(l)||u!==o.style.getPropertyPriority(l)?u===""?n.styleDiff[l]=a:n.styleDiff[l]=[a,u]:n._unchangedStyles[l]=[a,u]}for(const l of Array.from(o.style))t.style.getPropertyValue(l)===""&&(n.styleDiff[l]=!1)}else r==="open"&&t.tagName==="DIALOG"&&(t.matches("dialog:modal")?n.attributes.rr_open_mode="modal":n.attributes.rr_open_mode="non-modal");break}case"childList":{if(Q(e.target,this.blockClass,this.blockSelector,!0))return;if(e.target.tagName==="TEXTAREA"){this.genTextAreaValueMutation(e.target);return}e.addedNodes.forEach(t=>this.genAdds(t,e.target)),e.removedNodes.forEach(t=>{const r=this.mirror.getId(t),i=He(e.target)?this.mirror.getId(I.host(e.target)):this.mirror.getId(e.target);Q(e.target,this.blockClass,this.blockSelector,!1)||Ds(t,this.mirror,this.slimDOMOptions)||!Nh(t,this.mirror)||(this.addedSet.has(t)?(Fs(this.addedSet,t),this.droppedSet.add(t)):this.addedSet.has(e.target)&&r===-1||fo(e.target,this.mirror)||(this.movedSet.has(t)&&this.movedMap[bo(r,i)]?Fs(this.movedSet,t):(this.removes.push({parentId:i,id:r,isShadow:He(e.target)&&Ye(e.target)?!0:void 0}),Fh(t,this.removesSubTreeCache))),this.mapRemoves.push(t))});break}}}),w(this,"genAdds",(e,t)=>{if(!this.processedNodeManager.inOtherBuffer(e,this)&&!(this.addedSet.has(e)||this.movedSet.has(e))){if(this.mirror.hasNode(e)){if(Ds(e,this.mirror,this.slimDOMOptions))return;this.movedSet.add(e);let r=null;t&&this.mirror.hasNode(t)&&(r=this.mirror.getId(t)),r&&r!==-1&&(this.movedMap[bo(this.mirror.getId(e),r)]=!0)}else this.addedSet.add(e),this.droppedSet.delete(e);Q(e,this.blockClass,this.blockSelector,!1)||(I.childNodes(e).forEach(r=>this.genAdds(r)),_s(e)&&I.childNodes(I.shadowRoot(e)).forEach(r=>{this.processedNodeManager.add(r,this),this.genAdds(r,e)}))}})}init(e){["mutationCb","blockClass","blockSelector","maskTextClass","maskTextSelector","inlineStylesheet","maskInputOptions","maskTextFn","maskInputFn","keepIframeSrcFn","recordCanvas","inlineImages","slimDOMOptions","dataURLOptions","doc","mirror","iframeManager","stylesheetManager","shadowDomManager","canvasManager","processedNodeManager"].forEach(t=>{this[t]=e[t]})}freeze(){this.frozen=!0,this.canvasManager.freeze()}unfreeze(){this.frozen=!1,this.canvasManager.unfreeze(),this.emit()}isFrozen(){return this.frozen}lock(){this.locked=!0,this.canvasManager.lock()}unlock(){this.locked=!1,this.canvasManager.unlock(),this.emit()}reset(){this.shadowDomManager.reset(),this.canvasManager.reset()}}function Fs(s,e){s.delete(e),I.childNodes(e).forEach(t=>Fs(s,t))}function Fh(s,e){const t=[s];for(;t.length;){const r=t.pop();e.has(r)||(e.add(r),I.childNodes(r).forEach(i=>t.push(i)))}}function vo(s,e,t){return s.size===0?!1:Uh(s,e)}function Uh(s,e,t){const r=I.parentNode(e);return r?s.has(r):!1}function Co(s,e){return s.size===0?!1:Io(s,e)}function Io(s,e){const t=I.parentNode(e);return t?s.has(t)?!0:Io(s,t):!1}let ft;function Bh(s){ft=s}function zh(){ft=void 0}const A=s=>ft?(...t)=>{try{return s(...t)}catch(r){if(ft&&ft(r)===!0)return;throw r}}:s,Oe=[];function dt(s){try{if("composedPath"in s){const e=s.composedPath();if(e.length)return e[0]}else if("path"in s&&s.path.length)return s.path[0]}catch{}return s&&s.target}function xo(s,e){const t=new _h;Oe.push(t),t.init(s);const r=new(oo())(A(t.processMutations.bind(t)));return r.observe(e,{attributes:!0,attributeOldValue:!0,characterData:!0,characterDataOldValue:!0,childList:!0,subtree:!0}),r}function Wh({mousemoveCb:s,sampling:e,doc:t,mirror:r}){if(e.mousemove===!1)return()=>{};const i=typeof e.mousemove=="number"?e.mousemove:50,n=typeof e.mousemoveCallback=="number"?e.mousemoveCallback:500;let o=[],l;const a=ht(A(h=>{const p=Date.now()-l;s(o.map(m=>(m.timeOffset-=p,m)),h),o=[],l=null}),n),u=A(ht(A(h=>{const p=dt(h),{clientX:m,clientY:g}=Ls(h)?h.changedTouches[0]:h;l||(l=cr()),o.push({x:m,y:g,id:r.getId(p),timeOffset:cr()-l}),a(typeof DragEvent<"u"&&h instanceof DragEvent?x.Drag:h instanceof MouseEvent?x.MouseMove:x.TouchMove)}),i,{trailing:!1})),c=[X("mousemove",u,t),X("touchmove",u,t),X("drag",u,t)];return A(()=>{c.forEach(h=>h())})}function Gh({mouseInteractionCb:s,doc:e,mirror:t,blockClass:r,blockSelector:i,sampling:n}){if(n.mouseInteraction===!1)return()=>{};const o=n.mouseInteraction===!0||n.mouseInteraction===void 0?{}:n.mouseInteraction,l=[];let a=null;const u=c=>h=>{const p=dt(h);if(Q(p,r,i,!0))return;let m=null,g=c;if("pointerType"in h){switch(h.pointerType){case"mouse":m=ve.Mouse;break;case"touch":m=ve.Touch;break;case"pen":m=ve.Pen;break}m===ve.Touch?q[c]===q.MouseDown?g="TouchStart":q[c]===q.MouseUp&&(g="TouchEnd"):ve.Pen}else Ls(h)&&(m=ve.Touch);m!==null?(a=m,(g.startsWith("Touch")&&m===ve.Touch||g.startsWith("Mouse")&&m===ve.Mouse)&&(m=null)):q[c]===q.Click&&(m=a,a=null);const d=Ls(h)?h.changedTouches[0]:h;if(!d)return;const f=t.getId(p),{clientX:S,clientY:b}=d;A(s)({type:q[g],id:f,x:S,y:b,...m!==null&&{pointerType:m}})};return Object.keys(q).filter(c=>Number.isNaN(Number(c))&&!c.endsWith("_Departed")&&o[c]!==!1).forEach(c=>{let h=Ie(c);const p=u(c);if(window.PointerEvent)switch(q[c]){case q.MouseDown:case q.MouseUp:h=h.replace("mouse","pointer");break;case q.TouchStart:case q.TouchEnd:return}l.push(X(h,p,e))}),A(()=>{l.forEach(c=>c())})}function Ao({scrollCb:s,doc:e,mirror:t,blockClass:r,blockSelector:i,sampling:n}){const o=A(ht(A(l=>{const a=dt(l);if(!a||Q(a,r,i,!0))return;const u=t.getId(a);if(a===e&&e.defaultView){const c=lo(e.defaultView);s({id:u,x:c.left,y:c.top})}else s({id:u,x:a.scrollLeft,y:a.scrollTop})}),n.scroll||100));return X("scroll",o,e)}function Vh({viewportResizeCb:s},{win:e}){let t=-1,r=-1;const i=A(ht(A(()=>{const n=uo(),o=co();(t!==n||r!==o)&&(s({width:Number(o),height:Number(n)}),t=n,r=o)}),200));return X("resize",i,e)}const jh=["INPUT","TEXTAREA","SELECT"],Ro=new WeakMap;function Hh({inputCb:s,doc:e,mirror:t,blockClass:r,blockSelector:i,ignoreClass:n,ignoreSelector:o,maskInputOptions:l,maskInputFn:a,sampling:u,userTriggeredOnInput:c}){function h(b){let y=dt(b);const C=b.isTrusted,k=y&&y.tagName;if(y&&k==="OPTION"&&(y=I.parentElement(y)),!y||!k||jh.indexOf(k)<0||Q(y,r,i,!0)||y.classList.contains(n)||o&&y.matches(o))return;let D=y.value,H=!1;const B=wt(y)||"";B==="radio"||B==="checkbox"?H=y.checked:(l[k.toLowerCase()]||l[B])&&(D=yt({element:y,maskInputOptions:l,tagName:k,type:B,value:D,maskInputFn:a})),p(y,c?{text:D,isChecked:H,userTriggered:C}:{text:D,isChecked:H});const V=y.name;B==="radio"&&V&&H&&e.querySelectorAll(`input[type="radio"][name="${V}"]`).forEach(Z=>{if(Z!==y){const ie=Z.value;p(Z,c?{text:ie,isChecked:!H,userTriggered:!1}:{text:ie,isChecked:!H})}})}function p(b,y){const C=Ro.get(b);if(!C||C.text!==y.text||C.isChecked!==y.isChecked){Ro.set(b,y);const k=t.getId(b);A(s)({...y,id:k})}}const g=(u.input==="last"?["change"]:["input","change"]).map(b=>X(b,A(h),e)),d=e.defaultView;if(!d)return()=>{g.forEach(b=>b())};const f=d.Object.getOwnPropertyDescriptor(d.HTMLInputElement.prototype,"value"),S=[[d.HTMLInputElement.prototype,"value"],[d.HTMLInputElement.prototype,"checked"],[d.HTMLSelectElement.prototype,"value"],[d.HTMLTextAreaElement.prototype,"value"],[d.HTMLSelectElement.prototype,"selectedIndex"],[d.HTMLOptionElement.prototype,"selected"]];return f&&f.set&&g.push(...S.map(b=>ur(b[0],b[1],{set(){A(h)({target:this,isTrusted:!1})}},!1,d))),A(()=>{g.forEach(b=>b())})}function hr(s){const e=[];function t(r,i){if(fr("CSSGroupingRule")&&r.parentRule instanceof CSSGroupingRule||fr("CSSMediaRule")&&r.parentRule instanceof CSSMediaRule||fr("CSSSupportsRule")&&r.parentRule instanceof CSSSupportsRule||fr("CSSConditionRule")&&r.parentRule instanceof CSSConditionRule){const o=Array.from(r.parentRule.cssRules).indexOf(r);i.unshift(o)}else if(r.parentStyleSheet){const o=Array.from(r.parentStyleSheet.cssRules).indexOf(r);i.unshift(o)}return i}return t(s,e)}function Ce(s,e,t){let r,i;return s?(s.ownerNode?r=e.getId(s.ownerNode):i=t.getId(s),{styleId:i,id:r}):{}}function Yh({styleSheetRuleCb:s,mirror:e,stylesheetManager:t},{win:r}){if(!r.CSSStyleSheet||!r.CSSStyleSheet.prototype)return()=>{};const i=r.CSSStyleSheet.prototype.insertRule;r.CSSStyleSheet.prototype.insertRule=new Proxy(i,{apply:A((c,h,p)=>{const[m,g]=p,{id:d,styleId:f}=Ce(h,e,t.styleMirror);return(d&&d!==-1||f&&f!==-1)&&s({id:d,styleId:f,adds:[{rule:m,index:g}]}),c.apply(h,p)})}),r.CSSStyleSheet.prototype.addRule=function(c,h,p=this.cssRules.length){const m=`${c} { ${h} }`;return r.CSSStyleSheet.prototype.insertRule.apply(this,[m,p])};const n=r.CSSStyleSheet.prototype.deleteRule;r.CSSStyleSheet.prototype.deleteRule=new Proxy(n,{apply:A((c,h,p)=>{const[m]=p,{id:g,styleId:d}=Ce(h,e,t.styleMirror);return(g&&g!==-1||d&&d!==-1)&&s({id:g,styleId:d,removes:[{index:m}]}),c.apply(h,p)})}),r.CSSStyleSheet.prototype.removeRule=function(c){return r.CSSStyleSheet.prototype.deleteRule.apply(this,[c])};let o;r.CSSStyleSheet.prototype.replace&&(o=r.CSSStyleSheet.prototype.replace,r.CSSStyleSheet.prototype.replace=new Proxy(o,{apply:A((c,h,p)=>{const[m]=p,{id:g,styleId:d}=Ce(h,e,t.styleMirror);return(g&&g!==-1||d&&d!==-1)&&s({id:g,styleId:d,replace:m}),c.apply(h,p)})}));let l;r.CSSStyleSheet.prototype.replaceSync&&(l=r.CSSStyleSheet.prototype.replaceSync,r.CSSStyleSheet.prototype.replaceSync=new Proxy(l,{apply:A((c,h,p)=>{const[m]=p,{id:g,styleId:d}=Ce(h,e,t.styleMirror);return(g&&g!==-1||d&&d!==-1)&&s({id:g,styleId:d,replaceSync:m}),c.apply(h,p)})}));const a={};dr("CSSGroupingRule")?a.CSSGroupingRule=r.CSSGroupingRule:(dr("CSSMediaRule")&&(a.CSSMediaRule=r.CSSMediaRule),dr("CSSConditionRule")&&(a.CSSConditionRule=r.CSSConditionRule),dr("CSSSupportsRule")&&(a.CSSSupportsRule=r.CSSSupportsRule));const u={};return Object.entries(a).forEach(([c,h])=>{u[c]={insertRule:h.prototype.insertRule,deleteRule:h.prototype.deleteRule},h.prototype.insertRule=new Proxy(u[c].insertRule,{apply:A((p,m,g)=>{const[d,f]=g,{id:S,styleId:b}=Ce(m.parentStyleSheet,e,t.styleMirror);return(S&&S!==-1||b&&b!==-1)&&s({id:S,styleId:b,adds:[{rule:d,index:[...hr(m),f||0]}]}),p.apply(m,g)})}),h.prototype.deleteRule=new Proxy(u[c].deleteRule,{apply:A((p,m,g)=>{const[d]=g,{id:f,styleId:S}=Ce(m.parentStyleSheet,e,t.styleMirror);return(f&&f!==-1||S&&S!==-1)&&s({id:f,styleId:S,removes:[{index:[...hr(m),d]}]}),p.apply(m,g)})})}),A(()=>{r.CSSStyleSheet.prototype.insertRule=i,r.CSSStyleSheet.prototype.deleteRule=n,o&&(r.CSSStyleSheet.prototype.replace=o),l&&(r.CSSStyleSheet.prototype.replaceSync=l),Object.entries(a).forEach(([c,h])=>{h.prototype.insertRule=u[c].insertRule,h.prototype.deleteRule=u[c].deleteRule})})}function Oo({mirror:s,stylesheetManager:e},t){var r,i,n;let o=null;t.nodeName==="#document"?o=s.getId(t):o=s.getId(I.host(t));const l=t.nodeName==="#document"?(r=t.defaultView)==null?void 0:r.Document:(n=(i=t.ownerDocument)==null?void 0:i.defaultView)==null?void 0:n.ShadowRoot,a=l!=null&&l.prototype?Object.getOwnPropertyDescriptor(l==null?void 0:l.prototype,"adoptedStyleSheets"):void 0;return o===null||o===-1||!l||!a?()=>{}:(Object.defineProperty(t,"adoptedStyleSheets",{configurable:a.configurable,enumerable:a.enumerable,get(){var u;return(u=a.get)==null?void 0:u.call(this)},set(u){var c;const h=(c=a.set)==null?void 0:c.call(this,u);if(o!==null&&o!==-1)try{e.adoptStyleSheets(u,o)}catch{}return h}}),A(()=>{Object.defineProperty(t,"adoptedStyleSheets",{configurable:a.configurable,enumerable:a.enumerable,get:a.get,set:a.set})}))}function Zh({styleDeclarationCb:s,mirror:e,ignoreCSSAttributes:t,stylesheetManager:r},{win:i}){const n=i.CSSStyleDeclaration.prototype.setProperty;i.CSSStyleDeclaration.prototype.setProperty=new Proxy(n,{apply:A((l,a,u)=>{var c;const[h,p,m]=u;if(t.has(h))return n.apply(a,[h,p,m]);const{id:g,styleId:d}=Ce((c=a.parentRule)==null?void 0:c.parentStyleSheet,e,r.styleMirror);return(g&&g!==-1||d&&d!==-1)&&s({id:g,styleId:d,set:{property:h,value:p,priority:m},index:hr(a.parentRule)}),l.apply(a,u)})});const o=i.CSSStyleDeclaration.prototype.removeProperty;return i.CSSStyleDeclaration.prototype.removeProperty=new Proxy(o,{apply:A((l,a,u)=>{var c;const[h]=u;if(t.has(h))return o.apply(a,[h]);const{id:p,styleId:m}=Ce((c=a.parentRule)==null?void 0:c.parentStyleSheet,e,r.styleMirror);return(p&&p!==-1||m&&m!==-1)&&s({id:p,styleId:m,remove:{property:h},index:hr(a.parentRule)}),l.apply(a,u)})}),A(()=>{i.CSSStyleDeclaration.prototype.setProperty=n,i.CSSStyleDeclaration.prototype.removeProperty=o})}function Jh({mediaInteractionCb:s,blockClass:e,blockSelector:t,mirror:r,sampling:i,doc:n}){const o=A(a=>ht(A(u=>{const c=dt(u);if(!c||Q(c,e,t,!0))return;const{currentTime:h,volume:p,muted:m,playbackRate:g,loop:d}=c;s({type:a,id:r.getId(c),currentTime:h,volume:p,muted:m,playbackRate:g,loop:d})}),i.media||500)),l=[X("play",o(Ge.Play),n),X("pause",o(Ge.Pause),n),X("seeked",o(Ge.Seeked),n),X("volumechange",o(Ge.VolumeChange),n),X("ratechange",o(Ge.RateChange),n)];return A(()=>{l.forEach(a=>a())})}function Kh({fontCb:s,doc:e}){const t=e.defaultView;if(!t)return()=>{};const r=[],i=new WeakMap,n=t.FontFace;t.FontFace=function(a,u,c){const h=new n(a,u,c);return i.set(h,{family:a,buffer:typeof u!="string",descriptors:c,fontSource:typeof u=="string"?u:JSON.stringify(Array.from(new Uint8Array(u)))}),h};const o=Re(e.fonts,"add",function(l){return function(a){return setTimeout(A(()=>{const u=i.get(a);u&&(s(u),i.delete(a))}),0),l.apply(this,[a])}});return r.push(()=>{t.FontFace=n}),r.push(o),A(()=>{r.forEach(l=>l())})}function Xh(s){const{doc:e,mirror:t,blockClass:r,blockSelector:i,selectionCb:n}=s;let o=!0;const l=A(()=>{const a=e.getSelection();if(!a||o&&(a!=null&&a.isCollapsed))return;o=a.isCollapsed||!1;const u=[],c=a.rangeCount||0;for(let h=0;h<c;h++){const p=a.getRangeAt(h),{startContainer:m,startOffset:g,endContainer:d,endOffset:f}=p;Q(m,r,i,!0)||Q(d,r,i,!0)||u.push({start:t.getId(m),startOffset:g,end:t.getId(d),endOffset:f})}n({ranges:u})});return l(),X("selectionchange",l)}function Qh({doc:s,customElementCb:e}){const t=s.defaultView;return!t||!t.customElements?()=>{}:Re(t.customElements,"define",function(i){return function(n,o,l){try{e({define:{name:n}})}catch{console.warn(`Custom element callback failed for ${n}`)}return i.apply(this,[n,o,l])}})}function qh(s,e){const{mutationCb:t,mousemoveCb:r,mouseInteractionCb:i,scrollCb:n,viewportResizeCb:o,inputCb:l,mediaInteractionCb:a,styleSheetRuleCb:u,styleDeclarationCb:c,canvasMutationCb:h,fontCb:p,selectionCb:m,customElementCb:g}=s;s.mutationCb=(...d)=>{e.mutation&&e.mutation(...d),t(...d)},s.mousemoveCb=(...d)=>{e.mousemove&&e.mousemove(...d),r(...d)},s.mouseInteractionCb=(...d)=>{e.mouseInteraction&&e.mouseInteraction(...d),i(...d)},s.scrollCb=(...d)=>{e.scroll&&e.scroll(...d),n(...d)},s.viewportResizeCb=(...d)=>{e.viewportResize&&e.viewportResize(...d),o(...d)},s.inputCb=(...d)=>{e.input&&e.input(...d),l(...d)},s.mediaInteractionCb=(...d)=>{e.mediaInteaction&&e.mediaInteaction(...d),a(...d)},s.styleSheetRuleCb=(...d)=>{e.styleSheetRule&&e.styleSheetRule(...d),u(...d)},s.styleDeclarationCb=(...d)=>{e.styleDeclaration&&e.styleDeclaration(...d),c(...d)},s.canvasMutationCb=(...d)=>{e.canvasMutation&&e.canvasMutation(...d),h(...d)},s.fontCb=(...d)=>{e.font&&e.font(...d),p(...d)},s.selectionCb=(...d)=>{e.selection&&e.selection(...d),m(...d)},s.customElementCb=(...d)=>{e.customElement&&e.customElement(...d),g(...d)}}function ef(s,e={}){const t=s.doc.defaultView;if(!t)return()=>{};qh(s,e);let r;s.recordDOM&&(r=xo(s,s.doc));const i=Wh(s),n=Gh(s),o=Ao(s),l=Vh(s,{win:t}),a=Hh(s),u=Jh(s);let c=()=>{},h=()=>{},p=()=>{},m=()=>{};s.recordDOM&&(c=Yh(s,{win:t}),h=Oo(s,s.doc),p=Zh(s,{win:t}),s.collectFonts&&(m=Kh(s)));const g=Xh(s),d=Qh(s),f=[];for(const S of s.plugins)f.push(S.observer(S.callback,t,S.options));return A(()=>{Oe.forEach(S=>S.reset()),r==null||r.disconnect(),i(),n(),o(),l(),a(),u(),c(),h(),p(),m(),g(),d(),f.forEach(S=>S())})}function fr(s){return typeof window[s]<"u"}function dr(s){return!!(typeof window[s]<"u"&&window[s].prototype&&"insertRule"in window[s].prototype&&"deleteRule"in window[s].prototype)}class Eo{constructor(e){w(this,"iframeIdToRemoteIdMap",new WeakMap),w(this,"iframeRemoteIdToIdMap",new WeakMap),this.generateIdFn=e}getId(e,t,r,i){const n=r||this.getIdToRemoteIdMap(e),o=i||this.getRemoteIdToIdMap(e);let l=n.get(t);return l||(l=this.generateIdFn(),n.set(t,l),o.set(l,t)),l}getIds(e,t){const r=this.getIdToRemoteIdMap(e),i=this.getRemoteIdToIdMap(e);return t.map(n=>this.getId(e,n,r,i))}getRemoteId(e,t,r){const i=r||this.getRemoteIdToIdMap(e);if(typeof t!="number")return t;const n=i.get(t);return n||-1}getRemoteIds(e,t){const r=this.getRemoteIdToIdMap(e);return t.map(i=>this.getRemoteId(e,i,r))}reset(e){if(!e){this.iframeIdToRemoteIdMap=new WeakMap,this.iframeRemoteIdToIdMap=new WeakMap;return}this.iframeIdToRemoteIdMap.delete(e),this.iframeRemoteIdToIdMap.delete(e)}getIdToRemoteIdMap(e){let t=this.iframeIdToRemoteIdMap.get(e);return t||(t=new Map,this.iframeIdToRemoteIdMap.set(e,t)),t}getRemoteIdToIdMap(e){let t=this.iframeRemoteIdToIdMap.get(e);return t||(t=new Map,this.iframeRemoteIdToIdMap.set(e,t)),t}}class tf{constructor(e){w(this,"iframes",new WeakMap),w(this,"crossOriginIframeMap",new WeakMap),w(this,"crossOriginIframeMirror",new Eo(ti)),w(this,"crossOriginIframeStyleMirror"),w(this,"crossOriginIframeRootIdMap",new WeakMap),w(this,"mirror"),w(this,"mutationCb"),w(this,"wrappedEmit"),w(this,"loadListener"),w(this,"stylesheetManager"),w(this,"recordCrossOriginIframes"),this.mutationCb=e.mutationCb,this.wrappedEmit=e.wrappedEmit,this.stylesheetManager=e.stylesheetManager,this.recordCrossOriginIframes=e.recordCrossOriginIframes,this.crossOriginIframeStyleMirror=new Eo(this.stylesheetManager.styleMirror.generateId.bind(this.stylesheetManager.styleMirror)),this.mirror=e.mirror,this.recordCrossOriginIframes&&window.addEventListener("message",this.handleMessage.bind(this))}addIframe(e){this.iframes.set(e,!0),e.contentWindow&&this.crossOriginIframeMap.set(e.contentWindow,e)}addLoadListener(e){this.loadListener=e}attachIframe(e,t){var r,i;this.mutationCb({adds:[{parentId:this.mirror.getId(e),nextId:null,node:t}],removes:[],texts:[],attributes:[],isAttachIframe:!0}),this.recordCrossOriginIframes&&((r=e.contentWindow)==null||r.addEventListener("message",this.handleMessage.bind(this))),(i=this.loadListener)==null||i.call(this,e),e.contentDocument&&e.contentDocument.adoptedStyleSheets&&e.contentDocument.adoptedStyleSheets.length>0&&this.stylesheetManager.adoptStyleSheets(e.contentDocument.adoptedStyleSheets,this.mirror.getId(e.contentDocument))}handleMessage(e){const t=e;if(t.data.type!=="rrweb"||t.origin!==t.data.origin||!e.source)return;const i=this.crossOriginIframeMap.get(e.source);if(!i)return;const n=this.transformCrossOriginEvent(i,t.data.event);n&&this.wrappedEmit(n,t.data.isCheckout)}transformCrossOriginEvent(e,t){var r;switch(t.type){case R.FullSnapshot:{this.crossOriginIframeMirror.reset(e),this.crossOriginIframeStyleMirror.reset(e),this.replaceIdOnNode(t.data.node,e);const i=t.data.node.id;return this.crossOriginIframeRootIdMap.set(e,i),this.patchRootIdOnNode(t.data.node,i),{timestamp:t.timestamp,type:R.IncrementalSnapshot,data:{source:x.Mutation,adds:[{parentId:this.mirror.getId(e),nextId:null,node:t.data.node}],removes:[],texts:[],attributes:[],isAttachIframe:!0}}}case R.Meta:case R.Load:case R.DomContentLoaded:return!1;case R.Plugin:return t;case R.Custom:return this.replaceIds(t.data.payload,e,["id","parentId","previousId","nextId"]),t;case R.IncrementalSnapshot:switch(t.data.source){case x.Mutation:return t.data.adds.forEach(i=>{this.replaceIds(i,e,["parentId","nextId","previousId"]),this.replaceIdOnNode(i.node,e);const n=this.crossOriginIframeRootIdMap.get(e);n&&this.patchRootIdOnNode(i.node,n)}),t.data.removes.forEach(i=>{this.replaceIds(i,e,["parentId","id"])}),t.data.attributes.forEach(i=>{this.replaceIds(i,e,["id"])}),t.data.texts.forEach(i=>{this.replaceIds(i,e,["id"])}),t;case x.Drag:case x.TouchMove:case x.MouseMove:return t.data.positions.forEach(i=>{this.replaceIds(i,e,["id"])}),t;case x.ViewportResize:return!1;case x.MediaInteraction:case x.MouseInteraction:case x.Scroll:case x.CanvasMutation:case x.Input:return this.replaceIds(t.data,e,["id"]),t;case x.StyleSheetRule:case x.StyleDeclaration:return this.replaceIds(t.data,e,["id"]),this.replaceStyleIds(t.data,e,["styleId"]),t;case x.Font:return t;case x.Selection:return t.data.ranges.forEach(i=>{this.replaceIds(i,e,["start","end"])}),t;case x.AdoptedStyleSheet:return this.replaceIds(t.data,e,["id"]),this.replaceStyleIds(t.data,e,["styleIds"]),(r=t.data.styles)==null||r.forEach(i=>{this.replaceStyleIds(i,e,["styleId"])}),t}}return!1}replace(e,t,r,i){for(const n of i)!Array.isArray(t[n])&&typeof t[n]!="number"||(Array.isArray(t[n])?t[n]=e.getIds(r,t[n]):t[n]=e.getId(r,t[n]));return t}replaceIds(e,t,r){return this.replace(this.crossOriginIframeMirror,e,t,r)}replaceStyleIds(e,t,r){return this.replace(this.crossOriginIframeStyleMirror,e,t,r)}replaceIdOnNode(e,t){this.replaceIds(e,t,["id","rootId"]),"childNodes"in e&&e.childNodes.forEach(r=>{this.replaceIdOnNode(r,t)})}patchRootIdOnNode(e,t){e.type!==wo.Document&&!e.rootId&&(e.rootId=t),"childNodes"in e&&e.childNodes.forEach(r=>{this.patchRootIdOnNode(r,t)})}}class rf{constructor(e){w(this,"shadowDoms",new WeakSet),w(this,"mutationCb"),w(this,"scrollCb"),w(this,"bypassOptions"),w(this,"mirror"),w(this,"restoreHandlers",[]),this.mutationCb=e.mutationCb,this.scrollCb=e.scrollCb,this.bypassOptions=e.bypassOptions,this.mirror=e.mirror,this.init()}init(){this.reset(),this.patchAttachShadow(Element,document)}addShadowRoot(e,t){if(!Ye(e)||this.shadowDoms.has(e))return;this.shadowDoms.add(e);const r=xo({...this.bypassOptions,doc:t,mutationCb:this.mutationCb,mirror:this.mirror,shadowDomManager:this},e);this.restoreHandlers.push(()=>r.disconnect()),this.restoreHandlers.push(Ao({...this.bypassOptions,scrollCb:this.scrollCb,doc:e,mirror:this.mirror})),setTimeout(()=>{e.adoptedStyleSheets&&e.adoptedStyleSheets.length>0&&this.bypassOptions.stylesheetManager.adoptStyleSheets(e.adoptedStyleSheets,this.mirror.getId(I.host(e))),this.restoreHandlers.push(Oo({mirror:this.mirror,stylesheetManager:this.bypassOptions.stylesheetManager},e))},0)}observeAttachShadow(e){!e.contentWindow||!e.contentDocument||this.patchAttachShadow(e.contentWindow.Element,e.contentDocument)}patchAttachShadow(e,t){const r=this;this.restoreHandlers.push(Re(e.prototype,"attachShadow",function(i){return function(n){const o=i.call(this,n),l=I.shadowRoot(this);return l&&yo(this)&&r.addShadowRoot(l,t),o}}))}reset(){this.restoreHandlers.forEach(e=>{try{e()}catch{}}),this.restoreHandlers=[],this.shadowDoms=new WeakSet}}for(var Ve="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",sf=typeof Uint8Array>"u"?[]:new Uint8Array(256),pr=0;pr<Ve.length;pr++)sf[Ve.charCodeAt(pr)]=pr;var nf=function(s){var e=new Uint8Array(s),t,r=e.length,i="";for(t=0;t<r;t+=3)i+=Ve[e[t]>>2],i+=Ve[(e[t]&3)<<4|e[t+1]>>4],i+=Ve[(e[t+1]&15)<<2|e[t+2]>>6],i+=Ve[e[t+2]&63];return r%3===2?i=i.substring(0,i.length-1)+"=":r%3===1&&(i=i.substring(0,i.length-2)+"=="),i};const Mo=new Map;function of(s,e){let t=Mo.get(s);return t||(t=new Map,Mo.set(s,t)),t.has(e)||t.set(e,[]),t.get(e)}const $o=(s,e,t)=>{if(!s||!(ko(s,e)||typeof s=="object"))return;const r=s.constructor.name,i=of(t,r);let n=i.indexOf(s);return n===-1&&(n=i.length,i.push(s)),n};function mr(s,e,t){if(s instanceof Array)return s.map(r=>mr(r,e,t));if(s===null)return s;if(s instanceof Float32Array||s instanceof Float64Array||s instanceof Int32Array||s instanceof Uint32Array||s instanceof Uint8Array||s instanceof Uint16Array||s instanceof Int16Array||s instanceof Int8Array||s instanceof Uint8ClampedArray)return{rr_type:s.constructor.name,args:[Object.values(s)]};if(s instanceof ArrayBuffer){const r=s.constructor.name,i=nf(s);return{rr_type:r,base64:i}}else{if(s instanceof DataView)return{rr_type:s.constructor.name,args:[mr(s.buffer,e,t),s.byteOffset,s.byteLength]};if(s instanceof HTMLImageElement){const r=s.constructor.name,{src:i}=s;return{rr_type:r,src:i}}else if(s instanceof HTMLCanvasElement){const r="HTMLImageElement",i=s.toDataURL();return{rr_type:r,src:i}}else{if(s instanceof ImageData)return{rr_type:s.constructor.name,args:[mr(s.data,e,t),s.width,s.height]};if(ko(s,e)||typeof s=="object"){const r=s.constructor.name,i=$o(s,e,t);return{rr_type:r,index:i}}}}return s}const No=(s,e,t)=>s.map(r=>mr(r,e,t)),ko=(s,e)=>!!["WebGLActiveInfo","WebGLBuffer","WebGLFramebuffer","WebGLProgram","WebGLRenderbuffer","WebGLShader","WebGLShaderPrecisionFormat","WebGLTexture","WebGLUniformLocation","WebGLVertexArrayObject","WebGLVertexArrayObjectOES"].filter(i=>typeof e[i]=="function").find(i=>s instanceof e[i]);function af(s,e,t,r){const i=[],n=Object.getOwnPropertyNames(e.CanvasRenderingContext2D.prototype);for(const o of n)try{if(typeof e.CanvasRenderingContext2D.prototype[o]!="function")continue;const l=Re(e.CanvasRenderingContext2D.prototype,o,function(a){return function(...u){return Q(this.canvas,t,r,!0)||setTimeout(()=>{const c=No(u,e,this);s(this.canvas,{type:We["2D"],property:o,args:c})},0),a.apply(this,u)}});i.push(l)}catch{const l=ur(e.CanvasRenderingContext2D.prototype,o,{set(a){s(this.canvas,{type:We["2D"],property:o,args:[a],setter:!0})}});i.push(l)}return()=>{i.forEach(o=>o())}}function lf(s){return s==="experimental-webgl"?"webgl":s}function To(s,e,t,r){const i=[];try{const n=Re(s.HTMLCanvasElement.prototype,"getContext",function(o){return function(l,...a){if(!Q(this,e,t,!0)){const u=lf(l);if("__context"in this||(this.__context=u),r&&["webgl","webgl2"].includes(u))if(a[0]&&typeof a[0]=="object"){const c=a[0];c.preserveDrawingBuffer||(c.preserveDrawingBuffer=!0)}else a.splice(0,1,{preserveDrawingBuffer:!0})}return o.apply(this,[l,...a])}});i.push(n)}catch{console.error("failed to patch HTMLCanvasElement.prototype.getContext")}return()=>{i.forEach(n=>n())}}function Po(s,e,t,r,i,n){const o=[],l=Object.getOwnPropertyNames(s);for(const a of l)if(!["isContextLost","canvas","drawingBufferWidth","drawingBufferHeight"].includes(a))try{if(typeof s[a]!="function")continue;const u=Re(s,a,function(c){return function(...h){const p=c.apply(this,h);if($o(p,n,this),"tagName"in this.canvas&&!Q(this.canvas,r,i,!0)){const m=No(h,n,this),g={type:e,property:a,args:m};t(this.canvas,g)}return p}});o.push(u)}catch{const u=ur(s,a,{set(c){t(this.canvas,{type:e,property:a,args:[c],setter:!0})}});o.push(u)}return o}function uf(s,e,t,r){const i=[];return i.push(...Po(e.WebGLRenderingContext.prototype,We.WebGL,s,t,r,e)),typeof e.WebGL2RenderingContext<"u"&&i.push(...Po(e.WebGL2RenderingContext.prototype,We.WebGL2,s,t,r,e)),()=>{i.forEach(n=>n())}}const Do="KGZ1bmN0aW9uKCkgewogICJ1c2Ugc3RyaWN0IjsKICB2YXIgY2hhcnMgPSAiQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVphYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ejAxMjM0NTY3ODkrLyI7CiAgdmFyIGxvb2t1cCA9IHR5cGVvZiBVaW50OEFycmF5ID09PSAidW5kZWZpbmVkIiA/IFtdIDogbmV3IFVpbnQ4QXJyYXkoMjU2KTsKICBmb3IgKHZhciBpID0gMDsgaSA8IGNoYXJzLmxlbmd0aDsgaSsrKSB7CiAgICBsb29rdXBbY2hhcnMuY2hhckNvZGVBdChpKV0gPSBpOwogIH0KICB2YXIgZW5jb2RlID0gZnVuY3Rpb24oYXJyYXlidWZmZXIpIHsKICAgIHZhciBieXRlcyA9IG5ldyBVaW50OEFycmF5KGFycmF5YnVmZmVyKSwgaTIsIGxlbiA9IGJ5dGVzLmxlbmd0aCwgYmFzZTY0ID0gIiI7CiAgICBmb3IgKGkyID0gMDsgaTIgPCBsZW47IGkyICs9IDMpIHsKICAgICAgYmFzZTY0ICs9IGNoYXJzW2J5dGVzW2kyXSA+PiAyXTsKICAgICAgYmFzZTY0ICs9IGNoYXJzWyhieXRlc1tpMl0gJiAzKSA8PCA0IHwgYnl0ZXNbaTIgKyAxXSA+PiA0XTsKICAgICAgYmFzZTY0ICs9IGNoYXJzWyhieXRlc1tpMiArIDFdICYgMTUpIDw8IDIgfCBieXRlc1tpMiArIDJdID4+IDZdOwogICAgICBiYXNlNjQgKz0gY2hhcnNbYnl0ZXNbaTIgKyAyXSAmIDYzXTsKICAgIH0KICAgIGlmIChsZW4gJSAzID09PSAyKSB7CiAgICAgIGJhc2U2NCA9IGJhc2U2NC5zdWJzdHJpbmcoMCwgYmFzZTY0Lmxlbmd0aCAtIDEpICsgIj0iOwogICAgfSBlbHNlIGlmIChsZW4gJSAzID09PSAxKSB7CiAgICAgIGJhc2U2NCA9IGJhc2U2NC5zdWJzdHJpbmcoMCwgYmFzZTY0Lmxlbmd0aCAtIDIpICsgIj09IjsKICAgIH0KICAgIHJldHVybiBiYXNlNjQ7CiAgfTsKICBjb25zdCBsYXN0QmxvYk1hcCA9IC8qIEBfX1BVUkVfXyAqLyBuZXcgTWFwKCk7CiAgY29uc3QgdHJhbnNwYXJlbnRCbG9iTWFwID0gLyogQF9fUFVSRV9fICovIG5ldyBNYXAoKTsKICBhc3luYyBmdW5jdGlvbiBnZXRUcmFuc3BhcmVudEJsb2JGb3Iod2lkdGgsIGhlaWdodCwgZGF0YVVSTE9wdGlvbnMpIHsKICAgIGNvbnN0IGlkID0gYCR7d2lkdGh9LSR7aGVpZ2h0fWA7CiAgICBpZiAoIk9mZnNjcmVlbkNhbnZhcyIgaW4gZ2xvYmFsVGhpcykgewogICAgICBpZiAodHJhbnNwYXJlbnRCbG9iTWFwLmhhcyhpZCkpIHJldHVybiB0cmFuc3BhcmVudEJsb2JNYXAuZ2V0KGlkKTsKICAgICAgY29uc3Qgb2Zmc2NyZWVuID0gbmV3IE9mZnNjcmVlbkNhbnZhcyh3aWR0aCwgaGVpZ2h0KTsKICAgICAgb2Zmc2NyZWVuLmdldENvbnRleHQoIjJkIik7CiAgICAgIGNvbnN0IGJsb2IgPSBhd2FpdCBvZmZzY3JlZW4uY29udmVydFRvQmxvYihkYXRhVVJMT3B0aW9ucyk7CiAgICAgIGNvbnN0IGFycmF5QnVmZmVyID0gYXdhaXQgYmxvYi5hcnJheUJ1ZmZlcigpOwogICAgICBjb25zdCBiYXNlNjQgPSBlbmNvZGUoYXJyYXlCdWZmZXIpOwogICAgICB0cmFuc3BhcmVudEJsb2JNYXAuc2V0KGlkLCBiYXNlNjQpOwogICAgICByZXR1cm4gYmFzZTY0OwogICAgfSBlbHNlIHsKICAgICAgcmV0dXJuICIiOwogICAgfQogIH0KICBjb25zdCB3b3JrZXIgPSBzZWxmOwogIHdvcmtlci5vbm1lc3NhZ2UgPSBhc3luYyBmdW5jdGlvbihlKSB7CiAgICBpZiAoIk9mZnNjcmVlbkNhbnZhcyIgaW4gZ2xvYmFsVGhpcykgewogICAgICBjb25zdCB7IGlkLCBiaXRtYXAsIHdpZHRoLCBoZWlnaHQsIGRhdGFVUkxPcHRpb25zIH0gPSBlLmRhdGE7CiAgICAgIGNvbnN0IHRyYW5zcGFyZW50QmFzZTY0ID0gZ2V0VHJhbnNwYXJlbnRCbG9iRm9yKAogICAgICAgIHdpZHRoLAogICAgICAgIGhlaWdodCwKICAgICAgICBkYXRhVVJMT3B0aW9ucwogICAgICApOwogICAgICBjb25zdCBvZmZzY3JlZW4gPSBuZXcgT2Zmc2NyZWVuQ2FudmFzKHdpZHRoLCBoZWlnaHQpOwogICAgICBjb25zdCBjdHggPSBvZmZzY3JlZW4uZ2V0Q29udGV4dCgiMmQiKTsKICAgICAgY3R4LmRyYXdJbWFnZShiaXRtYXAsIDAsIDApOwogICAgICBiaXRtYXAuY2xvc2UoKTsKICAgICAgY29uc3QgYmxvYiA9IGF3YWl0IG9mZnNjcmVlbi5jb252ZXJ0VG9CbG9iKGRhdGFVUkxPcHRpb25zKTsKICAgICAgY29uc3QgdHlwZSA9IGJsb2IudHlwZTsKICAgICAgY29uc3QgYXJyYXlCdWZmZXIgPSBhd2FpdCBibG9iLmFycmF5QnVmZmVyKCk7CiAgICAgIGNvbnN0IGJhc2U2NCA9IGVuY29kZShhcnJheUJ1ZmZlcik7CiAgICAgIGlmICghbGFzdEJsb2JNYXAuaGFzKGlkKSAmJiBhd2FpdCB0cmFuc3BhcmVudEJhc2U2NCA9PT0gYmFzZTY0KSB7CiAgICAgICAgbGFzdEJsb2JNYXAuc2V0KGlkLCBiYXNlNjQpOwogICAgICAgIHJldHVybiB3b3JrZXIucG9zdE1lc3NhZ2UoeyBpZCB9KTsKICAgICAgfQogICAgICBpZiAobGFzdEJsb2JNYXAuZ2V0KGlkKSA9PT0gYmFzZTY0KSByZXR1cm4gd29ya2VyLnBvc3RNZXNzYWdlKHsgaWQgfSk7CiAgICAgIHdvcmtlci5wb3N0TWVzc2FnZSh7CiAgICAgICAgaWQsCiAgICAgICAgdHlwZSwKICAgICAgICBiYXNlNjQsCiAgICAgICAgd2lkdGgsCiAgICAgICAgaGVpZ2h0CiAgICAgIH0pOwogICAgICBsYXN0QmxvYk1hcC5zZXQoaWQsIGJhc2U2NCk7CiAgICB9IGVsc2UgewogICAgICByZXR1cm4gd29ya2VyLnBvc3RNZXNzYWdlKHsgaWQ6IGUuZGF0YS5pZCB9KTsKICAgIH0KICB9Owp9KSgpOwovLyMgc291cmNlTWFwcGluZ1VSTD1pbWFnZS1iaXRtYXAtZGF0YS11cmwtd29ya2VyLUlKcEM3Z19iLmpzLm1hcAo=",cf=s=>Uint8Array.from(atob(s),e=>e.charCodeAt(0)),Lo=typeof window<"u"&&window.Blob&&new Blob([cf(Do)],{type:"text/javascript;charset=utf-8"});function hf(s){let e;try{if(e=Lo&&(window.URL||window.webkitURL).createObjectURL(Lo),!e)throw"";const t=new Worker(e,{name:s==null?void 0:s.name});return t.addEventListener("error",()=>{(window.URL||window.webkitURL).revokeObjectURL(e)}),t}catch{return new Worker("data:text/javascript;base64,"+Do,{name:s==null?void 0:s.name})}finally{e&&(window.URL||window.webkitURL).revokeObjectURL(e)}}class ff{constructor(e){w(this,"pendingCanvasMutations",new Map),w(this,"rafStamps",{latestId:0,invokeId:null}),w(this,"mirror"),w(this,"mutationCb"),w(this,"resetObservers"),w(this,"frozen",!1),w(this,"locked",!1),w(this,"processMutation",(a,u)=>{(this.rafStamps.invokeId&&this.rafStamps.latestId!==this.rafStamps.invokeId||!this.rafStamps.invokeId)&&(this.rafStamps.invokeId=this.rafStamps.latestId),this.pendingCanvasMutations.has(a)||this.pendingCanvasMutations.set(a,[]),this.pendingCanvasMutations.get(a).push(u)});const{sampling:t="all",win:r,blockClass:i,blockSelector:n,recordCanvas:o,dataURLOptions:l}=e;this.mutationCb=e.mutationCb,this.mirror=e.mirror,o&&t==="all"&&this.initCanvasMutationObserver(r,i,n),o&&typeof t=="number"&&this.initCanvasFPSObserver(t,r,i,n,{dataURLOptions:l})}reset(){this.pendingCanvasMutations.clear(),this.resetObservers&&this.resetObservers()}freeze(){this.frozen=!0}unfreeze(){this.frozen=!1}lock(){this.locked=!0}unlock(){this.locked=!1}initCanvasFPSObserver(e,t,r,i,n){const o=To(t,r,i,!0),l=new Map,a=new hf;a.onmessage=g=>{const{id:d}=g.data;if(l.set(d,!1),!("base64"in g.data))return;const{base64:f,type:S,width:b,height:y}=g.data;this.mutationCb({id:d,type:We["2D"],commands:[{property:"clearRect",args:[0,0,b,y]},{property:"drawImage",args:[{rr_type:"ImageBitmap",args:[{rr_type:"Blob",data:[{rr_type:"ArrayBuffer",base64:f}],type:S}]},0,0]}]})};const u=1e3/e;let c=0,h;const p=()=>{const g=[];return t.document.querySelectorAll("canvas").forEach(d=>{Q(d,r,i,!0)||g.push(d)}),g},m=g=>{if(c&&g-c<u){h=requestAnimationFrame(m);return}c=g,p().forEach(async d=>{var f;const S=this.mirror.getId(d);if(l.get(S)||d.width===0||d.height===0)return;if(l.set(S,!0),["webgl","webgl2"].includes(d.__context)){const y=d.getContext(d.__context);((f=y==null?void 0:y.getContextAttributes())==null?void 0:f.preserveDrawingBuffer)===!1&&y.clear(y.COLOR_BUFFER_BIT)}const b=await createImageBitmap(d);a.postMessage({id:S,bitmap:b,width:d.width,height:d.height,dataURLOptions:n.dataURLOptions},[b])}),h=requestAnimationFrame(m)};h=requestAnimationFrame(m),this.resetObservers=()=>{o(),cancelAnimationFrame(h)}}initCanvasMutationObserver(e,t,r){this.startRAFTimestamping(),this.startPendingCanvasMutationFlusher();const i=To(e,t,r,!1),n=af(this.processMutation.bind(this),e,t,r),o=uf(this.processMutation.bind(this),e,t,r);this.resetObservers=()=>{i(),n(),o()}}startPendingCanvasMutationFlusher(){requestAnimationFrame(()=>this.flushPendingCanvasMutations())}startRAFTimestamping(){const e=t=>{this.rafStamps.latestId=t,requestAnimationFrame(e)};requestAnimationFrame(e)}flushPendingCanvasMutations(){this.pendingCanvasMutations.forEach((e,t)=>{const r=this.mirror.getId(t);this.flushPendingCanvasMutationFor(t,r)}),requestAnimationFrame(()=>this.flushPendingCanvasMutations())}flushPendingCanvasMutationFor(e,t){if(this.frozen||this.locked)return;const r=this.pendingCanvasMutations.get(e);if(!r||t===-1)return;const i=r.map(o=>{const{type:l,...a}=o;return a}),{type:n}=r[0];this.mutationCb({id:t,type:n,commands:i}),this.pendingCanvasMutations.delete(e)}}class df{constructor(e){w(this,"trackedLinkElements",new WeakSet),w(this,"mutationCb"),w(this,"adoptedStyleSheetCb"),w(this,"styleMirror",new Th),this.mutationCb=e.mutationCb,this.adoptedStyleSheetCb=e.adoptedStyleSheetCb}attachLinkElement(e,t){"_cssText"in t.attributes&&this.mutationCb({adds:[],removes:[],texts:[],attributes:[{id:t.id,attributes:t.attributes}]}),this.trackLinkElement(e)}trackLinkElement(e){this.trackedLinkElements.has(e)||(this.trackedLinkElements.add(e),this.trackStylesheetInLinkElement(e))}adoptStyleSheets(e,t){if(e.length===0)return;const r={id:t,styleIds:[]},i=[];for(const n of e){let o;this.styleMirror.has(n)?o=this.styleMirror.getId(n):(o=this.styleMirror.add(n),i.push({styleId:o,rules:Array.from(n.rules||CSSRule,(l,a)=>({rule:Xs(l,n.href),index:a}))})),r.styleIds.push(o)}i.length>0&&(r.styles=i),this.adoptedStyleSheetCb(r)}reset(){this.styleMirror.reset(),this.trackedLinkElements=new WeakSet}trackStylesheetInLinkElement(e){}}class pf{constructor(){w(this,"nodeMap",new WeakMap),w(this,"active",!1)}inOtherBuffer(e,t){const r=this.nodeMap.get(e);return r&&Array.from(r).some(i=>i!==t)}add(e,t){this.active||(this.active=!0,requestAnimationFrame(()=>{this.nodeMap=new WeakMap,this.active=!1})),this.nodeMap.set(e,(this.nodeMap.get(e)||new Set).add(t))}destroy(){}}let z,gr,Us,yr=!1;try{if(Array.from([1],s=>s*2)[0]!==2){const s=document.createElement("iframe");document.body.appendChild(s),Array.from=((js=s.contentWindow)==null?void 0:js.Array.from)||Array.from,document.body.removeChild(s)}}catch(s){console.debug("Unable to override Array.from",s)}const le=va();function de(s={}){const{emit:e,checkoutEveryNms:t,checkoutEveryNth:r,blockClass:i="rr-block",blockSelector:n=null,ignoreClass:o="rr-ignore",ignoreSelector:l=null,maskTextClass:a="rr-mask",maskTextSelector:u=null,inlineStylesheet:c=!0,maskAllInputs:h,maskInputOptions:p,slimDOMOptions:m,maskInputFn:g,maskTextFn:d,hooks:f,packFn:S,sampling:b={},dataURLOptions:y={},mousemoveWait:C,recordDOM:k=!0,recordCanvas:D=!1,recordCrossOriginIframes:H=!1,recordAfter:B=s.recordAfter==="DOMContentLoaded"?s.recordAfter:"load",userTriggeredOnInput:V=!1,collectFonts:Z=!1,inlineImages:ie=!1,plugins:pe,keepIframeSrcFn:O=()=>!1,ignoreCSSAttributes:pt=new Set([]),errorHandler:mt}=s;Bh(mt);const Y=H?window.parent===window:!0;let ne=!1;if(!Y)try{window.parent.document&&(ne=!1)}catch{ne=!0}if(Y&&!e)throw new Error("emit function is required");if(!Y&&!ne)return()=>{};C!==void 0&&b.mousemove===void 0&&(b.mousemove=C),le.reset();const Me=h===!0?{color:!0,date:!0,"datetime-local":!0,email:!0,month:!0,number:!0,range:!0,search:!0,tel:!0,text:!0,time:!0,url:!0,week:!0,textarea:!0,select:!0,password:!0}:p!==void 0?p:{password:!0},me=ai(m);kh();let zo,zs=0;const Wo=E=>{for(const ue of pe||[])ue.eventProcessor&&(E=ue.eventProcessor(E));return S&&!ne&&(E=S(E)),E};z=(E,ue)=>{var W;const G=E;if(G.timestamp=cr(),(W=Oe[0])!=null&&W.isFrozen()&&G.type!==R.FullSnapshot&&!(G.type===R.IncrementalSnapshot&&G.data.source===x.Mutation)&&Oe.forEach(te=>te.unfreeze()),Y)e==null||e(Wo(G),ue);else if(ne){const te={type:"rrweb",event:Wo(G),origin:window.location.origin,isCheckout:ue};window.parent.postMessage(te,"*")}if(G.type===R.FullSnapshot)zo=G,zs=0;else if(G.type===R.IncrementalSnapshot){if(G.data.source===x.Mutation&&G.data.isAttachIframe)return;zs++;const te=r&&zs>=r,T=t&&G.timestamp-zo.timestamp>t;(te||T)&&gr(!0)}};const br=E=>{z({type:R.IncrementalSnapshot,data:{source:x.Mutation,...E}})},Go=E=>z({type:R.IncrementalSnapshot,data:{source:x.Scroll,...E}}),Vo=E=>z({type:R.IncrementalSnapshot,data:{source:x.CanvasMutation,...E}}),Vf=E=>z({type:R.IncrementalSnapshot,data:{source:x.AdoptedStyleSheet,...E}}),$e=new df({mutationCb:br,adoptedStyleSheetCb:Vf}),Ne=new tf({mirror:le,mutationCb:br,stylesheetManager:$e,recordCrossOriginIframes:H,wrappedEmit:z});for(const E of pe||[])E.getMirror&&E.getMirror({nodeMirror:le,crossOriginIframeMirror:Ne.crossOriginIframeMirror,crossOriginIframeStyleMirror:Ne.crossOriginIframeStyleMirror});const Ws=new pf;Us=new ff({recordCanvas:D,mutationCb:Vo,win:window,blockClass:i,blockSelector:n,mirror:le,sampling:b.canvas,dataURLOptions:y});const vr=new rf({mutationCb:br,scrollCb:Go,bypassOptions:{blockClass:i,blockSelector:n,maskTextClass:a,maskTextSelector:u,inlineStylesheet:c,maskInputOptions:Me,dataURLOptions:y,maskTextFn:d,maskInputFn:g,recordCanvas:D,inlineImages:ie,sampling:b,slimDOMOptions:me,iframeManager:Ne,stylesheetManager:$e,canvasManager:Us,keepIframeSrcFn:O,processedNodeManager:Ws},mirror:le});gr=(E=!1)=>{if(!k)return;z({type:R.Meta,data:{href:window.location.href,width:co(),height:uo()}},E),$e.reset(),vr.init(),Oe.forEach(W=>W.lock());const ue=ja(document,{mirror:le,blockClass:i,blockSelector:n,maskTextClass:a,maskTextSelector:u,inlineStylesheet:c,maskAllInputs:Me,maskTextFn:d,maskInputFn:g,slimDOM:me,dataURLOptions:y,recordCanvas:D,inlineImages:ie,onSerialize:W=>{po(W,le)&&Ne.addIframe(W),mo(W,le)&&$e.trackLinkElement(W),_s(W)&&vr.addShadowRoot(I.shadowRoot(W),document)},onIframeLoad:(W,G)=>{Ne.attachIframe(W,G),vr.observeAttachShadow(W)},onStylesheetLoad:(W,G)=>{$e.attachLinkElement(W,G)},keepIframeSrcFn:O});if(!ue)return console.warn("Failed to snapshot the document");z({type:R.FullSnapshot,data:{node:ue,initialOffset:lo(window)}},E),Oe.forEach(W=>W.unlock()),document.adoptedStyleSheets&&document.adoptedStyleSheets.length>0&&$e.adoptStyleSheets(document.adoptedStyleSheets,le.getId(document))};try{const E=[],ue=G=>{var te;return A(ef)({mutationCb:br,mousemoveCb:(T,Gs)=>z({type:R.IncrementalSnapshot,data:{source:Gs,positions:T}}),mouseInteractionCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.MouseInteraction,...T}}),scrollCb:Go,viewportResizeCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.ViewportResize,...T}}),inputCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.Input,...T}}),mediaInteractionCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.MediaInteraction,...T}}),styleSheetRuleCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.StyleSheetRule,...T}}),styleDeclarationCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.StyleDeclaration,...T}}),canvasMutationCb:Vo,fontCb:T=>z({type:R.IncrementalSnapshot,data:{source:x.Font,...T}}),selectionCb:T=>{z({type:R.IncrementalSnapshot,data:{source:x.Selection,...T}})},customElementCb:T=>{z({type:R.IncrementalSnapshot,data:{source:x.CustomElement,...T}})},blockClass:i,ignoreClass:o,ignoreSelector:l,maskTextClass:a,maskTextSelector:u,maskInputOptions:Me,inlineStylesheet:c,sampling:b,recordDOM:k,recordCanvas:D,inlineImages:ie,userTriggeredOnInput:V,collectFonts:Z,doc:G,maskInputFn:g,maskTextFn:d,keepIframeSrcFn:O,blockSelector:n,slimDOMOptions:me,dataURLOptions:y,mirror:le,iframeManager:Ne,stylesheetManager:$e,shadowDomManager:vr,processedNodeManager:Ws,canvasManager:Us,ignoreCSSAttributes:pt,plugins:((te=pe==null?void 0:pe.filter(T=>T.observer))==null?void 0:te.map(T=>({observer:T.observer,options:T.options,callback:Gs=>z({type:R.Plugin,data:{plugin:T.name,payload:Gs}})})))||[]},f)};Ne.addLoadListener(G=>{try{E.push(ue(G.contentDocument))}catch(te){console.warn(te)}});const W=()=>{gr(),E.push(ue(document)),yr=!0};return["interactive","complete"].includes(document.readyState)?W():(E.push(X("DOMContentLoaded",()=>{z({type:R.DomContentLoaded,data:{}}),B==="DOMContentLoaded"&&W()})),E.push(X("load",()=>{z({type:R.Load,data:{}}),B==="load"&&W()},window))),()=>{E.forEach(G=>{try{G()}catch(te){String(te).toLowerCase().includes("cross-origin")||console.warn(te)}}),Ws.destroy(),yr=!1,zh()}}catch(E){console.warn(E)}}de.addCustomEvent=(s,e)=>{if(!yr)throw new Error("please add custom event after start recording");z({type:R.Custom,data:{tag:s,payload:e}})},de.freezePage=()=>{Oe.forEach(s=>s.freeze())},de.takeFullSnapshot=s=>{if(!yr)throw new Error("please take full snapshot after start recording");gr(s)},de.mirror=le;var _o;(function(s){s[s.NotStarted=0]="NotStarted",s[s.Running=1]="Running",s[s.Stopped=2]="Stopped"})(_o||(_o={}));const{addCustomEvent:sd}=de,{freezePage:id}=de,{takeFullSnapshot:nd}=de;async function mf(s){const t=new TextEncoder().encode(s),r=new CompressionStream("gzip"),i=r.writable.getWriter();i.write(t),i.close();const n=r.readable.getReader(),o=[];let l=0;for(;;){const{done:c,value:h}=await n.read();if(c)break;o.push(h),l+=h.length}const a=new Uint8Array(l);let u=0;for(const c of o)a.set(c,u),u+=c.length;return a}async function gf(s,e,t,r={}){if(r.keepalive)try{return fetch(s,{method:"POST",headers:{"Content-Type":"application/json",Authorization:`Bearer ${e}`},body:t,keepalive:!0}),!0}catch{return!1}const n=await mf(t);return(await fetch(s,{method:"POST",headers:{"Content-Type":"application/json","Content-Encoding":"gzip",Authorization:`Bearer ${e}`},body:n})).status===200}var yf=class{constructor(s={}){v(this,"segmentDuration");v(this,"current");v(this,"previous",null);v(this,"stopFn",null);v(this,"onSegmentReady");v(this,"onActivity");this.segmentDuration=s.segmentDuration??3e4,this.current=this.newSegment(),this.onSegmentReady=s.onSegmentReady,this.onActivity=s.onActivity}start(){this.stopFn=de({emit:s=>{this.onEvent(s)}})}stop(){this.stopFn&&(this.stopFn(),this.stopFn=null)}onEvent(s){if(this.onActivity)try{this.onActivity()}catch{}if(this.current.events.length>0&&s.timestamp-this.current.startedAtMs>=this.segmentDuration){const e=this.current,t=s.timestamp-this.current.startedAtMs>=2*this.segmentDuration;if(this.previous=t?null:this.current,this.current=this.newSegment(),de.takeFullSnapshot(),this.onSegmentReady&&e.events.length>0){const r=e.events[e.events.length-1].timestamp;try{this.onSegmentReady({events:e.events,startedAt:e.startedAt,endedAt:new Date(r).toISOString()})}catch{}}}this.current.events.length===0&&(this.current.startedAtMs=s.timestamp,this.current.startedAt=new Date(s.timestamp).toISOString()),this.current.events.push(s)}drainCurrent(){if(this.current.events.length===0)return null;const s=this.current;this.current=this.newSegment();const e=s.events[s.events.length-1].timestamp;return{events:s.events,startedAt:s.startedAt,endedAt:new Date(e).toISOString()}}newSegment(){const s=Date.now();return{events:[],startedAt:new Date(s).toISOString(),startedAtMs:s}}getSegments(){const s=[];return this.previous&&this.previous.events.length>0&&s.push({events:this.previous.events,timestamp:this.previous.startedAt}),this.current.events.length>0&&s.push({events:this.current.events,timestamp:this.current.startedAt}),s}hasSegments(){return this.current.events.length>0||this.previous!==null&&this.previous.events.length>0}flush(){this.previous=null,this.current=this.newSegment(),this.stopFn&&de.takeFullSnapshot()}},wf=class{constructor(s){v(this,"inactivityMs");v(this,"maxDurationMs");v(this,"checkIntervalMs");v(this,"onSessionEnd");v(this,"onSoftFlush");v(this,"onUnloading");v(this,"onSessionRestart");v(this,"startedAtMs");v(this,"lastActivityMs");v(this,"timer",null);v(this,"installed",!1);v(this,"ended",!1);v(this,"boundPagehide",()=>this.handlePagehide());v(this,"boundVisibility",()=>this.handleVisibility());v(this,"boundPageshow",s=>this.handlePageshow(s));this.inactivityMs=s.inactivityMs??15*6e4,this.maxDurationMs=s.maxDurationMs??60*6e4,this.checkIntervalMs=s.checkIntervalMs??3e4,this.onSessionEnd=s.onSessionEnd,this.onSoftFlush=s.onSoftFlush??(()=>{}),this.onUnloading=s.onUnloading??(()=>{}),this.onSessionRestart=s.onSessionRestart??(()=>{});const e=Date.now();this.startedAtMs=e,this.lastActivityMs=e}install(){this.installed||typeof window>"u"||(window.addEventListener("pagehide",this.boundPagehide),window.addEventListener("pageshow",this.boundPageshow),document.addEventListener("visibilitychange",this.boundVisibility),this.timer=setInterval(()=>this.tick(),this.checkIntervalMs),this.installed=!0)}uninstall(){this.installed&&(window.removeEventListener("pagehide",this.boundPagehide),window.removeEventListener("pageshow",this.boundPageshow),document.removeEventListener("visibilitychange",this.boundVisibility),this.timer!==null&&(clearInterval(this.timer),this.timer=null),this.installed=!1)}markActivity(){this.lastActivityMs=Date.now()}reset(){const s=Date.now();this.startedAtMs=s,this.lastActivityMs=s,this.ended=!1}startedAt(){return new Date(this.startedAtMs)}tick(){if(this.ended)return;const s=Date.now();(s-this.lastActivityMs>=this.inactivityMs||s-this.startedAtMs>=this.maxDurationMs)&&this.endSession()}handlePagehide(){try{this.onUnloading()}catch{}this.endSession()}handlePageshow(s){if(s.persisted){this.reset();try{this.onSessionRestart()}catch{}}}handleVisibility(){typeof document>"u"||document.visibilityState==="hidden"&&this.onSoftFlush()}endSession(){if(!this.ended){this.ended=!0;try{this.onSessionEnd()}catch{}}}};function Fo(){if(typeof window>"u")return{};const s={};try{s.url=window.location.href,s.path=window.location.pathname}catch{}try{document.referrer&&(s.referrer=document.referrer)}catch{}try{navigator.userAgent&&(s.userAgent=navigator.userAgent),navigator.language&&(s.language=navigator.language),navigator.platform&&(s.platform=navigator.platform)}catch{}try{const e=window.innerWidth,t=window.innerHeight;Number.isFinite(e)&&Number.isFinite(t)&&(s.viewport=`${e}x${t}`)}catch{}try{if(window.screen){const e=window.screen.width,t=window.screen.height;Number.isFinite(e)&&Number.isFinite(t)&&(s.screen=`${e}x${t}`)}}catch{}try{const e=Intl.DateTimeFormat().resolvedOptions().timeZone;e&&(s.timezone=e)}catch{}return s}function wr(s){return new Date(s).toISOString()}function Sf(s,e,t){if(s.length>0){const i=[];for(const n of s){const o=n==null?void 0:n.timestamp;typeof o=="number"&&Number.isFinite(o)&&i.push(o)}if(i.length>0)return{startedAt:wr(Math.min(...i)),endedAt:wr(Math.max(...i))}}const r=[...e.map(i=>Date.parse(i.timestamp)),...t.map(i=>Date.parse(i.timestamp))].filter(i=>Number.isFinite(i));return r.length===0?{}:{startedAt:wr(Math.min(...r)),endedAt:wr(Math.max(...r))}}var bf=["Failed to fetch","NetworkError when attempting to fetch resource","Load failed","Network Error","The operation was aborted",/timeout/i,/status code 4\d{2}/,/failed: 4\d{2}/],vf=class{constructor(s,e={}){v(this,"apiUrl");v(this,"token");v(this,"debug");v(this,"debounceMs");v(this,"retryDelayMs");v(this,"version");v(this,"pendingExceptions",[]);v(this,"pendingRecordings",[]);v(this,"pendingSessions",[]);v(this,"isSyncing",!1);v(this,"debounceTimer",null);v(this,"retryTimer",null);v(this,"recorder",null);v(this,"lifecycle",null);v(this,"recordAllSessions");v(this,"sessionId",null);v(this,"sessionStartedAt",null);v(this,"segmentIndex",0);v(this,"globalAttributes",{});v(this,"unloading",!1);v(this,"ignoreErrors");v(this,"beforeCapture");v(this,"captureLogs");v(this,"captureNetwork");v(this,"captureNavigation");v(this,"logs");v(this,"actions");const{token:t,apiUrl:r}=ge(s);this.apiUrl=r,this.token=t,this.debug=e.debug??!1,this.debounceMs=e.debounceMs??1500,this.retryDelayMs=e.retryDelayMs??1e4,this.version=e.version??"",this.ignoreErrors=e.ignoreErrors??bf,this.beforeCapture=e.beforeCapture??null,this.captureLogs=e.captureLogs??!0,this.captureNetwork=e.captureNetwork??!0,this.captureNavigation=e.captureNavigation??!0;const i=e.recordAllSessions===!0,n={windowMs:e.eventsWindowMs??(i?3e4:1e4),maxSize:e.eventsMaxCount??(i?600:200)};this.logs=new Vs(n),this.actions=new Vs(n),this.recordAllSessions=e.recordAllSessions??!1;const o=typeof window<"u";if(this.recordAllSessions&&o&&(this.beginSession(),this.lifecycle=new wf({onUnloading:()=>{this.unloading=!0},onSessionEnd:()=>this.endSession(),onSessionRestart:()=>this.restartSession(),onSoftFlush:()=>{(this.pendingSessions.length>0||this.pendingRecordings.length>0)&&this.scheduleSync()}}),this.lifecycle.install()),e.sessionRecording!==!1&&o){const l={segmentDuration:e.sessionRecordingSegmentDuration};this.recordAllSessions&&(l.onSegmentReady=a=>this.handleSegmentReady(a),l.onActivity=()=>{var a;return(a=this.lifecycle)==null?void 0:a.markActivity()}),this.recorder=new yf(l),this.recorder.start()}}beginSession(){this.sessionId=ce(),this.sessionStartedAt=J(),this.segmentIndex=0,this.pendingSessions.push({id:this.sessionId,startedAt:this.sessionStartedAt,attributes:this.composedSessionAttributes()}),this.scheduleSync()}composedSessionAttributes(){return{...Fo(),...this.globalAttributes}}setAttribute(s,e){s&&(this.globalAttributes[s]=e,this.refreshOpenSessionAttributes())}setAttributes(s){if(!s)return;let e=!1;for(const t of Object.keys(s))t&&(this.globalAttributes[t]=s[t],e=!0);e&&this.refreshOpenSessionAttributes()}removeAttribute(s){s in this.globalAttributes&&(delete this.globalAttributes[s],this.refreshOpenSessionAttributes())}clearAttributes(){Object.keys(this.globalAttributes).length!==0&&(this.globalAttributes={},this.refreshOpenSessionAttributes())}currentAttributes(){return{...this.globalAttributes}}refreshOpenSessionAttributes(){!this.sessionId||!this.sessionStartedAt||this.recordAllSessions===!0&&(this.pendingSessions.push({id:this.sessionId,startedAt:this.sessionStartedAt,attributes:this.composedSessionAttributes()}),this.scheduleSync())}endSession(){if(!(!this.sessionId||!this.sessionStartedAt)){if(this.recorder){const s=this.recorder.drainCurrent();s&&s.events.length>0&&this.queueSegment(s)}this.pendingSessions.push({id:this.sessionId,startedAt:this.sessionStartedAt,endedAt:J(),attributes:this.composedSessionAttributes()}),this.unloading?(this.debounceTimer!==null&&(clearTimeout(this.debounceTimer),this.debounceTimer=null),this.doSync()):this.scheduleSync()}}restartSession(){this.unloading=!1,this.beginSession()}handleSegmentReady(s){this.sessionId&&(this.queueSegment(s),this.scheduleSync())}queueSegment(s){if(!this.sessionId)return;const e=this.logs.snapshot(),t=this.actions.snapshot();e.length>0&&this.logs.clear(),t.length>0&&this.actions.clear();const r={sessionId:this.sessionId,segmentIndex:this.segmentIndex++,events:s.events,startedAt:s.startedAt,endedAt:s.endedAt};e.length>0&&(r.logs=e),t.length>0&&(r.actions=t),this.pendingRecordings.push(r)}currentSessionId(){return this.sessionId}recordLog(s,e){this.captureLogs&&this.logs.add({type:"log",timestamp:J(),level:s,message:e})}recordNetworkEvent(s){this.captureNetwork&&(this.isOwnReportUrl(s.url)||this.actions.add({type:"network",timestamp:s.timestamp??J(),method:s.method,url:s.url,durationMs:s.durationMs,statusCode:s.statusCode,requestBytes:s.requestBytes,responseBytes:s.responseBytes,error:s.error}))}isOwnReportUrl(s){return s?!!(s===this.apiUrl||s.startsWith(this.apiUrl+"?")||s.startsWith(this.apiUrl+"#")):!1}recordNavigationEvent(s){this.captureNavigation&&this.actions.add({type:"navigation",timestamp:s.timestamp??J(),action:s.action,from:s.from,to:s.to})}recordAction(s,e,t){this.actions.add({type:"custom",timestamp:J(),category:s,name:e,data:t})}bufferedLogs(){return this.logs.snapshot()}bufferedActions(){return this.actions.snapshot()}addException(s){if(this.shouldIgnore(s)){this.debug&&console.debug("Traceway: exception suppressed by filter",s.stackTrace.slice(0,120));return}s.attributes={...Fo(),...this.globalAttributes,...s.attributes??{}};const e=this.recorder&&this.recorder.hasSegments()?this.recorder.getSegments().flatMap(n=>n.events):[],t=this.logs.snapshot(),r=this.actions.snapshot(),i=e.length>0||t.length>0||r.length>0;if(this.recordAllSessions&&this.sessionId&&(s.sessionId=this.sessionId),i){const n=ce();s.sessionRecordingId=n;const o={exceptionId:n,events:e},l=Sf(e,t,r);l.startedAt&&(o.startedAt=l.startedAt),l.endedAt&&(o.endedAt=l.endedAt),t.length>0&&(o.logs=t),r.length>0&&(o.actions=r),this.pendingRecordings.push(o)}this.pendingExceptions.push(s),this.scheduleSync()}shouldIgnore(s){if(this.ignoreErrors.length>0){const e=s.stackTrace;for(const t of this.ignoreErrors)if(typeof t=="string"){if(e.includes(t))return!0}else if(t.lastIndex=0,t.test(e))return!0}if(this.beforeCapture!==null)try{if(this.beforeCapture(s)===!1)return!0}catch(e){this.debug&&console.error("Traceway: beforeCapture callback threw:",e)}return!1}scheduleSync(){this.debounceTimer!==null&&clearTimeout(this.debounceTimer),this.debounceTimer=setTimeout(()=>{this.debounceTimer=null,this.doSync()},this.debounceMs)}async doSync(){if(this.isSyncing||this.pendingExceptions.length===0&&this.pendingRecordings.length===0&&this.pendingSessions.length===0)return;this.isSyncing=!0;const s=this.pendingExceptions.splice(0),e=this.pendingRecordings.splice(0),t=this.pendingSessions.splice(0),i={collectionFrames:[{stackTraces:s,metrics:[],traces:[],sessionRecordings:e.length>0?e:void 0,sessions:t.length>0?t:void 0}],appVersion:this.version,serverName:""};let n=!1;try{await gf(this.apiUrl,this.token,JSON.stringify(i),this.unloading?{keepalive:!0}:void 0)||(n=!0,this.pendingExceptions.unshift(...s),this.pendingRecordings.unshift(...e),this.pendingSessions.unshift(...t),this.debug&&console.error("Traceway: sync failed, re-queued exceptions"))}catch(o){n=!0,this.pendingExceptions.unshift(...s),this.pendingRecordings.unshift(...e),this.pendingSessions.unshift(...t),this.debug&&console.error("Traceway: sync error:",o)}finally{this.isSyncing=!1,(this.pendingExceptions.length>0||this.pendingRecordings.length>0||this.pendingSessions.length>0)&&(n?this.scheduleRetry():this.doSync())}}scheduleRetry(){this.retryTimer===null&&(this.retryTimer=setTimeout(()=>{this.retryTimer=null,this.doSync()},this.retryDelayMs))}async flush(s){this.debounceTimer!==null&&(clearTimeout(this.debounceTimer),this.debounceTimer=null),this.retryTimer!==null&&(clearTimeout(this.retryTimer),this.retryTimer=null),this.lifecycle&&this.lifecycle.uninstall(),this.recordAllSessions&&this.endSession(),this.recorder&&this.recorder.stop();const e=this.doSync();s!==void 0?await Promise.race([e,new Promise(t=>setTimeout(t,s))]):await e}};function Bs(s){var r;const e=[],t=((r=s.constructor)==null?void 0:r.name)||"Error";if(e.push(`${t}: ${s.message}`),s.stack){const i=s.stack.split(`
-`);for(const n of i){const o=n.match(/^\s+at\s+(.+?)\s+\((.+):(\d+):(\d+)\)$/);if(o){const c=Uo(o[1]),h=Sr(o[2]);e.push(`${c}()`),e.push(` ${h}:${o[3]}:${o[4]}`);continue}const l=n.match(/^\s+at\s+(.+):(\d+):(\d+)$/);if(l){const c=Sr(l[1]);e.push("<anonymous>()"),e.push(` ${c}:${l[2]}:${l[3]}`);continue}const a=n.match(/^(.+)@(.+):(\d+):(\d+)$/);if(a){const c=Uo(a[1])||"<anonymous>",h=Sr(a[2]);e.push(`${c}()`),e.push(` ${h}:${a[3]}:${a[4]}`);continue}const u=n.match(/^@(.+):(\d+):(\d+)$/);if(u){const c=Sr(u[1]);e.push("<anonymous>()"),e.push(` ${c}:${u[2]}:${u[3]}`);continue}}}return e.join(`
+or you can use record.mirror to access the mirror instance during recording.`;let Qi={map:{},getId(){return console.error($e),-1},getNode(){return console.error($e),null},removeNodeFromMap(){console.error($e)},has(){return console.error($e),!1},reset(){console.error($e)}};typeof window<"u"&&window.Proxy&&window.Reflect&&(Qi=new Proxy(Qi,{get(e,t,r){return t==="map"&&console.error($e),Reflect.get(e,t,r)}}));function He(e,t,r={}){let o=null,s=0;return function(...u){const c=Date.now();!s&&r.leading===!1&&(s=c);const p=t-(c-s),l=this;p<=0||p>t?(o&&(clearTimeout(o),o=null),s=c,e.apply(l,u)):!o&&r.trailing!==!1&&(o=setTimeout(()=>{s=r.leading===!1?0:Date.now(),o=null,e.apply(l,u)},p))}}function bt(e,t,r,o,s=window){const u=s.Object.getOwnPropertyDescriptor(e,t);return s.Object.defineProperty(e,t,o?r:{set(c){setTimeout(()=>{r.set.call(this,c)},0),u&&u.set&&u.set.call(this,c)}}),()=>bt(e,t,u||{},!0)}function Zi(e){var t,r,o,s;const u=e.document;return{left:u.scrollingElement?u.scrollingElement.scrollLeft:e.pageXOffset!==void 0?e.pageXOffset:u.documentElement.scrollLeft||(u==null?void 0:u.body)&&((t=D.parentElement(u.body))==null?void 0:t.scrollLeft)||((r=u==null?void 0:u.body)==null?void 0:r.scrollLeft)||0,top:u.scrollingElement?u.scrollingElement.scrollTop:e.pageYOffset!==void 0?e.pageYOffset:(u==null?void 0:u.documentElement.scrollTop)||(u==null?void 0:u.body)&&((o=D.parentElement(u.body))==null?void 0:o.scrollTop)||((s=u==null?void 0:u.body)==null?void 0:s.scrollTop)||0}}function Ki(){return window.innerHeight||document.documentElement&&document.documentElement.clientHeight||document.body&&document.body.clientHeight}function Xi(){return window.innerWidth||document.documentElement&&document.documentElement.clientWidth||document.body&&document.body.clientWidth}function en(e){return e?e.nodeType===e.ELEMENT_NODE?e:D.parentElement(e):null}function se(e,t,r,o){if(!e)return!1;const s=en(e);if(!s)return!1;try{if(typeof t=="string"){if(s.classList.contains(t)||o&&s.closest("."+t)!==null)return!0}else if(st(s,t,o))return!0}catch{}return!!(r&&(s.matches(r)||o&&s.closest(r)!==null))}function ga(e,t){return t.getId(e)!==-1}function Kr(e,t,r){return e.tagName==="TITLE"&&r.headTitleMutations?!0:t.getId(e)===rt}function tn(e,t){if(We(e))return!1;const r=t.getId(e);if(!t.has(r))return!0;const o=D.parentNode(e);return o&&o.nodeType===e.DOCUMENT_NODE?!1:o?tn(o,t):!0}function Xr(e){return!!e.changedTouches}function ya(e=window){"NodeList"in e&&!e.NodeList.prototype.forEach&&(e.NodeList.prototype.forEach=Array.prototype.forEach),"DOMTokenList"in e&&!e.DOMTokenList.prototype.forEach&&(e.DOMTokenList.prototype.forEach=Array.prototype.forEach)}function rn(e,t){return!!(e.nodeName==="IFRAME"&&t.getMeta(e))}function sn(e,t){return!!(e.nodeName==="LINK"&&e.nodeType===e.ELEMENT_NODE&&e.getAttribute&&e.getAttribute("rel")==="stylesheet"&&t.getMeta(e))}function es(e){return e?e instanceof Jr&&"shadowRoot"in e?!!e.shadowRoot:!!D.shadowRoot(e):!1}class wa{constructor(){A(this,"id",1),A(this,"styleIDMap",new WeakMap),A(this,"idStyleMap",new Map)}getId(t){return this.styleIDMap.get(t)??-1}has(t){return this.styleIDMap.has(t)}add(t,r){if(this.has(t))return this.getId(t);let o;return r===void 0?o=this.id++:o=r,this.styleIDMap.set(t,o),this.idStyleMap.set(o,t),o}getStyle(t){return this.idStyleMap.get(t)||null}reset(){this.styleIDMap=new WeakMap,this.idStyleMap=new Map,this.id=1}generateId(){return this.id++}}function nn(e){var t;let r=null;return"getRootNode"in e&&((t=D.getRootNode(e))==null?void 0:t.nodeType)===Node.DOCUMENT_FRAGMENT_NODE&&D.host(D.getRootNode(e))&&(r=D.host(D.getRootNode(e))),r}function ba(e){let t=e,r;for(;r=nn(t);)t=r;return t}function Sa(e){const t=D.ownerDocument(e);if(!t)return!1;const r=ba(e);return D.contains(t,r)}function on(e){const t=D.ownerDocument(e);return t?D.contains(t,e)||Sa(e):!1}var F=(e=>(e[e.DomContentLoaded=0]="DomContentLoaded",e[e.Load=1]="Load",e[e.FullSnapshot=2]="FullSnapshot",e[e.IncrementalSnapshot=3]="IncrementalSnapshot",e[e.Meta=4]="Meta",e[e.Custom=5]="Custom",e[e.Plugin=6]="Plugin",e[e.Asset=7]="Asset",e))(F||{}),L=(e=>(e[e.Mutation=0]="Mutation",e[e.MouseMove=1]="MouseMove",e[e.MouseInteraction=2]="MouseInteraction",e[e.Scroll=3]="Scroll",e[e.ViewportResize=4]="ViewportResize",e[e.Input=5]="Input",e[e.TouchMove=6]="TouchMove",e[e.MediaInteraction=7]="MediaInteraction",e[e.StyleSheetRule=8]="StyleSheetRule",e[e.CanvasMutation=9]="CanvasMutation",e[e.Font=10]="Font",e[e.Log=11]="Log",e[e.Drag=12]="Drag",e[e.StyleDeclaration=13]="StyleDeclaration",e[e.Selection=14]="Selection",e[e.AdoptedStyleSheet=15]="AdoptedStyleSheet",e[e.CustomElement=16]="CustomElement",e))(L||{}),ne=(e=>(e[e.MouseUp=0]="MouseUp",e[e.MouseDown=1]="MouseDown",e[e.Click=2]="Click",e[e.ContextMenu=3]="ContextMenu",e[e.DblClick=4]="DblClick",e[e.Focus=5]="Focus",e[e.Blur=6]="Blur",e[e.TouchStart=7]="TouchStart",e[e.TouchMove_Departed=8]="TouchMove_Departed",e[e.TouchEnd=9]="TouchEnd",e[e.TouchCancel=10]="TouchCancel",e))(ne||{}),xe=(e=>(e[e.Mouse=0]="Mouse",e[e.Pen=1]="Pen",e[e.Touch=2]="Touch",e))(xe||{}),Fe=(e=>(e[e["2D"]=0]="2D",e[e.WebGL=1]="WebGL",e[e.WebGL2=2]="WebGL2",e))(Fe||{}),Ue=(e=>(e[e.Play=0]="Play",e[e.Pause=1]="Pause",e[e.Seeked=2]="Seeked",e[e.VolumeChange=3]="VolumeChange",e[e.RateChange=4]="RateChange",e))(Ue||{}),an=(e=>(e[e.Document=0]="Document",e[e.DocumentType=1]="DocumentType",e[e.Element=2]="Element",e[e.Text=3]="Text",e[e.CDATA=4]="CDATA",e[e.Comment=5]="Comment",e))(an||{});const ln=(e,t)=>`${e}@${t}`;class va{constructor(){A(this,"frozen",!1),A(this,"locked",!1),A(this,"texts",[]),A(this,"attributes",[]),A(this,"attributeMap",new WeakMap),A(this,"removes",[]),A(this,"mapRemoves",[]),A(this,"movedMap",{}),A(this,"addedSet",new Set),A(this,"movedSet",new Set),A(this,"droppedSet",new Set),A(this,"removesSubTreeCache",new Set),A(this,"mutationCb"),A(this,"blockClass"),A(this,"blockSelector"),A(this,"maskTextClass"),A(this,"maskTextSelector"),A(this,"inlineStylesheet"),A(this,"maskInputOptions"),A(this,"maskTextFn"),A(this,"maskInputFn"),A(this,"keepIframeSrcFn"),A(this,"recordCanvas"),A(this,"inlineImages"),A(this,"slimDOMOptions"),A(this,"dataURLOptions"),A(this,"doc"),A(this,"mirror"),A(this,"iframeManager"),A(this,"stylesheetManager"),A(this,"shadowDomManager"),A(this,"canvasManager"),A(this,"processedNodeManager"),A(this,"unattachedDoc"),A(this,"processMutations",t=>{t.forEach(this.processMutation),this.emit()}),A(this,"emit",()=>{if(this.frozen||this.locked)return;const t=[],r=new Set;for(;this.mapRemoves.length;)this.mirror.removeNodeFromMap(this.mapRemoves.shift());for(const a of this.movedSet){const n=D.parentNode(a);this.removesSubTreeCache.has(n)&&!this.movedSet.has(n)||this.addedSet.add(a)}let o=null,s=null,u=-1,c=null,p=!1;const l=new Set,d=this.addedSet.values();let i=d.next();for(;this.addedSet.size;){if(o!==null&&this.addedSet.has(D.previousSibling(o)))c=o,o=D.previousSibling(o);else{for(this.addedSet.has(i.value)||(i=d.next()),o=i.value;;){if(s=D.parentNode(o),this.addedSet.has(s)){o=s;continue}break}if(l.has(s))s=null;else if(s){if(on(s)?(p=un(this.droppedSet,s)||this.removesSubTreeCache.has(s),p&&un(this.movedSet,o)&&(p=!1)):p=!0,this.addedSet.has(D.lastChild(s)))o=D.lastChild(s),c=null;else for(;;){if(c=D.nextSibling(o),this.addedSet.has(c)){o=c;continue}break}if(u=We(s)?this.mirror.getId(nn(o)):this.mirror.getId(s),u===-1&&s.nodeType===Node.DOCUMENT_FRAGMENT_NODE){const h=D.host(s);u=this.mirror.getId(h)}}}if(this.addedSet.delete(o),!s||u===-1){l.add(o);continue}else if(p){this.droppedSet.add(o);continue}let a=!1;if(o.nodeType===Node.TEXT_NODE){const h=s.tagName;if(h==="TEXTAREA")continue;h==="STYLE"&&r.has(u)&&(a=!0)}let n=c?this.mirror.getId(c):null;for(;n===rt;)c=c&&D.nextSibling(c),n=c&&this.mirror.getId(c);if(n===-1){console.warn("Couldn't record new node. Couldn't find mirror id for nextSibling:",c),o=null;continue}const m=_e(o,{doc:this.doc,mirror:this.mirror,blockClass:this.blockClass,blockSelector:this.blockSelector,maskTextClass:this.maskTextClass,maskTextSelector:this.maskTextSelector,skipChild:!0,newlyAddedElement:!0,inlineStylesheet:this.inlineStylesheet,maskInputOptions:this.maskInputOptions,maskTextFn:this.maskTextFn,maskInputFn:this.maskInputFn,slimDOMOptions:this.slimDOMOptions,dataURLOptions:this.dataURLOptions,recordCanvas:this.recordCanvas,inlineImages:this.inlineImages,onSerialize:h=>{rn(h,this.mirror)&&this.iframeManager.addIframe(h),sn(h,this.mirror)&&this.stylesheetManager.trackLinkElement(h),es(o)&&this.shadowDomManager.addShadowRoot(D.shadowRoot(o),this.doc)},onIframeLoad:(h,g)=>{this.iframeManager.attachIframe(h,g),this.shadowDomManager.observeAttachShadow(h)},onStylesheetLoad:(h,g)=>{this.stylesheetManager.attachLinkElement(h,g)},cssCaptured:a});m&&(t.push({parentId:u,nextId:n,node:m}),r.add(m.id))}const f={texts:this.texts.map(a=>{const n=a.node,m=D.parentNode(n);return m&&m.tagName==="TEXTAREA"&&this.genTextAreaValueMutation(m),{id:this.mirror.getId(n),value:a.value}}).filter(a=>!r.has(a.id)).filter(a=>this.mirror.has(a.id)),attributes:this.attributes.map(a=>{const{attributes:n}=a;if(typeof n.style=="string"){const m=JSON.stringify(a.styleDiff),h=JSON.stringify(a._unchangedStyles);m.length<n.style.length&&(m+h).split("var(").length===n.style.split("var(").length&&(n.style=a.styleDiff)}return{id:this.mirror.getId(a.node),attributes:n}}).filter(a=>!r.has(a.id)).filter(a=>this.mirror.has(a.id)),removes:this.removes,adds:t};!f.texts.length&&!f.attributes.length&&!f.removes.length&&!f.adds.length||(this.texts=[],this.attributes=[],this.attributeMap=new WeakMap,this.removes=[],this.addedSet=new Set,this.movedSet=new Set,this.droppedSet=new Set,this.removesSubTreeCache=new Set,this.movedMap={},this.mutationCb(f))}),A(this,"genTextAreaValueMutation",t=>{let r=this.attributeMap.get(t);r||(r={node:t,attributes:{},styleDiff:{},_unchangedStyles:{}},this.attributes.push(r),this.attributeMap.set(t,r));const o=Array.from(D.childNodes(t),s=>D.textContent(s)||"").join("");r.attributes.value=Ke({element:t,maskInputOptions:this.maskInputOptions,tagName:t.tagName,type:Xe(t),value:o,maskInputFn:this.maskInputFn})}),A(this,"processMutation",t=>{if(!Kr(t.target,this.mirror,this.slimDOMOptions))switch(t.type){case"characterData":{const r=D.textContent(t.target);!se(t.target,this.blockClass,this.blockSelector,!1)&&r!==t.oldValue&&this.texts.push({value:Os(t.target,this.maskTextClass,this.maskTextSelector,!0)&&r?this.maskTextFn?this.maskTextFn(r,en(t.target)):r.replace(/[\S]/g,"*"):r,node:t.target});break}case"attributes":{const r=t.target,o=Re(r.tagName);let s=t.attributeName,u=t.target.getAttribute(s);if(s==="value"){const p=Xe(r);u=Ke({element:r,maskInputOptions:this.maskInputOptions,tagName:r.tagName,type:p,value:u,maskInputFn:this.maskInputFn})}if(se(t.target,this.blockClass,this.blockSelector,!1)||u===t.oldValue)return;let c=this.attributeMap.get(t.target);if(o==="iframe"&&s==="src"&&!this.keepIframeSrcFn(u))if(!r.contentDocument)s="rr_src";else return;if(c||(c={node:t.target,attributes:{},styleDiff:{},_unchangedStyles:{}},this.attributes.push(c),this.attributeMap.set(t.target,c)),s==="type"&&o==="input"&&(t.oldValue||"").toLowerCase()==="password"&&r.setAttribute("data-rr-is-password","true"),!Rs(o,s))if(c.attributes[s]=xs(this.doc,o,Re(s),u),s==="style"){if(!this.unattachedDoc)try{this.unattachedDoc=document.implementation.createHTMLDocument()}catch{this.unattachedDoc=this.doc}const p=this.unattachedDoc.createElement("span");t.oldValue&&p.setAttribute("style",t.oldValue);for(const l of Array.from(r.style)){const d=r.style.getPropertyValue(l),i=r.style.getPropertyPriority(l);d!==p.style.getPropertyValue(l)||i!==p.style.getPropertyPriority(l)?i===""?c.styleDiff[l]=d:c.styleDiff[l]=[d,i]:c._unchangedStyles[l]=[d,i]}for(const l of Array.from(p.style))r.style.getPropertyValue(l)===""&&(c.styleDiff[l]=!1)}else s==="open"&&r.tagName==="DIALOG"&&(r.matches("dialog:modal")?c.attributes.rr_open_mode="modal":c.attributes.rr_open_mode="non-modal");break}case"childList":{if(se(t.target,this.blockClass,this.blockSelector,!0))return;if(t.target.tagName==="TEXTAREA"){this.genTextAreaValueMutation(t.target);return}t.addedNodes.forEach(r=>this.genAdds(r,t.target)),t.removedNodes.forEach(r=>{const o=this.mirror.getId(r),s=We(t.target)?this.mirror.getId(D.host(t.target)):this.mirror.getId(t.target);se(t.target,this.blockClass,this.blockSelector,!1)||Kr(r,this.mirror,this.slimDOMOptions)||!ga(r,this.mirror)||(this.addedSet.has(r)?(ts(this.addedSet,r),this.droppedSet.add(r)):this.addedSet.has(t.target)&&o===-1||tn(t.target,this.mirror)||(this.movedSet.has(r)&&this.movedMap[ln(o,s)]?ts(this.movedSet,r):(this.removes.push({parentId:s,id:o,isShadow:We(t.target)&&qe(t.target)?!0:void 0}),Ca(r,this.removesSubTreeCache))),this.mapRemoves.push(r))});break}}}),A(this,"genAdds",(t,r)=>{if(!this.processedNodeManager.inOtherBuffer(t,this)&&!(this.addedSet.has(t)||this.movedSet.has(t))){if(this.mirror.hasNode(t)){if(Kr(t,this.mirror,this.slimDOMOptions))return;this.movedSet.add(t);let o=null;r&&this.mirror.hasNode(r)&&(o=this.mirror.getId(r)),o&&o!==-1&&(this.movedMap[ln(this.mirror.getId(t),o)]=!0)}else this.addedSet.add(t),this.droppedSet.delete(t);se(t,this.blockClass,this.blockSelector,!1)||(D.childNodes(t).forEach(o=>this.genAdds(o)),es(t)&&D.childNodes(D.shadowRoot(t)).forEach(o=>{this.processedNodeManager.add(o,this),this.genAdds(o,t)}))}})}init(t){["mutationCb","blockClass","blockSelector","maskTextClass","maskTextSelector","inlineStylesheet","maskInputOptions","maskTextFn","maskInputFn","keepIframeSrcFn","recordCanvas","inlineImages","slimDOMOptions","dataURLOptions","doc","mirror","iframeManager","stylesheetManager","shadowDomManager","canvasManager","processedNodeManager"].forEach(r=>{this[r]=t[r]})}freeze(){this.frozen=!0,this.canvasManager.freeze()}unfreeze(){this.frozen=!1,this.canvasManager.unfreeze(),this.emit()}isFrozen(){return this.frozen}lock(){this.locked=!0,this.canvasManager.lock()}unlock(){this.locked=!1,this.canvasManager.unlock(),this.emit()}reset(){this.shadowDomManager.reset(),this.canvasManager.reset()}}function ts(e,t){e.delete(t),D.childNodes(t).forEach(r=>ts(e,r))}function Ca(e,t){const r=[e];for(;r.length;){const o=r.pop();t.has(o)||(t.add(o),D.childNodes(o).forEach(s=>r.push(s)))}}function un(e,t){return e.size===0?!1:cn(e,t)}function cn(e,t){if(e.has(t))return!0;const r=D.parentNode(t);return r?cn(e,r):!1}let Je;function xa(e){Je=e}function Ra(){Je=void 0}const _=e=>Je?(...r)=>{try{return e(...r)}catch(o){if(Je&&Je(o)===!0)return;throw o}}:e,ke=[];function Ye(e){try{if("composedPath"in e){const t=e.composedPath();if(t.length)return t[0]}else if("path"in e&&e.path.length)return e.path[0]}catch{}return e&&e.target}function hn(e,t){const r=new va;ke.push(r),r.init(e);const[o,s]=Yi(),u=new o(_(r.processMutations.bind(r)));return u.observe(t,{attributes:!0,attributeOldValue:!0,characterData:!0,characterDataOldValue:!0,childList:!0,subtree:!0}),[u,s]}function Oa({mousemoveCb:e,sampling:t,doc:r,mirror:o}){if(t.mousemove===!1)return()=>{};const s=typeof t.mousemove=="number"?t.mousemove:50,u=typeof t.mousemoveCallback=="number"?t.mousemoveCallback:500;let c=[],p;const l=He(_(f=>{const a=Date.now()-p;e(c.map(n=>(n.timeOffset-=a,n)),f),c=[],p=null}),u),d=_(He(_(f=>{const a=Ye(f),{clientX:n,clientY:m}=Xr(f)?f.changedTouches[0]:f;p||(p=Ve()),c.push({x:n,y:m,id:o.getId(a),timeOffset:Ve()-p}),l(typeof DragEvent<"u"&&f instanceof DragEvent?L.Drag:f instanceof MouseEvent?L.MouseMove:L.TouchMove)}),s,{trailing:!1})),i=[re("mousemove",d,r),re("touchmove",d,r),re("drag",d,r)];return _(()=>{i.forEach(f=>f())})}function Ma({mouseInteractionCb:e,doc:t,mirror:r,blockClass:o,blockSelector:s,sampling:u}){if(u.mouseInteraction===!1)return()=>{};const c=u.mouseInteraction===!0||u.mouseInteraction===void 0?{}:u.mouseInteraction,p=[];let l=null;const d=i=>f=>{const a=Ye(f);if(se(a,o,s,!0))return;let n=null,m=i;if("pointerType"in f){switch(f.pointerType){case"mouse":n=xe.Mouse;break;case"touch":n=xe.Touch;break;case"pen":n=xe.Pen;break}n===xe.Touch?ne[i]===ne.MouseDown?m="TouchStart":ne[i]===ne.MouseUp&&(m="TouchEnd"):xe.Pen}else Xr(f)&&(n=xe.Touch);n!==null?(l=n,(m.startsWith("Touch")&&n===xe.Touch||m.startsWith("Mouse")&&n===xe.Mouse)&&(n=null)):ne[i]===ne.Click&&(n=l,l=null);const h=Xr(f)?f.changedTouches[0]:f;if(!h)return;const g=r.getId(a),{clientX:v,clientY:b}=h;_(e)({type:ne[m],id:g,x:v,y:b,...n!==null&&{pointerType:n}})};return Object.keys(ne).filter(i=>Number.isNaN(Number(i))&&!i.endsWith("_Departed")&&c[i]!==!1).forEach(i=>{let f=Re(i);const a=d(i);if(window.PointerEvent)switch(ne[i]){case ne.MouseDown:case ne.MouseUp:f=f.replace("mouse","pointer");break;case ne.TouchStart:case ne.TouchEnd:return}p.push(re(f,a,t))}),_(()=>{p.forEach(i=>i())})}function fn({scrollCb:e,doc:t,mirror:r,blockClass:o,blockSelector:s,sampling:u}){const c=_(He(_(p=>{const l=Ye(p);if(!l||se(l,o,s,!0))return;const d=r.getId(l);if(l===t&&t.defaultView){const i=Zi(t.defaultView);e({id:d,x:i.left,y:i.top})}else e({id:d,x:l.scrollLeft,y:l.scrollTop})}),u.scroll||100));return re("scroll",c,t)}function Ea({viewportResizeCb:e},{win:t}){let r=-1,o=-1;const s=_(He(_(()=>{const u=Ki(),c=Xi();(r!==u||o!==c)&&(e({width:Number(c),height:Number(u)}),r=u,o=c)}),200));return re("resize",s,t)}const Ia=["INPUT","TEXTAREA","SELECT"],dn=new WeakMap;function Aa({inputCb:e,doc:t,mirror:r,blockClass:o,blockSelector:s,ignoreClass:u,ignoreSelector:c,maskInputOptions:p,maskInputFn:l,sampling:d,userTriggeredOnInput:i}){function f(b){let S=Ye(b);const x=b.isTrusted,w=S&&S.tagName;if(S&&w==="OPTION"&&(S=D.parentElement(S)),!S||!w||Ia.indexOf(w)<0||se(S,o,s,!0)||S.classList.contains(u)||c&&S.matches(c))return;let y=S.value,C=!1;const O=Xe(S)||"";O==="radio"||O==="checkbox"?C=S.checked:(p[w.toLowerCase()]||p[O])&&(y=Ke({element:S,maskInputOptions:p,tagName:w,type:O,value:y,maskInputFn:l})),a(S,i?{text:y,isChecked:C,userTriggered:x}:{text:y,isChecked:C});const I=S.name;O==="radio"&&I&&C&&t.querySelectorAll(`input[type="radio"][name="${I}"]`).forEach(M=>{if(M!==S){const P=M.value;a(M,i?{text:P,isChecked:!C,userTriggered:!1}:{text:P,isChecked:!C})}})}function a(b,S){const x=dn.get(b);if(!x||x.text!==S.text||x.isChecked!==S.isChecked){dn.set(b,S);const w=r.getId(b);_(e)({...S,id:w})}}const m=(d.input==="last"?["change"]:["input","change"]).map(b=>re(b,_(f),t)),h=t.defaultView;if(!h)return()=>{m.forEach(b=>b())};const g=h.Object.getOwnPropertyDescriptor(h.HTMLInputElement.prototype,"value"),v=[[h.HTMLInputElement.prototype,"value"],[h.HTMLInputElement.prototype,"checked"],[h.HTMLSelectElement.prototype,"value"],[h.HTMLTextAreaElement.prototype,"value"],[h.HTMLSelectElement.prototype,"selectedIndex"],[h.HTMLOptionElement.prototype,"selected"]];return g&&g.set&&m.push(...v.map(b=>bt(b[0],b[1],{set(){_(f)({target:this,isTrusted:!1})}},!1,h))),_(()=>{m.forEach(b=>b())})}function St(e){const t=[];function r(o,s){if(vt("CSSGroupingRule")&&o.parentRule instanceof CSSGroupingRule||vt("CSSMediaRule")&&o.parentRule instanceof CSSMediaRule||vt("CSSSupportsRule")&&o.parentRule instanceof CSSSupportsRule||vt("CSSConditionRule")&&o.parentRule instanceof CSSConditionRule){const c=Array.from(o.parentRule.cssRules).indexOf(o);return s.unshift(c),r(o.parentRule,s)}else if(o.parentStyleSheet){const c=Array.from(o.parentStyleSheet.cssRules).indexOf(o);s.unshift(c)}return s}return r(e,t)}function Oe(e,t,r){let o,s;return e?(e.ownerNode?o=t.getId(e.ownerNode):s=r.getId(e),{styleId:s,id:o}):{}}function Na({styleSheetRuleCb:e,mirror:t,stylesheetManager:r},{win:o}){if(!o.CSSStyleSheet||!o.CSSStyleSheet.prototype)return()=>{};const s=o.CSSStyleSheet.prototype.insertRule;o.CSSStyleSheet.prototype.insertRule=new Proxy(s,{apply:_((i,f,a)=>{const[n,m]=a,{id:h,styleId:g}=Oe(f,t,r.styleMirror);return(h&&h!==-1||g&&g!==-1)&&e({id:h,styleId:g,adds:[{rule:n,index:m}]}),i.apply(f,a)})}),o.CSSStyleSheet.prototype.addRule=function(i,f,a=this.cssRules.length){const n=`${i} { ${f} }`;return o.CSSStyleSheet.prototype.insertRule.apply(this,[n,a])};const u=o.CSSStyleSheet.prototype.deleteRule;o.CSSStyleSheet.prototype.deleteRule=new Proxy(u,{apply:_((i,f,a)=>{const[n]=a,{id:m,styleId:h}=Oe(f,t,r.styleMirror);return(m&&m!==-1||h&&h!==-1)&&e({id:m,styleId:h,removes:[{index:n}]}),i.apply(f,a)})}),o.CSSStyleSheet.prototype.removeRule=function(i){return o.CSSStyleSheet.prototype.deleteRule.apply(this,[i])};let c;o.CSSStyleSheet.prototype.replace&&(c=o.CSSStyleSheet.prototype.replace,o.CSSStyleSheet.prototype.replace=new Proxy(c,{apply:_((i,f,a)=>{const[n]=a,{id:m,styleId:h}=Oe(f,t,r.styleMirror);return(m&&m!==-1||h&&h!==-1)&&e({id:m,styleId:h,replace:n}),i.apply(f,a)})}));let p;o.CSSStyleSheet.prototype.replaceSync&&(p=o.CSSStyleSheet.prototype.replaceSync,o.CSSStyleSheet.prototype.replaceSync=new Proxy(p,{apply:_((i,f,a)=>{const[n]=a,{id:m,styleId:h}=Oe(f,t,r.styleMirror);return(m&&m!==-1||h&&h!==-1)&&e({id:m,styleId:h,replaceSync:n}),i.apply(f,a)})}));const l={};Ct("CSSGroupingRule")?l.CSSGroupingRule=o.CSSGroupingRule:(Ct("CSSMediaRule")&&(l.CSSMediaRule=o.CSSMediaRule),Ct("CSSConditionRule")&&(l.CSSConditionRule=o.CSSConditionRule),Ct("CSSSupportsRule")&&(l.CSSSupportsRule=o.CSSSupportsRule));const d={};return Object.entries(l).forEach(([i,f])=>{d[i]={insertRule:f.prototype.insertRule,deleteRule:f.prototype.deleteRule},f.prototype.insertRule=new Proxy(d[i].insertRule,{apply:_((a,n,m)=>{const[h,g]=m,{id:v,styleId:b}=Oe(n.parentStyleSheet,t,r.styleMirror);return(v&&v!==-1||b&&b!==-1)&&e({id:v,styleId:b,adds:[{rule:h,index:[...St(n),g||0]}]}),a.apply(n,m)})}),f.prototype.deleteRule=new Proxy(d[i].deleteRule,{apply:_((a,n,m)=>{const[h]=m,{id:g,styleId:v}=Oe(n.parentStyleSheet,t,r.styleMirror);return(g&&g!==-1||v&&v!==-1)&&e({id:g,styleId:v,removes:[{index:[...St(n),h]}]}),a.apply(n,m)})})}),_(()=>{o.CSSStyleSheet.prototype.insertRule=s,o.CSSStyleSheet.prototype.deleteRule=u,c&&(o.CSSStyleSheet.prototype.replace=c),p&&(o.CSSStyleSheet.prototype.replaceSync=p),Object.entries(l).forEach(([i,f])=>{f.prototype.insertRule=d[i].insertRule,f.prototype.deleteRule=d[i].deleteRule})})}function pn({mirror:e,stylesheetManager:t},r){var o,s,u;let c=null;r.nodeName==="#document"?c=e.getId(r):c=e.getId(D.host(r));const p=r.nodeName==="#document"?(o=r.defaultView)==null?void 0:o.Document:(u=(s=r.ownerDocument)==null?void 0:s.defaultView)==null?void 0:u.ShadowRoot,l=p!=null&&p.prototype?Object.getOwnPropertyDescriptor(p==null?void 0:p.prototype,"adoptedStyleSheets"):void 0;return c===null||c===-1||!p||!l?()=>{}:(Object.defineProperty(r,"adoptedStyleSheets",{configurable:l.configurable,enumerable:l.enumerable,get(){var d;return(d=l.get)==null?void 0:d.call(this)},set(d){var i;const f=(i=l.set)==null?void 0:i.call(this,d);if(c!==null&&c!==-1)try{t.adoptStyleSheets(d,c)}catch{}return f}}),_(()=>{Object.defineProperty(r,"adoptedStyleSheets",{configurable:l.configurable,enumerable:l.enumerable,get:l.get,set:l.set})}))}function ka({styleDeclarationCb:e,mirror:t,ignoreCSSAttributes:r,stylesheetManager:o},{win:s}){const u=s.CSSStyleDeclaration.prototype.setProperty;s.CSSStyleDeclaration.prototype.setProperty=new Proxy(u,{apply:_((p,l,d)=>{var i;const[f,a,n]=d;if(r.has(f))return u.apply(l,[f,a,n]);const{id:m,styleId:h}=Oe((i=l.parentRule)==null?void 0:i.parentStyleSheet,t,o.styleMirror);return(m&&m!==-1||h&&h!==-1)&&e({id:m,styleId:h,set:{property:f,value:a,priority:n},index:St(l.parentRule)}),p.apply(l,d)})});const c=s.CSSStyleDeclaration.prototype.removeProperty;return s.CSSStyleDeclaration.prototype.removeProperty=new Proxy(c,{apply:_((p,l,d)=>{var i;const[f]=d;if(r.has(f))return c.apply(l,[f]);const{id:a,styleId:n}=Oe((i=l.parentRule)==null?void 0:i.parentStyleSheet,t,o.styleMirror);return(a&&a!==-1||n&&n!==-1)&&e({id:a,styleId:n,remove:{property:f},index:St(l.parentRule)}),p.apply(l,d)})}),_(()=>{s.CSSStyleDeclaration.prototype.setProperty=u,s.CSSStyleDeclaration.prototype.removeProperty=c})}function Pa({mediaInteractionCb:e,blockClass:t,blockSelector:r,mirror:o,sampling:s,doc:u}){const c=_(l=>He(_(d=>{const i=Ye(d);if(!i||se(i,t,r,!0))return;const{currentTime:f,volume:a,muted:n,playbackRate:m,loop:h}=i;e({type:l,id:o.getId(i),currentTime:f,volume:a,muted:n,playbackRate:m,loop:h})}),s.media||500)),p=[re("play",c(Ue.Play),u),re("pause",c(Ue.Pause),u),re("seeked",c(Ue.Seeked),u),re("volumechange",c(Ue.VolumeChange),u),re("ratechange",c(Ue.RateChange),u)];return _(()=>{p.forEach(l=>l())})}function Ta({fontCb:e,doc:t}){const r=t.defaultView;if(!r)return()=>{};const o=[],s=new WeakMap,u=r.FontFace;r.FontFace=function(l,d,i){const f=new u(l,d,i);return s.set(f,{family:l,buffer:typeof d!="string",descriptors:i,fontSource:typeof d=="string"?d:JSON.stringify(Array.from(new Uint8Array(d)))}),f};const c=Ne(t.fonts,"add",function(p){return function(l){return setTimeout(_(()=>{const d=s.get(l);d&&(e(d),s.delete(l))}),0),p.apply(this,[l])}});return o.push(()=>{r.FontFace=u}),o.push(c),_(()=>{o.forEach(p=>p())})}function Da(e){const{doc:t,mirror:r,blockClass:o,blockSelector:s,selectionCb:u}=e;let c=!0;const p=_(()=>{const l=t.getSelection();if(!l||c&&(l!=null&&l.isCollapsed))return;c=l.isCollapsed||!1;const d=[],i=l.rangeCount||0;for(let f=0;f<i;f++){const a=l.getRangeAt(f),{startContainer:n,startOffset:m,endContainer:h,endOffset:g}=a;se(n,o,s,!0)||se(h,o,s,!0)||d.push({start:r.getId(n),startOffset:m,end:r.getId(h),endOffset:g})}u({ranges:d})});return p(),re("selectionchange",p)}function La({doc:e,customElementCb:t}){const r=e.defaultView;return!r||!r.customElements?()=>{}:Ne(r.customElements,"define",function(s){return function(u,c,p){try{t({define:{name:u}})}catch{console.warn(`Custom element callback failed for ${u}`)}return s.apply(this,[u,c,p])}})}function _a(e,t){const{mutationCb:r,mousemoveCb:o,mouseInteractionCb:s,scrollCb:u,viewportResizeCb:c,inputCb:p,mediaInteractionCb:l,styleSheetRuleCb:d,styleDeclarationCb:i,canvasMutationCb:f,fontCb:a,selectionCb:n,customElementCb:m}=e;e.mutationCb=(...h)=>{t.mutation&&t.mutation(...h),r(...h)},e.mousemoveCb=(...h)=>{t.mousemove&&t.mousemove(...h),o(...h)},e.mouseInteractionCb=(...h)=>{t.mouseInteraction&&t.mouseInteraction(...h),s(...h)},e.scrollCb=(...h)=>{t.scroll&&t.scroll(...h),u(...h)},e.viewportResizeCb=(...h)=>{t.viewportResize&&t.viewportResize(...h),c(...h)},e.inputCb=(...h)=>{t.input&&t.input(...h),p(...h)},e.mediaInteractionCb=(...h)=>{t.mediaInteaction&&t.mediaInteaction(...h),l(...h)},e.styleSheetRuleCb=(...h)=>{t.styleSheetRule&&t.styleSheetRule(...h),d(...h)},e.styleDeclarationCb=(...h)=>{t.styleDeclaration&&t.styleDeclaration(...h),i(...h)},e.canvasMutationCb=(...h)=>{t.canvasMutation&&t.canvasMutation(...h),f(...h)},e.fontCb=(...h)=>{t.font&&t.font(...h),a(...h)},e.selectionCb=(...h)=>{t.selection&&t.selection(...h),n(...h)},e.customElementCb=(...h)=>{t.customElement&&t.customElement(...h),m(...h)}}function $a(e,t={}){const r=e.doc.defaultView;if(!r)return()=>{};_a(e,t);let o,s=()=>{};e.recordDOM&&([o,s]=hn(e,e.doc));const u=Oa(e),c=Ma(e),p=fn(e),l=Ea(e,{win:r}),d=Aa(e),i=Pa(e);let f=()=>{},a=()=>{},n=()=>{},m=()=>{};e.recordDOM&&(f=Na(e,{win:r}),a=pn(e,e.doc),n=ka(e,{win:r}),e.collectFonts&&(m=Ta(e)));const h=Da(e),g=La(e),v=[];for(const b of e.plugins)v.push(b.observer(b.callback,r,b.options));return _(()=>{ke.forEach(b=>b.reset()),o==null||o.disconnect(),s(),u(),c(),p(),l(),d(),i(),f(),a(),n(),m(),h(),g(),v.forEach(b=>b())})}function vt(e){return typeof window[e]<"u"}function Ct(e){return!!(typeof window[e]<"u"&&window[e].prototype&&"insertRule"in window[e].prototype&&"deleteRule"in window[e].prototype)}class mn{constructor(t){A(this,"iframeIdToRemoteIdMap",new WeakMap),A(this,"iframeRemoteIdToIdMap",new WeakMap),this.generateIdFn=t}getId(t,r,o,s){const u=o||this.getIdToRemoteIdMap(t),c=s||this.getRemoteIdToIdMap(t);let p=u.get(r);return p||(p=this.generateIdFn(),u.set(r,p),c.set(p,r)),p}getIds(t,r){const o=this.getIdToRemoteIdMap(t),s=this.getRemoteIdToIdMap(t);return r.map(u=>this.getId(t,u,o,s))}getRemoteId(t,r,o){const s=o||this.getRemoteIdToIdMap(t);if(typeof r!="number")return r;const u=s.get(r);return u||-1}getRemoteIds(t,r){const o=this.getRemoteIdToIdMap(t);return r.map(s=>this.getRemoteId(t,s,o))}reset(t){if(!t){this.iframeIdToRemoteIdMap=new WeakMap,this.iframeRemoteIdToIdMap=new WeakMap;return}this.iframeIdToRemoteIdMap.delete(t),this.iframeRemoteIdToIdMap.delete(t)}getIdToRemoteIdMap(t){let r=this.iframeIdToRemoteIdMap.get(t);return r||(r=new Map,this.iframeIdToRemoteIdMap.set(t,r)),r}getRemoteIdToIdMap(t){let r=this.iframeRemoteIdToIdMap.get(t);return r||(r=new Map,this.iframeRemoteIdToIdMap.set(t,r)),r}}class Fa{constructor(t){A(this,"iframes",new WeakMap),A(this,"crossOriginIframeMap",new WeakMap),A(this,"crossOriginIframeMirror",new mn(Ss)),A(this,"crossOriginIframeStyleMirror"),A(this,"crossOriginIframeRootIdMap",new WeakMap),A(this,"mirror"),A(this,"mutationCb"),A(this,"wrappedEmit"),A(this,"loadListener"),A(this,"stylesheetManager"),A(this,"recordCrossOriginIframes"),this.mutationCb=t.mutationCb,this.wrappedEmit=t.wrappedEmit,this.stylesheetManager=t.stylesheetManager,this.recordCrossOriginIframes=t.recordCrossOriginIframes,this.crossOriginIframeStyleMirror=new mn(this.stylesheetManager.styleMirror.generateId.bind(this.stylesheetManager.styleMirror)),this.mirror=t.mirror,this.recordCrossOriginIframes&&window.addEventListener("message",this.handleMessage.bind(this))}addIframe(t){this.iframes.set(t,!0),t.contentWindow&&this.crossOriginIframeMap.set(t.contentWindow,t)}addLoadListener(t){this.loadListener=t}attachIframe(t,r){var o,s;this.mutationCb({adds:[{parentId:this.mirror.getId(t),nextId:null,node:r}],removes:[],texts:[],attributes:[],isAttachIframe:!0}),this.recordCrossOriginIframes&&((o=t.contentWindow)==null||o.addEventListener("message",this.handleMessage.bind(this))),(s=this.loadListener)==null||s.call(this,t),t.contentDocument&&t.contentDocument.adoptedStyleSheets&&t.contentDocument.adoptedStyleSheets.length>0&&this.stylesheetManager.adoptStyleSheets(t.contentDocument.adoptedStyleSheets,this.mirror.getId(t.contentDocument))}handleMessage(t){const r=t;if(r.data.type!=="rrweb"||r.origin!==r.data.origin||!t.source)return;const s=this.crossOriginIframeMap.get(t.source);if(!s)return;const u=this.transformCrossOriginEvent(s,r.data.event);u&&this.wrappedEmit(u,r.data.isCheckout)}transformCrossOriginEvent(t,r){var o;switch(r.type){case F.FullSnapshot:{this.crossOriginIframeMirror.reset(t),this.crossOriginIframeStyleMirror.reset(t),this.replaceIdOnNode(r.data.node,t);const s=r.data.node.id;return this.crossOriginIframeRootIdMap.set(t,s),this.patchRootIdOnNode(r.data.node,s),{timestamp:r.timestamp,type:F.IncrementalSnapshot,data:{source:L.Mutation,adds:[{parentId:this.mirror.getId(t),nextId:null,node:r.data.node}],removes:[],texts:[],attributes:[],isAttachIframe:!0}}}case F.Meta:case F.Load:case F.DomContentLoaded:return!1;case F.Plugin:return r;case F.Custom:return this.replaceIds(r.data.payload,t,["id","parentId","previousId","nextId"]),r;case F.IncrementalSnapshot:switch(r.data.source){case L.Mutation:return r.data.adds.forEach(s=>{this.replaceIds(s,t,["parentId","nextId","previousId"]),this.replaceIdOnNode(s.node,t);const u=this.crossOriginIframeRootIdMap.get(t);u&&this.patchRootIdOnNode(s.node,u)}),r.data.removes.forEach(s=>{this.replaceIds(s,t,["parentId","id"])}),r.data.attributes.forEach(s=>{this.replaceIds(s,t,["id"])}),r.data.texts.forEach(s=>{this.replaceIds(s,t,["id"])}),r;case L.Drag:case L.TouchMove:case L.MouseMove:return r.data.positions.forEach(s=>{this.replaceIds(s,t,["id"])}),r;case L.ViewportResize:return!1;case L.MediaInteraction:case L.MouseInteraction:case L.Scroll:case L.CanvasMutation:case L.Input:return this.replaceIds(r.data,t,["id"]),r;case L.StyleSheetRule:case L.StyleDeclaration:return this.replaceIds(r.data,t,["id"]),this.replaceStyleIds(r.data,t,["styleId"]),r;case L.Font:return r;case L.Selection:return r.data.ranges.forEach(s=>{this.replaceIds(s,t,["start","end"])}),r;case L.AdoptedStyleSheet:return this.replaceIds(r.data,t,["id"]),this.replaceStyleIds(r.data,t,["styleIds"]),(o=r.data.styles)==null||o.forEach(s=>{this.replaceStyleIds(s,t,["styleId"])}),r}}return!1}replace(t,r,o,s){for(const u of s)!Array.isArray(r[u])&&typeof r[u]!="number"||(Array.isArray(r[u])?r[u]=t.getIds(o,r[u]):r[u]=t.getId(o,r[u]));return r}replaceIds(t,r,o){return this.replace(this.crossOriginIframeMirror,t,r,o)}replaceStyleIds(t,r,o){return this.replace(this.crossOriginIframeStyleMirror,t,r,o)}replaceIdOnNode(t,r){this.replaceIds(t,r,["id","rootId"]),"childNodes"in t&&t.childNodes.forEach(o=>{this.replaceIdOnNode(o,r)})}patchRootIdOnNode(t,r){t.type!==an.Document&&!t.rootId&&(t.rootId=r),"childNodes"in t&&t.childNodes.forEach(o=>{this.patchRootIdOnNode(o,r)})}}class Ua{constructor(t){A(this,"shadowDoms",new WeakSet),A(this,"mutationCb"),A(this,"scrollCb"),A(this,"bypassOptions"),A(this,"mirror"),A(this,"restoreHandlers",[]),this.mutationCb=t.mutationCb,this.scrollCb=t.scrollCb,this.bypassOptions=t.bypassOptions,this.mirror=t.mirror,this.init()}init(){this.reset(),this.patchAttachShadow(Element,document)}addShadowRoot(t,r){if(!qe(t)||this.shadowDoms.has(t))return;this.shadowDoms.add(t);const[o]=hn({...this.bypassOptions,doc:r,mutationCb:this.mutationCb,mirror:this.mirror,shadowDomManager:this},t);this.restoreHandlers.push(()=>o.disconnect()),this.restoreHandlers.push(fn({...this.bypassOptions,scrollCb:this.scrollCb,doc:t,mirror:this.mirror})),setTimeout(()=>{t.adoptedStyleSheets&&t.adoptedStyleSheets.length>0&&this.bypassOptions.stylesheetManager.adoptStyleSheets(t.adoptedStyleSheets,this.mirror.getId(D.host(t))),this.restoreHandlers.push(pn({mirror:this.mirror,stylesheetManager:this.bypassOptions.stylesheetManager},t))},0)}observeAttachShadow(t){!t.contentWindow||!t.contentDocument||this.patchAttachShadow(t.contentWindow.Element,t.contentDocument)}patchAttachShadow(t,r){const o=this;this.restoreHandlers.push(Ne(t.prototype,"attachShadow",function(s){return function(u){const c=s.call(this,u),p=D.shadowRoot(this);return p&&on(this)&&o.addShadowRoot(p,r),c}}))}reset(){this.restoreHandlers.forEach(t=>{try{t()}catch{}}),this.restoreHandlers=[],this.shadowDoms=new WeakSet}}for(var Be="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/",Ba=typeof Uint8Array>"u"?[]:new Uint8Array(256),xt=0;xt<Be.length;xt++)Ba[Be.charCodeAt(xt)]=xt;var za=function(e){var t=new Uint8Array(e),r,o=t.length,s="";for(r=0;r<o;r+=3)s+=Be[t[r]>>2],s+=Be[(t[r]&3)<<4|t[r+1]>>4],s+=Be[(t[r+1]&15)<<2|t[r+2]>>6],s+=Be[t[r+2]&63];return o%3===2?s=s.substring(0,s.length-1)+"=":o%3===1&&(s=s.substring(0,s.length-2)+"=="),s};const gn=new Map;function Wa(e,t){let r=gn.get(e);return r||(r=new Map,gn.set(e,r)),r.has(t)||r.set(t,[]),r.get(t)}const yn=(e,t,r)=>{if(!e||!(bn(e,t)||typeof e=="object"))return;const o=e.constructor.name,s=Wa(r,o);let u=s.indexOf(e);return u===-1&&(u=s.length,s.push(e)),u};function Rt(e,t,r){if(e instanceof Array)return e.map(o=>Rt(o,t,r));if(e===null)return e;if(e instanceof Float32Array||e instanceof Float64Array||e instanceof Int32Array||e instanceof Uint32Array||e instanceof Uint8Array||e instanceof Uint16Array||e instanceof Int16Array||e instanceof Int8Array||e instanceof Uint8ClampedArray)return{rr_type:e.constructor.name,args:[Object.values(e)]};if(e instanceof ArrayBuffer){const o=e.constructor.name,s=za(e);return{rr_type:o,base64:s}}else{if(e instanceof DataView)return{rr_type:e.constructor.name,args:[Rt(e.buffer,t,r),e.byteOffset,e.byteLength]};if(e instanceof HTMLImageElement){const o=e.constructor.name,{src:s}=e;return{rr_type:o,src:s}}else if(e instanceof HTMLCanvasElement){const o="HTMLImageElement",s=e.toDataURL();return{rr_type:o,src:s}}else{if(e instanceof ImageData)return{rr_type:e.constructor.name,args:[Rt(e.data,t,r),e.width,e.height]};if(bn(e,t)||typeof e=="object"){const o=e.constructor.name,s=yn(e,t,r);return{rr_type:o,index:s}}}}return e}const wn=(e,t,r)=>e.map(o=>Rt(o,t,r)),bn=(e,t)=>!!["WebGLActiveInfo","WebGLBuffer","WebGLFramebuffer","WebGLProgram","WebGLRenderbuffer","WebGLShader","WebGLShaderPrecisionFormat","WebGLTexture","WebGLUniformLocation","WebGLVertexArrayObject","WebGLVertexArrayObjectOES"].filter(s=>typeof t[s]=="function").find(s=>e instanceof t[s]);function qa(e,t,r,o){const s=[],u=Object.getOwnPropertyNames(t.CanvasRenderingContext2D.prototype);for(const c of u)try{if(typeof t.CanvasRenderingContext2D.prototype[c]!="function")continue;const p=Ne(t.CanvasRenderingContext2D.prototype,c,function(l){return function(...d){return se(this.canvas,r,o,!0)||setTimeout(()=>{const i=wn(d,t,this);e(this.canvas,{type:Fe["2D"],property:c,args:i})},0),l.apply(this,d)}});s.push(p)}catch{const p=bt(t.CanvasRenderingContext2D.prototype,c,{set(l){e(this.canvas,{type:Fe["2D"],property:c,args:[l],setter:!0})}});s.push(p)}return()=>{s.forEach(c=>c())}}function ja(e){return e==="experimental-webgl"?"webgl":e}function Sn(e,t,r,o){const s=[];try{const u=Ne(e.HTMLCanvasElement.prototype,"getContext",function(c){return function(p,...l){if(!se(this,t,r,!0)){const d=ja(p);if("__context"in this||(this.__context=d),o&&["webgl","webgl2"].includes(d))if(l[0]&&typeof l[0]=="object"){const i=l[0];i.preserveDrawingBuffer||(i.preserveDrawingBuffer=!0)}else l.splice(0,1,{preserveDrawingBuffer:!0})}return c.apply(this,[p,...l])}});s.push(u)}catch{console.error("failed to patch HTMLCanvasElement.prototype.getContext")}return()=>{s.forEach(u=>u())}}function vn(e,t,r,o,s,u){const c=[],p=Object.getOwnPropertyNames(e);for(const l of p)if(!["isContextLost","canvas","drawingBufferWidth","drawingBufferHeight"].includes(l))try{if(typeof e[l]!="function")continue;const d=Ne(e,l,function(i){return function(...f){const a=i.apply(this,f);if(yn(a,u,this),"tagName"in this.canvas&&!se(this.canvas,o,s,!0)){const n=wn(f,u,this),m={type:t,property:l,args:n};r(this.canvas,m)}return a}});c.push(d)}catch{const d=bt(e,l,{set(i){r(this.canvas,{type:t,property:l,args:[i],setter:!0})}});c.push(d)}return c}function Ga(e,t,r,o){const s=[];return typeof t.WebGLRenderingContext<"u"&&s.push(...vn(t.WebGLRenderingContext.prototype,Fe.WebGL,e,r,o,t)),typeof t.WebGL2RenderingContext<"u"&&s.push(...vn(t.WebGL2RenderingContext.prototype,Fe.WebGL2,e,r,o,t)),()=>{s.forEach(u=>u())}}const Cn=`(function() {
+ "use strict";
+ var chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+ var lookup = typeof Uint8Array === "undefined" ? [] : new Uint8Array(256);
+ for (var i = 0; i < chars.length; i++) {
+ lookup[chars.charCodeAt(i)] = i;
+ }
+ var encode = function(arraybuffer) {
+ var bytes = new Uint8Array(arraybuffer), i2, len = bytes.length, base64 = "";
+ for (i2 = 0; i2 < len; i2 += 3) {
+ base64 += chars[bytes[i2] >> 2];
+ base64 += chars[(bytes[i2] & 3) << 4 | bytes[i2 + 1] >> 4];
+ base64 += chars[(bytes[i2 + 1] & 15) << 2 | bytes[i2 + 2] >> 6];
+ base64 += chars[bytes[i2 + 2] & 63];
+ }
+ if (len % 3 === 2) {
+ base64 = base64.substring(0, base64.length - 1) + "=";
+ } else if (len % 3 === 1) {
+ base64 = base64.substring(0, base64.length - 2) + "==";
+ }
+ return base64;
+ };
+ const lastBlobMap = /* @__PURE__ */ new Map();
+ const transparentBlobMap = /* @__PURE__ */ new Map();
+ async function getTransparentBlobFor(width, height, dataURLOptions) {
+ const id = \`\${width}-\${height}\`;
+ if ("OffscreenCanvas" in globalThis) {
+ if (transparentBlobMap.has(id)) return transparentBlobMap.get(id);
+ const offscreen = new OffscreenCanvas(width, height);
+ offscreen.getContext("2d");
+ const blob = await offscreen.convertToBlob(dataURLOptions);
+ const arrayBuffer = await blob.arrayBuffer();
+ const base64 = encode(arrayBuffer);
+ transparentBlobMap.set(id, base64);
+ return base64;
+ } else {
+ return "";
+ }
+ }
+ const worker = self;
+ worker.onmessage = async function(e) {
+ if ("OffscreenCanvas" in globalThis) {
+ const { id, bitmap, width, height, dataURLOptions } = e.data;
+ const transparentBase64 = getTransparentBlobFor(
+ width,
+ height,
+ dataURLOptions
+ );
+ const offscreen = new OffscreenCanvas(width, height);
+ const ctx = offscreen.getContext("2d");
+ ctx.drawImage(bitmap, 0, 0);
+ bitmap.close();
+ const blob = await offscreen.convertToBlob(dataURLOptions);
+ const type = blob.type;
+ const arrayBuffer = await blob.arrayBuffer();
+ const base64 = encode(arrayBuffer);
+ if (!lastBlobMap.has(id) && await transparentBase64 === base64) {
+ lastBlobMap.set(id, base64);
+ return worker.postMessage({ id });
+ }
+ if (lastBlobMap.get(id) === base64) return worker.postMessage({ id });
+ worker.postMessage({
+ id,
+ type,
+ base64,
+ width,
+ height
+ });
+ lastBlobMap.set(id, base64);
+ } else {
+ return worker.postMessage({ id: e.data.id });
+ }
+ };
+})();
+//# sourceMappingURL=image-bitmap-data-url-worker-IJpC7g_b.js.map
+`,xn=typeof self<"u"&&self.Blob&&new Blob([Cn],{type:"text/javascript;charset=utf-8"});function Va(e){let t;try{if(t=xn&&(self.URL||self.webkitURL).createObjectURL(xn),!t)throw"";const r=new Worker(t,{name:e==null?void 0:e.name});return r.addEventListener("error",()=>{(self.URL||self.webkitURL).revokeObjectURL(t)}),r}catch{return new Worker("data:text/javascript;charset=utf-8,"+encodeURIComponent(Cn),{name:e==null?void 0:e.name})}finally{t&&(self.URL||self.webkitURL).revokeObjectURL(t)}}class Ha{constructor(t){A(this,"pendingCanvasMutations",new Map),A(this,"rafStamps",{latestId:0,invokeId:null}),A(this,"mirror"),A(this,"mutationCb"),A(this,"resetObservers"),A(this,"frozen",!1),A(this,"locked",!1),A(this,"processMutation",(l,d)=>{(this.rafStamps.invokeId&&this.rafStamps.latestId!==this.rafStamps.invokeId||!this.rafStamps.invokeId)&&(this.rafStamps.invokeId=this.rafStamps.latestId),this.pendingCanvasMutations.has(l)||this.pendingCanvasMutations.set(l,[]),this.pendingCanvasMutations.get(l).push(d)});const{sampling:r="all",win:o,blockClass:s,blockSelector:u,recordCanvas:c,dataURLOptions:p}=t;this.mutationCb=t.mutationCb,this.mirror=t.mirror,c&&r==="all"&&this.initCanvasMutationObserver(o,s,u),c&&typeof r=="number"&&this.initCanvasFPSObserver(r,o,s,u,{dataURLOptions:p})}reset(){this.pendingCanvasMutations.clear(),this.resetObservers&&this.resetObservers()}freeze(){this.frozen=!0}unfreeze(){this.frozen=!1}lock(){this.locked=!0}unlock(){this.locked=!1}initCanvasFPSObserver(t,r,o,s,u){const c=Sn(r,o,s,!0),p=new Map,l=new Va;l.onmessage=m=>{const{id:h}=m.data;if(p.set(h,!1),!("base64"in m.data))return;const{base64:g,type:v,width:b,height:S}=m.data;this.mutationCb({id:h,type:Fe["2D"],commands:[{property:"clearRect",args:[0,0,b,S]},{property:"drawImage",args:[{rr_type:"ImageBitmap",args:[{rr_type:"Blob",data:[{rr_type:"ArrayBuffer",base64:g}],type:v}]},0,0]}]})};const d=1e3/t;let i=0,f;const a=()=>{const m=[];return r.document.querySelectorAll("canvas").forEach(h=>{se(h,o,s,!0)||m.push(h)}),m},n=m=>{if(i&&m-i<d){f=requestAnimationFrame(n);return}i=m,a().forEach(async h=>{var g;const v=this.mirror.getId(h);if(p.get(v)||h.width===0||h.height===0)return;if(p.set(v,!0),["webgl","webgl2"].includes(h.__context)){const S=h.getContext(h.__context);((g=S==null?void 0:S.getContextAttributes())==null?void 0:g.preserveDrawingBuffer)===!1&&S.clear(S.COLOR_BUFFER_BIT)}const b=await createImageBitmap(h);l.postMessage({id:v,bitmap:b,width:h.width,height:h.height,dataURLOptions:u.dataURLOptions},[b])}),f=requestAnimationFrame(n)};f=requestAnimationFrame(n),this.resetObservers=()=>{c(),cancelAnimationFrame(f)}}initCanvasMutationObserver(t,r,o){this.startRAFTimestamping(),this.startPendingCanvasMutationFlusher();const s=Sn(t,r,o,!1),u=qa(this.processMutation.bind(this),t,r,o),c=Ga(this.processMutation.bind(this),t,r,o);this.resetObservers=()=>{s(),u(),c()}}startPendingCanvasMutationFlusher(){requestAnimationFrame(()=>this.flushPendingCanvasMutations())}startRAFTimestamping(){const t=r=>{this.rafStamps.latestId=r,requestAnimationFrame(t)};requestAnimationFrame(t)}flushPendingCanvasMutations(){this.pendingCanvasMutations.forEach((t,r)=>{const o=this.mirror.getId(r);this.flushPendingCanvasMutationFor(r,o)}),requestAnimationFrame(()=>this.flushPendingCanvasMutations())}flushPendingCanvasMutationFor(t,r){if(this.frozen||this.locked)return;const o=this.pendingCanvasMutations.get(t);if(!o||r===-1)return;const s=o.map(c=>{const{type:p,...l}=c;return l}),{type:u}=o[0];this.mutationCb({id:r,type:u,commands:s}),this.pendingCanvasMutations.delete(t)}}class Ja{constructor(t){A(this,"trackedLinkElements",new WeakSet),A(this,"mutationCb"),A(this,"adoptedStyleSheetCb"),A(this,"styleMirror",new wa),this.mutationCb=t.mutationCb,this.adoptedStyleSheetCb=t.adoptedStyleSheetCb}attachLinkElement(t,r){"_cssText"in r.attributes&&this.mutationCb({adds:[],removes:[],texts:[],attributes:[{id:r.id,attributes:r.attributes}]}),this.trackLinkElement(t)}trackLinkElement(t){this.trackedLinkElements.has(t)||(this.trackedLinkElements.add(t),this.trackStylesheetInLinkElement(t))}adoptStyleSheets(t,r){if(t.length===0)return;const o={id:r,styleIds:[]},s=[];for(const u of t){let c;this.styleMirror.has(u)?c=this.styleMirror.getId(u):(c=this.styleMirror.add(u),s.push({styleId:c,rules:Array.from(u.rules||CSSRule,(p,l)=>({rule:gs(p,u.href),index:l}))})),o.styleIds.push(c)}s.length>0&&(o.styles=s),this.adoptedStyleSheetCb(o)}reset(){this.styleMirror.reset(),this.trackedLinkElements=new WeakSet}trackStylesheetInLinkElement(t){}}class Ya{constructor(){A(this,"nodeMap",new WeakMap),A(this,"active",!1)}inOtherBuffer(t,r){const o=this.nodeMap.get(t);return o&&Array.from(o).some(s=>s!==r)}add(t,r){this.active||(this.active=!0,requestAnimationFrame(()=>{this.nodeMap=new WeakMap,this.active=!1})),this.nodeMap.set(t,(this.nodeMap.get(t)||new Set).add(r))}destroy(){}}let H,Ot,rs,Mt=!1;try{if(Array.from([1],e=>e*2)[0]!==2){const e=document.createElement("iframe");document.body.appendChild(e),Array.from=((ls=e.contentWindow)==null?void 0:ls.Array.from)||Array.from,document.body.removeChild(e)}}catch(e){console.debug("Unable to override Array.from",e)}const ge=so();function Se(e={}){const{emit:t,checkoutEveryNms:r,checkoutEveryNth:o,blockClass:s="rr-block",blockSelector:u=null,ignoreClass:c="rr-ignore",ignoreSelector:p=null,maskTextClass:l="rr-mask",maskTextSelector:d=null,inlineStylesheet:i=!0,maskAllInputs:f,maskInputOptions:a,slimDOMOptions:n,maskInputFn:m,maskTextFn:h,hooks:g,packFn:v,sampling:b={},dataURLOptions:S={},mousemoveWait:x,recordDOM:w=!0,recordCanvas:y=!1,recordCrossOriginIframes:C=!1,recordAfter:O=e.recordAfter==="DOMContentLoaded"?e.recordAfter:"load",userTriggeredOnInput:I=!1,collectFonts:M=!1,inlineImages:P=!1,plugins:N,keepIframeSrcFn:R=()=>!1,ignoreCSSAttributes:ye=new Set([]),errorHandler:ue}=e;xa(ue);const B=C?window.parent===window:!0;let U=!1;if(!B)try{window.parent.document&&(U=!1)}catch{U=!0}if(B&&!t)throw new Error("emit function is required");if(!B&&!U)return()=>{};x!==void 0&&b.mousemove===void 0&&(b.mousemove=x),ge.reset();const Y=f===!0?{color:!0,date:!0,"datetime-local":!0,email:!0,month:!0,number:!0,range:!0,search:!0,tel:!0,text:!0,time:!0,url:!0,week:!0,textarea:!0,select:!0,password:!0}:a!==void 0?a:{password:!0},te=Ms(n);ya();let Q,z=0;const Me=T=>{for(const K of N||[])K.eventProcessor&&(T=K.eventProcessor(T));return v&&!U&&(T=v(T)),T};H=(T,K)=>{var W;const J=T;if(J.timestamp=Ve(),(W=ke[0])!=null&&W.isFrozen()&&J.type!==F.FullSnapshot&&!(J.type===F.IncrementalSnapshot&&J.data.source===L.Mutation)&&ke.forEach(ce=>ce.unfreeze()),B)t==null||t(Me(J),K);else if(U){const ce={type:"rrweb",event:Me(J),origin:window.location.origin,isCheckout:K};window.parent.postMessage(ce,"*")}if(J.type===F.FullSnapshot)Q=J,z=0;else if(J.type===F.IncrementalSnapshot){if(J.data.source===L.Mutation&&J.data.isAttachIframe)return;z++;const ce=o&&z>=o,q=r&&J.timestamp-Q.timestamp>r;(ce||q)&&Ot(!0)}};const E=T=>{H({type:F.IncrementalSnapshot,data:{source:L.Mutation,...T}})},ve=T=>H({type:F.IncrementalSnapshot,data:{source:L.Scroll,...T}}),de=T=>H({type:F.IncrementalSnapshot,data:{source:L.CanvasMutation,...T}}),Qe=T=>H({type:F.IncrementalSnapshot,data:{source:L.AdoptedStyleSheet,...T}}),ie=new Ja({mutationCb:E,adoptedStyleSheetCb:Qe}),we=new Fa({mirror:ge,mutationCb:E,stylesheetManager:ie,recordCrossOriginIframes:C,wrappedEmit:H});for(const T of N||[])T.getMirror&&T.getMirror({nodeMirror:ge,crossOriginIframeMirror:we.crossOriginIframeMirror,crossOriginIframeStyleMirror:we.crossOriginIframeStyleMirror});const Te=new Ya;rs=new Ha({recordCanvas:y,mutationCb:de,win:window,blockClass:s,blockSelector:u,mirror:ge,sampling:b.canvas,dataURLOptions:S});const Ee=new Ua({mutationCb:E,scrollCb:ve,bypassOptions:{blockClass:s,blockSelector:u,maskTextClass:l,maskTextSelector:d,inlineStylesheet:i,maskInputOptions:Y,dataURLOptions:S,maskTextFn:h,maskInputFn:m,recordCanvas:y,inlineImages:P,sampling:b,slimDOMOptions:te,iframeManager:we,stylesheetManager:ie,canvasManager:rs,keepIframeSrcFn:R,processedNodeManager:Te},mirror:ge});Ot=(T=!1)=>{if(!w)return;H({type:F.Meta,data:{href:window.location.href,width:Xi(),height:Ki()}},T),ie.reset(),Ee.init(),ke.forEach(W=>W.lock());const K=Io(document,{mirror:ge,blockClass:s,blockSelector:u,maskTextClass:l,maskTextSelector:d,inlineStylesheet:i,maskAllInputs:Y,maskTextFn:h,maskInputFn:m,slimDOM:te,dataURLOptions:S,recordCanvas:y,inlineImages:P,onSerialize:W=>{rn(W,ge)&&we.addIframe(W),sn(W,ge)&&ie.trackLinkElement(W),es(W)&&Ee.addShadowRoot(D.shadowRoot(W),document)},onIframeLoad:(W,J)=>{we.attachIframe(W,J),Ee.observeAttachShadow(W)},onStylesheetLoad:(W,J)=>{ie.attachLinkElement(W,J)},keepIframeSrcFn:R});if(!K)return console.warn("Failed to snapshot the document");H({type:F.FullSnapshot,data:{node:K,initialOffset:Zi(window)}},T),ke.forEach(W=>W.unlock()),document.adoptedStyleSheets&&document.adoptedStyleSheets.length>0&&ie.adoptStyleSheets(document.adoptedStyleSheets,ge.getId(document))};try{const T=[],K=J=>{var ce;return _($a)({mutationCb:E,mousemoveCb:(q,os)=>H({type:F.IncrementalSnapshot,data:{source:os,positions:q}}),mouseInteractionCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.MouseInteraction,...q}}),scrollCb:ve,viewportResizeCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.ViewportResize,...q}}),inputCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.Input,...q}}),mediaInteractionCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.MediaInteraction,...q}}),styleSheetRuleCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.StyleSheetRule,...q}}),styleDeclarationCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.StyleDeclaration,...q}}),canvasMutationCb:de,fontCb:q=>H({type:F.IncrementalSnapshot,data:{source:L.Font,...q}}),selectionCb:q=>{H({type:F.IncrementalSnapshot,data:{source:L.Selection,...q}})},customElementCb:q=>{H({type:F.IncrementalSnapshot,data:{source:L.CustomElement,...q}})},blockClass:s,ignoreClass:c,ignoreSelector:p,maskTextClass:l,maskTextSelector:d,maskInputOptions:Y,inlineStylesheet:i,sampling:b,recordDOM:w,recordCanvas:y,inlineImages:P,userTriggeredOnInput:I,collectFonts:M,doc:J,maskInputFn:m,maskTextFn:h,keepIframeSrcFn:R,blockSelector:u,slimDOMOptions:te,dataURLOptions:S,mirror:ge,iframeManager:we,stylesheetManager:ie,shadowDomManager:Ee,processedNodeManager:Te,canvasManager:rs,ignoreCSSAttributes:ye,plugins:((ce=N==null?void 0:N.filter(q=>q.observer))==null?void 0:ce.map(q=>({observer:q.observer,options:q.options,callback:os=>H({type:F.Plugin,data:{plugin:q.name,payload:os}})})))||[]},g)};we.addLoadListener(J=>{try{T.push(K(J.contentDocument))}catch(ce){console.warn(ce)}});const W=()=>{Ot(),T.push(K(document)),Mt=!0};return["interactive","complete"].includes(document.readyState)?W():(T.push(re("DOMContentLoaded",()=>{H({type:F.DomContentLoaded,data:{}}),O==="DOMContentLoaded"&&W()})),T.push(re("load",()=>{H({type:F.Load,data:{}}),O==="load"&&W()},window))),()=>{T.forEach(J=>{try{J()}catch(ce){String(ce).toLowerCase().includes("cross-origin")||console.warn(ce)}}),Te.destroy(),Mt=!1,Ra()}}catch(T){console.warn(T)}}Se.addCustomEvent=(e,t)=>{if(!Mt)throw new Error("please add custom event after start recording");H({type:F.Custom,data:{tag:e,payload:t}})},Se.freezePage=()=>{ke.forEach(e=>e.freeze())},Se.takeFullSnapshot=e=>{if(!Mt)throw new Error("please take full snapshot after start recording");Ot(e)},Se.mirror=ge;var Rn;(function(e){e[e.NotStarted=0]="NotStarted",e[e.Running=1]="Running",e[e.Stopped=2]="Stopped"})(Rn||(Rn={}));const{addCustomEvent:_l}=Se,{freezePage:$l}=Se,{takeFullSnapshot:Fl}=Se;async function Qa(e){const r=new TextEncoder().encode(e),o=new CompressionStream("gzip"),s=o.writable.getWriter();s.write(r),s.close();const u=o.readable.getReader(),c=[];let p=0;for(;;){const{done:i,value:f}=await u.read();if(i)break;c.push(f),p+=f.length}const l=new Uint8Array(p);let d=0;for(const i of c)l.set(i,d),d+=i.length;return l}async function Za(e,t,r,o={}){if(o.keepalive)try{return fetch(e,{method:"POST",headers:{"Content-Type":"application/json",Authorization:`Bearer ${t}`},body:r,keepalive:!0}),!0}catch{return!1}const u=await Qa(r);return(await fetch(e,{method:"POST",headers:{"Content-Type":"application/json","Content-Encoding":"gzip",Authorization:`Bearer ${t}`},body:u})).status===200}var Ka=class{constructor(e={}){k(this,"segmentDuration");k(this,"current");k(this,"previous",null);k(this,"stopFn",null);k(this,"onSegmentReady");k(this,"onActivity");this.segmentDuration=e.segmentDuration??3e4,this.current=this.newSegment(),this.onSegmentReady=e.onSegmentReady,this.onActivity=e.onActivity}start(){this.stopFn=Se({emit:e=>{this.onEvent(e)}})}stop(){this.stopFn&&(this.stopFn(),this.stopFn=null)}onEvent(e){if(this.onActivity)try{this.onActivity()}catch{}if(this.current.events.length>0&&e.timestamp-this.current.startedAtMs>=this.segmentDuration){const t=this.current,r=e.timestamp-this.current.startedAtMs>=2*this.segmentDuration;if(this.previous=r?null:this.current,this.current=this.newSegment(),Se.takeFullSnapshot(),this.onSegmentReady&&t.events.length>0){const o=t.events[t.events.length-1].timestamp;try{this.onSegmentReady({events:t.events,startedAt:t.startedAt,endedAt:new Date(o).toISOString()})}catch{}}}this.current.events.length===0&&(this.current.startedAtMs=e.timestamp,this.current.startedAt=new Date(e.timestamp).toISOString()),this.current.events.push(e)}drainCurrent(){if(this.current.events.length===0)return null;const e=this.current;this.current=this.newSegment();const t=e.events[e.events.length-1].timestamp;return{events:e.events,startedAt:e.startedAt,endedAt:new Date(t).toISOString()}}newSegment(){const e=Date.now();return{events:[],startedAt:new Date(e).toISOString(),startedAtMs:e}}getSegments(){const e=[];return this.previous&&this.previous.events.length>0&&e.push({events:this.previous.events,timestamp:this.previous.startedAt}),this.current.events.length>0&&e.push({events:this.current.events,timestamp:this.current.startedAt}),e}getClipEvents(){return this.current.events.length>0?this.current.events:this.previous&&this.previous.events.length>0?this.previous.events:[]}hasSegments(){return this.current.events.length>0||this.previous!==null&&this.previous.events.length>0}flush(){this.previous=null,this.current=this.newSegment(),this.stopFn&&Se.takeFullSnapshot()}},Xa=class{constructor(e){k(this,"inactivityMs");k(this,"maxDurationMs");k(this,"checkIntervalMs");k(this,"onSessionEnd");k(this,"onSoftFlush");k(this,"onUnloading");k(this,"onSessionRestart");k(this,"startedAtMs");k(this,"lastActivityMs");k(this,"timer",null);k(this,"installed",!1);k(this,"ended",!1);k(this,"boundPagehide",()=>this.handlePagehide());k(this,"boundVisibility",()=>this.handleVisibility());k(this,"boundPageshow",e=>this.handlePageshow(e));this.inactivityMs=e.inactivityMs??15*6e4,this.maxDurationMs=e.maxDurationMs??60*6e4,this.checkIntervalMs=e.checkIntervalMs??3e4,this.onSessionEnd=e.onSessionEnd,this.onSoftFlush=e.onSoftFlush??(()=>{}),this.onUnloading=e.onUnloading??(()=>{}),this.onSessionRestart=e.onSessionRestart??(()=>{});const t=Date.now();this.startedAtMs=t,this.lastActivityMs=t}install(){this.installed||typeof window>"u"||(window.addEventListener("pagehide",this.boundPagehide),window.addEventListener("pageshow",this.boundPageshow),document.addEventListener("visibilitychange",this.boundVisibility),this.timer=setInterval(()=>this.tick(),this.checkIntervalMs),this.installed=!0)}uninstall(){this.installed&&(window.removeEventListener("pagehide",this.boundPagehide),window.removeEventListener("pageshow",this.boundPageshow),document.removeEventListener("visibilitychange",this.boundVisibility),this.timer!==null&&(clearInterval(this.timer),this.timer=null),this.installed=!1)}markActivity(){this.lastActivityMs=Date.now()}reset(){const e=Date.now();this.startedAtMs=e,this.lastActivityMs=e,this.ended=!1}startedAt(){return new Date(this.startedAtMs)}tick(){if(this.ended)return;const e=Date.now();(e-this.lastActivityMs>=this.inactivityMs||e-this.startedAtMs>=this.maxDurationMs)&&this.endSession()}handlePagehide(){try{this.onUnloading()}catch{}this.endSession()}handlePageshow(e){if(e.persisted){this.reset();try{this.onSessionRestart()}catch{}}}handleVisibility(){typeof document>"u"||document.visibilityState==="hidden"&&this.onSoftFlush()}endSession(){if(!this.ended){this.ended=!0;try{this.onSessionEnd()}catch{}}}};function On(){if(typeof window>"u")return{};const e={};try{e.url=window.location.href,e.path=window.location.pathname}catch{}try{document.referrer&&(e.referrer=document.referrer)}catch{}try{navigator.userAgent&&(e.userAgent=navigator.userAgent),navigator.language&&(e.language=navigator.language),navigator.platform&&(e.platform=navigator.platform)}catch{}try{const t=window.innerWidth,r=window.innerHeight;Number.isFinite(t)&&Number.isFinite(r)&&(e.viewport=`${t}x${r}`)}catch{}try{if(window.screen){const t=window.screen.width,r=window.screen.height;Number.isFinite(t)&&Number.isFinite(r)&&(e.screen=`${t}x${r}`)}}catch{}try{const t=Intl.DateTimeFormat().resolvedOptions().timeZone;t&&(e.timezone=t)}catch{}return e}var el=[/^\s+at\s+.+?\s+\((.+):\d+:\d+\)$/,/^\s+at\s+(.+):\d+:\d+$/,/^(?:.*@)?(.+):\d+:\d+$/];function tl(e){for(const t of e.split(`
+`))for(const r of el){const o=t.match(r);if(o){const s=o[1].split("/");return s[s.length-1]}}}function rl(e){return typeof e=="object"&&e!==null}var Mn=-1,ss={};function sl(){const e=globalThis,t=[e._tracewayDebugIds,e._sentryDebugIds].filter(rl);if(t.length===0)return;let r=0;for(const o of t)r+=Object.keys(o).length;if(r!==Mn){const o={};for(const s of t)for(const u of Object.keys(s)){const c=tl(u);c&&!(c in o)&&(o[c]=String(s[u]))}Mn=r,ss=o}return Object.keys(ss).length>0?ss:void 0}function il(e){const t=sl();if(!t)return;let r;for(const[o,s]of Object.entries(t))e.includes(o)&&((r??(r={}))[o]=s);return r}function Et(e){return new Date(e).toISOString()}function nl(e,t,r){if(e.length>0){const s=[];for(const u of e){const c=u==null?void 0:u.timestamp;typeof c=="number"&&Number.isFinite(c)&&s.push(c)}if(s.length>0)return{startedAt:Et(Math.min(...s)),endedAt:Et(Math.max(...s))}}const o=[...t.map(s=>Date.parse(s.timestamp)),...r.map(s=>Date.parse(s.timestamp))].filter(s=>Number.isFinite(s));return o.length===0?{}:{startedAt:Et(Math.min(...o)),endedAt:Et(Math.max(...o))}}var ol=["Failed to fetch","NetworkError when attempting to fetch resource","Load failed","Network Error","Network request failed","The operation was aborted",/timeout/i,/status code 4\d{2}/,/failed: 4\d{2}/],al=class{constructor(e,t={}){k(this,"apiUrl");k(this,"token");k(this,"debug");k(this,"debounceMs");k(this,"retryDelayMs");k(this,"version");k(this,"pendingExceptions",[]);k(this,"pendingRecordings",[]);k(this,"pendingSessions",[]);k(this,"isSyncing",!1);k(this,"debounceTimer",null);k(this,"retryTimer",null);k(this,"recorder",null);k(this,"lifecycle",null);k(this,"recordAllSessions");k(this,"sessionId",null);k(this,"sessionStartedAt",null);k(this,"segmentIndex",0);k(this,"globalAttributes",{});k(this,"unloading",!1);k(this,"ignoreErrors");k(this,"beforeCapture");k(this,"captureLogs");k(this,"captureNetwork");k(this,"captureNavigation");k(this,"captureHttpServerErrors");k(this,"logs");k(this,"actions");const{token:r,apiUrl:o}=Ce(e);this.apiUrl=o,this.token=r,this.debug=t.debug??!1,this.debounceMs=t.debounceMs??1500,this.retryDelayMs=t.retryDelayMs??1e4,this.version=t.version??"",this.globalAttributes={...t.attributes},this.ignoreErrors=t.ignoreErrors??ol,this.beforeCapture=t.beforeCapture??null,this.captureLogs=t.captureLogs??!0,this.captureNetwork=t.captureNetwork??!0,this.captureNavigation=t.captureNavigation??!0,this.captureHttpServerErrors=t.captureHttpServerErrors??!1;const s=t.recordAllSessions===!0,u={windowMs:t.eventsWindowMs??(s?3e4:1e4),maxSize:t.eventsMaxCount??(s?600:200)};this.logs=new as(u),this.actions=new as(u),this.recordAllSessions=t.recordAllSessions??!1;const c=typeof window<"u";if(this.recordAllSessions&&c&&(this.beginSession(),this.lifecycle=new Xa({onUnloading:()=>{this.unloading=!0},onSessionEnd:()=>this.endSession(),onSessionRestart:()=>this.restartSession(),onSoftFlush:()=>{(this.pendingSessions.length>0||this.pendingRecordings.length>0)&&this.scheduleSync()}}),this.lifecycle.install()),t.sessionRecording!==!1&&c){const p={segmentDuration:t.sessionRecordingSegmentDuration};this.recordAllSessions&&(p.onSegmentReady=l=>this.handleSegmentReady(l),p.onActivity=()=>{var l;return(l=this.lifecycle)==null?void 0:l.markActivity()}),this.recorder=new Ka(p),this.recorder.start()}}beginSession(){this.sessionId=be(),this.sessionStartedAt=X(),this.segmentIndex=0,this.pendingSessions.push({id:this.sessionId,startedAt:this.sessionStartedAt,attributes:this.composedSessionAttributes()}),this.scheduleSync()}composedSessionAttributes(){return{...On(),...this.globalAttributes}}setAttribute(e,t){e&&(this.globalAttributes[e]=t,this.refreshOpenSessionAttributes())}setAttributes(e){if(!e)return;let t=!1;for(const r of Object.keys(e))r&&(this.globalAttributes[r]=e[r],t=!0);t&&this.refreshOpenSessionAttributes()}removeAttribute(e){e in this.globalAttributes&&(delete this.globalAttributes[e],this.refreshOpenSessionAttributes())}clearAttributes(){Object.keys(this.globalAttributes).length!==0&&(this.globalAttributes={},this.refreshOpenSessionAttributes())}currentAttributes(){return{...this.globalAttributes}}refreshOpenSessionAttributes(){!this.sessionId||!this.sessionStartedAt||this.recordAllSessions===!0&&(this.pendingSessions.push({id:this.sessionId,startedAt:this.sessionStartedAt,attributes:this.composedSessionAttributes()}),this.scheduleSync())}endSession(){if(!(!this.sessionId||!this.sessionStartedAt)){if(this.recorder){const e=this.recorder.drainCurrent();e&&e.events.length>0&&this.queueSegment(e)}this.pendingSessions.push({id:this.sessionId,startedAt:this.sessionStartedAt,endedAt:X(),attributes:this.composedSessionAttributes()}),this.unloading?(this.debounceTimer!==null&&(clearTimeout(this.debounceTimer),this.debounceTimer=null),this.doSync()):this.scheduleSync()}}restartSession(){this.unloading=!1,this.beginSession()}handleSegmentReady(e){this.sessionId&&(this.queueSegment(e),this.scheduleSync())}queueSegment(e){if(!this.sessionId)return;const t=this.logs.snapshot(),r=this.actions.snapshot();t.length>0&&this.logs.clear(),r.length>0&&this.actions.clear();const o={sessionId:this.sessionId,segmentIndex:this.segmentIndex++,events:e.events,startedAt:e.startedAt,endedAt:e.endedAt};t.length>0&&(o.logs=t),r.length>0&&(o.actions=r),this.pendingRecordings.push(o)}currentSessionId(){return this.sessionId}recordLog(e,t){this.captureLogs&&this.logs.add({type:"log",timestamp:X(),level:e,message:t})}recordNetworkEvent(e){this.captureNetwork&&(this.isOwnReportUrl(e.url)||this.actions.add({type:"network",timestamp:e.timestamp??X(),method:e.method,url:e.url,durationMs:e.durationMs,statusCode:e.statusCode,requestBytes:e.requestBytes,responseBytes:e.responseBytes,error:e.error}))}isOwnReportUrl(e){return e?!!(e===this.apiUrl||e.startsWith(this.apiUrl+"?")||e.startsWith(this.apiUrl+"#")):!1}recordNavigationEvent(e){this.captureNavigation&&this.actions.add({type:"navigation",timestamp:e.timestamp??X(),action:e.action,from:e.from,to:e.to})}recordAction(e,t,r){this.actions.add({type:"custom",timestamp:X(),category:e,name:t,data:r})}bufferedLogs(){return this.logs.snapshot()}bufferedActions(){return this.actions.snapshot()}captureHttpServerError(e,t,r){this.addException({traceId:null,stackTrace:`HTTP ${r} ${e} ${t}`,recordedAt:X(),attributes:{"http.method":e,"http.url":t,"http.status_code":String(r)},isMessage:!0})}addException(e){if(this.shouldIgnore(e)){this.debug&&console.debug("Traceway: exception suppressed by filter",e.stackTrace.slice(0,120));return}if(e.attributes={...On(),...this.globalAttributes,...e.attributes??{}},!e.isMessage&&!e.debugIds){const u=il(e.stackTrace);u&&(e.debugIds=u)}const t=this.recorder&&this.recorder.hasSegments()?this.recorder.getClipEvents():[],r=this.logs.snapshot(),o=this.actions.snapshot(),s=t.length>0||r.length>0||o.length>0;if(this.recordAllSessions&&this.sessionId&&(e.sessionId=this.sessionId),s){const u=be();e.sessionRecordingId=u;const c={exceptionId:u,events:t},p=nl(t,r,o);p.startedAt&&(c.startedAt=p.startedAt),p.endedAt&&(c.endedAt=p.endedAt),r.length>0&&(c.logs=r),o.length>0&&(c.actions=o),this.pendingRecordings.push(c)}this.pendingExceptions.push(e),this.scheduleSync()}shouldIgnore(e){if(this.ignoreErrors.length>0){const t=e.stackTrace;for(const r of this.ignoreErrors)if(typeof r=="string"){if(t.includes(r))return!0}else if(r.lastIndex=0,r.test(t))return!0}if(this.beforeCapture!==null)try{if(this.beforeCapture(e)===!1)return!0}catch(t){this.debug&&console.error("Traceway: beforeCapture callback threw:",t)}return!1}scheduleSync(){this.debounceTimer!==null&&clearTimeout(this.debounceTimer),this.debounceTimer=setTimeout(()=>{this.debounceTimer=null,this.doSync()},this.debounceMs)}async doSync(){if(this.isSyncing||this.pendingExceptions.length===0&&this.pendingRecordings.length===0&&this.pendingSessions.length===0)return;this.isSyncing=!0;const e=this.pendingExceptions.splice(0),t=this.pendingRecordings.splice(0),r=this.pendingSessions.splice(0),s={collectionFrames:[{stackTraces:e,metrics:[],traces:[],sessionRecordings:t.length>0?t:void 0,sessions:r.length>0?r:void 0}],appVersion:this.version,serverName:""};let u=!1;try{await Za(this.apiUrl,this.token,JSON.stringify(s),this.unloading?{keepalive:!0}:void 0)||(u=!0,this.pendingExceptions.unshift(...e),this.pendingRecordings.unshift(...t),this.pendingSessions.unshift(...r),this.debug&&console.error("Traceway: sync failed, re-queued exceptions"))}catch(c){u=!0,this.pendingExceptions.unshift(...e),this.pendingRecordings.unshift(...t),this.pendingSessions.unshift(...r),this.debug&&console.error("Traceway: sync error:",c)}finally{this.isSyncing=!1,(this.pendingExceptions.length>0||this.pendingRecordings.length>0||this.pendingSessions.length>0)&&(u?this.scheduleRetry():this.doSync())}}scheduleRetry(){this.retryTimer===null&&(this.retryTimer=setTimeout(()=>{this.retryTimer=null,this.doSync()},this.retryDelayMs))}async flush(e){this.debounceTimer!==null&&(clearTimeout(this.debounceTimer),this.debounceTimer=null),this.retryTimer!==null&&(clearTimeout(this.retryTimer),this.retryTimer=null),this.lifecycle&&this.lifecycle.uninstall(),this.recordAllSessions&&this.endSession(),this.recorder&&this.recorder.stop();const t=this.doSync();e!==void 0?await Promise.race([t,new Promise(r=>setTimeout(r,e))]):await t}};function is(e){var o;const t=[],r=((o=e.constructor)==null?void 0:o.name)||"Error";if(t.push(`${r}: ${e.message}`),e.stack){const s=e.stack.split(`
+`);for(const u of s){const c=u.match(/^\s+at\s+(.+?)\s+\((.+):(\d+):(\d+)\)$/);if(c){const i=En(c[1]),[f,a,n]=ns(c[2],c[3],c[4]);t.push(`${i}()`),t.push(` ${It(f)}:${a}:${n}`);continue}const p=u.match(/^\s+at\s+(.+):(\d+):(\d+)$/);if(p){const i=It(p[1]);t.push("<anonymous>()"),t.push(` ${i}:${p[2]}:${p[3]}`);continue}const l=u.match(/^(.+)@(.+):(\d+):(\d+)$/);if(l){let i=l[1];const f=i.lastIndexOf("*");if(f!==-1&&(i=i.slice(f+1),!i))continue;const a=En(i)||"<anonymous>",[n,m,h]=ns(l[2],l[3],l[4]);t.push(`${a}()`),t.push(` ${It(n)}:${m}:${h}`);continue}const d=u.match(/^@(.+):(\d+):(\d+)$/);if(d){const[i,f,a]=ns(d[1],d[2],d[3]);t.push("<anonymous>()"),t.push(` ${It(i)}:${f}:${a}`);continue}}}return t.join(`
`)+`
-`}function Uo(s){const e=s.lastIndexOf("/");e>=0&&(s=s.slice(e+1));const t=s.indexOf(".");return t>=0&&(s=s.slice(t+1)),s}function Sr(s){const e=s.split("/");return e[e.length-1]}var Ee=null;function je(){return Ee}function Cf(s){Ee=s}function If(s){Ee===s&&(Ee=null)}function xf(s){try{return new URL(typeof s=="string"?s:s instanceof Request?s.url:s.href,window.location.origin).origin===window.location.origin}catch{return!0}}function Af(s){return typeof s=="string"?s:s instanceof URL?s.href:s instanceof Request?s.url:String(s)}function Rf(s,e){return e!=null&&e.method?e.method:s instanceof Request?s.method:"GET"}function Of(s){if(s!=null&&s.body){if(typeof s.body=="string")return s.body.length;if(s.body instanceof Blob)return s.body.size;if(s.body instanceof ArrayBuffer||ArrayBuffer.isView(s.body))return s.body.byteLength}}function Ef(s){const e=s.headers.get("content-length");if(e===null)return;const t=Number(e);return Number.isFinite(t)?t:void 0}function Mf(s){const e=window.fetch;window.fetch=function(t,r){const i=xf(t),n=Rf(t,r),o=Af(t),l=Of(r),a=performance.now(),u=new Date().toISOString();let c=r,h=null;if(i){h=ce(),Ee=h;const m=new Headers(r==null?void 0:r.headers);m.set("traceway-trace-id",h),c={...r,headers:m}}const p=(m,g)=>{if(s)try{s.recordNetworkEvent({method:n.toUpperCase(),url:o,durationMs:Math.round(performance.now()-a),statusCode:m==null?void 0:m.status,requestBytes:l,responseBytes:m?Ef(m):void 0,error:g?String(g):void 0,timestamp:u})}catch{}};return e.call(this,t,c).then(m=>(p(m,null),m)).catch(m=>{throw p(null,m),m}).finally(()=>{h!==null&&Ee===h&&(Ee=null)})}}function $f(s){const e=window.onerror;window.onerror=(r,i,n,o,l)=>(l?s.addException({traceId:null,stackTrace:Bs(l),recordedAt:J(),isMessage:!1,distributedTraceId:je()}):s.addException({traceId:null,stackTrace:String(r),recordedAt:J(),isMessage:!1,distributedTraceId:je()}),typeof e=="function"?e(r,i,n,o,l):!1);const t=window.onunhandledrejection;window.onunhandledrejection=r=>{const i=r.reason;i instanceof Error?s.addException({traceId:null,stackTrace:Bs(i),recordedAt:J(),isMessage:!1,distributedTraceId:je()}):s.addException({traceId:null,stackTrace:String(i),recordedAt:J(),isMessage:!1,distributedTraceId:je()}),typeof t=="function"&&t.call(window,r)}}function Nf(s){try{return new URL(s,window.location.origin).origin===window.location.origin}catch{return!0}}function kf(s){if(s!=null){if(typeof s=="string")return s.length;if(s instanceof Blob)return s.size;if(s instanceof ArrayBuffer||ArrayBuffer.isView(s))return s.byteLength;s instanceof FormData}}function Tf(s){const e=XMLHttpRequest.prototype.open,t=XMLHttpRequest.prototype.send;XMLHttpRequest.prototype.open=function(r,i,...n){const o=String(i);return this._tracewaySameOrigin=Nf(o),this._tracewayMethod=r,this._tracewayUrl=o,this._tracewayRecorded=!1,e.apply(this,[r,i,...n])},XMLHttpRequest.prototype.send=function(r){this._tracewayStart=performance.now(),this._tracewayStartedAt=new Date().toISOString(),this._tracewayRequestBytes=kf(r);let i=null;this._tracewaySameOrigin&&(i=ce(),Cf(i),this.setRequestHeader("traceway-trace-id",i));const n=c=>{if(!this._tracewayRecorded){if(this._tracewayRecorded=!0,s)try{const h=this.getResponseHeader("content-length"),p=h!==null&&Number.isFinite(Number(h))?Number(h):void 0;s.recordNetworkEvent({method:(this._tracewayMethod??"GET").toUpperCase(),url:this._tracewayUrl??"",durationMs:Math.round(performance.now()-(this._tracewayStart??performance.now())),statusCode:this.status||void 0,requestBytes:this._tracewayRequestBytes,responseBytes:p,error:c?`XHR ${this.statusText||"error"}`:void 0,timestamp:this._tracewayStartedAt??new Date().toISOString()})}catch{}i!==null&&If(i),this.removeEventListener("load",o),this.removeEventListener("error",l),this.removeEventListener("abort",a),this.removeEventListener("timeout",u)}},o=()=>n(!1),l=()=>n(!0),a=()=>n(!0),u=()=>n(!0);return this.addEventListener("load",o),this.addEventListener("error",l),this.addEventListener("abort",a),this.addEventListener("timeout",u),t.call(this,r)}}var Pf=[{name:"debug",level:"debug"},{name:"log",level:"info"},{name:"info",level:"info"},{name:"warn",level:"warn"},{name:"error",level:"error"}];function Df(s){if(typeof s=="string")return s;if(s instanceof Error)return s.stack??s.message;if(s==null)return String(s);if(typeof s=="object")try{return JSON.stringify(s)}catch{return Object.prototype.toString.call(s)}return String(s)}function Lf(s){if(!(typeof console>"u"))for(const{name:e,level:t}of Pf){const r=console[e];typeof r=="function"&&(console[e]=function(...i){try{const n=i.map(Df).join(" ");s.recordLog(t,n)}catch{}return r.apply(console,i)})}}function Bo(){return typeof window>"u"||!window.location?"":window.location.pathname+window.location.search+window.location.hash}function _f(s){if(typeof window>"u"||!window.history)return;let e=Bo();const t=n=>{const o=Bo();if(o!==e){try{s.recordNavigationEvent({action:n,from:e,to:o})}catch{}e=o}},r=window.history.pushState;window.history.pushState=function(...n){const o=r.apply(this,n);return t("push"),o};const i=window.history.replaceState;window.history.replaceState=function(...n){const o=i.apply(this,n);return t("replace"),o},window.addEventListener("popstate",()=>t("pop")),window.addEventListener("hashchange",()=>t("push"))}var ee=null;function Ff(s,e={}){ee=new vf(s,e),typeof window<"u"&&($f(ee),Mf(ee),Tf(ee),ee.captureLogs&&Lf(ee),ee.captureNavigation&&_f(ee))}function Uf(s,e,t){ee&&ee.addException({traceId:null,stackTrace:Bs(s),recordedAt:J(),attributes:e,isMessage:!1,distributedTraceId:(t==null?void 0:t.distributedTraceId)??je()})}function Bf(s,e){ee&&ee.setAttribute(s,e)}function zf(s){ee&&ee.setAttributes(s)}function Wf(s,e={}){Ff(s,e),Gf()}function Gf(){const s=typeof jQuery<"u"?jQuery:typeof $<"u"&&typeof $.ajax=="function"?$:null;s&&s(document).ajaxError(function(e,t,r,i){var c;const n=i||(t==null?void 0:t.statusText)||"AJAX Error",o=(r==null?void 0:r.url)||"unknown",l=((r==null?void 0:r.type)||"GET").toUpperCase(),a=(t==null?void 0:t.status)||0,u=((c=t==null?void 0:t.getResponseHeader)==null?void 0:c.call(t,"traceway-trace-id"))??je();Uf(new Error(`${l} ${o} failed: ${a} ${n}`),{url:o,method:l,status:String(a)},{distributedTraceId:u??void 0})})}Wf("frontend-dev-token@http://localhost:8082/api/report",{recordAllSessions:!0}),zf({userId:"u_42",tenant:"acme-corp",plan:"pro",build_channel:"canary"}),Bf("experiment.cart_redesign",Math.random()<.5?"control":"variant_b");function se(s){var e=new Date().toLocaleTimeString(),t=$("#log"),r=t.text();r==="Waiting for actions..."&&(r=""),t.text(r+"["+e+"] "+s+`
-`)}$(function(){$("#btn-error").on("click",function(){se("Calling GET /api/test-error ..."),$.ajax({url:"/api/test-error",method:"GET",complete:function(s){var e=s.getResponseHeader("traceway-trace-id");se("Response: "+s.status+" — traceway-trace-id: "+(e||"not present")),s.status>=400&&se("Exception auto-captured by @tracewayapp/jquery with distributedTraceId="+e)}})}),$("#btn-success").on("click",function(){se("Calling GET /api/test-success ..."),$.ajax({url:"/api/test-success",method:"GET",complete:function(s){var e=s.getResponseHeader("traceway-trace-id");se("Response: "+s.status+" — traceway-trace-id: "+(e||"not present"))}})}),$("#btn-logs").on("click",function(){se("Emitting console logs at every level ..."),console.debug("traceway-demo: debug ping",{step:1}),console.log("traceway-demo: cart loaded",{items:3,total:49.99}),console.info("traceway-demo: user resumed session",{userId:"u_42"}),console.warn("traceway-demo: slow response from /api/test-success"),console.error("traceway-demo: payment retry exhausted",{attempts:3}),se("5 console.* calls captured into the rolling log buffer.")}),$("#btn-network").on("click",function(){se("Firing 4 network requests (mix of fetch + jQuery, success + 404) ..."),fetch("/api/test-success").then(function(s){se("fetch GET /api/test-success -> "+s.status)}),fetch("/api/test-log-levels").then(function(s){se("fetch GET /api/test-log-levels -> "+s.status)}),fetch("/api/does-not-exist").then(function(s){se("fetch GET /api/does-not-exist -> "+s.status)}),$.ajax({url:"/api/test-spans-with-logs",method:"GET",complete:function(s){se("jQuery GET /api/test-spans-with-logs -> "+s.status)}})}),$("#btn-clear").on("click",function(){$("#log").text("Waiting for actions...")})})})();
+`}function ns(e,t,r){if(e.startsWith("eval at ")){const s=e.match(/\(([^()]+):(\d+):(\d+)\)/);if(s)return[s[1],s[2],s[3]]}const o=e.match(/^(.*?) line (\d+) > (?:eval|Function)(?: line \d+ > (?:eval|Function))*$/);return o?[o[1],o[2],"1"]:[e,t,r]}function En(e){const t=e.lastIndexOf("/");t>=0&&(e=e.slice(t+1));const r=e.indexOf(".");return r>=0&&(e=e.slice(r+1)),e}function It(e){const t=e.split("/");return t[t.length-1]}var Pe=null;function ze(){return Pe}function ll(e){Pe=e}function ul(e){Pe===e&&(Pe=null)}function cl(e){try{return new URL(typeof e=="string"?e:e instanceof Request?e.url:e.href,window.location.origin).origin===window.location.origin}catch{return!0}}function hl(e){return typeof e=="string"?e:e instanceof URL?e.href:e instanceof Request?e.url:String(e)}function fl(e,t){return t!=null&&t.method?t.method:e instanceof Request?e.method:"GET"}function dl(e){if(e!=null&&e.body){if(typeof e.body=="string")return e.body.length;if(e.body instanceof Blob)return e.body.size;if(e.body instanceof ArrayBuffer||ArrayBuffer.isView(e.body))return e.body.byteLength}}function pl(e){const t=e.headers.get("content-length");if(t===null)return;const r=Number(t);return Number.isFinite(r)?r:void 0}function ml(e){const t=window.fetch;window.fetch=function(r,o){const s=cl(r),u=fl(r,o),c=hl(r),p=dl(o),l=performance.now(),d=new Date().toISOString();let i=o,f=null;if(s){f=be(),Pe=f;const n=new Headers(o==null?void 0:o.headers);n.set("traceway-trace-id",f),i={...o,headers:n}}const a=(n,m)=>{if(e)try{e.recordNetworkEvent({method:u.toUpperCase(),url:c,durationMs:Math.round(performance.now()-l),statusCode:n==null?void 0:n.status,requestBytes:p,responseBytes:n?pl(n):void 0,error:m?String(m):void 0,timestamp:d})}catch{}};return t.call(this,r,i).then(n=>(a(n,null),e&&e.captureHttpServerErrors&&n.status>=500&&e.captureHttpServerError(u.toUpperCase(),c,n.status),n)).catch(n=>{throw a(null,n),n}).finally(()=>{f!==null&&Pe===f&&(Pe=null)})}}function gl(e){const t=window.onerror;window.onerror=(o,s,u,c,p)=>(p?e.addException({traceId:null,stackTrace:is(p),recordedAt:X(),isMessage:!1,distributedTraceId:ze()}):e.addException({traceId:null,stackTrace:String(o),recordedAt:X(),isMessage:!1,distributedTraceId:ze()}),typeof t=="function"?t(o,s,u,c,p):!1);const r=window.onunhandledrejection;window.onunhandledrejection=o=>{const s=o.reason;s instanceof Error?e.addException({traceId:null,stackTrace:is(s),recordedAt:X(),isMessage:!1,distributedTraceId:ze()}):e.addException({traceId:null,stackTrace:String(s),recordedAt:X(),isMessage:!1,distributedTraceId:ze()}),typeof r=="function"&&r.call(window,o)}}function yl(e){try{return new URL(e,window.location.origin).origin===window.location.origin}catch{return!0}}function wl(e){if(e!=null){if(typeof e=="string")return e.length;if(e instanceof Blob)return e.size;if(e instanceof ArrayBuffer||ArrayBuffer.isView(e))return e.byteLength;e instanceof FormData}}function bl(e){const t=XMLHttpRequest.prototype.open,r=XMLHttpRequest.prototype.send;XMLHttpRequest.prototype.open=function(o,s,...u){const c=String(s);return this._tracewaySameOrigin=yl(c),this._tracewayMethod=o,this._tracewayUrl=c,this._tracewayRecorded=!1,t.apply(this,[o,s,...u])},XMLHttpRequest.prototype.send=function(o){this._tracewayStart=performance.now(),this._tracewayStartedAt=new Date().toISOString(),this._tracewayRequestBytes=wl(o);let s=null;this._tracewaySameOrigin&&(s=be(),ll(s),this.setRequestHeader("traceway-trace-id",s));const u=i=>{if(!this._tracewayRecorded){if(this._tracewayRecorded=!0,e)try{const f=this.getResponseHeader("content-length"),a=f!==null&&Number.isFinite(Number(f))?Number(f):void 0;e.recordNetworkEvent({method:(this._tracewayMethod??"GET").toUpperCase(),url:this._tracewayUrl??"",durationMs:Math.round(performance.now()-(this._tracewayStart??performance.now())),statusCode:this.status||void 0,requestBytes:this._tracewayRequestBytes,responseBytes:a,error:i?`XHR ${this.statusText||"error"}`:void 0,timestamp:this._tracewayStartedAt??new Date().toISOString()})}catch{}s!==null&&ul(s),this.removeEventListener("load",c),this.removeEventListener("error",p),this.removeEventListener("abort",l),this.removeEventListener("timeout",d)}},c=()=>u(!1),p=()=>u(!0),l=()=>u(!0),d=()=>u(!0);return this.addEventListener("load",c),this.addEventListener("error",p),this.addEventListener("abort",l),this.addEventListener("timeout",d),r.call(this,o)}}var Sl=[{name:"debug",level:"debug"},{name:"log",level:"info"},{name:"info",level:"info"},{name:"warn",level:"warn"},{name:"error",level:"error"}];function vl(e){if(typeof e=="string")return e;if(e instanceof Error)return e.stack??e.message;if(e==null)return String(e);if(typeof e=="object")try{return JSON.stringify(e)}catch{return Object.prototype.toString.call(e)}return String(e)}function Cl(e){if(!(typeof console>"u"))for(const{name:t,level:r}of Sl){const o=console[t];typeof o=="function"&&(console[t]=function(...s){try{const u=s.map(vl).join(" ");e.recordLog(r,u)}catch{}return o.apply(console,s)})}}function In(){return typeof window>"u"||!window.location?"":window.location.pathname+window.location.search+window.location.hash}function xl(e){if(typeof window>"u"||!window.history)return;let t=In();const r=u=>{const c=In();if(c!==t){try{e.recordNavigationEvent({action:u,from:t,to:c})}catch{}t=c}},o=window.history.pushState;window.history.pushState=function(...u){const c=o.apply(this,u);return r("push"),c};const s=window.history.replaceState;window.history.replaceState=function(...u){const c=s.apply(this,u);return r("replace"),c},window.addEventListener("popstate",()=>r("pop")),window.addEventListener("hashchange",()=>r("push"))}var oe=null;function Rl(e,t={}){oe=new al(e,t),typeof window<"u"&&(gl(oe),ml(oe),bl(oe),oe.captureLogs&&Cl(oe),oe.captureNavigation&&xl(oe))}function Ol(e,t,r){oe&&oe.addException({traceId:null,stackTrace:is(e),recordedAt:X(),attributes:t,isMessage:!1,distributedTraceId:(r==null?void 0:r.distributedTraceId)??ze()})}function Ml(e,t){oe&&oe.setAttribute(e,t)}function El(e){oe&&oe.setAttributes(e)}function Il(e,t={}){Rl(e,t),Al()}function Al(){const e=typeof jQuery<"u"?jQuery:typeof $<"u"&&typeof $.ajax=="function"?$:null;e&&e(document).ajaxError(function(t,r,o,s){var i;const u=s||(r==null?void 0:r.statusText)||"AJAX Error",c=(o==null?void 0:o.url)||"unknown",p=((o==null?void 0:o.type)||"GET").toUpperCase(),l=(r==null?void 0:r.status)||0,d=((i=r==null?void 0:r.getResponseHeader)==null?void 0:i.call(r,"traceway-trace-id"))??ze();Ol(new Error(`${p} ${c} failed: ${l} ${u}`),{url:c,method:p,status:String(l)},{distributedTraceId:d??void 0})})}Il("frontend-dev-token@http://localhost:8082/api/report",{recordAllSessions:!0}),El({userId:"u_42",tenant:"acme-corp",plan:"pro",build_channel:"canary"}),Ml("experiment.cart_redesign",Math.random()<.5?"control":"variant_b");function fe(e){var t=new Date().toLocaleTimeString(),r=$("#log"),o=r.text();o==="Waiting for actions..."&&(o=""),r.text(o+"["+t+"] "+e+`
+`)}$(function(){$("#btn-error").on("click",function(){fe("Calling GET /api/test-error ..."),$.ajax({url:"/api/test-error",method:"GET",complete:function(e){var t=e.getResponseHeader("traceway-trace-id");fe("Response: "+e.status+" — traceway-trace-id: "+(t||"not present")),e.status>=400&&fe("Exception auto-captured by @tracewayapp/jquery with distributedTraceId="+t)}})}),$("#btn-success").on("click",function(){fe("Calling GET /api/test-success ..."),$.ajax({url:"/api/test-success",method:"GET",complete:function(e){var t=e.getResponseHeader("traceway-trace-id");fe("Response: "+e.status+" — traceway-trace-id: "+(t||"not present"))}})}),$("#btn-logs").on("click",function(){fe("Emitting console logs at every level ..."),console.debug("traceway-demo: debug ping",{step:1}),console.log("traceway-demo: cart loaded",{items:3,total:49.99}),console.info("traceway-demo: user resumed session",{userId:"u_42"}),console.warn("traceway-demo: slow response from /api/test-success"),console.error("traceway-demo: payment retry exhausted",{attempts:3}),fe("5 console.* calls captured into the rolling log buffer.")}),$("#btn-network").on("click",function(){fe("Firing 4 network requests (mix of fetch + jQuery, success + 404) ..."),fetch("/api/test-success").then(function(e){fe("fetch GET /api/test-success -> "+e.status)}),fetch("/api/test-log-levels").then(function(e){fe("fetch GET /api/test-log-levels -> "+e.status)}),fetch("/api/does-not-exist").then(function(e){fe("fetch GET /api/does-not-exist -> "+e.status)}),$.ajax({url:"/api/test-spans-with-logs",method:"GET",complete:function(e){fe("jQuery GET /api/test-spans-with-logs -> "+e.status)}})}),$("#btn-clear").on("click",function(){$("#log").text("Waiting for actions...")})})})();
diff --git a/frontend/src/lib/components/app-sidebar.svelte b/frontend/src/lib/components/app-sidebar.svelte
index ec0f2d3e..afbfe6d7 100644
--- a/frontend/src/lib/components/app-sidebar.svelte
+++ b/frontend/src/lib/components/app-sidebar.svelte
@@ -10,6 +10,7 @@
Workflow,
Bug,
Link2,
+ ChartGantt,
ChartNoAxesCombined,
FileText,
Film,
@@ -67,6 +68,7 @@
{ Icon: FileText, href: '/logs', title: 'Logs', stickyParams: ['preset', 'from', 'to'] },
{ Icon: Gauge, href: '/endpoints', title: 'Endpoints', stickyParams: ['preset', 'from', 'to'] },
{ Icon: ListEnd, href: '/tasks', title: 'Tasks', stickyParams: ['preset', 'from', 'to'] },
+ { Icon: ChartGantt, href: '/spans', title: 'Spans', stickyParams: ['preset', 'from', 'to'] },
{ Icon: Flame, href: '/profiles', title: 'Profiles', stickyParams: ['preset', 'from', 'to'] },
{ Icon: Film, href: '/sessions', title: 'Sessions', stickyParams: ['preset', 'from', 'to'] },
{
diff --git a/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte b/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte
index 5e31cce3..e76c9820 100644
--- a/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte
+++ b/frontend/src/lib/components/distributed-trace/distributed-trace-card.svelte
@@ -1,26 +1,29 @@
<script lang="ts">
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { gotoHref } from '$lib/utils/navigation';
import { api } from '$lib/api';
import * as Card from '$lib/components/ui/card';
import { Badge } from '$lib/components/ui/badge';
+ import SpanGraphNotice from '$lib/components/spans/span-graph-notice.svelte';
import { Button } from '$lib/components/ui/button';
import { LoadingCircle } from '$lib/components/ui/loading-circle';
import { formatDuration, getStatusColor } from '$lib/utils/formatters';
- import { ArrowRight, GitBranch } from '@lucide/svelte';
+ import { ArrowRight, GitBranch, ChevronRight, ChevronDown } from '@lucide/svelte';
+ import { SvelteSet } from 'svelte/reactivity';
+ import { distributedTraceTree } from '$lib/utils/distributed-trace-tree';
import type {
DistributedTraceResponse,
DistributedTraceNode
} from '$lib/types/distributed-trace';
interface Props {
- distributedTraceId: string;
+ traceId: string;
currentExceptionHash?: string;
currentNodeId?: string;
recordedAt?: string;
}
- let { distributedTraceId, currentExceptionHash, currentNodeId, recordedAt }: Props = $props();
+ let { traceId, currentExceptionHash, currentNodeId, recordedAt }: Props = $props();
function isCurrentNode(node: DistributedTraceNode): boolean {
if (currentExceptionHash && node.traceType === 'exception') {
@@ -36,20 +39,32 @@
return false;
}
- let response = $state<DistributedTraceResponse | null>(null);
+ let response = $state.raw<DistributedTraceResponse | null>(null);
let loading = $state(true);
+ const collapsed = new SvelteSet<string>();
+ const treeRows = $derived(distributedTraceTree(response?.nodes ?? [], collapsed));
+ function toggle(key: string) {
+ if (collapsed.has(key)) collapsed.delete(key);
+ else collapsed.add(key);
+ }
+
+ let loadSequence = 0;
async function loadTrace() {
+ const sequence = ++loadSequence;
loading = true;
try {
- response = (await api.post(
- `/distributed-traces/${distributedTraceId}`,
+ const result = (await api.post(
+ `/distributed-traces/${traceId}`,
recordedAt ? { recordedAt } : {}
)) as DistributedTraceResponse;
+ if (sequence !== loadSequence) return;
+ response = result;
} catch {
+ if (sequence !== loadSequence) return;
response = null;
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -72,8 +87,17 @@
}
}
- onMount(() => {
- loadTrace();
+ $effect(() => {
+ void traceId;
+ void recordedAt;
+ untrack(() => {
+ response = null;
+ collapsed.clear();
+ loadTrace();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
@@ -103,54 +127,75 @@
</Card.Header>
<Card.Content>
<div class="space-y-3">
- {#each response.nodes as node, i (i)}
- <div class="flex items-center gap-3 rounded-md border p-3 {i > 0 ? 'ml-6' : ''}">
- <div class="flex min-w-0 flex-1 items-center gap-3">
- <Badge variant="outline" class="shrink-0">{node.projectName}</Badge>
- <span class="truncate font-mono text-sm">
- {#if node.traceType === 'task'}
- {node.task?.taskName}
- {:else if node.traceType === 'ai_trace'}
- {node.aiTrace?.traceName}
- {:else if node.traceType === 'exception'}
- {node.exception?.stackTrace.split('\n')[0]}
- {:else}
- {node.endpoint?.endpoint}
- {/if}
- </span>
- {#if node.traceType === 'endpoint' && node.endpoint}
- <span class="shrink-0 font-mono text-sm {getStatusColor(node.endpoint.statusCode)}">
- {node.endpoint.statusCode}
- </span>
- {/if}
- {#if node.traceType === 'ai_trace' && node.aiTrace}
- <Badge variant="secondary" class="shrink-0"
- >{node.aiTrace.provider || node.aiTrace.model || 'AI'}</Badge
+ {#each treeRows as row (row.key)}
+ {@const node = row.node}
+ <div
+ class="flex flex-col gap-3 rounded-md border p-3"
+ style:margin-left={`${row.depth * 24}px`}
+ >
+ <div class="flex items-center gap-3">
+ {#if row.hasChildren}
+ <button
+ class="shrink-0 rounded p-1 hover:bg-muted"
+ onclick={() => toggle(row.key)}
+ aria-label={collapsed.has(row.key) ? 'Expand children' : 'Collapse children'}
+ aria-expanded={!collapsed.has(row.key)}
>
+ {#if collapsed.has(row.key)}<ChevronRight class="h-4 w-4" />{:else}<ChevronDown
+ class="h-4 w-4"
+ />{/if}
+ </button>
{/if}
- {#if node.traceType !== 'exception'}
- <span class="shrink-0 font-mono text-sm text-muted-foreground">
- {formatDuration(
- node.traceType === 'task'
- ? (node.task?.duration ?? 0)
- : node.traceType === 'ai_trace'
- ? (node.aiTrace?.duration ?? 0)
- : (node.endpoint?.duration ?? 0)
- )}
+ <div class="flex min-w-0 flex-1 items-center gap-3">
+ <Badge variant="outline" class="shrink-0">{node.projectName}</Badge>
+ <span class="truncate font-mono text-sm">
+ {#if node.traceType === 'task'}
+ {node.task?.taskName}
+ {:else if node.traceType === 'ai_trace'}
+ {node.aiTrace?.traceName}
+ {:else if node.traceType === 'exception'}
+ {node.exception?.stackTrace.split('\n')[0]}
+ {:else}
+ {node.endpoint?.endpoint}
+ {/if}
</span>
- {/if}
- {#if node.exception}
- <Badge variant="destructive" class="shrink-0">Exception</Badge>
+ {#if node.traceType === 'endpoint' && node.endpoint}
+ <span
+ class="shrink-0 font-mono text-sm {getStatusColor(node.endpoint.statusCode)}"
+ >
+ {node.endpoint.statusCode}
+ </span>
+ {/if}
+ {#if node.traceType === 'ai_trace' && node.aiTrace}
+ <Badge variant="secondary" class="shrink-0"
+ >{node.aiTrace.provider || node.aiTrace.model || 'AI'}</Badge
+ >
+ {/if}
+ {#if node.traceType !== 'exception'}
+ <span class="shrink-0 font-mono text-sm text-muted-foreground">
+ {formatDuration(
+ node.traceType === 'task'
+ ? (node.task?.duration ?? 0)
+ : node.traceType === 'ai_trace'
+ ? (node.aiTrace?.duration ?? 0)
+ : (node.endpoint?.duration ?? 0)
+ )}
+ </span>
+ {/if}
+ {#if node.exception}
+ <Badge variant="destructive" class="shrink-0">Exception</Badge>
+ {/if}
+ </div>
+ {#if isCurrentNode(node)}
+ <Badge class="bg-blue-500 text-white hover:bg-blue-500">You're here</Badge>
+ {:else}
+ <Button variant="ghost" size="sm" onclick={() => navigateToNode(node)}>
+ View
+ <ArrowRight class="ml-1 h-3 w-3" />
+ </Button>
{/if}
</div>
- {#if isCurrentNode(node)}
- <Badge class="bg-blue-500 text-white hover:bg-blue-500">You're here</Badge>
- {:else}
- <Button variant="ghost" size="sm" onclick={() => navigateToNode(node)}>
- View
- <ArrowRight class="ml-1 h-3 w-3" />
- </Button>
- {/if}
+ <SpanGraphNotice status={node.spanGraphStatus} />
</div>
{/each}
</div>
diff --git a/frontend/src/lib/components/issues/event-card.svelte b/frontend/src/lib/components/issues/event-card.svelte
index 74de4f3d..d2d4402f 100644
--- a/frontend/src/lib/components/issues/event-card.svelte
+++ b/frontend/src/lib/components/issues/event-card.svelte
@@ -36,6 +36,12 @@
timezone
}: Props = $props();
+ const RELATED_LABELS = {
+ endpoint: { label: 'Endpoint', route: 'endpoints' },
+ task: { label: 'Task', route: 'tasks' },
+ ai_trace: { label: 'AI Trace', route: 'ai-traces' }
+ } as const;
+
const tz = $derived(timezone ?? getTimezone());
const isFrontend = $derived(
projectsState.currentProject?.framework
@@ -185,18 +191,13 @@
<!-- Related Trace -->
{#if linkedTrace}
+ {@const related = RELATED_LABELS[linkedTrace.traceType]}
<hr class="border-border" />
<div>
- <p class="mb-3 text-sm font-medium">
- {linkedTrace.traceType === 'task' ? 'Related Task' : 'Related Endpoint'}
- </p>
+ <p class="mb-3 text-sm font-medium">Related {related.label}</p>
<div class="mb-4 grid grid-cols-2 gap-4 md:grid-cols-4">
- <LabelValue
- label={linkedTrace.traceType === 'task' ? 'Task' : 'Endpoint'}
- value={linkedTrace.endpoint}
- mono
- />
- {#if linkedTrace.traceType !== 'task'}
+ <LabelValue label={related.label} value={linkedTrace.endpoint} mono />
+ {#if linkedTrace.traceType === 'endpoint'}
<LabelValue
label="Status"
value={linkedTrace.statusCode}
@@ -212,43 +213,27 @@
mono
/>
</div>
- {#if linkedTrace.traceType === 'task'}
- <Button
- variant="outline"
- size="sm"
- onclick={() =>
- goto(
- resolve(
- `/tasks/${encodeURIComponent(linkedTrace.endpoint)}/${linkedTrace.id}?preset=24h`
- )
- )}
- >
- View Task Details
- <ArrowRight class="ml-2 h-4 w-4" />
- </Button>
- {:else}
- <Button
- variant="outline"
- size="sm"
- onclick={() =>
- goto(
- resolve(
- `/endpoints/${encodeURIComponent(linkedTrace.endpoint)}/${linkedTrace.id}?preset=24h`
- )
- )}
- >
- View Endpoint Details
- <ArrowRight class="ml-2 h-4 w-4" />
- </Button>
- {/if}
+ <Button
+ variant="outline"
+ size="sm"
+ onclick={() =>
+ goto(
+ resolve(
+ `/${related.route}/${encodeURIComponent(linkedTrace.endpoint)}/${linkedTrace.id}?preset=24h&t=${encodeURIComponent(linkedTrace.recordedAt)}`
+ )
+ )}
+ >
+ View {related.label} Details
+ <ArrowRight class="ml-2 h-4 w-4" />
+ </Button>
</div>
{/if}
</Card.Content>
</Card.Root>
-{#if linkedTrace?.distributedTraceId || occurrence.distributedTraceId}
+{#if occurrence.traceId}
<DistributedTraceCard
- distributedTraceId={(linkedTrace?.distributedTraceId ?? occurrence.distributedTraceId)!}
+ traceId={occurrence.traceId}
currentExceptionHash={occurrence.exceptionHash}
recordedAt={linkedTrace?.recordedAt ?? occurrence.recordedAt}
/>
diff --git a/frontend/src/lib/components/spans/span-graph-notice.svelte b/frontend/src/lib/components/spans/span-graph-notice.svelte
new file mode 100644
index 00000000..21cfa23c
--- /dev/null
+++ b/frontend/src/lib/components/spans/span-graph-notice.svelte
@@ -0,0 +1,34 @@
+<script lang="ts">
+ import type { SpanGraphStatus } from '$lib/types/spans';
+ import { WarningCallout } from '$lib/components/ui/warning-callout';
+
+ let { status }: { status?: SpanGraphStatus | null } = $props();
+</script>
+
+{#if status && status.state !== 'complete'}
+ <WarningCallout
+ title={status.state === 'unavailable'
+ ? 'Spans are temporarily unavailable'
+ : status.reasons?.includes('row_limit')
+ ? 'This trace is too large to show in full'
+ : 'Some trace details could not be loaded'}
+ >
+ {#if status.state === 'unavailable'}
+ The trace could not be loaded within the read limits. Refresh to try again.
+ {:else if status.reasons?.includes('most_important')}
+ The available spans prioritize errors, entry points and slowest spans. Other spans may be
+ missing.
+ {:else if status.reasons?.includes('row_limit')}
+ The spans shown are only part of the trace.
+ {:else}
+ The available spans are shown, but some attributes could not be loaded.
+ {/if}
+ {#if status.state === 'unavailable' || status.reasons?.includes('row_limit')}
+ Exceptions attached to missing spans may also be absent from this view.
+ {/if}
+ {#if status.omittedAttributes}
+ Attributes are missing from {status.omittedAttributes}
+ {status.omittedAttributes === 1 ? 'span' : 'spans'}.
+ {/if}
+ </WarningCallout>
+{/if}
diff --git a/frontend/src/lib/components/spans/span-graph-notice.test.ts b/frontend/src/lib/components/spans/span-graph-notice.test.ts
new file mode 100644
index 00000000..5177d6b4
--- /dev/null
+++ b/frontend/src/lib/components/spans/span-graph-notice.test.ts
@@ -0,0 +1,48 @@
+import { afterEach, expect, it } from 'vitest';
+import { cleanup, render } from '@testing-library/svelte';
+import SpanGraphNotice from './span-graph-notice.svelte';
+
+afterEach(cleanup);
+
+it('does not warn for a complete or legacy response', () => {
+ const { queryByRole, rerender } = render(SpanGraphNotice, { status: { state: 'complete' } });
+ expect(queryByRole('alert')).toBeNull();
+ return rerender({ status: undefined }).then(() => expect(queryByRole('alert')).toBeNull());
+});
+
+it('identifies a truncated graph', () => {
+ const { getByRole } = render(SpanGraphNotice, {
+ status: { state: 'partial', reasons: ['row_limit'] }
+ });
+ expect(getByRole('alert').textContent?.replace(/\s+/g, ' ')).toContain('only part of the trace');
+ expect(getByRole('alert').textContent?.replace(/\s+/g, ' ')).toContain(
+ 'Exceptions attached to missing spans'
+ );
+});
+
+it('explains omitted attributes', () => {
+ const { getByRole } = render(SpanGraphNotice, {
+ status: { state: 'partial', reasons: ['attribute_size'], omittedAttributes: 2 }
+ });
+ expect(getByRole('alert').textContent?.replace(/\s+/g, ' ')).toContain(
+ 'Attributes are missing from 2 spans'
+ );
+});
+
+it('distinguishes an unavailable graph from a trace with no spans', () => {
+ const { getByRole } = render(SpanGraphNotice, {
+ status: { state: 'unavailable', reasons: ['read_limit'] }
+ });
+ expect(getByRole('alert').textContent?.replace(/\s+/g, ' ')).toContain(
+ 'Spans are temporarily unavailable'
+ );
+});
+
+it('says which spans were kept when a whole trace is over the cap', () => {
+ const { getByRole } = render(SpanGraphNotice, {
+ status: { state: 'partial', reasons: ['row_limit', 'most_important'] }
+ });
+ expect(getByRole('alert').textContent?.replace(/\s+/g, ' ')).toContain(
+ 'errors, entry points and slowest spans'
+ );
+});
diff --git a/frontend/src/lib/components/spans/span-row.svelte b/frontend/src/lib/components/spans/span-row.svelte
index f4991fdd..2382edac 100644
--- a/frontend/src/lib/components/spans/span-row.svelte
+++ b/frontend/src/lib/components/spans/span-row.svelte
@@ -1,11 +1,16 @@
<script lang="ts">
- import type { Span } from '$lib/types/spans';
+ import type { Span, SpanAttributeLoader } from '$lib/types/spans';
import { cn } from '$lib/utils';
import { formatDuration, preciseTimeMs } from '$lib/utils/formatters';
import * as Popover from '$lib/components/ui/popover';
import CopyButton from '$lib/components/traceway/copy-button.svelte';
import ChevronRight from '@lucide/svelte/icons/chevron-right';
import ChevronDown from '@lucide/svelte/icons/chevron-down';
+ import CircleAlert from '@lucide/svelte/icons/circle-alert';
+ import Unlink from '@lucide/svelte/icons/unlink';
+ import { spanDisplayName } from '$lib/utils/span-display';
+ import { resolveHref } from '$lib/utils/links';
+ import { isErrorSpan, spanKindLabel, spanStatusLabel } from '$lib/utils/span-fields';
type Props = {
row: number;
@@ -15,6 +20,15 @@
isOdd: boolean;
depth: number;
hasChildren: boolean;
+ missingParent?: boolean;
+ missingParentLabel?: string;
+ descendants?: number;
+ errorBelow?: boolean;
+ loadAttributes?: SpanAttributeLoader;
+ projectName?: string;
+ showService?: boolean;
+ isSelected?: boolean;
+ traceHref?: string;
isCollapsed: boolean;
onToggle: () => void;
nameColumnWidth: number;
@@ -33,6 +47,15 @@
isOdd,
depth,
hasChildren,
+ missingParent = false,
+ missingParentLabel = 'Parent span is unavailable',
+ descendants = 0,
+ errorBelow = false,
+ loadAttributes,
+ projectName,
+ showService = false,
+ isSelected = false,
+ traceHref,
isCollapsed,
onToggle,
nameColumnWidth,
@@ -96,22 +119,63 @@
}
let nameElement: HTMLSpanElement;
+ let pillElement = $state<HTMLButtonElement>();
+
+ const isError = $derived(isErrorSpan(span));
+ const fieldEntries = $derived(
+ [
+ ['Project', projectName],
+ ['Service', span.serviceName],
+ ['Kind', spanKindLabel(span.spanKind)],
+ ['Status', spanStatusLabel(span.statusCode)],
+ ['Scope', span.scopeName]
+ ].filter((entry): entry is [string, string] => !!entry[1])
+ );
+
+ // Spans of a whole trace arrive without attributes. They are fetched once, when the popover first opens.
+ let lazyAttributes = $state<Record<string, string> | null>(null);
+ let lazyState = $state<'idle' | 'loading' | 'failed'>('idle');
+ let lazyOmitted = $state(false);
+
+ async function ensureAttributes(open: boolean) {
+ if (!open || !loadAttributes || span.attributes || lazyAttributes || lazyState === 'loading') {
+ return;
+ }
+ lazyState = 'loading';
+ try {
+ const loaded = await loadAttributes(span);
+ lazyAttributes = loaded.attributes ?? {};
+ lazyOmitted = !!loaded.attributesOmitted;
+ lazyState = 'idle';
+ } catch (e) {
+ console.error(e);
+ lazyState = 'failed';
+ }
+ }
const attributeEntries = $derived(
- Object.entries(span.attributes ?? {}).sort((a, b) => a[0].localeCompare(b[0]))
+ Object.entries(span.attributes ?? lazyAttributes ?? {}).sort((a, b) => a[0].localeCompare(b[0]))
);
+ const foldedCount = $derived(isCollapsed && hasChildren ? descendants : 0);
+
$effect(() => {
if (nameElement) {
// Measure the natural width needed, including indentation and chevron
- const naturalWidth = nameElement.scrollWidth + depth * 16 + 44;
+ const pillWidth = foldedCount > 0 && pillElement ? pillElement.offsetWidth + 12 : 0;
+ const naturalWidth = nameElement.scrollWidth + depth * 16 + 44 + pillWidth;
updateNameWidth?.(naturalWidth);
}
});
</script>
<div
- class={cn('flex items-center border-b border-border last:border-b-0', isOdd ? 'bg-muted/40' : '')}
+ class={cn(
+ 'flex items-center border-b border-border last:border-b-0',
+ isOdd ? 'bg-muted/40' : '',
+ isSelected && 'bg-primary/10'
+ )}
+ aria-current={isSelected ? 'true' : undefined}
>
<!-- Span name -->
<div
@@ -124,6 +188,7 @@
onclick={onToggle}
class="mr-1 shrink-0 rounded p-0.5 text-muted-foreground hover:bg-muted hover:text-foreground"
aria-label={isCollapsed ? 'Expand children' : 'Collapse children'}
+ aria-expanded={!isCollapsed}
>
{#if isCollapsed}
<ChevronRight class="h-3 w-3" />
@@ -134,18 +199,60 @@
{:else}
<span class="mr-1 w-4 shrink-0"></span>
{/if}
- <Popover.Root>
+ <Popover.Root onOpenChange={ensureAttributes}>
<Popover.Trigger class="min-w-0 flex-1 cursor-pointer text-left">
<span bind:this={nameElement} class="block truncate font-mono text-xs">
- {span.name}
+ {#if isError}<CircleAlert
+ class="mr-1 inline h-3 w-3 align-[-2px] text-destructive"
+ aria-label="Error"
+ />{/if}{#if missingParent}<Unlink
+ class="mr-1 inline h-3 w-3 align-[-2px] text-muted-foreground"
+ aria-label={missingParentLabel}
+ />{/if}{#if showService && span.serviceName}<span class="mr-1.5 text-muted-foreground"
+ >{span.serviceName}</span
+ >{/if}{spanDisplayName(span)}
</span>
</Popover.Trigger>
<Popover.Content class="w-auto max-w-sm" align="start">
<div class="flex max-h-[60vh] flex-col gap-2 overflow-y-auto">
+ {#if span.attributesOmitted || lazyOmitted || lazyState === 'failed'}<p
+ class="text-xs text-muted-foreground"
+ >
+ Attributes could not be loaded for this span.
+ </p>{/if}
+ {#if missingParent}<p class="text-xs text-muted-foreground">{missingParentLabel}.</p>{/if}
+ <div class="text-xs text-muted-foreground">
+ Trace
+ {#if traceHref}<a
+ class="underline underline-offset-2"
+ {...{ href: resolveHref(traceHref) }}><code>{span.traceId}</code></a
+ >{:else}<code>{span.traceId}</code>{/if}
+ </div>
+ <div class="text-xs text-muted-foreground">Span <code>{span.spanId}</code></div>
+ {#if fieldEntries.length > 0}
+ <dl class="grid grid-cols-[auto_1fr] gap-x-3 gap-y-0.5 text-xs">
+ {#each fieldEntries as [label, value] (label)}
+ <dt class="text-muted-foreground">{label}</dt>
+ <dd
+ class={cn(
+ 'font-mono break-all select-text',
+ label === 'Status' && isError && 'text-destructive'
+ )}
+ >
+ {value}
+ </dd>
+ {/each}
+ </dl>
+ {/if}
<div class="flex items-start gap-2">
<span class="font-mono text-xs break-all select-text">{span.name}</span>
<CopyButton bare text={span.name} iconClass="h-3.5 w-3.5" class="shrink-0" />
</div>
+ {#if lazyState === 'loading'}
+ <p class="border-t border-border pt-2 text-xs text-muted-foreground">
+ Loading attributes...
+ </p>
+ {/if}
{#if attributeEntries.length > 0}
<div class="flex flex-col gap-1.5 border-t border-border pt-2">
{#each attributeEntries as [key, value], __index (__index)}
@@ -162,12 +269,27 @@
</div>
</Popover.Content>
</Popover.Root>
+ {#if foldedCount > 0}
+ <button
+ bind:this={pillElement}
+ type="button"
+ onclick={onToggle}
+ class="ml-2 inline-flex shrink-0 cursor-pointer items-center gap-1 rounded-full bg-muted px-1.5 py-px font-mono text-[10px] text-muted-foreground hover:bg-muted-foreground/20 hover:text-foreground"
+ aria-label={`Show ${foldedCount} hidden ${foldedCount === 1 ? 'span' : 'spans'}`}
+ >
+ {#if errorBelow}<CircleAlert
+ class="h-2.5 w-2.5 text-destructive"
+ aria-label="Error in the hidden spans"
+ />{/if}+{foldedCount.toLocaleString()}
+ </button>
+ {/if}
</div>
<!-- Timeline bar -->
<div
class="relative flex min-w-[200px] flex-1 items-center self-stretch"
bind:this={containerElement}
+ role="presentation"
onmouseenter={containerSpanCellHandleMouseEnter}
onmousemove={containerSpanCellHandleMouseMove}
onmouseleave={spanCellHandleMouseLeave}
diff --git a/frontend/src/lib/components/spans/span-waterfall.svelte b/frontend/src/lib/components/spans/span-waterfall.svelte
index e02cbd6a..34f3e363 100644
--- a/frontend/src/lib/components/spans/span-waterfall.svelte
+++ b/frontend/src/lib/components/spans/span-waterfall.svelte
@@ -1,52 +1,151 @@
<script lang="ts">
- import type { Span } from '$lib/types/spans';
- import { SvelteSet } from 'svelte/reactivity';
+ import type { Span, SpanAttributeLoader } from '$lib/types/spans';
+ import { SvelteMap, SvelteSet } from 'svelte/reactivity';
+ import { untrack } from 'svelte';
import ScrollArea from '../ui/scroll-area/scroll-area.svelte';
import SpanRow from './span-row.svelte';
import { preciseTimeMs } from '$lib/utils/formatters';
- import { flattenSpanTree } from '$lib/utils/span-tree';
+ import { buildSpanTree, flattenBuiltTree, initiallyCollapsed } from '$lib/utils/span-tree';
+ import ChevronsDownUp from '@lucide/svelte/icons/chevrons-down-up';
+ import ChevronsUpDown from '@lucide/svelte/icons/chevrons-up-down';
+ import { spanServices } from '$lib/utils/span-fields';
+ import { spanTraceHref } from '$lib/utils/span-explorer';
+ import { resolveHref } from '$lib/utils/links';
type Props = {
spans: Span[];
traceDuration: number;
traceStartTime: string;
+ // The trace the waterfall belongs to, for the link to the whole trace. Without it the first span names the trace.
+ traceId?: string;
rootSpanId?: string;
+ linkTraces?: boolean;
+ selectedSpanId?: string | null;
+ // Set by a page whose spans arrive without attributes: the popover asks for them when it opens.
+ loadAttributes?: SpanAttributeLoader;
+ missingParentLabel?: string;
+ // Set by the whole trace view, whose spans come from every project of the organization.
+ projectNames?: Record<string, string>;
};
- let { spans: rawSpans, traceDuration, traceStartTime, rootSpanId }: Props = $props();
+ let {
+ spans: rawSpans,
+ traceDuration,
+ traceStartTime,
+ traceId,
+ rootSpanId,
+ linkTraces = true,
+ selectedSpanId = null,
+ loadAttributes,
+ missingParentLabel = 'Parent span is unavailable',
+ projectNames
+ }: Props = $props();
- let collapsed = $state(new Set<string>());
+ const services = $derived(spanServices(rawSpans));
+ const showService = $derived(services.length > 1);
- function toggleCollapse(id: string) {
- const next = new SvelteSet(collapsed);
- if (next.has(id)) {
- next.delete(id);
- } else {
- next.add(id);
- }
- collapsed = next;
+ // One colour per service once a trace crosses services, so the hops read at a glance. A single service keeps a colour per row.
+ function colorIndex(service: string | undefined, row: number): number {
+ return showService ? Math.max(0, services.indexOf(service ?? '')) : row;
+ }
+
+ // A complete trace can hold thousands of spans. Rendering them all at once stalls the page.
+ const ROW_PAGE = 500;
+ let visibleRows = $state(ROW_PAGE);
+
+ const wholeTraceHref = $derived.by(() => {
+ if (!linkTraces) return undefined;
+ if (traceId) return spanTraceHref({ traceId, spanId: '', recordedAt: traceStartTime });
+ return rawSpans.length ? spanTraceHref(rawSpans[0]) : undefined;
+ });
+
+ const tree = $derived(buildSpanTree(rawSpans, rootSpanId, !!projectNames));
+
+ // A large trace opens with its widest nodes folded. What someone clicks is kept as the difference from that default,
+ // so the default can follow the spans without an effect.
+ const foldedByDefault = $derived(initiallyCollapsed(tree));
+ const flipped = new SvelteSet<string>();
+ const collapsed = $derived(
+ new Set([
+ ...[...foldedByDefault].filter((key) => !flipped.has(key)),
+ ...[...flipped].filter((key) => !foldedByDefault.has(key))
+ ])
+ );
+
+ function toggleCollapse(key: string) {
+ if (!flipped.delete(key)) flipped.add(key);
+ }
+
+ function flipExactly(keys: Iterable<string>) {
+ flipped.clear();
+ for (const key of keys) flipped.add(key);
+ visibleRows = ROW_PAGE;
}
- const treeRows = $derived(flattenSpanTree(rawSpans, rootSpanId, collapsed));
+ const expandAll = () => flipExactly(foldedByDefault);
+ const collapseAll = () =>
+ flipExactly([...tree.childrenById.keys()].filter((key) => !foldedByDefault.has(key)));
+
+ const treeRows = $derived(flattenBuiltTree(tree, collapsed));
+
+ $effect(() => {
+ const currentTree = tree;
+ const selection = selectedSpanId;
+ if (!selection) return;
+ untrack(() => {
+ const parents = new SvelteMap<string, string>();
+ for (const [parent, children] of currentTree.childrenById) {
+ for (const child of children) parents.set(child, parent);
+ }
+ for (const [key, span] of currentTree.byId) {
+ if (span.spanId !== selection) continue;
+ for (let parent = parents.get(key); parent; parent = parents.get(parent)) {
+ if (foldedByDefault.has(parent)) flipped.add(parent);
+ else flipped.delete(parent);
+ }
+ }
+ const index = treeRows.findLastIndex((row) => row.span.spanId === selection);
+ visibleRows = Math.max(visibleRows, index + 1);
+ });
+ });
const traceStart = $derived(
- rawSpans.length === 0
- ? preciseTimeMs(traceStartTime)
- : rawSpans.reduce((earliest, s) => {
- const sTime = preciseTimeMs(s.startTime);
- return sTime < earliest ? sTime : earliest;
- }, preciseTimeMs(rawSpans[0].startTime))
+ rawSpans.reduce(
+ (earliest, span) => Math.min(earliest, preciseTimeMs(span.startTime)),
+ preciseTimeMs(traceStartTime)
+ )
);
- const durationMs = $derived(traceDuration / 1_000_000);
+ const timelineDuration = $derived(
+ Math.max(
+ traceDuration,
+ ...rawSpans.map(
+ (span) => (preciseTimeMs(span.startTime) - traceStart) * 1_000_000 + span.duration
+ ),
+ (preciseTimeMs(traceStartTime) - traceStart) * 1_000_000 + traceDuration,
+ 1
+ )
+ );
+ const durationMs = $derived(timelineDuration / 1_000_000);
- let nameColumnWidth = $state(180); // default minimum
+ let measuredNameWidth = $state(180); // default minimum
+ let containerWidth = $state(0);
+
+ // On a phone a 400px name column would push every bar off screen.
+ const nameColumnWidth = $derived(
+ containerWidth > 0
+ ? Math.min(measuredNameWidth, Math.max(140, Math.round(containerWidth * 0.5)))
+ : measuredNameWidth
+ );
+
+ // On a narrow screen the prefix would crowd out the span name. The bar colour and the popover still name the service.
+ const showServiceLabel = $derived(showService && (containerWidth === 0 || containerWidth >= 520));
function updateNameWidth(width: number) {
if (width > 400) {
width = 400;
}
- if (width > nameColumnWidth) {
- nameColumnWidth = width;
+ if (width > measuredNameWidth) {
+ measuredNameWidth = width;
}
}
@@ -68,67 +167,132 @@
isHovered = false;
}
- let timelineElement: HTMLDivElement;
+ let timelineElement = $state<HTMLDivElement>();
</script>
-<ScrollArea orientation="horizontal" class="p-relative rounded-md border border-border">
- <div class="relative overflow-hidden">
- <!-- Header -->
- <div class="flex border-b border-border bg-muted/30">
- <div
- class="flex-shrink-0 border-r border-border px-3 py-1.5 text-xs font-medium"
- style="min-width: {nameColumnWidth}px"
- >
- Span Name
- </div>
- <div bind:this={timelineElement} class="min-w-[200px] flex-1 px-3 py-1.5">
- <div class="flex justify-between text-xs text-muted-foreground">
- <span>0ms</span>
- <span>{(durationMs / 2).toFixed(0)}ms</span>
- <span>{durationMs.toFixed(0)}ms</span>
+<div bind:clientWidth={containerWidth}>
+ <ScrollArea orientation="horizontal" class="p-relative rounded-md border border-border">
+ <div class="relative overflow-hidden">
+ <!-- Header -->
+ <div class="flex border-b border-border bg-muted/30">
+ <div
+ class="flex-shrink-0 border-r border-border px-3 py-1.5 text-xs font-medium"
+ style="min-width: {nameColumnWidth}px"
+ >
+ <div class="flex items-center justify-between gap-2">
+ <span>Span Name</span>
+ {#if tree.childrenById.size > 0}
+ <span class="flex items-center gap-0.5">
+ <button
+ type="button"
+ class="cursor-pointer rounded p-0.5 text-muted-foreground hover:bg-muted hover:text-foreground"
+ aria-label="Expand all"
+ title="Expand all"
+ onclick={expandAll}
+ >
+ <ChevronsUpDown class="h-3.5 w-3.5" />
+ </button>
+ <button
+ type="button"
+ class="cursor-pointer rounded p-0.5 text-muted-foreground hover:bg-muted hover:text-foreground"
+ aria-label="Collapse all"
+ title="Collapse all"
+ onclick={collapseAll}
+ >
+ <ChevronsDownUp class="h-3.5 w-3.5" />
+ </button>
+ </span>
+ {/if}
+ </div>
+ </div>
+ <div bind:this={timelineElement} class="min-w-[200px] flex-1 px-3 py-1.5">
+ <div class="flex justify-between text-xs text-muted-foreground">
+ <span>0ms</span>
+ <span>{(durationMs / 2).toFixed(0)}ms</span>
+ <span>{durationMs.toFixed(0)}ms</span>
+ </div>
+ </div>
+ <div
+ class="w-[100px] flex-shrink-0 border-l border-border px-3 py-1.5 text-right text-xs font-medium"
+ >
+ Duration
</div>
</div>
- <div
- class="w-[100px] flex-shrink-0 border-l border-border px-3 py-1.5 text-right text-xs font-medium"
- >
- Duration
- </div>
- </div>
- <!-- Spans -->
- {#each treeRows as treeRow, i (treeRow.span.id)}
- <SpanRow
- row={i}
- span={treeRow.span}
- {traceStart}
- {traceDuration}
- isOdd={i % 2 === 1}
- depth={treeRow.depth}
- hasChildren={treeRow.hasChildren}
- isCollapsed={collapsed.has(treeRow.span.id)}
- onToggle={() => toggleCollapse(treeRow.span.id)}
- {nameColumnWidth}
- {updateNameWidth}
- spanCellHandleMouseEnter={handleMouseEnter}
- spanCellHandleMouseMove={handleMouseMove}
- spanCellHandleMouseLeave={handleMouseLeave}
- />
- {/each}
-
- {#if isHovered}
- <div
- class="pointer-events-none absolute top-[28px] bottom-0 border-l border-border"
- style="left: {tooltipX + nameColumnWidth}px"
- ></div>
- <div class="absolute top-[1px] -translate-x-1/2" style="left: {tooltipX + nameColumnWidth}px">
+ <!-- Spans -->
+ {#each treeRows.slice(0, visibleRows) as treeRow, i (treeRow.key)}
+ {#key treeRow.span}
+ <SpanRow
+ row={colorIndex(treeRow.span.serviceName, i)}
+ span={treeRow.span}
+ {traceStart}
+ traceDuration={timelineDuration}
+ isOdd={i % 2 === 1}
+ depth={treeRow.depth}
+ hasChildren={treeRow.hasChildren}
+ missingParent={treeRow.missingParent}
+ {missingParentLabel}
+ descendants={treeRow.descendants}
+ errorBelow={treeRow.errorBelow}
+ {loadAttributes}
+ projectName={projectNames?.[treeRow.span.projectId]}
+ showService={showServiceLabel}
+ isSelected={!!selectedSpanId && treeRow.span.spanId === selectedSpanId}
+ traceHref={linkTraces ? spanTraceHref(treeRow.span) : undefined}
+ isCollapsed={collapsed.has(treeRow.key)}
+ onToggle={() => toggleCollapse(treeRow.key)}
+ {nameColumnWidth}
+ {updateNameWidth}
+ spanCellHandleMouseEnter={handleMouseEnter}
+ spanCellHandleMouseMove={handleMouseMove}
+ spanCellHandleMouseLeave={handleMouseLeave}
+ />
+ {/key}
+ {/each}
+
+ {#if treeRows.length > visibleRows || wholeTraceHref}
+ <div
+ class="flex flex-wrap items-center justify-between gap-2 border-t border-border bg-muted/30 px-3 py-1.5 text-xs text-muted-foreground"
+ >
+ {#if treeRows.length > visibleRows}
+ <button
+ type="button"
+ class="cursor-pointer underline underline-offset-2 hover:text-foreground"
+ onclick={() => (visibleRows += ROW_PAGE)}
+ >
+ Show {Math.min(ROW_PAGE, treeRows.length - visibleRows)} more of {treeRows.length -
+ visibleRows} hidden spans
+ </button>
+ {:else}
+ <span></span>
+ {/if}
+ {#if wholeTraceHref}
+ <a
+ class="underline underline-offset-2 hover:text-foreground"
+ {...{ href: resolveHref(wholeTraceHref) }}>Open the whole trace</a
+ >
+ {/if}
+ </div>
+ {/if}
+
+ {#if isHovered}
+ <div
+ class="pointer-events-none absolute top-[28px] bottom-0 border-l border-border"
+ style="left: {tooltipX + nameColumnWidth}px"
+ ></div>
<div
- class="rounded-md border bg-popover px-2 py-1 text-xs whitespace-nowrap text-popover-foreground shadow-md"
+ class="absolute top-[1px] -translate-x-1/2"
+ style="left: {tooltipX + nameColumnWidth}px"
>
- <div class="font-medium">
- {Math.round(durationMs * ((tooltipX + 1) / (timelineElement?.clientWidth || 1)))}ms
+ <div
+ class="rounded-md border bg-popover px-2 py-1 text-xs whitespace-nowrap text-popover-foreground shadow-md"
+ >
+ <div class="font-medium">
+ {Math.round(durationMs * ((tooltipX + 1) / (timelineElement?.clientWidth || 1)))}ms
+ </div>
</div>
</div>
- </div>
- {/if}
- </div>
-</ScrollArea>
+ {/if}
+ </div>
+ </ScrollArea>
+</div>
diff --git a/frontend/src/lib/components/spans/span-waterfall.test.ts b/frontend/src/lib/components/spans/span-waterfall.test.ts
new file mode 100644
index 00000000..0ec28677
--- /dev/null
+++ b/frontend/src/lib/components/spans/span-waterfall.test.ts
@@ -0,0 +1,282 @@
+import { afterEach, beforeEach, expect, it, vi } from 'vitest';
+import { cleanup, fireEvent, render } from '@testing-library/svelte';
+vi.mock('$lib/state/timezone.svelte', () => ({ getTimezone: () => 'UTC' }));
+import SpanWaterfall from './span-waterfall.svelte';
+import type { Span } from '$lib/types/spans';
+
+beforeEach(() => {
+ vi.stubGlobal(
+ 'ResizeObserver',
+ class {
+ observe() {}
+ unobserve() {}
+ disconnect() {}
+ }
+ );
+});
+afterEach(() => {
+ cleanup();
+ vi.unstubAllGlobals();
+});
+
+it('collapses descendants and keeps siblings visible, then restores the subtree', async () => {
+ const makeSpan = (id: string, parentSpanId?: string): Span => ({
+ spanId: id,
+ parentSpanId,
+ traceId: 'trace',
+ projectId: 'project',
+ name: `span ${id}`,
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000
+ });
+ const { getAllByRole, getByRole, queryByText, getByText } = render(SpanWaterfall, {
+ spans: [makeSpan('01'), makeSpan('02', '01'), makeSpan('03', '02'), makeSpan('04')],
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ expect(getAllByRole('button', { name: 'Collapse children' })).toHaveLength(2);
+ await fireEvent.click(getAllByRole('button', { name: 'Collapse children' })[0]);
+ expect(queryByText('span 02')).toBeNull();
+ expect(queryByText('span 03')).toBeNull();
+ expect(getByText('span 04')).toBeTruthy();
+ const expand = getByRole('button', { name: 'Expand children' });
+ expect(expand.getAttribute('aria-expanded')).toBe('false');
+ await fireEvent.click(expand);
+ expect(getByText('span 02')).toBeTruthy();
+ expect(getByText('span 03')).toBeTruthy();
+ expect(getAllByRole('button', { name: 'Collapse children' })).toHaveLength(2);
+});
+
+it('marks errors, missing parents, the selected span, and names services once there are several', () => {
+ const makeSpan = (id: string, overrides: Partial<Span>): Span => ({
+ spanId: id,
+ traceId: 'trace',
+ projectId: 'project',
+ name: `span ${id}`,
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000,
+ ...overrides
+ });
+ const { getAllByLabelText, getByText, container } = render(SpanWaterfall, {
+ spans: [
+ makeSpan('01', { serviceName: 'web', statusCode: 2 }),
+ makeSpan('02', { serviceName: 'api', parentSpanId: '99' })
+ ],
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000,
+ selectedSpanId: '02'
+ });
+ expect(getAllByLabelText('Error')).toHaveLength(1);
+ expect(getAllByLabelText('Parent span is unavailable')).toHaveLength(1);
+ expect(getByText('web')).toBeTruthy();
+ expect(getByText('api')).toBeTruthy();
+ const selected = container.querySelectorAll('[aria-current="true"]');
+ expect(selected).toHaveLength(1);
+ expect(selected[0].textContent).toContain('span 02');
+});
+
+it('leaves the service name out when the whole trace is one service', () => {
+ const { queryByText } = render(SpanWaterfall, {
+ spans: [
+ {
+ spanId: '01',
+ traceId: 'trace',
+ projectId: 'project',
+ name: 'only span',
+ serviceName: 'web',
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000
+ }
+ ],
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ expect(queryByText('web')).toBeNull();
+});
+
+it('pages a very large trace and links to the whole trace', async () => {
+ const spans: Span[] = Array.from({ length: 620 }, (_, i) => ({
+ spanId: i.toString(16).padStart(16, '0'),
+ traceId: 'abc',
+ projectId: 'project',
+ name: `span ${i}`,
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000 + i
+ }));
+ const { getByRole, queryByText, getByText } = render(SpanWaterfall, {
+ spans,
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ expect(queryByText('span 619')).toBeNull();
+ await fireEvent.click(getByRole('button', { name: /Show 120 more/ }));
+ expect(getByText('span 619')).toBeTruthy();
+ expect(getByRole('link', { name: 'Open the whole trace' }).getAttribute('href')).toContain(
+ '/spans/abc?at='
+ );
+}, 30_000);
+
+it('links the whole trace by the trace id and start time of the page it sits on', () => {
+ const { getByRole } = render(SpanWaterfall, {
+ spans: [
+ {
+ spanId: '02',
+ parentSpanId: '01',
+ traceId: 'abc',
+ projectId: 'project',
+ name: 'child',
+ startTime: '2026-09-17T00:00:00.250Z',
+ recordedAt: '',
+ duration: 1_000_000
+ }
+ ],
+ traceId: 'abc',
+ rootSpanId: '01',
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ expect(getByRole('link', { name: 'Open the whole trace' }).getAttribute('href')).toBe(
+ '/spans/abc?at=2026-09-17T00%3A00%3A00Z'
+ );
+});
+
+it('has no whole-trace link inside the trace view itself', () => {
+ const { queryByRole } = render(SpanWaterfall, {
+ spans: [
+ {
+ spanId: '01',
+ traceId: 'abc',
+ projectId: 'project',
+ name: 'only span',
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000
+ }
+ ],
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000,
+ linkTraces: false
+ });
+ expect(queryByRole('link', { name: 'Open the whole trace' })).toBeNull();
+});
+
+function wideTrace(): Span[] {
+ const make = (id: number, parent?: number, statusCode?: number): Span => ({
+ spanId: id.toString(16).padStart(16, '0'),
+ parentSpanId: parent === undefined ? undefined : parent.toString(16).padStart(16, '0'),
+ traceId: 'abc',
+ projectId: 'project',
+ name: `span ${id}`,
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000,
+ statusCode
+ });
+ return [
+ make(1),
+ make(2, 1),
+ make(3, 1),
+ ...Array.from({ length: 600 }, (_, i) => make(i + 100, 2, i === 7 ? 2 : 0))
+ ];
+}
+
+it('opens a large trace with its widest node folded into a pill that flags the error inside', async () => {
+ const { getByRole, getByText, queryByText, getAllByLabelText } = render(SpanWaterfall, {
+ spans: wideTrace(),
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ expect(getByText('span 3')).toBeTruthy();
+ expect(queryByText('span 100')).toBeNull();
+ const pill = getByRole('button', { name: 'Show 600 hidden spans' });
+ expect(pill.textContent).toContain('+600');
+ expect(getAllByLabelText('Error in the hidden spans')).toHaveLength(1);
+ await fireEvent.click(pill);
+ expect(getByText('span 100')).toBeTruthy();
+});
+
+it('collapses and expands everything from the header', async () => {
+ const { getByRole, getByText, queryByText } = render(SpanWaterfall, {
+ spans: wideTrace(),
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ await fireEvent.click(getByRole('button', { name: 'Collapse all' }));
+ expect(getByText('span 1')).toBeTruthy();
+ expect(queryByText('span 2')).toBeNull();
+ expect(getByRole('button', { name: 'Show 602 hidden spans' })).toBeTruthy();
+ await fireEvent.click(getByRole('button', { name: 'Expand all' }));
+ expect(getByText('span 2')).toBeTruthy();
+ expect(getByText('span 100')).toBeTruthy();
+});
+
+it('reveals a selected span inside a folded subtree beyond the first page', async () => {
+ const { findByText, container, rerender } = render(SpanWaterfall, {
+ spans: wideTrace(),
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000
+ });
+ await rerender({ selectedSpanId: (699).toString(16).padStart(16, '0') });
+ expect(await findByText('span 699')).toBeTruthy();
+ expect(container.querySelector('[aria-current="true"]')?.textContent).toContain('span 699');
+}, 30_000);
+
+it('reloads lazy attributes after a new version of the same span arrives', async () => {
+ const loadAttributes = vi
+ .fn()
+ .mockResolvedValueOnce({ attributes: { version: 'before' } })
+ .mockResolvedValueOnce({ attributes: { version: 'after' } });
+ const span: Span = {
+ spanId: '01',
+ traceId: 'abc',
+ projectId: 'project',
+ name: 'refreshed span',
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000
+ };
+ const { getByText, findByText, queryByText, rerender } = render(SpanWaterfall, {
+ spans: [span],
+ traceStartTime: span.startTime,
+ traceDuration: 5_000_000,
+ loadAttributes
+ });
+ await fireEvent.click(getByText('refreshed span'));
+ expect(await findByText('before')).toBeTruthy();
+ await rerender({ spans: [{ ...span, duration: 2_000_000 }] });
+ await fireEvent.click(getByText('refreshed span'));
+ expect(await findByText('after')).toBeTruthy();
+ expect(queryByText('before')).toBeNull();
+ expect(loadAttributes).toHaveBeenCalledTimes(2);
+});
+
+it("asks for a span's attributes once, when its popover opens", async () => {
+ const loadAttributes = vi.fn(async () => ({ attributes: { 'url.path': '/checkout' } }));
+ const { getByText, getAllByText, findByText } = render(SpanWaterfall, {
+ spans: [
+ {
+ spanId: '01',
+ traceId: 'abc',
+ projectId: 'project',
+ name: 'lazy span',
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000
+ }
+ ],
+ traceStartTime: '2026-09-17T00:00:00Z',
+ traceDuration: 5_000_000,
+ loadAttributes
+ });
+ expect(loadAttributes).not.toHaveBeenCalled();
+ await fireEvent.click(getByText('lazy span'));
+ expect(await findByText('/checkout')).toBeTruthy();
+ // The popover repeats the name, so the row label is the first match.
+ await fireEvent.click(getAllByText('lazy span')[0]);
+ await fireEvent.click(getAllByText('lazy span')[0]);
+ expect(loadAttributes).toHaveBeenCalledTimes(1);
+});
diff --git a/frontend/src/lib/components/trace-logs/trace-logs-panel.svelte b/frontend/src/lib/components/trace-logs/trace-logs-panel.svelte
index dd8df991..ff6563d7 100644
--- a/frontend/src/lib/components/trace-logs/trace-logs-panel.svelte
+++ b/frontend/src/lib/components/trace-logs/trace-logs-panel.svelte
@@ -1,19 +1,16 @@
<script lang="ts">
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { SvelteMap } from 'svelte/reactivity';
import { api } from '$lib/api';
import * as Card from '$lib/components/ui/card';
import * as Table from '$lib/components/ui/table';
- import * as Tabs from '$lib/components/ui/tabs';
+ import { Button } from '$lib/components/ui/button';
import { LoadingCircle } from '$lib/components/ui/loading-circle';
import { SeverityBadge } from '$lib/components/ui/severity-badge';
import ExpandedLogRow from './expanded-log-row.svelte';
import LogMessage from './log-message.svelte';
import { formatDateTime } from '$lib/utils/formatters';
import { getTimezone } from '$lib/state/timezone.svelte';
- import { underlineTabTriggerClass, underlineTabListClass } from '$lib/utils/tabs';
- import { cn } from '$lib/utils';
- import { spanIdUuidToHex } from '$lib/utils/span-id';
import type { Span } from '$lib/types/spans';
type LogRecord = {
@@ -36,47 +33,45 @@
let {
projectId,
traceId,
- distributedTraceId = null,
spans,
+ rootSpan,
+ wholeTrace = false,
traceRecordedAt
}: {
projectId: string;
traceId: string;
- distributedTraceId?: string | null;
spans: Span[];
+ rootSpan?: Pick<Span, 'spanId' | 'name'>;
+ wholeTrace?: boolean;
traceRecordedAt: string;
} = $props();
const timezone = $derived(getTimezone());
+ let generation = 0;
- let activeTab = $state<'this-trace' | 'all-distributed'>('this-trace');
-
- const tabTriggerClass = cn(underlineTabTriggerClass, 'mb-0 pb-2');
-
- // This-trace data
let logs = $state<LogRecord[]>([]);
let loading = $state(true);
let error = $state('');
let expandedId = $state<string | null>(null);
+ let logsPage = $state(1);
+ let totalLogs = $state(0);
- // Distributed data (lazy)
- let distributedLogs = $state<LogRecord[]>([]);
- let distributedLoading = $state(false);
- let distributedLoaded = $state(false);
- let distributedError = $state('');
-
- const childSpanNameByHex = $derived.by(() => {
+ const spanNameByHex = $derived.by(() => {
const m = new SvelteMap<string, string>();
+ if (rootSpan) m.set(`${projectId}:${traceId}:${rootSpan.spanId.toLowerCase()}`, rootSpan.name);
for (const s of spans) {
- const hex = spanIdUuidToHex(s.id);
- if (hex) m.set(hex, s.name);
+ m.set(`${s.projectId}:${s.traceId}:${s.spanId.toLowerCase()}`, s.name);
}
return m;
});
function resolveSpanName(log: LogRecord): string | null {
if (!log.spanId) return null;
- return childSpanNameByHex.get(log.spanId.toLowerCase()) ?? null;
+ return (
+ spanNameByHex.get(
+ `${log.projectId || projectId}:${log.traceId || traceId}:${log.spanId.toLowerCase()}`
+ ) ?? null
+ );
}
function timeWindow(): { fromDate: string; toDate: string } {
@@ -88,7 +83,8 @@
};
}
- async function loadTraceLogs() {
+ async function loadTraceLogs(page = 1) {
+ const requestGeneration = generation;
loading = true;
error = '';
try {
@@ -100,53 +96,24 @@
toDate,
orderBy: 'timestamp',
sortDirection: 'asc',
- pagination: { page: 1, pageSize: 100 },
- traceId
+ pagination: { page, pageSize: 100 },
+ traceId,
+ ...(wholeTrace ? { wholeTrace: true } : {})
},
{ projectId: projectId || undefined }
- )) as { data: LogRecord[] };
- logs = response.data || [];
+ )) as { data: LogRecord[]; pagination?: { total: number } };
+ if (requestGeneration === generation) {
+ logs = response.data || [];
+ logsPage = page;
+ totalLogs = response.pagination?.total ?? logs.length;
+ expandedId = null;
+ }
} catch (e: unknown) {
+ if (requestGeneration !== generation) return;
const err = e as { message?: string };
error = err.message || 'Failed to load logs';
} finally {
- loading = false;
- }
- }
-
- async function loadDistributedLogs() {
- if (!distributedTraceId) return;
- distributedLoading = true;
- distributedError = '';
- try {
- const { fromDate, toDate } = timeWindow();
- const response = (await api.post(
- '/logs',
- {
- fromDate,
- toDate,
- orderBy: 'timestamp',
- sortDirection: 'asc',
- pagination: { page: 1, pageSize: 100 },
- distributedTraceId
- },
- { projectId: projectId || undefined }
- )) as { data: LogRecord[] };
- distributedLogs = response.data || [];
- distributedLoaded = true;
- } catch (e: unknown) {
- const err = e as { message?: string };
- distributedError = err.message || 'Failed to load distributed logs';
- } finally {
- distributedLoading = false;
- }
- }
-
- function onTabChange(value: string) {
- if (value !== 'this-trace' && value !== 'all-distributed') return;
- activeTab = value;
- if (value === 'all-distributed' && !distributedLoaded && !distributedLoading) {
- loadDistributedLogs();
+ if (requestGeneration === generation) loading = false;
}
}
@@ -154,149 +121,116 @@
expandedId = expandedId === id ? null : id;
}
- onMount(() => {
- loadTraceLogs();
+ $effect(() => {
+ // SvelteKit reuses this component when navigating between trace URLs.
+ const identity = [projectId, traceId, traceRecordedAt, wholeTrace];
+ untrack(() => {
+ generation++;
+ logs = [];
+ error = '';
+ logsPage = 1;
+ totalLogs = 0;
+ expandedId = null;
+ if (identity[0] && identity[1]) loadTraceLogs();
+ else loading = false;
+ });
+ return () => {
+ generation++;
+ };
});
</script>
+{#snippet pagination(page: number, total: number, busy: boolean, onPage: (page: number) => void)}
+ {#if total > 100}
+ <div
+ class="flex items-center justify-between gap-3 border-t px-6 py-3 text-xs text-muted-foreground"
+ >
+ <span>Page {page} of {Math.ceil(total / 100)} · {total.toLocaleString()} logs</span>
+ <div class="flex gap-2">
+ <Button
+ variant="outline"
+ size="sm"
+ disabled={busy || page <= 1}
+ onclick={() => onPage(page - 1)}>Previous</Button
+ >
+ <Button
+ variant="outline"
+ size="sm"
+ disabled={busy || page * 100 >= total}
+ onclick={() => onPage(page + 1)}>Next</Button
+ >
+ </div>
+ </div>
+ {/if}
+{/snippet}
+
<Card.Root class="gap-0 overflow-hidden pb-0">
- <Card.Header class={distributedTraceId ? '' : 'pb-4'}>
+ <Card.Header class="pb-4">
<Card.Title>Logs</Card.Title>
</Card.Header>
<Card.Content class="p-0">
- <Tabs.Root value={activeTab} onValueChange={onTabChange}>
- {#if distributedTraceId}
- <Tabs.List class={cn(underlineTabListClass, 'pb-2 pl-6')}>
- <Tabs.Trigger value="this-trace" class={tabTriggerClass}>This Trace</Tabs.Trigger>
- <Tabs.Trigger value="all-distributed" class={tabTriggerClass}>
- All Distributed Traces
- </Tabs.Trigger>
- </Tabs.List>
- {/if}
-
- <Tabs.Content value="this-trace" class={distributedTraceId ? 'mt-0' : 'mt-0 border-t'}>
- {#if loading}
- <div class="flex items-center justify-center py-6">
- <LoadingCircle size="lg" />
- </div>
- {:else if error}
- <div class="py-6 text-center text-sm text-red-500">{error}</div>
- {:else if logs.length === 0}
- <div class="py-6 text-center text-sm text-muted-foreground">
- No logs for this trace in the surrounding time window
- </div>
- {:else}
- <Table.Root>
- <Table.Header>
- <Table.Row>
- <Table.Head class="h-8 w-[180px] py-1.5 pl-6">Timestamp</Table.Head>
- <Table.Head class="h-8 w-[80px] py-1.5">Level</Table.Head>
- <Table.Head class="h-8 py-1.5">Message</Table.Head>
- <Table.Head class="h-8 w-[220px] py-1.5 pr-6">Span</Table.Head>
- </Table.Row>
- </Table.Header>
- <Table.Body>
- {#each logs as log (log.id)}
- {@const spanName = resolveSpanName(log)}
- <Table.Row class="h-8 cursor-pointer" onclick={() => toggleExpanded(log.id)}>
- <Table.Cell class="py-1.5 pl-6 text-xs text-muted-foreground tabular-nums">
- {formatDateTime(log.timestamp, { timezone })}
- </Table.Cell>
- <Table.Cell class="py-1.5">
- <SeverityBadge
- severityText={log.severityText}
- severityNumber={log.severityNumber}
- />
- </Table.Cell>
- <Table.Cell class="max-w-[600px] truncate py-1.5 font-mono text-xs">
- <LogMessage body={log.body} attributes={log.logAttributes} />
- </Table.Cell>
- <Table.Cell class="py-1.5 pr-6">
- {#if spanName}
- <span
- class="inline-block max-w-[200px] truncate rounded bg-muted px-2 py-0.5 font-mono text-xs text-muted-foreground"
- title={spanName}
- >
- {spanName}
- </span>
- {:else}
- <span class="text-xs text-muted-foreground">—</span>
- {/if}
- </Table.Cell>
- </Table.Row>
- {#if expandedId === log.id}
- <ExpandedLogRow {log} colspan={4} />
+ {#if loading}
+ <div class="flex items-center justify-center py-6">
+ <LoadingCircle size="lg" />
+ </div>
+ {:else if error}
+ <div class="py-6 text-center text-sm text-red-500">{error}</div>
+ {:else if logs.length === 0}
+ <div class="py-6 text-center text-sm text-muted-foreground">
+ No logs for this trace in the surrounding time window
+ </div>
+ {:else}
+ <Table.Root>
+ <Table.Header>
+ <Table.Row>
+ <Table.Head class="h-8 w-[180px] py-1.5 pl-6">Timestamp</Table.Head>
+ <Table.Head class="h-8 w-[80px] py-1.5">Level</Table.Head>
+ <Table.Head class="h-8 py-1.5">Message</Table.Head>
+ {#if wholeTrace}<Table.Head class="h-8 py-1.5">Service</Table.Head>{/if}
+ <Table.Head class="h-8 w-[220px] py-1.5 pr-6">Span</Table.Head>
+ </Table.Row>
+ </Table.Header>
+ <Table.Body>
+ {#each logs as log (log.id)}
+ {@const spanName = resolveSpanName(log)}
+ <Table.Row class="h-8 cursor-pointer" onclick={() => toggleExpanded(log.id)}>
+ <Table.Cell class="py-1.5 pl-6 text-xs text-muted-foreground tabular-nums">
+ {formatDateTime(log.timestamp, { timezone })}
+ </Table.Cell>
+ <Table.Cell class="py-1.5">
+ <SeverityBadge
+ severityText={log.severityText}
+ severityNumber={log.severityNumber}
+ />
+ </Table.Cell>
+ <Table.Cell class="max-w-[600px] truncate py-1.5 font-mono text-xs">
+ <LogMessage body={log.body} attributes={log.logAttributes} />
+ </Table.Cell>
+ {#if wholeTrace}
+ <Table.Cell class="py-1.5 text-xs text-muted-foreground"
+ >{log.serviceName || '—'}</Table.Cell
+ >
+ {/if}
+ <Table.Cell class="py-1.5 pr-6">
+ {#if spanName}
+ <span
+ class="inline-block max-w-[200px] truncate rounded bg-muted px-2 py-0.5 font-mono text-xs text-muted-foreground"
+ title={spanName}
+ >
+ {spanName}
+ </span>
+ {:else}
+ <span class="text-xs text-muted-foreground">—</span>
{/if}
- {/each}
- </Table.Body>
- </Table.Root>
- {/if}
- </Tabs.Content>
-
- {#if distributedTraceId}
- <Tabs.Content value="all-distributed" class="mt-0">
- {#if distributedLoading}
- <div class="flex items-center justify-center py-6">
- <LoadingCircle size="lg" />
- </div>
- {:else if distributedError}
- <div class="py-6 text-center text-sm text-red-500">{distributedError}</div>
- {:else if !distributedLoaded}
- <div class="py-6 text-center text-sm text-muted-foreground">
- Loading distributed logs…
- </div>
- {:else if distributedLogs.length === 0}
- <div class="py-6 text-center text-sm text-muted-foreground">
- No logs found in this distributed trace
- </div>
- {:else}
- <Table.Root>
- <Table.Header>
- <Table.Row>
- <Table.Head class="h-8 w-[180px] py-1.5 pl-6">Timestamp</Table.Head>
- <Table.Head class="h-8 w-[80px] py-1.5">Level</Table.Head>
- <Table.Head class="h-8 py-1.5">Message</Table.Head>
- <Table.Head class="h-8 w-[160px] py-1.5">Service</Table.Head>
- <Table.Head class="h-8 w-[130px] py-1.5 pr-6">Trace</Table.Head>
- </Table.Row>
- </Table.Header>
- <Table.Body>
- {#each distributedLogs as log (log.id)}
- <Table.Row class="h-8 cursor-pointer" onclick={() => toggleExpanded(log.id)}>
- <Table.Cell class="py-1.5 pl-6 text-xs text-muted-foreground tabular-nums">
- {formatDateTime(log.timestamp, { timezone })}
- </Table.Cell>
- <Table.Cell class="py-1.5">
- <SeverityBadge
- severityText={log.severityText}
- severityNumber={log.severityNumber}
- />
- </Table.Cell>
- <Table.Cell class="max-w-[500px] truncate py-1.5 font-mono text-xs">
- <LogMessage body={log.body} attributes={log.logAttributes} />
- </Table.Cell>
- <Table.Cell class="py-1.5 text-xs text-muted-foreground">
- {log.serviceName || '—'}
- </Table.Cell>
- <Table.Cell class="py-1.5 pr-6 font-mono text-xs">
- {#if log.traceId}
- <span class="text-muted-foreground" title={log.traceId}>
- {log.traceId.slice(0, 8)}…
- </span>
- {:else}
- <span class="text-muted-foreground">—</span>
- {/if}
- </Table.Cell>
- </Table.Row>
- {#if expandedId === log.id}
- <ExpandedLogRow {log} colspan={5} />
- {/if}
- {/each}
- </Table.Body>
- </Table.Root>
- {/if}
- </Tabs.Content>
- {/if}
- </Tabs.Root>
+ </Table.Cell>
+ </Table.Row>
+ {#if expandedId === log.id}
+ <ExpandedLogRow {log} colspan={wholeTrace ? 5 : 4} />
+ {/if}
+ {/each}
+ </Table.Body>
+ </Table.Root>
+ {/if}
+ {@render pagination(logsPage, totalLogs, loading, loadTraceLogs)}
</Card.Content>
</Card.Root>
diff --git a/frontend/src/lib/components/trace-logs/trace-logs-panel.test.ts b/frontend/src/lib/components/trace-logs/trace-logs-panel.test.ts
new file mode 100644
index 00000000..8ee1d4a4
--- /dev/null
+++ b/frontend/src/lib/components/trace-logs/trace-logs-panel.test.ts
@@ -0,0 +1,139 @@
+import { afterEach, expect, it, vi } from 'vitest';
+import { cleanup, fireEvent, render, waitFor } from '@testing-library/svelte';
+import { api } from '$lib/api';
+import TraceLogsPanel from './trace-logs-panel.svelte';
+
+vi.mock('$lib/api', () => ({ api: { post: vi.fn() } }));
+vi.mock('$lib/state/timezone.svelte', () => ({ getTimezone: () => 'UTC' }));
+
+afterEach(() => {
+ cleanup();
+ vi.clearAllMocks();
+});
+
+it('names logs emitted by the occurrence root when the child graph excludes that span', async () => {
+ vi.mocked(api.post).mockResolvedValue({
+ data: [
+ {
+ id: 'root-log',
+ body: 'root message',
+ spanId: 'ABCDEF0123456789',
+ timestamp: '2026-09-01T12:00:00Z'
+ }
+ ]
+ });
+ const { findByText } = render(TraceLogsPanel, {
+ projectId: 'project-a',
+ traceId: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
+ spans: [],
+ rootSpan: { spanId: 'abcdef0123456789', name: 'GET /root' },
+ traceRecordedAt: '2026-09-01T12:00:00Z'
+ });
+ expect(await findByText('GET /root')).toBeTruthy();
+});
+
+it('loads one trace with the selected project and bounded time window', async () => {
+ vi.mocked(api.post).mockResolvedValue({ data: [] });
+ const traceId = '1234567890abcdef1234567890abcdef';
+ const { queryByRole } = render(TraceLogsPanel, {
+ projectId: 'selected-project',
+ traceId,
+ spans: [],
+ traceRecordedAt: '2026-09-01T12:00:00Z'
+ });
+ await waitFor(() => expect(api.post).toHaveBeenCalledTimes(1));
+ expect(api.post).toHaveBeenLastCalledWith(
+ '/logs',
+ {
+ traceId,
+ fromDate: '2026-09-01T11:00:00.000Z',
+ toDate: '2026-09-01T13:00:00.000Z',
+ orderBy: 'timestamp',
+ sortDirection: 'asc',
+ pagination: { page: 1, pageSize: 100 }
+ },
+ { projectId: 'selected-project' }
+ );
+ expect(queryByRole('tab', { name: 'Related Traces' })).toBeNull();
+});
+
+it('reloads when trace props change and ignores a late response from the old trace', async () => {
+ let finishOld!: (value: unknown) => void;
+ vi.mocked(api.post).mockImplementationOnce(
+ () =>
+ new Promise((resolve) => {
+ finishOld = resolve;
+ })
+ );
+ vi.mocked(api.post).mockResolvedValue({ data: [] });
+ const { rerender, queryByText, findByText } = render(TraceLogsPanel, {
+ projectId: 'project-a',
+ traceId: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
+ spans: [],
+ traceRecordedAt: '2026-09-01T12:00:00Z'
+ });
+ await waitFor(() => expect(api.post).toHaveBeenCalledTimes(1));
+ await rerender({ projectId: 'project-b', traceId: 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' });
+ await waitFor(() => expect(api.post).toHaveBeenCalledTimes(2));
+ finishOld({
+ data: [{ id: 'old-log', body: 'stale trace log', timestamp: '2026-09-01T12:00:00Z' }]
+ });
+ expect(await findByText('No logs for this trace in the surrounding time window')).toBeTruthy();
+ expect(queryByText('stale trace log')).toBeNull();
+ expect(api.post).toHaveBeenLastCalledWith(
+ '/logs',
+ expect.objectContaining({ traceId: 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' }),
+ { projectId: 'project-b' }
+ );
+});
+
+it('pages beyond the first 100 logs instead of silently hiding them', async () => {
+ vi.mocked(api.post).mockResolvedValue({ data: [], pagination: { total: 101 } });
+ const { getByRole, findByText } = render(TraceLogsPanel, {
+ projectId: 'project-a',
+ traceId: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
+ spans: [],
+ traceRecordedAt: '2026-09-01T12:00:00Z'
+ });
+ expect(await findByText('Page 1 of 2 · 101 logs')).toBeTruthy();
+ await fireEvent.click(getByRole('button', { name: 'Next' }));
+ expect(await findByText('Page 2 of 2 · 101 logs')).toBeTruthy();
+ expect(api.post).toHaveBeenLastCalledWith(
+ '/logs',
+ expect.objectContaining({ pagination: { page: 2, pageSize: 100 } }),
+ { projectId: 'project-a' }
+ );
+ expect(getByRole('button', { name: 'Next' }).hasAttribute('disabled')).toBe(true);
+});
+
+it('requests one paginated whole-trace log stream and displays each service', async () => {
+ vi.mocked(api.post).mockResolvedValue({
+ data: [
+ { id: 'a', body: 'gateway log', serviceName: 'gateway', timestamp: '2026-09-01T12:00:00Z' },
+ { id: 'b', body: 'payments log', serviceName: 'payments', timestamp: '2026-09-01T12:00:01Z' }
+ ],
+ pagination: { total: 101 }
+ });
+ const { findByText, getByRole, rerender } = render(TraceLogsPanel, {
+ projectId: 'project-a',
+ traceId: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
+ wholeTrace: true,
+ spans: [],
+ traceRecordedAt: '2026-09-01T12:00:00Z'
+ });
+ expect(await findByText('gateway')).toBeTruthy();
+ expect(await findByText('payments')).toBeTruthy();
+ await fireEvent.click(getByRole('button', { name: 'Next' }));
+ await waitFor(() => expect(api.post).toHaveBeenCalledTimes(2));
+ expect(api.post).toHaveBeenLastCalledWith(
+ '/logs',
+ expect.objectContaining({
+ wholeTrace: true,
+ pagination: { page: 2, pageSize: 100 }
+ }),
+ { projectId: 'project-a' }
+ );
+ await rerender({ wholeTrace: false });
+ await waitFor(() => expect(api.post).toHaveBeenCalledTimes(3));
+ expect(vi.mocked(api.post).mock.calls[2][1]).not.toHaveProperty('wholeTrace');
+});
diff --git a/frontend/src/lib/types/distributed-trace.ts b/frontend/src/lib/types/distributed-trace.ts
index dad8a70e..e8a12785 100644
--- a/frontend/src/lib/types/distributed-trace.ts
+++ b/frontend/src/lib/types/distributed-trace.ts
@@ -1,9 +1,14 @@
export type DistributedTraceNode = {
+ spanGraphStatus?: SpanGraphStatus;
projectId: string;
projectName: string;
traceType: 'endpoint' | 'task' | 'ai_trace' | 'exception';
+ traceId: string;
+ spanId: string;
+ parentEntitySpanId?: string;
endpoint?: {
id: string;
+ parentSpanId?: string;
endpoint: string;
duration: number;
statusCode: number;
@@ -11,12 +16,14 @@ export type DistributedTraceNode = {
};
task?: {
id: string;
+ parentSpanId?: string;
taskName: string;
duration: number;
recordedAt: string;
};
aiTrace?: {
id: string;
+ parentSpanId?: string;
traceName: string;
model: string;
provider: string;
@@ -34,7 +41,7 @@ export type DistributedTraceNode = {
};
export type DistributedTraceResponse = {
- distributedTraceId: string;
+ traceId: string;
nodes: DistributedTraceNode[];
};
-import type { Span } from '$lib/types/spans';
+import type { Span, SpanGraphStatus } from '$lib/types/spans';
diff --git a/frontend/src/lib/types/exceptions.ts b/frontend/src/lib/types/exceptions.ts
index 0b7e72f9..1b3e551f 100644
--- a/frontend/src/lib/types/exceptions.ts
+++ b/frontend/src/lib/types/exceptions.ts
@@ -8,8 +8,10 @@ export type ExceptionGroup = {
export type ExceptionOccurrence = {
id: string;
- traceId: string | null;
- traceType: 'endpoint' | 'task';
+ // The trace and span the exception was recorded on, as lowercase hex. Empty when it arrived without a trace.
+ traceId: string;
+ spanId: string;
+ traceType: 'endpoint' | 'task' | 'ai_trace' | '';
exceptionHash: string;
stackTrace: string;
recordedAt: string;
@@ -18,7 +20,17 @@ export type ExceptionOccurrence = {
serverName: string;
isMessage: boolean;
endpoint: string;
- distributedTraceId?: string;
+};
+
+// The endpoint, task or AI trace an exception happened in, as the backend resolves it.
+export type RelatedEntity = {
+ traceType: 'endpoint' | 'task' | 'ai_trace';
+ id: string;
+ name: string;
+ statusCode: number;
+ duration: number;
+ recordedAt: string;
+ traceId: string;
};
export type LinkedTrace = {
@@ -27,10 +39,23 @@ export type LinkedTrace = {
duration: number;
statusCode: number;
recordedAt: string;
- traceType: 'endpoint' | 'task';
- distributedTraceId?: string;
+ traceType: 'endpoint' | 'task' | 'ai_trace';
+ traceId: string;
};
+export function linkedTraceFrom(related: RelatedEntity | null | undefined): LinkedTrace | null {
+ if (!related) return null;
+ return {
+ id: related.id,
+ endpoint: related.name,
+ duration: related.duration,
+ statusCode: related.statusCode,
+ recordedAt: related.recordedAt,
+ traceType: related.traceType,
+ traceId: related.traceId
+ };
+}
+
// Session recording shape returned by the backend's exception detail endpoints.
// Mirrors the wire format produced by the Flutter and JS SDKs.
diff --git a/frontend/src/lib/types/spans.ts b/frontend/src/lib/types/spans.ts
index a4c86b96..2c2ad02b 100644
--- a/frontend/src/lib/types/spans.ts
+++ b/frontend/src/lib/types/spans.ts
@@ -1,16 +1,43 @@
+// traceId, spanId and parentSpanId are the ids the span arrived with, as lowercase hex: 32 characters for a trace, 16
+// for an OTel span, 32 for a span of the native protocol.
export type Span = {
- id: string;
- traceId: string;
projectId: string;
+ traceId: string;
+ spanId: string;
+ parentSpanId?: string;
name: string;
startTime: string; // ISO datetime
duration: number; // nanoseconds
recordedAt: string;
- parentSpanId?: string;
+ spanKind?: number;
+ statusCode?: number;
+ serviceName?: string;
+ scopeName?: string;
attributes?: Record<string, string> | null;
+ attributesOmitted?: boolean;
+ dbStatement?: string;
+};
+
+export type TraceIdentity = {
+ traceId: string;
+ spanId: string;
+ parentSpanId?: string;
+};
+
+export type SpanAttributes = {
+ attributes: Record<string, string>;
+ attributesOmitted?: boolean;
+};
+
+export type SpanAttributeLoader = (span: Span) => Promise<SpanAttributes>;
+
+export type SpanGraphStatus = {
+ state: 'complete' | 'partial' | 'unavailable';
+ reasons?: string[];
+ omittedAttributes?: number;
};
-export type TraceDetail = {
+export type TraceDetail = TraceIdentity & {
id: string;
projectId: string;
endpoint: string;
@@ -22,8 +49,6 @@ export type TraceDetail = {
attributes: Record<string, string> | null;
appVersion: string;
serverName: string;
- distributedTraceId?: string;
- spanId?: string;
};
export type ExceptionInfo = {
@@ -41,6 +66,7 @@ export type MessageInfo = {
};
export type TraceDetailResponse = {
+ spanGraphStatus?: SpanGraphStatus;
endpoint: TraceDetail;
spans: Span[];
hasSpans: boolean;
diff --git a/frontend/src/lib/utils/ai-conversation.test.ts b/frontend/src/lib/utils/ai-conversation.test.ts
new file mode 100644
index 00000000..e5251e2c
--- /dev/null
+++ b/frontend/src/lib/utils/ai-conversation.test.ts
@@ -0,0 +1,63 @@
+import { expect, it } from 'vitest';
+import {
+ buildConversationTimeline,
+ extractMessages,
+ type ConversationTurn
+} from './ai-conversation';
+
+it('shows plain OTLP prompts and completions, including output-only captures', () => {
+ expect(extractMessages('请求 café 🚀', 'Review answer')).toEqual([
+ { role: 'user', content: '请求 café 🚀' },
+ { role: 'assistant', content: 'Review answer' }
+ ]);
+ expect(extractMessages('', 'only output')).toEqual([
+ { role: 'assistant', content: 'only output' }
+ ]);
+ expect(extractMessages('', '')).toBeNull();
+});
+
+function turn(input: unknown, output: unknown): ConversationTurn {
+ return {
+ id: 'turn',
+ traceName: 'chat',
+ recordedAt: '',
+ duration: 1,
+ model: 'model',
+ totalTokens: 1,
+ totalCost: 0,
+ input: JSON.stringify(input),
+ output: JSON.stringify(output)
+ };
+}
+
+it('merges repeated complete histories without repeating earlier messages', () => {
+ const first = { role: 'user', content: 'first question' };
+ const answer = { role: 'assistant', content: 'first answer' };
+ const next = { role: 'user', content: 'next question' };
+ const timeline = buildConversationTimeline([
+ turn([first], { choices: [{ message: answer }] }),
+ turn([first, answer, next], {
+ choices: [{ message: { role: 'assistant', content: 'next answer' } }]
+ })
+ ]);
+ expect(
+ timeline?.filter((entry) => entry.kind === 'message').map((entry) => entry.message.content)
+ ).toEqual(['first question', 'first answer', 'next question', 'next answer']);
+});
+
+it('falls back to separate turns when history text changes even if its roles match', () => {
+ const first = { role: 'user', content: 'original question' };
+ const answer = { role: 'assistant', content: 'answer' };
+ expect(
+ buildConversationTimeline([
+ turn([first], { choices: [{ message: answer }] }),
+ turn([{ ...first, content: 'replaced question' }, answer], { choices: [{ message: answer }] })
+ ])
+ ).toBeNull();
+});
+
+it('retains unrecognized response shapes in the per-turn fallback instead of claiming no payload', () => {
+ const value = turn([{ role: 'user', content: 'question' }], { vendor_response: 'answer' });
+ expect(buildConversationTimeline([value])).toBeNull();
+ expect(extractMessages(value.input, value.output)?.[1].content).toContain('vendor_response');
+});
diff --git a/frontend/src/lib/utils/ai-conversation.ts b/frontend/src/lib/utils/ai-conversation.ts
index 51b02927..b0fd2389 100644
--- a/frontend/src/lib/utils/ai-conversation.ts
+++ b/frontend/src/lib/utils/ai-conversation.ts
@@ -131,7 +131,11 @@ function normalizeInputMessage(value: unknown): ChatMessage | null {
function extractOutputMessage(output: string): ChatMessage | null {
const outputParsed = tryParseJson(output);
- if (!isRecord(outputParsed)) return null;
+ if (!isRecord(outputParsed)) {
+ return output.trim()
+ ? { role: 'assistant', content: typeof outputParsed === 'string' ? outputParsed : output }
+ : null;
+ }
// OpenAI-style: {choices: [{message: {role, content, tool_calls}}]}
const choices = outputParsed.choices;
@@ -193,10 +197,13 @@ export function parseInputMessages(input: string): ChatMessage[] | null {
}
export function extractMessages(input: string, output: string): ChatMessage[] | null {
- const messages = parseInputMessages(input);
- if (!messages) return null;
+ const messages =
+ parseInputMessages(input) ??
+ (input.trim() ? [{ role: 'user', content: formatConversationContent(input) }] : []);
- const outputMessage = extractOutputMessage(output);
+ const outputMessage =
+ extractOutputMessage(output) ??
+ (output.trim() ? { role: 'assistant', content: formatConversationContent(output) } : null);
if (outputMessage) messages.push(outputMessage);
return messages.length > 0 ? messages : null;
@@ -219,17 +226,17 @@ export function pairToolResults(messages: ChatMessage[]): Map<string, ChatMessag
return paired;
}
-function roleSequenceMatches(previous: ChatMessage[], current: ChatMessage[]): boolean {
+function historyPrefixMatches(previous: ChatMessage[], current: ChatMessage[]): boolean {
if (current.length < previous.length) return false;
for (let i = 0; i < previous.length; i++) {
- if (previous[i].role !== current[i].role) return false;
+ if (JSON.stringify(previous[i]) !== JSON.stringify(current[i])) return false;
}
return true;
}
// buildConversationTimeline merges per-call payloads into one chat timeline.
// Most SDKs resend the full history each turn, so each turn's input is
-// deduplicated against the previous turn's by role-sequence prefix matching;
+// deduplicated against the previous turn's by complete-message prefix matching;
// when that fails (compacted histories, unparsable payloads) the caller
// should fall back to per-turn rendering (signaled by returning null).
export function buildConversationTimeline(turns: ConversationTurn[]): TimelineEntry[] | null {
@@ -242,7 +249,7 @@ export function buildConversationTimeline(turns: ConversationTurn[]): TimelineEn
let fresh = inputMessages;
if (previousInput && previousInput.length > 0) {
- if (!roleSequenceMatches(previousInput, inputMessages)) return null;
+ if (!historyPrefixMatches(previousInput, inputMessages)) return null;
fresh = inputMessages.slice(previousInput.length);
}
for (const message of fresh) {
@@ -250,6 +257,7 @@ export function buildConversationTimeline(turns: ConversationTurn[]): TimelineEn
}
const outputMessage = extractOutputMessage(turn.output);
+ if (turn.output.trim() && !outputMessage) return null;
if (outputMessage) {
entries.push({ kind: 'message', message: outputMessage });
}
diff --git a/frontend/src/lib/utils/distributed-trace-tree.test.ts b/frontend/src/lib/utils/distributed-trace-tree.test.ts
new file mode 100644
index 00000000..17cca11c
--- /dev/null
+++ b/frontend/src/lib/utils/distributed-trace-tree.test.ts
@@ -0,0 +1,88 @@
+import { expect, it } from 'vitest';
+import { distributedTraceTree } from './distributed-trace-tree';
+import type { DistributedTraceNode } from '$lib/types/distributed-trace';
+
+function node(id: string, parent?: string, trace = 'trace'): DistributedTraceNode {
+ return {
+ projectId: `project-${id}`,
+ projectName: id,
+ traceType: 'endpoint',
+ traceId: trace,
+ spanId: id,
+ spans: [],
+ endpoint: {
+ id,
+ parentSpanId: parent,
+ endpoint: id,
+ recordedAt: '2026-09-17T00:00:00Z',
+ duration: 100,
+ statusCode: 200
+ }
+ };
+}
+
+it('connects authorized services through intermediate spans and collapses only descendants', () => {
+ const parent = node('01');
+ const child = node('03', '02');
+ const independent = node('04');
+ parent.spans = [
+ {
+ spanId: '02',
+ parentSpanId: '01',
+ traceId: 'trace',
+ projectId: parent.projectId,
+ name: 'client',
+ startTime: '',
+ recordedAt: '',
+ duration: 10
+ }
+ ];
+ const rows = distributedTraceTree([child, independent, parent], new Set());
+ expect(rows.map((r) => [r.node.endpoint?.id, r.depth, r.hasChildren])).toEqual([
+ ['01', 0, true],
+ ['03', 1, false],
+ ['04', 0, false]
+ ]);
+ expect(
+ distributedTraceTree([child, independent, parent], new Set([rows[0].key])).map(
+ (r) => r.node.endpoint?.id
+ )
+ ).toEqual(['01', '04']);
+});
+
+it('does not manufacture relationships from ordering or custom grouping', () => {
+ const rows = distributedTraceTree([node('01'), node('02', '01', 'other-trace')], new Set());
+ expect(rows.every((r) => r.depth === 0 && !r.hasChildren)).toBe(true);
+});
+
+it('keeps migrated occurrences with the same legacy ID in different traces', () => {
+ const rows = distributedTraceTree([node('01'), node('01', undefined, 'other-trace')], new Set());
+ expect(rows).toHaveLength(2);
+ expect(new Set(rows.map((row) => row.key)).size).toBe(2);
+ expect(rows.map((row) => row.node.traceId).sort()).toEqual(['other-trace', 'trace']);
+});
+
+it('keeps duplicate source identities in different projects visible without guessing a parent', () => {
+ const first = node('01');
+ const second = { ...node('01'), projectId: 'another-project' };
+ const child = node('02', '01');
+ const rows = distributedTraceTree([first, second, child], new Set());
+ expect(rows).toHaveLength(3);
+ expect(new Set(rows.map((r) => r.key)).size).toBe(3);
+ expect(rows.every((r) => r.depth === 0)).toBe(true);
+});
+
+it('nests across a hop that belongs to no entity when the server names the ancestor', () => {
+ const gateway = node('0000000000000001');
+ // The parent is a gRPC client span in a project with no promoted entity, so no loaded span leads back to gateway.
+ const warehouse = node('0000000000000005', '0000000000000004');
+ expect(distributedTraceTree([gateway, warehouse], new Set()).map((row) => row.depth)).toEqual([
+ 0, 0
+ ]);
+ const linked = { ...warehouse, parentEntitySpanId: '0000000000000001' };
+ const rows = distributedTraceTree([gateway, linked], new Set());
+ expect(rows.map((row) => [row.node.projectName, row.depth])).toEqual([
+ ['0000000000000001', 0],
+ ['0000000000000005', 1]
+ ]);
+});
diff --git a/frontend/src/lib/utils/distributed-trace-tree.ts b/frontend/src/lib/utils/distributed-trace-tree.ts
new file mode 100644
index 00000000..80289fef
--- /dev/null
+++ b/frontend/src/lib/utils/distributed-trace-tree.ts
@@ -0,0 +1,78 @@
+import type { DistributedTraceNode } from '$lib/types/distributed-trace';
+import type { Span } from '$lib/types/spans';
+import { flattenSpanTree, spanTreeKey } from './span-tree';
+
+const wireKey = (trace: string, span: string) => `${trace}:${span}`;
+
+export function distributedTraceTree(nodes: DistributedTraceNode[], collapsed: Set<string>) {
+ const parents = new Map<string, Set<string>>();
+ const addParent = (child: string, parent: string) => {
+ const candidates = parents.get(child) ?? new Set<string>();
+ candidates.add(parent);
+ parents.set(child, candidates);
+ };
+ const parentOf = (key: string) => {
+ const candidates = parents.get(key);
+ return candidates?.size === 1 ? candidates.values().next().value : undefined;
+ };
+ // The server walks the whole trace across projects and names the nearest promoted entity above each one. That beats
+ // the edges loaded here, which stop at any hop that belongs to no entity's subtree.
+ const ancestors = new Map<string, string>();
+ const roots = new Map<string, { projectId: string; span: Span }[]>();
+ const nodesByKey = new Map<string, DistributedTraceNode>();
+ const spans: Span[] = [];
+
+ for (const node of nodes) {
+ for (const span of node.spans) {
+ if (span.parentSpanId) {
+ addParent(wireKey(span.traceId, span.spanId), wireKey(span.traceId, span.parentSpanId));
+ }
+ }
+ const entity = node.endpoint ?? node.task ?? node.aiTrace;
+ const span: Span = {
+ // Virtual tree identities keep duplicated exports in different projects distinct.
+ projectId: 'distributed',
+ traceId: 'distributed',
+ spanId: `${node.projectId}:${node.traceType}:${node.traceId}:${node.spanId}:${entity?.id ?? `${node.exception?.exceptionHash}:${node.exception?.recordedAt}`}`,
+ name: '',
+ startTime: entity?.recordedAt ?? node.exception?.recordedAt ?? '',
+ recordedAt: entity?.recordedAt ?? '',
+ duration: entity?.duration ?? 0
+ };
+ if (entity && node.traceId && node.spanId) {
+ const key = wireKey(node.traceId, node.spanId);
+ const candidates = roots.get(key) ?? [];
+ candidates.push({ projectId: node.projectId, span });
+ roots.set(key, candidates);
+ if (entity.parentSpanId) addParent(key, wireKey(node.traceId, entity.parentSpanId));
+ if (node.parentEntitySpanId) {
+ ancestors.set(key, wireKey(node.traceId, node.parentEntitySpanId));
+ }
+ }
+ spans.push(span);
+ nodesByKey.set(spanTreeKey(span), node);
+ }
+
+ for (const [key, entries] of roots) {
+ for (const { projectId, span } of entries) {
+ const visited = new Set([key]);
+ let parent = ancestors.get(key) ?? parentOf(key);
+ while (parent && !visited.has(parent)) {
+ visited.add(parent);
+ const candidates = roots.get(parent) ?? [];
+ const local = candidates.filter((candidate) => candidate.projectId === projectId);
+ const eligible = local.length ? local : candidates;
+ if (eligible.length === 1) {
+ span.parentSpanId = eligible[0].span.spanId;
+ break;
+ }
+ if (eligible.length > 1) break;
+ parent = parentOf(parent);
+ }
+ }
+ }
+ return flattenSpanTree(spans, undefined, collapsed).map((row) => ({
+ ...row,
+ node: nodesByKey.get(row.key)!
+ }));
+}
diff --git a/frontend/src/lib/utils/formatters.ts b/frontend/src/lib/utils/formatters.ts
index 042c3ce5..8990926a 100644
--- a/frontend/src/lib/utils/formatters.ts
+++ b/frontend/src/lib/utils/formatters.ts
@@ -68,7 +68,14 @@ export function formatRelativeTimeAgo(dateStr: string, timezone?: string): strin
return relative === 'just now' ? relative : `${relative} ago`;
}
-export type DateTimeFormat = 'full' | 'short' | 'date' | 'time' | 'datetime' | 'iso';
+export type DateTimeFormat =
+ | 'full'
+ | 'short'
+ | 'date'
+ | 'time'
+ | 'datetime'
+ | 'datetime-seconds'
+ | 'iso';
export function formatDateTime(
dateInput: string | Date | number,
@@ -102,6 +109,8 @@ export function formatDateTime(
return dt.toLocaleString(DateTime.TIME_SIMPLE);
case 'iso':
return dt.toISO() ?? '';
+ case 'datetime-seconds':
+ return dt.toLocaleString(DateTime.DATETIME_MED_WITH_SECONDS);
case 'datetime':
default:
return dt.toLocaleString(DateTime.DATETIME_MED);
diff --git a/frontend/src/lib/utils/span-display.test.ts b/frontend/src/lib/utils/span-display.test.ts
new file mode 100644
index 00000000..f778eb0d
--- /dev/null
+++ b/frontend/src/lib/utils/span-display.test.ts
@@ -0,0 +1,48 @@
+import { describe, expect, it } from 'vitest';
+import { spanDisplayName } from './span-display';
+import type { Span } from '$lib/types/spans';
+
+function span(overrides: Partial<Span>): Span {
+ return {
+ traceId: 'trace',
+ spanId: '01',
+ projectId: 'project',
+ name: 'SELECT shop.orders',
+ startTime: '2026-09-17T00:00:00Z',
+ duration: 100,
+ recordedAt: '',
+ ...overrides
+ };
+}
+
+describe('spanDisplayName', () => {
+ it('previews the statement of a database span and prefers db.query.text', () => {
+ const attributes = { 'db.query.text': 'SELECT 1', 'db.statement': 'SELECT 2' };
+ expect(spanDisplayName(span({ attributes }))).toBe('SELECT 1');
+ expect(spanDisplayName(span({ attributes: { 'db.statement': 'SELECT 2' } }))).toBe('SELECT 2');
+ });
+
+ it('keeps the span name when there is no statement', () => {
+ expect(spanDisplayName(span({}))).toBe('SELECT shop.orders');
+ expect(spanDisplayName(span({ attributes: null }))).toBe('SELECT shop.orders');
+ expect(spanDisplayName(span({ attributes: { 'db.query.text': ' \n ' } }))).toBe(
+ 'SELECT shop.orders'
+ );
+ });
+
+ it('collapses whitespace and bounds a very long statement', () => {
+ const statement = `SELECT *\n\tFROM orders\n WHERE id IN (${'1, '.repeat(500)}2)`;
+ const preview = spanDisplayName(span({ attributes: { 'db.query.text': statement } }));
+ expect(preview.startsWith('SELECT * FROM orders WHERE id IN (1, 1,')).toBe(true);
+ expect(preview.length).toBe(201);
+ expect(preview.endsWith('…')).toBe(true);
+ });
+});
+
+describe('spanDisplayName without attributes', () => {
+ it('uses the statement preview a whole trace read sends along', () => {
+ expect(spanDisplayName(span({ dbStatement: 'SELECT *\n FROM orders' }))).toBe(
+ 'SELECT * FROM orders'
+ );
+ });
+});
diff --git a/frontend/src/lib/utils/span-display.ts b/frontend/src/lib/utils/span-display.ts
new file mode 100644
index 00000000..ca4e19e6
--- /dev/null
+++ b/frontend/src/lib/utils/span-display.ts
@@ -0,0 +1,14 @@
+import type { Span } from '$lib/types/spans';
+
+const STATEMENT_PREVIEW_LIMIT = 200;
+
+export function spanDisplayName(span: Span): string {
+ const statement =
+ span.attributes?.['db.query.text'] || span.attributes?.['db.statement'] || span.dbStatement;
+ if (!statement) return span.name;
+ const preview = statement.replace(/\s+/g, ' ').trim();
+ if (!preview) return span.name;
+ return preview.length > STATEMENT_PREVIEW_LIMIT
+ ? `${preview.slice(0, STATEMENT_PREVIEW_LIMIT)}…`
+ : preview;
+}
diff --git a/frontend/src/lib/utils/span-explorer.test.ts b/frontend/src/lib/utils/span-explorer.test.ts
new file mode 100644
index 00000000..aabaa1cc
--- /dev/null
+++ b/frontend/src/lib/utils/span-explorer.test.ts
@@ -0,0 +1,121 @@
+import { describe, expect, it } from 'vitest';
+import {
+ parseSpanAttributeFilter,
+ readSpanSearchFilters,
+ spanSearchBody,
+ spanSearchUrlParams,
+ spanTraceHref,
+ summarizeSpanTrace
+} from './span-explorer';
+import type { Span } from '$lib/types/spans';
+
+function urlOf(params: Record<string, string | string[] | undefined>): string {
+ const search = new URLSearchParams();
+ for (const [key, value] of Object.entries(params)) {
+ if (Array.isArray(value)) value.forEach((item) => search.append(key, item));
+ else if (value) search.set(key, value);
+ }
+ return `?${search}`;
+}
+
+describe('span search filters', () => {
+ it('survive a trip through the URL', () => {
+ const filters = {
+ name: 'GET /checkout',
+ service: 'cart',
+ kind: '2',
+ status: '2',
+ minMs: '12.5',
+ maxMs: '900',
+ traceId: '0af7651916cd43dd8448eb211c80319c',
+ attributes: [
+ { key: 'http.request.method', value: 'GET' },
+ { key: 'url.query', value: 'a=b&c=d' }
+ ]
+ };
+ expect(readSpanSearchFilters(urlOf(spanSearchUrlParams(filters)))).toEqual(filters);
+ });
+
+ it('drop attribute entries that have no key or no value', () => {
+ expect(readSpanSearchFilters('?attr=novalue&attr==x&attr=k=').attributes).toEqual([]);
+ });
+
+ it('send only the filters that are set', () => {
+ const body = spanSearchBody(readSpanSearchFilters('?status=0&maxMs=-3'));
+ expect(body.status).toBe(0);
+ expect(body.kind).toBeUndefined();
+ expect(body.minDurationMs).toBeUndefined();
+ expect(body.maxDurationMs).toBe(-3);
+ });
+
+ it('rejects malformed durations instead of silently broadening the search', () => {
+ for (const value of ['abc', 'Infinity', '1e999']) {
+ expect(() => spanSearchBody(readSpanSearchFilters(`?minMs=${value}`))).toThrow(
+ /finite duration/
+ );
+ }
+ });
+});
+
+describe('attribute filter dialog', () => {
+ it('trims and accepts a filter', () => {
+ expect(parseSpanAttributeFilter(' url.path ', ' /checkout ', [])).toEqual({
+ key: 'url.path',
+ value: '/checkout'
+ });
+ });
+
+ it('explains what is wrong', () => {
+ const existing = [{ key: 'url.path', value: '/checkout' }];
+ expect(parseSpanAttributeFilter('', 'x', [])).toMatch(/key/);
+ expect(parseSpanAttributeFilter('a"b', 'x', [])).toMatch(/quotes/);
+ expect(parseSpanAttributeFilter('url.path', '', [])).toMatch(/value/);
+ expect(parseSpanAttributeFilter('url.path', '/checkout', existing)).toMatch(/already/);
+ const full = Array.from({ length: 10 }, (_, i) => ({ key: `k${i}`, value: 'v' }));
+ expect(parseSpanAttributeFilter('k', 'v', full)).toMatch(/at most 10/);
+ });
+});
+
+describe('trace view', () => {
+ const span = (overrides: Partial<Span>): Span => ({
+ traceId: 'abc',
+ spanId: '01',
+ projectId: 'project',
+ name: 'span',
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000_000,
+ ...overrides
+ });
+
+ it('links a span to its trace and keeps the time the read is bound to', () => {
+ const href = spanTraceHref(span({ recordedAt: '2026-09-17T00:00:00.5Z' }), { focusSpan: true });
+ expect(href).toBe('/spans/abc?at=2026-09-17T00%3A00%3A00.5Z&span=01');
+ expect(spanTraceHref(span({ traceId: '' }))).toBeUndefined();
+ });
+
+ it.each(['1970-01-01T00:00:00Z', '2554-07-21T23:34:33.709551615Z'])(
+ 'opens spans with source time %s using their storage time',
+ (startTime) => {
+ const href = spanTraceHref(span({ startTime, recordedAt: '2026-09-20T12:34:56Z' }), {
+ focusSpan: true
+ });
+ expect(href).toBe('/spans/abc?at=2026-09-20T12%3A34%3A56Z&span=01');
+ }
+ );
+
+ it('measures the trace from its first start to its last end', () => {
+ const summary = summarizeSpanTrace([
+ span({ startTime: '2026-09-17T00:00:00.002Z', duration: 5_000_000, serviceName: 'api' }),
+ span({ startTime: '2026-09-17T00:00:00.000Z', duration: 3_000_000, serviceName: 'web' }),
+ span({ startTime: '2026-09-17T00:00:00.001Z', statusCode: 2, serviceName: 'api' })
+ ]);
+ expect(summary).toEqual({
+ startTime: '2026-09-17T00:00:00.000Z',
+ duration: 7_000_000,
+ services: 2,
+ errors: 1
+ });
+ expect(summarizeSpanTrace([])).toEqual({ startTime: '', duration: 0, services: 0, errors: 0 });
+ });
+});
diff --git a/frontend/src/lib/utils/span-explorer.ts b/frontend/src/lib/utils/span-explorer.ts
new file mode 100644
index 00000000..b8267202
--- /dev/null
+++ b/frontend/src/lib/utils/span-explorer.ts
@@ -0,0 +1,137 @@
+import type { Span } from '$lib/types/spans';
+import { addStickyParamsToHref } from './navigation';
+import { SPAN_STATUS_ERROR } from './span-fields';
+import { timestampNanos } from './timestamp-nanos';
+
+// A search reads every span of the project in the range, so the explorer opens on a short one.
+export const DEFAULT_SPAN_SEARCH_PRESET = '60m';
+export const MAX_SPAN_ATTRIBUTE_FILTERS = 10;
+
+export type SpanAttributeFilter = { key: string; value: string };
+
+export type SpanSearchFilters = {
+ name: string;
+ service: string;
+ kind: string;
+ status: string;
+ minMs: string;
+ maxMs: string;
+ traceId: string;
+ attributes: SpanAttributeFilter[];
+};
+
+export function spanTraceHref(
+ span: Pick<Span, 'traceId' | 'spanId' | 'recordedAt'>,
+ options: { focusSpan?: boolean } = {}
+): string | undefined {
+ if (!span.traceId) return undefined;
+ const focus = options.focusSpan && span.spanId ? `&span=${span.spanId}` : '';
+ return addStickyParamsToHref(
+ `/spans/${span.traceId}?at=${encodeURIComponent(span.recordedAt)}${focus}`
+ );
+}
+
+export function readSpanSearchFilters(search: string): SpanSearchFilters {
+ const params = new URLSearchParams(search);
+ const attributes = params
+ .getAll('attr')
+ .map((entry) => {
+ const split = entry.indexOf('=');
+ return split > 0 ? { key: entry.slice(0, split), value: entry.slice(split + 1) } : null;
+ })
+ .filter((filter): filter is SpanAttributeFilter => !!filter && !!filter.value)
+ .slice(0, MAX_SPAN_ATTRIBUTE_FILTERS);
+ return {
+ name: params.get('name') ?? '',
+ service: params.get('service') ?? '',
+ kind: params.get('kind') ?? '',
+ status: params.get('status') ?? '',
+ minMs: params.get('minMs') ?? '',
+ maxMs: params.get('maxMs') ?? '',
+ traceId: params.get('trace') ?? '',
+ attributes
+ };
+}
+
+export function spanSearchUrlParams(
+ filters: SpanSearchFilters
+): Record<string, string | string[] | undefined> {
+ return {
+ name: filters.name || undefined,
+ service: filters.service || undefined,
+ kind: filters.kind || undefined,
+ status: filters.status || undefined,
+ minMs: filters.minMs.trim() || undefined,
+ maxMs: filters.maxMs.trim() || undefined,
+ trace: filters.traceId.trim() || undefined,
+ attr: filters.attributes.map((filter) => `${filter.key}=${filter.value}`)
+ };
+}
+
+function durationNumber(text: string): number | undefined {
+ if (!text.trim()) return undefined;
+ const value = Number(text.trim());
+ if (!Number.isFinite(value)) {
+ throw Object.assign(new Error('Enter a finite duration in milliseconds.'), { status: 422 });
+ }
+ return value;
+}
+
+export function spanSearchBody(filters: SpanSearchFilters) {
+ return {
+ name: filters.name.trim(),
+ serviceName: filters.service,
+ traceId: filters.traceId.trim(),
+ kind: filters.kind === '' ? undefined : Number(filters.kind),
+ status: filters.status === '' ? undefined : Number(filters.status),
+ minDurationMs: durationNumber(filters.minMs),
+ maxDurationMs: durationNumber(filters.maxMs),
+ attributeFilters: filters.attributes
+ };
+}
+
+export function parseSpanAttributeFilter(
+ key: string,
+ value: string,
+ existing: SpanAttributeFilter[]
+): SpanAttributeFilter | string {
+ const filter = { key: key.trim(), value: value.trim() };
+ if (!filter.key) return 'Enter an attribute key.';
+ if (/["\\]/.test(filter.key)) return 'Attribute keys cannot contain quotes or backslashes.';
+ if (!filter.value) return 'Enter the value to match.';
+ if (existing.some((other) => other.key === filter.key && other.value === filter.value)) {
+ return 'This filter is already applied.';
+ }
+ if (existing.length >= MAX_SPAN_ATTRIBUTE_FILTERS) {
+ return `A search takes at most ${MAX_SPAN_ATTRIBUTE_FILTERS} attribute filters.`;
+ }
+ return filter;
+}
+
+export type SpanTraceSummary = {
+ startTime: string;
+ duration: number;
+ services: number;
+ errors: number;
+};
+
+export function summarizeSpanTrace(spans: Span[]): SpanTraceSummary {
+ let first: Span | undefined;
+ let start = 0n;
+ let end = 0n;
+ for (const span of spans) {
+ const spanStart = timestampNanos(span.startTime);
+ const spanEnd = spanStart + BigInt(Math.trunc(span.duration));
+ if (!first || spanStart < start) {
+ first = span;
+ start = spanStart;
+ }
+ if (spanEnd > end) end = spanEnd;
+ }
+ return {
+ startTime: first?.startTime ?? '',
+ duration: first ? Number(end - start) : 0,
+ services: new Set(spans.map((span) => span.serviceName).filter((name) => !!name)).size,
+ errors: spans.filter((span) => span.statusCode === SPAN_STATUS_ERROR).length
+ };
+}
diff --git a/frontend/src/lib/utils/span-fields.ts b/frontend/src/lib/utils/span-fields.ts
new file mode 100644
index 00000000..837f8808
--- /dev/null
+++ b/frontend/src/lib/utils/span-fields.ts
@@ -0,0 +1,35 @@
+import type { Span } from '$lib/types/spans';
+
+export const SPAN_KIND_OPTIONS = [
+ { value: '1', label: 'Internal' },
+ { value: '2', label: 'Server' },
+ { value: '3', label: 'Client' },
+ { value: '4', label: 'Producer' },
+ { value: '5', label: 'Consumer' }
+];
+
+export const SPAN_STATUS_OPTIONS = [
+ { value: '2', label: 'Error' },
+ { value: '1', label: 'Ok' },
+ { value: '0', label: 'Unset' }
+];
+
+export const SPAN_STATUS_ERROR = 2;
+
+export function spanKindLabel(kind: number | undefined): string {
+ return SPAN_KIND_OPTIONS.find((option) => option.value === String(kind))?.label ?? '';
+}
+
+export function spanStatusLabel(status: number | undefined): string {
+ return SPAN_STATUS_OPTIONS.find((option) => option.value === String(status ?? 0))?.label ?? '';
+}
+
+export function isErrorSpan(span: Pick<Span, 'statusCode'>): boolean {
+ return span.statusCode === SPAN_STATUS_ERROR;
+}
+
+export function spanServices(spans: Span[]): string[] {
+ return [...new Set(spans.map((span) => span.serviceName).filter((name) => !!name))].sort((a, b) =>
+ a!.localeCompare(b!)
+ ) as string[];
+}
diff --git a/frontend/src/lib/utils/span-id.test.ts b/frontend/src/lib/utils/span-id.test.ts
deleted file mode 100644
index 996b4321..00000000
--- a/frontend/src/lib/utils/span-id.test.ts
+++ /dev/null
@@ -1,30 +0,0 @@
-import { describe, expect, it } from 'vitest';
-import { logTraceId } from './span-id';
-
-const traceId = '8462dd06-1570-32df-3c6c-90ac5ff74ed7';
-const traceHex = '8462dd06157032df3c6c90ac5ff74ed7';
-const occurrenceId = '00000000-0000-0000-1985-a7abed0024db';
-const groupId = 'a5100000-0000-4000-8000-000000000001';
-
-describe('logTraceId', () => {
- it('correlates existing non-root occurrences without requiring a stored upstream root', () => {
- expect(logTraceId({ id: occurrenceId, distributedTraceId: traceId })).toBe(traceHex);
- });
-
- it('keeps native trace IDs independent of distributed grouping', () => {
- expect(logTraceId({ id: traceId, distributedTraceId: groupId })).toBe(traceHex);
- expect(logTraceId({ id: traceId, attributes: null })).toBe(traceHex);
- });
-
- it('uses the original OTel ID even when distributed grouping is overridden', () => {
- for (const id of [occurrenceId, traceId]) {
- expect(
- logTraceId({
- id,
- distributedTraceId: groupId,
- attributes: { 'traceway.otel.trace_id': traceHex.toUpperCase() }
- })
- ).toBe(traceHex);
- }
- });
-});
diff --git a/frontend/src/lib/utils/span-id.ts b/frontend/src/lib/utils/span-id.ts
deleted file mode 100644
index 34515527..00000000
--- a/frontend/src/lib/utils/span-id.ts
+++ /dev/null
@@ -1,26 +0,0 @@
-export function spanIdUuidToHex(spanUuid: string | null | undefined): string {
- if (!spanUuid) return '';
- return spanUuid.replace(/-/g, '').slice(-16).toLowerCase();
-}
-
-export function traceIdUuidToHex(traceUuid: string | null | undefined): string {
- if (!traceUuid) return '';
- return traceUuid.replace(/-/g, '').toLowerCase();
-}
-
-export function logTraceId(trace: {
- id: string;
- distributedTraceId?: string | null;
- attributes?: Record<string, string> | null;
-}): string {
- const otelTraceId = trace.attributes?.['traceway.otel.trace_id'];
- if (otelTraceId) return traceIdUuidToHex(otelTraceId);
-
- const occurrenceId = traceIdUuidToHex(trace.id);
- // Older OTel non-root occurrences store a zero-padded span ID as their ID.
- // Native traces keep their own trace ID, even when grouped into a distributed trace.
- if (occurrenceId.startsWith('0000000000000000') && trace.distributedTraceId) {
- return traceIdUuidToHex(trace.distributedTraceId);
- }
- return occurrenceId;
-}
diff --git a/frontend/src/lib/utils/span-tree.test.ts b/frontend/src/lib/utils/span-tree.test.ts
new file mode 100644
index 00000000..e82dc71b
--- /dev/null
+++ b/frontend/src/lib/utils/span-tree.test.ts
@@ -0,0 +1,185 @@
+import { describe, expect, it } from 'vitest';
+import {
+ buildSpanTree,
+ flattenBuiltTree,
+ flattenSpanTree,
+ initiallyCollapsed,
+ spanTreeKey
+} from './span-tree';
+import type { Span } from '$lib/types/spans';
+
+function span(id: string, parentSpanId?: string, trace = 'trace-a'): Span {
+ return {
+ spanId: id,
+ traceId: trace,
+ projectId: 'project',
+ name: id,
+ parentSpanId,
+ startTime: '2026-09-17T00:00:00Z',
+ duration: 100,
+ recordedAt: ''
+ };
+}
+
+describe('span graph', () => {
+ it('nests by protocol identity and hides only actual descendants', () => {
+ const parent = span('0000000000000001');
+ const child = span('0000000000000002', parent.spanId);
+ const grandchild = span('0000000000000003', child.spanId);
+ const sibling = span('0000000000000004');
+ const input = [grandchild, sibling, child, parent];
+ expect(
+ flattenSpanTree(input, undefined, new Set()).map((r) => [
+ r.span.spanId,
+ r.depth,
+ r.hasChildren
+ ])
+ ).toEqual([
+ [parent.spanId, 0, true],
+ [child.spanId, 1, true],
+ [grandchild.spanId, 2, false],
+ [sibling.spanId, 0, false]
+ ]);
+ expect(
+ flattenSpanTree(input, undefined, new Set([spanTreeKey(parent)])).map((r) => r.span.spanId)
+ ).toEqual([parent.spanId, sibling.spanId]);
+ });
+ it('keeps missing parents visible and does not join different traces or projects', () => {
+ const parent = span('01');
+ const orphan = span('02', '01', 'trace-b');
+ const otherProject = { ...span('03', '01'), projectId: 'other' };
+ const rows = flattenSpanTree([parent, orphan, otherProject], undefined, new Set());
+ expect(rows.every((r) => r.depth === 0 && !r.hasChildren)).toBe(true);
+ expect(rows.filter((r) => r.missingParent)).toHaveLength(2);
+ });
+ it('deduplicates retries and breaks cycles without phantom children', () => {
+ const a = span('01', '02');
+ const b = span('02', '01');
+ const self = span('03', '03');
+ const rows = flattenSpanTree([a, b, a, self], undefined, new Set());
+ expect(rows).toHaveLength(3);
+ expect(rows.filter((r) => r.hasChildren)).toHaveLength(1);
+ expect(flattenSpanTree([a, b], undefined, new Set([rows[0].key]))).toHaveLength(1);
+ });
+ it('orders siblings by exact start time and breaks ties by source ID', () => {
+ const at = (id: string, startTime: string) => ({ ...span(id), startTime });
+ const ordered = [
+ at('000000000000000a', '2026-09-17T00:00:00.000000001Z'),
+ at('000000000000000b', '2026-09-17T00:00:00.000000002Z'),
+ at('000000000000000c', '2026-09-17T00:00:00.0001Z'),
+ at('000000000000000d', '2026-09-17T00:00:00.0002Z'),
+ at('000000000000000e', '2026-09-17T00:00:01Z'),
+ at('000000000000000f', '2026-09-17T00:00:01Z')
+ ];
+ const expected = ordered.map((s) => s.spanId);
+ for (const input of [
+ [...ordered].reverse(),
+ [ordered[3], ordered[0], ordered[5], ordered[2], ordered[4], ordered[1]]
+ ]) {
+ expect(flattenSpanTree(input, undefined, new Set()).map((r) => r.span.spanId)).toEqual(
+ expected
+ );
+ }
+ });
+
+ it('handles deep traces without recursion', () => {
+ const spans = Array.from({ length: 15000 }, (_, i) =>
+ span(String(i + 1), i ? String(i) : undefined)
+ );
+ expect(flattenSpanTree(spans, undefined, new Set())).toHaveLength(15000);
+ });
+ it('nests spans of the native protocol by their 32 character ids', () => {
+ const a = span('11111111111111111111111111111111');
+ const b = span('22222222222222222222222222222222', a.spanId);
+ expect(flattenSpanTree([b, a], undefined, new Set()).map((r) => r.depth)).toEqual([0, 1]);
+ });
+ it('does not mark the spans directly under the root span as orphans', () => {
+ const child = span('02', '01');
+ const orphan = span('03', '09');
+ const rows = flattenSpanTree([child, orphan], '01', new Set());
+ expect(rows.map((r) => [r.span.spanId, r.missingParent])).toEqual([
+ ['02', false],
+ ['03', true]
+ ]);
+ });
+});
+
+describe('large traces', () => {
+ const make = (id: number, parent?: number, statusCode?: number): Span => ({
+ spanId: id.toString(16).padStart(16, '0'),
+ parentSpanId: parent === undefined ? undefined : parent.toString(16).padStart(16, '0'),
+ traceId: 'trace',
+ projectId: 'project',
+ name: `span ${id}`,
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000,
+ statusCode
+ });
+
+ it('counts what is below each node and knows where an error hides', () => {
+ const tree = buildSpanTree(
+ [make(1), make(2, 1), make(3, 2, 2), make(4, 1), make(5)],
+ undefined
+ );
+ const rows = flattenBuiltTree(tree, new Set());
+ expect(rows.map((row) => [row.span.name, row.descendants, row.errorBelow])).toEqual([
+ ['span 1', 3, true],
+ ['span 2', 1, true],
+ ['span 3', 0, false],
+ ['span 4', 0, false],
+ ['span 5', 0, false]
+ ]);
+ });
+
+ it('leave an ordinary trace fully open', () => {
+ const spans = [make(1), ...Array.from({ length: 300 }, (_, i) => make(i + 2, 1))];
+ expect(initiallyCollapsed(buildSpanTree(spans, undefined)).size).toBe(0);
+ });
+
+ it('open with only their widest nodes folded', () => {
+ const wide = Array.from({ length: 600 }, (_, i) => make(i + 100, 2));
+ const spans = [make(1), make(2, 1), make(3, 1), make(4, 3), ...wide];
+ const tree = buildSpanTree(spans, undefined);
+ const folded = initiallyCollapsed(tree);
+ expect([...folded]).toEqual([spanTreeKey(spans[1])]);
+ expect(flattenBuiltTree(tree, folded).map((row) => row.span.name)).toEqual([
+ 'span 1',
+ 'span 2',
+ 'span 3',
+ 'span 4'
+ ]);
+ });
+});
+
+describe('a trace read across projects', () => {
+ const hop = (projectId: string, spanId: string, parentSpanId?: string): Span => ({
+ spanId,
+ parentSpanId,
+ traceId: 'trace',
+ projectId,
+ name: `${projectId} ${spanId}`,
+ startTime: '2026-09-17T00:00:00Z',
+ recordedAt: '',
+ duration: 1_000
+ });
+ const spans = [hop('gateway', '01'), hop('payments', '02', '01'), hop('worker', '03', '02')];
+
+ it('hangs a child under its parent in another project', () => {
+ const rows = flattenBuiltTree(buildSpanTree(spans, undefined, true), new Set());
+ expect(rows.map((row) => [row.span.name, row.depth, row.missingParent])).toEqual([
+ ['gateway 01', 0, false],
+ ['payments 02', 1, false],
+ ['worker 03', 2, false]
+ ]);
+ });
+
+ it('keeps projects apart everywhere else', () => {
+ const rows = flattenBuiltTree(buildSpanTree(spans, undefined), new Set());
+ expect(rows.map((row) => [row.depth, row.missingParent])).toEqual([
+ [0, false],
+ [0, true],
+ [0, true]
+ ]);
+ });
+});
diff --git a/frontend/src/lib/utils/span-tree.ts b/frontend/src/lib/utils/span-tree.ts
index 9085135e..963bae8f 100644
--- a/frontend/src/lib/utils/span-tree.ts
+++ b/frontend/src/lib/utils/span-tree.ts
@@ -1,62 +1,162 @@
import type { Span } from '$lib/types/spans';
+import { timestampNanos } from './timestamp-nanos';
export type SpanTreeRow = {
+ key: string;
span: Span;
depth: number;
hasChildren: boolean;
+ missingParent: boolean;
+ descendants: number;
+ errorBelow: boolean;
};
-export function flattenSpanTree(
+export type SpanTree = {
+ byId: Map<string, Span>;
+ childrenById: Map<string, string[]>;
+ roots: string[];
+ missing: Set<string>;
+ descendants: Map<string, number>;
+ errorBelow: Set<string>;
+};
+
+const SPAN_STATUS_ERROR = 2;
+
+// A trace this large opens with its widest nodes folded, the way Datadog folds hidden spans into a numbered pill.
+export const COLLAPSE_MIN_SPANS = 500;
+export const COLLAPSE_MIN_CHILDREN = 50;
+
+// A span normally links only inside its own project, which keeps the same span exported to two projects apart. A whole
+// trace read across projects is the exception: there a child in one project hangs under its parent in another.
+function keyScope(span: Span, acrossProjects: boolean): string {
+ return acrossProjects ? 'trace' : span.projectId;
+}
+
+export function spanTreeKey(span: Span, acrossProjects = false): string {
+ return `${keyScope(span, acrossProjects)}:${span.traceId}:${span.spanId}`;
+}
+
+export function buildSpanTree(
spans: Span[],
rootSpanId: string | undefined,
- collapsed: Set<string>
-): SpanTreeRow[] {
- const byId = new Map(spans.map((s) => [s.id, s]));
- const childrenById = new Map<string, Span[]>();
- const roots: Span[] = [];
-
+ acrossProjects = false
+): SpanTree {
+ const byId = new Map<string, Span>();
for (const span of spans) {
- const parentId = span.parentSpanId;
- if (parentId && parentId !== rootSpanId && parentId !== span.id && byId.has(parentId)) {
- const siblings = childrenById.get(parentId);
- if (siblings) {
- siblings.push(span);
- } else {
- childrenById.set(parentId, [span]);
- }
- } else {
- roots.push(span);
+ const key = spanTreeKey(span, acrossProjects);
+ const old = byId.get(key);
+ if (!old || span.duration > old.duration) byId.set(key, span);
+ }
+ const parentById = new Map<string, string>();
+ const missing = new Set<string>();
+ for (const [key, span] of byId) {
+ if (!span.parentSpanId) continue;
+ const parentKey = `${keyScope(span, acrossProjects)}:${span.traceId}:${span.parentSpanId}`;
+ if (parentKey !== key && byId.has(parentKey)) {
+ parentById.set(key, parentKey);
+ } else if (span.parentSpanId !== rootSpanId) {
+ missing.add(key);
}
}
- const byStartTime = (a: Span, b: Span) =>
- new Date(a.startTime).getTime() - new Date(b.startTime).getTime();
+ // Corrupt cycles become roots; iterative traversal also handles very deep traces.
+ const checked = new Set<string>();
+ for (const key of byId.keys()) {
+ const path = new Set<string>();
+ let current: string | undefined = key;
+ while (current && !checked.has(current)) {
+ if (path.has(current)) {
+ parentById.delete(current);
+ missing.add(current);
+ break;
+ }
+ path.add(current);
+ current = parentById.get(current);
+ }
+ for (const visited of path) checked.add(visited);
+ }
- const rows: SpanTreeRow[] = [];
- const visited = new Set<string>();
-
- function walk(span: Span, depth: number, hidden: boolean) {
- if (visited.has(span.id)) return;
- visited.add(span.id);
- const children = childrenById.get(span.id) ?? [];
- if (!hidden) {
- rows.push({ span, depth, hasChildren: children.length > 0 });
+ const childrenById = new Map<string, string[]>();
+ const roots: string[] = [];
+ for (const key of byId.keys()) {
+ const parent = parentById.get(key);
+ if (!parent) roots.push(key);
+ else {
+ const children = childrenById.get(parent) ?? [];
+ children.push(key);
+ childrenById.set(parent, children);
}
- const hideChildren = hidden || collapsed.has(span.id);
- for (const child of [...children].sort(byStartTime)) {
- walk(child, depth + 1, hideChildren);
+ }
+ const startNanos = new Map<string, bigint>();
+ for (const [key, span] of byId) startNanos.set(key, timestampNanos(span.startTime));
+ const compare = (a: string, b: string) => {
+ const difference = startNanos.get(a)! - startNanos.get(b)!;
+ return difference < 0n ? -1 : difference > 0n ? 1 : a.localeCompare(b);
+ };
+ roots.sort(compare);
+ for (const children of childrenById.values()) children.sort(compare);
+
+ // Children before parents, without recursion: a trace can be thousands of spans deep.
+ const descendants = new Map<string, number>();
+ const errorBelow = new Set<string>();
+ const order: string[] = [];
+ const stack = [...roots];
+ while (stack.length) {
+ const key = stack.pop()!;
+ order.push(key);
+ for (const child of childrenById.get(key) ?? []) stack.push(child);
+ }
+ for (let i = order.length - 1; i >= 0; i--) {
+ const key = order[i];
+ let count = 0;
+ for (const child of childrenById.get(key) ?? []) {
+ count += 1 + (descendants.get(child) ?? 0);
+ if (errorBelow.has(child) || byId.get(child)!.statusCode === SPAN_STATUS_ERROR) {
+ errorBelow.add(key);
+ }
}
+ descendants.set(key, count);
}
- for (const root of [...roots].sort(byStartTime)) {
- walk(root, 0, false);
+ return { byId, childrenById, roots, missing, descendants, errorBelow };
+}
+
+export function initiallyCollapsed(tree: SpanTree): Set<string> {
+ const collapsed = new Set<string>();
+ if (tree.byId.size <= COLLAPSE_MIN_SPANS) return collapsed;
+ for (const [key, children] of tree.childrenById) {
+ if (children.length > COLLAPSE_MIN_CHILDREN) collapsed.add(key);
}
+ return collapsed;
+}
- for (const span of spans) {
- if (!visited.has(span.id)) {
- walk(span, 0, false);
+export function flattenBuiltTree(tree: SpanTree, collapsed: Set<string>): SpanTreeRow[] {
+ const rows: SpanTreeRow[] = [];
+ const pending = tree.roots.map((key) => ({ key, depth: 0 })).reverse();
+ while (pending.length) {
+ const { key, depth } = pending.pop()!;
+ const children = tree.childrenById.get(key) ?? [];
+ rows.push({
+ key,
+ span: tree.byId.get(key)!,
+ depth,
+ hasChildren: children.length > 0,
+ missingParent: tree.missing.has(key),
+ descendants: tree.descendants.get(key) ?? 0,
+ errorBelow: tree.errorBelow.has(key)
+ });
+ if (collapsed.has(key)) continue;
+ for (let i = children.length - 1; i >= 0; i--) {
+ pending.push({ key: children[i], depth: depth + 1 });
}
}
-
return rows;
}
+
+export function flattenSpanTree(
+ spans: Span[],
+ rootSpanId: string | undefined,
+ collapsed: Set<string>
+): SpanTreeRow[] {
+ return flattenBuiltTree(buildSpanTree(spans, rootSpanId), collapsed);
+}
diff --git a/frontend/src/lib/utils/timestamp-nanos.test.ts b/frontend/src/lib/utils/timestamp-nanos.test.ts
new file mode 100644
index 00000000..bd22fae3
--- /dev/null
+++ b/frontend/src/lib/utils/timestamp-nanos.test.ts
@@ -0,0 +1,21 @@
+import { describe, expect, it } from 'vitest';
+import { timestampNanos } from './timestamp-nanos';
+
+describe('timestampNanos', () => {
+ it('keeps every fractional digit a Go RFC3339Nano timestamp can carry', () => {
+ const base = BigInt(Date.UTC(2026, 8, 17)) * 1_000_000n;
+ expect(timestampNanos('2026-09-17T00:00:00Z')).toBe(base);
+ expect(timestampNanos('2026-09-17T00:00:00.000000001Z')).toBe(base + 1n);
+ expect(timestampNanos('2026-09-17T00:00:00.5Z')).toBe(base + 500_000_000n);
+ expect(timestampNanos('2026-09-17T00:00:00.123456789Z')).toBe(base + 123_456_789n);
+ expect(timestampNanos('2026-09-17T00:00:00.1234567891Z')).toBe(base + 123_456_789n);
+ });
+
+ it('applies zone offsets and treats unparseable input as zero', () => {
+ expect(timestampNanos('2026-09-17T02:00:00.000000007+02:00')).toBe(
+ timestampNanos('2026-09-17T00:00:00.000000007Z')
+ );
+ expect(timestampNanos('not a time')).toBe(0n);
+ expect(timestampNanos('')).toBe(0n);
+ });
+});
diff --git a/frontend/src/lib/utils/timestamp-nanos.ts b/frontend/src/lib/utils/timestamp-nanos.ts
new file mode 100644
index 00000000..ca2fa23b
--- /dev/null
+++ b/frontend/src/lib/utils/timestamp-nanos.ts
@@ -0,0 +1,10 @@
+const FRACTION = /^(.*?)(?:\.(\d+))?(Z|[+-]\d{2}:\d{2})?$/;
+
+// Exact epoch nanoseconds for ordering. preciseTimeMs is floating point and cannot separate spans a few hundred ns apart.
+export function timestampNanos(isoString: string): bigint {
+ const match = isoString.match(FRACTION);
+ const base = Date.parse(match ? `${match[1]}${match[3] ?? ''}` : isoString);
+ if (!match || Number.isNaN(base)) return 0n;
+ const fraction = (match[2] ?? '').padEnd(9, '0').slice(0, 9);
+ return BigInt(base) * 1_000_000n + BigInt(fraction || '0');
+}
diff --git a/frontend/src/routes/ai-traces/[traceName]/+page.svelte b/frontend/src/routes/ai-traces/[traceName]/+page.svelte
index d9ea40ad..250da11e 100644
--- a/frontend/src/routes/ai-traces/[traceName]/+page.svelte
+++ b/frontend/src/routes/ai-traces/[traceName]/+page.svelte
@@ -1,6 +1,6 @@
<script lang="ts">
import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { api } from '$lib/api';
import {
formatDuration,
@@ -156,7 +156,10 @@
loadData();
}
+ let loadSequence = 0;
+
async function loadData(pushToHistory = true) {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -179,11 +182,13 @@
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
traces = response.data || [];
stats = response.stats || null;
total = response.pagination.total;
totalPages = response.pagination.totalPages;
} catch (e) {
+ if (sequence !== loadSequence) return;
console.error(e);
errorStatus = getErrorStatus(e) || 0;
if (getErrorStatus(e) === 404) {
@@ -192,7 +197,7 @@
error = getErrorMessage(e) || 'Failed to load data';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -218,8 +223,24 @@
loadData(false);
}
- onMount(() => {
- loadData(false);
+ $effect(() => {
+ void data.traceName;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ traces = [];
+ stats = null;
+ page = 1;
+ const range = getInitialRange();
+ selectedPreset = range.preset;
+ fromDate = dateToCalendarDate(range.from, timezone);
+ toDate = dateToCalendarDate(range.to, timezone);
+ fromTime = dateToTimeString(range.from, timezone);
+ toTime = dateToTimeString(range.to, timezone);
+ loadData(false);
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
diff --git a/frontend/src/routes/ai-traces/[traceName]/[traceId]/+page.svelte b/frontend/src/routes/ai-traces/[traceName]/[traceId]/+page.svelte
index a2692937..bc6b4dad 100644
--- a/frontend/src/routes/ai-traces/[traceName]/[traceId]/+page.svelte
+++ b/frontend/src/routes/ai-traces/[traceName]/[traceId]/+page.svelte
@@ -1,5 +1,8 @@
<script lang="ts">
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
+ import SpanWaterfall from '$lib/components/spans/span-waterfall.svelte';
+ import SpanGraphNotice from '$lib/components/spans/span-graph-notice.svelte';
+ import type { Span, SpanGraphStatus, TraceIdentity } from '$lib/types/spans';
import { api } from '$lib/api';
import { formatDuration, formatDateTime } from '$lib/utils/formatters';
import { getTimezone } from '$lib/state/timezone.svelte';
@@ -13,14 +16,13 @@
import { createSmartBackHandler } from '$lib/utils/back-navigation';
import { resolve } from '$app/paths';
import TraceLogsPanel from '$lib/components/trace-logs/trace-logs-panel.svelte';
- import { logTraceId } from '$lib/utils/span-id';
import { formatCost } from '$lib/utils/ai-format';
import { extractMessages, formatConversationContent } from '$lib/utils/ai-conversation';
import ConversationMessages from '$lib/components/ai/conversation-messages.svelte';
import FlaggedBadge from '$lib/components/ai/flagged-badge.svelte';
import { addStickyParamsToHref } from '$lib/utils/navigation';
- type AiTrace = {
+ type AiTrace = TraceIdentity & {
id: string;
recordedAt: string;
duration: number;
@@ -43,7 +45,6 @@
serverName: string;
appVersion: string;
attributes: Record<string, string> | null;
- distributedTraceId?: string;
conversationId?: string;
toolCallCount?: number;
toolNames?: string[] | null;
@@ -52,6 +53,8 @@
};
type AiTraceDetailResponse = {
+ spanGraphStatus?: SpanGraphStatus;
+ spans?: Span[];
aiTrace: AiTrace;
conversation?: {
input: string;
@@ -63,14 +66,18 @@
const timezone = $derived(getTimezone());
- let response = $state<AiTraceDetailResponse | null>(null);
+ // Raw on purpose: the response can carry 20,000 spans and is only ever replaced, never edited.
+ let response = $state.raw<AiTraceDetailResponse | null>(null);
let loading = $state(true);
let error = $state('');
let notFound = $state(false);
let showRawJson = $state(false);
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -81,8 +88,10 @@
data.recordedAt ? { recordedAt: data.recordedAt } : {},
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
response = result;
} catch (e: unknown) {
+ if (sequence !== loadSequence) return;
console.error(e);
const err = e as { status?: number; message?: string };
if (err.status === 404) {
@@ -91,12 +100,22 @@
error = err.message || 'Failed to load AI trace details';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
- onMount(() => {
- loadData();
+ $effect(() => {
+ void data.traceId;
+ void data.recordedAt;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ response = null;
+ showRawJson = false;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
@@ -228,6 +247,24 @@
</Card.Root>
{/if}
+ {#if response.spans?.length || (response.spanGraphStatus && response.spanGraphStatus.state !== 'complete')}
+ <Card.Root>
+ <Card.Header><Card.Title>Spans</Card.Title></Card.Header>
+ <Card.Content class="space-y-3">
+ <SpanGraphNotice status={response.spanGraphStatus} />
+ {#if response.spans?.length}
+ <SpanWaterfall
+ spans={response.spans}
+ traceDuration={trace.duration}
+ traceStartTime={trace.recordedAt}
+ traceId={trace.traceId}
+ rootSpanId={trace.spanId}
+ />
+ {/if}
+ </Card.Content>
+ </Card.Root>
+ {/if}
+
{#if response.conversation}
{@const conv = response.conversation}
{@const chatMessages = extractMessages(conv.input, conv.output)}
@@ -284,9 +321,9 @@
<TraceLogsPanel
projectId={projectsState.currentProjectId ?? ''}
- traceId={logTraceId(trace)}
- distributedTraceId={trace.distributedTraceId ?? null}
- spans={[]}
+ traceId={trace.traceId}
+ spans={response.spans ?? []}
+ rootSpan={{ spanId: trace.spanId, name: trace.traceName }}
traceRecordedAt={trace.recordedAt}
/>
{/if}
diff --git a/frontend/src/routes/ai-traces/conversations/[conversationId]/+page.svelte b/frontend/src/routes/ai-traces/conversations/[conversationId]/+page.svelte
index 64245cf6..a305ec67 100644
--- a/frontend/src/routes/ai-traces/conversations/[conversationId]/+page.svelte
+++ b/frontend/src/routes/ai-traces/conversations/[conversationId]/+page.svelte
@@ -1,5 +1,5 @@
<script lang="ts">
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { api } from '$lib/api';
import { formatDuration, formatDateTime } from '$lib/utils/formatters';
import { getTimezone } from '$lib/state/timezone.svelte';
@@ -99,7 +99,10 @@
return `${(spanMs / 3_600_000).toFixed(1)}h`;
}
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -110,8 +113,10 @@
{ conversationId },
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
response = result;
} catch (e: unknown) {
+ if (sequence !== loadSequence) return;
console.error(e);
const err = e as { status?: number; message?: string };
if (err.status === 404) {
@@ -120,12 +125,20 @@
error = err.message || 'Failed to load conversation';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
- onMount(() => {
- loadData();
+ $effect(() => {
+ void conversationId;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ response = null;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
diff --git a/frontend/src/routes/dashboards/+page.svelte b/frontend/src/routes/dashboards/+page.svelte
index 1860aa72..5a1dc80d 100644
--- a/frontend/src/routes/dashboards/+page.svelte
+++ b/frontend/src/routes/dashboards/+page.svelte
@@ -1066,7 +1066,7 @@
</EmptyState>
{:else}
<Tabs.Root value={activeTabId} onValueChange={handleTabChange}>
- <div class="flex items-center gap-2" bind:this={tabsRowEl}>
+ <div class="flex flex-wrap items-center gap-2" bind:this={tabsRowEl}>
{#if hasOverflow}
<Select.Root
type="single"
@@ -1075,7 +1075,9 @@
if (v) handleTabChange(v);
}}
>
- <Select.Trigger size="sm">{activeTabName}</Select.Trigger>
+ <Select.Trigger size="sm" class="max-w-[70%] min-w-0">
+ <span class="truncate">{activeTabName}</span>
+ </Select.Trigger>
<Select.Content>
{#each dashboards as dashboard (dashboard.id)}
<Select.Item value={String(dashboard.id)}>{dashboard.name}</Select.Item>
@@ -1206,7 +1208,10 @@
</DropdownMenu.Root>
{/if}
{#if lastUpdated}
- <div class="ml-auto flex items-center gap-1" bind:this={updatedEl}>
+ <div
+ class="ml-auto flex w-full shrink-0 items-center justify-end gap-1 sm:w-auto"
+ bind:this={updatedEl}
+ >
<span class="text-sm whitespace-nowrap text-muted-foreground">
Updated: {lastUpdatedFormatted}
</span>
@@ -1214,6 +1219,7 @@
variant="ghost"
size="sm"
onclick={reloadDashboardWidgets}
+ aria-label="Refresh dashboard"
disabled={loadingDashboard}
>
<RefreshCw class="h-4 w-4 {loadingDashboard ? 'animate-spin' : ''}" />
diff --git a/frontend/src/routes/endpoints/[endpoint]/+page.svelte b/frontend/src/routes/endpoints/[endpoint]/+page.svelte
index 7beccd7b..6a4db86c 100644
--- a/frontend/src/routes/endpoints/[endpoint]/+page.svelte
+++ b/frontend/src/routes/endpoints/[endpoint]/+page.svelte
@@ -1,6 +1,6 @@
<script lang="ts">
import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
- import { onMount, onDestroy } from 'svelte';
+ import { onMount, onDestroy, untrack } from 'svelte';
import { SvelteURLSearchParams } from 'svelte/reactivity';
import { api } from '$lib/api';
import {
@@ -214,7 +214,10 @@
}
}
+ let loadSequence = 0;
+
async function loadData(pushToHistory = true) {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -248,11 +251,13 @@
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
transactions = response.data || [];
stats = response.stats || null;
total = response.pagination.total;
totalPages = response.pagination.totalPages;
} catch (e) {
+ if (sequence !== loadSequence) return;
console.error(e);
errorStatus = getErrorStatus(e) || 0;
if (getErrorStatus(e) === 404) {
@@ -261,7 +266,7 @@
error = getErrorMessage(e) || 'Failed to load data';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -287,15 +292,20 @@
loadData(false);
}
+ let slowSequence = 0;
+
async function loadSlowEndpoint() {
+ const sequence = ++slowSequence;
try {
const response = await api.get(
`/endpoints/slow?endpoint=${encodeURIComponent(data.endpoint)}`,
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== slowSequence) return;
offsetMs = response.offsetMs ?? 0;
reason = response.reason ?? '';
} catch {
+ if (sequence !== slowSequence) return;
offsetMs = 0;
reason = '';
}
@@ -340,8 +350,6 @@
onMount(() => {
window.addEventListener('popstate', handlePopState);
- loadData(false);
- loadSlowEndpoint();
});
onDestroy(() => {
@@ -349,6 +357,31 @@
window.removeEventListener('popstate', handlePopState);
}
});
+
+ $effect(() => {
+ void data.endpoint;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ transactions = [];
+ stats = null;
+ page = 1;
+ const range = getInitialRange();
+ selectedPreset = range.preset;
+ fromDate = dateToCalendarDate(range.from, timezone);
+ toDate = dateToCalendarDate(range.to, timezone);
+ fromTime = dateToTimeString(range.from, timezone);
+ toTime = dateToTimeString(range.to, timezone);
+ loadData(false);
+ offsetMs = 0;
+ reason = '';
+ showSlowDialog = false;
+ loadSlowEndpoint();
+ });
+ return () => {
+ loadSequence++;
+ slowSequence++;
+ };
+ });
</script>
<div class="space-y-6">
diff --git a/frontend/src/routes/endpoints/[endpoint]/[endpointId]/+page.svelte b/frontend/src/routes/endpoints/[endpoint]/[endpointId]/+page.svelte
index 6c8a6a44..d385e757 100644
--- a/frontend/src/routes/endpoints/[endpoint]/[endpointId]/+page.svelte
+++ b/frontend/src/routes/endpoints/[endpoint]/[endpointId]/+page.svelte
@@ -1,5 +1,5 @@
<script lang="ts">
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { goto } from '$app/navigation';
import { api } from '$lib/api';
import { formatDuration, getStatusColor, formatDateTime } from '$lib/utils/formatters';
@@ -14,6 +14,7 @@
import { LabelValue } from '$lib/components/ui/label-value';
import { AttributesGrid } from '$lib/components/ui/attributes-grid/index.js';
import SpanWaterfall from '$lib/components/spans/span-waterfall.svelte';
+ import SpanGraphNotice from '$lib/components/spans/span-graph-notice.svelte';
import SpanEmptyState from '$lib/components/spans/span-empty-state.svelte';
import type { TraceDetailResponse } from '$lib/types/spans';
import PageHeader from '$lib/components/traceway/page-header.svelte';
@@ -23,18 +24,21 @@
import { resolve } from '$app/paths';
import DistributedTraceCard from '$lib/components/distributed-trace/distributed-trace-card.svelte';
import TraceLogsPanel from '$lib/components/trace-logs/trace-logs-panel.svelte';
- import { logTraceId } from '$lib/utils/span-id';
let { data } = $props();
const timezone = $derived(getTimezone());
- let response = $state<TraceDetailResponse | null>(null);
+ // Raw on purpose: the response can carry 20,000 spans and is only ever replaced, never edited.
+ let response = $state.raw<TraceDetailResponse | null>(null);
let loading = $state(true);
let error = $state('');
let notFound = $state(false);
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -45,8 +49,10 @@
data.recordedAt ? { recordedAt: data.recordedAt } : {},
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
response = result;
} catch (e: unknown) {
+ if (sequence !== loadSequence) return;
console.error(e);
const err = e as { status?: number; message?: string };
if (err.status === 404) {
@@ -55,7 +61,7 @@
error = err.message || 'Failed to load endpoint details';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -69,8 +75,17 @@
}
}
- onMount(() => {
- loadData();
+ $effect(() => {
+ void data.endpointId;
+ void data.recordedAt;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ response = null;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
@@ -262,15 +277,17 @@
</Card.Description>
</Card.Header>
{/if}
- <Card.Content>
+ <Card.Content class="space-y-3">
+ <SpanGraphNotice status={response.spanGraphStatus} />
{#if response.hasSpans}
<SpanWaterfall
spans={response.spans}
traceDuration={response.endpoint.duration}
traceStartTime={response.endpoint.recordedAt}
+ traceId={response.endpoint.traceId}
rootSpanId={response.endpoint.spanId}
/>
- {:else}
+ {:else if !response.spanGraphStatus || response.spanGraphStatus.state === 'complete'}
<SpanEmptyState framework={projectsState.currentProject?.framework ?? 'gin'} />
{/if}
</Card.Content>
@@ -278,15 +295,15 @@
<TraceLogsPanel
projectId={projectsState.currentProjectId ?? ''}
- traceId={logTraceId(response.endpoint)}
- distributedTraceId={response.endpoint.distributedTraceId ?? null}
+ traceId={response.endpoint.traceId}
spans={response.spans ?? []}
+ rootSpan={{ spanId: response.endpoint.spanId, name: response.endpoint.endpoint }}
traceRecordedAt={response.endpoint.recordedAt}
/>
- {#if response.endpoint.distributedTraceId}
+ {#if response.endpoint.traceId}
<DistributedTraceCard
- distributedTraceId={response.endpoint.distributedTraceId}
+ traceId={response.endpoint.traceId}
currentNodeId={response.endpoint.id}
recordedAt={response.endpoint.recordedAt}
/>
diff --git a/frontend/src/routes/issues/[exceptionHash]/+page.svelte b/frontend/src/routes/issues/[exceptionHash]/+page.svelte
index 6ba2582c..a7430fbb 100644
--- a/frontend/src/routes/issues/[exceptionHash]/+page.svelte
+++ b/frontend/src/routes/issues/[exceptionHash]/+page.svelte
@@ -1,6 +1,6 @@
<script lang="ts">
import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import { api } from '$lib/api';
@@ -12,6 +12,7 @@
import { toast } from 'svelte-sonner';
import ArchiveConfirmationDialog from '$lib/components/archive-confirmation-dialog.svelte';
import OncallOwner from '$lib/components/traceway/oncall-owner.svelte';
+ import { linkedTraceFrom } from '$lib/types/exceptions';
import type {
ExceptionGroup,
ExceptionOccurrence,
@@ -53,7 +54,10 @@
latestOccurrence?.stackTrace.split('\n')[0] || 'Exception'
);
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -74,40 +78,15 @@
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
group = response.group;
occurrences = response.occurrences || [];
total = response.pagination.total;
sessionRecording = response.sessionRecording ?? null;
sessionId = response.sessionId ?? null;
-
- // Load linked trace if the latest occurrence has a traceId
- const firstOccurrence = occurrences[0];
- if (firstOccurrence?.traceId) {
- try {
- const isTask = firstOccurrence.traceType === 'task';
- const endpoint = isTask ? '/tasks' : '/endpoints';
- const txResponse = await api.post(
- `${endpoint}/${firstOccurrence.traceId}`,
- {},
- { projectId: projectsState.currentProjectId ?? undefined }
- );
- const txData = isTask ? txResponse.task : txResponse.endpoint;
- if (txData) {
- linkedTrace = {
- id: txData.id,
- endpoint: isTask ? txData.taskName : txData.endpoint,
- duration: txData.duration,
- statusCode: txData.statusCode || 0,
- recordedAt: txData.recordedAt,
- traceType: isTask ? 'task' : 'endpoint',
- distributedTraceId: txData.distributedTraceId
- };
- }
- } catch (txError) {
- console.warn('Could not load linked trace:', txError);
- }
- }
+ linkedTrace = linkedTraceFrom(response.relatedEntity);
} catch (e) {
+ if (sequence !== loadSequence) return;
console.error(e);
if (getErrorStatus(e) === 404) {
notFound = true;
@@ -115,7 +94,7 @@
error = getErrorMessage(e) || 'Failed to load exception details';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -137,8 +116,18 @@
}
}
- onMount(() => {
- loadData();
+ $effect(() => {
+ void page.params.exceptionHash;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ group = null;
+ occurrences = [];
+ total = 0;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
diff --git a/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte b/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte
index a8bb9ddf..83370482 100644
--- a/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte
+++ b/frontend/src/routes/issues/[exceptionHash]/[exceptionId]/+page.svelte
@@ -1,4 +1,5 @@
<script lang="ts">
+ import { untrack } from 'svelte';
import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
import { api } from '$lib/api';
import { LoadingCircle } from '$lib/components/ui/loading-circle';
@@ -8,6 +9,7 @@
import { formatDateTime } from '$lib/utils/formatters';
import { StackTraceCard, EventCard, EventsTable } from '$lib/components/issues';
import PageHeader from '$lib/components/traceway/page-header.svelte';
+ import { linkedTraceFrom } from '$lib/types/exceptions';
import type { ExceptionOccurrence, LinkedTrace, SessionRecording } from '$lib/types/exceptions';
import { createSmartBackHandler } from '$lib/utils/back-navigation';
import { resolve } from '$app/paths';
@@ -50,7 +52,10 @@
occurrence ? `Event from ${formatDateTime(occurrence.recordedAt, { timezone })}` : 'Loading...'
);
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -64,7 +69,9 @@
data.recordedAt ? { recordedAt: data.recordedAt } : {},
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
occurrence = exceptionResponse.exception;
+ linkedTrace = linkedTraceFrom(exceptionResponse.relatedEntity);
sessionRecording = exceptionResponse.sessionRecording ?? null;
sessionId = exceptionResponse.sessionId ?? null;
@@ -85,45 +92,11 @@
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
allOccurrences = response.occurrences || [];
total = response.pagination.total;
-
- // Load linked trace if this occurrence has a traceId
- if (occurrence.traceId) {
- try {
- const isTask = occurrence.traceType === 'task';
- console.log('DEBUG linked trace:', {
- traceId: occurrence.traceId,
- traceType: occurrence.traceType,
- isTask
- });
- const endpoint = isTask ? '/tasks' : '/endpoints';
- const txResponse = await api.post(
- `${endpoint}/${occurrence.traceId}`,
- {},
- { projectId: projectsState.currentProjectId ?? undefined }
- );
- const txData = isTask ? txResponse.task : txResponse.endpoint;
- if (txData) {
- linkedTrace = {
- id: txData.id,
- endpoint: isTask ? txData.taskName : txData.endpoint,
- duration: txData.duration,
- statusCode: txData.statusCode || 0,
- recordedAt: txData.recordedAt,
- traceType: isTask ? 'task' : 'endpoint',
- distributedTraceId: txData.distributedTraceId
- };
- }
- } catch (txError) {
- console.error('Failed to load linked trace:', {
- error: txError,
- traceId: occurrence.traceId,
- traceType: occurrence.traceType
- });
- }
- }
} catch (e) {
+ if (sequence !== loadSequence) return;
console.error(e);
if (getErrorStatus(e) === 404) {
notFound = true;
@@ -131,7 +104,7 @@
error = getErrorMessage(e) || 'Failed to load exception details';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -155,7 +128,18 @@
$effect(() => {
void data.exceptionId;
- loadData();
+ void data.exceptionHash;
+ void data.recordedAt;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ occurrence = null;
+ allOccurrences = [];
+ sessionId = null;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
diff --git a/frontend/src/routes/issues/[exceptionHash]/events/+page.svelte b/frontend/src/routes/issues/[exceptionHash]/events/+page.svelte
index 3bbe7be7..f6bdc73f 100644
--- a/frontend/src/routes/issues/[exceptionHash]/events/+page.svelte
+++ b/frontend/src/routes/issues/[exceptionHash]/events/+page.svelte
@@ -1,6 +1,6 @@
<script lang="ts">
import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { page } from '$app/state';
import { goto } from '$app/navigation';
import { createRowClickHandler } from '$lib/utils/navigation';
@@ -41,7 +41,7 @@
type ExceptionOccurrence = {
id: string;
- traceId: string | null;
+ traceId: string;
exceptionHash: string;
stackTrace: string;
recordedAt: string;
@@ -64,7 +64,10 @@
let total = $state(0);
let totalPages = $state(0);
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -82,11 +85,13 @@
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
group = response.group;
occurrences = response.occurrences || [];
total = response.pagination.total;
totalPages = response.pagination.totalPages;
} catch (e) {
+ if (sequence !== loadSequence) return;
console.error(e);
if (getErrorStatus(e) === 404) {
notFound = true;
@@ -94,7 +99,7 @@
error = getErrorMessage(e) || 'Failed to load exception details';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -129,8 +134,19 @@
}
}
- onMount(() => {
- loadData();
+ $effect(() => {
+ void page.params.exceptionHash;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ group = null;
+ occurrences = [];
+ currentPage = 1;
+ total = 0;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
diff --git a/frontend/src/routes/sessions/[sessionId]/+page.svelte b/frontend/src/routes/sessions/[sessionId]/+page.svelte
index 0c3607e5..d15273c4 100644
--- a/frontend/src/routes/sessions/[sessionId]/+page.svelte
+++ b/frontend/src/routes/sessions/[sessionId]/+page.svelte
@@ -143,7 +143,7 @@
<Card.Header>
<Card.Title>Overview</Card.Title>
</Card.Header>
- <Card.Content class="grid grid-cols-2 gap-4 sm:grid-cols-3">
+ <Card.Content class="grid grid-cols-1 gap-4 sm:grid-cols-3">
<LabelValue label="Started" value={formatDateTime(session.startedAt, { timezone })} />
<LabelValue label="Duration" value={durationLabel(session)} />
<LabelValue label="App version" value={session.appVersion || '—'} />
diff --git a/frontend/src/routes/spans/+page.svelte b/frontend/src/routes/spans/+page.svelte
new file mode 100644
index 00000000..d1e53535
--- /dev/null
+++ b/frontend/src/routes/spans/+page.svelte
@@ -0,0 +1,546 @@
+<script lang="ts">
+ import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
+ import { onMount, onDestroy } from 'svelte';
+ import { browser } from '$app/environment';
+ import { api } from '$lib/api';
+ import {
+ formatDuration,
+ formatDateTime,
+ toUTCISO,
+ calendarDateTimeToLuxon
+ } from '$lib/utils/formatters';
+ import { getTimezone } from '$lib/state/timezone.svelte';
+ import { projectsState } from '$lib/state/projects.svelte';
+ import * as Table from '$lib/components/ui/table';
+ import * as AlertDialog from '$lib/components/ui/alert-dialog';
+ import { Button } from '$lib/components/ui/button';
+ import { Input } from '$lib/components/ui/input';
+ import { LoadingCircle } from '$lib/components/ui/loading-circle';
+ import { TracewayTableHeader } from '$lib/components/ui/traceway-table-header';
+ import { TableEmptyState } from '$lib/components/ui/table-empty-state';
+ import { PaginationFooter } from '$lib/components/ui/pagination-footer';
+ import { TimeRangePicker } from '$lib/components/ui/time-range-picker';
+ import { SearchBar } from '$lib/components/ui/search-bar';
+ import { WarningCallout } from '$lib/components/ui/warning-callout';
+ import PageHeader from '$lib/components/traceway/page-header.svelte';
+ import TableContainer from '$lib/components/traceway/table-container.svelte';
+ import ToolbarSelect from '$lib/components/traceway/toolbar-select.svelte';
+ import FormField from '$lib/components/traceway/form-field.svelte';
+ import { CircleAlert, Plus, X } from '@lucide/svelte';
+ import { CalendarDate } from '@internationalized/date';
+ import { createRowClickHandler } from '$lib/utils/navigation';
+ import {
+ getTimeRangeFromPreset,
+ dateToCalendarDate,
+ dateToTimeString,
+ parseTimeRangeFromUrl,
+ getResolvedTimeRange,
+ updateUrl
+ } from '$lib/utils/url-params';
+ import {
+ getSortState,
+ setSortState,
+ handleSortClick,
+ type SortDirection
+ } from '$lib/utils/sort-storage';
+ import type { Span } from '$lib/types/spans';
+ import {
+ SPAN_KIND_OPTIONS,
+ SPAN_STATUS_OPTIONS,
+ isErrorSpan,
+ spanKindLabel,
+ spanStatusLabel
+ } from '$lib/utils/span-fields';
+ import {
+ type SpanAttributeFilter,
+ type SpanSearchFilters,
+ DEFAULT_SPAN_SEARCH_PRESET,
+ parseSpanAttributeFilter,
+ readSpanSearchFilters,
+ spanSearchBody,
+ spanSearchUrlParams,
+ spanTraceHref
+ } from '$lib/utils/span-explorer';
+
+ const timezone = $derived(getTimezone());
+ const initialTimezone = getTimezone();
+
+ type SortField = 'start_time' | 'duration';
+ type SpanSearchResponse = {
+ data: Span[] | null;
+ pagination: { total: number; totalPages: number };
+ services: string[] | null;
+ };
+
+ let spans = $state<Span[]>([]);
+ let services = $state<string[]>([]);
+ let loading = $state(true);
+ let loadSequence = 0;
+ let error = $state('');
+ // A 422 is advice about the search itself (range too long, search too slow), not a failure.
+ let guidance = $state('');
+
+ let page = $state(1);
+ let pageSize = $state(50);
+ let total = $state(0);
+ let totalPages = $state(0);
+
+ const initialUrlParams = parseTimeRangeFromUrl(initialTimezone, DEFAULT_SPAN_SEARCH_PRESET);
+ const initialRange = getResolvedTimeRange(initialUrlParams, initialTimezone);
+
+ function filtersFromUrl(): SpanSearchFilters {
+ return readSpanSearchFilters(browser ? window.location.search : '');
+ }
+
+ let filters = $state<SpanSearchFilters>(filtersFromUrl());
+
+ let selectedPreset = $state<string | null>(initialUrlParams.preset);
+ let fromDate = $state<CalendarDate>(dateToCalendarDate(initialRange.from, initialTimezone));
+ let toDate = $state<CalendarDate>(dateToCalendarDate(initialRange.to, initialTimezone));
+ let fromTime = $state(dateToTimeString(initialRange.from, initialTimezone));
+ let toTime = $state(dateToTimeString(initialRange.to, initialTimezone));
+
+ const SORT_STORAGE_KEY = 'spans';
+ const initialSort = getSortState(SORT_STORAGE_KEY, { field: 'start_time', direction: 'desc' });
+ let orderBy = $state<SortField>(initialSort.field === 'duration' ? 'duration' : 'start_time');
+ let sortDirection = $state<SortDirection>(initialSort.direction);
+
+ const serviceOptions = $derived([
+ { value: '', label: 'All services' },
+ ...[...new Set([...services, filters.service].filter((name) => !!name))].map((name) => ({
+ value: name,
+ label: name
+ }))
+ ]);
+ const kindOptions = [{ value: '', label: 'All kinds' }, ...SPAN_KIND_OPTIONS];
+ const statusOptions = [{ value: '', label: 'Any status' }, ...SPAN_STATUS_OPTIONS];
+
+ function getFromDateTimeUTC(): string {
+ const [hour, minute] = (fromTime || '00:00').split(':').map(Number);
+ return toUTCISO(
+ calendarDateTimeToLuxon(
+ { year: fromDate.year, month: fromDate.month, day: fromDate.day, hour, minute },
+ timezone
+ )
+ );
+ }
+
+ function getToDateTimeUTC(): string {
+ const [hour, minute] = (toTime || '23:59').split(':').map(Number);
+ return toUTCISO(
+ calendarDateTimeToLuxon(
+ { year: toDate.year, month: toDate.month, day: toDate.day, hour, minute },
+ timezone
+ ).endOf('minute')
+ );
+ }
+
+ function updateExplorerUrl(pushToHistory: boolean) {
+ const range: Record<string, string | string[] | undefined> = selectedPreset
+ ? { preset: selectedPreset }
+ : { from: getFromDateTimeUTC(), to: getToDateTimeUTC() };
+ updateUrl({ ...range, ...spanSearchUrlParams(filters) }, { pushToHistory });
+ }
+
+ function handlePopState() {
+ const urlParams = parseTimeRangeFromUrl(timezone, DEFAULT_SPAN_SEARCH_PRESET);
+ const range = getResolvedTimeRange(urlParams, timezone);
+ selectedPreset = urlParams.preset;
+ fromDate = dateToCalendarDate(range.from, timezone);
+ fromTime = dateToTimeString(range.from, timezone);
+ toDate = dateToCalendarDate(range.to, timezone);
+ toTime = dateToTimeString(range.to, timezone);
+ filters = filtersFromUrl();
+ page = 1;
+ loadData(false);
+ }
+
+ function handleTimeRangeChange(
+ from: { date: CalendarDate; time: string },
+ to: { date: CalendarDate; time: string },
+ preset: string | null
+ ) {
+ fromDate = from.date;
+ fromTime = from.time;
+ toDate = to.date;
+ toTime = to.time;
+ selectedPreset = preset;
+ page = 1;
+ loadData(true);
+ }
+
+ async function loadData(pushToHistory = true) {
+ const sequence = ++loadSequence;
+ loading = true;
+ error = '';
+ guidance = '';
+
+ if (selectedPreset) {
+ const range = getTimeRangeFromPreset(selectedPreset, timezone);
+ fromDate = dateToCalendarDate(range.from, timezone);
+ toDate = dateToCalendarDate(range.to, timezone);
+ fromTime = dateToTimeString(range.from, timezone);
+ toTime = dateToTimeString(range.to, timezone);
+ }
+
+ updateExplorerUrl(pushToHistory);
+
+ try {
+ const response = (await api.post(
+ '/spans/search',
+ {
+ fromDate: getFromDateTimeUTC(),
+ toDate: getToDateTimeUTC(),
+ orderBy: `${orderBy} ${sortDirection}`,
+ ...spanSearchBody(filters),
+ pagination: { page, pageSize }
+ },
+ { projectId: projectsState.currentProjectId ?? undefined }
+ )) as SpanSearchResponse;
+ if (sequence !== loadSequence) return;
+
+ spans = response.data || [];
+ services = response.services || [];
+ total = response.pagination.total;
+ totalPages = response.pagination.totalPages;
+ } catch (e) {
+ if (sequence !== loadSequence) return;
+ console.error(e);
+ spans = [];
+ total = 0;
+ totalPages = 0;
+ if (getErrorStatus(e) === 422) {
+ guidance = getErrorMessage(e);
+ } else {
+ error = getErrorMessage(e) || 'Failed to load spans';
+ }
+ } finally {
+ if (sequence === loadSequence) loading = false;
+ }
+ }
+
+ function applyFilters() {
+ page = 1;
+ loadData(true);
+ }
+
+ function handlePageChange(newPage: number) {
+ if (newPage >= 1 && newPage <= totalPages) {
+ page = newPage;
+ loadData(false);
+ }
+ }
+
+ function handlePageSizeChange(newPageSize: number) {
+ pageSize = newPageSize;
+ page = 1;
+ loadData(false);
+ }
+
+ function handleSort(field: SortField) {
+ const newSort = handleSortClick(field, orderBy, sortDirection);
+ orderBy = newSort.field as SortField;
+ sortDirection = newSort.direction;
+ setSortState(SORT_STORAGE_KEY, newSort);
+ page = 1;
+ loadData(false);
+ }
+
+ let filterDialogOpen = $state(false);
+ let dialogKey = $state('');
+ let dialogValue = $state('');
+ let dialogError = $state('');
+
+ function openFilterDialog() {
+ dialogKey = '';
+ dialogValue = '';
+ dialogError = '';
+ filterDialogOpen = true;
+ }
+
+ function submitFilterDialog() {
+ const parsed = parseSpanAttributeFilter(dialogKey, dialogValue, filters.attributes);
+ if (typeof parsed === 'string') {
+ dialogError = parsed;
+ return;
+ }
+ filters.attributes = [...filters.attributes, parsed];
+ filterDialogOpen = false;
+ applyFilters();
+ }
+
+ function handleDialogKeydown(e: KeyboardEvent) {
+ if (e.key === 'Enter') {
+ e.preventDefault();
+ submitFilterDialog();
+ }
+ }
+
+ function removeAttributeFilter(filter: SpanAttributeFilter) {
+ filters.attributes = filters.attributes.filter((existing) => existing !== filter);
+ applyFilters();
+ }
+
+ function handleInputKeydown(e: KeyboardEvent) {
+ if (e.key === 'Enter') applyFilters();
+ }
+
+ onMount(() => {
+ window.addEventListener('popstate', handlePopState);
+ loadData(false);
+ });
+
+ onDestroy(() => {
+ if (typeof window !== 'undefined') {
+ window.removeEventListener('popstate', handlePopState);
+ }
+ });
+</script>
+
+<div class="space-y-4">
+ <PageHeader title="Spans" description="Every OpenTelemetry span this project has received.">
+ {#snippet actions()}
+ <TimeRangePicker
+ bind:fromDate
+ bind:toDate
+ bind:fromTime
+ bind:toTime
+ bind:preset={selectedPreset}
+ onApply={handleTimeRangeChange}
+ />
+ {/snippet}
+ </PageHeader>
+
+ <SearchBar
+ placeholder="Search span names..."
+ bind:value={filters.name}
+ onSearch={applyFilters}
+ disabled={loading}
+ >
+ {#snippet pillEnd()}
+ <ToolbarSelect
+ bind:value={filters.service}
+ options={serviceOptions}
+ ariaLabel="Service"
+ class="h-9 w-[170px] shrink-0 shadow-none sm:rounded-none sm:border-r-0"
+ onChange={applyFilters}
+ />
+ {/snippet}
+ </SearchBar>
+
+ <div class="flex flex-wrap items-center gap-2">
+ <ToolbarSelect
+ bind:value={filters.kind}
+ options={kindOptions}
+ ariaLabel="Span kind"
+ class="w-[130px]"
+ onChange={applyFilters}
+ />
+ <ToolbarSelect
+ bind:value={filters.status}
+ options={statusOptions}
+ ariaLabel="Span status"
+ class="w-[130px]"
+ onChange={applyFilters}
+ />
+ <Input
+ inputmode="decimal"
+ placeholder="Min ms"
+ aria-label="Minimum duration in milliseconds"
+ class="h-9 w-[100px]"
+ bind:value={filters.minMs}
+ onchange={applyFilters}
+ onkeydown={handleInputKeydown}
+ />
+ <Input
+ inputmode="decimal"
+ placeholder="Max ms"
+ aria-label="Maximum duration in milliseconds"
+ class="h-9 w-[100px]"
+ bind:value={filters.maxMs}
+ onchange={applyFilters}
+ onkeydown={handleInputKeydown}
+ />
+ <Input
+ placeholder="Trace ID"
+ aria-label="Trace ID"
+ class="h-9 w-[280px] font-mono text-xs"
+ bind:value={filters.traceId}
+ onchange={applyFilters}
+ onkeydown={handleInputKeydown}
+ />
+ </div>
+
+ <div class="flex flex-wrap items-center gap-2">
+ <button
+ type="button"
+ class="inline-flex items-center gap-1 rounded-full border border-dashed px-3 py-0.5 text-xs font-medium text-muted-foreground transition-colors hover:border-foreground/40 hover:text-foreground"
+ onclick={openFilterDialog}
+ disabled={loading}
+ >
+ <Plus class="h-3 w-3" />
+ Add attribute filter
+ </button>
+ {#each filters.attributes as filter (filter.key + '=' + filter.value)}
+ <span
+ class="inline-flex items-center gap-1 rounded-full bg-muted px-2 py-0.5 font-mono text-xs"
+ >
+ <span class="text-muted-foreground">{filter.key}</span>
+ <span>=</span>
+ <span>{filter.value}</span>
+ <button
+ type="button"
+ aria-label="Remove filter"
+ class="ml-1 cursor-pointer text-muted-foreground hover:text-foreground"
+ onclick={() => removeAttributeFilter(filter)}
+ >
+ <X class="h-3 w-3" />
+ </button>
+ </span>
+ {/each}
+ </div>
+
+ <AlertDialog.Root open={filterDialogOpen} onOpenChange={(open) => (filterDialogOpen = open)}>
+ <AlertDialog.Content>
+ <AlertDialog.Header>
+ <AlertDialog.Title>Add attribute filter</AlertDialog.Title>
+ <AlertDialog.Description>
+ Matches spans whose attribute has exactly this value. Keys are the OpenTelemetry names,
+ such as <code class="font-mono">http.request.method</code>.
+ </AlertDialog.Description>
+ </AlertDialog.Header>
+ <div class="flex flex-col gap-3">
+ <FormField label="Attribute key">
+ <Input
+ placeholder="http.request.method"
+ bind:value={dialogKey}
+ onkeydown={handleDialogKeydown}
+ />
+ </FormField>
+ <FormField label="Value">
+ <Input placeholder="GET" bind:value={dialogValue} onkeydown={handleDialogKeydown} />
+ </FormField>
+ {#if dialogError}
+ <p class="text-xs text-red-500">{dialogError}</p>
+ {/if}
+ </div>
+ <AlertDialog.Footer>
+ <Button variant="outline" onclick={() => (filterDialogOpen = false)}>Cancel</Button>
+ <Button onclick={submitFilterDialog}><Plus class="h-4 w-4" /> Add filter</Button>
+ </AlertDialog.Footer>
+ </AlertDialog.Content>
+ </AlertDialog.Root>
+
+ {#if guidance}
+ <WarningCallout title="Narrow the search">{guidance}</WarningCallout>
+ {/if}
+
+ <TableContainer minWidth="860px" empty={loading || !!error || spans.length === 0}>
+ <Table.Root>
+ {#if loading}
+ <Table.Body>
+ <Table.Row>
+ <Table.Cell colspan={7} class="h-48">
+ <div class="flex h-full items-center justify-center">
+ <LoadingCircle size="xlg" />
+ </div>
+ </Table.Cell>
+ </Table.Row>
+ </Table.Body>
+ {:else if error}
+ <Table.Body>
+ <Table.Row>
+ <Table.Cell colspan={7} class="h-24 text-center text-red-500">
+ {error}
+ </Table.Cell>
+ </Table.Row>
+ </Table.Body>
+ {:else if spans.length === 0}
+ <Table.Body>
+ <TableEmptyState
+ colspan={7}
+ message={guidance ? 'No results to show' : 'No spans match in this time range'}
+ />
+ </Table.Body>
+ {:else}
+ <Table.Header>
+ <Table.Row>
+ <TracewayTableHeader
+ label="Started"
+ tooltip="When the span started"
+ sortField="start_time"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[210px]"
+ />
+ <TracewayTableHeader label="Service" tooltip="The service.name of the resource" />
+ <TracewayTableHeader label="Span" tooltip="The span name" />
+ <TracewayTableHeader
+ label="Kind"
+ tooltip="Server, client, producer, consumer or internal"
+ class="w-[100px]"
+ />
+ <TracewayTableHeader label="Status" tooltip="The span status" class="w-[90px]" />
+ <TracewayTableHeader
+ label="Duration"
+ tooltip="How long the span took"
+ sortField="duration"
+ currentSortField={orderBy}
+ {sortDirection}
+ onSort={(field) => handleSort(field as SortField)}
+ class="w-[110px]"
+ />
+ <TracewayTableHeader
+ label="Trace"
+ tooltip="Start of the trace ID. Open the row to see the whole trace"
+ class="w-[110px]"
+ />
+ </Table.Row>
+ </Table.Header>
+ <Table.Body>
+ {#each spans as span (`${span.traceId}:${span.spanId}`)}
+ <Table.Row
+ class="cursor-pointer"
+ onclick={createRowClickHandler(spanTraceHref(span, { focusSpan: true }) ?? '')}
+ >
+ <Table.Cell class="text-sm whitespace-nowrap text-muted-foreground tabular-nums">
+ {formatDateTime(span.startTime, { timezone, format: 'datetime-seconds' })}
+ </Table.Cell>
+ <Table.Cell class="text-sm">{span.serviceName || '-'}</Table.Cell>
+ <Table.Cell class="max-w-[420px] truncate font-mono text-sm">
+ {#if isErrorSpan(span)}<CircleAlert
+ class="mr-1 inline h-3.5 w-3.5 align-[-2px] text-destructive"
+ aria-label="Error"
+ />{/if}{span.name}
+ </Table.Cell>
+ <Table.Cell class="text-sm text-muted-foreground">
+ {spanKindLabel(span.spanKind) || '-'}
+ </Table.Cell>
+ <Table.Cell class={isErrorSpan(span) ? 'text-sm text-destructive' : 'text-sm'}>
+ {spanStatusLabel(span.statusCode)}
+ </Table.Cell>
+ <Table.Cell class="font-mono text-sm tabular-nums">
+ {formatDuration(span.duration)}
+ </Table.Cell>
+ <Table.Cell class="font-mono text-xs text-muted-foreground">
+ {span.traceId.slice(0, 8)}
+ </Table.Cell>
+ </Table.Row>
+ {/each}
+ </Table.Body>
+ {/if}
+ </Table.Root>
+ </TableContainer>
+
+ <PaginationFooter
+ currentPage={page}
+ {totalPages}
+ {pageSize}
+ totalItems={total}
+ onPageChange={handlePageChange}
+ onPageSizeChange={handlePageSizeChange}
+ {loading}
+ itemLabel="span"
+ />
+</div>
diff --git a/frontend/src/routes/spans/+page.ts b/frontend/src/routes/spans/+page.ts
new file mode 100644
index 00000000..9f2ce0c4
--- /dev/null
+++ b/frontend/src/routes/spans/+page.ts
@@ -0,0 +1,9 @@
+import { redirect } from '@sveltejs/kit';
+import { authState } from '$lib/state/auth.svelte';
+import type { PageLoad } from './$types';
+
+export const load: PageLoad = () => {
+ if (!authState.isAuthenticated) {
+ throw redirect(302, '/login');
+ }
+};
diff --git a/frontend/src/routes/spans/[traceId]/+page.svelte b/frontend/src/routes/spans/[traceId]/+page.svelte
new file mode 100644
index 00000000..1393e3ae
--- /dev/null
+++ b/frontend/src/routes/spans/[traceId]/+page.svelte
@@ -0,0 +1,179 @@
+<script lang="ts">
+ import { untrack } from 'svelte';
+ import { resolve } from '$app/paths';
+ import { api } from '$lib/api';
+ import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
+ import { projectsState } from '$lib/state/projects.svelte';
+ import { getTimezone } from '$lib/state/timezone.svelte';
+ import { formatDateTime, formatDuration } from '$lib/utils/formatters';
+ import { createSmartBackHandler } from '$lib/utils/back-navigation';
+ import { summarizeSpanTrace } from '$lib/utils/span-explorer';
+ import * as Card from '$lib/components/ui/card';
+ import { LoadingCircle } from '$lib/components/ui/loading-circle';
+ import { ErrorDisplay } from '$lib/components/ui/error-display';
+ import CopyButton from '$lib/components/traceway/copy-button.svelte';
+ import PageHeader from '$lib/components/traceway/page-header.svelte';
+ import SpanWaterfall from '$lib/components/spans/span-waterfall.svelte';
+ import SpanGraphNotice from '$lib/components/spans/span-graph-notice.svelte';
+ import TraceLogsPanel from '$lib/components/trace-logs/trace-logs-panel.svelte';
+ import type { Span, SpanAttributes, SpanGraphStatus } from '$lib/types/spans';
+
+ let { data } = $props();
+
+ type SpanTraceResponse = {
+ spanGraphStatus?: SpanGraphStatus | null;
+ spans: Span[] | null;
+ projects?: { id: string; name: string }[] | null;
+ };
+
+ const timezone = $derived(getTimezone());
+ const onBack = createSmartBackHandler({ fallbackPath: resolve('/spans') });
+
+ // Raw on purpose: a deep proxy over 20,000 spans took minutes to read. The array is only ever replaced.
+ let spans = $state.raw<Span[]>([]);
+ let status = $state<SpanGraphStatus | null>(null);
+ // The trace is read from every project of the organization, so a row may live in another project than this page.
+ let projectNames = $state.raw<Record<string, string>>({});
+ const projectCount = $derived(Object.keys(projectNames).length);
+ let loading = $state(true);
+ let error = $state('');
+ let notFound = $state(false);
+ let generation = 0;
+
+ const summary = $derived(summarizeSpanTrace(spans));
+
+ const truncated = $derived(!!status?.reasons?.includes('most_important'));
+
+ function loadAttributes(span: Span): Promise<SpanAttributes> {
+ return api.get(
+ `/spans/traces/${data.traceId}/spans/${span.spanId}/attributes?at=${encodeURIComponent(span.recordedAt)}`,
+ { projectId: span.projectId }
+ );
+ }
+
+ async function loadData() {
+ const requestGeneration = ++generation;
+ loading = true;
+ error = '';
+ notFound = false;
+ spans = [];
+ status = null;
+ projectNames = {};
+ if (!data.at) {
+ notFound = true;
+ loading = false;
+ return;
+ }
+ try {
+ const response = (await api.get(
+ `/spans/traces/${data.traceId}?at=${encodeURIComponent(data.at)}`,
+ { projectId: projectsState.currentProjectId ?? undefined }
+ )) as SpanTraceResponse;
+ if (requestGeneration !== generation) return;
+ spans = response.spans || [];
+ status = response.spanGraphStatus ?? null;
+ projectNames = Object.fromEntries((response.projects ?? []).map((p) => [p.id, p.name]));
+ } catch (e) {
+ if (requestGeneration !== generation) return;
+ console.error(e);
+ if (getErrorStatus(e) === 404) {
+ notFound = true;
+ } else {
+ error = getErrorMessage(e) || 'Failed to load the trace';
+ }
+ } finally {
+ if (requestGeneration === generation) loading = false;
+ }
+ }
+
+ $effect(() => {
+ const identity = [data.traceId, data.at, projectsState.currentProjectId];
+ if (identity[0]) untrack(loadData);
+ return () => {
+ generation++;
+ };
+ });
+</script>
+
+<div class="space-y-6">
+ <PageHeader title="Trace" {onBack}>
+ {#snippet meta()}
+ <div class="flex items-center gap-1 font-mono text-sm text-muted-foreground">
+ <span class="break-all">{data.traceId}</span>
+ <CopyButton bare text={data.traceId} iconClass="h-3.5 w-3.5" class="shrink-0" />
+ </div>
+ {/snippet}
+ </PageHeader>
+
+ {#if loading}
+ <div class="flex items-center justify-center py-20">
+ <LoadingCircle size="xlg" />
+ </div>
+ {:else if notFound}
+ <ErrorDisplay
+ status={404}
+ title="Trace Not Found"
+ description="No spans of this trace were recorded within 24 hours of the linked time. It may have expired, or the link may be incomplete."
+ {onBack}
+ backLabel="Back to Spans"
+ onRetry={loadData}
+ identifier={data.traceId}
+ />
+ {:else if error}
+ <ErrorDisplay
+ status={400}
+ title="Failed to Load Trace"
+ description={error}
+ {onBack}
+ backLabel="Back to Spans"
+ onRetry={loadData}
+ />
+ {:else}
+ <Card.Root>
+ <Card.Header>
+ <Card.Title>Spans</Card.Title>
+ <Card.Description>
+ {spans.length.toLocaleString()}
+ {spans.length === 1 ? 'span' : 'spans'} across {summary.services}
+ {summary.services === 1 ? 'service' : 'services'}{projectCount > 1
+ ? ` in ${projectCount} projects`
+ : ''}, {formatDuration(summary.duration)} end to end{#if summary.startTime}, started {formatDateTime(
+ summary.startTime,
+ {
+ timezone,
+ format: 'datetime-seconds'
+ }
+ )}{/if}{#if summary.errors > 0}<span class="text-destructive"
+ >, {summary.errors}
+ {summary.errors === 1 ? 'error' : 'errors'}</span
+ >{/if}
+ </Card.Description>
+ </Card.Header>
+ <Card.Content class="space-y-3">
+ <SpanGraphNotice {status} />
+ {#if spans.length > 0}
+ <SpanWaterfall
+ {spans}
+ traceDuration={summary.duration}
+ traceStartTime={summary.startTime}
+ linkTraces={false}
+ {loadAttributes}
+ {projectNames}
+ missingParentLabel={truncated ? 'Parent span is not among the spans shown' : undefined}
+ selectedSpanId={data.spanId}
+ />
+ {/if}
+ </Card.Content>
+ </Card.Root>
+
+ {#if spans.length > 0}
+ <TraceLogsPanel
+ wholeTrace
+ projectId={projectsState.currentProjectId ?? ''}
+ traceId={data.traceId}
+ {spans}
+ traceRecordedAt={data.at ?? spans[0]?.recordedAt ?? summary.startTime}
+ />
+ {/if}
+ {/if}
+</div>
diff --git a/frontend/src/routes/spans/[traceId]/+page.ts b/frontend/src/routes/spans/[traceId]/+page.ts
new file mode 100644
index 00000000..6ebbcbb7
--- /dev/null
+++ b/frontend/src/routes/spans/[traceId]/+page.ts
@@ -0,0 +1,17 @@
+import { redirect } from '@sveltejs/kit';
+import { authState } from '$lib/state/auth.svelte';
+import type { PageLoad } from './$types';
+
+export const prerender = false;
+
+export const load: PageLoad = ({ params, url }) => {
+ if (!authState.isAuthenticated) {
+ throw redirect(302, '/login');
+ }
+
+ return {
+ traceId: params.traceId.toLowerCase(),
+ at: url.searchParams.get('at'),
+ spanId: url.searchParams.get('span')
+ };
+};
diff --git a/frontend/src/routes/spans/span-trace.test.ts b/frontend/src/routes/spans/span-trace.test.ts
new file mode 100644
index 00000000..9a8c9bd4
--- /dev/null
+++ b/frontend/src/routes/spans/span-trace.test.ts
@@ -0,0 +1,63 @@
+import { afterEach, beforeEach, expect, it, vi } from 'vitest';
+import { cleanup, fireEvent, render } from '@testing-library/svelte';
+
+const { get } = vi.hoisted(() => ({ get: vi.fn() }));
+vi.mock('$lib/api', () => ({ api: { get } }));
+vi.mock('$lib/state/projects.svelte', () => ({
+ projectsState: { currentProjectId: 'project', currentProject: null },
+ isFrontendFramework: () => false
+}));
+vi.mock('$lib/state/timezone.svelte', () => ({ getTimezone: () => 'UTC' }));
+
+import SpanTracePage from '../../test/span-trace-page.svelte';
+
+beforeEach(() => {
+ vi.stubGlobal(
+ 'ResizeObserver',
+ class {
+ observe() {}
+ unobserve() {}
+ disconnect() {}
+ }
+ );
+});
+
+afterEach(() => {
+ cleanup();
+ vi.clearAllMocks();
+ vi.unstubAllGlobals();
+});
+
+it.each(['1970-01-01T00:00:00Z', '2554-07-21T23:34:33.709551615Z'])(
+ 'loads attributes using the stored time when the source start is %s',
+ async (startTime) => {
+ const recordedAt = '2026-09-20T12:34:56Z';
+ get.mockResolvedValueOnce({
+ spans: [
+ {
+ projectId: 'project',
+ traceId: 'abc',
+ spanId: '01',
+ name: 'fallback span',
+ startTime,
+ recordedAt,
+ duration: 1_000_000
+ }
+ ]
+ });
+ get.mockResolvedValueOnce({ attributes: { 'test.case': 'reopened' } });
+ const { findByText } = render(SpanTracePage, {
+ data: { traceId: 'abc', at: recordedAt, spanId: null }
+ });
+ await fireEvent.click(await findByText('fallback span'));
+ expect(await findByText('reopened')).toBeTruthy();
+ expect(get).toHaveBeenNthCalledWith(1, '/spans/traces/abc?at=2026-09-20T12%3A34%3A56Z', {
+ projectId: 'project'
+ });
+ expect(get).toHaveBeenNthCalledWith(
+ 2,
+ '/spans/traces/abc/spans/01/attributes?at=2026-09-20T12%3A34%3A56Z',
+ { projectId: 'project' }
+ );
+ }
+);
diff --git a/frontend/src/routes/tasks/[task]/+page.svelte b/frontend/src/routes/tasks/[task]/+page.svelte
index 1704d56e..f39110fd 100644
--- a/frontend/src/routes/tasks/[task]/+page.svelte
+++ b/frontend/src/routes/tasks/[task]/+page.svelte
@@ -1,6 +1,6 @@
<script lang="ts">
import { getErrorMessage, getErrorStatus } from '$lib/utils/errors';
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { api } from '$lib/api';
import {
formatDuration,
@@ -161,7 +161,10 @@
loadData();
}
+ let loadSequence = 0;
+
async function loadData(pushToHistory = true) {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -187,11 +190,13 @@
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
tasks = response.data || [];
stats = response.stats || null;
total = response.pagination.total;
totalPages = response.pagination.totalPages;
} catch (e) {
+ if (sequence !== loadSequence) return;
console.error(e);
errorStatus = getErrorStatus(e) || 0;
if (getErrorStatus(e) === 404) {
@@ -200,7 +205,7 @@
error = getErrorMessage(e) || 'Failed to load data';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
@@ -226,8 +231,24 @@
loadData(false);
}
- onMount(() => {
- loadData(false);
+ $effect(() => {
+ void data.task;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ tasks = [];
+ stats = null;
+ page = 1;
+ const range = getInitialRange();
+ selectedPreset = range.preset;
+ fromDate = dateToCalendarDate(range.from, timezone);
+ toDate = dateToCalendarDate(range.to, timezone);
+ fromTime = dateToTimeString(range.from, timezone);
+ toTime = dateToTimeString(range.to, timezone);
+ loadData(false);
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
diff --git a/frontend/src/routes/tasks/[task]/[taskId]/+page.svelte b/frontend/src/routes/tasks/[task]/[taskId]/+page.svelte
index 2b6f89b5..a42cca9e 100644
--- a/frontend/src/routes/tasks/[task]/[taskId]/+page.svelte
+++ b/frontend/src/routes/tasks/[task]/[taskId]/+page.svelte
@@ -1,5 +1,5 @@
<script lang="ts">
- import { onMount } from 'svelte';
+ import { untrack } from 'svelte';
import { goto } from '$app/navigation';
import { api } from '$lib/api';
import { formatDuration, formatDateTime } from '$lib/utils/formatters';
@@ -14,6 +14,7 @@
import { LabelValue } from '$lib/components/ui/label-value';
import { AttributesGrid } from '$lib/components/ui/attributes-grid/index.js';
import SpanWaterfall from '$lib/components/spans/span-waterfall.svelte';
+ import SpanGraphNotice from '$lib/components/spans/span-graph-notice.svelte';
import SpanEmptyState from '$lib/components/spans/span-empty-state.svelte';
import PageHeader from '$lib/components/traceway/page-header.svelte';
import TableContainer from '$lib/components/traceway/table-container.svelte';
@@ -22,11 +23,11 @@
import { resolve } from '$app/paths';
import DistributedTraceCard from '$lib/components/distributed-trace/distributed-trace-card.svelte';
import TraceLogsPanel from '$lib/components/trace-logs/trace-logs-panel.svelte';
- import { logTraceId } from '$lib/utils/span-id';
- import type { Span } from '$lib/types/spans';
+ import type { Span, SpanGraphStatus, TraceIdentity } from '$lib/types/spans';
type TaskDetailResponse = {
- task: {
+ spanGraphStatus?: SpanGraphStatus;
+ task: TraceIdentity & {
id: string;
taskName: string;
duration: number;
@@ -35,8 +36,6 @@
attributes: Record<string, string> | null;
serverName: string;
appVersion: string;
- distributedTraceId?: string;
- spanId?: string;
};
exception?: {
exceptionHash: string;
@@ -57,12 +56,16 @@
const timezone = $derived(getTimezone());
- let response = $state<TaskDetailResponse | null>(null);
+ // Raw on purpose: the response can carry 20,000 spans and is only ever replaced, never edited.
+ let response = $state.raw<TaskDetailResponse | null>(null);
let loading = $state(true);
let error = $state('');
let notFound = $state(false);
+ let loadSequence = 0;
+
async function loadData() {
+ const sequence = ++loadSequence;
loading = true;
error = '';
notFound = false;
@@ -73,8 +76,10 @@
data.recordedAt ? { recordedAt: data.recordedAt } : {},
{ projectId: projectsState.currentProjectId ?? undefined }
);
+ if (sequence !== loadSequence) return;
response = result;
} catch (e: unknown) {
+ if (sequence !== loadSequence) return;
console.error(e);
const err = e as { status?: number; message?: string };
if (err.status === 404) {
@@ -83,12 +88,21 @@
error = err.message || 'Failed to load task details';
}
} finally {
- loading = false;
+ if (sequence === loadSequence) loading = false;
}
}
- onMount(() => {
- loadData();
+ $effect(() => {
+ void data.taskId;
+ void data.recordedAt;
+ void projectsState.currentProjectId;
+ untrack(() => {
+ response = null;
+ loadData();
+ });
+ return () => {
+ loadSequence++;
+ };
});
</script>
@@ -262,15 +276,17 @@
{/if}
</Card.Description>
</Card.Header>
- <Card.Content>
+ <Card.Content class="space-y-3">
+ <SpanGraphNotice status={response.spanGraphStatus} />
{#if response.hasSpans}
<SpanWaterfall
spans={response.spans}
traceDuration={response.task.duration}
traceStartTime={response.task.recordedAt}
+ traceId={response.task.traceId}
rootSpanId={response.task.spanId}
/>
- {:else}
+ {:else if !response.spanGraphStatus || response.spanGraphStatus.state === 'complete'}
<SpanEmptyState framework={projectsState.currentProject?.framework ?? 'gin'} />
{/if}
</Card.Content>
@@ -278,15 +294,15 @@
<TraceLogsPanel
projectId={projectsState.currentProjectId ?? ''}
- traceId={logTraceId(response.task)}
- distributedTraceId={response.task.distributedTraceId ?? null}
+ traceId={response.task.traceId}
spans={response.spans ?? []}
+ rootSpan={{ spanId: response.task.spanId, name: response.task.taskName }}
traceRecordedAt={response.task.recordedAt}
/>
- {#if response.task.distributedTraceId}
+ {#if response.task.traceId}
<DistributedTraceCard
- distributedTraceId={response.task.distributedTraceId}
+ traceId={response.task.traceId}
currentNodeId={response.task.id}
recordedAt={response.task.recordedAt}
/>
diff --git a/frontend/src/test/span-trace-page.svelte b/frontend/src/test/span-trace-page.svelte
new file mode 100644
index 00000000..d6872b4c
--- /dev/null
+++ b/frontend/src/test/span-trace-page.svelte
@@ -0,0 +1,10 @@
+<script lang="ts">
+ import * as Tooltip from '$lib/components/ui/tooltip';
+ import SpanTracePage from '../routes/spans/[traceId]/+page.svelte';
+
+ let { data }: { data: { traceId: string; at: string; spanId: string | null } } = $props();
+</script>
+
+<Tooltip.Provider>
+ <SpanTracePage {data} />
+</Tooltip.Provider>
diff --git a/skills/traceway-setup/data-model.md b/skills/traceway-setup/data-model.md
index d2e2c72c..2567c799 100644
--- a/skills/traceway-setup/data-model.md
+++ b/skills/traceway-setup/data-model.md
@@ -32,20 +32,20 @@ Traceway turns OTel spans into five distinct concepts, each with its own dashboa
For every incoming span, Traceway applies these rules in order:
-1. **Endpoint**: `SpanKind` is `SERVER` or `INTERNAL`, the span has at least one HTTP attribute (`http.request.method`, `http.method`, `http.route`, or `url.path`), AND it is either a root span (no parent) or its parent span is not present in the same export batch (cross-process tracing, e.g. behind a proxy that injects `traceparent`). See the carve-out below.
+1. **Endpoint**: `SERVER` with HTTP attributes and no observed local HTTP SERVER ancestor before a remote/CLIENT/PRODUCER boundary; alternatively a root `INTERNAL` span with HTTP attributes. A missing parent cannot establish an HTTP ancestor. HTTP attributes are `http.request.method`, `http.method`, `http.route`, or `url.path`. See the carve-out below.
2. **Task**: `SpanKind` is `CONSUMER`. This applies to ANY consumer span, root or not.
3. **Task**: a root `INTERNAL` span with a `console.command` attribute (CLI command instrumentation, e.g. Laravel/Symfony console).
4. **AI Trace**: the span has any attribute starting with `gen_ai.`.
-5. **Child span**: anything else with a parent, stored as a generic span and attached to the nearest promoted ancestor (Endpoint/Task/AI Trace) by walking up the parent chain.
-6. **Dropped**: anything else that is a root span. A root span that matches none of the rules above is silently discarded: no entity row, no span row. One exception: if the span carries an exception signal (an `exception` event or `exception.*` attributes), the exception is still captured as an Issue, linked by the raw OTel trace ID. Only the span itself is lost.
+5. **Span only**: anything else. The span is stored with the trace id, span id and parent span id it arrived with, and that is all. It shows in the waterfall of whichever Endpoint, Task or AI Trace sits above it in its trace, which Traceway works out when the page is opened, not at ingest. So the order spans arrive in does not matter.
+6. **No page of its own**: a root span that matches none of the rules above gets no Endpoint, Task or AI Trace row. The span itself is still stored, and you can find it in the span explorer. If it carries an exception signal (an `exception` event or `exception.*` attributes), the exception becomes an Issue that records the span's trace id and span id.
One carve-out is checked before all six rules: an `INTERNAL` span carrying `exception.*` **attributes** (`exception.type`, `exception.message` or `exception.stacktrace` set as span attributes, not as an `exception` event) is never promoted to anything. Not an Endpoint, not an AI Trace, not a `console.command` Task. The reason is browser SDKs (e.g. Honeycomb's), which stamp page context like `url.path` onto their zero-duration error spans, and those must become Issues, not endpoints. The consequence beyond browsers: do not set `exception.*` attributes on an `INTERNAL` span you also want promoted, or you lose the entity row and keep only the Issue. An agent that stamps `exception.type` onto its own `INTERNAL` `gen_ai.*` span loses the AI Trace row this way and gets only an Issue. Record the error as an `exception` event instead (`span.recordException(error)`), which does not suppress promotion.
-A note on "the same export batch": Traceway resolves parents inside one `ResourceSpans` block, not across the whole request. Spans from one service normally share a Resource and land in one block, so this is invisible. But if an exporter splits a trace across Resource blocks, the children in the other block are treated as cross-process. They link by raw OTel trace ID, and a `SERVER`/`INTERNAL` child carrying HTTP attributes is promoted to a second Endpoint.
+A note on export batches: ingest looks at one span at a time. The one check that looks further is the nested server span rule (a `SERVER` span under another local `SERVER` span is one request, not two), and it only sees the spans of the same `ResourceSpans` block. When a parent is absent its kind is unknown, even when flags say it is local, so the HTTP SERVER span is counted as an entry point. Separately exported nested servers can therefore create extra projections.
-The consequences of rule 6 are the most common integration bug: a custom root span created with `tracer.startActiveSpan("my-job")` and default `SpanKind.INTERNAL` produces nothing on the Endpoints or Tasks pages (errors recorded on it do still reach Issues, but with no Endpoint/Task to link back to). Background work must use `SpanKind.CONSUMER` (see Tasks below).
+The consequences of rule 6 are the most common integration bug: a custom root span created with `tracer.startActiveSpan("my-job")` and default `SpanKind.INTERNAL` produces nothing on the Endpoints or Tasks pages. It is only visible in the span explorer, and errors recorded on it reach Issues with no Endpoint or Task to link back to. Background work must use `SpanKind.CONSUMER` (see Tasks below).
-One project-level gate sits above all of these rules: a project whose framework is one of the five browser SDK types (`react`, `svelte`, `vuejs`, `jquery`, `react-native`) never promotes OTel spans to Endpoints/Tasks/AI Traces, and stores no child span rows either. Only exceptions are extracted. Server-side JS frameworks (`nextjs`, `nestjs`, `express`, `remix`) and the mobile frameworks (`ios`, `android`, `flutter`) are NOT gated. Backend OTel exporters must point at a backend-framework project token.
+One project-level gate sits above all of these rules: a project whose framework is one of the five browser SDK types (`react`, `svelte`, `vuejs`, `jquery`, `react-native`) never promotes OTel spans to Endpoints/Tasks/AI Traces. Its spans are still stored and show in the span explorer, and its exceptions become Issues. Server-side JS frameworks (`nextjs`, `nestjs`, `express`, `remix`) and the mobile frameworks (`ios`, `android`, `flutter`) are NOT gated. Backend OTel exporters must point at a backend-framework project token.
## Endpoints: Name and Route Parameters
@@ -131,8 +131,8 @@ Quirks:
- **The span name IS the task name, and Traceway groups tasks by name.** Use a stable identifier like `cleanup-expired-sessions` or `process-email-queue`. Never put job IDs, timestamps, or user IDs in the span name. Each unique name becomes a separate task group.
- **Every `CONSUMER` span becomes a Task, even non-root ones.** If a queue library's auto-instrumentation already emits `CONSUMER` spans (e.g. Kafka/RabbitMQ consumers, Symfony Messenger), do NOT wrap them in another `CONSUMER` span. You'd get duplicate Task entries.
-- A root span with `SpanKind.INTERNAL` (the default!) is **dropped silently**. This is the most common reason "my cron job doesn't show up". The kind must be `CONSUMER`. (An exception recorded on the dropped span still reaches Issues, but the task run itself is lost: name, duration, history.)
-- **Tasks are timestamped when they finish, endpoints when they start.** A task run lands in the time bucket of its end time, so a nightly job triggered at 01:00 and finishing at 03:00 shows up at 03:00. Widen the time range when a long-running job seems missing.
+- A root span with `SpanKind.INTERNAL` (the default!) is **stored as a span without a Task projection**. This is the most common reason "my cron job doesn't show up". The kind must be `CONSUMER`. (An exception recorded on the dropped span still reaches Issues, but there is no task-specific aggregation; span name and duration remain available in the explorer.)
+- **Tasks and endpoints are timestamped when they start in V2.** A job starting at 01:00 and finishing at 03:00 appears at 01:00. Widen the time range when a long-running job seems missing.
- Per-job context (job ID, batch size) belongs in span **attributes**, where it shows on the task detail page without affecting grouping.
## Database Queries and Child Spans
@@ -142,9 +142,9 @@ Work inside an endpoint or task (DB queries, cache hits, outgoing HTTP calls, cu
How Traceway handles them:
- **The span name is replaced by the SQL text when present**: if the span has `db.query.text` (current semconv) or `db.statement` (old semconv), that value is displayed instead of the span name. So a `pg.query` span carrying `db.statement = "SELECT * FROM users WHERE id = $1"` shows the query itself in the waterfall. This is what makes auto-instrumented DB clients (`pg`, `mysql2`, `mongodb`, `ioredis`, Prisma via `@prisma/instrumentation`) useful out of the box.
-- **Child spans attach to the nearest promoted ancestor.** The parent chain is walked until an Endpoint/Task/AI Trace is found. This means DB spans must be created **inside the active context** of the request/task span (`startActiveSpan`, or the framework middleware's context). A DB span created outside any active context becomes a root `INTERNAL` span, which is dropped.
-- **A promoted span is never also a child span.** Once a span becomes an Endpoint, Task or AI Trace it leaves the waterfall of whatever contains it, and its own children re-root under it. So a `CONSUMER` span created inside a request handler (an inline queue publish, a Symfony Messenger sync dispatch) disappears from that endpoint's Spans tab and shows up on the Tasks page instead, with its sub-spans attached to the Task. If you want the work to stay in the request waterfall, use a plain child span (default `INTERNAL`), not `CONSUMER`.
-- Spans whose parent lives in another process/batch and never resolves to a promoted entity are kept but linked by the raw OTel trace ID instead.
+- **A waterfall is the spans below the entry point.** An Endpoint, Task or AI Trace page shows every span of its trace that sits below its own span, in the same project. This means DB spans must be created **inside the active context** of the request/task span (`startActiveSpan`, or the framework middleware's context). A DB span created outside any active context becomes a root `INTERNAL` span of a trace of its own, and no page shows it except the span explorer.
+- **A promoted span stays in its trace.** A span that becomes an Endpoint, Task or AI Trace is still an ordinary span of its trace. So a `CONSUMER` span created inside an HTTP request gets its own Task page, and it also appears, with its children, in the waterfall of the Endpoint that started it when both report to the same project. Across projects each page shows its own project's spans, and the Distributed Trace card links the pages.
+- A span whose parent never arrives (a collector dropped it, say) is shown on the page of the trace's root entry point, marked "Parent span is unavailable". It takes its proper place as soon as the parent is stored.
- For databases without auto-instrumentation (e.g. SQLite), create a manual child span and set `db.system` + `db.statement` attributes so it renders like the auto-instrumented ones.
## Issues: Exceptions
@@ -169,8 +169,8 @@ An `exception.type: exception.message` header is prepended unless the stack trac
Quirks:
-- **Exceptions survive span-dropping, but their context doesn't.** An exception recorded on an unpromoted root span (see classification rule 6) still becomes an Issue, but there is no Endpoint/Task for it to link to. The occurrence carries only the raw trace ID. Such an occurrence is stored with trace type `task`, which is why orphaned errors look task-shaped in the UI. Record exceptions on spans that live inside an Endpoint/Task, or fix the span kind, so Issues stay correlated with the request or job that raised them.
-- **`exception.stacktrace` is stripped from every stored attribute map.** The raw blob lives in the Issue's stack trace field only, never duplicated onto the endpoint, task, span or exception rows. Don't re-add it under another key. For JS and Android telemetry the exception's attribute map additionally carries `telemetry.sdk.language`, copied down from the Resource.
+- **An exception without an entry point has nothing to link to.** An exception recorded on a root span that matched no rule (see classification rule 6) still becomes an Issue, but there is no Endpoint or Task above it. The occurrence shows the trace id, and the span is in the span explorer, but the Related Endpoint section is empty. An exception on any span below an Endpoint or Task is linked to it, even when the two arrived in different exports.
+- **`exception.stacktrace` is stripped from every stored attribute map.** The readable attribute maps omit the blob; the Issue keeps the symbolicated stack trace, while the preserved OTLP payload also retains the original source stack trace. Don't re-add it under another key. For JS and Android telemetry the exception's attribute map additionally carries `telemetry.sdk.language`, copied down from the Resource.
- **Grouping is automatic.** Before hashing (SHA-256, truncated to 16 hex chars), Traceway normalizes the stack trace: error messages are stripped (only the error type is kept, including on `Caused by:` lines), JS function-name lines collapse to `<fn>`, URL origins are removed so the same bundle served from different hosts or CDNs groups together, absolute paths reduce to `filename:line`, column numbers are dropped from frames on line 2 or later (the column survives only on line 1, where minified bundles put everything and it is the only thing telling frames apart), dependency version suffixes (`@v1.2.3`) are removed, and runtime values are replaced with placeholders: hex addresses -> `<hex>`, UUIDs -> `<uuid>`, standalone numbers of 5+ digits -> `<id>` (never a number preceded by `:`, so `handler.go:123456` line numbers stay significant), emails -> `<email>`, IPs -> `<ip>`, goroutine IDs -> `goroutine <n>`. JVM frames additionally lose their line numbers (`(Foo.java:123)` -> `(Foo.java)`, same for `.kt` and `.scala`) and `... N more` becomes `... more`. The same logical error therefore groups into one Issue even when runtime values differ, so don't try to pre-group errors client-side.
- **JS projects get source-map symbolication, once the project has a source-map token.** Symbolication is triggered when the `telemetry.sdk.language` resource attribute is a JS value (`nodejs`, `webjs`, `javascript` or `typescript`, all set automatically by OTel SDKs), or when the instrumentation scope name is npm-scoped (`@scope/pkg`) or `next.js`. It is a silent no-op until the project has a source-map token: the trace is canonicalized and stored unresolved, with no error and no warning, and uploaded maps are never consulted. Only iOS projects get a token automatically, so for an OpenTelemetry project generate one with `POST /api/projects/source-map-token` first. Minified frames are resolved server-side **before** grouping, so uploading source maps improves grouping too.
- **Source maps are matched by filename (and debug ID when embedded), not by version.** Upload via `POST /api/sourcemaps/upload` (multipart `files` field, `.map`/`.js`/`.cjs`/`.mjs`, max 50 MB per file) using a token from `POST /api/projects/source-map-token`. A frame referencing `app.min.js` resolves against the uploaded `app.min.js.map` by name. Keep bundle filenames unique per release (content hashes do this) or rely on debug IDs.
@@ -244,7 +244,6 @@ Traceway recognizes these non-standard span attributes:
| Attribute | Type | Purpose |
|---|---|---|
| `traceway.is_stream` | boolean | Mark an endpoint span as streaming (SSE/long-poll) so its duration is excluded from latency percentiles. |
-| `traceway.distributed_trace_id` | string (UUID) | Override the distributed trace ID used to link entities across services. Rarely needed. The OTel trace ID is used by default. |
## Verification Checklist
@@ -264,3 +263,7 @@ traceway exceptions list --since 1h
traceway logs query --since 1h --min-severity 17
traceway metrics query --name <metric-name> --since 1h
```
+
+Canonical span storage is subject to permission and healthcheck filtering and rejected-row handling. Graphs are limited by authorized projects, a lookup time window, row count and UTF-8 attribute byte budgets; arrival order does not remove those limits.
+
+Browser/server correlation uses standard W3C trace context and the same `traceId`. Span links retain their target trace and span IDs without merging traces.
diff --git a/skills/traceway-setup/frontend-js.md b/skills/traceway-setup/frontend-js.md
index e94f73bc..cc9b63ea 100644
--- a/skills/traceway-setup/frontend-js.md
+++ b/skills/traceway-setup/frontend-js.md
@@ -209,7 +209,7 @@ clearAttributes();
#### Backend side
-The browser id only reaches Traceway if the backend puts it on its server span. Traceway reads the span attribute `traceway.distributed_trace_id` and uses it as the row's distributed trace id, overriding the one derived from the OTel trace id. That override is what joins the browser exception and the backend endpoint into one trace. The value must be a bare UUID; anything else is ignored silently.
+The browser id only reaches Traceway if the backend puts it on its server span. Traceway reads the span attribute `traceway.distributed_trace_id` and stores it on the row as its linked trace id, next to the row's own OTel trace id. That link is what joins the browser exception and the backend endpoint into one trace, whichever of the two ids you open it by. The value must be a bare UUID (or the same 32 hex characters without dashes); anything else is ignored silently.
Only the Symfony bundle sets the attribute for you. Node, NestJS, Next.js, Hono, Cloudflare, Laravel and Django install vanilla OpenTelemetry, which has never heard of the header, so add one middleware:
diff --git a/skills/traceway/SKILL.md b/skills/traceway/SKILL.md
index ca0eaae3..fe8cd19c 100644
--- a/skills/traceway/SKILL.md
+++ b/skills/traceway/SKILL.md
@@ -43,7 +43,7 @@ Users paste dashboard URLs (`https://<instance>/<route>`) as references in any f
| `/tasks/<task>/<taskId>` | Single task run | `traceway tasks show <taskId> --recorded-at <t>` (`t` = the URL's `?t=` param) |
| `/sessions/<sessionId>` | Session (the exceptions that fired during it; replay stays dashboard-only) | `traceway sessions show <sessionId> --started-at <t>`. The URL has no `?t=`; use the session's start, the URL's `from=`, or a linked occurrence's `recordedAt` (it falls inside the window). Occurrences reference sessions via their `sessionId` |
| `/ai-traces/<traceName>` | AI trace group | No CLI for the group; for one trace use the next row |
-| `/ai-traces/<traceName>/<traceId>` | Single AI trace | `traceway ai-traces show <traceId> --recorded-at <t>` (`t` = the URL's `?t=` param); returns token/cost stats + the conversation |
+| `/ai-traces/<traceName>/<traceId>` | Single AI trace | `traceway ai-traces show <aiTraceId> --recorded-at <t>` (the UUID in the URL, not a 32 hex trace id) (`t` = the URL's `?t=` param); returns token/cost stats + the conversation |
| `/logs` | Logs page (its filters are not stored in the URL) | `traceway logs query` with flags taken from the user's description |
| `/issues`, `/endpoints`, `/metrics`, `/` | List and dashboard pages | The matching `list` / `query` command |
@@ -92,7 +92,7 @@ So from a notification, go straight to the occurrence (fast), then pivot reusing
```bash
traceway exceptions occurrence <Exception ID> --recorded-at <Occurred at → RFC3339> --output json
-# the result carries distributedTraceId and sessionId → traces show / sessions show below
+# the result carries traceId, relatedEntity and sessionId → traces show / endpoints show / sessions show below
```
### Performance notification (an endpoint became slow or critical)
@@ -214,7 +214,7 @@ traceway exceptions list --since 7d --search "checkout" --output json \
traceway exceptions show <hash>
```
-This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `distributedTraceId` / `sessionId` per occurrence. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.
+This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `traceId` / `spanId` / `sessionId` per occurrence. `traceId` is the OpenTelemetry trace id (32 hex characters), the same id the request's logs and spans carry. The response also has `relatedEntity`: the endpoint, task or AI trace the newest occurrence happened in, with the `id` and `recordedAt` that `endpoints show` / `tasks show` / `ai-traces show` take. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.
**When the user gave an issue URL (or hash), fix the LAST occurrence — not "the group".** A single hash can bundle *several distinct errors*: the hash is computed from a normalized stack trace with the message stripped, so two unrelated failures that share their top frames (e.g. both captured at the same middleware/recovery frame) collapse into one group. The group's representative stack trace and `firstSeen` may belong to a different, now-dormant error than the one the user is looking at. Anchor on the most recent occurrence and fix that specific failure path:
@@ -222,7 +222,7 @@ This is the high-value call: full stack trace, occurrence list with `recordedAt`
# The occurrence the user actually wants: the latest one. Pin its exact message + attributes + trace.
traceway exceptions show <hash> --output json \
| jq '.occurrences | sort_by(.recordedAt) | last
- | {recordedAt, message: (.stackTrace | split("\n")[0]), attributes, traceId, distributedTraceId}'
+ | {recordedAt, message: (.stackTrace | split("\n")[0]), attributes, traceId, spanId}'
# Then confirm whether the group is homogeneous or mixed — distinct first lines = distinct bugs:
traceway exceptions show <hash> --output json \
@@ -249,7 +249,7 @@ Severity is an OTel number, not a name: 1 TRACE, 5 DEBUG, 9 INFO, 13 WARN, 17 ER
**Correlate by trace**: when an occurrence or log line carries a trace ID, pull the whole request timeline; this is usually the fastest route to a root cause:
```bash
-traceway exceptions show $HASH --output json | jq -r '.occurrences[0].distributedTraceId' \
+traceway exceptions show $HASH --output json | jq -r '.occurrences[0].traceId' \
| xargs -I{} traceway logs query --trace-id {} --output json
```
@@ -258,13 +258,13 @@ Pull the whole cross-service trace and the user's session, reusing the occurrenc
```bash
OCC=$(traceway exceptions show $HASH --output json | jq -c '.occurrences[0]')
TS=$(jq -r '.recordedAt' <<<"$OCC")
-DT=$(jq -r '.distributedTraceId // empty' <<<"$OCC")
+DT=$(jq -r '.traceId // empty' <<<"$OCC")
SID=$(jq -r '.sessionId // empty' <<<"$OCC")
-[ -n "$DT" ] && traceway traces show "$DT" --recorded-at "$TS" # every endpoint/task/ai-trace/exception node across services
+[ -n "$DT" ] && traceway traces show "$DT" --recorded-at "$TS" # available endpoint/task/ai-trace/exception nodes across services
[ -n "$SID" ] && traceway sessions show "$SID" --started-at "$TS" # the session + the exceptions that fired in it
```
-`traces show` is usually the single highest-value RCA call: it stitches one logical request together end to end across services.
+`traces show` connects available nodes across services. Inspect each node's `spanGraphStatus` (the `GRAPH` column in table output): time windows, retention and read limits can leave spans and their related exceptions absent.
**Check metrics for systemic causes** (spikes lining up with `firstSeen` suggest saturation rather than a code bug):
@@ -336,7 +336,7 @@ The loop, using the read commands documented in this skill:
1. **Quantify and localize.** `traceway endpoints list --since 24h --order-by p95` (or `impact`). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note that `impact`/`impactReason` are offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not.
2. **Check the accepted baseline.** Before calling an endpoint slow, run `traceway endpoints slow "<endpoint>"`. A non-zero `offsetMs` means an operator accepted that much latency (with a `reason`), so only latency *beyond* the offset is a real regression; `offsetMs 0` means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown.
3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".
-4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <distributedTraceId> --recorded-at <t>` for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.
+4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <traceId> --recorded-at <t>` (the `traceId` that `endpoints show` prints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.
5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.
6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --aggregation max`, then `mem.used`, `go.gc_pause`. A spike that lines up with the latency onset points at saturation, not a code bug.
7. **Correlate with code and deploys.** With the onset time from step 3, check what shipped then: `git log --since '<onset - 30m>' --until '<onset + 30m>'` or the deploy history. A jump at a release is a regression; a gradual ramp with no deploy is data growth (N+1, missing index, no pagination).
diff --git a/skills/traceway/performance.md b/skills/traceway/performance.md
index 0d146d8d..2bdbe5bc 100644
--- a/skills/traceway/performance.md
+++ b/skills/traceway/performance.md
@@ -17,7 +17,7 @@ All commands are read-only. Never archive or mutate anything during a performanc
2. **Get a representative slow trace.** Drill into one slow request to see where the time goes. Capture an id and its `recordedAt` together, then:
```bash
traceway endpoints show <endpointId> --recorded-at <t> # span waterfall for one request
- traceway traces show <distributedTraceId> --recorded-at <t> # cross-service timeline
+ traceway traces show <traceId> --recorded-at <t> # cross-service timeline (traceId from endpoints show)
```
`traces show` is the highest-value call when more than one service is involved: it stitches the whole logical request together end to end.
diff --git a/testing/flutter-symbolication/run-local.sh b/testing/flutter-symbolication/run-local.sh
index c1ed188e..0196bd71 100755
--- a/testing/flutter-symbolication/run-local.sh
+++ b/testing/flutter-symbolication/run-local.sh
@@ -85,7 +85,7 @@ echo "==> reading back from $TELDB"
got=
for _ in $(seq 1 25); do
if [ -f "$TELDB" ]; then
- got="$(sqlite3 "$TELDB" "SELECT stack_trace FROM exception_stack_traces WHERE stack_trace LIKE '%chargeCard%' LIMIT 1;" 2>/dev/null || true)"
+ got="$(sqlite3 "$TELDB" "SELECT stack_trace FROM exceptions_v2 WHERE stack_trace LIKE '%chargeCard%' LIMIT 1;" 2>/dev/null || true)"
[ -n "$got" ] && break
fi
sleep 1
@@ -102,7 +102,7 @@ echo "--- stored symbolicated stack trace ---"
echo "$got" | head -6
echo "$got" | grep -q "main.dart:20:3" || { echo "FAIL: chargeCard not resolved to main.dart:20:3"; exit 1; }
-counts="$(sqlite3 "$TELDB" "SELECT COUNT(*), COUNT(DISTINCT exception_hash) FROM exception_stack_traces WHERE stack_trace LIKE '%chargeCard%';")"
+counts="$(sqlite3 "$TELDB" "SELECT COUNT(*), COUNT(DISTINCT exception_hash) FROM exceptions_v2 WHERE stack_trace LIKE '%chargeCard%';")"
total="${counts%%|*}"
distinct="${counts##*|}"
echo "==> grouping: $total row(s), $distinct distinct hash(es)"
diff --git a/testing/testingapispec.md b/testing/testingapispec.md
index ac144bb7..152de5d1 100644
--- a/testing/testingapispec.md
+++ b/testing/testingapispec.md
@@ -105,7 +105,7 @@ curl http://localhost:8080/test-exception
```
**Verification:**
-- `exception_stack_traces` table: row with `type` containing panic/Error type, `value` containing `"test panic from /test-exception"`
+- `exceptions_v2` table: row with `stack_trace` containing the panic/Error type and `"test panic from /test-exception"`
- Stack trace should be present
- `transactions` table: row with `status_code` = `500`
@@ -127,7 +127,7 @@ curl http://localhost:8080/test-error-simple
```
**Verification:**
-- `exception_stack_traces` table: row with `value` containing `"simple error without stack"`
+- `exceptions_v2` table: row with `stack_trace` containing `"simple error without stack"`
- `transactions` table: row with `status_code` = `500`
---
@@ -148,7 +148,7 @@ curl http://localhost:8080/test-error-stacktrace
```
**Verification:**
-- `exception_stack_traces` table: row with `value` containing `"error with stack trace"`
+- `exceptions_v2` table: row with `stack_trace` containing `"error with stack trace"`
- Stack trace should contain the handler function name
- `transactions` table: row with `status_code` = `500`
@@ -170,7 +170,7 @@ curl http://localhost:8080/test-error-wrapped
```
**Verification:**
-- `exception_stack_traces` table: row with `value` containing `"layer 2"` and `"base error"` in the chain
+- `exceptions_v2` table: row with `stack_trace` containing `"layer 2"` and `"base error"` in the chain
- Error chain should preserve wrapping (Go: `fmt.Errorf %w`, JS: `Error.cause`)
- `transactions` table: row with `status_code` = `500`
@@ -192,7 +192,7 @@ curl http://localhost:8080/test-error-nested
```
**Verification:**
-- `exception_stack_traces` table: row with `value` containing `"error from inner function"`
+- `exceptions_v2` table: row with `stack_trace` containing `"error from inner function"`
- Stack trace should show `innerFunction`, `middleFunction`, `outerFunction` in the call chain
- `transactions` table: row with `status_code` = `500`
@@ -214,7 +214,7 @@ curl http://localhost:8080/test-message
```
**Verification:**
-- `exception_stack_traces` table: row with `value` = `"test message from /test-message"` and `type` = `"message"` (or similar message type marker)
+- `exceptions_v2` table: row with `stack_trace` = `"test message from /test-message"` and `is_message` = true
- `transactions` table: row with `status_code` = `200`
---
@@ -235,7 +235,7 @@ curl http://localhost:8080/test-message-attributes
```
**Verification:**
-- `exception_stack_traces` table: row with `value` = `"test message with attributes"`
+- `exceptions_v2` table: row with `stack_trace` = `"test message with attributes"`
- **Go only:** `tags` map should contain `source` = `"test-message-attributes"` and `priority` = `"high"`
- **JS limitation:** `captureMessage` does not support attributes; message is captured without custom tags
@@ -327,7 +327,7 @@ curl http://localhost:8080/test-attributes
```
**Verification:**
-- `exception_stack_traces` table: row with `value` containing `"exception with custom attributes"`
+- `exceptions_v2` table: row with `stack_trace` containing `"exception with custom attributes"`
- `tags` map should contain:
- `user_id` = `"usr_123"`
- `request_id` = `"req_456"`
@@ -368,7 +368,7 @@ Normal response:
```
**Verification (panic case):**
-- `exception_stack_traces` table: row with `value` containing `"panic triggered by /test-recording"`
+- `exceptions_v2` table: row with `stack_trace` containing `"panic triggered by /test-recording"`
- `tags` map should contain recorded request details (depending on framework recording support):
- URL/path information
- Request body content
@@ -457,6 +457,6 @@ curl -H "Authorization: Bearer <jwt>" \
| Table | Data From Endpoints |
|-------|-------------------|
-| `transactions` | All endpoints (1-14) generate transaction records |
-| `exception_stack_traces` | Endpoints 3-9, 13-14 generate exception/message records |
+| `endpoints_v2` / `tasks_v2` | All endpoints (1-14) generate transaction records |
+| `exceptions_v2` | Endpoints 3-9, 13-14 generate exception/message records |
| `metric_records` | Endpoint 12 generates custom metric + automatic system metrics |
diff --git a/website/app/globals.css b/website/app/globals.css
index 91a8dc96..f25206a5 100644
--- a/website/app/globals.css
+++ b/website/app/globals.css
@@ -267,7 +267,7 @@
.wrap {
max-width: var(--container-max);
margin: 0 auto;
- padding: 0 28px;
+ padding-inline: 28px;
}
/* Light band: local token remap so a run of sections renders on white.
@@ -1360,7 +1360,7 @@
padding: 48px 24px;
}
.wrap {
- padding: 0 20px;
+ padding-inline: 20px;
}
}
diff --git a/website/app/layout.tsx b/website/app/layout.tsx
index 0345f066..78cf60b7 100644
--- a/website/app/layout.tsx
+++ b/website/app/layout.tsx
@@ -33,9 +33,9 @@ export const metadata: Metadata = {
alternates: {
canonical: "./",
},
- title: "Traceway · Open Source APM Built on OpenTelemetry",
+ title: "Traceway · Your OpenTelemetry home",
description:
- "Traceway is an open-source APM built on OpenTelemetry. Complete observability: logs, traces, metrics, session replay, and exceptions, all connected. MIT licensed. Self-host free or run on Traceway Cloud.",
+ "Bring your data. Get the whole picture. Open-source observability. Self-host or run on Traceway Cloud.",
icons: {
icon: [
{ url: "/favicon.svg", type: "image/svg+xml" },
diff --git a/website/app/page.tsx b/website/app/page.tsx
index 969f6047..16465eae 100644
--- a/website/app/page.tsx
+++ b/website/app/page.tsx
@@ -33,7 +33,7 @@ const PILLARS = [
{
icon: Network,
title: "Traces",
- description: "End-to-end span waterfalls across every service",
+ description: "Search OTel spans and follow cross-service waterfalls",
href: "/product/traces",
},
{
@@ -75,6 +75,40 @@ const STACK_ICONS = [
export default function Home() {
return (
<main className="relative">
+ <aside aria-label="Traceway 2.0 release" className="wrap mt-6">
+ <Link
+ href="/blog/traceway-2-0"
+ aria-labelledby="v2-release-title v2-release-cta"
+ className="grid grid-cols-[64px_minmax(0,1fr)] items-center gap-x-5 gap-y-4 rounded-xl border border-white/10 bg-[#152e76] p-6 transition-colors hover:bg-[#193886] focus-visible:outline-2 focus-visible:outline-offset-4 focus-visible:outline-blue-bright md:grid-cols-[96px_minmax(0,1fr)_auto] md:gap-x-8 md:gap-y-2 md:px-9 md:py-7"
+ >
+ <Image
+ src="/blog/traceway-2-0/party-popper.png"
+ alt=""
+ width={96}
+ height={96}
+ className="size-16 md:row-span-2 md:size-24"
+ priority
+ />
+ <p
+ id="v2-release-title"
+ className="text-2xl font-semibold leading-snug tracking-tight text-white md:text-3xl"
+ >
+ Traceway 2.0 is here.
+ </p>
+ <p className="col-span-2 max-w-2xl text-pretty text-base leading-relaxed text-[#d2e2ff] md:col-span-1 md:col-start-2">
+ More reliable traces, better OpenTelemetry compatibility, and a
+ clearer picture of your app.
+ </p>
+ <span
+ id="v2-release-cta"
+ className="col-span-2 inline-flex min-h-11 w-fit items-center gap-2 rounded-md bg-white px-4 py-2 text-sm font-semibold text-[#152e76] md:col-span-1 md:col-start-3 md:row-span-2 md:row-start-1"
+ >
+ Read the V2 post
+ <ArrowRight className="size-4" aria-hidden />
+ </span>
+ </Link>
+ </aside>
+
{/* HERO: centered chip, title, subhead, email form */}
<section className="hero">
<div className="wrap">
@@ -87,11 +121,10 @@ export default function Home() {
className="mb-5"
/>
<h1 className="mt-6">
- The open-source APM <em>built on OpenTelemetry.</em>
+ Your <em>OpenTelemetry home.</em>
</h1>
<p className="hero-sub text-pretty">
- Complete observability. Logs, traces, metrics, session replay,
- and exceptions, all connected.
+ Bring your data. Get the whole picture.
</p>
<div className="mt-10 w-full">
diff --git a/website/app/product/traces/page.tsx b/website/app/product/traces/page.tsx
index e016aafe..b916f80e 100644
--- a/website/app/product/traces/page.tsx
+++ b/website/app/product/traces/page.tsx
@@ -1,14 +1,22 @@
-import Image from "next/image";
+import type { Metadata } from "next";
import Link from "next/link";
-import { ArrowRight, Network, Video, FileCode, TrendingUp } from "lucide-react";
+import { ArrowRight, Braces, GitBranch, Network, Radio } from "lucide-react";
import { Chip } from "@/components/chip";
import { SectionHead } from "@/components/section-head";
-import { FeatureRow } from "@/components/feature-row";
-import { BentoGrid, BentoCell } from "@/components/bento-grid";
import { FaqList } from "@/components/faq-list";
import { FinalCTA } from "@/components/final-cta";
import { AuroraBackground } from "@/components/aurora-background";
+import { TracingShowcase } from "@/components/tracing-showcase";
+import { ScreenshotZoom } from "@/components/screenshot-zoom";
+import { tracingScreenshots } from "@/lib/tracing-screenshots";
+
+export const metadata: Metadata = {
+ title: "OpenTelemetry Tracing & Span Explorer | Traceway",
+ description: "Search OpenTelemetry spans by service, duration, status and attributes. Follow a trace across services, inspect its waterfall and read correlated logs in Traceway.",
+};
+
+const tracingDocs = "https://docs.tracewayapp.com/client/otel/traces";
export default function TracesPage() {
return (
@@ -16,21 +24,18 @@ export default function TracesPage() {
<section className="hero hero-product relative">
<AuroraBackground variant="hero" />
<div className="wrap relative z-10">
- <Chip>
- <Network className="h-3 w-3 inline mr-1" />
- Traces
- </Chip>
- <h1 className="mt-6">
- See the user behind <em>every backend error.</em>
+ <Chip><Network className="mr-1 inline h-3 w-3" />OpenTelemetry tracing</Chip>
+ <h1 className="mt-6 max-w-5xl">
+ Search any span.<br /><em>Follow the whole trace.</em>
</h1>
<p className="hero-sub">
- When a backend service throws an exception, Traceway shows you the
- user&apos;s session replay, the cross-service trace, and the exact
- span that failed. No log-digging. No guessing what happened.
+ Find the slow query, failed request or background job. Open its trace
+ to see the services involved, where the time went, and what the logs
+ say—all connected by the original OpenTelemetry trace ID.
</p>
<div className="hero-cta-row">
- <Link href="https://docs.tracewayapp.com" className="btn btn-accent">
- Get Started <ArrowRight className="h-4 w-4" />
+ <Link href={tracingDocs} className="btn btn-accent">
+ Connect OpenTelemetry <ArrowRight className="h-4 w-4" />
</Link>
<Link href="https://cloud.tracewayapp.com/register" className="btn btn-ghost">
Try Traceway Cloud
@@ -39,149 +44,115 @@ export default function TracesPage() {
</div>
</section>
- {/* Distributed trace example, absorbed from home */}
- {/* WHITE BAND: feature sections render on white */}
+ <section className="wrap pb-16 sm:pb-20" aria-label="Explore tracing in Traceway">
+ <TracingShowcase />
+ </section>
+
<div className="band-light">
- <section className="wrap py-20">
+ <section className="wrap py-16 sm:py-20" aria-label="OpenTelemetry features">
<SectionHead
- eyebrow="Cross-service"
- title="Trace requests across every service"
- description="Follow a single user action from the browser through your API gateway, backend services, and database calls. See the full picture in one distributed trace."
+ eyebrow="OpenTelemetry throughout"
+ title={<>Keep the context <em>your tools already send.</em></>}
+ description="Send traces from your existing OpenTelemetry SDK or Collector over OTLP/HTTP. Your services keep their trace IDs, span IDs and parent relationships."
/>
- <div className="mt-8 max-w-4xl mx-auto">
- <Image
- src="/images/traces-cross-service.png"
- alt="Distributed trace across multiple services"
- width={1600}
- height={500}
- className="w-full h-auto rounded-[12px]"
- style={{ border: "1px solid var(--hair)" }}
- />
+ <div className="grid gap-10 md:grid-cols-3">
+ {[
+ {
+ icon: Radio,
+ title: "Search beyond HTTP",
+ text: "Database calls, internal operations, producers and consumers are all searchable spans. Filter the work itself, even when it has no endpoint or task page.",
+ },
+ {
+ icon: GitBranch,
+ title: "Follow service boundaries",
+ text: "See one trace across services and the projects you can access in your organization. Expand branches, spot errors and jump straight to a selected span.",
+ },
+ {
+ icon: Braces,
+ title: "Preserve the OTel detail",
+ text: "Original OTLP payloads retain typed attributes, resource and scope metadata, events and span links. Retrieve a stored span through the OTLP export API when you need its full payload.",
+ },
+ ].map(({ icon: Icon, title, text }) => (
+ <div key={title} className="border-t border-hair-2 pt-6">
+ <Icon className="mb-5 h-5 w-5 text-a2" aria-hidden="true" />
+ <h3 className="text-xl">{title}</h3>
+ <p className="mt-3 text-sm leading-relaxed text-fg-2">{text}</p>
+ </div>
+ ))}
</div>
</section>
- {/* Full distributed trace screenshot */}
- <section className="wrap">
- <FeatureRow
- eyebrow="Waterfall"
- title={
- <>
- Every span. <em>Every hop.</em>
- </>
- }
- description="Visualize a request as a waterfall: each hop between services shows duration, status, and the exact span where latency or errors appear. Click any span to open its logs, exceptions, and attributes."
- bullets={[
- "Cross-service distributed trace propagation",
- "W3C Trace Context standard headers",
- "Drill-down to span attributes, logs, events",
- "Async workflows (Kafka, RabbitMQ, SQS)",
- ]}
- image={{ src: "/images/traces-spans-waterfall.png", alt: "Span waterfall timing breakdown", width: 1600, height: 500 }}
- />
- </section>
-
- {/* Session replay + backend errors */}
- <section className="wrap">
- <FeatureRow
- reverse
- eyebrow="Replay"
- title="See what the user did when the backend broke"
- description="Traceway connects frontend session replays to backend exceptions. When your payment service returns a 500, you don't just see the stack trace. You see the user clicking Checkout, filling in their card, and hitting submit."
- bullets={[
- "Frontend replay linked to backend errors",
- "Automatic correlation via trace ID",
- "No manual reproduction needed",
- "Works across browser and server",
- ]}
- image={{ src: "/images/session-replay-viewer.png", alt: "Session replay linked to trace" }}
+ <section className="wrap pb-16 sm:pb-20" aria-label="Correlated logs">
+ <SectionHead
+ eyebrow="Correlated logs"
+ title={<>The request failed. <em>Here is what it logged.</em></>}
+ description="Read logs carrying the same trace ID alongside the waterfall. See the emitting service and span, then expand a record for its attributes. Logs stay scoped to the selected project."
/>
+ <figure className="overflow-hidden rounded-xl border border-hair-2 bg-ink-1">
+ <ScreenshotZoom images={tracingScreenshots} index={3} />
+ <figcaption className="px-5 py-4 text-sm leading-relaxed text-fg-2">
+ The same checkout shown above: a 950 ms payment timeout, followed by
+ a failed-order event and the gateway&apos;s 502 response. Demo data.
+ </figcaption>
+ </figure>
</section>
- {/* Two card bento */}
- <section className="wrap py-10">
- <SectionHead
- eyebrow="Deep context"
- title={
- <>
- More than spans, <em>full context</em>
- </>
- }
- />
- <BentoGrid>
- <BentoCell
- size="wide"
- icon={FileCode}
- title="Source map stack trace resolution"
- iconColor="var(--a4)"
- >
- <p>
- Minified JavaScript stack traces are resolved to original source
- files and line numbers automatically. Upload your source maps and
- every frontend error shows readable, actionable traces. Works with
- webpack, esbuild, and Vite.
+ <section className="wrap border-t border-hair py-16" aria-label="Browser tracing">
+ <div className="grid items-start gap-8 md:grid-cols-2 md:gap-16">
+ <h2>From the browser <em>to the backend.</em></h2>
+ <div>
+ <p className="leading-relaxed text-fg-2">
+ The Traceway browser SDK propagates W3C trace context on configured
+ requests. When your backend instrumentation continues that context,
+ captured errors and sessions can carry the same trace ID. Add session
+ replay to see what the user was doing alongside the technical context.
</p>
- </BentoCell>
- <BentoCell
- size="tall"
- icon={TrendingUp}
- title="Impact propagation"
- iconColor="var(--a2)"
- >
- <p>
- The Impact Score extends across service boundaries. When an
- upstream service degrades, its impact propagates to every
- downstream consumer, so you fix the root cause, not the symptoms.
- </p>
- </BentoCell>
- </BentoGrid>
+ <Link href="https://docs.tracewayapp.com/client/js-sdk/distributed-tracing" className="mt-6 inline-flex items-center gap-2 text-sm font-medium text-a2 underline-offset-4 hover:underline">
+ Set up browser-to-backend tracing <ArrowRight className="h-4 w-4" />
+ </Link>
+ </div>
+ </div>
</section>
</div>
- <FinalCTA
- title={
- <>
- Trace every hop, <em>not just the first</em>.
- </>
- }
- description="Connect an SDK, propagate one trace ID, and see the whole story."
- primary={{ label: "Get Started", href: "https://docs.tracewayapp.com" }}
- />
-
- <section className="wrap pt-10 pb-24">
- <div className="max-w-3xl mx-auto">
- <SectionHead align="center" eyebrow="FAQ" title="Questions about traces" />
- <div className="mt-4">
- <FaqList
- items={[
- {
- q: "How does distributed tracing work?",
- a: "Traceway propagates a trace ID across every service in a request chain. The frontend SDK generates the ID and passes it to your backend via the traceparent header (W3C Trace Context). Each backend service forwards it to downstream calls. Every span, exception, and session replay recorded with that trace ID is linked together, giving you a complete picture of a single user action across your entire architecture.",
- },
- {
- q: "Does Traceway connect frontend and backend issues?",
- a: "Yes. When a backend service returns an error, Traceway links it to the frontend session replay that triggered the request. You see the user's clicks and navigations alongside the server-side stack trace and span waterfall. Both sides are connected automatically via the shared trace ID.",
- },
- {
- q: "How does distributed tracing connect to session replay?",
- a: "Traceway's frontend SDK generates a trace ID for each user interaction and passes it to your backend via the traceparent header. When the backend reports a span or exception with that trace ID, Traceway links the frontend session replay to the backend trace automatically.",
- },
- {
- q: "What protocols does Traceway use for trace propagation?",
- a: "Traceway supports W3C Trace Context (traceparent/tracestate headers) and is compatible with any OpenTelemetry-instrumented service. If your services already propagate trace context, Traceway picks it up automatically.",
- },
- {
- q: "Do I need to instrument both frontend and backend?",
- a: "For the full experience (session replay linked to backend traces), yes. The frontend SDK captures user interactions and the backend middleware captures server-side spans. Both connect via the shared trace ID. However, backend-only distributed tracing works independently.",
- },
- {
- q: "Does distributed tracing work with message queues and async workflows?",
- a: "Yes. Traceway uses W3C Trace Context, and OpenTelemetry instrumentation libraries for Kafka, RabbitMQ, SQS, and other message brokers propagate the trace context through message headers automatically. When a consumer processes a message, its spans are linked to the original producer's trace, so an API that publishes to Kafka, which triggers a worker, which calls a downstream service, appears as a single connected trace.",
- },
- ]}
- />
- </div>
+ <section className="wrap py-16 sm:py-20" aria-label="Tracing questions">
+ <div className="mx-auto max-w-3xl">
+ <SectionHead align="center" eyebrow="FAQ" title="Questions about tracing" />
+ <FaqList items={[
+ {
+ q: "Can I use my existing OpenTelemetry instrumentation?",
+ a: "Yes. Send traces with an OpenTelemetry SDK or Collector using OTLP over HTTP, in protobuf or JSON format. Traceway keeps the original trace and span IDs. Backend tracing works independently of the Traceway browser SDK.",
+ },
+ {
+ q: "What can I search in the span explorer?",
+ a: "Search received spans by name, service, span kind, status, minimum or maximum duration, trace ID and exact attribute values within a time range. HTTP calls, database queries, internal spans and messaging operations are all available. Open a result to view its trace with that span selected.",
+ },
+ {
+ q: "How are spans connected across services?",
+ a: "Your instrumentation propagates W3C trace context between services. Spans in the same trace share a trace ID, and parent span IDs describe their relationships. Traceway assembles that trace across the projects you can access in the selected organization.",
+ },
+ {
+ q: "Are span links the same as parent-child relationships?",
+ a: "No. OpenTelemetry span links can reference spans in the same trace or a different trace, for example in a batch or asynchronous workflow. Traceway preserves those links in the original OTLP payload and export. They do not merge separate traces into one waterfall. Whether a queue consumer continues a trace or links to it depends on your instrumentation.",
+ },
+ {
+ q: "How do logs connect to a trace?",
+ a: "Export logs with a trace ID and, when available, a span ID. The trace page shows matching logs in the selected project and surrounding time window, with their service and span names. A span ID lets you narrow a log search to a specific operation.",
+ },
+ {
+ q: "Can I connect browser errors and session replay?",
+ a: "Yes. Configure W3C propagation in the Traceway browser SDK and context extraction in your backend instrumentation. Errors and sessions carrying that trace ID can be correlated with the backend trace. For cross-origin requests, allow the destination in the SDK and permit traceparent and tracestate in your server's CORS policy.",
+ },
+ ]} />
</div>
</section>
+
+ <FinalCTA
+ title={<>Bring your spans. <em>See the request.</em></>}
+ description="Connect your OpenTelemetry exporter and start with the span explorer."
+ primary={{ label: "Connect OpenTelemetry", href: tracingDocs }}
+ secondary={{ label: "Try Traceway Cloud", href: "https://cloud.tracewayapp.com/register" }}
+ />
</main>
);
}
diff --git a/website/components/home-tabs.tsx b/website/components/home-tabs.tsx
index 89e43c00..61fe9e6e 100644
--- a/website/components/home-tabs.tsx
+++ b/website/components/home-tabs.tsx
@@ -55,14 +55,14 @@ const tabs: Tab[] = [
label: "Microservice",
icon: Network,
color: "var(--ok)",
- heading: "Follow a request across every service.",
+ heading: "Search spans. Follow the whole request.",
description:
- "Distributed tracing connects frontend sessions to backend errors across your entire microservice topology. When a backend service throws an exception, you see the user's session replay, the full cross-service trace, and the exact span that failed.",
+ "Search OpenTelemetry spans by service, duration, status and attributes. Open a result to follow its trace across services, inspect the span waterfall and read logs carrying the same trace ID.",
bullets: [
- "Cross-service distributed trace propagation",
- "Frontend sessions linked to backend errors",
- "Full trace context across service boundaries",
- "Exception pinpointing across services",
+ "Span explorer with service and attribute filters",
+ "One trace across services and accessible projects",
+ "Original OTel trace IDs and parent relationships",
+ "Correlated logs and span-level error context",
],
href: "/product/traces",
},
diff --git a/website/components/screenshot-zoom.module.css b/website/components/screenshot-zoom.module.css
new file mode 100644
index 00000000..47efb999
--- /dev/null
+++ b/website/components/screenshot-zoom.module.css
@@ -0,0 +1,45 @@
+.overlay {
+ position: fixed;
+ inset: 0;
+ z-index: 150;
+ display: flex;
+ flex-direction: column;
+ align-items: center;
+ overflow-y: auto;
+ overscroll-behavior: contain;
+ padding: max(16px, env(safe-area-inset-top)) max(16px, env(safe-area-inset-right)) max(16px, env(safe-area-inset-bottom)) max(16px, env(safe-area-inset-left));
+ background: rgba(9, 10, 12, 0.94);
+ backdrop-filter: blur(10px);
+ -webkit-backdrop-filter: blur(10px);
+ animation: fade-in 160ms ease-out;
+}
+
+.panel {
+ position: relative;
+ flex-shrink: 0;
+ width: 100%;
+ max-width: 1152px;
+ margin-block: auto;
+ outline: none;
+ animation: rise-in 220ms cubic-bezier(0.2, 0.8, 0.2, 1);
+}
+
+@media (min-width: 640px) {
+ .overlay {
+ padding: 32px;
+ }
+}
+
+@keyframes fade-in {
+ from { opacity: 0; }
+ to { opacity: 1; }
+}
+
+@keyframes rise-in {
+ from { opacity: 0; transform: translateY(12px) scale(0.985); }
+ to { opacity: 1; transform: none; }
+}
+
+@media (prefers-reduced-motion: reduce) {
+ .overlay, .panel { animation: none; }
+}
diff --git a/website/components/screenshot-zoom.tsx b/website/components/screenshot-zoom.tsx
new file mode 100644
index 00000000..f2a73e79
--- /dev/null
+++ b/website/components/screenshot-zoom.tsx
@@ -0,0 +1,102 @@
+"use client";
+
+import Image from "next/image";
+import { useState } from "react";
+import * as Dialog from "@radix-ui/react-dialog";
+import { ChevronLeft, ChevronRight, Expand, X } from "lucide-react";
+import styles from "./screenshot-zoom.module.css";
+
+type Screenshot = {
+ src: string;
+ width: number;
+ height: number;
+ alt: string;
+ label: string;
+ title: string;
+ description: string;
+};
+
+export function ScreenshotZoom({ images, index = 0, eager = false }: {
+ images: readonly Screenshot[];
+ index?: number;
+ eager?: boolean;
+}) {
+ const [open, setOpen] = useState(false);
+ const [selected, setSelected] = useState(index);
+ const preview = images[index];
+ const image = images[selected];
+ const step = (delta: number) => setSelected(current => (current + delta + images.length) % images.length);
+ const controlClass = "inline-flex h-11 w-11 shrink-0 cursor-pointer items-center justify-center rounded-full bg-white/10 text-white transition-colors hover:bg-white/20 focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-white";
+
+ return (
+ <Dialog.Root open={open} onOpenChange={setOpen}>
+ <Dialog.Trigger asChild>
+ <button
+ type="button"
+ className="group relative block w-full cursor-zoom-in text-left focus-visible:outline-2 focus-visible:outline-offset-[-4px] focus-visible:outline-a2"
+ onClick={() => setSelected(index)}
+ aria-label={`Enlarge: ${preview.label}`}
+ >
+ <Image
+ src={preview.src}
+ alt={preview.alt}
+ width={preview.width}
+ height={preview.height}
+ sizes="(max-width: 1280px) 100vw, 1240px"
+ loading={eager ? "eager" : "lazy"}
+ className="block h-auto w-full"
+ />
+ <span className="absolute right-3 bottom-3 flex items-center gap-2 rounded-md border border-white/15 bg-[#05070c] px-3 py-2 text-xs text-white/80 group-hover:text-white">
+ <Expand className="h-3.5 w-3.5" aria-hidden="true" />
+ Full size
+ </span>
+ </button>
+ </Dialog.Trigger>
+ <Dialog.Portal>
+ <Dialog.Overlay className={styles.overlay} onClick={event => {
+ if (event.target === event.currentTarget) setOpen(false);
+ }}>
+ <Dialog.Content className={styles.panel} onKeyDown={event => {
+ if (event.key === "ArrowRight" || event.key === "ArrowLeft") {
+ event.preventDefault();
+ step(event.key === "ArrowRight" ? 1 : -1);
+ }
+ }}>
+ <div className="mb-3 flex items-center justify-between text-sm text-white/70">
+ <span aria-live="polite">{selected + 1} / {images.length}</span>
+ <Dialog.Close className={controlClass} aria-label="Close image viewer">
+ <X className="h-5 w-5" aria-hidden="true" />
+ </Dialog.Close>
+ </div>
+ <div className="overflow-hidden rounded-xl border border-white/15 bg-[#0a0d14]">
+ <Image
+ key={image.src}
+ src={image.src}
+ alt={image.alt}
+ width={image.width}
+ height={image.height}
+ sizes="100vw"
+ loading="eager"
+ className="max-h-[60dvh] w-full object-contain"
+ />
+ </div>
+ <div className="mt-4 flex flex-col items-start justify-between gap-4 sm:flex-row sm:gap-6">
+ <div className="min-w-0" aria-live="polite">
+ <Dialog.Title className="text-xl font-semibold text-white">{image.title}</Dialog.Title>
+ <Dialog.Description className="mt-2 max-w-3xl text-[15px] leading-relaxed text-white/70">{image.description}</Dialog.Description>
+ </div>
+ {images.length > 1 && <div className="flex shrink-0 gap-2">
+ <button type="button" className={controlClass} onClick={() => step(-1)} aria-label="Previous screenshot">
+ <ChevronLeft className="h-5 w-5" aria-hidden="true" />
+ </button>
+ <button type="button" className={controlClass} onClick={() => step(1)} aria-label="Next screenshot">
+ <ChevronRight className="h-5 w-5" aria-hidden="true" />
+ </button>
+ </div>}
+ </div>
+ </Dialog.Content>
+ </Dialog.Overlay>
+ </Dialog.Portal>
+ </Dialog.Root>
+ );
+}
diff --git a/website/components/site-header.tsx b/website/components/site-header.tsx
index f455ab04..c0b216bd 100644
--- a/website/components/site-header.tsx
+++ b/website/components/site-header.tsx
@@ -43,7 +43,7 @@ const PILLARS: NavItem[] = [
},
{
title: "Traces",
- description: "Follow a request across every service.",
+ description: "Search OTel spans. Follow the whole trace.",
href: "/product/traces",
icon: Network,
},
diff --git a/website/components/tracing-showcase.tsx b/website/components/tracing-showcase.tsx
new file mode 100644
index 00000000..8cdc1e5d
--- /dev/null
+++ b/website/components/tracing-showcase.tsx
@@ -0,0 +1,44 @@
+"use client";
+
+import { useState } from "react";
+import { ListFilter, Network, ScanSearch } from "lucide-react";
+import { ScreenshotZoom } from "@/components/screenshot-zoom";
+import { tracingScreenshots } from "@/lib/tracing-screenshots";
+
+const icons = [ListFilter, Network, ScanSearch];
+
+export function TracingShowcase() {
+ const [active, setActive] = useState(0);
+ const view = tracingScreenshots[active];
+
+ return (
+ <div className="overflow-hidden rounded-xl border border-hair-2 bg-ink-1">
+ <div className="grid grid-cols-3 border-b border-hair-2" role="group" aria-label="Tracing screenshots">
+ {tracingScreenshots.slice(0, 3).map(({ label }, index) => {
+ const Icon = icons[index];
+ return (
+ <button
+ key={label}
+ type="button"
+ aria-pressed={active === index}
+ aria-controls="tracing-screenshot"
+ onClick={() => setActive(index)}
+ className={`flex cursor-pointer flex-col items-center justify-center gap-2 border-b-2 px-2 py-4 text-xs font-medium transition-colors focus-visible:outline-2 focus-visible:outline-offset-[-4px] focus-visible:outline-a2 sm:flex-row sm:gap-3 sm:px-5 sm:text-sm ${active === index ? "border-a2 bg-ink-3 text-fg-0" : "border-transparent text-fg-2 hover:bg-ink-2 hover:text-fg-0"}`}
+ >
+ <Icon className="h-4 w-4 shrink-0" aria-hidden="true" />
+ {label}
+ </button>
+ );
+ })}
+ </div>
+ <figure id="tracing-screenshot">
+ <ScreenshotZoom images={tracingScreenshots} index={active} eager={active === 0} />
+ <figcaption className="border-t border-hair-2 px-5 py-5 sm:px-7" aria-live="polite" aria-atomic="true">
+ <p className="font-medium text-fg-0">{view.title}</p>
+ <p className="mt-2 max-w-3xl text-sm leading-relaxed text-fg-2">{view.description}</p>
+ <p className="mt-4 font-mono text-[11px] text-fg-2">Captured in Traceway · checkout demo data</p>
+ </figcaption>
+ </figure>
+ </div>
+ );
+}
diff --git a/website/content/blog/traceway-2-0.mdx b/website/content/blog/traceway-2-0.mdx
new file mode 100644
index 00000000..affb6be4
--- /dev/null
+++ b/website/content/blog/traceway-2-0.mdx
@@ -0,0 +1,24 @@
+---
+title: "Traceway 2.0: a new foundation for traces"
+date: "2026-09-20"
+category: engineering
+author: dusan
+image: /blog/traceway-2-0/og.png
+description: >-
+ A request starts a task. The task calls an AI model. You should be able
+ to follow the whole thing. That's why we've rebuilt tracing in Traceway 2.0.
+---
+
+![Traceway 2.0 release, with a large party popper and a smaller construction emoji.](/blog/traceway-2-0/og.png)
+
+Traceway 2.0 is here. 🎉
+
+We've rebuilt tracing around one shared span model. Incoming OpenTelemetry spans keep their original IDs, parent relationships, and full payloads. That makes Traceway much more OTel compatible, with more reliable data linkage and a more faithful view of your traces in the dashboard, even when child spans arrive before their parents. You can also search individual spans, including the slow query buried inside a request.
+
+The cost is **a bigger database and more work on reads and writes**, which can make them slower. We're accepting that overhead to preserve the data and its relationships more faithfully. Loading the trace outline before the full details helps limit how much data we fetch while you browse.
+
+There is one thing to plan for: **this is a breaking upgrade**. The trace tables and parts of the API have changed. New tables are created automatically, but copying your history is opt-in. Until you copy it, those views show only new data.
+
+Back up first, upgrade the backend and dashboard together, and stop all old backend writers. Then enable `V2_MOVE_OVER=true` on **one** upgraded backend instance to copy history in the background, newest day first. This works on **ClickHouse, SQLite, and DuckDB**. Fresh installs can skip it.
+
+[Read the migration guide before upgrading](https://docs.tracewayapp.com/server/upgrade-v2-tables). It covers retention, checking the copy, and rollback.
diff --git a/website/lib/tracing-screenshots.ts b/website/lib/tracing-screenshots.ts
new file mode 100644
index 00000000..036c93f7
--- /dev/null
+++ b/website/lib/tracing-screenshots.ts
@@ -0,0 +1,38 @@
+export const tracingScreenshots = [
+ {
+ label: "Search spans",
+ src: "/images/tracing/span-explorer.png",
+ width: 2528,
+ height: 1640,
+ alt: "Span explorer filtered to checkout operations over 40 milliseconds, showing services, span kinds, errors, durations and trace IDs",
+ title: "Start with the operation that needs attention.",
+ description: "Filter by service, name, kind, status, duration, trace ID or an exact attribute value. Open a result to see where it belongs in the request.",
+ },
+ {
+ label: "Follow the trace",
+ src: "/images/tracing/trace-waterfall.png",
+ width: 2528,
+ height: 1640,
+ alt: "A checkout trace with 14 spans across four services and two projects, including PostgreSQL, Redis, a failed payment call and a Kafka worker",
+ title: "One checkout. Four services. The full sequence.",
+ description: "The gateway, checkout API, inventory service and worker share one trace ID. The waterfall shows their parent-child relationships and the payment call that took 950 ms.",
+ },
+ {
+ label: "Inspect a span",
+ src: "/images/tracing/span-attributes.png",
+ width: 2528,
+ height: 1640,
+ alt: "Payment span details open over the waterfall, displaying the original trace and span IDs, service, client kind, error status, instrumentation scope and HTTP attributes",
+ title: "Read the context behind the failed span.",
+ description: "Select a span to inspect its service, kind, status, instrumentation scope and attributes. Copy the trace or span ID to continue the investigation in your own tools.",
+ },
+ {
+ label: "Correlated logs",
+ src: "/images/tracing/trace-logs.png",
+ width: 2464,
+ height: 712,
+ alt: "Seven logs for the checkout trace: order loaded, inventory reserved, payment timeout, failure published and the final 502, each with its severity and emitting span",
+ title: "The request failed. Here is what it logged.",
+ description: "The same checkout: a 950 ms payment timeout, followed by a failed-order event and the gateway’s 502 response. Each log carries the trace ID and its emitting span. Demo data.",
+ },
+] as const;
diff --git a/website/public/blog/traceway-2-0/og.png b/website/public/blog/traceway-2-0/og.png
new file mode 100644
index 00000000..2deffd35
Binary files /dev/null and b/website/public/blog/traceway-2-0/og.png differ
diff --git a/website/public/blog/traceway-2-0/party-popper.png b/website/public/blog/traceway-2-0/party-popper.png
new file mode 100644
index 00000000..c59a2b8c
Binary files /dev/null and b/website/public/blog/traceway-2-0/party-popper.png differ
diff --git a/website/public/images/tracing/span-attributes.png b/website/public/images/tracing/span-attributes.png
new file mode 100644
index 00000000..f2b0176d
Binary files /dev/null and b/website/public/images/tracing/span-attributes.png differ
diff --git a/website/public/images/tracing/span-explorer.png b/website/public/images/tracing/span-explorer.png
new file mode 100644
index 00000000..bf905de3
Binary files /dev/null and b/website/public/images/tracing/span-explorer.png differ
diff --git a/website/public/images/tracing/trace-logs.png b/website/public/images/tracing/trace-logs.png
new file mode 100644
index 00000000..0555469d
Binary files /dev/null and b/website/public/images/tracing/trace-logs.png differ
diff --git a/website/public/images/tracing/trace-waterfall.png b/website/public/images/tracing/trace-waterfall.png
new file mode 100644
index 00000000..b14a9ad2
Binary files /dev/null and b/website/public/images/tracing/trace-waterfall.png differ