The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
feat: better docs and ai skills (#287)
narrow-adventurec4ea50c7c630
diff --git a/CLAUDE.md b/CLAUDE.mdindex 27939900..e6c3ac9c 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -8,7 +8,6 @@ Traceway is an error tracking and monitoring platform consisting of:- **Frontend**: SvelteKit 2 dashboard application with Svelte 5- **Backend**: Go/Gin API server with ClickHouse database- **CLI**: Go/Cobra command-line client for the backend HTTP API (`/cli`)-- **Go Client SDK**: Distributed tracing SDK for Go applications (external repo)---@@ -33,7 +32,6 @@ Traceway is an error tracking and monitoring platform consisting of:| CLI | `cd cli && just test` | Runs unit tests || CLI | `cd cli && just check` | Lint + test + vulncheck (pre-commit gate) || CLI | `cd cli && just smoke-test` | Live E2E (needs `TRACEWAY_SMOKE_*` env vars) |-| Go Client | External repo at `/Users/dusanstanojevic/Documents/workspace/go-client` | Build with `go build ./...` |### Tech Stack- **Frontend**: SvelteKit 2.49, Svelte 5.45, Tailwind CSS v4, shadcn-svelte, Vite 7@@ -1073,20 +1071,28 @@ backend/app/migrations/ch/The backend normalizes stack traces before hashing to group identical errors despite different runtime values. This happens in `backend/app/controllers/clientcontrollers/client.controller.go`.-**Normalization Steps:**-1. Extract error type only (remove error message content)-2. Collapse JS SDK function-name lines (ending in `()`, directly above a 4-space-indented `file:line:col` location line) to `<fn>` so resolved function names never affect grouping; anchoring on the location line keeps Go traces (tab-indented file lines) untouched-3. Remove absolute file paths (keep `filename:line` only)-4. Replace hexadecimal addresses with `<hex>`-5. Replace UUIDs with `<uuid>`-6. Replace IP addresses with `<ip>`-7. Replace timestamps with `<timestamp>`-8. Replace numeric IDs in paths with `<id>`-9. Normalize whitespace-10. Remove ANSI color codes-11. Hash the normalized string with SHA-256, truncate to 16 chars--**Note:** columns are intentionally kept in the hash. Minified bundles put everything on line 1, so the column is the only frame disambiguator when no source map matched. Function names are excluded (step 2) because they are derived from the location and change as symbolication improves.+**Normalization Steps** (`ComputeExceptionHash`, applied in this order, and skipped entirely when `isMessage` is true):+1. Strip the message from `Caused by:` lines, keeping the class name (`causedByRe`)+2. Strip the message from the error line, keeping the error type (`errorMessageRe`)+3. Collapse JS SDK function-name lines (ending in `()`, directly above a 4-space-indented `file:line:col` location line) to `<fn>` so resolved function names never affect grouping; anchoring on the location line keeps Go traces (tab-indented file lines) untouched (`jsFuncLineRe`)+4. Remove URL origins such as `https://cdn.example.com` (`urlOriginRe`)+5. Remove absolute file paths, keeping `filename:line` (`absolutePathRe`)+6. Drop the column from any frame whose line number is 2 or higher (`laterLineColRe`)+7. Remove `@v1.2.3` module version suffixes (`versionRe`)+8. Replace hexadecimal addresses with `<hex>` (`hexRe`)+9. Replace UUIDs with `<uuid>` (`uuidRe`)+10. Replace runs of 5 or more digits with `<id>`, unless preceded by a colon or another digit (`largeNumberRe`)+11. Replace email addresses with `<email>` (`emailRe`)+12. Replace IP addresses, with an optional port, with `<ip>` (`ipRe`)+13. Replace Go goroutine ids with `goroutine <n>` (`goroutineRe`)+14. Drop line numbers from Java, Kotlin and Scala frames (`javaLineNumRe`)+15. Collapse `... 12 more` to `... more` (`javaEllipsisRe`)+16. Collapse runs of spaces and tabs, then runs of newlines (`spacesRe`, `newlinesRe`)+17. Trim, hash with SHA-256, truncate to 16 hex chars++**Not normalized:** timestamps and ANSI color codes. Two otherwise identical traces that differ only in an embedded timestamp, or only in `\x1b[31m` escapes, produce two separate Issues. Add a regex to the block in `client.controller.go` if that ever matters.++**Note:** the column is kept only on line 1 (step 6). Minified bundles put everything on line 1, so there the column is the only frame disambiguator when no source map matched, while on real source lines the column is noise. Function names are excluded (step 3) because they are derived from the location and change as symbolication improves.**Result:** Same logical error gets same hash, even if:- Error message contains different user IDs@@ -1188,209 +1194,19 @@ if err != nil {----## Go Client SDK+## Native `/api/report` Protocol-**Location:** External repository at `/Users/dusanstanojevic/Documents/workspace/go-client`+There is **no Traceway Go SDK**. It was retired, and every backend, Go included, instruments with+OpenTelemetry and exports over OTLP/HTTP. The docs carry no Go SDK pages, and `docs/public/_redirects`+301s the old `/client/sdk` and `/client/*-middleware` URLs to `/client/otel`. Do not reintroduce them.-The SDK is a separate Go module that applications import to send telemetry to Traceway.--### Installation-```go-import (- "github.com/user/traceway" // Core SDK- "github.com/user/traceway/traceway_gin" // Gin middleware-)-```--### Connection String Format-```-<project_token>@<server_url>--Examples:-abc123@http://localhost:8082/api/report-abc123@https://traceway.example.com/api/report-```--### Architecture--#### Ring Buffer Batching-The SDK uses a ring buffer to batch telemetry data:-1. Data collected into current "frame" (transactions, exceptions, metrics)-2. Frame rotates every 5 seconds (configurable via `WithCollectionInterval`)-3. Completed frames uploaded async via HTTP POST-4. Failed uploads retry with exponential backoff--#### Data Types Collected-- **Transactions**: HTTP requests (endpoint, duration, status code)-- **Exceptions**: Errors/panics with full stack traces-- **Metrics**: System metrics (CPU, memory, Go runtime)--### Gin Middleware Integration--```go-package main--import (- "github.com/gin-gonic/gin"- "traceway"- "traceway/traceway_gin"-)--func main() {- router := gin.Default()-- // Initialize with app name and connection string- traceway_gin.Use(router, "myapp", "token@http://localhost:8082/api/report")-- // Or with options- traceway_gin.Use(router, "myapp", "token@http://localhost:8082/api/report",- traceway.WithDebug(true),- traceway.WithCollectionInterval(10 * time.Second),- )-- router.GET("/api/users", getUsers)- router.Run(":8080")-}-```--The middleware automatically:-- Tracks request duration and status code-- Captures endpoint as `"METHOD /path"` (e.g., `"GET /api/users"`)-- Recovers from panics and reports them as exceptions-- Attaches request scope for contextual tags--### Capture Methods--#### Exceptions-```go-// Basic capture-traceway.CaptureException(err)--// With additional scope-traceway.CaptureExceptionWithScope(err, scope)--// Panic recovery (use in defer)-defer traceway.Recover()--// Recover with custom scope-defer traceway.RecoverWithScope(scope)-```--#### Metrics-```go-// Capture custom metric-traceway.CaptureMetric("custom.metric.name", 42.0)--// Metrics are batched and sent with the next frame-```--#### Transactions (Manual)-```go-// For non-HTTP transactions or custom tracking-txn := traceway.StartTransaction("operation_name")-defer txn.End()--// Add segments for sub-operations-seg := txn.StartSegment("database_query")-// ... do work ...-seg.End()-```--#### Tasks and Segments-```go-// Capture a task with duration-traceway.CaptureTask("background_job", startTime, endTime, nil)--// Measure a function-result := traceway.MeasureTask("compute", func() interface{} {- return heavyComputation()-})--// Segments within tasks-traceway.CaptureSegment(taskID, "subtask", startTime, endTime)-```--### Scope System--Scopes attach contextual tags to exceptions and transactions.--#### Global Scope-```go-// Configure tags that apply to all telemetry-traceway.ConfigureScope(func(s *traceway.Scope) {- s.SetTag("environment", "production")- s.SetTag("version", "1.2.3")- s.SetTag("region", "us-east-1")-})-```--#### Request Scope (Gin)-```go-func handler(c *gin.Context) {- // Get request-scoped scope from Gin context- scope := traceway_gin.GetScopeFromGin(c)-- // Tags only apply to this request- scope.SetTag("user_id", userID)- scope.SetTag("tenant", tenantID)-- // Any exceptions captured in this request will include these tags-}-```--#### Scope Methods-```go-scope.SetTag("key", "value") // Set a single tag-scope.SetTags(map[string]string{ // Set multiple tags- "key1": "value1",- "key2": "value2",-})-scope.SetUser(userID) // Shorthand for user_id tag-scope.SetExtra("key", anyValue) // Set extra data (serialized to JSON)-```--### Configuration Options--```go-traceway.Init(appName, connectionString,- // Debug mode - logs all telemetry to stdout- traceway.WithDebug(true),-- // Max frames to keep in ring buffer (default: 20)- traceway.WithMaxCollectionFrames(20),-- // How often to rotate frames (default: 5s)- traceway.WithCollectionInterval(5 * time.Second),-- // HTTP upload timeout (default: 3s)- traceway.WithUploadTimeout(3 * time.Second),-- // Metric collection interval (default: 30s)- traceway.WithMetricInterval(30 * time.Second),-- // Disable automatic metric collection- traceway.WithMetricsDisabled(),-- // Custom HTTP transport- traceway.WithTransport(customTransport),-)-```--### Default Collected Metrics--| Metric | Description | Unit |-|--------|-------------|------|-| `cpu.used_pcnt` | CPU usage percentage | % |-| `mem.used` | Memory usage | MB |-| `mem.used_pcnt` | Memory usage percentage | % |-| `go.go_routines` | Active goroutine count | count |-| `go.heap_objects` | Heap object count | count |-| `go.num_gc` | Total GC cycles | count |-| `go.gc_pause` | Last GC pause time | nanoseconds |+The `/api/report` endpoint below still exists and is still served. The browser and mobile SDKs+(`@tracewayapp/*`, the iOS and Android libraries, Flutter) speak it, and the backend uses it to report+its own telemetry. Keep this section accurate for those clients.### Data Format (Frame)-The SDK sends data as gzipped JSON. The wire shape is `ReportRequest` in `backend/app/controllers/clientcontrollers/client.controller.go` wrapping `CollectionFrame` from `backend/app/models/clientmodels/`:+Clients send data as gzipped JSON. The wire shape is `ReportRequest` in `backend/app/controllers/clientcontrollers/client.controller.go` wrapping `CollectionFrame` from `backend/app/models/clientmodels/`:```json{@@ -1631,18 +1447,24 @@ type PaginationParams struct {### Adding a New Framework-Every framework addition touches multiple files across backend, frontend, and docs. Use this checklist to avoid missing any:--1. **Backend** — `backend/app/controllers/project.controller.go`: Add to `validFrameworks` map and update the validation error message-2. **Frontend state** — `frontend/src/lib/state/projects.svelte.ts`: Add to `Framework` type union and `FRAMEWORK_LABELS` map-3. **Frontend combobox** — `frontend/src/lib/components/framework-combobox.svelte`: Add entry with correct group (Go / JavaScript / PHP / etc.)-4. **Frontend icon** — `frontend/src/lib/components/framework-icon.svelte`: Add icon mapping for the new framework value-5. **Framework code** — `frontend/src/lib/utils/framework-code.ts`: Add install command, integration code snippet, label, code language, and testing routes-6. **Connection page** — `frontend/src/routes/connection/+page.svelte`: Add highlight language mapping and install description-7. **Dashboard page** — `frontend/src/routes/+page.svelte`: Add highlight language mapping-8. **Docs sidebar** — `docs/pages/client/_meta.json`: Add navigation entry-9. **Docs SDK selector** — `docs/components/SdkContext.jsx`: Add to `SDK_OPTIONS` array and `PATH_SDK_MAP` object-10. **Docs framework picker** — `docs/components/FrameworkPicker.jsx`: Add card to `FRAMEWORKS` array-11. **Docs icon** — `docs/public/`: Add framework icon (PNG, ~45x45)-12. **Docs page** — `docs/pages/client/<framework>/`: Create `_meta.json` and `index.mdx`-13. **Docs OTel language table** — `docs/pages/client/otel/index.mdx`: Add row if the framework uses OTLP+**Backends do not get a new framework value.** OpenTelemetry is the single backend integration path, and `opentelemetry` is the only backend option in the project-creation picker (plus the preselected default). A new backend language or web framework is a *documentation and Connection-page* change, never a new project framework:++1. **Connection page targets** — `frontend/src/lib/utils/otel-setup.ts`: add the framework to the matching `OTEL_TARGETS` language entry (or add a new language target) and its setup steps+2. **Docs guide** — `docs/pages/client/otel/<framework>/`: create `_meta.json` and `index.mdx`, then list it in `docs/pages/client/otel/_meta.json` and the "Next Steps" block of `docs/pages/client/otel/index.mdx`+3. **Docs OTel language table** — `docs/pages/client/otel/index.mdx`: add a row+4. **Docs framework picker** — nothing to do. `docs/components/FrameworkPicker.jsx` carries exactly one backend card, **OpenTelemetry**, pointing at `/client/otel`, so it mirrors the dashboard's project-creation picker. Do not add a per-framework backend card; the guide is discovered from `/client/otel` instead.+5. **Combobox search keywords** — `frontend/src/lib/components/framework-combobox.svelte`: append the name to the OpenTelemetry entry's `keywords` so searching for it still finds the right option+6. **README** — add to the supported-frameworks table if it warrants a row++Only a **browser or mobile** framework needs a real new framework value, because those use platform SDKs with their own Connection-page content:++1. **Backend** — `backend/app/controllers/project.controller.go`: add to `validFrameworks` and update the validation error message+2. **Frontend state** — `frontend/src/lib/state/projects.svelte.ts`: add to the `Framework` union, `FRAMEWORK_LABELS`, and `MOBILE_FRAMEWORKS`/`FRONTEND_FRAMEWORKS`+3. **Frontend combobox** — `frontend/src/lib/components/framework-combobox.svelte`: add an entry to the `Browser` or `Mobile` group with `keywords`+4. **Frontend icon** — `frontend/src/lib/components/framework-icon.svelte`: add the icon mapping+5. **Framework code** — `frontend/src/lib/utils/framework-code.ts`: add install command, integration snippet, label, code language, and testing routes+6. **Connection page** — `frontend/src/routes/connection/+page.svelte`: add highlight language mapping and install description+7. **Dashboard page** — `frontend/src/routes/+page.svelte`: add highlight language mapping+8. **Docs** — `docs/pages/client/<framework>/`, `docs/pages/client/_meta.json`, `SDK_OPTIONS` + `FOLDER_SDK` in `docs/components/SdkContext.jsx`, `SDK_VISIBILITY` in `docs/theme.config.jsx`, `SDK_QUICK_START` in `docs/components/SdkSelector.jsx`, and a `FrameworkPicker.jsx` card++Values removed from the picker (`gin`, `fiber`, `chi`, `fasthttp`, `stdlib`, `custom`, `nextjs`, `nestjs`, `express`, `remix`, `hono`, `cloudflare`, `symfony`, `laravel`, `django`) stay valid in `validFrameworks` and in `FRAMEWORK_LABELS` — existing projects still carry them and must keep rendering.diff --git a/README.md b/README.mdindex fef74987..73c6709b 100644--- a/README.md+++ b/README.md@@ -138,43 +138,43 @@ Open `http://localhost:8082`, log in, and hit your app to see traces appear. Ful## Supported Integrations-Traceway integrates with the tools you already use. Every integration ships traces, metrics, and logs over **OTLP/HTTP**. No proprietary SDK required.+Traceway integrates with the tools you already use. Backends report over **OTLP/HTTP**, so any OpenTelemetry SDK works. Browser and mobile apps use the Traceway SDKs.> View the full list in the [documentation](https://docs.tracewayapp.com/client). Missing a framework? [Open an issue](https://github.com/tracewayapp/traceway/issues) to request it.### Backend-<table width="100%">+OpenTelemetry is the integration path for every backend language. Point your OTLP exporter at Traceway and you are done.++<table><tbody><tr>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/gin-middleware"><img src="./docs/public/gin.png" height="28" alt="Gin" /><br/><b>Gin</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/chi-middleware"><img src="./docs/public/chi.png" height="28" alt="Chi" /><br/><b>Chi</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/fiber-middleware"><img src="./docs/public/fiber.svg" height="28" alt="Fiber" /><br/><b>Fiber</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/fasthttp-middleware"><img src="./docs/public/fasthttp.png" height="28" alt="FastHTTP" /><br/><b>FastHTTP</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/http-middleware"><img src="./docs/public/stdlib.png" height="28" alt="net/http" /><br/><b>net/http</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/sdk"><img src="./docs/public/custom.png" height="28" alt="Go Generic" /><br/><b>Go Generic</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/node-sdk"><img src="./docs/public/node.png" height="28" alt="Node.js" /><br/><b>Node.js</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel"><img src="./docs/public/otel.png" height="28" alt="OpenTelemetry" /><br/><b>OpenTelemetry</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/nodejs"><img src="./docs/public/node.png" height="28" alt="Node.js" /><br/><b>Node.js</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/nestjs"><img src="./docs/public/nestjs.png" height="28" alt="NestJS" /><br/><b>NestJS</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/nextjs"><img src="./docs/public/nextjs.png" height="28" alt="Next.js" /><br/><b>Next.js</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/hono"><img src="./docs/public/hono.png" height="28" alt="Hono" /><br/><b>Hono</b></a></td></tr><tr>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/nestjs"><img src="./docs/public/nestjs.png" height="28" alt="NestJS" /><br/><b>NestJS</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/hono"><img src="./docs/public/hono.png" height="28" alt="Hono" /><br/><b>Hono</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/symfony"><img src="./docs/public/symfony.png" height="28" alt="Symfony" /><br/><b>Symfony</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/laravel"><img src="./docs/public/laravel.png" height="28" alt="Laravel" /><br/><b>Laravel</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/django"><img src="./docs/public/django.png" height="28" alt="Django" /><br/><b>Django</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/cloudflare"><img src="./docs/public/cloudflare.png" height="28" alt="Cloudflare Workers" /><br/><b>Cloudflare</b></a></td>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel"><img src="./docs/public/otel.png" height="28" alt="OpenTelemetry" /><br/><b>OpenTelemetry</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/cloudflare"><img src="./docs/public/cloudflare.png" height="28" alt="Cloudflare Workers" /><br/><b>Cloudflare</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/symfony"><img src="./docs/public/symfony.png" height="28" alt="Symfony" /><br/><b>Symfony</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/laravel"><img src="./docs/public/laravel.png" height="28" alt="Laravel" /><br/><b>Laravel</b></a></td>+<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/otel/django"><img src="./docs/public/django.png" height="28" alt="Django" /><br/><b>Django</b></a></td></tr></tbody></table>+Go applications use OpenTelemetry too. Point [opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go) at the same OTLP endpoint. See the [OpenTelemetry guide](https://docs.tracewayapp.com/client/otel).+### Frontend> Session Replay is included with every frontend integration, and with Flutter too.+A full-stack framework has two halves and two Traceway projects. Next.js and Remix use the React integration below for the browser, SvelteKit uses Svelte, and Nuxt uses Vue. Their server half is a separate OpenTelemetry project from the table above. See [Project Structure](https://docs.tracewayapp.com/learn/projects).+<table width="100%"><tbody><tr>-<td align="center" width="150"><a href="https://docs.tracewayapp.com/client/nextjs"><img src="./docs/public/nextjs.png" height="28" alt="Next.js" /><br/><b>Next.js</b></a></td><td align="center" width="150"><a href="https://docs.tracewayapp.com/client/react"><img src="./docs/public/react.png" height="28" alt="React" /><br/><b>React</b></a></td><td align="center" width="150"><a href="https://docs.tracewayapp.com/client/vue"><img src="./docs/public/vue.png" height="28" alt="Vue" /><br/><b>Vue</b></a></td><td align="center" width="150"><a href="https://docs.tracewayapp.com/client/svelte"><img src="./docs/public/svelte.png" height="28" alt="Svelte" /><br/><b>Svelte</b></a></td>diff --git a/backend/go.mod b/backend/go.modindex 1dbe6774..742711fb 100644--- a/backend/go.mod+++ b/backend/go.mod@@ -19,7 +19,7 @@ require (github.com/markbates/goth v1.82.0github.com/modelcontextprotocol/go-sdk v1.6.1github.com/tracewayapp/lit/v2 v2.0.2- github.com/tracewayapp/traceway/cli v0.0.0-00010101000000-000000000000+ github.com/tracewayapp/traceway/cli v1.9.15go.opentelemetry.io/collector/component v1.60.0go.opentelemetry.io/collector/component/componenttest v0.154.0go.opentelemetry.io/collector/consumer v1.60.0@@ -199,4 +199,8 @@ require (golang.org/x/text v0.37.0 // indirect)+// Local development only. `replace` is ignored by anything that imports this+// module, so the `require` above must name a real published cli version or+// `go get github.com/tracewayapp/traceway/backend` fails to resolve for+// consumers. Bump that version when a release depends on newer cli code.replace github.com/tracewayapp/traceway/cli => ../clidiff --git a/cli/pkg/mcpserver/knowledge/debug-flow.md b/cli/pkg/mcpserver/knowledge/debug-flow.mdindex 49d85305..36437c16 100644--- a/cli/pkg/mcpserver/knowledge/debug-flow.md+++ b/cli/pkg/mcpserver/knowledge/debug-flow.md@@ -94,7 +94,42 @@ The CLI also accepts `p50|p95|p99`, but the server has no quantile aggregation f3. Form a hypothesis that explains the targeted occurrence's full observation set (its message, affected endpoint, timing, volume) — not just the first stack frame. When a URL/hash anchored you to the last occurrence (step 2), explain *that* error; do not stretch one hypothesis to cover sibling clusters that merely share the hash.4. Propose or implement the fix per the user's instruction, scoped to the targeted occurrence's failure path.-### 5. Report and clean up+### 5. When the evidence is not enough, instrument instead of guessing++Before writing a fix, answer one question honestly: **does the telemetry actually explain this failure?** You have enough when you can name the failing line, the input that reached it, and why that input was wrong. A stack trace on its own is usually not enough. Neither is a plausible story that the data neither confirms nor contradicts.++If you cannot explain it, do not ship a speculative fix. A guess that changes behaviour hides the symptom, destroys the next reproduction, and costs another incident cycle. Instead, add the instrumentation that would answer the open question, and let the next occurrence do the diagnosing.++Pick the signal that closes the specific gap:++| What you could not determine | What to add |+|---|---|+| Which branch ran, or what the inputs were | A log at the decision point with the deciding values as attributes. Use ERROR severity on the failure path so `--min-severity 17` finds it |+| Where the time went, or which step failed | Child spans around each suspect step, opened inside the request's active context |+| Who or what it happened to (user, tenant, release, payload shape) | Attributes on the request span, or attributes passed to `recordException` |+| How often it happens, or under which conditions | A counter or histogram metric, tagged with the low-cardinality dimension you want to slice by |+| Where it broke across a service boundary | Trace context propagation on the call, plus a span or log on both sides |+| Whether the code path runs at all | A log or counter at entry, so absence of data becomes evidence |++Rules for what you add:++- Emit it inside the active span of the request or task. That is what stamps the trace id on the log and makes it show up on the trace.+- Put identifiers in attributes, never in the message text. Messages are stripped before grouping, and attributes are filterable.+- Keep cardinality low. User ids and request ids belong in span attributes, never in a metric tag or a span name.+- Never log secrets, tokens, or personal data.+- Ship it as permanent instrumentation, not a temporary debug print. Something you remove tomorrow answers nothing.+- If the area has no instrumentation at all, set it up first with the `traceway-setup` skill rather than bolting on one log line.++Then close the loop with the user. State what you could not determine, what you added, and the exact command to run when it fires again:++```bash+traceway logs query --trace-id <id> --output json+traceway exceptions show <hash> --output json | jq '.occurrences | last | .attributes'+```++Adding instrumentation is a legitimate outcome of a debug session, not a failure to fix. Report it as the deliverable it is.++### 6. Report and clean upSummarize: symptom, evidence (exception hashes, log excerpts, metric anomalies), root cause, fix. Include `traceway exceptions show <hash>` references so the user can verify. After a fix is deployed and verified, archive only when the user asks:diff --git a/cli/pkg/mcpserver/knowledge/ground-rules.md b/cli/pkg/mcpserver/knowledge/ground-rules.mdindex 143f7985..090923e8 100644--- a/cli/pkg/mcpserver/knowledge/ground-rules.md+++ b/cli/pkg/mcpserver/knowledge/ground-rules.md@@ -6,3 +6,4 @@- **Time windows**: always bound queries, default `--since 1h` for "now" questions, `--since 24h` otherwise. `--since` accepts `s`, `m`, `h`, lowercase `Nd` (no `1w`, no `7d2h`). Absolute windows via `--from` / `--to` (RFC3339).- **Exit codes**: 0 ok, 1 generic/API, 2 usage, 3 connection, 4 auth, 5 not found, 6 rate limited, 7 server 5xx. Errors emit `{"error":"<stable_id>","message":"...","hint":"...","exit_code":N}` on stderr; branch on the `error` field.- On exit code 4 (auth), do not run `traceway login` yourself; switch to the Login flow and let the user enter credentials.+- **Never guess a fix**: if the telemetry does not explain the failure, do not change behaviour on a hunch. Add the instrumentation that would explain it next time, and say what is still unknown. See "When the evidence is not enough" in the Debug flow.diff --git a/cli/pkg/mcpserver/knowledge/performance-flow.md b/cli/pkg/mcpserver/knowledge/performance-flow.mdindex bce0a73b..fd2bf0ae 100644--- a/cli/pkg/mcpserver/knowledge/performance-flow.md+++ b/cli/pkg/mcpserver/knowledge/performance-flow.md@@ -11,6 +11,8 @@ The loop, using the read commands documented in this skill:3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <distributedTraceId> --recorded-at <t>` for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.-6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --aggregation max`, then `mem.used_pcnt`, `go.gc_pause`. A spike that lines up with the latency onset points at saturation, not a code bug.+6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --aggregation max`, then `mem.used`, `go.gc_pause`. A spike that lines up with the latency onset points at saturation, not a code bug.7. **Correlate with code and deploys.** With the onset time from step 3, check what shipped then: `git log --since '<onset - 30m>' --until '<onset + 30m>'` or the deploy history. A jump at a release is a regression; a gradual ramp with no deploy is data growth (N+1, missing index, no pagination).8. **Report.** The bottleneck, the evidence (endpoint percentiles, the onset time, the dominating span, any correlated metric anomaly), the root cause mapped to a checklist item, and a concrete fix, with the `endpoints show` / `traces show` references so the user can verify. If the endpoint is marked slow, state whether the latency exceeded the accepted offset.++If the waterfall does not localize the cost, stop before proposing a fix. An unexplained gap, one fat span covering the whole handler, or a tail you cannot reproduce all mean the same thing: the trace is too coarse to act on. Add child spans around the suspect steps inside that handler, then wait for the next slow request to point at the real one. See "When the evidence is not enough" in the Debug flow.diff --git a/cli/pkg/mcpserver/knowledge/performance.md b/cli/pkg/mcpserver/knowledge/performance.mdindex 1ae7a352..ad3a61de 100644--- a/cli/pkg/mcpserver/knowledge/performance.md+++ b/cli/pkg/mcpserver/knowledge/performance.md@@ -25,7 +25,7 @@ All commands are read-only. Never archive or mutate anything during a performanc4. **Separate code from saturation.** A slow endpoint can be a code bug or an overloaded host. Pull infrastructure and runtime metrics over the same window:```bashtraceway metrics query --name system.cpu.utilization --aggregation max --since 24h- traceway metrics query --name mem.used_pcnt --aggregation max --since 24h+ traceway metrics query --name mem.used --aggregation max --since 24htraceway metrics query --name go.gc_pause --aggregation max --since 24h```A metric spike that lines up with the latency rise points at saturation; flat metrics under rising latency point at a code or query problem.@@ -119,7 +119,7 @@ Each row: the suspect, and how to confirm it from Traceway telemetry.| **Synchronous work that belongs in a background task** | A long handler span runs before the response returns, for work the user does not wait on (emails, exports, image processing). It should be a `CaptureTask`. || **CPU-bound serialization** | A compute gap between IO spans (large JSON encode/decode); `system.cpu.utilization` up during the request. || **Inefficient algorithm / O(n^2) hot path** | Latency scales superlinearly with input size across traces; no IO to blame in the waterfall. |-| **Unbounded in-memory work / large allocations** | `mem.used_pcnt` and `go.num_gc` climb with p99; large compute spans. |+| **Unbounded in-memory work / large allocations** | `mem.used` and `go.num_gc` climb with p99; large compute spans. || **Excessive logging / telemetry in the hot path** | Many log lines per request in `logs query`; logging spans visible in the waterfall. |### Network, IO, and external dependencies@@ -143,12 +143,12 @@ Each row: the suspect, and how to confirm it from Traceway telemetry.| Suspect | How to confirm in Traceway ||---|---|| **CPU saturation** | `system.cpu.utilization` (max) spike lines up with the latency rise across many endpoints at once. |-| **Memory pressure / GC pauses** | `mem.used_pcnt`, `go.gc_pause`, `go.num_gc` correlate with p99; pauses visible as gaps in the waterfall. |+| **Memory pressure / GC pauses** | `mem.used`, `go.gc_pause`, `go.num_gc` correlate with p99; pauses visible as gaps in the waterfall. || **Disk / IO bottleneck** | `system.disk.*` or IO-wait metrics up; slow IO spans with no CPU cost. || **Undersized / noisy-neighbor instance** | Broad latency rise with no single span culprit; infra metrics elevated with no code change. |## Notes on the signals-- Host and runtime metrics live under `system.*` (Traceway OTel Agent) and the SDK runtime names (`go.gc_pause`, `mem.used_pcnt`, `go.go_routines`). The server has no quantile aggregation for metric points: `p50/p95/p99` on `metrics query` silently return `avg`, so never present those as percentiles. Real latency percentiles come only from `endpoints list`, computed from raw request durations.+- Host and runtime metrics live under `system.*` (Traceway OTel Agent) and the SDK runtime names (`go.gc_pause`, `mem.used`, `go.go_routines`). The server has no quantile aggregation for metric points: `p50/p95/p99` on `metrics query` silently return `avg`, so never present those as percentiles. Real latency percentiles come only from `endpoints list`, computed from raw request durations.- OTLP histogram metrics are stored as two series, `<name>.avg` and `<name>.count`. A bogus metric name returns an empty `series: {}` cleanly, so probing names is safe.- When the waterfall shows a gap with no span, that gap is the finding: it is time the request spent waiting (for a connection, a lock, the scheduler), not time spent doing work.diff --git a/cli/test/contract/go.mod b/cli/test/contract/go.modindex 0a2e9f72..7ff7ceac 100644--- a/cli/test/contract/go.mod+++ b/cli/test/contract/go.mod@@ -6,7 +6,7 @@ require (github.com/google/uuid v1.6.0github.com/modelcontextprotocol/go-sdk v1.6.1github.com/tracewayapp/traceway/backend v0.0.0- github.com/tracewayapp/traceway/cli v0.0.0+ github.com/tracewayapp/traceway/cli v1.9.15)require (diff --git a/docs/CLAUDE.md b/docs/CLAUDE.mdindex 20d10ff7..5ad756a5 100644--- a/docs/CLAUDE.md+++ b/docs/CLAUDE.md@@ -5,11 +5,18 @@- The header navbar items must always be aligned to the left (next to the logo), not centered or spread across the header. This is achieved via CSS in `styles/custom.css`.- Navbar order: Docs, Learn, Symbolicator, Protocol, Self Host (controlled by key order in `pages/_meta.json`).+## Backend docs live under `/client/otel`++- OpenTelemetry is the one backend integration path. Every OTel-based framework guide is a subfolder of `pages/client/otel/` (`nodejs`, `nestjs`, `nextjs`, `hono`, `cloudflare`, `symfony`, `laravel`, `python`, `django`), listed in `pages/client/otel/_meta.json`. A new backend framework guide goes there — it does **not** get a top-level `/client` entry, an `SDK_OPTIONS` value, or a `SDK_VISIBILITY` key.+- **The `/client` picker carries one backend card.** `components/FrameworkPicker.jsx` lists **OpenTelemetry** (href `/client/otel`) and nothing else for backends, matching the dashboard's project-creation picker, which offers `opentelemetry` as the only backend option. Per-framework backend cards were removed on purpose: the framework guides are reached from `/client/otel`. Do not re-add backend cards.+- **The native Go SDK is gone from the docs.** `client/sdk` and the five `*-middleware` folders were deleted, and `public/_redirects` 301s every one of those URLs to `/client/otel`. Go uses OpenTelemetry like every other backend. Do not re-add Go SDK pages, `go-*` SDK options, or `FOLDER_SDK` entries for them.+- Old flat URLs (`/client/django`, `/client/symfony`, …) are 301'd to the nested paths by `public/_redirects`, which Cloudflare serves from `out/`. Do not delete those lines when adding new ones.+## SDK Selection (path-derived)- The URL **path is the single source of truth** for the active SDK. There is **no `?sdk=` query parameter** — do not reintroduce one, and never append `?sdk=` to internal `/client` links.-- `resolveSdkFromPath(pathname, lastSdk)` in `components/SdkContext.jsx` maps the first path segment after `/client/` to an SDK via `FOLDER_SDK`. Folders that map 1:1 to a framework (e.g. `gin-middleware` → `go-gin`, `react` → `js-react`) are authoritative.-- Two folders are shared/ambiguous (`SHARED_FOLDERS`): `sdk` (Go SDK Reference, any `go-*`) and `js-sdk` (JS SDK Reference, the JS frontend group). These disambiguate via the remembered framework in `localStorage['traceway-docs-sdk']` (`lastSdk`) when it belongs to the folder's group, otherwise fall back to `go-generic` / `js-generic`.+- `resolveSdkFromPath(pathname, lastSdk)` in `components/SdkContext.jsx` maps the first path segment after `/client/` to an SDK via `FOLDER_SDK`. Folders that map 1:1 to a framework (e.g. `react` → `js-react`, `otel` → `otel`) are authoritative.+- One folder is shared/ambiguous (`SHARED_FOLDERS`): `js-sdk` (JS SDK Reference, the JS frontend group). It disambiguates via the remembered framework in `localStorage['traceway-docs-sdk']` (`lastSdk`) when that belongs to the group, otherwise falls back to `js-generic`.- `localStorage` stores only `lastSdk`. It is written when a specific SDK is resolved from the path, or when the user picks a framework from the dropdown (`SdkSelector`) or the `/client` picker (`FrameworkPicker`).- Sidebar item visibility (`SDK_VISIBILITY` in `theme.config.jsx`) and the dropdown label both derive from the path-resolved SDK. `SDK_VISIBILITY` must stay complete (every `/client/<folder>` needs an entry) and matching is by exact first path segment, not substring.- Stray `?sdk=` params on `/client` URLs are self-healed: `SdkProvider` strips them via a shallow `router.replace`, so old bookmarks/links correct themselves.diff --git a/docs/components/AgentSkillsCallout.jsx b/docs/components/AgentSkillsCallout.jsxnew file mode 100644index 00000000..872d7cdc--- /dev/null+++ b/docs/components/AgentSkillsCallout.jsx@@ -0,0 +1,76 @@+import { useState } from "react";++const COMMAND = "npx skills add tracewayapp/traceway";++export default function AgentSkillsCallout() {+ const [copied, setCopied] = useState(false);++ async function copyCommand() {+ try {+ await navigator.clipboard.writeText(COMMAND);+ setCopied(true);+ setTimeout(() => setCopied(false), 2000);+ } catch {+ setCopied(false);+ }+ }++ return (+ <div className="agent-skills">+ <p className="agent-skills-text">+ Using a coding agent?{" "}+ <a+ href="https://tracewayapp.com/product/agent-skills"+ target="_blank"+ rel="noreferrer"+ >+ Install the Traceway skills+ </a>{" "}+ and run <code>/traceway-setup</code> to have it wire this up for you.+ </p>+ <button+ type="button"+ className="agent-skills-cmd"+ onClick={copyCommand}+ title={copied ? "Copied" : "Copy install command"}+ >+ <span className="agent-skills-prompt" aria-hidden>+ $+ </span>+ <span className="agent-skills-cmd-text">{COMMAND}</span>+ {copied ? (+ <svg+ className="agent-skills-cmd-icon agent-skills-cmd-icon--ok"+ width="15"+ height="15"+ viewBox="0 0 24 24"+ fill="none"+ stroke="currentColor"+ strokeWidth="2"+ strokeLinecap="round"+ strokeLinejoin="round"+ aria-hidden+ >+ <path d="M20 6 9 17l-5-5" />+ </svg>+ ) : (+ <svg+ className="agent-skills-cmd-icon"+ width="15"+ height="15"+ viewBox="0 0 24 24"+ fill="none"+ stroke="currentColor"+ strokeWidth="2"+ strokeLinecap="round"+ strokeLinejoin="round"+ aria-hidden+ >+ <rect width="14" height="14" x="8" y="8" rx="2" ry="2" />+ <path d="M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2" />+ </svg>+ )}+ </button>+ </div>+ );+}diff --git a/docs/components/FrameworkPicker.jsx b/docs/components/FrameworkPicker.jsxindex a362e114..72ba6985 100644--- a/docs/components/FrameworkPicker.jsx+++ b/docs/components/FrameworkPicker.jsx@@ -2,128 +2,21 @@ import { useRouter } from "next/router";import { useSdk } from "./SdkContext";const FRAMEWORKS = [- {- value: "openrouter",- label: "OpenRouter",- description: "AI observability for OpenRouter with automatic OTLP trace export.",- icon: "/openrouter.png",- href: "/client/openrouter",- },+ // Backends are deliberately one card. OpenTelemetry is the only backend+ // option in the dashboard's project picker, so the docs picker matches it.+ // Per-framework guides (Node.js, NestJS, Next.js, Hono, Cloudflare, Symfony,+ // Laravel, Python, Django) are listed on /client/otel, not here.{value: "otel",- label: "OpenTelemetry (otel)",- description: "Send traces and metrics from any OTel-instrumented app to Traceway.",+ label: "OpenTelemetry",+ description: "Every backend, whatever the language: Go, Node.js, Python, PHP, Java, .NET, Ruby. Start here.",icon: "/otel.png",href: "/client/otel",},- {- value: "cloudflare",- label: "Cloudflare Workers",- description: "Cloudflare Workers with automatic request tracing via OTLP.",- icon: "/cloudflare.png",- href: "/client/cloudflare",- },- {- value: "php-symfony",- label: "Symfony",- description: "Symfony framework with OpenTelemetry auto-instrumentation.",- icon: "/symfony.png",- href: "/client/symfony",- },- {- value: "php-laravel",- label: "Laravel",- description: "Laravel framework with OpenTelemetry auto-instrumentation.",- icon: "/laravel.png",- href: "/client/laravel",- },- {- value: "python-django",- label: "Django",- description: "Django framework with OpenTelemetry auto-instrumentation.",- icon: "/django.png",- href: "/client/django",- },- {- value: "go-gin",- label: "Go Gin",- description:- "Gin Gonic web framework with automatic request tracing and panic recovery.",- icon: "/gin.png",- href: "/client/gin-middleware",- },- {- value: "go-chi",- label: "Go Chi",- description:- "Lightweight Chi router with automatic request tracing and panic recovery.",- icon: "/chi.png",- href: "/client/chi-middleware",- },- {- value: "go-fiber",- label: "Go Fiber",- description:- "Express-inspired Fiber framework with request tracing and error capture.",- icon: "/fiber.svg",- href: "/client/fiber-middleware",- },- {- value: "go-fasthttp",- label: "Go FastHTTP",- description:- "High-performance FastHTTP server with request tracing and panic recovery.",- icon: "/fasthttp.png",- href: "/client/fasthttp-middleware",- },- {- value: "go-http",- label: "Go net/http",- description:- "Standard library HTTP middleware for request tracing and error capture.",- icon: "/stdlib.png",- href: "/client/http-middleware",- },- {- value: "go-generic",- label: "Go Generic",- description:- "Framework-agnostic SDK for manual instrumentation of any Go application.",- icon: "/custom.png",- href: "/client/sdk",- },- {- value: "js-nextjs",- label: "Next.js",- description: "Next.js applications with OpenTelemetry auto-instrumentation.",- icon: "/nextjs.png",- href: "/client/nextjs",- },- {- value: "js-node",- label: "Node.js",- description: "Node.js backend with OpenTelemetry traces and metrics.",- icon: "/node.png",- href: "/client/node-sdk",- },- {- value: "js-nestjs",- label: "NestJS",- description: "NestJS framework with OpenTelemetry auto-instrumentation.",- icon: "/nestjs.png",- href: "/client/nestjs",- },- {- value: "js-hono",- label: "Hono",- description: "Lightweight multi-runtime framework with OpenTelemetry.",- icon: "/hono.png",- href: "/client/hono",- },{value: "js-react",label: "React",- description: "React applications with error boundaries and hooks.",+ description: "React apps in the browser, including Next.js and Remix front ends. Their server code uses OpenTelemetry.",icon: "/react.png",href: "/client/react",},@@ -162,6 +55,13 @@ const FRAMEWORKS = [icon: "/flutter.png",href: "/client/flutter",},+ {+ value: "react-native",+ label: "React Native",+ description: "React Native and Expo apps with automatic exception, fetch / XHR, and console capture. Works in Expo Go.",+ icon: "/react.png",+ href: "/client/react-native",+ },{value: "android",label: "Android",@@ -178,11 +78,11 @@ const FRAMEWORKS = [href: "/client/ios",},{- value: "react-native",- label: "React Native",- description: "React Native and Expo apps with automatic exception, fetch / XHR, and console capture. Works in Expo Go.",- icon: "/react.png",- href: "/client/react-native",+ value: "openrouter",+ label: "OpenRouter",+ description: "AI observability for OpenRouter with automatic OTLP trace export.",+ icon: "/openrouter.png",+ href: "/client/openrouter",},];@@ -198,19 +98,16 @@ export default function FrameworkPicker() {return (<div className="framework-picker"><h2 className="framework-picker-heading">Choose your framework</h2>- <p className="framework-picker-subheading">- Select the framework you're using to get started with Traceway.- </p><div className="framework-picker-grid">{FRAMEWORKS.map((fw) => (<button- key={fw.value}+ key={fw.href}className="framework-picker-card"onClick={() => handleSelect(fw)}><imgsrc={fw.icon}- alt={fw.label}+ alt=""className={`framework-picker-icon${fw.iconClassName ? ` ${fw.iconClassName}` : ""}`}/><span className="framework-picker-label">{fw.label}</span>diff --git a/docs/components/SdkContext.jsx b/docs/components/SdkContext.jsxindex a0c3e2b4..bfb7ac3e 100644--- a/docs/components/SdkContext.jsx+++ b/docs/components/SdkContext.jsx@@ -1,32 +1,21 @@import { createContext, useContext, useState, useEffect, useCallback } from 'react'import { useRouter } from 'next/router'+// One entry per integration path. Every backend (Go, Node, Python, PHP, Java,+// .NET, Ruby) is the single `otel` option, and the per-framework guides live+// inside /client/otel.export const SDK_OPTIONS = [- { value: 'go-gin', label: 'Go Gin' },- { value: 'go-chi', label: 'Go Chi' },- { value: 'go-fiber', label: 'Go Fiber' },- { value: 'go-fasthttp', label: 'Go FastHTTP' },- { value: 'go-http', label: 'Go Http' },- { value: 'go-generic', label: 'Go Generic' },- { value: 'js-node', label: 'Node.js (OTel)' },- { value: 'js-nestjs', label: 'NestJS (OTel)' },- { value: 'js-hono', label: 'Hono' },+ { value: 'otel', label: 'OpenTelemetry (any backend)' },{ value: 'js-react', label: 'React' },{ value: 'js-vue', label: 'Vue.js' },{ value: 'js-svelte', label: 'Svelte' },{ value: 'js-jquery', label: 'jQuery' },{ value: 'js-generic', label: 'JS Generic' },- { value: 'openrouter', label: 'OpenRouter' },- { value: 'otel', label: 'OpenTelemetry (otel)' },- { value: 'cloudflare', label: 'Cloudflare Workers' },- { value: 'js-nextjs', label: 'Next.js (OTel)' },- { value: 'php-symfony', label: 'Symfony' },- { value: 'php-laravel', label: 'Laravel' },- { value: 'python-django', label: 'Django' },{ value: 'flutter', label: 'Flutter' },{ value: 'android', label: 'Android' },{ value: 'ios', label: 'iOS' },{ value: 'react-native', label: 'React Native' },+ { value: 'openrouter', label: 'OpenRouter' },]const STORAGE_KEY = 'traceway-docs-sdk'@@ -35,39 +24,22 @@ const VALID_VALUES = new Set(SDK_OPTIONS.map((o) => o.value))// First path segment after /client/ → exact SDK value. These folders map 1:1// to a single framework, so the path alone is authoritative.const FOLDER_SDK = {- 'gin-middleware': 'go-gin',- 'chi-middleware': 'go-chi',- 'fiber-middleware': 'go-fiber',- 'fasthttp-middleware': 'go-fasthttp',- 'http-middleware': 'go-http',- 'node-sdk': 'js-node',- 'nestjs': 'js-nestjs',- 'hono': 'js-hono',- 'nextjs': 'js-nextjs',+ 'otel': 'otel','react': 'js-react','react-native': 'react-native','vue': 'js-vue','svelte': 'js-svelte','jquery': 'js-jquery','openrouter': 'openrouter',- 'otel': 'otel',- 'cloudflare': 'cloudflare',- 'symfony': 'php-symfony',- 'laravel': 'php-laravel',- 'django': 'python-django','flutter': 'flutter','android': 'android','ios': 'ios',}-// Shared reference folders — the SDK cannot be derived from the path alone.-// Disambiguated by the remembered framework (lastSdk) when it belongs to the-// folder's group, otherwise by a generic fallback.+// Shared reference folders. The SDK cannot be derived from the path alone, so+// it is disambiguated by the remembered framework (lastSdk) when that belongs+// to the folder's group, otherwise by a generic fallback.const SHARED_FOLDERS = {- 'sdk': {- members: new Set(['go-gin', 'go-chi', 'go-fiber', 'go-fasthttp', 'go-http', 'go-generic']),- fallback: 'go-generic',- },'js-sdk': {members: new Set(['js-react', 'js-vue', 'js-svelte', 'js-jquery', 'js-generic']),fallback: 'js-generic',diff --git a/docs/components/SdkSelector.jsx b/docs/components/SdkSelector.jsxindex 6e9ad377..a978d829 100644--- a/docs/components/SdkSelector.jsx+++ b/docs/components/SdkSelector.jsx@@ -3,30 +3,17 @@ import { useRouter } from 'next/router'import { useSdk, SDK_OPTIONS } from './SdkContext'const SDK_QUICK_START = {- 'go-gin': '/client/gin-middleware',- 'go-chi': '/client/chi-middleware',- 'go-fiber': '/client/fiber-middleware',- 'go-fasthttp': '/client/fasthttp-middleware',- 'go-http': '/client/http-middleware',- 'go-generic': '/client/sdk',- 'js-node': '/client/node-sdk',- 'js-nestjs': '/client/nestjs',- 'js-hono': '/client/hono',- 'js-nextjs': '/client/nextjs',+ 'otel': '/client/otel','js-react': '/client/react','js-vue': '/client/vue','js-svelte': '/client/svelte','js-jquery': '/client/jquery','js-generic': '/client/js-sdk',- 'openrouter': '/client/openrouter',- 'otel': '/client/otel',- 'cloudflare': '/client/cloudflare',- 'php-symfony': '/client/symfony',- 'php-laravel': '/client/laravel',- 'python-django': '/client/django','flutter': '/client/flutter','android': '/client/android',+ 'ios': '/client/ios','react-native': '/client/react-native',+ 'openrouter': '/client/openrouter',}export default function SdkSelector() {diff --git a/docs/pages/client/_meta.json b/docs/pages/client/_meta.jsonindex 90d63527..1a20b464 100644--- a/docs/pages/client/_meta.json+++ b/docs/pages/client/_meta.json@@ -5,27 +5,13 @@"display": "hidden","theme": { "sidebar": false, "pagination": false }},- "gin-middleware": "Gin Middleware",- "chi-middleware": "Chi Middleware",- "fiber-middleware": "Fiber Middleware",- "fasthttp-middleware": "FastHTTP Middleware",- "http-middleware": "HTTP Middleware",- "sdk": "SDK Reference",- "node-sdk": "Node.js (OTel)",- "nestjs": "NestJS (OTel)",- "hono": "Hono",- "nextjs": "Next.js (OTel)",+ "otel": "OpenTelemetry","react": "React","vue": "Vue.js","svelte": "Svelte","jquery": "jQuery","js-sdk": "JS SDK Reference","openrouter": "OpenRouter",- "otel": "OpenTelemetry",- "cloudflare": "Cloudflare Workers",- "symfony": "Symfony",- "laravel": "Laravel",- "django": "Django","flutter": "Flutter","android": "Android","ios": "iOS",diff --git a/docs/pages/client/android/index.mdx b/docs/pages/client/android/index.mdxindex 2dfae1a8..8a0ce565 100644--- a/docs/pages/client/android/index.mdx+++ b/docs/pages/client/android/index.mdx@@ -35,17 +35,18 @@ class MyApp : Application() {}```-Register the application class in `AndroidManifest.xml` and add the `INTERNET` permission so reports can leave the device:+Register the application class in `AndroidManifest.xml`, or `onCreate()` never runs and the SDK is never initialized:```xml<manifest xmlns:android="http://schemas.android.com/apk/res/android">- <uses-permission android:name="android.permission.INTERNET" /><application android:name=".MyApp" ...>...</application></manifest>```+You do not need to add a permission. The library's own manifest declares `INTERNET` and `ACCESS_NETWORK_STATE`, and the Android manifest merger folds both into your app. Re-declaring `INTERNET` yourself is harmless if your app already has it.+## Capture Errors Manually```kotlin@@ -114,7 +115,7 @@ Every captured exception ships with the last ~10 seconds of session context, attEvery `println` and any other write to `System.out` / `System.err` is mirrored into the log buffer. Console output behaves normally; it just gets a copy.-Calls that go straight to `android.util.Log` bypass `System.out` and are **not** captured automatically. To pick up Logcat lines, install a Timber tree (or any other logger) that calls `TracewayClient.instance?.recordLog(message, level)`:+Calls that go straight to `android.util.Log` bypass `System.out` and are **not** captured automatically. To pick up Logcat lines, install a Timber tree (or any other logger) that calls `TracewayClient.instance?.recordLog(message, level)`. Timber is not pulled in by the SDK, so add it yourself with `implementation("com.jakewharton.timber:timber:5.0.1")`:```kotlinimport android.util.Log@@ -141,14 +142,22 @@ Three kinds of actions are collected:- **Network**: every HTTP request routed through an OkHttp client with `TracewayOkHttpInterceptor` installed (method, URL, status, duration, request/response byte counts). Catches Retrofit, Coil, Glide-via-OkHttp, and your own usage:```kotlin+ import com.tracewayapp.traceway.network.TracewayOkHttpInterceptor+ import okhttp3.OkHttpClient+val client = OkHttpClient.Builder().addInterceptor(TracewayOkHttpInterceptor()).build()```+ The SDK's only declared dependency is `kotlin-stdlib`, so the interceptor compiles against your app's own OkHttp (`implementation("com.squareup.okhttp3:okhttp:4.12.0")`).+For HTTP clients that aren't OkHttp, record events manually:```kotlin+ import com.tracewayapp.traceway.TracewayClient+ import com.tracewayapp.traceway.events.NetworkEvent+TracewayClient.instance?.recordNetworkEvent(NetworkEvent(method = "GET", url = "https://...", durationMs = 42L, statusCode = 200))@@ -225,7 +234,7 @@ Release builds run R8, so crash traces arrive with renamed classes and rewritten```kotlinTraceway.init(application = this,- connectionString = BuildConfig.TRACEWAY_DSN,+ connectionString = "your-token@https://traceway.example.com/api/report",options = TracewayOptions(version = "1.0.0",proguardUuid = BuildConfig.TRACEWAY_PROGUARD_UUID,@@ -233,17 +242,28 @@ Traceway.init()```+`TRACEWAY_PROGUARD_UUID` is the only field the plugin adds to `BuildConfig`. To keep the connection string out of source too, declare your own field with `buildConfigField("String", "TRACEWAY_DSN", "\"$tracewayDsn\"")` and read it back as `BuildConfig.TRACEWAY_DSN`.+See the [Android symbolicator docs](/symbolicator/android) for the full plugin configuration, the upload token, and the manual upload path.## Test Your Integration-Add a button that throws an exception to verify everything is wired up:+Make your activity show a button that throws, so you can verify everything is wired up:```kotlin-Button(this).apply {- text = "Send Test Error"- setOnClickListener {- throw RuntimeException("Test error from Traceway")+import android.app.Activity+import android.os.Bundle+import android.widget.Button++class MainActivity : Activity() {+ override fun onCreate(savedInstanceState: Bundle?) {+ super.onCreate(savedInstanceState)+ setContentView(Button(this).apply {+ text = "Send Test Error"+ setOnClickListener {+ throw RuntimeException("Test error from Traceway")+ }+ })}}```diff --git a/docs/pages/client/chi-middleware/_meta.json b/docs/pages/client/chi-middleware/_meta.jsondeleted file mode 100644index a5183284..00000000--- a/docs/pages/client/chi-middleware/_meta.json+++ /dev/null@@ -1,5 +0,0 @@-{- "index": "Quick Start",- "configuration": { "display": "hidden" },- "error-recording": "Error Recording"-}diff --git a/docs/pages/client/chi-middleware/configuration.mdx b/docs/pages/client/chi-middleware/configuration.mdxdeleted file mode 100644index 0b46e0b0..00000000--- a/docs/pages/client/chi-middleware/configuration.mdx+++ /dev/null@@ -1,188 +0,0 @@-# Chi Middleware Configuration--## Middleware Options--| Option | Description | Default |-|--------|-------------|---------|-| `WithRepanic(bool)` | Re-panic after capturing, letting upstream recovery handle it | `true` |-| `WithIgnoredPaths(...string)` | Skip recording for specific URL paths | none |-| `WithStreamingPaths(...string)` | Mark paths as long-lived streaming responses (SSE / WS / long-poll) | none |-| `WithFilter(func(*http.Request) bool)` | Custom filter function — return `false` to skip recording | none |-| `WithOnErrorRecording(RecordingFlag)` | What additional data to capture on errors | `0` (none) |--## Passthrough Core Options--These options are forwarded to the core `traceway` client:--| Option | Description | Default |-|--------|-------------|---------|-| `WithDebug(bool)` | Enable debug logging | `false` |-| `WithVersion(string)` | Application version shown in dashboard | `""` |-| `WithServerName(string)` | Server hostname identifier | Auto-detected |-| `WithSampleRate(float64)` | Percentage of normal traces to record (0.0–1.0) | `1.0` |-| `WithErrorSampleRate(float64)` | Percentage of error traces to record (0.0–1.0) | `1.0` |-| `WithCollectionInterval(time.Duration)` | How often batched data is sent | `5s` |-| `WithMaxCollectionFrames(int)` | Maximum frames to buffer | `12` |-| `WithUploadTimeout(time.Duration)` | HTTP upload timeout | `2s` |-| `WithMetricsInterval(time.Duration)` | System metrics collection interval | `30s` |--## Repanic Behavior--When `WithRepanic(true)` (default), the middleware re-panics after capturing the stack trace. This allows upstream recovery middleware (like Chi's `middleware.Recoverer`) to handle the panic as well.--When `WithRepanic(false)`, the middleware responds with HTTP 500 and does not re-panic.--## Path Filtering--### WithIgnoredPaths--Skip recording for specific paths:--```go-tracewaychi.New(- connectionString,- tracewaychi.WithIgnoredPaths("/health", "/ready", "/metrics"),-)-```--Ignored paths are still served — the request passes through to your handler, but no trace is recorded.--### WithFilter--For dynamic filtering logic, use `WithFilter`. Return `false` to skip recording:--```go-tracewaychi.New(- connectionString,- tracewaychi.WithFilter(func(r *http.Request) bool {- // Skip internal service-to-service calls- if r.Header.Get("X-Internal") == "true" {- return false- }- // Skip static assets- if strings.HasPrefix(r.URL.Path, "/static/") {- return false- }- return true- }),-)-```--When the filter returns `false`, the request is passed directly to your handler without any Traceway wrapping.--## Streaming Responses (SSE / WebSocket / long-poll)--Long-lived streaming routes have a request lifetime that equals the connection duration, not request processing latency. Recording them as normal endpoints would dominate P50/P95/P99 rankings, tank Apdex, and trigger slow-endpoint alerts.--The middleware **auto-detects** streaming responses from the response headers:--| Signal | Detection |-|--------|-----------|-| Server-Sent Events | `Content-Type: text/event-stream` |-| WebSocket upgrade | HTTP status `101 Switching Protocols`, or `Upgrade: websocket` |--Streaming requests are still recorded — status code, error rate, and throughput remain visible — but their duration is **excluded** from P50/P95/P99, Apdex, and impact scoring.--Declare paths the SDK can't auto-detect (long-polling, chunked JSON streams):--```go-tracewaychi.New(- connectionString,- tracewaychi.WithStreamingPaths("/api/long-poll", "/api/stream/json"),-)-```--Or mark dynamically from any handler:--```go-func eventsHandler(w http.ResponseWriter, r *http.Request) {- traceway.MarkStream(r.Context())- // ... write stream ...-}-```--## Complete Example--```go-package main--import (- "net/http"- "time"-- "github.com/go-chi/chi/v5"- traceway "go.tracewayapp.com"- tracewaychi "go.tracewayapp.com/tracewaychi"-)--func main() {- traceway.ConfigureAttributes(func(s *traceway.Attributes) {- s.SetTag("environment", "production")- s.SetTag("service", "user-api")- })-- r := chi.NewRouter()-- r.Use(tracewaychi.New(- "your-token@http://localhost:8082/api/report",- tracewaychi.WithVersion("1.2.3"),- tracewaychi.WithServerName("api-server-01"),- tracewaychi.WithRepanic(true),- tracewaychi.WithIgnoredPaths("/health", "/ready"),- tracewaychi.WithFilter(func(r *http.Request) bool {- return r.Header.Get("X-Internal") != "true"- }),- tracewaychi.WithOnErrorRecording(- tracewaychi.RecordingUrl | tracewaychi.RecordingQuery,- ),- tracewaychi.WithCollectionInterval(5 * time.Second),- tracewaychi.WithMetricsInterval(30 * time.Second),- ))-- r.Get("/api/users/{id}", getUser)- http.ListenAndServe(":8080", r)-}-```--## Configuration Recommendations--### Development--```go-tracewaychi.New(- connectionString,- tracewaychi.WithDebug(true),- tracewaychi.WithOnErrorRecording(- tracewaychi.RecordingUrl |- tracewaychi.RecordingQuery |- tracewaychi.RecordingHeader |- tracewaychi.RecordingBody,- ),-)-```--### Production--```go-tracewaychi.New(- connectionString,- tracewaychi.WithVersion(os.Getenv("APP_VERSION")),- tracewaychi.WithServerName(os.Getenv("HOSTNAME")),- tracewaychi.WithOnErrorRecording(- tracewaychi.RecordingUrl | tracewaychi.RecordingQuery,- ),-)-```--### High-Traffic Services--```go-tracewaychi.New(- connectionString,- tracewaychi.WithSampleRate(0.1),- tracewaychi.WithErrorSampleRate(1.0),- tracewaychi.WithCollectionInterval(10 * time.Second),- tracewaychi.WithMaxCollectionFrames(20),- tracewaychi.WithMetricsInterval(60 * time.Second),-)-```diff --git a/docs/pages/client/chi-middleware/error-recording.mdx b/docs/pages/client/chi-middleware/error-recording.mdxdeleted file mode 100644index 78adcfd0..00000000--- a/docs/pages/client/chi-middleware/error-recording.mdx+++ /dev/null@@ -1,83 +0,0 @@-# Chi Error Recording--When an exception occurs during a request, you can control what additional request data is captured alongside the stack trace.--## Recording Flags--Configure error recording with `WithOnErrorRecording`:--```go-tracewaychi.New(- connectionString,- tracewaychi.WithOnErrorRecording(- tracewaychi.RecordingUrl | tracewaychi.RecordingQuery,- ),-)-```--### Available Flags--| Flag | Description | Privacy Impact |-|------|-------------|----------------|-| `RecordingUrl` | Request URL path | Low |-| `RecordingQuery` | Query parameters as JSON | Medium |-| `RecordingBody` | Request body (JSON content-type only, max 64KB) | High |-| `RecordingHeader` | Request headers as JSON | High |--### Combining Flags--Flags are combined with the `|` operator:--```go-// Record URL and query parameters-tracewaychi.RecordingUrl | tracewaychi.RecordingQuery--// Record everything-tracewaychi.RecordingUrl | tracewaychi.RecordingQuery | tracewaychi.RecordingBody | tracewaychi.RecordingHeader-```--## Body Capture Details--Body recording has two constraints:--- **Content-Type must be `application/json`** — non-JSON bodies are not captured-- **Maximum size is 64KB** — bodies larger than 64KB are truncated--After reading the body for capture, it is **restored** so downstream handlers can still read it.--## Privacy Considerations--Body and header recording may capture sensitive data:--- Authentication tokens in headers-- Personal information in request bodies-- API keys in query parameters--Use minimal recording flags in production:--```go-// Production — limited data-tracewaychi.WithOnErrorRecording(- tracewaychi.RecordingUrl | tracewaychi.RecordingQuery,-)--// Development — full request data-tracewaychi.WithOnErrorRecording(- tracewaychi.RecordingUrl |- tracewaychi.RecordingQuery |- tracewaychi.RecordingBody |- tracewaychi.RecordingHeader,-)-```--## Captured Data Format--Recorded data is attached as attributes on the issue:--| Attribute Key | Value |-|---------------|-------|-| `url` | URL path string |-| `query` | JSON-encoded query parameters |-| `body` | Raw request body string |-| `headers` | JSON-encoded request headers |-| `user agent` | User-Agent header value (always captured on error) |diff --git a/docs/pages/client/chi-middleware/index.mdx b/docs/pages/client/chi-middleware/index.mdxdeleted file mode 100644index 05f6174a..00000000--- a/docs/pages/client/chi-middleware/index.mdx+++ /dev/null@@ -1,106 +0,0 @@-# Chi Middleware--import { Callout } from "nextra/components";--<Callout type="warning">- The Traceway Go SDK is no longer the recommended way to integrate with Traceway. For new projects, instrument with [OpenTelemetry](/client/otel) — it works across languages and Traceway natively ingests OTLP. The Go SDK is still supported, but new features land in the OTel path first.-</Callout>--The `tracewaychi` package provides middleware for the [Chi](https://github.com/go-chi/chi) router. It automatically creates a trace for each request, captures panics, and detects client IP addresses.--Chi is fully compatible with `net/http` middleware, so `tracewaychi` delegates to the `tracewayhttp` package internally while providing a Chi-specific API surface.--## Installation--```bash-go get go.tracewayapp.com-go get go.tracewayapp.com/tracewaychi-```--## Basic Setup--```go-package main--import (- "net/http"-- "github.com/go-chi/chi/v5"- tracewaychi "go.tracewayapp.com/tracewaychi"-)--func main() {- r := chi.NewRouter()-- // Add Traceway middleware (must be before routes)- r.Use(tracewaychi.New(- "your-token@http://localhost:8082/api/report",- ))-- r.Get("/api/users", getUsers)- r.Post("/api/users", createUser)-- http.ListenAndServe(":8080", r)-}-```--`New` returns `func(http.Handler) http.Handler` and calls `traceway.Init` internally — do not call `Init` separately.--## What Gets Captured--The middleware automatically captures for every request:--| Data | Description |-| --------------- | ----------------------------------------------- |-| **Endpoint** | HTTP method + URL path (e.g., `GET /api/users`) |-| **Duration** | Request processing time |-| **Status Code** | HTTP response status |-| **Body Size** | Response body size in bytes |-| **Client IP** | Detected from headers or connection |-| **Panics** | Recovered and captured with full stack trace |--## Client IP Detection--Client IP is resolved in this order:--1. `X-Forwarded-For` header (first IP in the list)-2. `X-Real-IP` header-3. `RemoteAddr` from the connection--## Error Detection--A request is considered an error when:--- A **panic** occurs during handler execution-- The response status code is **>= 500**--Error traces use the error sample rate for sampling decisions.--## Panic Handling--When a panic occurs, the middleware:--1. Recovers the panic-2. Captures the full stack trace as an issue-3. Either re-panics (default, `WithRepanic(true)`) or responds with 500--## Connection package--The connection page in the sidebar will have your projects unique API key and the integration instructions.--<img- src="/connection.png"- alt="SDK Connection Setup"- style={{- maxWidth: "100%",- height: "auto",- marginTop: "16px",- borderRadius: "8px",- border: "1px solid #e5e7eb",- }}-/>--## Next Steps--- [Configuration](/client/chi-middleware/configuration) — All options including filtering and passthrough settings-- [Error Recording](/client/chi-middleware/error-recording) — Control what request data is captured on errorsdiff --git a/docs/pages/client/cloudflare/index.mdx b/docs/pages/client/cloudflare/index.mdxdeleted file mode 100644index e6d08538..00000000--- a/docs/pages/client/cloudflare/index.mdx+++ /dev/null@@ -1,94 +0,0 @@-# Cloudflare Workers--Cloudflare Workers has built-in observability that can export traces and logs via OTLP. No SDK or code changes are needed in your Worker — just configure a destination in the Cloudflare dashboard and enable it in your `wrangler.jsonc`.--## Traces--### Step 1: Create a traces destination in the Cloudflare dashboard--Navigate to **Compute (Workers) → Observability → Destinations**, then click **"+ Add Destination"**.----Fill in the following fields:--- **Destination Type**: Traces-- **OTLP Endpoint**: `https://cloud.tracewayapp.com/api/otel/v1/traces` (or your self-hosted equivalent)-- **Destination Name**: `traceway-traces`-- **Custom Header**: `Authorization` = `Bearer <project_token>`--Replace `<project_token>` with your project token from the Traceway dashboard under **Project Settings**.----### Step 2: Enable traces in wrangler.jsonc--Add the `observability.traces` configuration to your `wrangler.jsonc`, referencing the destination name you created above:--```jsonc-{- "observability": {- "traces": {- "enabled": true,- "destinations": ["traceway-traces"],- "head_sampling_rate": 1- }- }-}-```--Set `head_sampling_rate` to a value between `0` and `1` to control what fraction of requests are traced. Use `1` to trace every request.--## Logs--Logs work exactly like traces — create a second destination in the dashboard, add a second block to `wrangler.jsonc`, and redeploy. No Worker code changes.--### Step 1: Create a logs destination in the Cloudflare dashboard--Back in **Workers → Observability → Destinations**, click **"+ Add Destination"** a second time and fill in:--- **Destination Type**: Logs-- **OTLP Endpoint**: `https://cloud.tracewayapp.com/api/otel/v1/logs` (or your self-hosted equivalent)-- **Destination Name**: `traceway-logs`-- **Custom Header**: `Authorization` = `Bearer <project_token>`--Use the same project token as the traces destination.--### Step 2: Enable logs in wrangler.jsonc--Add the `observability.logs` block alongside `observability.traces`:--```jsonc-{- "observability": {- "traces": {- "enabled": true,- "destinations": ["traceway-traces"],- "head_sampling_rate": 1- },- "logs": {- "enabled": true,- "destinations": ["traceway-logs"],- "head_sampling_rate": 1,- "persist": false- }- }-}-```--- `head_sampling_rate` (0 to 1) controls what fraction of logs are exported. Use `1` to export every log.-- `persist: false` disables Cloudflare's own dashboard log storage to reduce billing when Traceway is your primary destination. Set to `true` (or omit) if you still want to view logs in the Cloudflare dashboard as well.--When both traces and logs are enabled, each log carries the `trace_id` and `span_id` of the request that produced it — so Traceway shows those logs on the matching trace detail page automatically.--## Deploy--Deploy your Worker:--```bash-npx wrangler deploy-```--Traces and logs will appear in Traceway automatically.--See [OTel Logs](/client/otel/logs) for the full OTLP logs mapping and severity reference.diff --git a/docs/pages/client/django/index.mdx b/docs/pages/client/django/index.mdxdeleted file mode 100644index 4fd49a2b..00000000--- a/docs/pages/client/django/index.mdx+++ /dev/null@@ -1,208 +0,0 @@-# Django--The official [`opentelemetry-instrumentation-django`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-django) package (from `opentelemetry-python-contrib`) automatically instruments your Django application and exports traces, metrics, and logs to Traceway's OTLP endpoints — no manual instrumentation needed.--## Prerequisites--- Python 3.9+-- Django 4.x or 5.x-- pip-- A Traceway project (create one in the dashboard)--## Step 1: Install Packages--```bash-pip install \- opentelemetry-distro \- opentelemetry-exporter-otlp \- opentelemetry-instrumentation-django-```--Then let the OTel bootstrapper auto-install the rest of the instrumentation that matches your installed dependencies (Postgres / MySQL / Redis / `requests` / `urllib3` / Celery / logging / …):--```bash-opentelemetry-bootstrap -a install-```--`opentelemetry-bootstrap` inspects your `site-packages` and only installs instrumentation packages that have a matching library present — re-run it whenever you add or remove dependencies.--## Step 2: Configure Environment Variables--Add the following to your `.env` (or your container / systemd unit / `manage.py` shell), replacing the endpoint and token with your project values:--```bash-OTEL_SERVICE_NAME=my-django-app-OTEL_TRACES_EXPORTER=otlp-OTEL_METRICS_EXPORTER=otlp-OTEL_LOGS_EXPORTER=otlp-OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf-OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel-OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20your-project-token--# Forward Python logging records (Django + your loggers) as OTel log records.-OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true-```--Your Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.--> **Note:** `OTEL_EXPORTER_OTLP_HEADERS` is a comma-separated list of `key=value` pairs, so the `Bearer ` prefix's space must be URL-encoded as `%20`. The exporter URL-decodes header values automatically.--## Step 3: Launch Django Through the OTel Agent--Don't run `manage.py runserver` (or your WSGI/ASGI server) directly — wrap it with `opentelemetry-instrument` so the agent can patch Django, the database driver, and the HTTP client libraries before your app imports them:--```bash-# Development-opentelemetry-instrument python manage.py runserver--# Production (Gunicorn / WSGI)-opentelemetry-instrument gunicorn myproject.wsgi:application--# Production (Uvicorn / ASGI)-opentelemetry-instrument uvicorn myproject.asgi:application-```--That's it — `DjangoInstrumentor` injects its middleware at index `0` of the `MIDDLEWARE` list at startup, so every inbound HTTP request is now traced. No `settings.py` edits required.--> **Want to instrument manually instead of via the agent?** You can call `DjangoInstrumentor().instrument()` from a tiny entrypoint module (e.g. `myproject/otel.py`) that you import at the top of `manage.py` and `wsgi.py`. See [Manual Initialization](#manual-initialization) below — using `opentelemetry-instrument` is strongly preferred because it also wires up exporters and resource detection for you.--## What Gets Captured--Once configured, the Django instrumentation automatically captures:--- **Endpoints** — every HTTP request with method and route template (e.g., `GET /users/<int:id>/`), set as `http.route`-- **Status codes** — 2xx, 4xx, 5xx responses-- **Exceptions** — unhandled exceptions are recorded on the request span via `span.record_exception()` and the span is marked `ERROR`-- **Active request counter** — concurrent in-flight request count, exported as a metric-- **Duration histogram** — request latency histogram (`http.server.request.duration`)--And once `opentelemetry-bootstrap -a install` has been run, the matching auto-instrumentations also capture:--| Library | What it captures |-|---------|------------------|-| `psycopg2` / `psycopg` / `mysqlclient` / `pymysql` / `sqlite3` | Every DB query — SQL statement, duration, error |-| `redis` / `aioredis` | Redis commands (GET/SET/…) |-| `requests` / `urllib` / `urllib3` / `httpx` | Outbound HTTP — propagates W3C trace context |-| `celery` | Producer + consumer spans for queued tasks — see [Tasks](/client/django/tasks) |-| `logging` | Standard `logging` records as OTel logs (when `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`) |-| `boto3` / `pika` / `kafka-python` / `pymongo` / `elasticsearch` / … | Bundled in `opentelemetry-bootstrap`'s registry; auto-installed if the library is present |--Disable any instrumentation you don't want by uninstalling its package, or by setting the matching `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS` env var:--```bash-# Comma-separated entry-point names — e.g. disable Redis + outbound requests-OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis,requests-```--## Tuning the Django Instrumentation--The middleware reads a handful of environment variables at startup:--| Variable | Default | Purpose |-|----------|---------|---------|-| `OTEL_PYTHON_DJANGO_INSTRUMENT` | `True` | Set to `False` to disable Django instrumentation while keeping the rest. |-| `OTEL_PYTHON_DJANGO_EXCLUDED_URLS` | _(unset)_ | Comma-separated regexes — matching paths are not traced. Example: `healthcheck,client/.*/info`. |-| `OTEL_PYTHON_DJANGO_TRACED_REQUEST_ATTRS` | _(unset)_ | Comma-separated `HttpRequest` attributes to attach as span attributes (e.g. `path_info,content_type`). |-| `OTEL_PYTHON_DJANGO_MIDDLEWARE_POSITION` | `0` | Index in the `MIDDLEWARE` list where the OTel middleware is inserted. |-| `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST` | _(unset)_ | Request headers to capture (regex list). Captured as `http.request.header.<lowercased_name>`. |-| `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE` | _(unset)_ | Response headers to capture (regex list). |-| `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS` | _(unset)_ | Regexes of header names whose values should be redacted (e.g. `.*session.*,set-cookie`). |--## Request & Response Hooks--For attribute work that needs Django-specific context (the authenticated user, the resolved view name, the response body size), register a `response_hook` from the manual entrypoint:--```python-# myproject/otel.py-from opentelemetry.instrumentation.django import DjangoInstrumentor---def response_hook(span, request, response):- if not span or not span.is_recording():- return-- if hasattr(request, "user") and request.user.is_authenticated:- span.set_attribute("enduser.id", request.user.pk)- span.set_attribute("enduser.role", request.user.groups.values_list("name", flat=True).first() or "")-- span.set_attribute("http.response.body.size", len(response.content) if hasattr(response, "content") else 0)---DjangoInstrumentor().instrument(response_hook=response_hook)-```--Then import this module **before** Django is imported — typically at the very top of `manage.py` / `wsgi.py` / `asgi.py`:--```python-# manage.py-import myproject.otel # noqa: F401 — must come first--# ...standard manage.py body...-```--> **Why `response_hook` and not `request_hook`?** The `request_hook` runs **before** Django middleware, so middleware-populated attributes like `request.user` are still anonymous when it fires. Use `response_hook` for anything that depends on auth, sessions, or view resolution.--## Manual Initialization--If you can't use the `opentelemetry-instrument` agent (for example you're in a managed environment that forbids wrapping the command line), set up the SDK and instrumentation programmatically:--```python-# myproject/otel.py-from opentelemetry import trace-from opentelemetry.sdk.resources import Resource-from opentelemetry.sdk.trace import TracerProvider-from opentelemetry.sdk.trace.export import BatchSpanProcessor-from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter-from opentelemetry.instrumentation.django import DjangoInstrumentor---resource = Resource.create({"service.name": "my-django-app"})-provider = TracerProvider(resource=resource)-provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))-trace.set_tracer_provider(provider)--DjangoInstrumentor().instrument()-```--Import this module before Django loads:--```python-# manage.py-import myproject.otel # noqa: F401-```--The `OTLPSpanExporter()` constructor reads `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` from the environment, so you still configure it the same way.--## Test Your Integration--Add a test route to verify data is flowing:--```python-# myapp/views.py-from django.http import HttpResponse---def testing(request):- raise RuntimeError("Test error from Traceway integration")-```--```python-# myproject/urls.py-from django.urls import path-from myapp import views--urlpatterns = [- path("testing/", views.testing),-]-```--Visit `/testing/` in your browser, then check the Traceway dashboard — the exception should appear within a few seconds.--## Next Steps--- [Exceptions](/client/django/exceptions) — manually capture caught exceptions with context-- [Spans](/client/django/spans) — create custom spans to measure sub-operations-- [Tasks](/client/django/tasks) — trace Celery tasks and Django management commands as background tasks-- [Metrics](/client/django/metrics) — track custom counters, histograms, and gauges with the OTel Metrics API-- [Logs](/client/django/logs) — forward Django + Python `logging` records to Traceway over OTLP-- [OpenTelemetry docs](/client/otel) — how OTel traces, metrics, and logs map to Tracewaydiff --git a/docs/pages/client/django/logs.mdx b/docs/pages/client/django/logs.mdxdeleted file mode 100644index d4f1475e..00000000--- a/docs/pages/client/django/logs.mdx+++ /dev/null@@ -1,158 +0,0 @@-# Logs--`opentelemetry-instrumentation-logging` (auto-installed by `opentelemetry-bootstrap`) bridges Python's standard `logging` module to the OpenTelemetry Logs SDK and OTLP exporter. Once enabled, every `logger.info(...)` / `logger.error(...)` — including records emitted by Django itself, your views, your Celery tasks, and any third-party library that uses `logging` — is forwarded to Traceway and linked to the active trace and span.--## Step 1: Enable OTLP Logs--Extend the env config from the [Quick Start](/client/django) to also turn on the logs exporter and the Python logging auto-instrumentation:--```bash-# Existing traces + metrics config from the Quick Start-OTEL_SERVICE_NAME=my-django-app-OTEL_TRACES_EXPORTER=otlp-OTEL_METRICS_EXPORTER=otlp--# Enable OTLP logs-OTEL_LOGS_EXPORTER=otlp--# Auto-forward Python `logging` records as OTel log records-OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true--OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf-OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel-OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20your-project-token-```--`OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL — the logs exporter appends `/v1/logs` automatically. No extra endpoint config is needed.--> **`OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` is what flips the bridge on.** Without it, the logging instrumentation only injects `trace_id` / `span_id` into your existing log formatters — your records do **not** get shipped to OTLP. The variable is read once at startup, so set it in your environment (not in a `.env` loaded after Django imports).--## Step 2: Use Python's `logging` Module as Usual--Use the standard Python logger. Because the handler runs inside the request / Celery task span, every record emitted from there carries that span's `trace_id` and `span_id` automatically:--```python-# myapp/views.py-import logging--from django.http import JsonResponse--logger = logging.getLogger(__name__)---def create_order(request):- order_id = "ord_123"- logger.info("order received", extra={"order.id": order_id})-- try:- # ... business logic ...- logger.info("order processed", extra={"order.id": order_id})- return JsonResponse({"status": "ok"})- except Exception as exc:- logger.error("order failed", extra={"order.id": order_id, "exception": str(exc)})- raise-```--Open the endpoint's trace in the Traceway dashboard and the **Logs** tab will show these records attached. The same is true for logs emitted inside Celery tasks, management commands, and any other code that runs under an active span.--> **Attributes via `extra={...}`:** OTel's logging handler reads the `extra` dict on a `LogRecord` and attaches each key as an attribute on the OTel log record. So `logger.info("order received", extra={"order.id": "ord_123"})` produces an OTel log with attribute `order.id="ord_123"`. Use dotted keys (e.g. `order.id`, `user.id`) to match OTel semantic conventions.--## Step 3 (optional): Inject `trace_id` Into Console / File Logs--If you also write logs to the console or a file (Django's default `LOGGING` config does this in dev), the logging instrumentation injects the active `trace_id` and `span_id` into the `LogRecord` itself. Add them to your formatter to see them in your tail / Cloud Logging / wherever you read raw logs:--```python-# settings.py-LOGGING = {- "version": 1,- "disable_existing_loggers": False,- "formatters": {- "with_trace": {- "format": "[{levelname}] {message} | trace_id={otelTraceID} span_id={otelSpanID}",- "style": "{",- },- },- "handlers": {- "console": {- "class": "logging.StreamHandler",- "formatter": "with_trace",- },- },- "root": {"handlers": ["console"], "level": "INFO"},-}-```--`otelTraceID` and `otelSpanID` are populated by `opentelemetry-instrumentation-logging` for every record emitted under an active span. Outside a span they are `"0"` placeholders.--> **Don't add a Python `OTLPHandler` manually.** When `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`, the agent already attaches a `LoggingHandler` to the root logger. Adding another one yourself will produce duplicate log records in Traceway.--## Severity Mapping--Python's `logging` levels are mapped to OpenTelemetry severity numbers (and Traceway's TRACE → FATAL labels) as follows:--| Python Level | Numeric | OTLP Severity Number | Traceway Severity |-| ------------ | ------- | -------------------- | ----------------- |-| `DEBUG` | 10 | 5 | DEBUG |-| `INFO` | 20 | 9 | INFO |-| `WARNING` | 30 | 13 | WARN |-| `ERROR` | 40 | 17 | ERROR |-| `CRITICAL` | 50 | 21 | FATAL |--Only records at or above the root logger's configured `level` are emitted to handlers — so set `LOGGING["root"]["level"] = "INFO"` (or whatever threshold you want) to control volume.--## Logging Exceptions With Full Tracebacks--`logger.exception(...)` and `logger.error(..., exc_info=True)` both attach the traceback to the OTel log record's `exception.stacktrace` attribute, which Traceway extracts into the Logs view:--```python-import logging--logger = logging.getLogger(__name__)---def withdraw(request):- try:- _withdraw(request.user, request.POST["amount"])- except InsufficientFundsError:- logger.exception("withdraw failed", extra={"user.id": request.user.pk})- raise-```--Note that the *unhandled* exception is **also** captured on the request span by the Django middleware (see [Exceptions](/client/django/exceptions)). `logger.exception(...)` is for logging the same event into the Logs feed with structured attributes — not a substitute for re-raising.--## Test Your Integration--Add a route that logs at multiple levels, hit it once, and check the **Logs** page in the Traceway dashboard:--```python-# myapp/views.py-import logging--from django.http import JsonResponse--logger = logging.getLogger(__name__)---def log_test(request):- logger.debug("debug sample")- logger.info("info sample", extra={"request.id": "req_1"})- logger.warning("warning sample")- logger.error("error sample", extra={"order.id": "ord_1"})- return JsonResponse({"emitted": 4})-```--```python-# myproject/urls.py-urlpatterns = [- # ...- path("log-test/", views.log_test),-]-```--Visit `/log-test/`, then open **Logs** in the dashboard. You should see records at or above your root logger's `level` within a few seconds, each linked to the trace for that request.--## Next Steps--- [Exceptions](/client/django/exceptions) — record caught exceptions with context-- [Metrics](/client/django/metrics) — custom counters, histograms, and gauges-- [OTel Logs reference](/client/otel/logs) — full OTLP logs mapping and severity detailsdiff --git a/docs/pages/client/django/tasks.mdx b/docs/pages/client/django/tasks.mdxdeleted file mode 100644index 62724170..00000000--- a/docs/pages/client/django/tasks.mdx+++ /dev/null@@ -1,146 +0,0 @@-# Tasks--Django itself doesn't ship a built-in task queue, so most Django apps use **Celery**. The official [`opentelemetry-instrumentation-celery`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-celery) package auto-traces Celery: when a task is dispatched, a `PRODUCER` span is created; when the worker picks it up, a `CONSUMER` span runs the handler. W3C trace context is propagated through the task headers so the consumer span links back to the dispatch trace — the full lifecycle is visible as one continuous trace in Traceway, and each consumed task appears as a **task** with duration, status, and any child spans.--This page covers Celery (the typical setup) and Django management commands. Other Django task systems (RQ, Huey, Django-Q, Dramatiq) follow the same OTel idioms — wrap the handler in `tracer.start_as_current_span(...)` and you get the same Traceway view.--## Celery Setup--`opentelemetry-bootstrap -a install` already installed `opentelemetry-instrumentation-celery` for you when you followed the [Quick Start](/client/django). If you skipped that step:--```bash-pip install opentelemetry-instrumentation-celery-```--The Celery instrumentation hooks signal handlers via Celery's own signal API — it does **not** auto-activate from `opentelemetry-instrument` alone in every layout. Make sure it's invoked once, before workers start consuming, by adding it to the OTel entrypoint module you already import from `manage.py` / `wsgi.py`:--```python-# myproject/otel.py-from opentelemetry.instrumentation.django import DjangoInstrumentor-from opentelemetry.instrumentation.celery import CeleryInstrumentor--DjangoInstrumentor().instrument()-CeleryInstrumentor().instrument()-```--For the worker process, launch it the same way as the web server — wrap it with the agent so the SDK + exporters are configured:--```bash-opentelemetry-instrument celery -A myproject worker -l info-opentelemetry-instrument celery -A myproject beat -l info-```--That's it — every Celery task is now traced. The same setup applies whether you use `redis`, `rabbitmq`, `sqs`, or `database` brokers.--## What Gets Captured--For each Celery task the instrumentation creates two spans:--- **Publish span** — name `apply_async/<task-name>`, `PRODUCER` kind, ended when Celery finishes pushing the payload to the broker.-- **Run span** — name `run/<task-name>`, `CONSUMER` kind, ended when the handler returns or fails. On a task failure the span records the exception and is marked `STATUS_ERROR`.--Both spans carry these attributes (using OTel messaging semantic conventions):--| Attribute | Value |-|---|---|-| `messaging.system` | `celery` |-| `messaging.destination` | The queue name (e.g. `celery`, `priority-high`) |-| `messaging.message_id` | Celery's task UUID |-| `messaging.conversation_id` | Celery's `root_id` (for chained tasks) |-| `celery.task_name` | Fully-qualified task name (e.g. `myapp.tasks.send_welcome_email`) |-| `celery.action` | `apply_async` (producer) or `run` (consumer) |-| `celery.state` | Task state on completion — `SUCCESS`, `FAILURE`, `RETRY` |--W3C trace context is also injected into the task headers, so the run span is linked to the trace where the task was originally dispatched — even if dispatch and execution happen on different services. Any spans you create inside the task body automatically nest under the run span.--## Adding Attributes in Tasks--Since the instrumentation activates the consumer span before your task body runs, use `trace.get_current_span()` to add custom attributes:--```python-# myapp/tasks.py-from celery import shared_task-from opentelemetry import trace---@shared_task-def send_order_email(order_id: int, recipient: str) -> None:- span = trace.get_current_span()- span.set_attribute("order.id", order_id)- span.set_attribute("email.to", recipient)-- # ... business logic ...-```--## Child Spans in Tasks--For tasks with distinct sub-operations, use the `Tracer` API to create child spans. They automatically nest under the task's run span:--```python-# myapp/tasks.py-from celery import shared_task-from opentelemetry import trace--tracer = trace.get_tracer(__name__)---@shared_task-def process_payment(payment_id: int, amount: float) -> None:- with tracer.start_as_current_span("validate.payment") as span:- span.set_attribute("payment.id", payment_id)- _validate(payment_id)-- with tracer.start_as_current_span("charge.gateway") as span:- span.set_attribute("payment.amount", amount)- _charge_gateway(payment_id, amount)-```--## Periodic Tasks (Celery Beat)--Tasks scheduled by Celery Beat run through the same code path as `apply_async`, so they get the same producer + consumer span pair automatically. The producer span is opened by Beat when it dispatches the schedule entry; the consumer span is opened by whichever worker picks the task up.--If you want to attribute beat-dispatched tasks differently in the dashboard, set an attribute on the consumer side:--```python-@shared_task(bind=True)-def nightly_cleanup(self) -> None:- span = trace.get_current_span()- span.set_attribute("celery.scheduled_by", "beat")-- # ... work ...-```--## Management Commands--Django management commands run outside the HTTP request cycle, so the Django middleware never opens a root span for them. Unlike Celery tasks, management commands are **not traced automatically** — wrap the command body in a span yourself:--```python-# myapp/management/commands/cleanup_orphans.py-from django.core.management.base import BaseCommand-from opentelemetry import trace--tracer = trace.get_tracer(__name__)---class Command(BaseCommand):- help = "Delete orphaned rows that have no foreign key parent."-- def add_arguments(self, parser):- parser.add_argument("--dry-run", action="store_true")-- def handle(self, *args, **options):- with tracer.start_as_current_span("cleanup.orphans") as span:- span.set_attribute("cleanup.dry_run", bool(options["dry_run"]))-- # ... do work ...-```--Each wrapped command opens as a root span on entry and ends on exit. Inside the block, `trace.get_current_span()` returns the active span and any further spans (DB queries, outbound HTTP) nest under it.--Run the command via the OTel agent so the SDK + exporters are configured exactly like the web process:--```bash-opentelemetry-instrument python manage.py cleanup_orphans-```--If you skip the wrapping span, the command's DB queries and HTTP calls still produce spans — but each one is a disconnected root span in Traceway. The wrapping span is what ties them into a single task you can filter on.diff --git a/docs/pages/client/fasthttp-middleware/_meta.json b/docs/pages/client/fasthttp-middleware/_meta.jsondeleted file mode 100644index a5183284..00000000--- a/docs/pages/client/fasthttp-middleware/_meta.json+++ /dev/null@@ -1,5 +0,0 @@-{- "index": "Quick Start",- "configuration": { "display": "hidden" },- "error-recording": "Error Recording"-}diff --git a/docs/pages/client/fasthttp-middleware/configuration.mdx b/docs/pages/client/fasthttp-middleware/configuration.mdxdeleted file mode 100644index b816ec87..00000000--- a/docs/pages/client/fasthttp-middleware/configuration.mdx+++ /dev/null@@ -1,186 +0,0 @@-# FastHTTP Middleware Configuration--## Middleware Options--| Option | Description | Default |-|--------|-------------|---------|-| `WithRepanic(bool)` | Re-panic after capturing, letting upstream recovery handle it | `true` |-| `WithIgnoredPaths(...string)` | Skip recording for specific URL paths | none |-| `WithStreamingPaths(...string)` | Mark paths as long-lived streaming responses (SSE / WS / long-poll) | none |-| `WithFilter(func(*fasthttp.RequestCtx) bool)` | Custom filter function — return `false` to skip recording | none |-| `WithOnErrorRecording(RecordingFlag)` | What additional data to capture on errors | `0` (none) |--## Passthrough Core Options--These options are forwarded to the core `traceway` client:--| Option | Description | Default |-|--------|-------------|---------|-| `WithDebug(bool)` | Enable debug logging | `false` |-| `WithVersion(string)` | Application version shown in dashboard | `""` |-| `WithServerName(string)` | Server hostname identifier | Auto-detected |-| `WithSampleRate(float64)` | Percentage of normal traces to record (0.0–1.0) | `1.0` |-| `WithErrorSampleRate(float64)` | Percentage of error traces to record (0.0–1.0) | `1.0` |-| `WithCollectionInterval(time.Duration)` | How often batched data is sent | `5s` |-| `WithMaxCollectionFrames(int)` | Maximum frames to buffer | `12` |-| `WithUploadTimeout(time.Duration)` | HTTP upload timeout | `2s` |-| `WithMetricsInterval(time.Duration)` | System metrics collection interval | `30s` |--## Repanic Behavior--When `WithRepanic(true)` (default), the middleware re-panics after capturing the stack trace. This allows upstream recovery handlers to handle the panic as well.--When `WithRepanic(false)`, the middleware responds with HTTP 500 and does not re-panic.--## Request Filtering--### WithIgnoredPaths--Skip recording for specific paths:--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithIgnoredPaths("/health", "/ready", "/metrics"),-)-```--Ignored paths are still served — the request passes through to your handler, but no trace is recorded.--### WithFilter--For dynamic filtering logic, use `WithFilter`. Return `false` to skip recording:--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithFilter(func(ctx *fasthttp.RequestCtx) bool {- // Skip internal service-to-service calls- if string(ctx.Request.Header.Peek("X-Internal")) == "true" {- return false- }- // Skip static assets- if strings.HasPrefix(string(ctx.Path()), "/static/") {- return false- }- return true- }),-)-```--When the filter returns `false`, the request is passed directly to your handler without any Traceway wrapping.--## Streaming Responses (SSE / WebSocket / long-poll)--Long-lived streaming routes have a request lifetime that equals the connection duration, not request processing latency. The middleware **auto-detects** streaming responses from the response headers (SSE `Content-Type: text/event-stream`, WebSocket upgrade status `101` or `Upgrade: websocket`) and excludes them from P50/P95/P99, Apdex, and impact scoring while keeping their status code, error rate, and throughput visible on the dashboard.--For paths the SDK can't auto-detect:--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithStreamingPaths("/api/long-poll", "/api/stream/json"),-)-```--Or mark dynamically from any handler:--```go-func eventsHandler(ctx *fasthttp.RequestCtx) {- tracewayfasthttp.MarkStream(ctx)- // ... write stream ...-}-```--## Complete Example--```go-package main--import (- "strings"- "time"-- "github.com/valyala/fasthttp"- traceway "go.tracewayapp.com"- tracewayfasthttp "go.tracewayapp.com/tracewayfasthttp"-)--func main() {- traceway.ConfigureAttributes(func(s *traceway.Attributes) {- s.SetTag("environment", "production")- s.SetTag("service", "user-api")- })-- handler := func(ctx *fasthttp.RequestCtx) {- switch string(ctx.Path()) {- case "/api/users":- ctx.SetStatusCode(200)- ctx.SetBodyString(`{"users": []}`)- default:- ctx.SetStatusCode(404)- }- }-- tracedHandler := tracewayfasthttp.New(- "your-token@http://localhost:8082/api/report",- tracewayfasthttp.WithVersion("1.2.3"),- tracewayfasthttp.WithServerName("api-server-01"),- tracewayfasthttp.WithRepanic(true),- tracewayfasthttp.WithIgnoredPaths("/health", "/ready"),- tracewayfasthttp.WithFilter(func(ctx *fasthttp.RequestCtx) bool {- return string(ctx.Request.Header.Peek("X-Internal")) != "true"- }),- tracewayfasthttp.WithOnErrorRecording(- tracewayfasthttp.RecordingUrl | tracewayfasthttp.RecordingQuery,- ),- tracewayfasthttp.WithCollectionInterval(5 * time.Second),- tracewayfasthttp.WithMetricsInterval(30 * time.Second),- )(handler)-- fasthttp.ListenAndServe(":8080", tracedHandler)-}-```--## Configuration Recommendations--### Development--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithDebug(true),- tracewayfasthttp.WithOnErrorRecording(- tracewayfasthttp.RecordingUrl |- tracewayfasthttp.RecordingQuery |- tracewayfasthttp.RecordingHeader |- tracewayfasthttp.RecordingBody,- ),-)-```--### Production--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithVersion(os.Getenv("APP_VERSION")),- tracewayfasthttp.WithServerName(os.Getenv("HOSTNAME")),- tracewayfasthttp.WithOnErrorRecording(- tracewayfasthttp.RecordingUrl | tracewayfasthttp.RecordingQuery,- ),-)-```--### High-Traffic Services--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithSampleRate(0.1),- tracewayfasthttp.WithErrorSampleRate(1.0),- tracewayfasthttp.WithCollectionInterval(10 * time.Second),- tracewayfasthttp.WithMaxCollectionFrames(20),- tracewayfasthttp.WithMetricsInterval(60 * time.Second),-)-```diff --git a/docs/pages/client/fasthttp-middleware/error-recording.mdx b/docs/pages/client/fasthttp-middleware/error-recording.mdxdeleted file mode 100644index 9378b942..00000000--- a/docs/pages/client/fasthttp-middleware/error-recording.mdx+++ /dev/null@@ -1,83 +0,0 @@-# FastHTTP Error Recording--When an exception occurs during a request, you can control what additional request data is captured alongside the stack trace.--## Recording Flags--Configure error recording with `WithOnErrorRecording`:--```go-tracewayfasthttp.New(- connectionString,- tracewayfasthttp.WithOnErrorRecording(- tracewayfasthttp.RecordingUrl | tracewayfasthttp.RecordingQuery,- ),-)-```--### Available Flags--| Flag | Description | Privacy Impact |-|------|-------------|----------------|-| `RecordingUrl` | Request URL path | Low |-| `RecordingQuery` | Query parameters as JSON | Medium |-| `RecordingBody` | Request body (JSON content-type only, max 64KB) | High |-| `RecordingHeader` | Request headers as JSON | High |--### Combining Flags--Flags are combined with the `|` operator:--```go-// Record URL and query parameters-tracewayfasthttp.RecordingUrl | tracewayfasthttp.RecordingQuery--// Record everything-tracewayfasthttp.RecordingUrl | tracewayfasthttp.RecordingQuery | tracewayfasthttp.RecordingBody | tracewayfasthttp.RecordingHeader-```--## Body Capture Details--Body recording has two constraints:--- **Content-Type must be `application/json`** — non-JSON bodies are not captured-- **Maximum size is 64KB** — bodies larger than 64KB are truncated--After reading the body for capture, it is restored so downstream handlers can still read it.--## Privacy Considerations--Body and header recording may capture sensitive data:--- Authentication tokens in headers-- Personal information in request bodies-- API keys in query parameters--Use minimal recording flags in production:--```go-// Production — limited data-tracewayfasthttp.WithOnErrorRecording(- tracewayfasthttp.RecordingUrl | tracewayfasthttp.RecordingQuery,-)--// Development — full request data-tracewayfasthttp.WithOnErrorRecording(- tracewayfasthttp.RecordingUrl |- tracewayfasthttp.RecordingQuery |- tracewayfasthttp.RecordingBody |- tracewayfasthttp.RecordingHeader,-)-```--## Captured Data Format--Recorded data is attached as attributes on the issue:--| Attribute Key | Value |-|---------------|-------|-| `url` | URL path string |-| `query` | JSON-encoded query parameters |-| `body` | Raw request body string |-| `headers` | JSON-encoded request headers |-| `user agent` | User-Agent header value (always captured on error) |diff --git a/docs/pages/client/fasthttp-middleware/index.mdx b/docs/pages/client/fasthttp-middleware/index.mdxdeleted file mode 100644index 19c30bb9..00000000--- a/docs/pages/client/fasthttp-middleware/index.mdx+++ /dev/null@@ -1,119 +0,0 @@-# FastHTTP Middleware--import { Callout } from "nextra/components";--<Callout type="warning">- The Traceway Go SDK is no longer the recommended way to integrate with Traceway. For new projects, instrument with [OpenTelemetry](/client/otel) — it works across languages and Traceway natively ingests OTLP. The Go SDK is still supported, but new features land in the OTel path first.-</Callout>--The `tracewayfasthttp` package provides middleware for the [FastHTTP](https://github.com/valyala/fasthttp) server. It automatically creates a trace for each request, captures panics, and detects client IP addresses.--## Installation--```bash-go get go.tracewayapp.com-go get go.tracewayapp.com/tracewayfasthttp-```--## Basic Setup--```go-package main--import (- "github.com/valyala/fasthttp"- tracewayfasthttp "go.tracewayapp.com/tracewayfasthttp"-)--func main() {- handler := func(ctx *fasthttp.RequestCtx) {- ctx.SetStatusCode(200)- ctx.SetBodyString("Hello, World!")- }-- // Wrap with Traceway middleware- tracedHandler := tracewayfasthttp.New(- "your-token@http://localhost:8082/api/report",- )(handler)-- fasthttp.ListenAndServe(":8080", tracedHandler)-}-```--`New` returns `func(fasthttp.RequestHandler) fasthttp.RequestHandler` and calls `traceway.Init` internally — do not call `Init` separately.--## What Gets Captured--The middleware automatically captures for every request:--| Data | Description |-| --------------- | ----------------------------------------------- |-| **Endpoint** | HTTP method + URL path (e.g., `GET /api/users`) |-| **Duration** | Request processing time |-| **Status Code** | HTTP response status |-| **Body Size** | Response body size in bytes |-| **Client IP** | Detected from headers or connection |-| **Panics** | Recovered and captured with full stack trace |--## Client IP Detection--Client IP is resolved in this order:--1. `X-Forwarded-For` header (first IP in the list)-2. `X-Real-IP` header-3. `RemoteAddr` from the connection--## Error Detection--A request is considered an error when:--- A **panic** occurs during handler execution-- The response status code is **>= 500**--Error traces use the error sample rate for sampling decisions.--## Panic Handling--When a panic occurs, the middleware:--1. Recovers the panic-2. Captures the full stack trace as an issue-3. Either re-panics (default, `WithRepanic(true)`) or responds with 500--## Context Access--Access Traceway context from within your handlers:--```go-func handler(ctx *fasthttp.RequestCtx) {- // Get attributes for this request- attributes := tracewayfasthttp.GetAttributesFromCtx(ctx)- attributes.SetTag("user_id", "123")-- // Get trace context- trace := tracewayfasthttp.GetTraceFromCtx(ctx)-}-```--Context is stored in FastHTTP's `UserValue` store, not in Go's standard `context.Context`.--## Connection package--The connection page in the sidebar will have your projects unique API key and the integration instructions.--<img- src="/connection.png"- alt="SDK Connection Setup"- style={{- maxWidth: "100%",- height: "auto",- marginTop: "16px",- borderRadius: "8px",- border: "1px solid #e5e7eb",- }}-/>--## Next Steps--- [Configuration](/client/fasthttp-middleware/configuration) — All options including filtering and passthrough settings-- [Error Recording](/client/fasthttp-middleware/error-recording) — Control what request data is captured on errorsdiff --git a/docs/pages/client/fiber-middleware/_meta.json b/docs/pages/client/fiber-middleware/_meta.jsondeleted file mode 100644index a5183284..00000000--- a/docs/pages/client/fiber-middleware/_meta.json+++ /dev/null@@ -1,5 +0,0 @@-{- "index": "Quick Start",- "configuration": { "display": "hidden" },- "error-recording": "Error Recording"-}diff --git a/docs/pages/client/fiber-middleware/configuration.mdx b/docs/pages/client/fiber-middleware/configuration.mdxdeleted file mode 100644index 63e91628..00000000--- a/docs/pages/client/fiber-middleware/configuration.mdx+++ /dev/null@@ -1,187 +0,0 @@-# Fiber Middleware Configuration--## Middleware Options--| Option | Description | Default |-|--------|-------------|---------|-| `WithRepanic(bool)` | Re-panic after capturing, letting upstream recovery handle it | `true` |-| `WithRecordUnmatched(bool)` | Record requests for unregistered routes (404s) | `false` |-| `WithIgnoredPaths(...string)` | Skip recording for specific route paths | none |-| `WithStreamingPaths(...string)` | Mark routes as long-lived streaming responses (SSE / WS / long-poll) | none |-| `WithFilter(func(*fiber.Ctx) bool)` | Custom filter function — return `false` to skip recording | none |-| `WithOnErrorRecording(RecordingFlag)` | What additional data to capture on errors | `0` (none) |--## Passthrough Core Options--These options are forwarded to the core `traceway` client:--| Option | Description | Default |-|--------|-------------|---------|-| `WithDebug(bool)` | Enable debug logging | `false` |-| `WithVersion(string)` | Application version shown in dashboard | `""` |-| `WithServerName(string)` | Server hostname identifier | Auto-detected |-| `WithSampleRate(float64)` | Percentage of normal traces to record (0.0–1.0) | `1.0` |-| `WithErrorSampleRate(float64)` | Percentage of error traces to record (0.0–1.0) | `1.0` |-| `WithCollectionInterval(time.Duration)` | How often batched data is sent | `5s` |-| `WithMaxCollectionFrames(int)` | Maximum frames to buffer | `12` |-| `WithUploadTimeout(time.Duration)` | HTTP upload timeout | `2s` |-| `WithMetricsInterval(time.Duration)` | System metrics collection interval | `30s` |--## Repanic Behavior--When `WithRepanic(true)` (default), the middleware re-panics after capturing the stack trace. This allows Fiber's built-in error handler or upstream recovery middleware to handle the panic.--When `WithRepanic(false)`, the middleware returns a 500 error and does not re-panic.--## Route Filtering--### Unmatched Routes--By default, requests to unregistered routes (404s) are **not recorded**. The request is passed through but no trace is created.--```go-tracewayfiber.WithRecordUnmatched(true) // Record 404 requests-```--### Ignored Paths--Skip recording for specific route paths:--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithIgnoredPaths("/health", "/ready", "/metrics"),-)-```--Ignored paths are still served — the request passes through to your handler, but no trace is recorded.--### WithFilter--For dynamic filtering logic, use `WithFilter`. Return `false` to skip recording:--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithFilter(func(c *fiber.Ctx) bool {- // Skip internal service-to-service calls- if c.Get("X-Internal") == "true" {- return false- }- // Skip static assets- if strings.HasPrefix(c.Path(), "/static/") {- return false- }- return true- }),-)-```--When the filter returns `false`, the request is passed directly to your handler without any Traceway wrapping.--## Streaming Responses (SSE / WebSocket / long-poll)--Long-lived streaming routes have a request lifetime that equals the connection duration, not request processing latency. The middleware **auto-detects** streaming responses from the response headers (SSE `Content-Type: text/event-stream`, WebSocket upgrade status `101` or `Upgrade: websocket`) and excludes them from P50/P95/P99, Apdex, and impact scoring while keeping their status code, error rate, and throughput visible on the dashboard.--For routes the SDK can't auto-detect:--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithStreamingPaths("/api/long-poll", "/api/stream/json"),-)-```--Or mark dynamically from any handler:--```go-func eventsHandler(c *fiber.Ctx) error {- tracewayfiber.MarkStream(c)- // ... write stream ...- return nil-}-```--## Complete Example--```go-package main--import (- "strings"- "time"-- "github.com/gofiber/fiber/v2"- traceway "go.tracewayapp.com"- tracewayfiber "go.tracewayapp.com/tracewayfiber"-)--func main() {- traceway.ConfigureAttributes(func(s *traceway.Attributes) {- s.SetTag("environment", "production")- s.SetTag("service", "user-api")- })-- app := fiber.New()-- app.Use(tracewayfiber.New(- "your-token@http://localhost:8082/api/report",- tracewayfiber.WithVersion("1.2.3"),- tracewayfiber.WithServerName("api-server-01"),- tracewayfiber.WithRepanic(true),- tracewayfiber.WithRecordUnmatched(false),- tracewayfiber.WithIgnoredPaths("/health", "/ready"),- tracewayfiber.WithOnErrorRecording(- tracewayfiber.RecordingUrl | tracewayfiber.RecordingQuery,- ),- tracewayfiber.WithCollectionInterval(5 * time.Second),- tracewayfiber.WithMetricsInterval(30 * time.Second),- ))-- app.Get("/api/users/:id", getUser)- app.Listen(":8080")-}-```--## Configuration Recommendations--### Development--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithDebug(true),- tracewayfiber.WithOnErrorRecording(- tracewayfiber.RecordingUrl |- tracewayfiber.RecordingQuery |- tracewayfiber.RecordingHeader |- tracewayfiber.RecordingBody,- ),-)-```--### Production--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithVersion(os.Getenv("APP_VERSION")),- tracewayfiber.WithServerName(os.Getenv("HOSTNAME")),- tracewayfiber.WithOnErrorRecording(- tracewayfiber.RecordingUrl | tracewayfiber.RecordingQuery,- ),-)-```--### High-Traffic Services--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithSampleRate(0.1),- tracewayfiber.WithErrorSampleRate(1.0),- tracewayfiber.WithCollectionInterval(10 * time.Second),- tracewayfiber.WithMaxCollectionFrames(20),- tracewayfiber.WithMetricsInterval(60 * time.Second),-)-```diff --git a/docs/pages/client/fiber-middleware/error-recording.mdx b/docs/pages/client/fiber-middleware/error-recording.mdxdeleted file mode 100644index bc84ce6d..00000000--- a/docs/pages/client/fiber-middleware/error-recording.mdx+++ /dev/null@@ -1,83 +0,0 @@-# Fiber Error Recording--When an exception occurs during a request (panic, handler error, or status >= 500), you can control what additional request data is captured alongside the stack trace.--## Recording Flags--Configure error recording with `WithOnErrorRecording`:--```go-tracewayfiber.New(- connectionString,- tracewayfiber.WithOnErrorRecording(- tracewayfiber.RecordingUrl | tracewayfiber.RecordingQuery,- ),-)-```--### Available Flags--| Flag | Description | Privacy Impact |-|------|-------------|----------------|-| `RecordingUrl` | Request URL path | Low |-| `RecordingQuery` | Query parameters as JSON | Medium |-| `RecordingBody` | Request body (JSON content-type only, max 64KB) | High |-| `RecordingHeader` | Request headers as JSON | High |--### Combining Flags--Flags are combined with the `|` operator:--```go-// Record URL and query parameters-tracewayfiber.RecordingUrl | tracewayfiber.RecordingQuery--// Record everything-tracewayfiber.RecordingUrl | tracewayfiber.RecordingQuery | tracewayfiber.RecordingBody | tracewayfiber.RecordingHeader-```--## Body Capture Details--Body recording has two constraints:--- **Content-Type must be `application/json`** — non-JSON bodies are not captured-- **Maximum size is 64KB** — bodies larger than 64KB are truncated--After reading the body for capture, it is restored so downstream handlers can still read it.--## Privacy Considerations--Body and header recording may capture sensitive data:--- Authentication tokens in headers-- Personal information in request bodies-- API keys in query parameters--Use minimal recording flags in production:--```go-// Production — limited data-tracewayfiber.WithOnErrorRecording(- tracewayfiber.RecordingUrl | tracewayfiber.RecordingQuery,-)--// Development — full request data-tracewayfiber.WithOnErrorRecording(- tracewayfiber.RecordingUrl |- tracewayfiber.RecordingQuery |- tracewayfiber.RecordingBody |- tracewayfiber.RecordingHeader,-)-```--## Captured Data Format--Recorded data is attached as attributes on the issue:--| Attribute Key | Value |-|---------------|-------|-| `url` | URL path string |-| `query` | JSON-encoded query parameters |-| `body` | Raw request body string |-| `headers` | JSON-encoded request headers |-| `user agent` | User-Agent header value (always captured on error) |diff --git a/docs/pages/client/fiber-middleware/index.mdx b/docs/pages/client/fiber-middleware/index.mdxdeleted file mode 100644index 6f6d3474..00000000--- a/docs/pages/client/fiber-middleware/index.mdx+++ /dev/null@@ -1,137 +0,0 @@-# Fiber Middleware--import { Callout } from "nextra/components";--<Callout type="warning">- The Traceway Go SDK is no longer the recommended way to integrate with Traceway. For new projects, instrument with [OpenTelemetry](/client/otel) — it works across languages and Traceway natively ingests OTLP. The Go SDK is still supported, but new features land in the OTel path first.-</Callout>--The `tracewayfiber` package provides middleware for the [Fiber](https://gofiber.io/) web framework. It automatically creates a trace for each request, captures panics and handler errors, and detects client IP addresses.--## Installation--```bash-go get go.tracewayapp.com-go get go.tracewayapp.com/tracewayfiber-```--## Basic Setup--```go-package main--import (- "github.com/gofiber/fiber/v2"- tracewayfiber "go.tracewayapp.com/tracewayfiber"-)--func main() {- app := fiber.New()-- // Add Traceway middleware (must be before routes)- app.Use(tracewayfiber.New(- "your-token@http://localhost:8082/api/report",- ))-- app.Get("/api/users", getUsers)- app.Post("/api/users", createUser)-- app.Listen(":8080")-}-```--`New` returns a `fiber.Handler` and calls `traceway.Init` internally — do not call `Init` separately.--## What Gets Captured--The middleware automatically captures for every request:--| Data | Description |-| ------------------ | ----------------------------------------------------- |-| **Endpoint** | HTTP method + route path (e.g., `GET /api/users/:id`) |-| **Duration** | Request processing time |-| **Status Code** | HTTP response status |-| **Body Size** | Response body size in bytes |-| **Client IP** | From Fiber's `c.IP()` |-| **Panics** | Recovered and captured with full stack trace |-| **Handler Errors** | Errors returned from handlers |--## Error Detection--A request is considered an error when any of these are true:--- A **panic** occurred during handler execution-- A handler **returned an error**-- The response **status code is >= 500**--Error traces use the error sample rate for sampling decisions.--## Panic Recovery--When a panic occurs, the middleware:--1. Recovers the panic-2. Captures the full stack trace as an issue-3. Either re-panics (default, `WithRepanic(true)`) or returns a 500 error--```go-app.Get("/risky", func(c *fiber.Ctx) error {- panic("something went wrong")- // Stack trace is captured, linked to this request's trace- return nil-})-```--## Handler Errors--Errors returned from Fiber handlers are captured as issues:--```go-app.Get("/api/users/:id", func(c *fiber.Ctx) error {- user, err := findUser(c.Params("id"))- if err != nil {- return err // Captured as an issue- }- return c.JSON(user)-})-```--## Context Access--Access Traceway context from within your handlers:--```go-app.Get("/api/users", func(c *fiber.Ctx) error {- // Get attributes for this request- attributes := tracewayfiber.GetAttributesFromCtx(c)- attributes.SetTag("user_id", "123")-- // Get trace context- trace := tracewayfiber.GetTraceFromCtx(c)-- return c.SendString("OK")-})-```--Context is stored in Fiber's `Locals` store.--## Connection package--The connection page in the sidebar will have your projects unique API key and the integration instructions.--<img- src="/connection.png"- alt="SDK Connection Setup"- style={{- maxWidth: "100%",- height: "auto",- marginTop: "16px",- borderRadius: "8px",- border: "1px solid #e5e7eb",- }}-/>--## Next Steps--- [Configuration](/client/fiber-middleware/configuration) — All options including filtering and passthrough settings-- [Error Recording](/client/fiber-middleware/error-recording) — Control what request data is captured on errorsdiff --git a/docs/pages/client/flutter/index.mdx b/docs/pages/client/flutter/index.mdxindex 9eca397b..de8f1c90 100644--- a/docs/pages/client/flutter/index.mdx+++ b/docs/pages/client/flutter/index.mdx@@ -88,6 +88,8 @@ Traceway.run(Traceway needs network access to send error reports. Depending on the platform, you may need to add permissions manually.+The package ships native plugins, so iOS and macOS builds go through CocoaPods. If `flutter run` stops with `Error: CocoaPods not installed or not in valid state.`, install it once with `brew install cocoapods` and run again.+### AndroidAdd the `INTERNET` permission to `android/app/src/main/AndroidManifest.xml`:@@ -129,20 +131,20 @@ When `screenCapture` is set to `true`:## Logs & Actions-Every captured exception ships with the last ~10 seconds of session context, attached to the same session recording as the video. Two independent rolling buffers — **logs** and **actions** — are kept in memory, each capped at 200 entries by default.+Every captured exception ships with the last ~10 seconds of session context, attached to the same session recording as the video. Two independent rolling buffers, **logs** and **actions**, are kept in memory, each capped at 200 entries by default.### LogsEvery `print` and `debugPrint` line is mirrored into the log buffer via a Zone print override installed by `Traceway.run()`. Console output behaves normally; it just gets a copy.-`dart:developer.log` is **not** captured — it doesn't flow through the Zone print hook.+`dart:developer.log` is **not** captured, because it doesn't flow through the Zone print hook.### ActionsThree kinds of actions are collected:-- **Network** — every dart:io HTTP request (method, URL, status, duration, request/response byte counts). `HttpOverrides.global` is installed by `Traceway.run()`, so this catches `package:http`, Dio, Firebase, and any other library that sits on the dart:io HttpClient on native platforms.-- **Navigation** — push / pop / replace / remove transitions, recorded by `Traceway.navigatorObserver`. Wire it into your app's navigator:+- **Network**: every dart:io HTTP request (method, URL, status, duration, request/response byte counts). `HttpOverrides.global` is installed by `Traceway.run()`, so this catches `package:http`, Dio, Firebase, and any other library that sits on the dart:io HttpClient on native platforms.+- **Navigation**: push / pop / replace / remove transitions, recorded by `Traceway.navigatorObserver`. Wire it into your app's navigator:```dartMaterialApp(@@ -151,7 +153,7 @@ Three kinds of actions are collected:);```-- **Custom** — anything your app records explicitly:+- **Custom**: anything your app records explicitly, like this:```dartTraceway.recordAction(@@ -163,7 +165,7 @@ Three kinds of actions are collected:### Wire shape-Logs and actions are sent to the backend inside the same `sessionRecordings` entry that carries the MP4 — they're stored under separate `logs` and `actions` arrays. Each recording also carries `startedAt` and `endedAt` ISO 8601 timestamps so the backend can align events onto the video timeline (`offsetIntoVideoMs = event.timestamp − recording.startedAt`):+Logs and actions are sent to the backend inside the same `sessionRecordings` entry that carries the MP4. They're stored under separate `logs` and `actions` arrays. Each recording also carries `startedAt` and `endedAt` ISO 8601 timestamps so the backend can align events onto the video timeline (`offsetIntoVideoMs = event.timestamp − recording.startedAt`):```json{@@ -190,7 +192,7 @@ When the screen recorder is on, `startedAt` and `endedAt` come from the first an### Network capture on Flutter web-`HttpOverrides.global` does not run on Flutter web. Use `TracewayHttpClient` instead — it's a drop-in `http.Client` you can use directly or pass to libraries that accept a custom client (Dio, Chopper, etc.):+`HttpOverrides.global` does not run on Flutter web. Use `TracewayHttpClient` instead. It's a drop-in `http.Client` you can use directly or pass to libraries that accept a custom client (Dio, Chopper, etc.):```dartimport 'package:traceway/traceway.dart';@@ -218,7 +220,7 @@ Traceway.run(## Privacy Masking-Use `TracewayPrivacyMask` to hide sensitive content in screen recordings. Masked regions are applied to the recorded frames only — the user sees no visual change in the live app.+Use `TracewayPrivacyMask` to hide sensitive content in screen recordings. Masked regions are applied to the recorded frames only. The user sees no visual change in the live app.**Blur (pixelation):**@@ -252,9 +254,9 @@ This replaces the region with a solid color (black by default) in the recording.`Traceway.run()` sets up capture for the following error sources with no additional configuration:-- **Flutter framework errors** — rendering, layout, and gesture errors reported through `FlutterError.onError`.-- **Platform errors** — native platform channel errors.-- **Uncaught async errors** — any unhandled exceptions in asynchronous code, captured via Dart's `Zone` mechanism.+- **Flutter framework errors**: rendering, layout, and gesture errors reported through `FlutterError.onError`.+- **Platform errors**: native platform channel errors.+- **Uncaught async errors**: any unhandled exceptions in asynchronous code, captured via Dart's `Zone` mechanism.## Release Symbolication (obfuscated builds)@@ -302,7 +304,7 @@ Tap the button and check your Traceway dashboard to verify the error appears.| macOS | Yes | Yes || Web | No | No |-For Flutter web, use the JS SDK instead — see [Flutter Web](#flutter-web) below.+For Flutter web, use the JS SDK instead; see [Flutter Web](#flutter-web) below.## Flutter Webdiff --git a/docs/pages/client/gin-middleware/_meta.json b/docs/pages/client/gin-middleware/_meta.jsondeleted file mode 100644index a5183284..00000000--- a/docs/pages/client/gin-middleware/_meta.json+++ /dev/null@@ -1,5 +0,0 @@-{- "index": "Quick Start",- "configuration": { "display": "hidden" },- "error-recording": "Error Recording"-}diff --git a/docs/pages/client/gin-middleware/configuration.mdx b/docs/pages/client/gin-middleware/configuration.mdxdeleted file mode 100644index 7ab81a2e..00000000--- a/docs/pages/client/gin-middleware/configuration.mdx+++ /dev/null@@ -1,176 +0,0 @@-# Gin Middleware Configuration--## Middleware Options--| Option | Description | Default |-|--------|-------------|---------|-| `WithRepanic(bool)` | Re-panic after capturing, letting upstream recovery handle it | `true` |-| `WithRecordUnmatched(bool)` | Record requests for unregistered routes (404s) | `false` |-| `WithRecordStatic(bool)` | Record requests for static file routes | `false` |-| `WithIgnoredPaths(...string)` | Skip recording for specific route paths | none |-| `WithStreamingPaths(...string)` | Mark routes as long-lived streaming responses (SSE / WS / long-poll) | none |-| `WithOnErrorRecording(RecordingFlag)` | What additional data to capture on errors | `0` (none) |--## Passthrough Core Options--These options are forwarded to the core `traceway` client:--| Option | Description | Default |-|--------|-------------|---------|-| `WithDebug(bool)` | Enable debug logging | `false` |-| `WithVersion(string)` | Application version shown in dashboard | `""` |-| `WithServerName(string)` | Server hostname identifier | Auto-detected |-| `WithSampleRate(float64)` | Percentage of normal traces to record (0.0–1.0) | `1.0` |-| `WithErrorSampleRate(float64)` | Percentage of error traces to record (0.0–1.0) | `1.0` |-| `WithCollectionInterval(time.Duration)` | How often batched data is sent | `5s` |-| `WithMaxCollectionFrames(int)` | Maximum frames to buffer | `12` |-| `WithUploadTimeout(time.Duration)` | HTTP upload timeout | `2s` |-| `WithMetricsInterval(time.Duration)` | System metrics collection interval | `30s` |--## Repanic Behavior--When `WithRepanic(true)` (default), the middleware re-panics after capturing the stack trace. This allows upstream recovery middleware (like `gin.Recovery()`) to handle the panic.--When `WithRepanic(false)`, the middleware calls `c.AbortWithStatus(500)` and does not re-panic.--## Route Filtering--### Unmatched Routes--By default, requests to unregistered routes (404s) are **not recorded**. The request is passed through but no trace is created.--```go-tracewaygin.WithRecordUnmatched(true) // Record 404 requests-```--When enabled, the raw URL path is used as the endpoint name instead of a route template.--### Static Routes--Requests served by Gin's `StaticFile`, `Static`, or `StaticFS` handlers are **not recorded** by default:--```go-tracewaygin.WithRecordStatic(true) // Record static file requests-```--### Ignored Paths--Skip recording for specific route paths:--```go-tracewaygin.WithIgnoredPaths("/health", "/ready", "/metrics")-```--Ignored paths are still served — the request passes through to your handler, but no trace is recorded.--## Streaming Responses (SSE / WebSocket / long-poll)--Long-lived streaming routes — Server-Sent Events, WebSocket connections, long-polling — have a request lifetime that equals the connection duration, not request processing latency. If they were recorded as normal endpoints they would dominate P50/P95/P99 rankings, tank Apdex, and trigger slow-endpoint alerts even when behaving correctly.--The middleware **auto-detects** streaming responses by inspecting the response headers after the handler returns:--| Signal | Detection |-|--------|-----------|-| Server-Sent Events | `Content-Type: text/event-stream` |-| WebSocket upgrade | HTTP status `101 Switching Protocols`, or `Upgrade: websocket` |--When a streaming response is detected the trace is still recorded — its status code, error rate, and throughput remain visible on the dashboard — but its duration is **excluded** from P50/P95/P99, Apdex, and impact scoring.--For routes the SDK can't auto-detect (long-polling, chunked JSON streams, custom protocols), declare them up-front:--```go-tracewaygin.WithStreamingPaths("/api/long-poll", "/api/stream/json")-```--Or mark dynamically from any handler reachable on the request — useful when the same route is sometimes streaming, sometimes not:--```go-func eventsHandler(c *gin.Context) {- traceway.MarkStream(c.Request.Context())- // ... write stream ...-}-```--## Complete Example--```go-package main--import (- "time"-- "github.com/gin-gonic/gin"- traceway "go.tracewayapp.com"- tracewaygin "go.tracewayapp.com/tracewaygin"-)--func main() {- traceway.ConfigureAttributes(func(s *traceway.Attributes) {- s.SetTag("environment", "production")- s.SetTag("service", "user-api")- })-- router := gin.Default()-- router.Use(tracewaygin.New(- "your-token@http://localhost:8082/api/report",- tracewaygin.WithVersion("1.2.3"),- tracewaygin.WithServerName("api-server-01"),- tracewaygin.WithRepanic(true),- tracewaygin.WithRecordUnmatched(false),- tracewaygin.WithRecordStatic(false),- tracewaygin.WithIgnoredPaths("/health", "/ready"),- tracewaygin.WithOnErrorRecording(- tracewaygin.RecordingUrl | tracewaygin.RecordingQuery,- ),- tracewaygin.WithCollectionInterval(5 * time.Second),- tracewaygin.WithMetricsInterval(30 * time.Second),- ))-- router.GET("/api/users/:id", getUser)- router.Run(":8080")-}-```--## Configuration Recommendations--### Development--```go-tracewaygin.New(- connectionString,- tracewaygin.WithDebug(true),- tracewaygin.WithOnErrorRecording(- tracewaygin.RecordingUrl |- tracewaygin.RecordingQuery |- tracewaygin.RecordingHeader |- tracewaygin.RecordingBody,- ),-)-```--### Production--```go-tracewaygin.New(- connectionString,- tracewaygin.WithVersion(os.Getenv("APP_VERSION")),- tracewaygin.WithServerName(os.Getenv("HOSTNAME")),- tracewaygin.WithOnErrorRecording(- tracewaygin.RecordingUrl | tracewaygin.RecordingQuery,- ),-)-```--### High-Traffic Services--```go-tracewaygin.New(- connectionString,- tracewaygin.WithSampleRate(0.1),- tracewaygin.WithErrorSampleRate(1.0),- tracewaygin.WithCollectionInterval(10 * time.Second),- tracewaygin.WithMaxCollectionFrames(20),- tracewaygin.WithMetricsInterval(60 * time.Second),-)-```diff --git a/docs/pages/client/gin-middleware/error-recording.mdx b/docs/pages/client/gin-middleware/error-recording.mdxdeleted file mode 100644index fe34cc6e..00000000--- a/docs/pages/client/gin-middleware/error-recording.mdx+++ /dev/null@@ -1,83 +0,0 @@-# Gin Error Recording--When an exception occurs during a request (panic, Gin error, or status >= 500), you can control what additional request data is captured alongside the stack trace.--## Recording Flags--Configure error recording with `WithOnErrorRecording`:--```go-tracewaygin.New(- connectionString,- tracewaygin.WithOnErrorRecording(- tracewaygin.RecordingUrl | tracewaygin.RecordingQuery,- ),-)-```--### Available Flags--| Flag | Description | Privacy Impact |-|------|-------------|----------------|-| `RecordingUrl` | Request URL path | Low |-| `RecordingQuery` | Query parameters as JSON | Medium |-| `RecordingBody` | Request body (JSON content-type only, max 64KB) | High |-| `RecordingHeader` | Request headers as JSON | High |--### Combining Flags--Flags are combined with the `|` operator:--```go-// Record URL and query parameters-tracewaygin.RecordingUrl | tracewaygin.RecordingQuery--// Record everything-tracewaygin.RecordingUrl | tracewaygin.RecordingQuery | tracewaygin.RecordingBody | tracewaygin.RecordingHeader-```--## Body Capture Details--Body recording has two constraints:--- **Content-Type must be `application/json`** — non-JSON bodies are not captured-- **Maximum size is 64KB** — bodies larger than 64KB are truncated--After reading the body for capture, it is restored so downstream handlers can still read it.--## Privacy Considerations--Body and header recording may capture sensitive data:--- Authentication tokens in headers-- Personal information in request bodies-- API keys in query parameters--Use minimal recording flags in production:--```go-// Production — limited data-tracewaygin.WithOnErrorRecording(- tracewaygin.RecordingUrl | tracewaygin.RecordingQuery,-)--// Development — full request data-tracewaygin.WithOnErrorRecording(- tracewaygin.RecordingUrl |- tracewaygin.RecordingQuery |- tracewaygin.RecordingBody |- tracewaygin.RecordingHeader,-)-```--## Captured Data Format--Recorded data is attached as attributes on the issue:--| Attribute Key | Value |-|---------------|-------|-| `url` | URL path string |-| `query` | JSON-encoded query parameters |-| `body` | Raw request body string |-| `headers` | JSON-encoded request headers |-| `user agent` | User-Agent header value (always captured on error) |diff --git a/docs/pages/client/gin-middleware/index.mdx b/docs/pages/client/gin-middleware/index.mdxdeleted file mode 100644index c1af773e..00000000--- a/docs/pages/client/gin-middleware/index.mdx+++ /dev/null@@ -1,119 +0,0 @@-# Gin Middleware--import { Callout } from "nextra/components";--<Callout type="warning">- The Traceway Go SDK is no longer the recommended way to integrate with Traceway. For new projects, instrument with [OpenTelemetry](/client/otel) — it works across languages and Traceway natively ingests OTLP. The Go SDK is still supported, but new features land in the OTel path first.-</Callout>--The `tracewaygin` package provides middleware for the [Gin](https://gin-gonic.com/) web framework. It automatically creates a trace for each request, captures panics, and detects errors from Gin's error chain.--## Installation--```bash-go get go.tracewayapp.com-go get go.tracewayapp.com/tracewaygin-```--## Basic Setup--```go-package main--import (- "github.com/gin-gonic/gin"- tracewaygin "go.tracewayapp.com/tracewaygin"-)--func main() {- router := gin.Default()-- // Add Traceway middleware (must be before routes)- router.Use(tracewaygin.New(- "your-token@http://localhost:8082/api/report",- ))-- router.GET("/api/users", getUsers)- router.POST("/api/users", createUser)-- router.Run(":8080")-}-```--`New` returns a `gin.HandlerFunc` and calls `traceway.Init` internally — do not call `Init` separately.--## What Gets Captured--The middleware automatically captures for every request:--| Data | Description |-| --------------- | --------------------------------------------------------- |-| **Endpoint** | HTTP method + route template (e.g., `GET /api/users/:id`) |-| **Duration** | Request processing time |-| **Status Code** | HTTP response status |-| **Body Size** | Response body size in bytes |-| **Client IP** | From Gin's `ClientIP()` |-| **Panics** | Recovered and captured with full stack trace |-| **Gin Errors** | Errors added via `c.Error()` |--## Error Detection--A request is considered an error when any of these are true:--- A **panic** occurred during handler execution-- **Gin context errors** exist (`c.Errors` is non-empty)-- The response **status code is >= 500**--Error traces use the error sample rate for sampling decisions.--## Panic Recovery--When a panic occurs, the middleware:--1. Recovers the panic-2. Captures the full stack trace as an issue-3. Either re-panics (default, `WithRepanic(true)`) or calls `c.AbortWithStatus(500)`--```go-func riskyHandler(c *gin.Context) {- panic("something went wrong")- // Stack trace is captured, linked to this request's trace-}-```--## Gin Error Chain--Errors added via `c.Error()` are captured as separate issues:--```go-func handler(c *gin.Context) {- if err := doSomething(); err != nil {- c.Error(err) // Captured as an issue- c.AbortWithStatus(500)- return- }-}-```--If the error wraps a `StackTraceError` (from `traceway.NewStackTraceErrorf`), the embedded stack frames are used. Otherwise, the error type and message are recorded.--## Connection package--The connection page in the sidebar will have your projects unique API key and the integration instructions.--<img- src="/connection.png"- alt="SDK Connection Setup"- style={{- maxWidth: "100%",- height: "auto",- marginTop: "16px",- borderRadius: "8px",- border: "1px solid #e5e7eb",- }}-/>--## Next Steps--- [Configuration](/client/gin-middleware/configuration) — All options including route filtering and passthrough settings-- [Error Recording](/client/gin-middleware/error-recording) — Control what request data is captured on errorsdiff --git a/docs/pages/client/hono/index.mdx b/docs/pages/client/hono/index.mdxdeleted file mode 100644index d09e7046..00000000--- a/docs/pages/client/hono/index.mdx+++ /dev/null@@ -1,179 +0,0 @@-# Hono--Instrument your Hono application with OpenTelemetry and export traces and metrics to Traceway. Hono runs on multiple runtimes — this guide covers the Node.js setup. For Cloudflare Workers, see the [Cloudflare guide](/client/cloudflare).--## Installation--```bash-npm install hono @hono/node-server @hono/otel \- @opentelemetry/sdk-node \- @opentelemetry/auto-instrumentations-node \- @opentelemetry/exporter-trace-otlp-http \- @opentelemetry/exporter-metrics-otlp-http-```--## Setup--Create an `instrumentation.ts` file at the root of your project:--```typescript-import { NodeSDK } from "@opentelemetry/sdk-node";-import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";-import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";-import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";-import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";--const sdk = new NodeSDK({- traceExporter: new OTLPTraceExporter({- url: "https://your-traceway-instance.com/api/otel/v1/traces",- headers: { Authorization: "Bearer your-project-token" },- }),-- metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: { Authorization: "Bearer your-project-token" },- }),- exportIntervalMillis: 30_000,- }),-- // Disable instrumentation-http — @hono/otel handles HTTP spans at the middleware level- instrumentations: [- getNodeAutoInstrumentations({- "@opentelemetry/instrumentation-http": { enabled: false },- }),- ],-});--sdk.start();-```--> **Why disable `instrumentation-http`?** Hono uses `@hono/node-server`, which imports Node's `http` module via ESM. On Node.js 22+, OTel's `import-in-the-middle` hook cannot intercept ESM imports of built-in modules, so `instrumentation-http` never patches the server — no spans are created. The [`@hono/otel`](https://www.npmjs.com/package/@hono/otel) middleware solves this by creating spans at the Hono middleware level instead, which works on all Node.js versions and runtimes.--## Application Code--The `@hono/otel` middleware handles everything automatically — endpoint grouping via `http.route`, status codes, and exception recording:--```typescript-import { serve } from "@hono/node-server";-import { Hono } from "hono";-import { otel } from "@hono/otel";--const app = new Hono();--app.use(otel());--app.get("/api/users", (c) => {- return c.json({ users: [] });-});--app.get("/api/users/:id", (c) => {- const id = c.req.param("id");- return c.json({ id, name: "John" });-});--serve({ fetch: app.fetch, port: 3000 });-```--## Running Your App--Load the instrumentation file before your application code:--```bash-# CommonJS-node --require ./instrumentation.js server.js--# ESM / TypeScript (with tsx)-node --import ./instrumentation.ts server.ts-```--Or set via environment variable:--```bash-export NODE_OPTIONS="--require ./instrumentation.js"-node server.js-```--## What Gets Captured--With `@hono/otel` and the auto-instrumentations enabled, the following is captured automatically:--| Layer | What's Captured |-|-------|-----------------|-| `@hono/otel` middleware | Incoming requests with method, route, status code, duration |-| `@hono/otel` middleware | `http.route` for endpoint grouping (e.g., `GET /api/users/:id`) |-| `@hono/otel` middleware | Exceptions recorded as Issues with stack traces |-| Database (`pg`, `mysql2`, `mongodb`) | Query spans with SQL statements, connection info |-| Cache (`redis`, `ioredis`) | Cache operation spans |-| Outgoing `fetch()` | Outgoing HTTP request spans (auto via `instrumentation-undici`) |-| DNS | DNS lookup spans |--Root SERVER spans become **Endpoints** in Traceway, child spans become **Spans**, and exception events become **Issues**.--### Database Auto-Instrumentation--Database libraries like `pg`, `mysql2`, `mongodb`, and `ioredis` are CommonJS packages. Even in an ESM Hono app, OTel's `require-in-the-middle` hook patches them successfully. Database queries automatically appear as child spans under the `@hono/otel` root span — no manual instrumentation needed:--```typescript-import pg from "pg";-import { otel } from "@hono/otel";--const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });-const app = new Hono();--app.use(otel());--app.get("/api/users/:id", async (c) => {- const id = c.req.param("id");- // This query automatically creates a child span:- // pg.query → SELECT * FROM users WHERE id = $1- const { rows } = await pool.query("SELECT * FROM users WHERE id = $1", [id]);- return c.json(rows[0]);-});-```--The resulting trace in Traceway:--```-GET /api/users/:id ← Endpoint (from @hono/otel)- └─ pg.query ← Span (from instrumentation-pg)- ├─ dns.lookup ← Span (from instrumentation-dns)- └─ tcp.connect ← Span (from instrumentation-net)-```--> **Note:** This works because `pg`, `mysql2`, and similar packages use CommonJS internally. The ESM import issue that affects `http` (a Node.js built-in) does not affect third-party CJS npm packages.--## Environment Variable Alternative--```bash-export OTEL_SERVICE_NAME="my-hono-app"-export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"-export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--## Test Your Integration--Add a test endpoint:--```typescript-app.get("/test-error", () => {- throw new Error("Test error from Hono");-});-```--Visit `/test-error` and check your Traceway dashboard — the error should appear under **Issues** with a full stack trace.--## Multi-Runtime Notes--| Runtime | Instrumentation |-|---------|----------------|-| **Node.js** | OTel Node SDK (this guide) |-| **Cloudflare Workers** | Native OTLP export — see [Cloudflare guide](/client/cloudflare) |-| **Deno** | Use [Deno OTel](https://docs.deno.com/runtime/fundamentals/open_telemetry/) with OTLP/HTTP export to Traceway |-| **Bun** | Use `@opentelemetry/sdk-node` (Bun supports most Node.js OTel packages) |--## Next Steps--- [Traces](/client/hono/traces) — manual spans, middleware instrumentation, and exception recording-- [Node.js OTel Guide](/client/node-sdk) — general Node.js setup details-- [OTel Overview](/client/otel) — protocol details and authenticationdiff --git a/docs/pages/client/http-middleware/_meta.json b/docs/pages/client/http-middleware/_meta.jsondeleted file mode 100644index a5183284..00000000--- a/docs/pages/client/http-middleware/_meta.json+++ /dev/null@@ -1,5 +0,0 @@-{- "index": "Quick Start",- "configuration": { "display": "hidden" },- "error-recording": "Error Recording"-}diff --git a/docs/pages/client/http-middleware/configuration.mdx b/docs/pages/client/http-middleware/configuration.mdxdeleted file mode 100644index be02207b..00000000--- a/docs/pages/client/http-middleware/configuration.mdx+++ /dev/null@@ -1,150 +0,0 @@-# HTTP Middleware Configuration--## Middleware Options--| Option | Description | Default |-|--------|-------------|---------|-| `WithRepanic(bool)` | Re-panic after capturing, letting upstream recovery handle it | `true` |-| `WithIgnoredPaths(...string)` | Skip recording for specific URL paths | none |-| `WithStreamingPaths(...string)` | Mark paths as long-lived streaming responses (SSE / WS / long-poll) | none |-| `WithFilter(func(*http.Request) bool)` | Custom filter function — return `false` to skip recording | none |-| `WithOnErrorRecording(RecordingFlag)` | What additional data to capture on errors | `0` (none) |--## Passthrough Core Options--These options are forwarded to the core `traceway` client:--| Option | Description | Default |-|--------|-------------|---------|-| `WithDebug(bool)` | Enable debug logging | `false` |-| `WithVersion(string)` | Application version shown in dashboard | `""` |-| `WithServerName(string)` | Server hostname identifier | Auto-detected |-| `WithSampleRate(float64)` | Percentage of normal traces to record (0.0–1.0) | `1.0` |-| `WithErrorSampleRate(float64)` | Percentage of error traces to record (0.0–1.0) | `1.0` |-| `WithCollectionInterval(time.Duration)` | How often batched data is sent | `5s` |-| `WithMaxCollectionFrames(int)` | Maximum frames to buffer | `12` |-| `WithUploadTimeout(time.Duration)` | HTTP upload timeout | `2s` |-| `WithMetricsInterval(time.Duration)` | System metrics collection interval | `30s` |--## Repanic Behavior--When `WithRepanic(true)` (default), the middleware re-panics after capturing the stack trace. This allows upstream recovery middleware to handle the panic as well.--When `WithRepanic(false)`, the middleware responds with HTTP 500 and does not re-panic.--## Path Filtering--### WithIgnoredPaths--Skip recording for specific paths:--```go-tracewayhttp.New(- connectionString,- tracewayhttp.WithIgnoredPaths("/health", "/ready", "/metrics"),-)-```--Ignored paths are still served — the request passes through to your handler, but no trace is recorded.--### WithFilter--For dynamic filtering logic, use `WithFilter`. Return `false` to skip recording:--```go-tracewayhttp.New(- connectionString,- tracewayhttp.WithFilter(func(r *http.Request) bool {- // Skip internal service-to-service calls- if r.Header.Get("X-Internal") == "true" {- return false- }- // Skip static assets- if strings.HasPrefix(r.URL.Path, "/static/") {- return false- }- return true- }),-)-```--When the filter returns `false`, the request is passed directly to your handler without any Traceway wrapping.--## Streaming Responses (SSE / WebSocket / long-poll)--Long-lived streaming routes have a request lifetime that equals the connection duration, not request processing latency. Recording them as normal endpoints would dominate P50/P95/P99 rankings, tank Apdex, and trigger slow-endpoint alerts.--The middleware **auto-detects** streaming responses by inspecting the response headers after the handler returns:--| Signal | Detection |-|--------|-----------|-| Server-Sent Events | `Content-Type: text/event-stream` |-| WebSocket upgrade | HTTP status `101 Switching Protocols`, or `Upgrade: websocket` |--Streaming requests are still recorded — status code, error rate, and throughput remain visible on the dashboard — but their duration is **excluded** from P50/P95/P99, Apdex, and impact scoring.--For paths the SDK can't auto-detect (long-polling, chunked JSON streams, custom protocols), declare them up-front:--```go-tracewayhttp.New(- connectionString,- tracewayhttp.WithStreamingPaths("/api/long-poll", "/api/stream/json"),-)-```--Or mark dynamically from any handler:--```go-func eventsHandler(w http.ResponseWriter, r *http.Request) {- traceway.MarkStream(r.Context())- // ... write stream ...-}-```--## Differences from Gin Middleware--The HTTP middleware does **not** have `WithRecordUnmatched` or `WithRecordStatic` options — these are Gin-specific because `net/http` has no built-in route registration to distinguish matched vs unmatched routes.--Instead, the HTTP middleware provides `WithFilter` for custom request filtering, which is not available in the Gin middleware.--## Complete Example--```go-package main--import (- "net/http"- "time"-- traceway "go.tracewayapp.com"- tracewayhttp "go.tracewayapp.com/tracewayhttp"-)--func main() {- traceway.ConfigureAttributes(func(s *traceway.Attributes) {- s.SetTag("environment", "production")- s.SetTag("service", "order-api")- })-- mux := http.NewServeMux()- mux.HandleFunc("/api/orders", handleOrders)-- handler := tracewayhttp.New(- "your-token@http://localhost:8082/api/report",- tracewayhttp.WithVersion("1.2.3"),- tracewayhttp.WithServerName("api-server-01"),- tracewayhttp.WithRepanic(true),- tracewayhttp.WithIgnoredPaths("/health", "/ready"),- tracewayhttp.WithFilter(func(r *http.Request) bool {- return r.Header.Get("X-Internal") != "true"- }),- tracewayhttp.WithOnErrorRecording(- tracewayhttp.RecordingUrl | tracewayhttp.RecordingQuery,- ),- tracewayhttp.WithCollectionInterval(5 * time.Second),- tracewayhttp.WithMetricsInterval(30 * time.Second),- )(mux)-- http.ListenAndServe(":8080", handler)-}-```diff --git a/docs/pages/client/http-middleware/error-recording.mdx b/docs/pages/client/http-middleware/error-recording.mdxdeleted file mode 100644index ab3c184a..00000000--- a/docs/pages/client/http-middleware/error-recording.mdx+++ /dev/null@@ -1,85 +0,0 @@-# HTTP Error Recording--When an exception occurs during a request, you can control what additional request data is captured alongside the stack trace.--## Recording Flags--Configure error recording with `WithOnErrorRecording`:--```go-tracewayhttp.New(- connectionString,- tracewayhttp.WithOnErrorRecording(- tracewayhttp.RecordingUrl | tracewayhttp.RecordingQuery,- ),-)-```--### Available Flags--| Flag | Description | Privacy Impact |-|------|-------------|----------------|-| `RecordingUrl` | Request URL path | Low |-| `RecordingQuery` | Query parameters as JSON | Medium |-| `RecordingBody` | Request body (JSON content-type only, max 64KB) | High |-| `RecordingHeader` | Request headers as JSON | High |--### Combining Flags--Flags are combined with the `|` operator:--```go-// Record URL and query parameters-tracewayhttp.RecordingUrl | tracewayhttp.RecordingQuery--// Record everything-tracewayhttp.RecordingUrl | tracewayhttp.RecordingQuery | tracewayhttp.RecordingBody | tracewayhttp.RecordingHeader-```--## Body Capture Details--Body recording has two constraints:--- **Content-Type must be `application/json`** — non-JSON bodies are not captured-- **Maximum size is 64KB** — bodies larger than 64KB are truncated--After reading the body for capture, it is **restored** so downstream handlers can still read it. The body is reconstructed using `io.MultiReader` with the captured bytes and the remaining body.--## Privacy Considerations--Body and header recording may capture sensitive data:--- Authentication tokens in headers-- Personal information in request bodies-- API keys in query parameters--Use minimal recording flags in production. A recommended production setup:--```go-tracewayhttp.WithOnErrorRecording(- tracewayhttp.RecordingUrl | tracewayhttp.RecordingQuery,-)-```--Reserve full recording for development environments:--```go-tracewayhttp.WithOnErrorRecording(- tracewayhttp.RecordingUrl |- tracewayhttp.RecordingQuery |- tracewayhttp.RecordingBody |- tracewayhttp.RecordingHeader,-)-```--## Captured Data Format--Recorded data is attached as attributes on the issue:--| Attribute Key | Value |-|---------------|-------|-| `url` | URL path string |-| `query` | JSON-encoded query parameters |-| `body` | Raw request body string |-| `headers` | JSON-encoded request headers |-| `user agent` | User-Agent header value (always captured on error) |diff --git a/docs/pages/client/http-middleware/index.mdx b/docs/pages/client/http-middleware/index.mdxdeleted file mode 100644index ffc76204..00000000--- a/docs/pages/client/http-middleware/index.mdx+++ /dev/null@@ -1,106 +0,0 @@-# HTTP Middleware--import { Callout } from "nextra/components";--<Callout type="warning">- The Traceway Go SDK is no longer the recommended way to integrate with Traceway. For new projects, instrument with [OpenTelemetry](/client/otel) — it works across languages and Traceway natively ingests OTLP. The Go SDK is still supported, but new features land in the OTel path first.-</Callout>--The `tracewayhttp` package provides middleware for standard `net/http` servers. It automatically creates a trace for each request, captures panics, and detects client IP addresses.--## Installation--```bash-go get go.tracewayapp.com-go get go.tracewayapp.com/tracewayhttp-```--## Basic Setup--```go-package main--import (- "net/http"-- tracewayhttp "go.tracewayapp.com/tracewayhttp"-)--func main() {- mux := http.NewServeMux()- mux.HandleFunc("/api/users", getUsers)- mux.HandleFunc("/api/orders", getOrders)-- // Wrap with Traceway middleware- handler := tracewayhttp.New(- "your-token@http://localhost:8082/api/report",- )(mux)-- http.ListenAndServe(":8080", handler)-}-```--The `New` function returns `func(http.Handler) http.Handler`, so you chain it around your handler.--## What Gets Captured--The middleware automatically captures for every request:--| Data | Description |-| --------------- | ----------------------------------------------- |-| **Endpoint** | HTTP method + URL path (e.g., `GET /api/users`) |-| **Duration** | Request processing time |-| **Status Code** | HTTP response status |-| **Body Size** | Response body size in bytes |-| **Client IP** | Detected from headers or connection |-| **Panics** | Recovered and captured with full stack trace |--## ResponseWriter Wrapping--The middleware wraps the `http.ResponseWriter` to track the response status code and bytes written. The wrapped writer preserves the original writer's `Flusher`, `Hijacker`, and `Unwrap` interfaces.--## Client IP Detection--Client IP is resolved in this order:--1. `X-Forwarded-For` header (first IP in the list)-2. `X-Real-IP` header-3. `RemoteAddr` from the connection--## Error Detection--A request is considered an error when:--- A **panic** occurs during handler execution-- The response status code is **>= 500**--Error traces use the error sample rate for sampling decisions.--## Panic Handling--When a panic occurs, the middleware:--1. Recovers the panic-2. Captures the full stack trace as an issue-3. Either re-panics (default, `WithRepanic(true)`) or responds with 500--## Connection package--The connection page in the sidebar will have your projects unique API key and the integration instructions.--<img- src="/connection.png"- alt="SDK Connection Setup"- style={{- maxWidth: "100%",- height: "auto",- marginTop: "16px",- borderRadius: "8px",- border: "1px solid #e5e7eb",- }}-/>--## Next Steps--- [Configuration](/client/http-middleware/configuration) — All options and passthrough settings-- [Error Recording](/client/http-middleware/error-recording) — Control what request data is captured on errorsdiff --git a/docs/pages/client/index.mdx b/docs/pages/client/index.mdxindex a7957ff4..23defe55 100644--- a/docs/pages/client/index.mdx+++ b/docs/pages/client/index.mdx@@ -1,7 +1,14 @@import FrameworkPicker from "../../components/FrameworkPicker";+import AgentSkillsCallout from "../../components/AgentSkillsCallout";# Traceway-Traceway is an open-source error tracking and monitoring platform for Go applications. It provides real-time visibility into your application's health through exception tracking, endpoint monitoring, background tasks, and system metrics. To learn more about the core concepts of Traceway <a href="/learn">Click Here</a>.+Traceway is an open-source error tracking and monitoring platform. It gives you real-time visibility into your application's health through exception tracking, endpoint monitoring, background tasks, and system metrics.++There are two ways to send data to it: backends of any language report over [OpenTelemetry](/client/otel), and browser or mobile apps use the Traceway SDKs.++> **First time here?** Read [Project Structure](/learn/projects) first: one project per backend system, per browser app, and per mobile app.++<AgentSkillsCallout /><FrameworkPicker />diff --git a/docs/pages/client/ios/index.mdx b/docs/pages/client/ios/index.mdxindex f139b62f..9c9e9e94 100644--- a/docs/pages/client/ios/index.mdx+++ b/docs/pages/client/ios/index.mdx@@ -23,12 +23,12 @@ dependencies: [],```-Then add `"Traceway"` to your target's dependencies:+Then add the `Traceway` product to your target's dependencies. The package identity is `traceway-ios` and the library product is `Traceway`, so the two names differ and SwiftPM needs the explicit `.product(name:package:)` form:```swift.target(name: "MyApp",- dependencies: ["Traceway"]+ dependencies: [.product(name: "Traceway", package: "traceway-ios")]),```@@ -139,15 +139,35 @@ Traceway.start(## Test Your Integration-Add a button that triggers a crash to verify everything is wired up:+Start with a caught error. It uploads immediately and works with the Xcode debugger attached, so it is the fastest way to confirm the wiring:```swift-Button("Send Test Error") {- fatalError("Test error from Traceway integration")+import SwiftUI+import Traceway++struct TestError: Error {}++struct TracewayTestView: View {+ var body: some View {+ Button("Send Test Error") {+ Traceway.capture(TestError())+ Traceway.flush(timeout: 5)+ }+ }+}+```++Tap it and check your Traceway dashboard. The error should appear on the Issues page within a few seconds.++To verify **hard-crash** capture as well, add a second button:++```swift+Button("Force Crash") {+ fatalError("Test crash from Traceway integration")}```-When the Xcode debugger is attached, lldb intercepts signals first, so a crash never reaches the SDK. Test crash capture by running **without** the debugger: launch the app from the home screen, trigger the crash, then relaunch and watch the report upload on the next launch. Caught errors sent through `Traceway.capture(...)` upload immediately and do not need this.+This one does nothing while you are running from Xcode. Hard crashes are intercepted with POSIX signal handlers, and when the debugger is attached lldb intercepts the signal first, so the SDK's handler never runs. Test it **without** the debugger: install once from Xcode, stop the debugger, launch the app from the home screen, tap the button, then relaunch and watch the report upload on the next launch.## Release Symbolication (dSYM upload)diff --git a/docs/pages/client/jquery/index.mdx b/docs/pages/client/jquery/index.mdxindex b9c612d7..2fbfc86f 100644--- a/docs/pages/client/jquery/index.mdx+++ b/docs/pages/client/jquery/index.mdx@@ -7,20 +7,26 @@ Integrate Traceway into your jQuery application with the `@tracewayapp/jquery` p### Via npm```bash-npm install @tracewayapp/jquery+npm install @tracewayapp/jquery jquery``````javascript+import $ from "jquery";import { init } from "@tracewayapp/jquery";+// The AJAX hook binds to the global jQuery, so publish it before init().+window.jQuery = window.$ = $;+init("your-token@https://cloud.tracewayapp.com/api/report");```-Session recording is on by default — the last ~30s of DOM events ship with every captured exception.+`init()` binds `$(document).ajaxError()` on `window.jQuery` (or `window.$`). If jQuery is only a bundled module and never lands on `window`, uncaught errors and `captureException()` still work but AJAX errors are silently skipped. Call `init()` after jQuery is on the page. If jQuery already comes from a `<script>` tag, it is global already and you can drop the two jQuery lines above.++Session recording is on by default. The last ~30s of DOM events ship with every captured exception.### Via CDN-No build step needed — add two script tags after jQuery:+No build step needed. Add two script tags after jQuery:```html<script src="https://code.jquery.com/jquery-3.7.1.min.js"></script>@@ -30,11 +36,12 @@ No build step needed — add two script tags after jQuery:</script>```+The CDN build exposes everything on the `TracewayJQuery` global: `TracewayJQuery.captureException`, `TracewayJQuery.captureMessage`, `TracewayJQuery.setAttribute`, `TracewayJQuery.flush`, and so on. The `import` examples below are for the npm build, so prefix them with `TracewayJQuery.` when you use the CDN.+## What It DoesOnce initialized, the SDK automatically:-This automatically:- Captures uncaught JavaScript errors and unhandled promise rejections- Captures jQuery AJAX errors via `$(document).ajaxError()`- Injects `traceway-trace-id` headers into same-origin `$.ajax()` requests@@ -50,7 +57,7 @@ $.ajax({url: "/api/users",method: "GET",error: function (jqXHR, textStatus, errorThrown) {- // Your error handling — Traceway also captures this+ // Your error handling; Traceway also captures this},});```@@ -84,7 +91,7 @@ init("your-token@https://cloud.tracewayapp.com/api/report", {| Option | Type | Default | Description ||--------|------|---------|-------------|-| `debug` | `boolean` | `false` | Log captured events to the browser console |+| `debug` | `boolean` | `false` | Log filtered-out events and failed uploads to the browser console. Successful captures are not logged || `debounceMs` | `number` | `1500` | Batch delay in milliseconds || `retryDelayMs` | `number` | `10000` | Retry delay for failed uploads || `version` | `string` | `undefined` | Your application version |@@ -96,6 +103,9 @@ init("your-token@https://cloud.tracewayapp.com/api/report", {| `captureLogs` | `boolean` | `true` | Mirror `console.*` calls into the rolling log buffer || `captureNetwork` | `boolean` | `true` | Record `fetch` / `XHR` calls as network actions || `captureNavigation` | `boolean` | `true` | Record History API push / replace / pop transitions |+| `eventsWindowMs` | `number` | `10000` (`30000` w/ `recordAllSessions`) | Rolling window the log and action buffers retain |+| `eventsMaxCount` | `number` | `200` (`600` w/ `recordAllSessions`) | Hard cap on entries kept independently in the log and action buffers |+| `captureHttpServerErrors` | `boolean` | `false` | Report every `fetch` response with status >= 500 as a synthetic exception. It is wired into the `fetch` wrapper only, so `$.ajax()` does not trigger it. Use the `ajaxError` capture below for jQuery requests |## Error Filtering@@ -148,17 +158,19 @@ Layering on each event: **defaults < global scope < per-call**. See [Sessions](/The SDK automatically instruments both `XMLHttpRequest` (used by `$.ajax()`) and `fetch` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them.+The backend has to hold up its end: it must set the incoming header on its server span as the `traceway.distributed_trace_id` attribute. The Symfony bundle does this for you, everything else needs one small middleware. See [Distributed Tracing](/client/js-sdk/distributed-tracing#backend-setup).+### With Symfony + OpenTelemetry-If your backend uses the [Symfony OpenTelemetry Bundle](/client/symfony), distributed tracing works out of the box. The bundle reads the `traceway-trace-id` header from incoming requests and attaches it as a span attribute, linking your jQuery frontend errors to Symfony backend traces.+If your backend uses the [Symfony OpenTelemetry Bundle](/client/otel/symfony), distributed tracing works out of the box. The bundle reads the `traceway-trace-id` header from incoming requests and attaches it as a span attribute, linking your jQuery frontend errors to Symfony backend traces.```javascript-// Frontend — trace header is injected automatically+// Frontend: trace header is injected automatically$.post("/api/orders", { item: "widget" });``````php-// Backend — the bundle reads traceway-trace-id automatically+// Backend: the bundle reads traceway-trace-id automatically// No extra configuration needed in your Symfony controllers```@@ -172,9 +184,11 @@ $("#test-button").on("click", function () {});```+On the CDN build, call `TracewayJQuery.captureException(new Error("Test error from jQuery"))` instead.+Click the button and check your Traceway dashboard to verify the error appears.## Next Steps-- [Distributed Tracing](/client/js-sdk/distributed-tracing) — Frontend ↔ backend correlation-- [JS SDK Reference](/client/js-sdk) — Full API reference+- [Distributed Tracing](/client/js-sdk/distributed-tracing): Frontend ↔ backend correlation+- [JS SDK Reference](/client/js-sdk): Full API referencediff --git a/docs/pages/client/js-sdk/distributed-tracing.mdx b/docs/pages/client/js-sdk/distributed-tracing.mdxindex 3b209399..b5e23c1e 100644--- a/docs/pages/client/js-sdk/distributed-tracing.mdx+++ b/docs/pages/client/js-sdk/distributed-tracing.mdx@@ -6,13 +6,15 @@ Correlate frontend errors with the backend requests that caused them. The Tracew1. The frontend SDK generates a unique trace ID for each outgoing HTTP request2. The ID is sent as a `traceway-trace-id` header to your backend-3. Your backend middleware reads the header and attaches it to the trace context-4. Any exceptions or traces captured on either side are tagged with the same ID+3. Your backend sets that ID on its server span as the `traceway.distributed_trace_id` attribute+4. Traceway uses that attribute as the row's distributed trace ID, so both sides land on one trace5. In the Traceway dashboard, you can jump between the frontend error and the backend request-## With fetch (Automatic)+Step 3 is the one thing you have to add yourself. See [Backend Setup](#backend-setup) below.-When you call `init()`, the SDK automatically instruments `window.fetch` to inject the `traceway-trace-id` header on **same-origin** requests. No extra code is needed.+## Browser Side (Automatic)++When you call `init()`, the SDK instruments both `window.fetch` and `XMLHttpRequest` to inject the `traceway-trace-id` header on **same-origin** requests. No extra code is needed.```javascriptimport { init } from "@tracewayapp/frontend";@@ -26,110 +28,99 @@ const res = await fetch("/api/orders", {});```-Cross-origin requests are **not** instrumented to prevent leaking the header to third-party services.+Cross-origin requests are **not** instrumented, to prevent leaking the header to third-party services.-If an error is thrown during or after a fetch call (e.g., in a `.then()` handler), the SDK automatically tags the captured exception with the distributed trace ID of that in-flight request.+An exception captured while a request is still **in flight** is automatically tagged with that request's distributed trace ID. Once the request settles, the ID is cleared, so a `.then()` handler or any code after `await` sees nothing and the exception is stored unlinked. See [Manual Capture After an Await](#manual-capture-after-an-await) for that case.-## With Axios (Manual Interceptor)+## With Axios-For Axios, use the `createAxiosInterceptor` helper to add the header to all requests:+Axios uses `XMLHttpRequest` in the browser, and the SDK already instruments it. The header is injected and the active trace ID is set, so Axios requests correlate on their own with no setup at all.```javascriptimport axios from "axios";-import { createAxiosInterceptor } from "@tracewayapp/frontend";const api = axios.create({ baseURL: "/api" });-api.interceptors.request.use(createAxiosInterceptor());-// All requests through this instance include the traceway-trace-id header+// Already carries the traceway-trace-id headerconst res = await api.post("/orders", order);```-### Exception Correlation with Axios--The Axios interceptor injects the header but does not set the active trace ID used by global error handlers. To correlate caught exceptions with Axios requests, pass the trace ID explicitly:--```javascript-import { captureException } from "@tracewayapp/frontend";--try {- await api.post("/orders", order);-} catch (err) {- captureException(err, {- distributedTraceId: err.config?.headers?.["traceway-trace-id"],- });-}-```+> **Do not register `createAxiosInterceptor()` in a browser app.** It generates a *second*, different ID under the same header name. `XMLHttpRequest.setRequestHeader` combines duplicate header names, so the backend receives `"<uuid1>, <uuid2>"`, which is not a valid UUID and is dropped without an error. Adding the interceptor breaks correlation that already worked.## Backend Setup-Your backend must read the `traceway-trace-id` header and pass it into the trace context. This is handled automatically by Traceway's backend middleware.--### NestJS+Your backend closes the loop in one middleware, in whatever language it uses:-The `TracewayMiddleware` from `@tracewayapp/nestjs` automatically:-- Reads `traceway-trace-id` from incoming request headers-- Echoes the ID back in the response header-- Stores it in the trace context for the duration of the request+1. Read the `traceway-trace-id` request header.+2. Set it on the active server span as the attribute `traceway.distributed_trace_id`. Traceway reads that attribute and uses it as the row's distributed trace ID, overriding the one derived from the OTel trace ID. This is what puts the browser exception and the backend endpoint on the same distributed trace. The value must be a bare UUID; anything else is ignored silently.+3. Echo the header back on the response so manual `captureException` calls after `await fetch` can read it. This step is optional. Cross-origin, also add `traceway-trace-id` to `Access-Control-Expose-Headers`.-No additional configuration is needed — if you have the middleware installed, distributed tracing works out of the box.+The [Symfony bundle](/client/otel/symfony) does step 2 for you; add step 3 yourself if you want it. Node.js, NestJS, Next.js, Hono, Cloudflare Workers, Laravel, and Django install vanilla OpenTelemetry, which knows nothing about the header, so do all three:-```typescript-import { MiddlewareConsumer, Module, NestModule } from "@nestjs/common";-import { TracewayModule, TracewayMiddleware } from "@tracewayapp/nestjs";+```javascript+import { trace } from "@opentelemetry/api";-@Module({- imports: [- TracewayModule.forRoot({- connectionString: "your-token@https://traceway.example.com/api/report",- }),- ],-})-export class AppModule implements NestModule {- configure(consumer: MiddlewareConsumer) {- consumer.apply(TracewayMiddleware).forRoutes("*");+app.use((req, res, next) => {+ const id = req.headers["traceway-trace-id"];+ if (id) {+ trace.getActiveSpan()?.setAttribute("traceway.distributed_trace_id", id);+ res.setHeader("traceway-trace-id", id);}-}+ next();+});```-### Node.js (Express / Custom)+Register it after your OpenTelemetry instrumentation, so there is an active server span to annotate. See the [OpenTelemetry guides](/client/otel) for the exporter setup itself.-If you're using the `@tracewayapp/backend` package directly, extract the header and pass it to `withTraceContext`:+To verify, open the backend endpoint's detail page and select **View distributed trace**. The browser exception appears as a second node.-```javascript-import { withTraceContext, captureCurrentTrace, setTraceResponseInfo } from "@tracewayapp/backend";+## Manual Capture After an Await-app.use((req, res, next) => {- const distributedTraceId = req.headers["traceway-trace-id"];+The SDK holds the active trace ID only while the request is in flight and clears it as soon as the request settles. A `captureException` call that runs after `await fetch` therefore sees `null` and stores the exception unlinked, even though the request header was sent correctly.++Read the ID synchronously, after starting the request and before awaiting it:- if (distributedTraceId) {- res.setHeader("traceway-trace-id", distributedTraceId);+```javascript+import {+ captureExceptionWithAttributes,+ getActiveDistributedTraceId,+} from "@tracewayapp/frontend";++async function request(path, options) {+ const pending = fetch(path, options);+ const distributedTraceId = getActiveDistributedTraceId() || undefined;+ const res = await pending;++ if (!res.ok) {+ const err = new Error(`Request to ${path} failed (${res.status})`);+ captureExceptionWithAttributes(+ err,+ { path, method: options?.method || "GET", status: String(res.status) },+ distributedTraceId ? { distributedTraceId } : undefined+ );+ throw err;}- withTraceContext({ endpoint: "", clientIP: req.ip, distributedTraceId }, () => {- res.on("finish", () => {- setTraceResponseInfo(res.statusCode, 0);- captureCurrentTrace();- });- next();- });-});+ return res.json();+}```+No other code can run between the `fetch` call and the `getActiveDistributedTraceId` call, so the ID always belongs to this request. If your backend echoes the header (step 3 above), `res.headers.get("traceway-trace-id")` gives you the same value after the await.+## API Reference| Export | Description ||--------|-------------|-| `createAxiosInterceptor()` | Returns an Axios request interceptor that adds the `traceway-trace-id` header |-| `getActiveDistributedTraceId()` | Returns the trace ID of the currently in-flight fetch request, or `null` |+| `getActiveDistributedTraceId()` | Returns the trace ID of the request currently in flight, or `null` || `DISTRIBUTED_TRACE_HEADER` | The header name constant: `"traceway-trace-id"` |+| `createAxiosInterceptor()` | Returns an Axios request interceptor that adds the header. Not for browser apps: see the warning above |## How the Active Trace ID Works-The SDK keeps a module-level variable that tracks which fetch call is currently in-flight:+The SDK keeps a module-level variable that tracks which request is currently in flight:-1. Before each same-origin `fetch()`, a new UUID is generated and stored as the "active" trace ID+1. Before each same-origin `fetch()` or `XMLHttpRequest.send()`, a new UUID is generated and stored as the "active" trace ID2. The same ID is injected as the `traceway-trace-id` header-3. When the fetch promise settles (resolves or rejects), the active ID is cleared+3. When the request settles (resolves, rejects, or errors), the active ID is cleared-If an exception is captured (via `captureException`, `window.onerror`, or `window.onunhandledrejection`) while a fetch is in-flight, the SDK automatically attaches the active trace ID to the exception. This is how frontend errors get linked to their triggering backend request without any manual work.+If an exception is captured (via `captureException`, `window.onerror`, or `window.onunhandledrejection`) while a request is in flight, the SDK automatically attaches the active trace ID to the exception. This is how frontend errors get linked to their triggering backend request without any manual work.diff --git a/docs/pages/client/js-sdk/exceptions.mdx b/docs/pages/client/js-sdk/exceptions.mdxindex c9b7df33..20e60583 100644--- a/docs/pages/client/js-sdk/exceptions.mdx+++ b/docs/pages/client/js-sdk/exceptions.mdx@@ -72,18 +72,28 @@ class ValidationError extends Error {captureException(new ValidationError("Invalid email format"));```+The type shown in Traceway comes from the error's **class name**, not from `this.name`. Minifiers rename classes, so in a production build the issue above reads `r: Invalid email format` rather than `ValidationError: Invalid email format`. Put anything you need to search or group on into the message or into attributes.+## Non-Error Values-If you capture a non-Error value, the SDK wraps it:+Pass real `Error` objects. The SDK reads only the value's class name and its `message` property, so anything else loses information and arrives with no stack trace:```javascript-// These work, but Error objects are preferred+// Message is lost. Stored as "String: undefined".captureException("Something went wrong");++// Stored as "Object: Server error". The `code` field is dropped.captureException({ code: 500, message: "Server error" });```-For best stack traces, always throw and catch actual `Error` objects.+Wrap the value in an `Error` instead, and pass the extra fields as attributes:++```javascript+import { captureExceptionWithAttributes } from "@tracewayapp/frontend";++captureExceptionWithAttributes(new Error("Server error"), { code: "500" });+```## React Error Boundaries-For React applications, use the `@tracewayapp/react` package. Its `<TracewayProvider>` itself acts as an error boundary — render-time exceptions are captured automatically and re-thrown so the app behaves exactly as without Traceway. See the [React documentation](/client/react) for setup, or the [Error Boundary page](/client/react/error-boundary) if you need a custom fallback UI for a specific subtree.+For React applications, use the `@tracewayapp/react` package. Its `<TracewayProvider>` itself acts as an error boundary: render-time exceptions are captured automatically and re-thrown so the app behaves exactly as without Traceway. See the [React documentation](/client/react) for setup, or the [Error Boundary page](/client/react/error-boundary) if you need a custom fallback UI for a specific subtree.diff --git a/docs/pages/client/js-sdk/flush.mdx b/docs/pages/client/js-sdk/flush.mdxindex 7c3d8d77..4465bed8 100644--- a/docs/pages/client/js-sdk/flush.mdx+++ b/docs/pages/client/js-sdk/flush.mdx@@ -98,7 +98,7 @@ Or use framework-specific packages that handle this automatically.Adjust timing based on your needs:```javascript-init("your-token@...", {+init("your-token@https://traceway.example.com/api/report", {// Faster batching for critical appsdebounceMs: 500,retryDelayMs: 5000,@@ -106,7 +106,7 @@ init("your-token@...", {``````javascript-init("your-token@...", {+init("your-token@https://traceway.example.com/api/report", {// More aggressive batching for high-traffic appsdebounceMs: 3000,retryDelayMs: 15000,diff --git a/docs/pages/client/js-sdk/index.mdx b/docs/pages/client/js-sdk/index.mdxindex ec7cfa6a..bc8da348 100644--- a/docs/pages/client/js-sdk/index.mdx+++ b/docs/pages/client/js-sdk/index.mdx@@ -29,7 +29,7 @@ npm install @tracewayapp/jquery### Via CDN-The frontend and jQuery packages are also available as standalone scripts — no bundler required:+The frontend and jQuery packages are also available as standalone scripts (no bundler required):```html<!-- Framework-agnostic (exposes window.Traceway) -->@@ -96,8 +96,8 @@ captureMessage("User completed checkout");## Features- **Automatic batching**: Events are batched and sent periodically to reduce network overhead-- **Retry with backoff**: Failed uploads are automatically retried+- **Automatic retry**: Failed uploads are retried after `retryDelayMs` (10 seconds by default)- **Stack trace capture**: Full stack traces are captured for exceptions- **Custom attributes**: Attach metadata to exceptions for better debugging-- **Distributed tracing**: Automatically correlates frontend errors with backend requests via `fetch` instrumentation, with manual support for Axios+- **Distributed tracing**: Automatically correlates frontend errors with backend requests by instrumenting `fetch` and `XMLHttpRequest`, which covers `$.ajax()` and browser Axios with no extra setup- **Lightweight**: Minimal bundle size impactdiff --git a/docs/pages/client/js-sdk/initialization.mdx b/docs/pages/client/js-sdk/initialization.mdxindex bb2d46c3..04a5aada 100644--- a/docs/pages/client/js-sdk/initialization.mdx+++ b/docs/pages/client/js-sdk/initialization.mdx@@ -26,7 +26,7 @@ init("your-token@https://traceway.example.com/api/report", {| Option | Type | Default | Description ||--------|------|---------|-------------|-| `debug` | `boolean` | `false` | Log captured events to the browser console |+| `debug` | `boolean` | `false` | Log dropped events and failed uploads to the browser console. Captured events are not logged || `debounceMs` | `number` | `1500` | Milliseconds to wait before sending batched events || `retryDelayMs` | `number` | `10000` | Milliseconds to wait before retrying failed uploads || `version` | `string` | `undefined` | Your application version (shown in Traceway dashboard) |@@ -34,36 +34,33 @@ init("your-token@https://traceway.example.com/api/report", {| `beforeCapture` | `(exception) => boolean` | `undefined` | Callback to programmatically suppress errors. Return `false` to drop. See [Error Filtering](#error-filtering) || `sessionRecording` | `boolean` | `true` | Enable the rrweb session recorder. Required for both per-exception clips and always-on session recording || `sessionRecordingSegmentDuration` | `number` | `30000` | Length of each rrweb segment in milliseconds. Always-on recording uploads one row per segment, so a longer value means fewer rows and S3 reads at the cost of replay granularity |-| `recordAllSessions` | `boolean` | `false` | Always-on session recording — upload every segment continuously and create a parent `sessions` row, not just exception-bound clips. See [Sessions](/learn/sessions) for the full feature |+| `recordAllSessions` | `boolean` | `false` | Always-on session recording: upload every segment continuously and create a parent `sessions` row, not just exception-bound clips. See [Sessions](/learn/sessions) for the full feature || `captureLogs` | `boolean` | `true` | Mirror `console.{debug,log,info,warn,error}` into the rolling log buffer that ships with each clip / segment || `captureNetwork` | `boolean` | `true` | Record `fetch` and `XMLHttpRequest` calls as network actions || `captureNavigation` | `boolean` | `true` | Record History API push / replace / pop transitions as navigation actions || `eventsWindowMs` | `number` | `10000` (`30000` w/ `recordAllSessions`) | Rolling window the log/action buffers retain || `eventsMaxCount` | `number` | `200` (`600` w/ `recordAllSessions`) | Hard cap on entries kept independently in the log and action buffers |+| `captureHttpServerErrors` | `boolean` | `false` | Report every `fetch` response with status >= 500 as a synthetic exception. 4xx is never included, and it is wired into the `fetch` wrapper only, so `XMLHttpRequest` (including `$.ajax()` and browser Axios) does not trigger it |## Auto-Capture Behavior-By default, the SDK does **not** automatically capture unhandled errors. You need to either:--1. Use a framework integration (React, Vue, Svelte) that provides automatic capture-2. Set up your own global error handler:+`init()` installs the global handlers for you. Uncaught errors (`window.onerror`) and unhandled promise rejections (`window.onunhandledrejection`) are captured automatically in every JS SDK: plain, React, Vue, Svelte, and jQuery alike.```javascript-import { init, captureException } from "@tracewayapp/frontend";+import { init } from "@tracewayapp/frontend";init("your-token@https://traceway.example.com/api/report");-// Capture unhandled errors-window.addEventListener("error", (event) => {- captureException(event.error);-});--// Capture unhandled promise rejections-window.addEventListener("unhandledrejection", (event) => {- captureException(event.reason);+// Nothing else to wire up. This is already reported:+setTimeout(() => {+ throw new Error("Boom");});```+> **Do not add your own `window.addEventListener("error")` or `"unhandledrejection"` listener that calls `captureException`.** The SDK's handlers are already attached, so every error would be reported twice and show up with double the count in the Issues feed.++The framework packages add render-time capture on top of this: React's `TracewayProvider` doubles as an error boundary, and the Vue plugin installs `app.config.errorHandler`. Manual `captureException` calls are only needed where the error never becomes uncaught, such as inside a `try`/`catch` or an event handler that swallows it.+## Error FilteringBy default, the SDK ignores common non-actionable errors: **4xx HTTP errors**, **network errors**, and **timeouts**. These are typically expected application behavior (e.g., form validation returning 422, auth redirects from 401) rather than bugs that need tracking.@@ -72,7 +69,7 @@ By default, the SDK ignores common non-actionable errors: **4xx HTTP errors**, *The following patterns are ignored out of the box:-- `"Failed to fetch"` (Chrome), `"Load failed"` (Safari), `"NetworkError when attempting to fetch resource"` (Firefox), `"Network Error"` (Axios)+- `"Failed to fetch"` (Chrome), `"Load failed"` (Safari), `"NetworkError when attempting to fetch resource"` (Firefox), `"Network Error"` (Axios), `"Network request failed"` (React Native style)- `"The operation was aborted"` (AbortController), any message matching `/timeout/i`- Axios 4xx errors matching `/status code 4\d{2}/`- jQuery/custom 4xx errors matching `/failed: 4\d{2}/`@@ -134,11 +131,13 @@ init("your-token@https://traceway.example.com/api/report", {});```-`beforeCapture` is checked **after** `ignoreErrors` — if a pattern already suppresses the error, the callback is not called. If the callback throws, the error is captured normally (safe default).+`beforeCapture` is checked **after** `ignoreErrors`. If a pattern already suppresses the error, the callback is not called. If the callback throws, the error is captured normally (safe default).## Debug Mode-When `debug: true` is set, the SDK logs all captured events to the browser console. This is useful during development to verify events are being captured correctly.+When `debug: true` is set, the SDK logs what it throws away: errors suppressed by `ignoreErrors`, a `beforeCapture` callback that threw, and failed uploads. Captured events produce no console output at all, so an empty console does not mean the SDK is broken.++To confirm an event was captured, watch for the `POST` to `/api/report` in the Network tab, or call `await flush()` and open the Issues page.```javascriptinit("your-token@https://traceway.example.com/api/report", {@@ -148,7 +147,7 @@ init("your-token@https://traceway.example.com/api/report", {## Custom Attributes (Global Scope)-The SDK auto-collects browser context (`url`, `userAgent`, `viewport`, etc.) on every session and exception. To attach app-level identifiers — `userId`, `tenant`, feature flags — call the imperative scope API once and they ride along every subsequent event:+The SDK auto-collects browser context (`url`, `userAgent`, `viewport`, etc.) on every session and exception. To attach app-level identifiers (`userId`, `tenant`, feature flags), call the imperative scope API once and they ride along every subsequent event:```javascriptimport {diff --git a/docs/pages/client/js-sdk/session-replay.mdx b/docs/pages/client/js-sdk/session-replay.mdxindex ed01689a..e6818d90 100644--- a/docs/pages/client/js-sdk/session-replay.mdx+++ b/docs/pages/client/js-sdk/session-replay.mdx@@ -35,15 +35,15 @@ init("your-token@https://traceway.example.com/api/report", {|--------|------|---------|-------------|| `sessionRecording` | `boolean` | `true` | Enable or disable session recording || `sessionRecordingSegmentDuration` | `number` | `30000` | Duration of each recording segment in milliseconds |-| `recordAllSessions` | `boolean` | `false` | Always-on session recording — upload every segment regardless of whether an exception fires. See [Always-on Session Recording](#always-on-session-recording) |+| `recordAllSessions` | `boolean` | `false` | Always-on session recording: upload every segment regardless of whether an exception fires. See [Always-on Session Recording](#always-on-session-recording) |## Privacy & Masking-Because Session Replay runs on [rrweb](https://github.com/rrweb-io/rrweb) under the hood, you can drop rrweb's standard class hooks onto any element in your app to keep sensitive content out of recordings. No SDK configuration needed — these classes are honored automatically.+Because Session Replay runs on [rrweb](https://github.com/rrweb-io/rrweb) under the hood, you can drop rrweb's standard class hooks onto any element in your app to keep sensitive content out of recordings. No SDK configuration is needed. These classes are honored automatically.-### `rr-mask` — block all text in an element+### `rr-mask`: block all text in an element-Add `class="rr-mask"` to any element whose **text content** should never be recorded. The element is still captured (layout, clicks, and scrolls all work in the replay), but every character of text — including the text of child elements — is replaced with asterisks.+Add `class="rr-mask"` to any element whose **text content** should never be recorded. The element is still captured (layout, clicks, and scrolls all work in the replay), but every character of text (including the text of child elements) is replaced with asterisks.```html<!-- Name visible in the app, hidden in replays -->@@ -58,7 +58,7 @@ Add `class="rr-mask"` to any element whose **text content** should never be recoUse this for anything you treat as PII or regulated data: names, emails, customer IDs, dollar amounts, addresses, health info, chat content, support tickets.-### `rr-block` — hide the element entirely+### `rr-block`: hide the element entirelyAdd `class="rr-block"` when you don't want the element visible at all in the replay. It's replaced by a neutral gray placeholder with the same dimensions. Good for avatars, user-uploaded images, or custom widgets that shouldn't be reconstructed.@@ -66,7 +66,7 @@ Add `class="rr-block"` when you don't want the element visible at all in the rep<img class="rr-block" src={user.avatarUrl} />```-### `rr-ignore` — don't record input values+### `rr-ignore`: don't record input valuesAdd `class="rr-ignore"` on form inputs when you want interaction events (focus, blur, click) captured but **not the characters the user typed**.@@ -76,13 +76,11 @@ Add `class="rr-ignore"` on form inputs when you want interaction events (focus,### What's masked by default-You don't need to add any classes to get the basics — these are always masked:+Exactly one thing is masked without any markup on your side:-- `<input type="password">` — value is never captured-- Elements with the `data-sensitive` attribute-- Elements with the `aria-masked` attribute+- `<input type="password">`. The value is never captured.-Reach for the `rr-*` classes whenever the sensitive content isn't in a form field, or when you need stricter masking than the defaults.+Nothing else is automatic. The recorder does not recognize attributes such as `data-sensitive` or `aria-masked`, so text inside those elements is recorded in full. Every other piece of sensitive content needs one of the `rr-*` classes above.## Segment Rotation@@ -124,5 +122,5 @@ When an exception includes session recording data, a **Session Replay** card app## Next Steps-- [Exceptions](/client/js-sdk/exceptions) — capturing errors with the JS SDK-- [Initialization](/client/js-sdk/initialization) — all SDK configuration options+- [Exceptions](/client/js-sdk/exceptions): capturing errors with the JS SDK+- [Initialization](/client/js-sdk/initialization): all SDK configuration optionsdiff --git a/docs/pages/client/js-sdk/sourcemap-upload.mdx b/docs/pages/client/js-sdk/sourcemap-upload.mdxindex ce46d261..51c384cf 100644--- a/docs/pages/client/js-sdk/sourcemap-upload.mdx+++ b/docs/pages/client/js-sdk/sourcemap-upload.mdx@@ -2,6 +2,23 @@Upload source maps to Traceway so that minified stack traces in production exceptions are automatically deobfuscated.+## Emit the Maps First++Most bundlers do not write `.map` files for a production build unless you ask them to. Turn it on before you wire up any upload:++| Bundler | Setting |+|---------|---------|+| Vite | `build: { sourcemap: true }` in `vite.config.js` |+| Rollup | `output: { sourcemap: true }` |+| webpack | `devtool: "source-map"` |+| Next.js | `productionBrowserSourceMaps: true` in `next.config.js` |++This matters because the uploader treats an empty result as success. With no maps on disk, `traceway-sourcemaps` prints `No .map files found in <dir>` and exits with code 0, so a `postbuild` hook or CI step looks green while nothing was ever uploaded. After a build, confirm the files exist:++```bash+ls dist/assets/*.map+```+## Installation```bash@@ -143,6 +160,6 @@ The SDK's `version` option is unrelated to source maps: it is plain metadata sho## Next Steps-- [Debug IDs](/client/js-sdk/debug-ids) — exact build-to-map matching with the bundler plugin-- [Exceptions](/client/js-sdk/exceptions) — capturing errors with the JS SDK-- [Initialization](/client/js-sdk/initialization) — configuring the SDK `version` option+- [Debug IDs](/client/js-sdk/debug-ids): exact build-to-map matching with the bundler plugin+- [Exceptions](/client/js-sdk/exceptions): capturing errors with the JS SDK+- [Initialization](/client/js-sdk/initialization): configuring the SDK `version` optiondiff --git a/docs/pages/client/laravel/index.mdx b/docs/pages/client/laravel/index.mdxdeleted file mode 100644index 6c2a2668..00000000--- a/docs/pages/client/laravel/index.mdx+++ /dev/null@@ -1,171 +0,0 @@-# Laravel--The [`keepsuit/laravel-opentelemetry`](https://github.com/keepsuit/laravel-opentelemetry) package automatically instruments your Laravel application and exports traces, metrics, and logs to Traceway's OTLP endpoints — no manual instrumentation needed.--## Prerequisites--- PHP 8.2+-- Laravel 10, 11, or 12-- Composer-- A Traceway project (create one in the dashboard)--## Step 1: Install Packages--```bash-composer require keepsuit/laravel-opentelemetry open-telemetry/exporter-otlp php-http/guzzle7-adapter-```--The `open-telemetry/exporter-otlp` and `php-http/guzzle7-adapter` packages provide the OTLP/HTTP exporter and its HTTP client. They are required at runtime to actually ship spans to Traceway.--## Step 2: Publish the Config--```bash-php artisan vendor:publish \- --provider="Keepsuit\LaravelOpenTelemetry\LaravelOpenTelemetryServiceProvider" \- --tag="opentelemetry-config"-```--This creates `config/opentelemetry.php` where you can disable individual instrumentations, configure sampling, set propagators, and tune exporter settings.--## Step 3: Configure Environment Variables--Add the following to your `.env` file, replacing the endpoint and token with your project values:--```bash-OTEL_SERVICE_NAME=my-laravel-app-OTEL_TRACES_EXPORTER=otlp-OTEL_METRICS_EXPORTER=otlp-OTEL_LOGS_EXPORTER=otlp-OTEL_EXPORTER_OTLP_PROTOCOL=http/json-OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel-OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--Your Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.--> **Note:** Use `http/json` instead of `http/protobuf`. The protobuf encoder has known deprecation warnings on PHP 8.4+ that can cause noisy logs. The package defaults to `http/protobuf` — explicitly set `http/json` if you see deprecation noise.--That's it. The package's service provider auto-registers `Keepsuit\LaravelOpenTelemetry\Support\HttpServer\TraceRequestMiddleware` as a global middleware, so every inbound HTTP request is now traced — no manual middleware registration needed.--## Tune Which Instrumentations Are Enabled--The package's instrumentation list lives in the `instrumentation` array of `config/opentelemetry.php`. Each entry can be removed (to disable it) or toggled per-environment via the matching env var. A reasonable default — what ships out of the box — is:--```php-// config/opentelemetry.php-use Keepsuit\LaravelOpenTelemetry\Instrumentation;--return [- // ...- 'instrumentation' => [- Instrumentation\HttpServerInstrumentation::class => [- 'enabled' => env('OTEL_INSTRUMENTATION_HTTP_SERVER', true),- 'excluded_paths' => ['/health', '/up'],- 'excluded_methods' => [],- 'allowed_headers' => [],- 'sensitive_headers' => [],- ],-- Instrumentation\HttpClientInstrumentation::class => [- 'enabled' => env('OTEL_INSTRUMENTATION_HTTP_CLIENT', true),- 'manual' => false,- ],-- Instrumentation\QueryInstrumentation::class => env('OTEL_INSTRUMENTATION_QUERY', true),- Instrumentation\RedisInstrumentation::class => env('OTEL_INSTRUMENTATION_REDIS', true),- Instrumentation\QueueInstrumentation::class => env('OTEL_INSTRUMENTATION_QUEUE', true),- Instrumentation\CacheInstrumentation::class => env('OTEL_INSTRUMENTATION_CACHE', true),- Instrumentation\ViewInstrumentation::class => env('OTEL_INSTRUMENTATION_VIEW', true),- Instrumentation\LivewireInstrumentation::class => env('OTEL_INSTRUMENTATION_LIVEWIRE', true),-- Instrumentation\ConsoleInstrumentation::class => [- 'enabled' => env('OTEL_INSTRUMENTATION_CONSOLE', true),- 'commands' => [],- ],- ],-];-```--| Instrumentation | What it captures |-|-----------------|------------------|-| `HttpServerInstrumentation` | Inbound HTTP requests, status codes, route templates (`http.route`) |-| `HttpClientInstrumentation` | Outbound `Http::` / Guzzle calls |-| `QueryInstrumentation` | Eloquent / DB queries with SQL and timing |-| `RedisInstrumentation` | Redis commands |-| `QueueInstrumentation` | Queued jobs — `PRODUCER` span at dispatch, `CONSUMER` span at execute (see [Tasks](/client/laravel/tasks)) |-| `CacheInstrumentation` | Cache hits / misses / writes (as span events) |-| `ViewInstrumentation` | Blade view rendering |-| `LivewireInstrumentation` | Livewire component renders |-| `ConsoleInstrumentation` | Artisan commands — **opt-in per command**, see [Tasks](/client/laravel/tasks) |-| `ScoutInstrumentation` | Laravel Scout (requires the `opentelemetry` PHP extension) |--Disable any instrumentation you don't want by removing it from the array or setting the matching env var to `false`. Every enabled instrumentation adds a small amount of overhead.--> **Exceptions:** there is no separate "exception" instrumentation. The package's service provider hooks Laravel's exception handler via `reportable()` and calls `$span->recordException($e)` + `setStatus(ERROR)` on the active span for every reported exception. Queue failures are captured by `QueueInstrumentation` on the `JobFailed` event. See [Exceptions](/client/laravel/exceptions) for adding extra context to caught errors.--## What Gets Captured--Once configured, the package automatically captures:--- **Endpoints** — every HTTP request with method and route template (e.g., `GET /users/{id}`)-- **Status codes** — 2xx, 4xx, 5xx responses-- **Exceptions** — recorded on the request span with stack traces-- **Database queries** — Eloquent / DB facade queries with statement and duration-- **Outbound HTTP** — `Http::` facade and Guzzle calls-- **Cache & Redis** — keys, hits, misses-- **Queued jobs** — `PRODUCER` + `CONSUMER` spans (see [Tasks](/client/laravel/tasks))-- **User context** — the authenticated user's id is added as `user.id` to every span and log (configurable via `opentelemetry.user_context`)--## User Context--When `opentelemetry.user_context` is `true` (the default), the authenticated user's id is attached as `user.id` to every span and log. You can customize the attributes:--```php-// app/Providers/AppServiceProvider.php-use Illuminate\Contracts\Auth\Authenticatable;-use Keepsuit\LaravelOpenTelemetry\Facades\OpenTelemetry;--public function boot(): void-{- OpenTelemetry::user(function (Authenticatable $user) {- return [- 'user.id' => $user->getAuthIdentifier(),- 'user.email' => $user->email,- ];- });-}-```--## Octane / Horizon / Queue Workers--The package detects worker mode automatically (Octane, Horizon, default queue workers). By default, span exports are batched and flushed periodically or on process shutdown. If you want every iteration (request, job) to flush immediately, set:--```bash-OTEL_WORKER_MODE_FLUSH_AFTER_EACH_ITERATION=true-```--This trades a small amount of throughput for guaranteed delivery of every span — useful in low-traffic apps where the next flush might be far away.--## Test Your Integration--Add a test route to verify data is flowing:--```php-// routes/web.php-use Illuminate\Support\Facades\Route;--Route::get('/testing', function () {- throw new \RuntimeException('Test error from Traceway integration');-});-```--Visit `/testing` in your browser, then check the Traceway dashboard — the exception should appear within a few seconds.--## Next Steps--- [Exceptions](/client/laravel/exceptions) — manually capture caught exceptions with context-- [Spans](/client/laravel/spans) — create custom spans to measure sub-operations-- [Tasks](/client/laravel/tasks) — trace Laravel queue jobs and console commands as background tasks-- [Metrics](/client/laravel/metrics) — track custom counters, histograms, and gauges with the `Meter` facade-- [Logs](/client/laravel/logs) — forward Laravel logs to Traceway via the auto-injected `otlp` channel-- [OpenTelemetry docs](/client/otel) — how OTel traces, metrics, and logs map to Tracewaydiff --git a/docs/pages/client/laravel/tasks.mdx b/docs/pages/client/laravel/tasks.mdxdeleted file mode 100644index 78e722a1..00000000--- a/docs/pages/client/laravel/tasks.mdx+++ /dev/null@@ -1,159 +0,0 @@-# Tasks--The `QueueInstrumentation` automatically traces Laravel queue jobs. When a job is dispatched, a `PRODUCER` span is created; when the worker picks it up, a `CONSUMER` span runs the handler. W3C trace context is propagated through the job payload so the consumer span links back to the dispatch trace — the full lifecycle is visible as one continuous trace in Traceway, and each consumed job appears as a **task** with duration, status, and any child spans.--## Setup--`QueueInstrumentation` is enabled by default. If it's not already in your `config/opentelemetry.php`:--```php-use Keepsuit\LaravelOpenTelemetry\Instrumentation;--return [- 'instrumentation' => [- // ...- Instrumentation\QueueInstrumentation::class => env('OTEL_INSTRUMENTATION_QUEUE', true),- ],-];-```--That's it — every queued job is now traced automatically. The same setup applies whether you use `database`, `redis`, `sqs`, or `beanstalkd` drivers.--## What Gets Captured--For each queued job the instrumentation creates two spans:--- **Dispatch span** — name `send {queue}` (e.g., `send default`), `PRODUCER` kind, ended when Laravel finishes pushing the payload onto the queue.-- **Process span** — name `process {queue}` (e.g., `process default`), `CONSUMER` kind, ended when the handler returns or fails. On `JobFailed` the span records the exception and is marked `STATUS_ERROR`.--Both spans carry these attributes (using OTel messaging semantic conventions):--| Attribute | Value |-|---|---|-| `messaging.system` | The queue driver — `redis`, `database`, `sqs`, `beanstalkd`, … |-| `messaging.operation.type` | `send` (producer) or `process` (consumer) |-| `messaging.message.id` | Laravel's job UUID |-| `messaging.destination.name` | The queue name |-| `messaging.message.envelope.size` | Payload size in bytes |-| `messaging.message.job_name` | The job class name (e.g., `App\Jobs\SendOrderEmail`) |-| `messaging.message.attempts` | Current attempt number |-| `messaging.message.max_tries` / `max_exceptions` / `retry_until` / `timeout` | Job configuration |--W3C trace context is also injected into the job payload, so the `process` span is linked to the trace where the job was originally dispatched — even if dispatch and execution happen on different services. Any spans you create inside `handle()` automatically nest under the `process` span.--## Adding Attributes in Handlers--Since the instrumentation activates the consumer span before your `handle()` runs, use `Span::getCurrent()` to add custom attributes:--```php-namespace App\Jobs;--use Illuminate\Bus\Queueable;-use Illuminate\Contracts\Queue\ShouldQueue;-use OpenTelemetry\API\Trace\Span;--class SendOrderEmail implements ShouldQueue-{- use Queueable;-- public function __construct(- public readonly int $orderId,- public readonly string $recipient,- ) {}-- public function handle(): void- {- $span = Span::getCurrent();- $span->setAttribute('order.id', $this->orderId);- $span->setAttribute('email.to', $this->recipient);-- // ... business logic ...- }-}-```--## Child Spans in Handlers--For jobs with distinct sub-operations, use the `Tracer` facade to create child spans. They automatically nest under the task's consumer span:--```php-namespace App\Jobs;--use Illuminate\Bus\Queueable;-use Illuminate\Contracts\Queue\ShouldQueue;-use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;-use OpenTelemetry\API\Trace\Span;--class ProcessPayment implements ShouldQueue-{- use Queueable;-- public function __construct(- public readonly int $paymentId,- public readonly float $amount,- ) {}-- public function handle(): void- {- Tracer::newSpan('validate.payment')->measure(function () {- Span::getCurrent()->setAttribute('payment.id', $this->paymentId);- $this->validate();- });-- Tracer::newSpan('charge.gateway')->measure(function () {- Span::getCurrent()->setAttribute('payment.amount', $this->amount);- $this->chargeGateway();- });- }-}-```--## Console Commands--Console commands are **not traced by default** — only commands you opt-in are wrapped in a span. Add them to the `commands` array of `ConsoleInstrumentation`:--```php-// config/opentelemetry.php-use Keepsuit\LaravelOpenTelemetry\Instrumentation;-use App\Console\Commands\CleanupOrphans;--return [- 'instrumentation' => [- // ...- Instrumentation\ConsoleInstrumentation::class => [- 'enabled' => env('OTEL_INSTRUMENTATION_CONSOLE', true),- 'commands' => [- // by class name- CleanupOrphans::class,- // or by signature- 'app:rebuild-search-index',- ],- ],- ],-];-```--Each listed command is opened as a root span at start and ended on exit. Inside `handle()`, use `Span::getCurrent()` to add attributes:--```php-namespace App\Console\Commands;--use Illuminate\Console\Command;-use OpenTelemetry\API\Trace\Span;--class CleanupOrphans extends Command-{- protected $signature = 'cleanup:orphans';-- public function handle(): int- {- Span::getCurrent()->setAttribute('cleanup.dry_run', false);-- // ... do work ...-- return self::SUCCESS;- }-}-```--Unlisted commands run without a span, so `Span::getCurrent()` inside them returns a no-op span — attribute and event calls are silently dropped, which is safe but not useful.diff --git a/docs/pages/client/nestjs/custom-instrumentation.mdx b/docs/pages/client/nestjs/custom-instrumentation.mdxdeleted file mode 100644index 0f5b3335..00000000--- a/docs/pages/client/nestjs/custom-instrumentation.mdx+++ /dev/null@@ -1,164 +0,0 @@-# Custom Instrumentation--Add manual spans, custom decorators, and explicit exception capture to your NestJS services using the OpenTelemetry API.--## Setup--Install the OTel API package (if not already installed via `@opentelemetry/sdk-node`):--```bash-npm install @opentelemetry/api-```--## Manual Spans in Services--Create spans for operations that aren't auto-instrumented:--```typescript-import { Injectable } from "@nestjs/common";-import { trace, SpanStatusCode } from "@opentelemetry/api";--const tracer = trace.getTracer("my-nestjs-app");--@Injectable()-export class UsersService {- async findAll() {- return tracer.startActiveSpan("db.users.findAll", async (span) => {- try {- const users = await this.userRepository.find();- span.setAttribute("user.count", users.length);- return users;- } catch (error) {- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- span.recordException(error);- throw error;- } finally {- span.end();- }- });- }-}-```--## Custom @Span Decorator--Create a reusable decorator that wraps methods in OTel spans:--```typescript-import { trace, SpanStatusCode } from "@opentelemetry/api";--const tracer = trace.getTracer("my-nestjs-app");--export function Span(name?: string): MethodDecorator {- return (target, propertyKey, descriptor: PropertyDescriptor) => {- const originalMethod = descriptor.value;- const spanName = name || `${target.constructor.name}.${String(propertyKey)}`;-- descriptor.value = function (...args: any[]) {- return tracer.startActiveSpan(spanName, async (span) => {- try {- const result = await originalMethod.apply(this, args);- span.setStatus({ code: SpanStatusCode.OK });- return result;- } catch (error) {- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- span.recordException(error);- throw error;- } finally {- span.end();- }- });- };-- return descriptor;- };-}-```--Use it on any service method:--```typescript-import { Injectable } from "@nestjs/common";-import { Span } from "./span.decorator";--@Injectable()-export class OrdersService {- @Span("orders.process")- async processOrder(orderId: string) {- // Span is automatically created and ended- await this.validate(orderId);- await this.charge(orderId);- await this.ship(orderId);- }-- @Span()- async validate(orderId: string) {- // Span name defaults to "OrdersService.validate"- }-}-```--## Exception Capture--Record exceptions on the active span:--```typescript-import { Injectable } from "@nestjs/common";-import { trace, SpanStatusCode } from "@opentelemetry/api";--@Injectable()-export class PaymentService {- async charge(amount: number) {- const span = trace.getActiveSpan();-- try {- await this.gateway.charge(amount);- } catch (error) {- if (span) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- }- throw error;- }- }-}-```--Exceptions recorded as span events appear as **Issues** in Traceway with full stack traces.--## Span Attributes--Add context to spans for richer trace data:--```typescript-const span = trace.getActiveSpan();-if (span) {- span.setAttribute("user.id", userId);- span.setAttribute("order.total", orderTotal);- span.setAttributes({- "payment.method": "credit_card",- "payment.currency": "USD",- });-}-```--## What Gets Auto-Instrumented vs Manual--| Operation | Auto-Instrumented? | Notes |-|-----------|-------------------|-------|-| HTTP requests (incoming) | Yes | Via `instrumentation-http` + `instrumentation-express` |-| `http.route` grouping | Yes | Express instrumentation sets this automatically |-| Thrown exceptions | Yes | Recorded as Issues with stack traces |-| Outgoing `fetch()` calls | Yes | Via `instrumentation-undici` |-| Prisma queries | Yes | Via `@prisma/instrumentation` (install separately) |-| Database queries (`pg`, `mysql2`, `mongodb`) | Yes | CJS packages — patched by `require-in-the-middle` |-| Cache (`redis`, `ioredis`) | Yes | CJS packages — patched automatically |-| NestJS handlers | Yes | Via `instrumentation-nestjs-core` |-| Custom business logic | No | Use `tracer.startActiveSpan()` or `@Span()` decorator |-| SQLite (`better-sqlite3`) | No | Use manual spans |--## Next Steps--- [Quick Start](/client/nestjs) — setup and auto-instrumentation-- [Node.js Traces](/client/node-sdk/traces) — more on manual spans and context propagation-- [Node.js Metrics](/client/node-sdk/metrics) — custom metrics with OTeldiff --git a/docs/pages/client/nestjs/index.mdx b/docs/pages/client/nestjs/index.mdxdeleted file mode 100644index 9f33efed..00000000--- a/docs/pages/client/nestjs/index.mdx+++ /dev/null@@ -1,162 +0,0 @@-# NestJS (OpenTelemetry)--Instrument your NestJS application with OpenTelemetry and export traces and metrics to Traceway. No vendor-specific NestJS modules needed — OTel auto-instrumentation handles everything.--## Installation--```bash-npm install @opentelemetry/sdk-node \- @opentelemetry/auto-instrumentations-node \- @opentelemetry/exporter-trace-otlp-http \- @opentelemetry/exporter-metrics-otlp-http \- @opentelemetry/api-```--If you use Prisma, also install its OTel instrumentation for automatic query tracing:--```bash-npm install @prisma/instrumentation-```--## Setup--Create an `instrumentation.ts` file at the root of your NestJS project:--```typescript-import { NodeSDK } from "@opentelemetry/sdk-node";-import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";-import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";-import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";-import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";-import { Resource } from "@opentelemetry/resources";--const sdk = new NodeSDK({- resource: new Resource({- "service.name": "my-nestjs-app",- "service.version": "1.0.0",- }),-- traceExporter: new OTLPTraceExporter({- url: "https://your-traceway-instance.com/api/otel/v1/traces",- headers: { Authorization: "Bearer your-project-token" },- }),-- metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: { Authorization: "Bearer your-project-token" },- }),- exportIntervalMillis: 30_000,- }),-- instrumentations: [getNodeAutoInstrumentations()],-});--sdk.start();-```--> If you use Prisma, add `import { PrismaInstrumentation } from "@prisma/instrumentation"` and include `new PrismaInstrumentation()` in the `instrumentations` array alongside `getNodeAutoInstrumentations()`.--## Running Your App--The instrumentation file must load **before** NestJS boots. Use the `--require` flag:--```bash-# Development (with ts-node)-node --require ts-node/register --require ./instrumentation.ts src/main.ts--# Production (compiled JS)-node --require ./instrumentation.js dist/main.js-```--Add these as scripts in `package.json`:--```json-{- "scripts": {- "start": "node --require ./instrumentation.js dist/main.js",- "start:dev": "node --require ts-node/register --require ./instrumentation.ts src/main.ts"- }-}-```--If you use the NestJS CLI (`nest start`), set `NODE_OPTIONS` in your environment:--```bash-export NODE_OPTIONS="--require ./instrumentation.js"-npm run start-```--## What Gets Captured--NestJS uses Express (or Fastify) under the hood — both are CJS packages that OTel auto-instrumentation patches successfully. The following is captured automatically:--| Layer | What's Captured |-|-------|-----------------|-| HTTP (Express/Fastify) | Incoming requests with route, method, status code, duration |-| NestJS handlers | Controller method spans via `instrumentation-nestjs-core` |-| `http.route` | Parameterized routes (e.g., `GET /api/users/:id`) — set automatically |-| Exceptions | Thrown errors recorded as Issues with stack traces |-| Prisma (`@prisma/instrumentation`) | Query spans with operation name, model, duration |-| Database (`pg`, `mysql2`, `mongodb`) | Query spans with SQL statements |-| Cache (`redis`, `ioredis`) | Cache operation spans |-| Outgoing `fetch()` | Outgoing HTTP request spans (auto via `instrumentation-undici`) |-| Outgoing HTTP (`axios`) | Outgoing request spans |-| DNS | DNS lookup spans |--Root SERVER spans become **Endpoints** in Traceway, child spans become **Spans**, and exception events become **Issues**.--### Why NestJS doesn't need a route helper--Unlike Hono and Next.js, NestJS uses Express (or Fastify) internally. The `@opentelemetry/instrumentation-express` package automatically sets `http.route` on the root HTTP span. Endpoint grouping works out of the box — `GET /api/users/1` and `GET /api/users/2` are grouped as `GET /api/users/:id` without any extra code.--### Prisma Auto-Instrumentation--With `@prisma/instrumentation` registered in the SDK, every Prisma query automatically creates child spans:--```typescript-// Just use Prisma normally — spans are created automatically-const users = await this.prismaService.user.findMany();-const user = await this.prismaService.user.create({ data: { name, email } });-```--The resulting trace in Traceway:--```-GET /api/users ← Endpoint- └─ prisma:client:query ← Span (findMany on User)-```--## Environment Variable Alternative--Instead of hardcoding the endpoint and token, you can use standard OTel environment variables:--```bash-export OTEL_SERVICE_NAME="my-nestjs-app"-export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"-export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--## Test Your Integration--Add a test endpoint:--```typescript-import { Controller, Get } from "@nestjs/common";--@Controller("api")-export class AppController {- @Get("test-error")- testError() {- throw new Error("Test error from NestJS");- }-}-```--Visit `/api/test-error` and check your Traceway dashboard — the error should appear under **Issues** with a full stack trace.--## Next Steps--- [Custom Instrumentation](/client/nestjs/custom-instrumentation) — manual spans, decorators, and exception capture-- [Node.js OTel Guide](/client/node-sdk) — general Node.js setup details-- [OTel Trace Mapping](/client/otel/traces) — how OTel spans map to Traceway conceptsdiff --git a/docs/pages/client/nextjs/index.mdx b/docs/pages/client/nextjs/index.mdxdeleted file mode 100644index fce752cf..00000000--- a/docs/pages/client/nextjs/index.mdx+++ /dev/null@@ -1,508 +0,0 @@-# Next.js--Traceway instruments Next.js on two sides:--- **Frontend (Browser)** — uncaught errors, render errors, and session recording via [`@tracewayapp/react`](https://www.npmjs.com/package/@tracewayapp/react).-- **Backend (Server)** — request traces, database spans, and server-side exceptions via OpenTelemetry.--The two work side by side on the same project. Most apps will want both.-------## Quick Start--Minimum configuration to get errors flowing in. Drop these in, replace the connection string, and you're done.--**Frontend** — install `@tracewayapp/react`, wrap your app in `TracewayProvider`:--```bash-npm install @tracewayapp/react-```--```tsx-// app/traceway-provider.tsx-"use client";--import { TracewayProvider } from "@tracewayapp/react";--export function TracewayClientProvider({ children }: { children: React.ReactNode }) {- return (- <TracewayProvider connectionString="your-token@https://traceway.example.com/api/report">- {children}- </TracewayProvider>- );-}-```--```tsx-// app/layout.tsx-import { TracewayClientProvider } from "./traceway-provider";--export default function RootLayout({ children }: { children: React.ReactNode }) {- return (- <html lang="en">- <body>- <TracewayClientProvider>{children}</TracewayClientProvider>- </body>- </html>- );-}-```--**Backend** — install the OTel Node SDK and add a single `instrumentation.ts`:--```bash-npm install @opentelemetry/sdk-node \- @opentelemetry/auto-instrumentations-node \- @opentelemetry/exporter-trace-otlp-http \- @opentelemetry/api-```--```typescript-// instrumentation.ts (project root)-export async function register() {- if (process.env.NEXT_RUNTIME === "nodejs") {- const { NodeSDK } = await import("@opentelemetry/sdk-node");- const { OTLPTraceExporter } = await import("@opentelemetry/exporter-trace-otlp-http");- const { getNodeAutoInstrumentations } = await import("@opentelemetry/auto-instrumentations-node");- const { Resource } = await import("@opentelemetry/resources");-- new NodeSDK({- resource: new Resource({ "service.name": "my-nextjs-app" }),- traceExporter: new OTLPTraceExporter({- url: "https://your-traceway-instance.com/api/otel/v1/traces",- headers: { Authorization: "Bearer your-project-token" },- }),- instrumentations: [getNodeAutoInstrumentations()],- }).start();- }-}-```--> Next.js < 15: also set `experimental.instrumentationHook: true` in `next.config.js`. Not needed for 15+.--That's the whole base setup. The rest of this page is for the cases where the defaults aren't enough.-------## Frontend (Browser)--Captures browser errors, render errors, and session recordings. Works with both App Router and Pages Router.--### Pages Router setup--If you're still on `pages/`, mount the provider directly in `_app.tsx`:--```tsx-// pages/_app.tsx-import type { AppProps } from "next/app";-import { TracewayProvider } from "@tracewayapp/react";--export default function App({ Component, pageProps }: AppProps) {- return (- <TracewayProvider connectionString="your-token@https://traceway.example.com/api/report">- <Component {...pageProps} />- </TracewayProvider>- );-}-```--### What you get by default--Defaults match the rest of the JS SDK family — no flags needed:--| Option | Default | Purpose |-|---|---|---|-| `sessionRecording` | `true` | rrweb recorder runs; segments ride along with each exception |-| `recordAllSessions` | `false` | Only exception-bound clips upload (no always-on recording) |-| `captureLogs` | `true` | `console.{log,info,warn,error}` mirrored into the rolling buffer |-| `captureNetwork` | `true` | `fetch` and XHR calls recorded as network actions |-| `captureNavigation` | `true` | History API push/replace/pop recorded as navigation actions |--The `connectionString` uses the standard `{token}@{apiUrl}` format from your Traceway project page.--`TracewayProvider` also acts as a React error boundary — render-time exceptions are captured and re-thrown so your app behaves exactly as without Traceway. No separate `<TracewayErrorBoundary>` is required.--### Capture errors manually--Inside any client component, use the `useTraceway` hook:--```tsx-"use client";--import { useTraceway } from "@tracewayapp/react";--export function CheckoutButton() {- const { captureException } = useTraceway();-- async function handleClick() {- try {- await placeOrder();- } catch (e) {- captureException(e as Error);- }- }-- return <button onClick={handleClick}>Place order</button>;-}-```--### Test the frontend integration--Drop a throw into any client component and reload:--```tsx-"use client";--export function TestErrorButton() {- return (- <button onClick={() => { throw new Error("Test error from Next.js client"); }}>- Send test error- </button>- );-}-```--The error should appear in your Traceway dashboard under **Issues** within a few seconds, with the session recording attached.-------## Backend (Server)--Captures server-side traces, database spans, and exceptions thrown in Server Components, route handlers, and `getServerSideProps` — none of which the frontend SDK can see.--### Setup with Prisma + metrics--The Quick Start above is the minimum. If you use Prisma or want metric export too, the fuller `instrumentation.ts` looks like:--```bash-npm install @prisma/instrumentation @opentelemetry/exporter-metrics-otlp-http-```--```typescript-export async function register() {- if (process.env.NEXT_RUNTIME === "nodejs") {- const { NodeSDK } = await import("@opentelemetry/sdk-node");- const { OTLPTraceExporter } = await import("@opentelemetry/exporter-trace-otlp-http");- const { OTLPMetricExporter } = await import("@opentelemetry/exporter-metrics-otlp-http");- const { PeriodicExportingMetricReader } = await import("@opentelemetry/sdk-metrics");- const { getNodeAutoInstrumentations } = await import("@opentelemetry/auto-instrumentations-node");- const { Resource } = await import("@opentelemetry/resources");- const { PrismaInstrumentation } = await import("@prisma/instrumentation");-- new NodeSDK({- resource: new Resource({- "service.name": "my-nextjs-app",- "service.version": "1.0.0",- }),- traceExporter: new OTLPTraceExporter({- url: "https://your-traceway-instance.com/api/otel/v1/traces",- headers: { Authorization: "Bearer your-project-token" },- }),- metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: { Authorization: "Bearer your-project-token" },- }),- exportIntervalMillis: 30_000,- }),- instrumentations: [- getNodeAutoInstrumentations(),- new PrismaInstrumentation(),- ],- }).start();- }-}-```--The `process.env.NEXT_RUNTIME === "nodejs"` guard is required — Next.js runs `instrumentation.ts` in both the Node.js and Edge runtimes, but the OTel Node SDK only works in Node.js. Dynamic `import()` keeps OTel out of the Edge bundle.--### Enabling the Instrumentation Hook (Next.js 13.4–14.x)--For Next.js versions before 15, enable the instrumentation hook in `next.config.js`:--```javascript-/** @type {import('next').NextConfig} */-const nextConfig = {- experimental: {- instrumentationHook: true,- },-};--module.exports = nextConfig;-```--Not needed for Next.js 15+, where the hook is stable by default.--### Route Helper (Required)--Next.js (App Router) doesn't use Express or Fastify under the hood, so OTel auto-instrumentation can't set `http.route` automatically. Without it, Traceway falls back to the literal URL path — every `/api/users/1`, `/api/users/2` becomes a separate endpoint instead of grouping under `GET /api/users/[id]`.--Create a `withRoute` helper that wraps your API route handlers. It sets `http.route` for endpoint grouping and records exceptions as Issues:--```typescript-// lib/with-route.ts-import { trace, SpanStatusCode } from "@opentelemetry/api";--type RouteHandler = (- req: Request,- context: { params: Promise<Record<string, string>> }-) => Response | Promise<Response>;--export function withRoute(route: string, handler: RouteHandler): RouteHandler {- return async (req, context) => {- const span = trace.getActiveSpan();- if (span) {- span.setAttribute("http.route", route);- }- try {- return await handler(req, context);- } catch (error) {- if (span) {- span.recordException(error as Error);- span.setStatus({- code: SpanStatusCode.ERROR,- message: (error as Error).message,- });- }- throw error;- }- };-}-```--Wrap every API route handler with `withRoute`, passing the route pattern that matches your file path:--```typescript-// app/api/users/route.ts-import { withRoute } from "@/lib/with-route";-import { prisma } from "@/lib/db";--export const GET = withRoute("/api/users", async () => {- const users = await prisma.user.findMany();- return Response.json(users);-});-```--```typescript-// app/api/users/[id]/route.ts-import { withRoute } from "@/lib/with-route";-import { prisma } from "@/lib/db";--export const GET = withRoute("/api/users/[id]", async (req, { params }) => {- const { id } = await params;- const user = await prisma.user.findUnique({ where: { id: parseInt(id) } });- if (!user) {- return Response.json({ error: "User not found" }, { status: 404 });- }- return Response.json(user);-});-```--### What gets captured--With the setup above, OTel captures the following automatically:--| Layer | What's Captured |-|-------|-----------------|-| HTTP server | Incoming requests with method, status code, duration |-| `withRoute` helper | `http.route` for endpoint grouping + exception recording |-| Prisma (`@prisma/instrumentation`) | Query spans with operation name, model, duration |-| Database (`pg`, `mysql2`, `mongodb`) | Query spans (CJS packages — auto-instrumented) |-| Cache (`redis`, `ioredis`) | Cache operation spans (auto-instrumented) |-| Outgoing `fetch()` | Outgoing HTTP request spans (auto via `instrumentation-undici`) |--Root SERVER spans become **Endpoints** in Traceway (e.g., `GET /api/users/[id]`), child spans become **Spans**, and exception events become **Issues**.--#### Prisma auto-instrumentation--With `@prisma/instrumentation` registered, every Prisma query creates child spans — no manual wrapping needed:--```typescript-const users = await prisma.user.findMany();-const user = await prisma.user.create({ data: { name, email } });-```--Resulting trace:--```-GET /api/users ← Endpoint- └─ prisma:client:query ← Span (findMany on User)-```--#### Outgoing fetch() auto-instrumentation--Outgoing `fetch()` calls are traced by `@opentelemetry/instrumentation-undici`:--```typescript-export const GET = withRoute("/api/external", async () => {- const res = await fetch("https://api.example.com/data");- const data = await res.json();- return Response.json(data);-});-```--Resulting trace:--```-GET /api/external ← Endpoint- └─ GET ← Span (https://api.example.com/data)-```--### Manual spans--For operations that aren't auto-instrumented (SQLite via `better-sqlite3`, Server Component rendering, custom business logic), open a span by hand with `@opentelemetry/api`:--```typescript-import { trace, SpanStatusCode } from "@opentelemetry/api";-import { withRoute } from "@/lib/with-route";--const tracer = trace.getTracer("my-nextjs-app");--export const GET = withRoute("/api/orders/[id]", async (req, { params }) => {- const { id } = await params;-- return tracer.startActiveSpan("fetch-order", async (span) => {- try {- span.setAttribute("order.id", id);- const order = await db.orders.findById(id);-- if (!order) {- span.setStatus({ code: SpanStatusCode.ERROR, message: "Order not found" });- span.end();- return Response.json({ error: "Not found" }, { status: 404 });- }-- span.end();- return Response.json(order);- } catch (error) {- span.recordException(error as Error);- span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });- span.end();- throw error;- }- });-});-```--#### Server Components--Wrap Server Component bodies in a span to see their render time in the trace:--```typescript-import { trace } from "@opentelemetry/api";--const tracer = trace.getTracer("my-nextjs-app");--export default async function UsersPage() {- return tracer.startActiveSpan("UsersPage.render", async (span) => {- try {- const users = await fetchUsers();- span.setAttribute("user.count", users.length);- return (- <ul>- {users.map((user) => (<li key={user.id}>{user.name}</li>))}- </ul>- );- } finally {- span.end();- }- });-}-```--Server Component spans appear as **Spans** in Traceway, nested under the parent HTTP request trace.--#### Nested spans--Spans created inside an active span are automatically linked as children:--```typescript-async function processPayment(orderId: string) {- return tracer.startActiveSpan("process-payment", async (span) => {- try {- await validateCard(orderId);- await chargeAmount(orderId);- span.setStatus({ code: SpanStatusCode.OK });- } catch (error) {- span.recordException(error as Error);- span.setStatus({ code: SpanStatusCode.ERROR });- throw error;- } finally {- span.end();- }- });-}-```--#### Recording an exception without re-throwing--`withRoute` already records errors that bubble up. If you catch an error and want to log it on the active span without throwing:--```typescript-import { trace, SpanStatusCode } from "@opentelemetry/api";--export const POST = withRoute("/api/checkout", async () => {- const span = trace.getActiveSpan();- try {- await processPayment();- } catch (error) {- if (span) {- span.recordException(error as Error);- span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });- }- return Response.json({ error: "Payment failed" }, { status: 500 });- }- return Response.json({ status: "ok" });-});-```--### Auto vs manual reference--| Operation | Auto-instrumented? | Notes |-|-----------|-------------------|-------|-| HTTP requests (incoming) | Partial | `withRoute` sets `http.route` + catches exceptions |-| Outgoing `fetch()` calls | Yes | Via `instrumentation-undici` |-| Prisma queries | Yes | Via `@prisma/instrumentation` |-| Database queries (`pg`, `mysql2`, `mongodb`) | Yes | CJS packages — patched by `require-in-the-middle` |-| Cache (`redis`, `ioredis`) | Yes | CJS packages — patched automatically |-| SQLite (`better-sqlite3`) | No | Use manual spans |-| Server Component rendering | No | Use `tracer.startActiveSpan()` |-| Custom business logic | No | Use `tracer.startActiveSpan()` |--### Environment variable alternative--Instead of hardcoding the endpoint and token, use standard OTel environment variables:--```bash-export OTEL_SERVICE_NAME="my-nextjs-app"-export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"-export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--With these set, your `instrumentation.ts` can omit the exporter URLs and headers — the SDK picks them up automatically.--### Test the backend integration--Add a test API route:--```typescript-// app/api/test-error/route.ts-import { withRoute } from "@/lib/with-route";--export const GET = withRoute("/api/test-error", async () => {- throw new Error("Test error from Next.js server");-});-```--Visit `/api/test-error` and check your Traceway dashboard — the error should appear under **Issues** with a full stack trace, and the endpoint should be grouped as `GET /api/test-error`.-------## Next Steps--- [React SDK Reference](/client/react) — provider options, hooks, advanced usage-- [Sessions](/learn/sessions) — how session recording is stored and replayed-- [Node.js OTel Guide](/client/node-sdk) — general Node.js setup details-- [OTel Overview](/client/otel) — protocol details and authentication-- [OTel Trace Mapping](/client/otel/traces) — how OTel spans map to Traceway conceptsdiff --git a/docs/pages/client/node-sdk/index.mdx b/docs/pages/client/node-sdk/index.mdxdeleted file mode 100644index 47e68e51..00000000--- a/docs/pages/client/node-sdk/index.mdx+++ /dev/null@@ -1,164 +0,0 @@-# Node.js (OpenTelemetry)--Instrument any Node.js backend — Express, Fastify, Koa, Hono, NestJS, or plain HTTP — with OpenTelemetry and export traces and metrics to Traceway.--## Installation--```bash-npm install @opentelemetry/sdk-node \- @opentelemetry/auto-instrumentations-node \- @opentelemetry/exporter-trace-otlp-http \- @opentelemetry/exporter-metrics-otlp-http-```--## Setup--Create an `instrumentation.ts` (or `.js`) file at the root of your project. This file must be loaded **before** your application code.--```typescript-import { NodeSDK } from "@opentelemetry/sdk-node";-import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";-import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";-import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";-import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";-import { Resource } from "@opentelemetry/resources";--const sdk = new NodeSDK({- resource: new Resource({- "service.name": "my-service",- "service.version": "1.0.0",- }),-- traceExporter: new OTLPTraceExporter({- url: "https://your-traceway-instance.com/api/otel/v1/traces",- headers: { Authorization: "Bearer your-project-token" },- }),-- metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: { Authorization: "Bearer your-project-token" },- }),- exportIntervalMillis: 30_000,- }),-- instrumentations: [getNodeAutoInstrumentations()],-});--sdk.start();-```--## Running Your App--Load the instrumentation file before everything else using Node's `--require` (CommonJS) or `--import` (ESM) flag:--```bash-# CommonJS-node --require ./instrumentation.js server.js--# ESM / TypeScript (with tsx or ts-node)-node --import ./instrumentation.ts server.ts-```--Or set it via environment variable:--```bash-export NODE_OPTIONS="--require ./instrumentation.js"-node server.js-```--## Environment Variable Alternative--Instead of hardcoding the endpoint and token, you can use standard OTel environment variables:--```bash-export OTEL_SERVICE_NAME="my-service"-export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"-export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--With these set, the SDK picks them up automatically — your `instrumentation.ts` can omit the exporter URLs and headers.--## What Gets Auto-Instrumented--The `@opentelemetry/auto-instrumentations-node` package automatically instruments:--| Library | What's Captured |-|---------|-----------------|-| `http` / `https` | Incoming and outgoing HTTP requests |-| `express` | Route-level spans with `http.route` |-| `fastify` | Route-level spans |-| `koa` | Middleware and route spans |-| `hono` | HTTP request spans (via `http` instrumentation) |-| `pg` / `mysql2` / `mongodb` | Database query spans |-| `redis` / `ioredis` | Cache operation spans |-| `grpc` | gRPC call spans |-| `fs` | File system operation spans |--Root SERVER spans become **Endpoints** in Traceway, child spans become **Spans**, and exception events become **Issues**. See [Traces](/client/otel/traces) for the full mapping.--## Framework Quick Start--### Express--```typescript-import express from "express";--const app = express();--app.get("/api/users", (req, res) => {- res.json({ users: [] });-});--app.listen(3000);-```--No extra code needed — auto-instrumentation captures all requests, routes, and database calls.--### Fastify--```typescript-import Fastify from "fastify";--const app = Fastify();--app.get("/api/users", async () => {- return { users: [] };-});--app.listen({ port: 3000 });-```--### Hono--See the dedicated [Hono guide](/client/hono) for multi-runtime setup.--```typescript-import { serve } from "@hono/node-server";-import { Hono } from "hono";--const app = new Hono();--app.get("/api/users", (c) => c.json({ users: [] }));--serve({ fetch: app.fetch, port: 3000 });-```--## Test Your Integration--Add a route that throws an error:--```typescript-app.get("/test-error", () => {- throw new Error("Test error from Traceway integration");-});-```--Visit `/test-error` and check your Traceway dashboard — the error should appear under **Issues** with a full stack trace.--## Next Steps--- [Traces](/client/node-sdk/traces) — manual spans, exception recording, context propagation-- [Metrics](/client/node-sdk/metrics) — custom counters, histograms, and gauges-- [OTel Overview](/client/otel) — protocol details and authentication-- [OTel Trace Mapping](/client/otel/traces) — how OTel spans map to Traceway conceptsdiff --git a/docs/pages/client/node-sdk/metrics.mdx b/docs/pages/client/node-sdk/metrics.mdxdeleted file mode 100644index d34d1c87..00000000--- a/docs/pages/client/node-sdk/metrics.mdx+++ /dev/null@@ -1,104 +0,0 @@-# Metrics--Export custom application metrics from your Node.js app to Traceway using the OpenTelemetry Metrics API.--## Setup--Metrics export is configured in your `instrumentation.ts` via the `PeriodicExportingMetricReader`. If you followed the [Quick Start](/client/node-sdk), metrics are already enabled.--```typescript-import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";-import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";--// In your NodeSDK config:-metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: { Authorization: "Bearer your-project-token" },- }),- exportIntervalMillis: 30_000,-}),-```--## Creating Custom Metrics--Use `@opentelemetry/api` to create meters and instruments:--```typescript-import { metrics } from "@opentelemetry/api";--const meter = metrics.getMeter("my-service");-```--## Metric Types--### Counter--Monotonically increasing value — use for counts of events:--```typescript-const requestCounter = meter.createCounter("app.requests.total", {- description: "Total number of requests",-});--// Increment-requestCounter.add(1, { endpoint: "/api/users", method: "GET" });-```--### UpDownCounter--Value that can increase or decrease — use for gauges like active connections:--```typescript-const activeConnections = meter.createUpDownCounter("app.connections.active", {- description: "Number of active connections",-});--activeConnections.add(1); // connection opened-activeConnections.add(-1); // connection closed-```--### Histogram--Distribution of values — use for durations and sizes:--```typescript-const requestDuration = meter.createHistogram("app.request.duration", {- description: "Request duration in milliseconds",- unit: "ms",-});--const start = performance.now();-await handleRequest();-requestDuration.record(performance.now() - start, {- endpoint: "/api/users",-});-```--### Observable Gauge--Asynchronously observed value — use for system metrics:--```typescript-meter.createObservableGauge("app.memory.heap_used", {- description: "Heap memory usage in bytes",- unit: "bytes",-}).addCallback((result) => {- result.observe(process.memoryUsage().heapUsed);-});-```--## How Metrics Appear in Traceway--| OTel Type | Traceway Handling |-|-----------|-------------------|-| Gauge | Stored as-is |-| Sum (Counter) | Stored as-is |-| Histogram | Converted to `{name}.avg` and `{name}.count` |--See [OTel Metrics](/client/otel/metrics) for full details on histogram handling and supported types.--## Next Steps--- [Traces](/client/node-sdk/traces) — manual spans and exception recording-- [Quick Start](/client/node-sdk) — setup and installationdiff --git a/docs/pages/client/node-sdk/traces.mdx b/docs/pages/client/node-sdk/traces.mdxdeleted file mode 100644index 7f85a8c8..00000000--- a/docs/pages/client/node-sdk/traces.mdx+++ /dev/null@@ -1,120 +0,0 @@-# Traces--OpenTelemetry auto-instrumentation handles most tracing automatically. This page covers manual span creation and exception recording for cases where you need more control.--## How Spans Map to Traceway--| OTel Span | Traceway Concept |-|-----------|-----------------|-| Root SERVER span with HTTP attributes | **Endpoint** (e.g., `GET /api/users`) |-| Root span (other kinds) | **Task** |-| Child span | **Span** |-| Exception event on any span | **Issue** |--See [OTel Trace Mapping](/client/otel/traces) for full details on attributes and conventions.--## Manual Span Creation--Use `@opentelemetry/api` to create custom spans for operations that aren't auto-instrumented:--```typescript-import { trace, SpanStatusCode } from "@opentelemetry/api";--const tracer = trace.getTracer("my-service");--async function processOrder(orderId: string) {- return tracer.startActiveSpan("process-order", async (span) => {- try {- span.setAttribute("order.id", orderId);-- await validateOrder(orderId);- await chargePayment(orderId);- await sendConfirmation(orderId);-- span.setStatus({ code: SpanStatusCode.OK });- } catch (error) {- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- span.recordException(error);- throw error;- } finally {- span.end();- }- });-}-```--## Nested Spans--Spans created inside an active span are automatically linked as children:--```typescript-async function chargePayment(orderId: string) {- return tracer.startActiveSpan("charge-payment", async (span) => {- try {- const result = await paymentGateway.charge(orderId);- span.setAttribute("payment.status", result.status);- return result;- } finally {- span.end();- }- });-}-```--## Recording Exceptions--Exceptions recorded as span events appear as **Issues** in Traceway:--```typescript-import { trace, SpanStatusCode } from "@opentelemetry/api";--function riskyOperation() {- const span = trace.getActiveSpan();-- try {- doSomethingDangerous();- } catch (error) {- if (span) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });- }- throw error;- }-}-```--The `recordException` call adds an event with `exception.type`, `exception.message`, and `exception.stacktrace` attributes. Traceway normalizes and hashes the stack trace to group identical errors into a single Issue.--## Span Attributes--Add attributes to enrich your spans with context:--```typescript-span.setAttribute("user.id", userId);-span.setAttribute("db.statement", "SELECT * FROM users");-span.setAttributes({- "order.id": orderId,- "order.total": 99.99,- "order.items": 3,-});-```--## Context Propagation--OTel automatically propagates trace context across async operations. For manual propagation across service boundaries, use the `W3C Trace Context` headers:--```typescript-import { propagation, context } from "@opentelemetry/api";--// Inject trace context into outgoing request headers-const headers = {};-propagation.inject(context.active(), headers);--// The headers now contain `traceparent` and optionally `tracestate`-await fetch("https://other-service.com/api", { headers });-```--## Next Steps--- [Metrics](/client/node-sdk/metrics) — custom counters, histograms, and gauges-- [Quick Start](/client/node-sdk) — setup and installationdiff --git a/docs/pages/client/openrouter/index.mdx b/docs/pages/client/openrouter/index.mdxindex 76610490..8137b66e 100644--- a/docs/pages/client/openrouter/index.mdx+++ b/docs/pages/client/openrouter/index.mdx@@ -5,22 +5,22 @@ Send AI traces from [OpenRouter](https://openrouter.ai) to Traceway with zero co## How It Works1. **OpenRouter routes your AI calls** to the best model provider (OpenAI, Anthropic, Google, etc.)-2. **Observability broadcasts traces** — OpenRouter sends an OTLP span for each completion with `gen_ai.*` attributes-3. **Traceway captures and stores** — Model, tokens, costs, duration, and conversation content appear in your AI Traces dashboard+2. **Observability broadcasts traces**: OpenRouter sends an OTLP span for each completion with `gen_ai.*` attributes+3. **Traceway captures and stores**: Model, tokens, costs, duration, and conversation content appear in your AI Traces dashboard-No SDK or code changes needed — OpenRouter handles the instrumentation automatically.+No SDK or code changes are needed. OpenRouter handles the instrumentation automatically.## Setup### 1. Get Your Traceway Project Token-AI Tracing works with any backend or OpenTelemetry project in Traceway. If you don't have a project yet, create a new one and select **OpenTelemetry** as the framework — this gives you a project token you can use for the OpenRouter integration.+AI Tracing works with any backend or OpenTelemetry project in Traceway. If you don't have a project yet, create a new one and select **OpenTelemetry** as the framework. This gives you a project token you can use for the OpenRouter integration.In your Traceway dashboard, go to **Connection** and copy your project token.### 2. Configure OpenRouter Observability-1. Go to [OpenRouter Settings → Observability](https://openrouter.ai/settings/integrations)+1. Go to [OpenRouter Settings → Observability](https://openrouter.ai/settings/observability)2. Enable **Broadcast**3. Click **Add Destination** next to **OpenTelemetry Collector**4. Fill in the connection details:@@ -33,13 +33,12 @@ In your Traceway dashboard, go to **Connection** and copy your project token.Replace `YOUR_PROJECT_TOKEN` with your actual project token. If you're self-hosting, replace the endpoint with your instance URL.-5. Click **Test Connection** to verify-6. Click **Send Trace** to send a test trace-7. Click **Add** to save+5. Click **Test Connection** to verify. OpenRouter only saves the destination if the test passes.+6. Click **Add** to save### 3. Verify in Traceway-Open your Traceway dashboard and navigate to **AI Traces**. You should see your test trace appear within a few seconds.+Make an API request through OpenRouter, then open your Traceway dashboard and navigate to **AI Traces**. The call appears within a few seconds.## Configuration Options@@ -93,10 +92,12 @@ Every OpenRouter completion generates a trace with:| Prompt | `gen_ai.prompt` | Full messages JSON || Completion | `gen_ai.completion` | Full response JSON || User ID | `user.id` | The `user` field from your request; falls back to your OpenRouter account id |-| Conversation | `gen_ai.conversation.id` / `session.id` | The `session_id` field from your request |-| Trace Name | `trace.name` | Workflow identifier |+| Conversation | `session.id` | The `session_id` field from your request |+| Trace Name | Span name | Set it with `trace_name` inside the request's `trace` object || Temperature | `gen_ai.request.temperature` | `0.7` |+Any other key you pass in the request's `trace` object arrives as a `trace.metadata.*` span attribute and shows up in the metadata list on the call detail page.+## Self-Hosted SetupIf you're running Traceway on your own infrastructure, use your instance URL as the endpoint:@@ -105,9 +106,9 @@ If you're running Traceway on your own infrastructure, use your instance URL ashttps://your-traceway-instance.com/api/otel/v1/traces```-The rest of the configuration is identical — project token in the Authorization header, same format.+The rest of the configuration is identical: project token in the Authorization header, same format.## Next Steps-- [AI Tracing concepts](/learn/ai-tracing) — Understand how AI traces are stored and queried-- [OpenTelemetry](/client/otel) — Send traces from any OTel-instrumented application+- [AI Tracing concepts](/learn/ai-tracing): Understand how AI traces are stored and queried+- [OpenTelemetry](/client/otel): Send traces from any OTel-instrumented applicationdiff --git a/docs/pages/client/otel/_meta.json b/docs/pages/client/otel/_meta.jsonindex 710d171c..f2e08f2d 100644--- a/docs/pages/client/otel/_meta.json+++ b/docs/pages/client/otel/_meta.json@@ -2,5 +2,14 @@"index": "Overview","traces": "Traces","metrics": "Metrics",- "logs": "Logs"+ "logs": "Logs",+ "nodejs": "Node.js",+ "nestjs": "NestJS",+ "nextjs": "Next.js",+ "hono": "Hono",+ "cloudflare": "Cloudflare Workers",+ "symfony": "Symfony",+ "laravel": "Laravel",+ "python": "Python",+ "django": "Django"}diff --git a/docs/pages/client/cloudflare/_meta.json b/docs/pages/client/otel/cloudflare/_meta.jsonsimilarity index 100%rename from docs/pages/client/cloudflare/_meta.jsonrename to docs/pages/client/otel/cloudflare/_meta.jsondiff --git a/docs/pages/client/otel/cloudflare/index.mdx b/docs/pages/client/otel/cloudflare/index.mdxnew file mode 100644index 00000000..6bc633e0--- /dev/null+++ b/docs/pages/client/otel/cloudflare/index.mdx@@ -0,0 +1,215 @@+# Cloudflare Workers++import { Callout } from "nextra/components";++<Callout type="warning">+ **Workers Paid plan only, and still in beta.** Cloudflare lists OTLP export as "Not available" on the Workers Free plan. On the Free plan you can follow every step on this page, redeploy, and receive nothing. Export is free on Workers Paid for now, during the beta. Cloudflare starts billing tracing on October 1, 2026, with 10 million events per month included for traces and another 10 million for logs, then $0.05 per additional million. The **Destinations** tab in the Cloudflare dashboard still carries a *Beta* badge. Check [Exporting OpenTelemetry Data](https://developers.cloudflare.com/workers/observability/exporting-opentelemetry-data/) for the current terms before you rely on this in production.+</Callout>++Cloudflare Workers has built-in observability that exports **traces and logs** over OTLP. No SDK and no code changes are needed in your Worker. You create a destination in the Cloudflare dashboard, point it at Traceway, and enable it in your `wrangler.jsonc`.++Create the Traceway project with framework **OpenTelemetry**, then copy its project token from **Project Settings**. You need that token twice below.++**Metrics are not part of this path.** Cloudflare states that exporting Worker infrastructure metrics and custom metrics over OpenTelemetry is not available yet, and the destination form only offers Traces and Logs. Metric dashboards stay empty for a Workers-only project. To get metrics into the same project, run the [Traceway OTel Agent](/learn/otel-agent) on a host you control, or send them from a backend service you instrument yourself.++## Traces++### Step 1: Create a traces destination in the Cloudflare dashboard++Open [Workers Observability](https://dash.cloudflare.com/?to=/:account/workers-and-pages/observability/pipelines) in the Cloudflare dashboard (sidebar: **Compute → Observability**), select the **Destinations** tab, then click **+ Add Destination**.++++Fill in the following fields:++- **Destination Type**: Traces+- **OTLP Traces Endpoint**: `https://cloud.tracewayapp.com/api/otel/v1/traces` (or your self-hosted equivalent)+- **Destination Name**: `traceway-traces`+- **Custom Headers**: `Authorization` = `Bearer <project_token>`++Replace `<project_token>` with your project token from the Traceway dashboard under **Project Settings**. The header value is the word `Bearer`, a space, then the token.++++The destination name is yours to choose (the screenshot above uses `traceway`). Whatever you pick has to match the `wrangler.jsonc` value in the next step exactly.++### Step 2: Enable traces in wrangler.jsonc++Add the `observability.traces` configuration to your `wrangler.jsonc`, referencing the destination name you created above:++```jsonc+{+ "observability": {+ "traces": {+ "enabled": true,+ "destinations": ["traceway-traces"],+ "head_sampling_rate": 1,+ "persist": false+ }+ }+}+```++- `head_sampling_rate` is a number between `0` and `1` and controls what fraction of requests are traced. `1` traces every request. Tracing is billed per exported event once the beta pricing ends, and Cloudflare does not say whether a request counts as one event or one per span, so lower it on a high-traffic Worker.+- `persist` defaults to `true`, which keeps a copy in the Cloudflare dashboard on top of exporting it. Set it to `false` when Traceway is your only destination. Keep it `true` if you still want to query the data in Cloudflare.++### Endpoint names: Workers traces carry no route template++Cloudflare's automatic tracing puts `url.path`, `url.full` and `http.request.method` on the invocation span. It never sets `http.route`, because the platform does not know your router's patterns. Traceway uses `http.route` when it is there and falls back to `url.path`, so you get one endpoint row per distinct URL:++```+GET /orders/8f21c0+GET /orders/a934bb+GET /orders/c710de+```++instead of a single `GET /orders/:id`. For a Worker serving a handful of fixed paths that is fine. For a Worker with dynamic path segments, a Hono or itty-router app for example, the Endpoints page fills with one-hit rows, per-endpoint P50 and P95 stop meaning anything, and endpoint-scoped notification rules never match.++There is no way to set `http.route` from inside a Worker today. The [custom span API](https://developers.cloudflare.com/workers/observability/traces/custom-spans/) can only annotate spans you create yourself, not the platform's root invocation span. What you can do is name a child span after the route pattern, so the pattern is at least visible and searchable on the trace:++```ts+import { tracing } from "cloudflare:workers";++export default {+ async fetch(request: Request): Promise<Response> {+ return tracing.enterSpan("GET /orders/:id", async () => {+ const id = new URL(request.url).pathname.split("/").pop();+ return Response.json({ id });+ });+ },+};+```++That span shows up under the trace in Traceway, but the endpoint row still carries the raw path. If per-route aggregates matter more to you than a zero-code setup, instrument the service behind the Worker with an OTel SDK that does set `http.route` (see the [Node.js guide](/client/otel/nodejs)) and send it to the same project.++## Logs++Logs work like traces. Create a second destination in the dashboard, add a second block to `wrangler.jsonc`, and redeploy. No Worker code changes.++### Step 1: Create a logs destination in the Cloudflare dashboard++Back on the **Destinations** tab, click **+ Add Destination** a second time and fill in:++- **Destination Type**: Logs+- **OTLP Logs Endpoint**: `https://cloud.tracewayapp.com/api/otel/v1/logs` (or your self-hosted equivalent)+- **Destination Name**: `traceway-logs`+- **Custom Headers**: `Authorization` = `Bearer <project_token>`++Use the same project token as the traces destination.++### Step 2: Enable logs in wrangler.jsonc++Add the `observability.logs` block alongside `observability.traces`:++```jsonc+{+ "observability": {+ "traces": {+ "enabled": true,+ "destinations": ["traceway-traces"],+ "head_sampling_rate": 1,+ "persist": false+ },+ "logs": {+ "enabled": true,+ "destinations": ["traceway-logs"],+ "head_sampling_rate": 1,+ "persist": false+ }+ }+}+```++`head_sampling_rate` and `persist` mean the same thing in both blocks. `persist` is easy to miss on the traces block: leave it out there and you keep paying for Cloudflare-side trace storage you did not ask for.++When both traces and logs are enabled, each log carries the `trace_id` and `span_id` of the request that produced it, so Traceway shows those logs on the matching trace detail page automatically. Anything you `console.log` in your Worker lands in **Logs**, with the Worker's name as the service.++See [OTel Logs](/client/otel/logs) for the full OTLP logs mapping and severity reference.++## Errors: an uncaught exception does not become an Issue on its own++When your Worker throws, Cloudflare returns an error response and records the failure. It does not attach an OTel exception to the span. The invocation span arrives with `cloudflare.outcome` set to `exception`, so you see the failed request on the Endpoints page, and the error text arrives in **Logs**. Nothing reaches **Issues**, so there is no grouped issue and no stack trace.++To get an Issue with a stack trace, catch the error yourself and put the three OTel exception attributes on a span. Traceway turns any span carrying `exception.type`, `exception.message` or `exception.stacktrace` into an issue, grouped by the normalized stack trace:++```ts+import { tracing } from "cloudflare:workers";++async function handle(request: Request): Promise<Response> {+ if (new URL(request.url).pathname === "/boom") {+ throw new Error("payment gateway timeout");+ }+ return new Response("ok");+}++export default {+ async fetch(request: Request): Promise<Response> {+ try {+ return await handle(request);+ } catch (err) {+ const e = err instanceof Error ? err : new Error(String(err));+ tracing.enterSpan("uncaught error", (span) => {+ span.setAttribute("exception.type", e.name);+ span.setAttribute("exception.message", e.message);+ span.setAttribute("exception.stacktrace", e.stack ?? "");+ });+ return new Response("Internal Server Error", { status: 500 });+ }+ },+};+```++Deploy that and request `/boom`. You get an HTTP 500, an endpoint row with status 500, and an Issue titled `Error: payment gateway timeout` with the frames from `e.stack`.++The frames point into your bundled Worker, not your sources. Cloudflare sets `telemetry.sdk.language` to `javascript`, so Traceway runs these traces through JavaScript symbolication once you upload your Worker's source maps. See [JavaScript symbolication](/symbolicator/javascript).++## Cron, queue, and email handlers++Cloudflare traces scheduled, queue and email invocations too, but those root spans carry no HTTP attributes. Traceway promotes a root span to an **Endpoint** when it has HTTP attributes and to a **Task** when its span kind is `CONSUMER`. A root span that is neither is dropped. Its child spans are still stored, but with no Endpoint or Task to hang under they have no page to appear on, so the invocation is invisible either way. Cloudflare does not document the span kind it uses for these handlers, so treat background invocations as "may not show up" rather than "will show up as a Task".++Exceptions are the exception: a span carrying the `exception.*` attributes still produces an Issue even when its root span was dropped. So the error handler above is worth wrapping around your `scheduled` and `queue` handlers as well.++If you need reliable task rows for background work, do the work in a service you instrument with an OTel SDK, where you can set the span kind to `CONSUMER` yourself.++## What Traceway shows++Cloudflare stamps its own attributes on the spans it exports and Traceway maps them for you. Nothing to configure:++| Cloudflare attribute | Traceway field |+|---|---|+| `service.name` (your Worker's name) | **Server Name** on endpoints, tasks and issues |+| `cloudflare.script_version.id` | **App Version**, using the part after the last `-`, so you can filter issues to a single deployment |+| `telemetry.sdk.language` = `javascript` | turns on JavaScript source-map symbolication for stack traces |++Cloudflare's [known limitations](https://developers.cloudflare.com/workers/observability/traces/known-limitations/) note that `service.name` is not on every span yet, so a span here and there can arrive with no server name. Cloudflare also warns that span and attribute names are not final during the beta, so re-check this table when the beta ends.++Cloudflare also puts `cloudflare.colo`, `cloudflare.ray_id`, `cloudflare.outcome`, `cloudflare.cpu_time_ms`, `cloudflare.wall_time_ms` and `faas.trigger` on the invocation span. Those arrive as attributes on the endpoint and are visible on the trace detail page. `cloudflare.ray_id` is the useful one: it lets you go from a Cloudflare support ticket straight to the matching Traceway trace.++## Deploy++Observability config only takes effect on a new deployment, so deploy your Worker:++```bash+npx wrangler deploy+```++## Verify++Send a few requests to your Worker, wait about a minute, then open Traceway:++1. **Endpoints**: one row per URL path your Worker served (see the endpoint-naming note above).+2. **Logs**: filter by your Worker's name in the Service field. Anything you `console.log` shows up here.+3. Open an endpoint, then a single trace. The log lines from that request are attached to it, because Cloudflare stamps the trace and span ids onto the exported log records.++Nothing showing up? Work through these in order:++- **Are you on a Workers Paid plan?** OTLP export is unavailable on Workers Free and it fails quietly.+- **Did you redeploy?** The observability config only applies to a new deployment.+- **Do the names match?** The strings in `destinations` must be identical to the destination names in the dashboard.+- **Is the header right?** The custom header value is `Bearer <project_token>`: the word `Bearer`, a space, then the token. A wrong token is rejected at the Traceway end. Cloudflare does show it: on the **Destinations** tab the destination's status reads **Error** instead of **Last: n minutes ago**.++## Next Steps++- [OpenTelemetry Overview](/client/otel): endpoint, authentication, and project setup+- [Traces](/client/otel/traces): how OTel spans map to Traceway endpoints, tasks, and issues+- [Logs](/client/otel/logs): OTLP log ingestion and trace linkagediff --git a/docs/pages/client/django/_meta.json b/docs/pages/client/otel/django/_meta.jsonsimilarity index 100%rename from docs/pages/client/django/_meta.jsonrename to docs/pages/client/otel/django/_meta.jsondiff --git a/docs/pages/client/django/exceptions.mdx b/docs/pages/client/otel/django/exceptions.mdxsimilarity index 68%rename from docs/pages/client/django/exceptions.mdxrename to docs/pages/client/otel/django/exceptions.mdxindex 48c3fad2..146a7278 100644--- a/docs/pages/client/django/exceptions.mdx+++ b/docs/pages/client/otel/django/exceptions.mdx@@ -2,8 +2,8 @@`opentelemetry-instrumentation-django` records unhandled exceptions automatically. When a view raises and Django doesn't catch it before the response is sent, the middleware calls `span.record_exception(exc)` and `span.set_status(StatusCode.ERROR)` on the request span. So:-- Any exception thrown from a **view function / class-based view** is captured on the request span (which becomes a Traceway Issue).-- Any exception thrown from a **Celery task** is captured on the consumer span by `opentelemetry-instrumentation-celery` (see [Tasks](/client/django/tasks)).+- Any exception thrown from a **view function / class-based view** is captured on the request span. Django returns a 500, the endpoint row in Traceway shows status `500`, and the error appears under **Issues** with the full Python traceback.+- Any exception thrown from a **Celery task** is captured on the consumer span by `opentelemetry-instrumentation-celery` (see [Tasks](/client/otel/django/tasks)).- Any exception thrown inside a span you created with `tracer.start_as_current_span(...)` is captured by the context manager's `__exit__` when `set_status_on_exception=True` (the default).- Any exception raised inside a `requests` / `httpx` / DB driver call is captured by that library's instrumentation on its own client span.@@ -61,37 +61,40 @@ def process_order(request, order_id):The `attributes={...}` payload passed to `record_exception` is attached to the exception event itself, not the span.-## Adding Context to the Auto-Capture (Response Hook)+## Adding Context to the Auto-Capture (Middleware)-The Django instrumentation lets you register a `response_hook` that runs **after** middleware has populated `request.user` / `request.session`. Use it to attach request-level diagnostic fields that should appear on every span — including spans that ended with an exception event:+To attach request-level diagnostic fields to every span, including spans that ended with an exception event, set them from your own middleware. Your middleware runs inside the OTel span, so `trace.get_current_span()` is the request span:```python-# myproject/otel.py-from opentelemetry.instrumentation.django import DjangoInstrumentor+# myproject/traceway_context.py+from opentelemetry import trace-def response_hook(span, request, response):- if not span or not span.is_recording():- return+class TracewayContextMiddleware:+ def __init__(self, get_response):+ self.get_response = get_response- if hasattr(request, "user") and request.user.is_authenticated:- span.set_attribute("enduser.id", request.user.pk)+ def __call__(self, request):+ response = self.get_response(request)+ span = trace.get_current_span()- rid = request.headers.get("X-Request-Id")- if rid:- span.set_attribute("request.id", rid)+ if hasattr(request, "user") and request.user.is_authenticated:+ span.set_attribute("enduser.id", request.user.pk)+ rid = request.headers.get("X-Request-Id")+ if rid:+ span.set_attribute("request.id", rid)-DjangoInstrumentor().instrument(response_hook=response_hook)+ return response```-This module must be imported before Django (top of `manage.py` / `wsgi.py` / `asgi.py`) — see the [Quick Start](/client/django#manual-initialization).+Add `"myproject.traceway_context.TracewayContextMiddleware"` to `MIDDLEWARE` in `settings.py`, the same way you added the route middleware in the [Quick Start](/client/otel/django#step-3-add-the-route-middleware-required). You can put both jobs in one middleware if you prefer.-> **Why not `request_hook`?** The `request_hook` runs **before** Django's middleware chain, so `request.user` is anonymous and the resolved view name isn't on the request yet. Use `response_hook` for anything that depends on auth, sessions, or view resolution.+> **What about the instrumentation's `response_hook`?** It works, but only if you uninstrument Django first. Under `opentelemetry-instrument` the agent has already instrumented Django, and a second `.instrument(response_hook=...)` is a no-op: the hook never fires. All you get is one easily missed startup line on stderr, `WARNING ... Attempting to instrument while already instrumented`. See [Request & Response Hooks](/client/otel/django#request--response-hooks) for the working version. A middleware needs none of that.## Capturing Exceptions in Services-For exceptions in services that run outside a request (or are several call frames away from one), get the current span from context — it will be whatever span is active, including the Celery consumer span or a custom span you created:+For exceptions in services that run outside a request (or are several call frames away from one), get the current span from context. It will be whatever span is active, including the Celery consumer span or a custom span you created:```python# myapp/services/payments.py@@ -114,11 +117,11 @@ class PaymentService:raise```-If no span is currently active (you're in a management command without a wrapping span, for example), `trace.get_current_span()` returns a no-op span — `record_exception` is silently dropped. Wrap the entry point in a span first; see [Spans](/client/django/spans) and [Tasks](/client/django/tasks#management-commands).+If no span is currently active (you're in a management command without a wrapping span, for example), `trace.get_current_span()` returns a no-op span, so `record_exception` is silently dropped. Wrap the entry point in a `CONSUMER`-kind span first; see [Spans](/client/otel/django/spans) and [Tasks](/client/otel/django/tasks#cron-jobs-scripts-and-management-commands).## Custom Exception Types-Custom exception classes work the same way — their type name appears in the Traceway dashboard:+Custom exception classes work the same way. Their type name appears in the Traceway dashboard:```pythonclass InsufficientFundsError(RuntimeError):@@ -146,7 +149,7 @@ def withdraw(request):## Custom 500 Handler-If you've configured a `handler500` in `urls.py`, exceptions are already handled by Django's BaseHandler before your handler runs — the middleware has caught the exception, recorded it, and set the span status. Your `handler500` just renders the response. No extra instrumentation work is needed.+If you've configured a `handler500` in `urls.py`, exceptions are already handled by Django's BaseHandler before your handler runs. The middleware has caught the exception, recorded it, and set the span status. Your `handler500` just renders the response. No extra instrumentation work is needed.If you replace the default exception handling with custom middleware that swallows exceptions, record them on the span manually before returning a response:diff --git a/docs/pages/client/otel/django/index.mdx b/docs/pages/client/otel/django/index.mdxnew file mode 100644index 00000000..1778e3da--- /dev/null+++ b/docs/pages/client/otel/django/index.mdx@@ -0,0 +1,390 @@+# Django++The official [`opentelemetry-instrumentation-django`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-django) package (from `opentelemetry-python-contrib`) automatically instruments your Django application and exports traces, metrics, and logs to Traceway's OTLP endpoints. Setup is four steps: install the packages, set the environment variables, add one small middleware so endpoint names come out right, and launch through the OTel agent.++## Prerequisites++- Python 3.9+. On Python 3.9 `pip` installs the last release that still supported it (`opentelemetry-instrumentation-*` 0.62b1). Everything on this page works there. Use Python 3.10 or newer to get the current releases.+- Django 4.2, 5.x or 6.x+- pip+- A Traceway project (create one in the dashboard)++## Step 1: Install Packages++```bash+pip install \+ opentelemetry-distro \+ opentelemetry-exporter-otlp \+ opentelemetry-instrumentation-django+```++Then let the OTel bootstrapper auto-install the rest of the instrumentation that matches your installed dependencies (Postgres / MySQL / Redis / `requests` / `urllib3` / Celery / logging / …):++```bash+opentelemetry-bootstrap -a install+```++`opentelemetry-bootstrap` inspects your `site-packages` and only installs instrumentation packages that have a matching library present. Re-run it whenever you add or remove dependencies.++## Step 2: Configure Environment Variables++Set these in the **process environment**: your container `env:` block, a systemd `Environment=` line, a Procfile, or a plain shell `export`. `opentelemetry-instrument` reads them at interpreter startup, so a `.env` file that `settings.py` loads is always too late.++```bash+# Required. The agent instruments Django at interpreter startup, which is before+# manage.py / wsgi.py get a chance to set this. Leave it out and Django boots+# with empty settings. See Step 4 for what that looks like.+DJANGO_SETTINGS_MODULE=myproject.settings++OTEL_SERVICE_NAME=my-django-app+OTEL_TRACES_EXPORTER=otlp+OTEL_METRICS_EXPORTER=otlp+OTEL_LOGS_EXPORTER=otlp+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20your-project-token++# Puts otelTraceID / otelSpanID on every LogRecord so you can print them in your+# own console or file format. See the Logs page.+OTEL_PYTHON_LOG_CORRELATION=true++# Optional. Metrics are exported every 60s by default. Lower it while testing.+# OTEL_METRIC_EXPORT_INTERVAL=10000+```++If you keep these in a file for local development, source the file into your shell before launching:++```bash+set -a; . ./.env; set +a+```++Your Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.++> **Note:** `OTEL_EXPORTER_OTLP_HEADERS` is a comma-separated list of `key=value` pairs, so the `Bearer ` prefix's space must be URL-encoded as `%20`. The exporter URL-decodes header values automatically.++> **Do not set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`.** It is deprecated as of `opentelemetry-instrumentation-logging` 0.65b0. Setting it forces the SDK's deprecated log handler and prints a warning on every process start, which then gets shipped to Traceway as noise. `OTEL_LOGS_EXPORTER=otlp` is all you need. See [Logs](/client/otel/django/logs).++## Step 3: Add the Route Middleware (Required)++Django sets `http.route` to the URLconf pattern **without** a leading slash (`api/users/<int:user_id>/`). Traceway only accepts an `http.route` that starts with `/`, so without this step every endpoint shows up with a doubled method and no slash, like `GET GET api/users/<int:user_id>/`, and grouping is fragile.++Create one small middleware that normalizes the route:++```python+# myproject/traceway_route.py+from opentelemetry import trace+++class TracewayRouteMiddleware:+ """Give http.route a leading slash so Traceway groups Django routes."""++ def __init__(self, get_response):+ self.get_response = get_response++ def __call__(self, request):+ response = self.get_response(request)+ match = getattr(request, "resolver_match", None)+ route = getattr(match, "route", None) if match else None+ if route is not None:+ trace.get_current_span().set_attribute("http.route", "/" + route.lstrip("/"))+ return response+```++Add it to `MIDDLEWARE` in `settings.py`. Position does not matter as long as it is in the list, because the OTel middleware is injected above it at index `0` and its span is still active when yours runs:++```python+# myproject/settings.py+MIDDLEWARE = [+ "django.middleware.security.SecurityMiddleware",+ "django.contrib.sessions.middleware.SessionMiddleware",+ "django.middleware.common.CommonMiddleware",+ "django.middleware.csrf.CsrfViewMiddleware",+ "django.contrib.auth.middleware.AuthenticationMiddleware",+ "django.contrib.messages.middleware.MessageMiddleware",+ "django.middleware.clickjacking.XFrameOptionsMiddleware",+ "myproject.traceway_route.TracewayRouteMiddleware",+]+```++With this in place, `/api/users/11/`, `/api/users/12/` and `/api/users/13/` all land on a single `GET /api/users/<int:user_id>/` row in Traceway.++On ASGI, a sync-only middleware makes Django adapt the whole chain through a thread. If you run async Django, use the async-capable form instead (Django 4.2+, which ships `markcoroutinefunction` in `asgiref`):++```python+# myproject/traceway_route.py+from asgiref.sync import iscoroutinefunction, markcoroutinefunction+from opentelemetry import trace+++def _stamp_route(request):+ match = getattr(request, "resolver_match", None)+ route = getattr(match, "route", None) if match else None+ if route is not None:+ trace.get_current_span().set_attribute("http.route", "/" + route.lstrip("/"))+++class TracewayRouteMiddleware:+ """Give http.route a leading slash so Traceway groups Django routes."""++ async_capable = True+ sync_capable = True++ def __init__(self, get_response):+ self.get_response = get_response+ if iscoroutinefunction(self.get_response):+ markcoroutinefunction(self)++ def __call__(self, request):+ if iscoroutinefunction(self.get_response):+ return self.__acall__(request)+ response = self.get_response(request)+ _stamp_route(request)+ return response++ async def __acall__(self, request):+ response = await self.get_response(request)+ _stamp_route(request)+ return response+```++> **Why a middleware and not a `response_hook`?** The instrumentation's `response_hook` never fires when you launch through `opentelemetry-instrument`, because the agent has already instrumented Django and a second `.instrument(...)` call is a silent no-op. A middleware works with the agent as-is. If you do need a hook for other reasons, see [Request & Response Hooks](#request--response-hooks) below.++## Step 4: Launch Django Through the OTel Agent++Don't run `manage.py runserver` (or your WSGI/ASGI server) directly. Wrap it with `opentelemetry-instrument` so the agent can patch Django, the database driver, and the HTTP client libraries before your app imports them:++```bash+# Required: the agent instruments Django before manage.py / wsgi.py can set this.+export DJANGO_SETTINGS_MODULE=myproject.settings++# Development+opentelemetry-instrument python manage.py runserver++# Production (Gunicorn / WSGI)+opentelemetry-instrument gunicorn myproject.wsgi:application++# Production (Uvicorn / ASGI)+opentelemetry-instrument uvicorn myproject.asgi:application+```++`DjangoInstrumentor` then injects its middleware at index `0` of the `MIDDLEWARE` list at startup, so every inbound HTTP request is traced.++> **If you forget `DJANGO_SETTINGS_MODULE` the app will not boot.** The agent runs from `sitecustomize` at interpreter startup, before `manage.py`'s `main()` (or `wsgi.py`'s module body) runs `os.environ.setdefault("DJANGO_SETTINGS_MODULE", ...)`. The instrumentor finds no settings, calls `settings.configure()` with empty defaults, and your real `settings.py` is ignored from then on. `runserver` fails with `CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False.` even though your file says `DEBUG = True`, and Gunicorn boots but 500s every request with `AttributeError: module 'django.conf.global_settings' has no attribute 'ROOT_URLCONF'`.++> **Want to instrument manually instead of via the agent?** You can call `DjangoInstrumentor().instrument()` from a tiny entrypoint module (e.g. `myproject/otel.py`) that you import at the top of `manage.py` and `wsgi.py`. See [Manual Initialization](#manual-initialization) below. Using `opentelemetry-instrument` is strongly preferred because it also wires up exporters and resource detection for you.++## What Gets Captured++Once configured, the Django instrumentation automatically captures:++- **Endpoints**: every HTTP request with method and route template, set as `http.route`. Django writes the raw URLconf pattern with no leading slash (`api/users/<int:user_id>/`), which is why [Step 3](#step-3-add-the-route-middleware-required) is required. With it, endpoints read `GET /api/users/<int:user_id>/`.+- **Status codes**: 2xx, 4xx, 5xx responses+- **Exceptions**: unhandled exceptions are recorded on the request span via `span.record_exception()` and the span is marked `ERROR`. The response is a 500 and the error appears under **Issues** with the full Python traceback.+- **Active request counter**: concurrent in-flight request count, exported as `http.server.active_requests`+- **Duration histogram**: request latency, exported as `http.server.duration` (Traceway stores it as `http.server.duration.avg` and `http.server.duration.count`). Set `OTEL_SEMCONV_STABILITY_OPT_IN=http` if you want the stable name `http.server.request.duration` instead.++And once `opentelemetry-bootstrap -a install` has been run, the matching auto-instrumentations also capture:++| Library | What it captures |+|---------|------------------|+| `psycopg2` / `psycopg` / `mysqlclient` / `pymysql` / `sqlite3` | Every DB query: SQL statement, duration, error |+| `redis` / `aioredis` | Redis commands (GET/SET/…) |+| `requests` / `urllib` / `urllib3` / `httpx` | Outbound HTTP (propagates W3C trace context) |+| `celery` | Producer + consumer spans for queued tasks. See [Tasks](/client/otel/django/tasks) |+| `logging` | Standard `logging` records as OTel logs (attaches a handler to the root logger, needs `OTEL_LOGS_EXPORTER=otlp`) |+| `boto3` / `pika` / `kafka-python` / `pymongo` / `elasticsearch` / … | Bundled in `opentelemetry-bootstrap`'s registry; auto-installed if the library is present |++Disable any instrumentation you don't want by uninstalling its package, or by setting the matching `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS` env var:++```bash+# Comma-separated entry-point names, e.g. disable Redis + outbound requests+OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis,requests+```++## How Traceway Reads Your Spans++Traceway turns each incoming span into an endpoint, a task, an AI trace, a child in the waterfall, or nothing. Knowing the rules saves you from writing a span that is silently discarded.++| Your span | Shows up as |+|---|---|+| Root SERVER span with `http.route` or `http.method` | **Endpoint**, named `METHOD route`. The route must start with `/` |+| Any CONSUMER-kind span (Celery `run/...`) | **Task**, named after the span |+| Root INTERNAL span with a `console.command` attribute | **Task**, named after the span |+| Any non-root span | A child in the trace waterfall |+| Any span with `gen_ai.*` attributes | **AI trace** |+| **A root INTERNAL span with none of the above** | **Nothing. It is dropped, with no error** |++That last row is the one to remember. A bare `tracer.start_as_current_span("my.job")` at the top of a script, a cron entry, or a management command creates a root INTERNAL span, and Traceway discards it. Pass `kind=SpanKind.CONSUMER` to make it a Task. See [Tasks](/client/otel/django/tasks#cron-jobs-scripts-and-management-commands).++## Tuning the Django Instrumentation++The middleware reads a handful of environment variables at startup:++| Variable | Default | Purpose |+|----------|---------|---------|+| `OTEL_PYTHON_DJANGO_INSTRUMENT` | `True` | Set to `False` to disable Django instrumentation while keeping the rest. |+| `OTEL_PYTHON_DJANGO_EXCLUDED_URLS` | _(unset)_ | Comma-separated regexes. Matching paths are not traced. Example: `healthcheck,client/.*/info`. |+| `OTEL_PYTHON_DJANGO_TRACED_REQUEST_ATTRS` | _(unset)_ | Comma-separated `HttpRequest` attributes to attach as span attributes (e.g. `path_info,content_type`). |+| `OTEL_PYTHON_DJANGO_MIDDLEWARE_POSITION` | `0` | Index in the `MIDDLEWARE` list where the OTel middleware is inserted. |+| `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST` | _(unset)_ | Request headers to capture (regex list). Captured as `http.request.header.<lowercased_name>`. |+| `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE` | _(unset)_ | Response headers to capture (regex list). |+| `OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS` | _(unset)_ | Regexes of header names whose values should be redacted (e.g. `.*session.*,set-cookie`). |++`OTEL_PYTHON_DJANGO_INSTRUMENT` is compared against the exact string `"False"`, so a lowercase `false` will not disable it.++## Request & Response Hooks++Most attribute work is easier in your own middleware, the way [Step 3](#step-3-add-the-route-middleware-required) does it: your middleware runs inside the OTel span, so `trace.get_current_span()` is the request span and you can set anything on it. Use that first.++The instrumentation also supports a `response_hook`, which runs **after** Django middleware has populated `request.user` and `request.session`. There is one catch: under `opentelemetry-instrument` the agent has already instrumented Django, `BaseInstrumentor` is a singleton, and a second `.instrument(...)` call does nothing. The only sign is one line on stderr at startup, `WARNING ... Attempting to instrument while already instrumented`, which is easy to miss. You have to uninstrument first:++```python+# myproject/otel.py+from opentelemetry.instrumentation.django import DjangoInstrumentor+++def response_hook(span, request, response):+ if not span or not span.is_recording():+ return++ if hasattr(request, "user") and request.user.is_authenticated:+ span.set_attribute("enduser.id", request.user.pk)+ span.set_attribute("enduser.role", request.user.groups.values_list("name", flat=True).first() or "")++ span.set_attribute("http.response.body.size", len(response.content) if hasattr(response, "content") else 0)+++# Under `opentelemetry-instrument` Django is already instrumented by the time this+# module is imported, so a bare .instrument(...) is a silent no-op and the hook+# never fires. Uninstrument first, then re-instrument with the hook.+DjangoInstrumentor().uninstrument()+DjangoInstrumentor().instrument(response_hook=response_hook)+```++Then import this module **before** Django is imported, typically at the very top of `manage.py` / `wsgi.py` / `asgi.py`:++```python+# manage.py+import myproject.otel # noqa: F401 (must come first)++# ...standard manage.py body...+```++Note that a Celery worker never imports `manage.py` or `wsgi.py`, so a hook module only reaches the worker if `myproject/__init__.py` imports it.++> **Why `response_hook` and not `request_hook`?** The `request_hook` runs **before** Django middleware, so middleware-populated attributes like `request.user` are still anonymous when it fires. Use `response_hook` for anything that depends on auth, sessions, or view resolution.++## Manual Initialization++If you can't use the `opentelemetry-instrument` agent (for example you're in a managed environment that forbids wrapping the command line), set up the SDK and instrumentation programmatically. You have to build all three providers yourself, otherwise you get traces but no metrics and no logs:++```python+# myproject/otel.py+import logging+import os++# Must come before the DjangoInstrumentor import below. This module is imported+# before manage.py sets DJANGO_SETTINGS_MODULE, and without it the instrumentor+# falls back to empty Django settings and the app will not boot.+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")++from opentelemetry import metrics, trace # noqa: E402+from opentelemetry._logs import set_logger_provider # noqa: E402+from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter # noqa: E402+from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter # noqa: E402+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter # noqa: E402+from opentelemetry.instrumentation.django import DjangoInstrumentor # noqa: E402+from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler # noqa: E402+from opentelemetry.sdk._logs.export import BatchLogRecordProcessor # noqa: E402+from opentelemetry.sdk.metrics import MeterProvider # noqa: E402+from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader # noqa: E402+from opentelemetry.sdk.resources import Resource # noqa: E402+from opentelemetry.sdk.trace import TracerProvider # noqa: E402+from opentelemetry.sdk.trace.export import BatchSpanProcessor # noqa: E402++resource = Resource.create({"service.name": "my-django-app"})++tracer_provider = TracerProvider(resource=resource)+tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))+trace.set_tracer_provider(tracer_provider)++metrics.set_meter_provider(+ MeterProvider(+ resource=resource,+ metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter())],+ )+)++logger_provider = LoggerProvider(resource=resource)+logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))+set_logger_provider(logger_provider)+logging.getLogger().addHandler(LoggingHandler(logger_provider=logger_provider))++DjangoInstrumentor().instrument()+```++Import this module before Django loads:++```python+# manage.py+import myproject.otel # noqa: F401+```++The exporter constructors read `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` from the environment, so you still configure them the same way.++You still need the [route middleware from Step 3](#step-3-add-the-route-middleware-required) here. The providers register an `atexit` hook by default, so a clean exit still flushes the last batch.++> **If you also set `LOGGING` in `settings.py`, register the OTel handler there.** Django applies your `LOGGING` dict during `django.setup()`, which happens after this module has run, and a `dictConfig` clears the root logger's handlers first. The `addHandler` line above is wiped and not one log record reaches Traceway. Traces and metrics keep flowing, so nothing looks broken. The agent path does not have this problem, because `opentelemetry-instrumentation-logging` wraps `logging.config.dictConfig` and re-attaches its own handler afterwards. On the manual path, drop the `addHandler` line and list the handler in `LOGGING` instead:+>+> ```python+> # myproject/settings.py+> LOGGING = {+> "version": 1,+> "disable_existing_loggers": False,+> "handlers": {+> "console": {"class": "logging.StreamHandler"},+> "otel": {"class": "opentelemetry.sdk._logs.LoggingHandler"},+> },+> "root": {"handlers": ["console", "otel"], "level": "INFO"},+> }+> ```+>+> `LoggingHandler` with no arguments uses the global logger provider, which `set_logger_provider(...)` above already installed. Do not add this entry when you launch through `opentelemetry-instrument`. You would then have two OTel handlers on the root logger and every record would arrive in Traceway twice.++## Test Your Integration++Add a test route to verify data is flowing:++```python+# myapp/views.py+from django.http import HttpResponse+++def testing(request):+ raise RuntimeError("Test error from Traceway integration")+```++```python+# myproject/urls.py+from django.urls import path+from myapp import views++urlpatterns = [+ path("testing/", views.testing),+]+```++Visit `/testing/` in your browser. The request returns a 500, the endpoint appears as `GET /testing/` with status `500`, and the `RuntimeError` shows up under **Issues** with the full Python traceback.++> **How quickly does data appear?** Spans and logs are batched and flushed within a few seconds. Metrics are exported every **60 seconds** by default, so a brand new custom counter or gauge can take up to a minute to show up. Set `OTEL_METRIC_EXPORT_INTERVAL=10000` (milliseconds) while you are testing.+>+> Short-lived processes such as management commands and one-off scripts flush on exit, under `opentelemetry-instrument` and with a hand-built provider alike, because the SDK registers an `atexit` hook. Only an abrupt end such as `os._exit` or `SIGKILL` loses the batch.++## Next Steps++- [Exceptions](/client/otel/django/exceptions): manually capture caught exceptions with context+- [Spans](/client/otel/django/spans): create custom spans to measure sub-operations+- [Tasks](/client/otel/django/tasks): trace Celery tasks and Django management commands as background tasks+- [Metrics](/client/otel/django/metrics): track custom counters, histograms, and gauges with the OTel Metrics API+- [Logs](/client/otel/django/logs): forward Django + Python `logging` records to Traceway over OTLP+- [OpenTelemetry docs](/client/otel): how OTel traces, metrics, and logs map to Tracewaydiff --git a/docs/pages/client/otel/django/logs.mdx b/docs/pages/client/otel/django/logs.mdxnew file mode 100644index 00000000..e8bc450c--- /dev/null+++ b/docs/pages/client/otel/django/logs.mdx@@ -0,0 +1,186 @@+# Logs++`opentelemetry-instrumentation-logging` (auto-installed by `opentelemetry-bootstrap`) bridges Python's standard `logging` module to the OpenTelemetry Logs SDK and OTLP exporter. Once enabled, every `logger.info(...)` / `logger.error(...)` is forwarded to Traceway and linked to the active trace and span. That includes records emitted by Django itself, your views, your Celery tasks, and any third-party library that uses `logging`.++Two steps, in this order: turn the exporter on, then set the root logger level explicitly.++## Step 1: Enable OTLP Logs++Extend the env config from the [Quick Start](/client/otel/django) to also turn on the logs exporter:++```bash+# Existing config from the Quick Start+DJANGO_SETTINGS_MODULE=myproject.settings+OTEL_SERVICE_NAME=my-django-app+OTEL_TRACES_EXPORTER=otlp+OTEL_METRICS_EXPORTER=otlp++# Enable OTLP logs+OTEL_LOGS_EXPORTER=otlp++# Put otelTraceID / otelSpanID on every LogRecord (see Step 2)+OTEL_PYTHON_LOG_CORRELATION=true++OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20your-project-token+```++`OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL. The logs exporter appends `/v1/logs` automatically. No extra endpoint config is needed.++These variables are read once at interpreter startup, so set them in your process environment, not in a `.env` file that `settings.py` loads.++> **`OTEL_LOGS_EXPORTER=otlp` is all you need to ship logs.** `opentelemetry-bootstrap -a install` installs `opentelemetry-instrumentation-logging`, which attaches a `LoggingHandler` to the root logger for you.+>+> **Do not set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`.** It is deprecated as of `opentelemetry-instrumentation-logging` 0.65b0. Setting it swaps in the SDK's deprecated handler and prints a warning on every process start, and that warning is itself shipped to Traceway as a log record. Leave it unset. If you want to turn the handler off entirely, set `OTEL_PYTHON_LOG_AUTO_INSTRUMENTATION=false`.++## Step 2: Set the Root Logger Level (Required)++A stock Django project leaves the **root logger at `WARNING`**. Python drops `logger.debug(...)` and `logger.info(...)` before they reach any handler, so they never reach the OTel handler either. `OTEL_PYTHON_LOG_CORRELATION=true` from Step 1 happens to raise the level to `INFO`, because it calls `logging.basicConfig(level=logging.INFO)` behind your back. Do not rely on that side effect. Configure `LOGGING` in `settings.py`:++```python+# myproject/settings.py+LOGGING = {+ "version": 1,+ "disable_existing_loggers": False,+ "formatters": {+ "with_trace": {+ "format": "[{levelname}] {message} | trace_id={otelTraceID} span_id={otelSpanID}",+ "style": "{",+ },+ },+ "handlers": {+ "console": {+ "class": "logging.StreamHandler",+ "formatter": "with_trace",+ },+ },+ "root": {"handlers": ["console"], "level": "INFO"},+}+```++Two things about this block:++- The `level` is what decides whether a record exists at all. `"INFO"` ships `INFO` and above. Use `"DEBUG"` if you want debug records in Traceway too.+- The `{otelTraceID}` / `{otelSpanID}` placeholders in the formatter only work when `OTEL_PYTHON_LOG_CORRELATION=true` is set (Step 1). Trace-context injection is opt-in. Without it those fields do not exist on the `LogRecord` and **every console line** raises `ValueError: Formatting field not found in record: 'otelTraceID'`. If you would rather not set that variable, drop the `| trace_id=... span_id=...` part of the format string.++This `dictConfig` does not remove the OTel handler. After Django applies it the root logger holds both your `StreamHandler` and `opentelemetry.instrumentation.logging.handler.LoggingHandler`, so console output and OTLP export both keep working.++`OTEL_PYTHON_LOG_CORRELATION=true` also stamps `otelTraceID`, `otelSpanID`, `otelServiceName` and `otelTraceSampled` as attributes on the OTel log record, so they are visible in Traceway next to your own `extra` keys.++## Step 3: Use Python's `logging` Module as Usual++Use the standard Python logger. Because the handler runs inside the request / Celery task span, every record emitted from there carries that span's `trace_id` and `span_id` automatically:++```python+# myapp/views.py+import logging++from django.http import JsonResponse++logger = logging.getLogger(__name__)+++def create_order(request):+ order_id = "ord_123"+ logger.info("order received", extra={"order.id": order_id})++ try:+ # ... business logic ...+ logger.info("order processed", extra={"order.id": order_id})+ return JsonResponse({"status": "ok"})+ except Exception as exc:+ logger.error("order failed", extra={"order.id": order_id, "exception": str(exc)})+ raise+```++Open the endpoint's trace in the Traceway dashboard and the **Logs** tab will show these records attached. The same is true for logs emitted inside Celery tasks, management commands, and any other code that runs under an active span.++> **Attributes via `extra={...}`:** OTel's logging handler reads the `extra` dict on a `LogRecord` and attaches each key as an attribute on the OTel log record. So `logger.info("order received", extra={"order.id": "ord_123"})` produces an OTel log with attribute `order.id="ord_123"`. Use dotted keys (e.g. `order.id`, `user.id`) to match OTel semantic conventions.++## Reading `trace_id` in Console / File Logs++With `OTEL_PYTHON_LOG_CORRELATION=true` and the `with_trace` formatter from Step 2, your raw console or file output carries the same ids Traceway shows, which makes it easy to jump from a log line to a trace:++```+[INFO] fetching user | trace_id=36d1a9e2918feaa0487fb7deb4e5d001 span_id=729a141ab07be6fc+[INFO] order received | trace_id=852364b111d5fb102122f0743a722986 span_id=80637045ab27cd48+[INFO] Watching for file changes with StatReloader | trace_id=0 span_id=0+```++Records emitted outside any active span (startup messages, for example) get `"0"` placeholders, as in the last line.++> **Don't add an OTel `LoggingHandler` yourself.** `opentelemetry-instrumentation-logging` already attaches one to the root logger. Adding another produces duplicate log records in Traceway.++## Severity Mapping++Python's `logging` levels are mapped to OpenTelemetry severity numbers (and Traceway's TRACE → FATAL labels) as follows:++| Python Level | Numeric | OTLP Severity Number | Traceway Severity |+| ------------ | ------- | -------------------- | ----------------- |+| `DEBUG` | 10 | 5 | DEBUG |+| `INFO` | 20 | 9 | INFO |+| `WARNING` | 30 | 13 | WARN |+| `ERROR` | 40 | 17 | ERROR |+| `CRITICAL` | 50 | 21 | FATAL |++Only records at or above the root logger's configured `level` are emitted to handlers, which is why [Step 2](#step-2-set-the-root-logger-level-required) is required. Raise or lower `LOGGING["root"]["level"]` to control volume.++## Logging Exceptions With Full Tracebacks++`logger.exception(...)` and `logger.error(..., exc_info=True)` both attach the traceback to the OTel log record's `exception.stacktrace` attribute, which Traceway extracts into the Logs view:++```python+import logging++logger = logging.getLogger(__name__)+++def withdraw(request):+ try:+ _withdraw(request.user, request.POST["amount"])+ except InsufficientFundsError:+ logger.exception("withdraw failed", extra={"user.id": request.user.pk})+ raise+```++Note that the *unhandled* exception is **also** captured on the request span by the Django middleware (see [Exceptions](/client/otel/django/exceptions)). `logger.exception(...)` is for logging the same event into the Logs feed with structured attributes. It is not a substitute for re-raising.++## Test Your Integration++Add a route that logs at multiple levels, hit it once, and check the **Logs** page in the Traceway dashboard:++```python+# myapp/views.py+import logging++from django.http import JsonResponse++logger = logging.getLogger(__name__)+++def log_test(request):+ logger.debug("debug sample")+ logger.info("info sample", extra={"request.id": "req_1"})+ logger.warning("warning sample")+ logger.error("error sample", extra={"order.id": "ord_1"})+ return JsonResponse({"ok": True})+```++```python+# myproject/urls.py+urlpatterns = [+ # ...+ path("log-test/", views.log_test),+]+```++Visit `/log-test/`, then open **Logs** in the dashboard. With the root level at `INFO` you get **three** records within a few seconds (`info`, `warning`, `error`), each linked to the trace for that request. The `debug` line is dropped by the level, not by OTel. Set the root level to `"DEBUG"` if you want all four.++If you see nothing at all, check in this order: the root logger level (Step 2), `OTEL_LOGS_EXPORTER=otlp` in the process environment (Step 1), and that you launched through `opentelemetry-instrument`.++## Next Steps++- [Exceptions](/client/otel/django/exceptions): record caught exceptions with context+- [Metrics](/client/otel/django/metrics): custom counters, histograms, and gauges+- [OTel Logs reference](/client/otel/logs): full OTLP logs mapping and severity detailsdiff --git a/docs/pages/client/django/metrics.mdx b/docs/pages/client/otel/django/metrics.mdxsimilarity index 64%rename from docs/pages/client/django/metrics.mdxrename to docs/pages/client/otel/django/metrics.mdxindex af12669b..23a2d4a6 100644--- a/docs/pages/client/django/metrics.mdx+++ b/docs/pages/client/otel/django/metrics.mdx@@ -1,10 +1,12 @@# Metrics-The OTel auto-instrumentation captures default request, DB, and outbound-HTTP metrics automatically (e.g. `http.server.request.duration`, `http.server.active_requests`, `db.client.operation.duration`). To track custom application metrics, use the OTel Metrics API.+The OTel auto-instrumentation captures default request and outbound-HTTP metrics automatically: `http.server.duration`, `http.server.active_requests` and `http.client.duration`. The stable names `http.server.request.duration` / `http.client.request.duration` only appear if you set `OTEL_SEMCONV_STABILITY_OPT_IN=http`. The Python DB instrumentations produce spans only, so there is no `db.client.operation.duration` metric. To track custom application metrics, use the OTel Metrics API.++> **Metrics are exported every 60 seconds** by default, so a new instrument can take up to a minute to show up in the dashboard. Set `OTEL_METRIC_EXPORT_INTERVAL=10000` (milliseconds) while you are testing.## Getting a Meter-Call `metrics.get_meter(__name__)` once per module — meters are cheap and cached:+Call `metrics.get_meter(__name__)` once per module. Meters are cheap and cached:```pythonfrom opentelemetry import metrics@@ -22,7 +24,7 @@ Instruments created from the same meter + same name return the same instance, so| `meter.create_up_down_counter(...)` | Counter that can go up or down || `meter.create_observable_counter(...)` | Counter whose value is read via callback at export time || `meter.create_observable_up_down_counter(...)` | Up/down counter read via callback |-| `meter.create_gauge(...)` | Synchronous gauge — record a value at a point in time |+| `meter.create_gauge(...)` | Synchronous gauge. Call `.set(value)` with a point-in-time value (it is `.set()`, not `.record()`) || `meter.create_observable_gauge(...)` | Gauge read via callback at export time || `meter.create_histogram(name, unit, description)` | Distribution of values (durations, sizes) |@@ -77,16 +79,16 @@ Traceway converts histograms into average and count metrics (see [OTel metrics](Gauges track point-in-time values that can go up or down (queue depth, active connections):```python-# Synchronous — record a value when you know it+# Synchronous: set a value when you know itcache_size_gauge = meter.create_gauge("cache.size_mb",unit="MB",description="Cache memory usage",)-cache_size_gauge.record(42.3)+cache_size_gauge.set(42.3)-# Observable — value is read by a callback at export time+# Observable: value is read by a callback at export timefrom opentelemetry.metrics import Observation@@ -102,28 +104,43 @@ meter.create_observable_gauge()```-## Batch Observation+The synchronous gauge is the one people get wrong. `.record()` belongs to `Histogram`. A gauge only has `.set()`, and calling `.record()` on one raises `AttributeError: '_Gauge' object has no attribute 'record'`, which in a Django view means a 500.++## Sharing One Expensive Reading Between Instruments-When several observable instruments share an expensive data source, use a single callback that yields multiple `Observation`s — the source is only consulted once per export cycle:+OpenTelemetry Python has no batch-callback API (unlike the Go and Java SDKs). Each instrument's callbacks are invoked separately at collection time, so registering one callback against two instruments calls it twice, not once. Cache the reading yourself:```python+import time+from opentelemetry.metrics import Observation+_cache = {"at": 0.0, "usage": 0.0, "pressure": 0.0}+-def system_callback(options):- usage, pressure = expensive_system_call()- return [- Observation(usage, {"source": "system"}),- Observation(pressure, {"source": "system"}),- ]+def _refresh():+ now = time.monotonic()+ if now - _cache["at"] > 5:+ _cache["usage"], _cache["pressure"] = expensive_system_call()+ _cache["at"] = now-# Register the same callback against both instruments. The OTel SDK-# de-duplicates the call internally so expensive_system_call() runs once.-meter.create_observable_counter("usage", callbacks=[system_callback], description="count of items used")-meter.create_observable_gauge("pressure", callbacks=[system_callback], description="force per unit area")+def usage_callback(options):+ _refresh()+ return [Observation(_cache["usage"], {"source": "system"})]+++def pressure_callback(options):+ _refresh()+ return [Observation(_cache["pressure"], {"source": "system"})]+++meter.create_observable_counter("usage", callbacks=[usage_callback], description="count of items used")+meter.create_observable_gauge("pressure", callbacks=[pressure_callback], description="force per unit area")```+One more rule that makes the naive version silently wrong: multiple `Observation`s returned by a single callback must differ in their attribute set. Return `Observation(10, {"source": "system"})` and `Observation(20, {"source": "system"})` from the same callback and they are aggregated into one data point (a counter reports 30, a gauge reports 20). Give each observation its own attributes, or use one callback per instrument as above.+## Use Cases### Business Metricsdiff --git a/docs/pages/client/django/spans.mdx b/docs/pages/client/otel/django/spans.mdxsimilarity index 75%rename from docs/pages/client/django/spans.mdxrename to docs/pages/client/otel/django/spans.mdxindex 5fe4b3b7..1e363435 100644--- a/docs/pages/client/django/spans.mdx+++ b/docs/pages/client/otel/django/spans.mdx@@ -4,7 +4,7 @@ The instrumentation creates spans for HTTP requests, DB queries, Celery tasks, c## Getting a Tracer-Call `trace.get_tracer(__name__)` once per module — tracers are cheap, cached by name, and safe to keep at module scope:+Call `trace.get_tracer(__name__)` once per module. Tracers are cheap, cached by name, and safe to keep at module scope:```pythonfrom opentelemetry import trace@@ -12,7 +12,7 @@ from opentelemetry import tracetracer = trace.get_tracer(__name__)```-The recommended pattern is `start_as_current_span` as a context manager — it activates the span, runs the block, ends the span on exit, and records any exception that propagates out:+The recommended pattern is `start_as_current_span` as a context manager. It activates the span, runs the block, ends the span on exit, and records any exception that propagates out:```pythonclass OrderService:@@ -24,7 +24,7 @@ class OrderService:self._charge_payment(order_id)```-> **Exception handling inside `start_as_current_span`:** by default, if the block raises, the span calls `record_exception(exc)` *and* `set_status(ERROR)` before re-raising — both happen automatically. This is different from some other OTel SDKs (e.g. PHP's `keepsuit/laravel-opentelemetry::measure`) which only call `record_exception` and leave the status unset. If you want to suppress the auto-status (for example you're treating a `Http404` as a normal outcome, not an error), pass `set_status_on_exception=False`:+> **Exception handling inside `start_as_current_span`:** by default, if the block raises, the span calls `record_exception(exc)` *and* `set_status(ERROR)` before re-raising. Both happen automatically. This is different from some other OTel SDKs (e.g. PHP's `keepsuit/laravel-opentelemetry::measure`) which only call `record_exception` and leave the status unset. If you want to suppress the auto-status (for example you're treating a `Http404` as a normal outcome, not an error), pass `set_status_on_exception=False`:>> ```python> with tracer.start_as_current_span("lookup-user", set_status_on_exception=False) as span:@@ -34,6 +34,8 @@ class OrderService:> span.set_attribute("user.found", False)> return None> ```+>+> `set_status_on_exception=False` only suppresses the span status. The `exception` event is still recorded if the exception escapes the block, and Traceway still opens an Issue from it. Catch the exception inside the block, as above, if you don't want that.### With Attributes@@ -84,7 +86,9 @@ with trace.use_span(span, end_on_exit=True):do_work()```-Setting `end_on_exit=True` makes `use_span` close the span when the block exits. Always pair `start_span` with either `use_span(..., end_on_exit=True)` or an explicit `span.end()` in a `finally` block — otherwise the span leaks and is never exported.+Setting `end_on_exit=True` makes `use_span` close the span when the block exits. Always pair `start_span` with either `use_span(..., end_on_exit=True)` or an explicit `span.end()` in a `finally` block. Otherwise the span leaks and is never exported.++> **Root spans need a kind.** Everything above assumes there is already a parent span, which is the case inside a request or a Celery task. A span you open at the top of a script, a cron entry, or a management command is a **root** span, and a root span with the default `INTERNAL` kind is dropped by Traceway with no error. Pass `kind=SpanKind.CONSUMER` so it becomes a Task. See [Tasks](/client/otel/django/tasks#cron-jobs-scripts-and-management-commands).## Other Tracer Utilities@@ -98,7 +102,7 @@ trace.get_current_span() # the currently active Spantrace.get_current_span().get_span_context() # SpanContext (trace_id, span_id, flags)format(trace.get_current_span().get_span_context().trace_id, "032x") # trace id as hex string-# Inject the active trace context into outbound headers — most users don't need this,+# Inject the active trace context into outbound headers. Most users don't need this,# because requests/httpx/urllib3 instrumentation already injects W3C headers automatically.carrier: dict[str, str] = {}TraceContextTextMapPropagator().inject(carrier)@@ -117,7 +121,7 @@ with tracer.start_as_current_span("db-query") as span:span.set_attribute("db.row_count", len(rows))```-For DB queries you go through Django's ORM, you don't need to do this — `opentelemetry-instrumentation-psycopg2` (or your DB driver's matching instrumentation) sets `db.system` / `db.statement` automatically.+For DB queries you go through Django's ORM, you don't need to do this. `opentelemetry-instrumentation-psycopg2` (or your DB driver's matching instrumentation) sets `db.system` / `db.statement` automatically.## Recording Errors on Spansdiff --git a/docs/pages/client/otel/django/tasks.mdx b/docs/pages/client/otel/django/tasks.mdxnew file mode 100644index 00000000..8d5ff866--- /dev/null+++ b/docs/pages/client/otel/django/tasks.mdx@@ -0,0 +1,202 @@+# Tasks++Django itself doesn't ship a built-in task queue, so most Django apps use **Celery**. The official [`opentelemetry-instrumentation-celery`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-celery) package auto-traces Celery: when a task is dispatched, a `PRODUCER` span is created; when the worker picks it up, a `CONSUMER` span runs the handler. W3C trace context is propagated through the task headers so the consumer span links back to the dispatch trace. The full lifecycle is visible as one continuous trace in Traceway, and each consumed task appears as a **task** with duration, status, and any child spans.++This page covers Celery (the typical setup), plus cron jobs, scripts and Django management commands. Other Django task systems (RQ, Huey, Django-Q, Dramatiq) follow the same OTel idioms.++## The One Rule to Know First++Traceway turns a span into a **Task** when the span is `CONSUMER` kind, or when it is a root span carrying a `console.command` attribute. A root span with the default `INTERNAL` kind and neither of those is **dropped silently**: no task, no row, no error anywhere.++Celery gets this right for free, because the instrumentation already opens a `CONSUMER` span. Everything you write by hand needs `kind=SpanKind.CONSUMER` at the root, or it produces nothing.++## Celery Setup++`opentelemetry-bootstrap -a install` only installs instrumentation for libraries that are already present, so a project that added Celery after the [Quick Start](/client/otel/django) does not have it yet. Install it directly, or re-run the bootstrapper:++```bash+pip install opentelemetry-instrumentation-celery+# or re-run: opentelemetry-bootstrap -a install+```++Then launch the worker through the agent. That is all that is required:++```bash+export DJANGO_SETTINGS_MODULE=myproject.settings++opentelemetry-instrument celery -A myproject worker -l info+opentelemetry-instrument celery -A myproject beat -l info+```++The same setup applies whether you use `redis`, `rabbitmq`, `sqs`, or `database` brokers.++> **Don't call `CeleryInstrumentor().instrument()` yourself when using the agent.** The agent already did it. `BaseInstrumentor` is a singleton, so a second call is a silent no-op. It would also be the wrong place: the worker's entrypoint is `celery`, so it never imports `manage.py` or `wsgi.py`, and a module you put there is only loaded by the worker if `myproject/__init__.py` imports it (the file that holds `from .celery import app` in the standard Celery-Django layout).++## What Gets Captured++For each Celery task the instrumentation creates two spans:++- **Publish span**: name `apply_async/<task-name>`, `PRODUCER` kind, ended when Celery finishes pushing the payload to the broker.+- **Run span**: name `run/<task-name>`, `CONSUMER` kind, ended when the handler returns or fails. On a task failure the span records the exception and is marked `STATUS_ERROR`.++The two spans carry different attributes. This is what `opentelemetry-instrumentation-celery` 0.65b0 actually emits:++| Attribute | Producer | Consumer | Value |+|---|---|---|---|+| `messaging.destination` | yes | yes | The queue name (e.g. `celery`, `priority-high`) |+| `messaging.destination_kind` | yes | no | `queue` |+| `messaging.message.id` | yes | yes | Celery's task UUID (note the dots, not `message_id`) |+| `messaging.conversation_id` | no | yes | Celery's `root_id`, for chained tasks |+| `celery.task_name` | yes | yes | Fully-qualified task name (e.g. `myapp.tasks.send_welcome_email`) |+| `celery.action` | yes | yes | `apply_async` (producer) or `run` (consumer) |+| `celery.state` | no | yes | Task state on completion: `SUCCESS`, `FAILURE`, `RETRY` |+| `celery.hostname`, `celery.reply_to`, `celery.delivery_info` | no | yes | Worker and delivery metadata |++There is no `messaging.system` attribute on either span.++W3C trace context is also injected into the task headers, so the run span is linked to the trace where the task was originally dispatched, even if dispatch and execution happen on different services. Any spans you create inside the task body automatically nest under the run span.++## Adding Attributes in Tasks++Since the instrumentation activates the consumer span before your task body runs, use `trace.get_current_span()` to add custom attributes:++```python+# myapp/tasks.py+from celery import shared_task+from opentelemetry import trace+++@shared_task+def send_order_email(order_id: int, recipient: str) -> None:+ span = trace.get_current_span()+ span.set_attribute("order.id", order_id)+ span.set_attribute("email.to", recipient)++ # ... business logic ...+```++## Child Spans in Tasks++For tasks with distinct sub-operations, use the `Tracer` API to create child spans. They automatically nest under the task's run span:++```python+# myapp/tasks.py+from celery import shared_task+from opentelemetry import trace++tracer = trace.get_tracer(__name__)+++@shared_task+def process_payment(payment_id: int, amount: float) -> None:+ with tracer.start_as_current_span("validate.payment") as span:+ span.set_attribute("payment.id", payment_id)+ _validate(payment_id)++ with tracer.start_as_current_span("charge.gateway") as span:+ span.set_attribute("payment.amount", amount)+ _charge_gateway(payment_id, amount)+```++## Periodic Tasks (Celery Beat)++Tasks scheduled by Celery Beat run through the same code path as `apply_async`, so they get the same producer + consumer span pair automatically. The producer span is opened by Beat when it dispatches the schedule entry; the consumer span is opened by whichever worker picks the task up.++If you want to attribute beat-dispatched tasks differently in the dashboard, set an attribute on the consumer side:++```python+@shared_task(bind=True)+def nightly_cleanup(self) -> None:+ span = trace.get_current_span()+ span.set_attribute("celery.scheduled_by", "beat")++ # ... work ...+```++## Cron Jobs, Scripts and Management Commands++Django management commands run outside the HTTP request cycle, so the Django middleware never opens a root span for them. Unlike Celery tasks, they are **not traced automatically**. You open the root span yourself, and it must be `CONSUMER` kind or Traceway drops it.++```python+# myapp/management/commands/cleanup_orphans.py+from django.core.management.base import BaseCommand+from opentelemetry import trace+from opentelemetry.trace import SpanKind, Status, StatusCode++tracer = trace.get_tracer(__name__)+++class Command(BaseCommand):+ help = "Delete orphaned rows that have no foreign key parent."++ def add_arguments(self, parser):+ parser.add_argument("--dry-run", action="store_true")++ def handle(self, *args, **options):+ with tracer.start_as_current_span(+ "cleanup_orphans", # this is the Task name in Traceway+ kind=SpanKind.CONSUMER, # required, or the span is dropped+ attributes={"console.command": "cleanup_orphans"},+ ) as span:+ span.set_attribute("cleanup.dry_run", bool(options["dry_run"]))++ try:+ with tracer.start_as_current_span("cleanup.scan") as child:+ deleted = self._scan_and_delete(options["dry_run"])+ child.set_attribute("rows.deleted", deleted)+ except Exception as exc:+ span.record_exception(exc)+ span.set_status(Status(StatusCode.ERROR, str(exc)))+ raise+```++Run the command via the OTel agent so the SDK and exporters are configured exactly like the web process, and so the span is flushed when the process exits:++```bash+export DJANGO_SETTINGS_MODULE=myproject.settings+opentelemetry-instrument python manage.py cleanup_orphans+```++The command now appears under **Tasks** as `cleanup_orphans`, with `cleanup.scan` nested underneath it.++Two things are doing work in that snippet:++- `kind=SpanKind.CONSUMER` is what makes it a Task. Leave it out and you get a root `INTERNAL` span, which Traceway discards without an error. The span really is exported by the SDK, it just never becomes a row, so this failure is invisible unless you know to look for it.+- `console.command` is the second rule that promotes a root span to a Task. Setting both means the span survives either way.++The same pattern covers plain scripts and cron entries that never touch `manage.py`:++```python+# scripts/nightly_report.py+import django+import os++os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")+django.setup()++from opentelemetry import trace # noqa: E402+from opentelemetry.trace import SpanKind # noqa: E402++tracer = trace.get_tracer(__name__)+++def main():+ with tracer.start_as_current_span(+ "nightly_report",+ kind=SpanKind.CONSUMER,+ attributes={"console.command": "nightly_report"},+ ) as span:+ rows = build_report()+ span.set_attribute("report.rows", rows)+++if __name__ == "__main__":+ main()+```++```bash+export DJANGO_SETTINGS_MODULE=myproject.settings+opentelemetry-instrument python scripts/nightly_report.py+```++If you skip the wrapping span entirely, the command's DB queries and HTTP calls still produce spans, but each one is a disconnected root span in Traceway. The wrapping span is what ties them into a single task you can filter on.diff --git a/docs/pages/client/hono/_meta.json b/docs/pages/client/otel/hono/_meta.jsonsimilarity index 100%rename from docs/pages/client/hono/_meta.jsonrename to docs/pages/client/otel/hono/_meta.jsondiff --git a/docs/pages/client/otel/hono/index.mdx b/docs/pages/client/otel/hono/index.mdxnew file mode 100644index 00000000..78484b67--- /dev/null+++ b/docs/pages/client/otel/hono/index.mdx@@ -0,0 +1,418 @@+# Hono++Instrument your Hono application with OpenTelemetry and export traces, metrics, and logs to Traceway. Hono runs on multiple runtimes. This guide covers the Node.js setup. For Cloudflare Workers, see the [Cloudflare guide](/client/otel/cloudflare).++## Installation++```bash+npm install hono @hono/node-server @hono/otel \+ @opentelemetry/api \+ @opentelemetry/api-logs \+ @opentelemetry/sdk-node \+ @opentelemetry/sdk-metrics \+ @opentelemetry/sdk-logs \+ @opentelemetry/auto-instrumentations-node \+ @opentelemetry/exporter-trace-otlp-http \+ @opentelemetry/exporter-metrics-otlp-http \+ @opentelemetry/exporter-logs-otlp-http+```++Every package in that list is imported by code on this page. Install them all, even the ones npm would otherwise hoist for you, or the imports break under pnpm and Yarn PnP.++## Setup++### 1. Set the module type++The two files below use ESM `import` syntax, so `package.json` needs `"type": "module"`:++```json+{+ "name": "my-hono-app",+ "type": "module"+}+```++Skip this and Node stops at the first line with `SyntaxError: Cannot use import statement outside a module`.++### 2. Create `instrumentation.ts`++Put it at the root of your project, next to `package.json`:++```typescript+import { NodeSDK } from "@opentelemetry/sdk-node";+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";+import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";+import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";++const OTEL_URL = "https://your-traceway-instance.com/api/otel";+const HEADERS = { Authorization: "Bearer your-project-token" };++const sdk = new NodeSDK({+ serviceName: "my-hono-app",++ traceExporter: new OTLPTraceExporter({+ url: `${OTEL_URL}/v1/traces`,+ headers: HEADERS,+ }),++ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({+ url: `${OTEL_URL}/v1/metrics`,+ headers: HEADERS,+ }),+ exportIntervalMillis: 30_000,+ }),+ ],++ logRecordProcessors: [+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({+ url: `${OTEL_URL}/v1/logs`,+ headers: HEADERS,+ }),+ }),+ ],++ // @hono/otel owns the incoming request span. See the note below.+ instrumentations: [+ getNodeAutoInstrumentations({+ "@opentelemetry/instrumentation-http": { enabled: false },+ }),+ ],+});++sdk.start();++const shutdown = async () => {+ await sdk.shutdown();+ process.exit(0);+};+process.on("SIGTERM", shutdown);+process.on("SIGINT", shutdown);+```++Replace `your-traceway-instance.com` and `your-project-token` with your own values. The token is on the Connection page of your Traceway dashboard.++Three details on that file are easy to get wrong:++- `serviceName` is not optional in practice. Leave it out and every endpoint, span, log, and metric lands under `unknown_service:node`, which makes the service filter useless as soon as a second app reports to the same project.+- The option is `metricReaders` (an array). The older `metricReader` still works but prints a deprecation warning.+- The `shutdown` handler matters more than it looks. Telemetry is batched before it is sent, so stopping the app without it throws away the last few seconds of data. That is usually the very request you just made to test the setup.++> **Why disable `instrumentation-http`?** `@opentelemetry/instrumentation-http` patches Node's `http` module, so it never sees Hono's router. Its spans carry no `http.route`, which means `/api/users/1` and `/api/users/2` arrive as two separate endpoints in Traceway instead of one `GET /api/users/:id` row, and a thrown error is not recorded as an exception. It also wins: once `instrumentation-http` is actually patching, it owns the root span and demotes the `@hono/otel` span to a child, so grouping breaks even though `@hono/otel` is installed. In the ESM setup on this page that line is a safety belt, not a fix. `instrumentation-http` cannot reach `node:http` through an ESM import until you register the loader hook, so it is inert here either way. It becomes load-bearing the moment you register the hook (see [Tracing Outgoing HTTP Clients](#tracing-outgoing-http-clients)) or run the app as CommonJS. Let [`@hono/otel`](https://www.npmjs.com/package/@hono/otel) own the incoming request span.+>+> The cost of turning it off is outgoing calls. Requests made through the `http` and `https` modules stop being traced, and that covers axios, got, node-fetch v2, and a lot of vendor SDKs. Calls through the global `fetch()` are still traced, by `@opentelemetry/instrumentation-undici`. See [Tracing Outgoing HTTP Clients](#tracing-outgoing-http-clients) for a config that keeps both.++## Application Code++Create `server.ts` next to `instrumentation.ts`. The `@hono/otel` middleware handles endpoint grouping via `http.route`, status codes, and exception recording:++```typescript+import { serve } from "@hono/node-server";+import { Hono } from "hono";+import { httpInstrumentationMiddleware } from "@hono/otel";++const app = new Hono();++app.use(httpInstrumentationMiddleware());++app.get("/api/users", (c) => {+ return c.json({ users: [] });+});++app.get("/api/users/:id", (c) => {+ const id = c.req.param("id");+ return c.json({ id, name: "John" });+});++serve({ fetch: app.fetch, port: 3000 });+```++> The export is `httpInstrumentationMiddleware`. Older guides show `import { otel } from "@hono/otel"`, which was removed in `@hono/otel` 1.0.0. On any version npm installs today that import fails at startup with `SyntaxError: The requested module '@hono/otel' does not provide an export named 'otel'`.++The middleware takes an optional config object with `serviceName`, `serviceVersion`, `captureRequestHeaders`, `captureResponseHeaders`, `captureActiveRequests`, and `spanNameFactory`. The defaults are fine for most apps.++## Running Your App++Load the instrumentation file before your application code:++```bash+node --import ./instrumentation.ts server.ts+```++Use `--import`. On Node 22.12 and newer, `--require` also loads this file, because `require()` can now load ESM that has no top-level await. On older versions it fails, so `--import` is the portable choice.++Node 22.18 and newer (including Node 24) run TypeScript files directly, with no build step and no extra flag. On older versions you have two options. Write both files in plain JavaScript, rename them to `instrumentation.js` and `server.js`, keep `"type": "module"`, and run:++```bash+node --import ./instrumentation.js server.js+```++Or install [tsx](https://tsx.is):++```bash+npm install -D tsx+node --import tsx --import ./instrumentation.ts server.ts+```++`tsx` has to come first so its loader is in place before Node reads `instrumentation.ts`.++## Test Your Integration++Add a route that throws:++```typescript+app.get("/test-error", () => {+ throw new Error("Test error from Hono");+});+```++Start the app, then send a few requests:++```bash+curl -i http://localhost:3000/test-error+curl http://localhost:3000/api/users/1+curl http://localhost:3000/api/users/2+curl http://localhost:3000/api/users/3+```++The first one returns `HTTP/1.1 500 Internal Server Error`. Hono's default error handler turns an uncaught throw into a 500, and `@hono/otel` records the exception on the span and sets `http.response.status_code` to 500.++Telemetry is batched, so wait about 10 seconds, or stop the app with Ctrl-C to flush it immediately. Then check the dashboard:++- **Endpoints** shows one row, `GET /api/users/:id`, with a count of 3. Three separate rows with literal ids means `http.route` is missing. Check that `app.use(httpInstrumentationMiddleware())` runs before your routes. If you also registered the loader hook from [Tracing Outgoing HTTP Clients](#tracing-outgoing-http-clients), check that you set `disableIncomingRequestInstrumentation: true` there, because otherwise `instrumentation-http` takes the root span.+- **Endpoints** shows `GET /test-error` with status 500.+- **Issues** shows `Error: Test error from Hono` with the full stack trace.++## Logs++The `logRecordProcessors` block in `instrumentation.ts` already ships logs. Emit them from inside a handler and they carry the request's trace id, so Traceway shows them on that trace:++```typescript+import { logs, SeverityNumber } from "@opentelemetry/api-logs";++const logger = logs.getLogger("my-hono-app");++app.get("/api/users/:id", (c) => {+ const id = c.req.param("id");++ logger.emit({+ severityNumber: SeverityNumber.INFO,+ severityText: "INFO",+ body: `fetching user ${id}`,+ attributes: { "user.id": id, route: "/api/users/:id" },+ });++ return c.json({ id, name: "John" });+});+```++There is no context to pass around. `@hono/otel` runs your handler inside the request span, so the log picks up that span's `trace_id` and `span_id` on its own.++> Pass the exporter to `BatchLogRecordProcessor` as `{ exporter }`, the way `instrumentation.ts` above does it. The older positional form, `new BatchLogRecordProcessor(exporter)`, still constructs without an error on current `@opentelemetry/sdk-logs` and then drops every log silently.++## Background Jobs++Cron jobs and queue consumers are not HTTP requests, so `@hono/otel` never sees them. Create the span yourself and set `kind: SpanKind.CONSUMER`. That is what files it under **Tasks** in Traceway. Replace `buildReport` with your own job:++```typescript+import { trace, context, ROOT_CONTEXT, SpanKind, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-hono-app");++async function nightlyReport() {+ // ROOT_CONTEXT detaches the job from whatever request happened to trigger it,+ // so it gets its own trace instead of hanging off an HTTP request.+ await context.with(ROOT_CONTEXT, async () => {+ await tracer.startActiveSpan(+ "nightly-report",+ { kind: SpanKind.CONSUMER, attributes: { "messaging.system": "cron" } },+ async (span) => {+ try {+ await buildReport();+ } catch (error) {+ span.recordException(error as Error);+ span.setStatus({ code: SpanStatusCode.ERROR });+ throw error;+ } finally {+ span.end();+ }+ }+ );+ });+}++setInterval(nightlyReport, 60 * 60 * 1000);+```++The task name in Traceway is the span name, `nightly-report`. To see the row without waiting an hour, call `nightlyReport()` once at startup as well.++> **Always set the kind.** A root span left at the default `SpanKind.INTERNAL` with no HTTP attributes matches nothing Traceway stores, so it is dropped on arrival: no endpoint, no task, no span row. Logs you emitted inside it still arrive, pointing at a trace that does not exist, which is a confusing thing to debug. `kind: SpanKind.CONSUMER` is the whole fix.++## Custom Metrics++The `metricReaders` block in `instrumentation.ts` exports anything you record through the OTel metrics API. Replace `processPayment` with your own work:++```typescript+import { metrics } from "@opentelemetry/api";++const meter = metrics.getMeter("my-hono-app");++const ordersProcessed = meter.createCounter("orders.processed", {+ description: "Orders processed",+});++const checkoutLatency = meter.createHistogram("checkout.latency", {+ description: "Checkout latency",+ unit: "ms",+});++app.post("/api/checkout", async (c) => {+ const started = Date.now();+ await processPayment();+ ordersProcessed.add(1, { plan: "pro" });+ checkoutLatency.record(Date.now() - started, { outcome: "ok" });+ return c.json({ status: "ok" });+});+```++Tag keys become filterable dimensions in Traceway. Counters and gauges are stored as they are. A histogram arrives as two metrics, `checkout.latency.avg` and `checkout.latency.count`. `ExponentialHistogram` and `Summary` instruments are dropped, so prefer a plain histogram.++`@hono/otel` also emits `http.server.request.duration` (tagged with `http.route` and `http.response.status_code`) and `http.server.active_requests` for free.++## What Gets Captured++With `@hono/otel` and the auto-instrumentations enabled, the following is captured automatically:++| Layer | What's Captured |+|-------|-----------------|+| `@hono/otel` middleware | Incoming requests with method, route, status code, duration |+| `@hono/otel` middleware | `http.route` for endpoint grouping (for example `GET /api/users/:id`) |+| `@hono/otel` middleware | Thrown errors, recorded as exception events and shown as Issues |+| `@hono/otel` middleware | `http.server.request.duration` and `http.server.active_requests` metrics |+| Database (`pg`, `mysql2`, `mongodb`) | Query spans with SQL statements, connection info |+| Cache (`redis`, `ioredis`) | Cache operation spans |+| Outgoing global `fetch()` | HTTP client spans, via `instrumentation-undici` |+| Outgoing `axios`, `got`, `node-fetch` v2 | **Not captured** with the config above. See [Tracing Outgoing HTTP Clients](#tracing-outgoing-http-clients) |+| DNS and TCP | `dns.lookup` and `tcp.connect` spans |++### How Traceway Classifies Your Spans++| Span shape | Where it lands |+|-----------|----------------|+| Root span with `http.route` or `url.path`, which is what `@hono/otel` emits | **Endpoints**, named `METHOD /route` from `http.route` |+| Any span with `kind: CONSUMER` | **Tasks**, named after the span |+| Any non-root span | **Spans**, in the trace waterfall |+| An `exception` event on any span, from `span.recordException()` or a throw caught by `@hono/otel` | **Issues**, with the stack trace |+| A root span with no HTTP attributes and a kind other than CONSUMER | Dropped. Nothing is stored |++A request that matched no Hono route has no `http.route`, so all such 404s group into a single endpoint named `UNMATCHED` instead of one row per bad URL. That is deliberate. It keeps scanners and typo traffic out of your endpoint list.++### Database Auto-Instrumentation++Database libraries like `pg`, `mysql2`, `mongodb`, and `ioredis` are CommonJS packages. Even in an ESM Hono app, OTel's `require-in-the-middle` hook patches them successfully. Database queries appear as child spans under the `@hono/otel` root span with no manual instrumentation:++```typescript+import { Hono } from "hono";+import { httpInstrumentationMiddleware } from "@hono/otel";+import pg from "pg";++const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });+const app = new Hono();++app.use(httpInstrumentationMiddleware());++app.get("/api/users/:id", async (c) => {+ const id = c.req.param("id");+ // This query creates a child span named after the SQL, with+ // db.system.name=postgresql and db.query.text set.+ const { rows } = await pool.query("SELECT * FROM users WHERE id = $1", [id]);+ return c.json(rows[0]);+});+```++The resulting trace in Traceway:++```+GET /api/users/:id ← Endpoint (from @hono/otel)+ ├─ pg-pool.connect ← Span (only when the pool opens a connection)+ │ └─ pg.connect+ │ ├─ dns.lookup ← Span (from instrumentation-dns)+ │ └─ tcp.connect ← Span (from instrumentation-net)+ └─ SELECT * FROM users WHERE id = $1 ← Span (from instrumentation-pg, named after the SQL)+```++There is no span called `pg.query`. `instrumentation-pg` names the query span after the SQL text. The `pg.connect`, `dns.lookup`, and `tcp.connect` spans only show up on the request that opens a new pooled connection. Later requests to the same route show `pg-pool.connect` and the query span, with nothing under `pg-pool.connect`.++> **Note:** This works because `pg`, `mysql2`, and similar packages use CommonJS internally, and the CommonJS require hook is always active. ESM imports are a different story, which is what [Tracing Outgoing HTTP Clients](#tracing-outgoing-http-clients) is about.++## Tracing Outgoing HTTP Clients++`axios`, `got`, `node-fetch` v2, and many vendor SDKs send requests through Node's `http` module, which the setup above deliberately leaves unpatched. To trace those calls while `@hono/otel` keeps ownership of the incoming request span, register OTel's ESM loader hook and disable only the incoming half of `instrumentation-http`.++Install the hook package as a direct dependency:++```bash+npm install @opentelemetry/instrumentation+```++Then make two changes to `instrumentation.ts`. Add the loader hook at the top of the file, and swap `enabled: false` for `disableIncomingRequestInstrumentation: true`:++```typescript+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);++import { NodeSDK } from "@opentelemetry/sdk-node";+// ...the rest of the imports, unchanged++const sdk = new NodeSDK({+ // ...serviceName, traceExporter, metricReaders, logRecordProcessors, unchanged++ instrumentations: [+ getNodeAutoInstrumentations({+ // @hono/otel keeps the incoming request span, the one that knows http.route.+ // instrumentation-http now only traces outgoing calls.+ "@opentelemetry/instrumentation-http": {+ disableIncomingRequestInstrumentation: true,+ },+ }),+ ],+});+```++The hook has to be registered in the preloaded file, which is why it goes in `instrumentation.ts` and not in `server.ts`. With this config, both `fetch()` and `http.get()` produce child spans, endpoints are still named from `http.route`, and there is no duplicate root span. On Node 26 you will see a `module.register() is deprecated` warning at startup. It is harmless and the hook still works.++A CommonJS app loaded with `node --require ./instrumentation.js` does not need the loader hook at all, because the CommonJS require hook patches `http` on its own. It still needs `disableIncomingRequestInstrumentation: true`, otherwise `instrumentation-http` takes the root span and endpoint grouping breaks.++## Environment Variable Alternative++If you would rather not hardcode the endpoint and token, the SDK reads them from the environment. `OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL, without a signal path. The SDK appends `/v1/traces` and `/v1/metrics` itself:++```bash+export OTEL_SERVICE_NAME="my-hono-app"+export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+```++With those set you can drop `serviceName`, `traceExporter`, and `metricReaders` from the SDK config. Keep the `logRecordProcessors` block, the `instrumentations` block, and the shutdown handler.++## Multi-Runtime Notes++| Runtime | Instrumentation |+|---------|----------------|+| **Node.js** | OTel Node SDK (this guide) |+| **Cloudflare Workers** | Platform OTLP export, no SDK (Workers Paid, beta). Cloudflare does not emit `http.route`, so endpoints are named by raw URL path instead of your Hono route pattern. Read the [Cloudflare guide](/client/otel/cloudflare) before you switch |+| **Deno** | Use [Deno OTel](https://docs.deno.com/runtime/fundamentals/open_telemetry/) with OTLP/HTTP export to Traceway |+| **Bun** | Use `@opentelemetry/sdk-node` (Bun supports most Node.js OTel packages) |++## Next Steps++- [Traces](/client/otel/hono/traces): manual spans, middleware instrumentation, and exception recording+- [Node.js OTel Guide](/client/otel/nodejs): general Node.js setup details+- [Logs](/client/otel/logs): severity levels, attributes, and how logs are stored+- [Metrics](/client/otel/metrics): instrument types and how Traceway stores them+- [OTel Overview](/client/otel): protocol details and authenticationdiff --git a/docs/pages/client/hono/traces.mdx b/docs/pages/client/otel/hono/traces.mdxsimilarity index 52%rename from docs/pages/client/hono/traces.mdxrename to docs/pages/client/otel/hono/traces.mdxindex 7e5bc36a..79dbfe3c 100644--- a/docs/pages/client/hono/traces.mdx+++ b/docs/pages/client/otel/hono/traces.mdx@@ -2,9 +2,11 @@The `@hono/otel` middleware automatically traces all HTTP requests. Database queries from CJS libraries (`pg`, `mysql2`, `mongodb`, `ioredis`) are also auto-instrumented. This page covers manual spans for operations that aren't covered by auto-instrumentation.+Everything here assumes the setup from the [Quick Start](/client/otel/hono), including `app.use(httpInstrumentationMiddleware())`.+## Manual Spans-Use `@opentelemetry/api` to create custom spans for operations that aren't auto-instrumented. These spans automatically become children of the `@hono/otel` root span:+Use `@opentelemetry/api` to create custom spans for operations that aren't auto-instrumented. These spans automatically become children of the `@hono/otel` root span. Replace `findOrder` with your own lookup:```typescriptimport { trace, SpanStatusCode } from "@opentelemetry/api";@@ -17,7 +19,7 @@ app.get("/api/orders/:id", async (c) => {return tracer.startActiveSpan("fetch-order", async (span) => {try {span.setAttribute("order.id", orderId);- const order = await db.orders.findById(orderId);+ const order = await findOrder(orderId);if (!order) {span.setStatus({ code: SpanStatusCode.ERROR, message: "Order not found" });@@ -28,15 +30,18 @@ app.get("/api/orders/:id", async (c) => {span.end();return c.json(order);} catch (error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ const e = error as Error;+ span.recordException(e);+ span.setStatus({ code: SpanStatusCode.ERROR, message: e.message });span.end();- throw error;+ throw e;}});});```+TypeScript binds `catch (error)` to `unknown`, so the `error as Error` cast is what makes `recordException` and `.message` compile. The same cast is used in every snippet below.+## Tracing MiddlewareCreate a Hono middleware that adds attributes to the current span:@@ -47,7 +52,7 @@ import { createMiddleware } from "hono/factory";const addUserContext = createMiddleware(async (c, next) => {const span = trace.getActiveSpan();- const userId = c.get("userId");+ const userId = c.req.header("x-user-id");if (span && userId) {span.setAttribute("user.id", userId);@@ -59,11 +64,13 @@ const addUserContext = createMiddleware(async (c, next) => {app.use("/api/*", addUserContext);```+The attribute lands on the endpoint's root span, so `user.id` shows up in the attributes panel of every request that route handled.+## Exception Recording-The `@hono/otel` middleware automatically records thrown errors as exception events on the span — they appear as **Issues** in Traceway with full stack traces.+The `@hono/otel` middleware records thrown errors as exception events on the span, so they appear as **Issues** in Traceway with full stack traces. Hono's default error handler answers the request with HTTP 500, and the middleware writes `http.response.status_code` 500 on the endpoint row to match.-For cases where you catch an error and want to record it without re-throwing:+For cases where you catch an error and want to record it without re-throwing. Replace `processPayment` with your own work:```typescriptimport { trace, SpanStatusCode } from "@opentelemetry/api";@@ -73,9 +80,10 @@ app.get("/api/checkout", async (c) => {try {await processPayment();} catch (error) {+ const e = error as Error;if (span) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ span.recordException(e);+ span.setStatus({ code: SpanStatusCode.ERROR, message: e.message });}return c.json({ error: "Payment failed" }, 500);}@@ -83,6 +91,8 @@ app.get("/api/checkout", async (c) => {});```+A caught error recorded this way still becomes its own Issue, with the stack trace from where it was thrown.+## Nested SpansSpans created inside an active span are automatically linked as children:@@ -97,7 +107,7 @@ async function processPayment(orderId: string) {await chargeAmount(orderId);span.setStatus({ code: SpanStatusCode.OK });} catch (error) {- span.recordException(error);+ span.recordException(error as Error);span.setStatus({ code: SpanStatusCode.ERROR });throw error;} finally {@@ -107,14 +117,18 @@ async function processPayment(orderId: string) {}```+## Background Jobs++A cron job or queue consumer runs outside any request, so `@hono/otel` never sees it and the span you create is a root span. Root spans need `kind: SpanKind.CONSUMER` or Traceway drops them. The full example is in the [Quick Start](/client/otel/hono#background-jobs).+## Outgoing HTTP Requests (Auto-Instrumented)-Outgoing `fetch()` calls are automatically traced by `@opentelemetry/instrumentation-undici` — no manual spans needed:+Calls through the global `fetch()` are traced by `@opentelemetry/instrumentation-undici`, with no manual spans:```typescriptapp.get("/api/external", async (c) => {- // This fetch automatically creates a child span with:- // url.full, http.response.status_code, server.address, etc.+ // This fetch creates a child span with url.full,+ // http.response.status_code, server.address, and more.const res = await fetch("https://api.example.com/data");const data = await res.json();return c.json(data);@@ -130,16 +144,26 @@ GET /api/external ← Endpoint (from @hono/otel)http.response.status_code = 200```+Opening a new connection adds `dns.lookup` and `tcp.connect` spans underneath.++This covers the global `fetch()` only. `axios`, `got`, `node-fetch` v2, and vendor SDKs built on Node's `http` module produce no span at all with the Quick Start config, because it turns `instrumentation-http` off. To trace those too, see [Tracing Outgoing HTTP Clients](/client/otel/hono#tracing-outgoing-http-clients).+## SQLite / Custom Database Spans (Manual)-SQLite has no OTel auto-instrumentation. Wrap queries in manual spans:+SQLite has no OTel auto-instrumentation. Wrap queries in manual spans. Install the driver first:++```bash+npm install better-sqlite3+``````typescriptimport { trace, SpanStatusCode } from "@opentelemetry/api";+import Database from "better-sqlite3";+const db = new Database("app.db");const tracer = trace.getTracer("my-hono-app");-function dbSpan(name, query, fn) {+function dbSpan<T>(name: string, query: string, fn: () => T): T {return tracer.startActiveSpan(name, (span) => {span.setAttribute("db.system", "sqlite");span.setAttribute("db.statement", query);@@ -148,10 +172,11 @@ function dbSpan(name, query, fn) {span.end();return result;} catch (error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ const e = error as Error;+ span.recordException(e);+ span.setStatus({ code: SpanStatusCode.ERROR, message: e.message });span.end();- throw error;+ throw e;}});}@@ -164,21 +189,26 @@ app.get("/api/users", (c) => {});```+Traceway renames any span that carries `db.statement` (or `db.query.text`) after the SQL itself. The snippet sets `db.statement`, so the child span under `GET /api/users` shows as `SELECT * FROM users`, not as `db.query`. That is the same naming you get from `pg` and `mysql2`. If you would rather see your own span name in the waterfall, drop the `db.statement` line.+## What Gets Auto-Instrumented vs Manual| Operation | Auto-Instrumented? | Notes ||-----------|-------------------|-------|| HTTP requests (incoming) | Yes | Via `@hono/otel` middleware |-| Outgoing `fetch()` calls | Yes | Via `instrumentation-undici` |-| Database queries (`pg`, `mysql2`, `mongodb`) | Yes | CJS packages — patched by `require-in-the-middle` |-| Cache (`redis`, `ioredis`) | Yes | CJS packages — patched automatically |+| Outgoing global `fetch()` calls | Yes | Via `instrumentation-undici` |+| Outgoing `axios`, `got`, `node-fetch` v2 | No | They use the `http` module, which the Quick Start config disables. [Re-enable outgoing only](/client/otel/hono#tracing-outgoing-http-clients) |+| Database queries (`pg`, `mysql2`, `mongodb`) | Yes | CJS packages, patched by `require-in-the-middle` |+| Cache (`redis`, `ioredis`) | Yes | CJS packages, patched automatically || DNS lookups | Yes | Via `instrumentation-dns` || TCP connections | Yes | Via `instrumentation-net` || SQLite (`better-sqlite3`) | No | Use manual spans (see above) |+| Background jobs and cron | No | Use `tracer.startActiveSpan()` with `kind: SpanKind.CONSUMER` || Custom business logic | No | Use `tracer.startActiveSpan()` |## Next Steps-- [Quick Start](/client/hono) — setup and installation-- [Node.js Traces](/client/node-sdk/traces) — more on manual spans and context propagation-- [Node.js Metrics](/client/node-sdk/metrics) — custom metrics with OTel+- [Quick Start](/client/otel/hono): setup, logs, background jobs, and custom metrics+- [Node.js Traces](/client/otel/nodejs/traces): more on manual spans and context propagation+- [Logs](/client/otel/logs): severity levels and attributes+- [Metrics](/client/otel/metrics): instrument types and how Traceway stores themdiff --git a/docs/pages/client/otel/index.mdx b/docs/pages/client/otel/index.mdxindex a13a7320..99a0a55f 100644--- a/docs/pages/client/otel/index.mdx+++ b/docs/pages/client/otel/index.mdx@@ -1,14 +1,30 @@# OpenTelemetry Integration-Send traces, metrics, and logs from any OpenTelemetry-instrumented application to Traceway — no vendor SDK required.+**OpenTelemetry is the integration path for every Traceway backend.** Go, Node.js, Python, PHP, Java, .NET, Ruby: whatever your server runs on, it exports traces, metrics, and logs over OTLP/HTTP to Traceway. There is no separate vendor SDK to learn per framework, and no separate Traceway project per backend service.[OpenTelemetry](https://opentelemetry.io/) (OTel) is the industry-standard, vendor-neutral framework for collecting telemetry. If your app already uses OTel, you can point it at Traceway with a few lines of config. If you're starting fresh, any OTel SDK will work.+Create the Traceway project with framework **OpenTelemetry**, then send your API, background workers, scheduled jobs, AI calls, and host metrics to that one project. See [Project Structure](/learn/projects).+## How It Works1. **Instrument your app** with an OpenTelemetry SDK (or auto-instrumentation).-2. **Export via OTLP/HTTP** — configure the SDK (or an OTel Collector) to send data to your Traceway instance.-3. **Traceway maps the data** — spans become endpoints and traces, metrics appear on your dashboards, and logs are indexed and linked to their originating traces.+2. **Export via OTLP/HTTP**: configure the SDK (or an OTel Collector) to send data to your Traceway instance.+3. **Traceway maps the data**: spans become endpoints and traces, metrics appear on your dashboards, and logs are indexed and linked to their originating traces.++### Three signals, three exporters++OTel treats traces, metrics and logs as three independent signals. Each one needs its own exporter wired into the SDK. Configuring a trace exporter does **not** start sending logs, and auto-instrumentation packages do not add a log exporter for you. This is the most common reason the Logs page stays empty.++| Signal | Path | What you must add | Without it |+|--------|------|-------------------|------------|+| Traces | `/v1/traces` | a trace exporter (`traceExporter` / `BatchSpanProcessor`) | No endpoints, tasks, spans or issues |+| Metrics | `/v1/metrics` | a metric reader (`PeriodicExportingMetricReader`) | Empty dashboard widgets |+| Logs | `/v1/logs` | a log record processor **and** a bridge from your logging library | Empty Logs page |++Logs need one extra step the other two do not. The OTel log exporter only ships records emitted through the OTel logs API, so you also need a bridge from whatever you actually log with: `LoggingHandler` for Python's `logging`, an `slog` handler for Go, a Monolog handler for PHP, the built-in appenders for the Java agent. A plain `console.log` or `print` sends nothing.++All three signals go to the same host and use the same project token. See [Logs](/client/otel/logs) for the per-language wiring.## Supported Languages@@ -16,15 +32,17 @@ Any language with an OTel SDK can export to Traceway. Here are the most common o| Language | OTel SDK | Install ||----------|----------|---------|-| Java | [opentelemetry-java](https://github.com/open-telemetry/opentelemetry-java) | [Agent JAR](https://opentelemetry.io/docs/zero-code/java/agent/) |-| Python | [opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) | `pip install opentelemetry-sdk` |-| Python / Django | [opentelemetry-python-contrib](https://github.com/open-telemetry/opentelemetry-python-contrib) | `pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-django` — see [Django guide](/client/django) |-| C# / .NET | [opentelemetry-dotnet](https://github.com/open-telemetry/opentelemetry-dotnet) | `dotnet add package OpenTelemetry` |-| Go | [opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go) | `go get go.opentelemetry.io/otel` |-| Node.js | [opentelemetry-js](https://github.com/open-telemetry/opentelemetry-js) | `npm install @opentelemetry/sdk-node` — see [Node.js guide](/client/node-sdk), [NestJS guide](/client/nestjs), [Hono guide](/client/hono), [Next.js guide](/client/nextjs) |-| PHP / Symfony | [opentelemetry-php](https://github.com/open-telemetry/opentelemetry-php) | `composer require open-telemetry/sdk` — see [Symfony guide](/client/symfony) |-| PHP / Laravel | [opentelemetry-php](https://github.com/open-telemetry/opentelemetry-php) | `composer require keepsuit/laravel-opentelemetry` — see [Laravel guide](/client/laravel) |-| Cloudflare Workers | [workers-sdk](https://developers.cloudflare.com/workers/observability/otel/) | Built-in — see [Cloudflare guide](/client/otel-cloudflare) |+| Java | [opentelemetry-java](https://github.com/open-telemetry/opentelemetry-java) | [Agent JAR](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar), 2.x (see the Spring Boot quick start below) |+| Python | [opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) | `pip install opentelemetry-distro opentelemetry-exporter-otlp`, then `opentelemetry-bootstrap -a install`, then run under `opentelemetry-instrument`. See [Python guide](/client/otel/python) for FastAPI, Flask, and any WSGI/ASGI app |+| Python / Django | [opentelemetry-python-contrib](https://github.com/open-telemetry/opentelemetry-python-contrib) | `pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-django`. See [Django guide](/client/otel/django) |+| C# / .NET | [opentelemetry-dotnet](https://github.com/open-telemetry/opentelemetry-dotnet) | `dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol` (pulls in `OpenTelemetry`) |+| Go | [opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go) | `go get go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` |+| Node.js | [opentelemetry-js](https://github.com/open-telemetry/opentelemetry-js) | `npm install @opentelemetry/sdk-node`. See [Node.js guide](/client/otel/nodejs), [NestJS guide](/client/otel/nestjs), [Hono guide](/client/otel/hono), [Next.js guide](/client/otel/nextjs) |+| PHP / Symfony | [opentelemetry-php](https://github.com/open-telemetry/opentelemetry-php) | `composer require open-telemetry/sdk`. See [Symfony guide](/client/otel/symfony) |+| PHP / Laravel | [opentelemetry-php](https://github.com/open-telemetry/opentelemetry-php) | `composer require keepsuit/laravel-opentelemetry`. See [Laravel guide](/client/otel/laravel) |+| Cloudflare Workers | [Workers OTel export](https://developers.cloudflare.com/workers/observability/exporting-opentelemetry-data/) | Built-in, traces and logs only, Workers Paid. See [Cloudflare guide](/client/otel/cloudflare) |++Every OTel ecosystem ships the API, the SDK and the OTLP exporter as separate packages. If you install only the SDK, you get an app that records spans and throws them away. The install commands above cover the exporter too.## Configuration@@ -35,61 +53,220 @@ Any language with an OTel SDK can export to Traceway. Here are the most common o| **Metrics path** | `/v1/metrics` || **Logs path** | `/v1/logs` || **Auth header** | `Authorization: Bearer <project_token>` |-| **Protocol** | OTLP/HTTP (Protobuf or JSON) |+| **Protocol** | OTLP/HTTP only, Protobuf or JSON. **gRPC is not supported**, so `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` is required || **Compression** | Gzip supported (`Content-Encoding: gzip`) |-| **Max body size** | 10 MB |+| **Content-Type** | `application/x-protobuf` (or `application/protobuf`) for OTLP/Protobuf. Anything else, including a missing header, is parsed as OTLP/JSON |+| **Max body size** | 10 MB **after decompression**. Gzip does not raise it |+| **Auth failure** | `401` with an empty body. The header value must literally start with `Bearer ` |++Traceway serves OTLP over HTTP only. There is no gRPC endpoint, so set your exporter's protocol to `http/protobuf` (or `http/json`) explicitly. Several SDKs default to gRPC when the variable is unset: with `OTEL_TRACES_EXPORTER=otlp` and no `OTEL_EXPORTER_OTLP_PROTOCOL`, the Python SDK resolves to gRPC, and the Java agent does the same. The failure is silent. The app runs, exits 0, prints no warning, and every export lands nowhere.++```bash+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+```++Set that wherever you configure the exporter. The one exception is PHP, which needs `http/json` instead.++An export larger than 10 MB is truncated and then fails to parse, so Traceway answers `400 Bad Request` with a decoding error. OTLP exporters treat 4xx as permanent and drop the batch instead of retrying, so that data is lost. Keep batches comfortably under the limit by capping the exporter's batch size: `maxExportBatchSize` in the JS SDK, `max_export_batch_size` in Python, `send_batch_max_size` on the Collector's `batch` processor. The SDK default of 512 spans per export is far below the limit.++If ingest is saturated, Traceway answers `503` with a `Retry-After` header instead. That one is retryable, and every OTLP exporter and the Collector will back off and resend.## Quick Start: Direct SDK Export-The simplest setup — your app exports directly to Traceway with no extra infrastructure.+The simplest setup: your app exports directly to Traceway, with no extra infrastructure.++Here is a complete Node.js setup that wires all three signals. Install the packages first:++```bash+npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \+ @opentelemetry/instrumentation @opentelemetry/api \+ @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http \+ @opentelemetry/exporter-logs-otlp-http @opentelemetry/sdk-metrics \+ @opentelemetry/sdk-logs @opentelemetry/resources @opentelemetry/api-logs+```++`@opentelemetry/instrumentation` is listed because the loader hook below is loaded by path from it, and `@opentelemetry/api` because your own code imports it. Both also arrive as transitive dependencies of the packages above, so npm's flat `node_modules` resolves them even when they are not declared. Declare them anyway, since your code names them directly.-Here's a Node.js example:+Then create `telemetry.mjs`:```typescript+// telemetry.mjs+// Load it before your app: node --import ./telemetry.mjs server.js+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);++import { NodeSDK } from "@opentelemetry/sdk-node";import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";+import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";+import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";+import { resourceFromAttributes } from "@opentelemetry/resources";+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";-const traceExporter = new OTLPTraceExporter({- url: "https://your-traceway-instance.com/api/otel/v1/traces",- headers: {- Authorization: "Bearer your-project-token",- },+const BASE = "https://your-traceway-instance.com/api/otel";+const headers = { Authorization: "Bearer your-project-token" };++const sdk = new NodeSDK({+ resource: resourceFromAttributes({+ "service.name": "my-service",+ "service.version": "1.0.0",+ }),+ traceExporter: new OTLPTraceExporter({ url: `${BASE}/v1/traces`, headers }),+ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({ url: `${BASE}/v1/metrics`, headers }),+ exportIntervalMillis: 30000,+ }),+ ],+ logRecordProcessors: [+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({ url: `${BASE}/v1/logs`, headers }),+ }),+ ],+ instrumentations: [getNodeAutoInstrumentations()],});-const metricExporter = new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: {- Authorization: "Bearer your-project-token",+sdk.start();+```++Start your app with `node --import ./telemetry.mjs server.js`. Loading the file first is what lets auto-instrumentation patch your HTTP server and database clients before they are imported.++Four details that break this silently if you get them wrong:++- The two `node:module` lines at the top are required for an **ESM** app (`.mjs`, or `"type": "module"`). Without the hook, no framework is patched, `http.route` is never set, and your endpoints are named after raw span names. A CommonJS app (`require`, `.cjs`) does not need them.+- `resourceFromAttributes` replaced the old `new Resource(...)`, which was removed in `@opentelemetry/resources` 2.x. The old form throws at import time.+- `BatchLogRecordProcessor` takes an **options object**, not a bare exporter. `new BatchLogRecordProcessor(exporter)` constructs fine and then exports nothing, with no error.+- `metricReaders` is the current option. The singular `metricReader` still works but is deprecated.++The same pattern applies to any language: set the OTLP/HTTP endpoint and the `Authorization` header in your exporter config, once per signal.++## End to End: One Request, Its Spans, and Its Logs++Traces and logs are separate signals with separate exporters, but they meet again in the dashboard. A log emitted inside an active span carries that span's `trace_id`, and the Endpoint detail page uses that id to show the request's log lines next to its waterfall.++Here is the request side, using the `telemetry.mjs` above. Auto-instrumentation produces spans like these for you; they are written out by hand so the correlation is visible.++```typescript+import { trace, SpanKind } from "@opentelemetry/api";+import { logs, SeverityNumber } from "@opentelemetry/api-logs";++const tracer = trace.getTracer("shop");+const logger = logs.getLogger("shop");++await tracer.startActiveSpan(+ "GET /orders/:id",+ {+ kind: SpanKind.SERVER,+ attributes: { "http.request.method": "GET", "http.route": "/orders/:id" },},-});+ async (serverSpan) => {+ logger.emit({+ severityNumber: SeverityNumber.INFO,+ severityText: "INFO",+ body: "handling order lookup",+ attributes: { "order.id": "ord_123" },+ });++ await tracer.startActiveSpan(+ "SELECT orders",+ {+ kind: SpanKind.CLIENT,+ attributes: {+ "db.system": "postgresql",+ "db.query.text": "SELECT * FROM orders WHERE id = $1",+ },+ },+ async (dbSpan) => {+ // ... run the query ...+ dbSpan.end();+ },+ );++ serverSpan.setAttribute("http.response.status_code", 200);+ serverSpan.end();+ },+);```-The same pattern applies to any language — set the OTLP/HTTP endpoint and the `Authorization` header in your exporter config.+What lands in Traceway:++| What you wrote | Where it shows up |+|----------------|-------------------|+| The root SERVER span with `http.route` | **Endpoints**, as `GET /orders/:id`, with its duration and status |+| The child CLIENT span | The **Spans** waterfall on that endpoint's detail page, labelled with the SQL from `db.query.text` |+| The `logger.emit` call | The **Logs** card on the same page, **This Trace** tab |+| Anything you `recordException` on either span | **Issues**, linked back to this endpoint |++The glue is that the log record carries the root span's `trace_id`, and a root span's Endpoint row is keyed by that same trace id.++Note the `SpanKind.SERVER` on the outer span. With the default `INTERNAL` kind and no HTTP attributes, the span is dropped and none of this appears. See [Traces](/client/otel/traces) for the full classification rules.## Quick Start: Spring Boot (Java Agent)-The [OpenTelemetry Java agent](https://opentelemetry.io/docs/zero-code/java/agent/) instruments Spring Boot applications with zero code changes. Download the agent JAR and pass it to the JVM:+The [OpenTelemetry Java agent](https://opentelemetry.io/docs/zero-code/java/agent/) instruments Spring Boot applications with zero code changes. Download [`opentelemetry-javaagent.jar`](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar) from the latest release, use the **2.x** agent, and pass it to the JVM:```bashjava \-javaagent:opentelemetry-javaagent.jar \-Dotel.service.name=my-spring-app \+ -Dotel.exporter.otlp.protocol=http/protobuf \-Dotel.exporter.otlp.endpoint=https://<your-instance>/api/otel \-Dotel.exporter.otlp.headers="Authorization=Bearer <project_token>" \- -Dotel.metrics.exporter=none \- -Dotel.logs.exporter=none \-jar target/my-app.jar```-Setting `OTEL_EXPORTER_OTLP_ENDPOINT` to `…/api/otel` works too — the agent appends `/v1/traces` automatically, which matches Traceway's ingest path.+`-Dotel.exporter.otlp.endpoint` takes the **base** URL. The agent appends `/v1/traces`, `/v1/metrics` and `/v1/logs` itself, which is exactly how Traceway's ingest paths are laid out. (The per-signal settings such as `otel.exporter.otlp.traces.endpoint` are used as-is, so those need the full path.) `-Dotel.service.name` becomes the **Server Name** on every endpoint, task and issue.++Set `-Dotel.exporter.otlp.protocol=http/protobuf` explicitly even though it is the default on the 2.x agent. Traceway speaks OTLP/HTTP only. A 1.x agent JAR, or any environment that already exports `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`, will send telemetry to a port that does not exist, and nothing in the dashboard will tell you.++**All three signals are exported by this command.** `otel.traces.exporter`, `otel.metrics.exporter` and `otel.logs.exporter` all default to `otlp`, so JVM and HTTP server metrics land on your dashboards and your Logback or Log4j output lands in **Logs**, with no extra flags and no appender to register. If you deliberately want traces only, add `-Dotel.metrics.exporter=none -Dotel.logs.exporter=none`.++Leave `otel.exporter.otlp.metrics.default.histogram.aggregation` at its default `EXPLICIT_BUCKET_HISTOGRAM`. Traceway reads explicit-bucket histograms and exposes each one as `<name>.avg` and `<name>.count`. Base-2 exponential histograms are not read and are dropped without an error, so `http.server.request.duration` and the JVM latency histograms would silently disappear from your dashboards.++In a container or a Kubernetes manifest, use the environment-variable form instead. Same settings, uppercased, with `.` and `-` replaced by `_`:++```bash+JAVA_TOOL_OPTIONS=-javaagent:/app/opentelemetry-javaagent.jar+OTEL_SERVICE_NAME=my-spring-app+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-instance>/api/otel+OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <project_token>+```++Do not quote the `OTEL_EXPORTER_OTLP_HEADERS` value in a Kubernetes manifest. The quotes become part of the header value and the request comes back `401`.++What lands where, with the command above:++| Spring Boot | Traceway |+|-------------|----------|+| Controller requests (`@GetMapping("/orders/{id}")`) | **Endpoints**, named `GET /orders/{id}`, because the Spring Web MVC instrumentation sets `http.route` |+| `@Scheduled` jobs, `@Async` work, queue consumers | **Tasks** |+| JDBC, HTTP client and Redis calls inside a request | Child spans on the trace detail page |+| Uncaught controller exceptions | **Issues**, with the full JVM stack trace |+| Logback / Log4j output | **Logs**, correlated to the trace that produced it |+| JVM and HTTP server metrics (`jvm.memory.used`, `http.server.request.duration`) | **Metrics** and dashboard widgets. Histograms appear as `<name>.avg` and `<name>.count` |-Exceptions thrown in your controllers are automatically captured as span events and appear in **Issues**, grouped by exception class across deployments regardless of differing error messages or line numbers.+Exceptions that reach the servlet container are recorded as span events and show up in **Issues**. Traceway groups them by the exception class **plus the normalized frame list**. The message is stripped and Java, Kotlin and Scala line numbers are removed, so one failure keeps a single issue across redeploys and across differing error messages. A different call site, or a frame added or removed in the same path, is a different issue by design.++Two caveats worth knowing. The message is stripped only from the first line of the header, and only when the exception class name contains no `$`. Exceptions from nested classes, and exceptions whose message spans several lines, can therefore split into one issue per distinct message.## Quick Start: OTel Collector-If you already run an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) or want a central pipeline that fans out to multiple backends, you can route data through it. **This is optional** — the direct SDK export above works without a Collector.+If you already run an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) or want a central pipeline that fans out to multiple backends, you can route data through it. **This is optional.** The direct SDK export above works without a Collector.```yaml+receivers:+ otlp:+ protocols:+ grpc:+ endpoint: 0.0.0.0:4317+ http:+ endpoint: 0.0.0.0:4318++processors:+ batch:+exporters:otlphttp:endpoint: "https://your-traceway-instance.com/api/otel"@@ -99,13 +276,69 @@ exporters:service:pipelines:traces:+ receivers: [otlp]+ processors: [batch]exporters: [otlphttp]metrics:+ receivers: [otlp]+ processors: [batch]exporters: [otlphttp]logs:+ receivers: [otlp]+ processors: [batch]exporters: [otlphttp]```+Every pipeline must declare at least one receiver. A pipeline with only an `exporters:` line fails validation with `must have at least one receiver` and the Collector never starts.++The `otlphttp` exporter appends `/v1/traces`, `/v1/metrics` and `/v1/logs` to the base `endpoint`, so point it at `/api/otel` and nothing else. On recent Collector builds the component has been renamed to `otlp_http`, with `otlphttp` kept as a deprecated alias. Both work today. Use `otlp_http` if your build warns about the alias.++Your apps then send to the Collector on `localhost:4318` (OTLP/HTTP) or `localhost:4317` (OTLP/gRPC), and the Collector is the only thing that needs your Traceway token.++## Nothing Is Showing Up++Traceway accepts telemetry it cannot use rather than rejecting the whole batch, so a misconfiguration usually looks like silence rather than an error. Work down this list.++**Check the wire first.** Post an empty span batch and watch the status code:++```bash+curl -i -X POST https://<your-instance>/api/otel/v1/traces \+ -H 'Content-Type: application/json' \+ -H 'Authorization: Bearer <project_token>' \+ -d '{"resourceSpans":[]}'+```++- `200` with the body `{}` means the endpoint and token are good. The problem is in your SDK, or in how your spans are shaped.+- `401` with an empty body means a bad token, or a header that does not start with `Bearer `.+- `503` with `Retry-After` means ingest is saturated. Exporters retry this automatically.+- `404` means the path under `/api` is wrong, for example `/api/v1/traces` instead of `/api/otel/v1/traces`.+- `200` with an HTML body means the `/api` prefix is missing. The dashboard is served from the same origin, so its single-page fallback answers `/otel/v1/traces` with the dashboard itself and it looks like a success. Check that the response `Content-Type` is `application/json`, not `text/html`. The base is `/api/otel`, not `/otel`.++**Then check the shape.**++| Symptom | Cause | Fix |+|---------|-------|-----|+| Background job or cron never appears on Tasks | Its root span uses the default `SpanKind.INTERNAL`, so it is discarded | Set `kind: SpanKind.CONSUMER`, or add a `console.command` attribute for CLI commands |+| Endpoint is named after the span, not the route | No `http.request.method` or `http.method` on the span. Without a method the route is ignored entirely | Set the method attribute alongside `http.route` |+| One endpoint row per URL, thousands of them | Only `url.path` is set, so the concrete URL becomes the name | Set `http.route` to the low-cardinality template, `/users/:id` |+| Requests appear as `UNMATCHED` | They are 404s with no matched route, collapsed on purpose | Expected. A matched route returning 404 keeps its name |+| The same route is listed twice with a **Mixed** chip | A batch processor split the parent and child spans across two exports, so the child was promoted to its own endpoint | Raise `maxExportBatchSize` / `scheduledDelayMillis`, or stop emitting the redundant sub-handler span |+| Nothing at all from an ESM Node app | The `@opentelemetry/instrumentation/hook.mjs` loader hook is missing, so nothing was patched | Add the two `node:module` lines shown above |+| Logs page is empty | No log exporter is wired (a trace exporter does not send logs), or there is no bridge from your logging library into the OTel logs API | See [Logs](/client/otel/logs) |+| Logs exist but the endpoint's Logs card is empty | The endpoint was promoted from a non-root span, so it is keyed by span id and not by the trace id the logs carry | Search the trace id on the Logs page |+| A whole metric family is missing | It is an ExponentialHistogram or a Summary. Both are dropped without an error | Switch to explicit-bucket histograms, or convert in the Collector |+| Metrics arrive but every series looks identical | The distinguishing attribute is on the Resource and is not in the allowlist | Emit it as a data-point attribute instead. See [Metrics](/client/otel/metrics) |+| Large exports vanish and the exporter logs a 400 | The batch exceeded 10 MB after decompression, was truncated, and failed to parse | Lower the exporter's batch size |++**Turn on the SDK's own diagnostics.** Most silent client-side failures, including a mis-constructed exporter or processor, only surface here:++```typescript+import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";+diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);+```++The equivalents are `OTEL_LOG_LEVEL=debug` for Python and the Java agent, and `otel.SetErrorHandler` for Go.+## Connection PageYour Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.@@ -114,6 +347,12 @@ Your Traceway dashboard includes a **Connection** page with a ready-made config## Next Steps-- [Traces](/client/otel/traces) — how OTel spans map to Traceway concepts-- [Metrics](/client/otel/metrics) — supported metric types and histogram handling-- [Logs](/client/otel/logs) — export OTel logs and link them to your traces+- [Traces](/client/otel/traces): how OTel spans map to Traceway concepts+- [Metrics](/client/otel/metrics): supported metric types and histogram handling+- [Logs](/client/otel/logs): export OTel logs and link them to your traces++Framework guides, all on this same OTLP path:++- [Node.js](/client/otel/nodejs) · [NestJS](/client/otel/nestjs) · [Next.js](/client/otel/nextjs) · [Hono](/client/otel/hono) · [Cloudflare Workers](/client/otel/cloudflare)+- [Python](/client/otel/python) · [Django](/client/otel/django)+- [Symfony](/client/otel/symfony) · [Laravel](/client/otel/laravel)diff --git a/docs/pages/client/laravel/_meta.json b/docs/pages/client/otel/laravel/_meta.jsonsimilarity index 100%rename from docs/pages/client/laravel/_meta.jsonrename to docs/pages/client/otel/laravel/_meta.jsondiff --git a/docs/pages/client/laravel/exceptions.mdx b/docs/pages/client/otel/laravel/exceptions.mdxsimilarity index 68%rename from docs/pages/client/laravel/exceptions.mdxrename to docs/pages/client/otel/laravel/exceptions.mdxindex 36e83d9d..f24b402c 100644--- a/docs/pages/client/laravel/exceptions.mdx+++ b/docs/pages/client/otel/laravel/exceptions.mdx@@ -4,8 +4,8 @@- Any exception thrown from a **controller / route closure** is captured on the request span (which becomes a Traceway Issue).- Any exception thrown from a **queued job** that triggers a `JobFailed` event is captured on the consumer span by `QueueInstrumentation`.-- Any exception thrown inside `Tracer::newSpan(...)->measure(fn)` is captured by `measure()` (see [Spans](/client/laravel/spans) for one caveat about span status).-- Any exception that propagates through Laravel's `report()` pipeline — including ones you catch and rethrow via `throw $e` — also goes through the reportable callback and is recorded on whatever span is active at that moment.+- Any exception thrown inside `Tracer::newSpan(...)->measure(fn)` is captured by `measure()` (see [Spans](/client/otel/laravel/spans) for one caveat about span status).+- Any exception that propagates through Laravel's `report()` pipeline (including ones you catch and rethrow via `throw $e`) also goes through the reportable callback and is recorded on whatever span is active at that moment.You only need the patterns on this page when:1. You **catch** an exception and don't rethrow it, so Laravel's reporter never sees it.@@ -13,10 +13,6 @@ You only need the patterns on this page when:## Recording an Exception on the Current Span-When you catch an error but don't rethrow, record it manually:--## Recording an Exception on the Current Span-When you catch an error but want it reported to Traceway, record it as an event on the active span:```php@@ -65,9 +61,11 @@ The additional attributes passed to `recordException` are attached to the except## Adding Context to the Auto-Capture-The package's built-in `reportable()` callback runs **before** any callback you register, but you can register your own to add request-level attributes to the span before the exception event is finalized. Use this when you want every error to carry the same diagnostic fields:+Register your own `reportable()` callback to add request-level attributes to the span. Use this when you want every error to carry the same diagnostic fields.-### Laravel 11 / 12+Your callback and the package's both run through Laravel's `reportable()` pipeline. On Laravel 11 and newer a callback registered in `bootstrap/app.php` actually runs *before* the package's, because `withExceptions()` is wired up while the application is being built, ahead of any service provider. The order does not matter either way: both act on the same active span, so attributes you set are attached whether they land before or after `recordException`.++### Laravel 11, 12 and 13```php// bootstrap/app.php@@ -81,8 +79,8 @@ return Application::configure(basePath: dirname(__DIR__))->withRouting(/* ... */)->withExceptions(function (Exceptions $exceptions) {$exceptions->report(function (\Throwable $e) {- // The package already calls recordException + setStatus(ERROR) on- // the active span — here we just attach extra attributes.+ // The package also calls recordException + setStatus(ERROR) on this+ // span. Here we just attach extra attributes.$span = Span::getCurrent();$span->setAttribute('request.id', request()->header('X-Request-Id'));$span->setAttribute('user.id', optional(auth()->user())->id);@@ -91,7 +89,9 @@ return Application::configure(basePath: dirname(__DIR__))->create();```-### Laravel 10+### Laravel 10 (package 1.x)++Version 2.x of the package dropped Laravel 10. If you are still on the 1.x line, the same callback goes in the exception handler class:```php// app/Exceptions/Handler.php@@ -100,8 +100,8 @@ use OpenTelemetry\API\Trace\Span;public function register(): void{$this->reportable(function (\Throwable $e) {- // The package already calls recordException + setStatus(ERROR) on- // the active span — here we just attach extra attributes.+ // The package also calls recordException + setStatus(ERROR) on this+ // span. Here we just attach extra attributes.$span = Span::getCurrent();$span->setAttribute('request.id', request()->header('X-Request-Id'));$span->setAttribute('user.id', optional(auth()->user())->id);@@ -141,7 +141,7 @@ class PaymentService## Custom Exception Types-Custom exception classes work the same way — their type name appears in the Traceway dashboard:+Custom exception classes work the same way. Their type name appears in the Traceway dashboard:```phpclass InsufficientFundsException extends \RuntimeException@@ -167,3 +167,11 @@ try {$span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());}```++**Keep the message constant on namespaced classes.** Traceway groups Issues by a normalized stack trace, and it strips the message only when the part before the first colon looks like a type name (letters, digits, dots and underscores). A real app class arrives as `App\Exceptions\InsufficientFundsException`, whose backslashes are not recognised, so the message stays in the hash and every distinct message becomes its own Issue. Build the message from fixed text and keep the varying values on the span, where they are still visible per occurrence:++```php+parent::__construct('Insufficient funds');+```++The `amount.requested` and `amount.available` attributes above already carry the numbers, so nothing is lost.diff --git a/docs/pages/client/otel/laravel/index.mdx b/docs/pages/client/otel/laravel/index.mdxnew file mode 100644index 00000000..f7261a85--- /dev/null+++ b/docs/pages/client/otel/laravel/index.mdx@@ -0,0 +1,219 @@+# Laravel++The [`keepsuit/laravel-opentelemetry`](https://github.com/keepsuit/laravel-opentelemetry) package automatically instruments your Laravel application and exports traces, metrics, and logs to Traceway's OTLP endpoints. No manual instrumentation is needed.++## Prerequisites++- PHP 8.2 or newer. Laravel 13 itself requires PHP 8.3+.+- Laravel 11.31+, 12, or 13. Version 2.x of the package dropped Laravel 10, so a Laravel 10 app resolves to the older 1.x line, which has a different file layout and config shape than this guide describes.+- Composer+- A Traceway project (create one in the dashboard)++No PECL extension is required. `ext-opentelemetry` is only needed by `ScoutInstrumentation`.++## Step 1: Install Packages++```bash+composer require keepsuit/laravel-opentelemetry open-telemetry/exporter-otlp php-http/guzzle7-adapter+```++The `open-telemetry/exporter-otlp` and `php-http/guzzle7-adapter` packages provide the OTLP/HTTP exporter and its HTTP client. They are required at runtime to actually ship spans to Traceway.++## Step 2: Publish the Config++```bash+php artisan vendor:publish \+ --provider="Keepsuit\LaravelOpenTelemetry\LaravelOpenTelemetryServiceProvider" \+ --tag="opentelemetry-config"+```++This creates `config/opentelemetry.php` where you can disable individual instrumentations, configure sampling, set propagators, and tune exporter settings.++## Step 3: Configure Environment Variables++Add the following to your `.env` file, replacing the endpoint and token with your project values:++```bash+OTEL_SERVICE_NAME=my-laravel-app+OTEL_TRACES_EXPORTER=otlp+OTEL_METRICS_EXPORTER=otlp+OTEL_LOGS_EXPORTER=otlp+OTEL_EXPORTER_OTLP_PROTOCOL=http/json+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"++# Logs need one more line. OTEL_LOGS_EXPORTER only wires up the exporter.+# Nothing writes to the package's auto-injected `otlp` log channel until you+# point Laravel's logger at it.+LOG_CHANNEL=stack+LOG_STACK=single,otlp+```++Your Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.++> **Protocol:** Traceway accepts both `http/protobuf` (the package default) and `http/json`. Protobuf payloads are smaller and slightly faster, `http/json` is easier to inspect through a proxy. gRPC is **not** supported. Traceway is OTLP/HTTP only, with a 10 MB limit per request body.++That's it for traces and metrics. The package's service provider auto-registers `Keepsuit\LaravelOpenTelemetry\Instrumentation\Support\Http\Server\TraceRequestMiddleware` as a global middleware, so every inbound HTTP request is traced without any manual middleware registration.++Logs are the one signal that needs the extra step above. On Laravel 11 and newer the shipped `stack` channel reads `LOG_STACK`, so `LOG_STACK=single,otlp` keeps writing `storage/logs/laravel.log` and ships a copy to Traceway. Set `LOG_CHANNEL=otlp` instead if you only want Traceway. Full details on [Logs](/client/otel/laravel/logs).++## Tune Which Instrumentations Are Enabled++The package's instrumentation list lives in the `instrumentation` array of `config/opentelemetry.php`. Each entry can be removed (to disable it) or toggled per-environment via the matching env var. This is what version 2.x publishes, so you can compare it against your own file:++```php+// config/opentelemetry.php+use Keepsuit\LaravelOpenTelemetry\Instrumentation;++return [+ // ...+ 'instrumentation' => [+ Instrumentation\HttpServerInstrumentation::class => [+ 'enabled' => filter_var(env('OTEL_INSTRUMENTATION_HTTP_SERVER', true), FILTER_VALIDATE_BOOLEAN),+ 'excluded_paths' => [], // e.g. ['/health', '/up'] to skip health checks+ 'excluded_methods' => [],+ 'allowed_headers' => [],+ 'sensitive_headers' => [],+ 'sensitive_query_parameters' => [],+ ],++ Instrumentation\HttpClientInstrumentation::class => [+ 'enabled' => filter_var(env('OTEL_INSTRUMENTATION_HTTP_CLIENT', true), FILTER_VALIDATE_BOOLEAN),+ 'manual' => false,+ 'allowed_headers' => [],+ 'sensitive_headers' => [],+ 'sensitive_query_parameters' => [],+ ],++ Instrumentation\QueryInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_QUERY', true), FILTER_VALIDATE_BOOLEAN),+ Instrumentation\RedisInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_REDIS', true), FILTER_VALIDATE_BOOLEAN),+ Instrumentation\QueueInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_QUEUE', true), FILTER_VALIDATE_BOOLEAN),+ Instrumentation\CacheInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_CACHE', true), FILTER_VALIDATE_BOOLEAN),++ Instrumentation\EventInstrumentation::class => [+ 'enabled' => filter_var(env('OTEL_INSTRUMENTATION_EVENT', true), FILTER_VALIDATE_BOOLEAN),+ 'excluded' => [],+ ],++ Instrumentation\ViewInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_VIEW', true), FILTER_VALIDATE_BOOLEAN),+ Instrumentation\LivewireInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_LIVEWIRE', true), FILTER_VALIDATE_BOOLEAN),++ Instrumentation\ConsoleInstrumentation::class => [+ 'enabled' => filter_var(env('OTEL_INSTRUMENTATION_CONSOLE', true), FILTER_VALIDATE_BOOLEAN),+ 'commands' => [],+ ],++ Instrumentation\ScoutInstrumentation::class => filter_var(env('OTEL_INSTRUMENTATION_SCOUT', true), FILTER_VALIDATE_BOOLEAN),+ ],+];+```++Note that `excluded_paths` really is empty by default, so Laravel's own `/up` health route shows up as a Traceway endpoint until you exclude it.++| Instrumentation | What it captures |+|-----------------|------------------|+| `HttpServerInstrumentation` | Inbound HTTP requests, status codes, route templates (`http.route`) |+| `HttpClientInstrumentation` | Outbound `Http::` / Guzzle calls |+| `QueryInstrumentation` | Eloquent / DB queries with SQL and timing |+| `RedisInstrumentation` | Redis commands |+| `QueueInstrumentation` | Queued jobs: `PRODUCER` span at dispatch, `CONSUMER` span at execute (see [Tasks](/client/otel/laravel/tasks)) |+| `CacheInstrumentation` | Cache hits / misses / writes (as span events) |+| `EventInstrumentation` | Application events, added to the active span as `event fired` span events. Framework events (`Illuminate\*`, Octane, Horizon, Scout) are skipped. |+| `ViewInstrumentation` | Blade view rendering |+| `LivewireInstrumentation` | Livewire component renders |+| `ConsoleInstrumentation` | Artisan commands (**opt-in per command**, see [Tasks](/client/otel/laravel/tasks)) |+| `ScoutInstrumentation` | Laravel Scout (requires the `opentelemetry` PHP extension) |++Disable any instrumentation you don't want by removing it from the array or setting the matching env var to `false`. Every enabled instrumentation adds a small amount of overhead.++> **Exceptions:** there is no separate "exception" instrumentation. The package's service provider hooks Laravel's exception handler via `reportable()` and calls `$span->recordException($e)` + `setStatus(ERROR)` on the active span for every reported exception. Queue failures are captured by `QueueInstrumentation` on the `JobFailed` event. See [Exceptions](/client/otel/laravel/exceptions) for adding extra context to caught errors.++## What Gets Captured++Once configured, the package automatically captures:++- **Endpoints**: every HTTP request with method and route template (e.g., `GET /users/{id}`)+- **Status codes**: 2xx, 4xx, 5xx responses+- **Exceptions**: recorded on the request span with stack traces+- **Database queries**: Eloquent / DB facade queries with statement and duration+- **Outbound HTTP**: `Http::` facade and Guzzle calls+- **Cache & Redis**: keys, hits, misses+- **Queued jobs**: `PRODUCER` + `CONSUMER` spans (see [Tasks](/client/otel/laravel/tasks))+- **Logs**: every `Log::` call, linked to the active trace and span. This one needs the `LOG_CHANNEL` / `LOG_STACK` lines from Step 3.+- **User context**: the authenticated user's id is added as `user.id` to every span and log (configurable via `opentelemetry.user_context`)++## User Context++When `opentelemetry.user_context` is `true` (the default), the authenticated user's id is attached as `user.id` to every span and log. You can customize the attributes:++```php+// app/Providers/AppServiceProvider.php+use Illuminate\Contracts\Auth\Authenticatable;+use Keepsuit\LaravelOpenTelemetry\Facades\OpenTelemetry;++public function boot(): void+{+ OpenTelemetry::user(function (Authenticatable $user) {+ return [+ 'user.id' => $user->getAuthIdentifier(),+ 'user.email' => $user->email,+ ];+ });+}+```++## Flushing: Short-Lived Processes and Long-Running Workers++Spans, metrics and logs are batched, not sent one by one. Where that batch gets flushed depends on how the process runs.++**Short-lived processes** (a php-fpm request, a one-off `php artisan …` run) flush on process shutdown. Telemetry lands in Traceway a couple of seconds after the process exits, so poll the dashboard rather than judging it while the command is still running. Killing the process before it exits cleanly (`Ctrl-C`, `kill -9`, a container `SIGKILL`) loses the pending batch.++**Long-running workers** are detected automatically (Octane, Horizon, default queue workers). There the default is plain batching with **no timer for spans**. A finished job's spans leave the worker only when a later job ends a span, or when the worker itself shuts down cleanly. A single job on a worker that then sits idle stays unexported for as long as that worker keeps running, so waiting does not help. To flush after every iteration (request or job) instead:++```bash+OTEL_WORKER_MODE_FLUSH_AFTER_EACH_ITERATION=true+```++This trades a small amount of throughput for guaranteed delivery of every span. Set it on any low-traffic worker, and set it while you are testing the integration, otherwise one test job looks like it never arrived.++When `flush_after_each_iteration` is off, `OTEL_WORKER_MODE_COLLECT_INTERVAL` (default `60`, in seconds) controls how often the worker collects **metrics**. It does not affect span export.++## Test Your Integration++Add a test route to verify data is flowing:++```php+// routes/web.php+use Illuminate\Support\Facades\Route;++Route::get('/testing', function () {+ throw new \RuntimeException('Test error from Traceway integration');+});+```++Visit `/testing` in your browser, then check the Traceway dashboard. The exception should appear within a few seconds.++> **Heads-up on the error page.** With `APP_DEBUG=true`, Laravel renders its own exception page through Blade, so `ViewInstrumentation` records a few hundred `view render` spans on every 500. A request that would otherwise have four child spans ends up with roughly 275. The Issue and the 500 status are still recorded correctly, only the waterfall is noisy. It goes away with `APP_DEBUG=false`, or you can turn view spans off with `OTEL_INSTRUMENTATION_VIEW=false`.++## Troubleshooting++Every failure mode here is silent, so work down this table before assuming data is being lost.++| Symptom | Cause | Fix |+|---|---|---|+| Nothing at all in the dashboard | Wrong endpoint or token | `OTEL_EXPORTER_OTLP_ENDPOINT` must be the **base** URL ending in `/api/otel`. The SDK appends `/v1/traces`, `/v1/metrics` and `/v1/logs` itself. The header format is `OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <token>"`. Export failures are printed to stderr as `OpenTelemetry: [error] Export failure …`, so check your server or worker output. |+| Traces arrive but logs do not | Laravel's logger is not pointed at the `otlp` channel | Set `LOG_CHANNEL=stack` plus `LOG_STACK=single,otlp`, or `LOG_CHANNEL=otlp`. See [Logs](/client/otel/laravel/logs). |+| A cron job or hand-rolled background task is missing entirely | Its root span is `INTERNAL` | Traceway drops root `INTERNAL` spans and everything under them. Start the span with `->setSpanKind(SpanKind::KIND_CONSUMER)`, or run the work as an Artisan command listed in `ConsoleInstrumentation`. See [Tasks](/client/otel/laravel/tasks). |+| Queued jobs never appear under Tasks | No worker is consuming them, or the worker has not flushed | Run `php artisan queue:work` (or Horizon), and set `OTEL_WORKER_MODE_FLUSH_AFTER_EACH_ITERATION=true` before you start it. Without that flag an idle worker holds the job's spans until the next job runs or the worker stops. With `QUEUE_CONNECTION=sync` the job runs inline instead and lands as `process sync` inside the dispatching request's trace. |+| An Artisan command produces no span | Not listed in `ConsoleInstrumentation` | Add it to the `commands` array. Unlisted commands get a no-op span, so `Span::getCurrent()` calls inside them are silently discarded. |+| Data appears seconds late | Batching | Expected. See [Flushing](#flushing-short-lived-processes-and-long-running-workers) above. |+| gRPC connection refused | Not supported | Traceway is OTLP/HTTP only. Use `http/protobuf` or `http/json`. |++## Next Steps++- [Exceptions](/client/otel/laravel/exceptions): manually capture caught exceptions with context+- [Spans](/client/otel/laravel/spans): create custom spans to measure sub-operations+- [Tasks](/client/otel/laravel/tasks): trace Laravel queue jobs and console commands as background tasks+- [Metrics](/client/otel/laravel/metrics): track custom counters, histograms, and gauges with the `Meter` facade+- [Logs](/client/otel/laravel/logs): forward Laravel logs to Traceway via the auto-injected `otlp` channel+- [OpenTelemetry docs](/client/otel): how OTel traces, metrics, and logs map to Tracewaydiff --git a/docs/pages/client/laravel/logs.mdx b/docs/pages/client/otel/laravel/logs.mdxsimilarity index 62%rename from docs/pages/client/laravel/logs.mdxrename to docs/pages/client/otel/laravel/logs.mdxindex 2fd41873..77aeaf4e 100644--- a/docs/pages/client/laravel/logs.mdx+++ b/docs/pages/client/otel/laravel/logs.mdx@@ -4,7 +4,7 @@## Step 1: Enable the OTLP Logs Exporter-Extend the env config from the [Quick Start](/client/laravel) to also enable the logs exporter:+Extend the env config from the [Quick Start](/client/otel/laravel) to also enable the logs exporter:```bash# Existing traces + metrics config from the Quick Start@@ -20,49 +20,56 @@ OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otelOTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"```-`OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL — the logs exporter appends `/v1/logs` automatically. No extra endpoint config is needed.+`OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL. The logs exporter appends `/v1/logs` automatically. No extra endpoint config is needed.-## Step 2: Use the `otlp` Channel+## Step 2: Route Laravel's Logger at the `otlp` Channel-The package's service provider **automatically injects** a log channel named `otlp` into `config/logging.php` at runtime — no manual channel registration is required. You just need to point your default log destination at it.+The package's service provider **automatically injects** a log channel named `otlp` into `config/logging.php` at runtime, so no manual channel registration is required. But nothing writes to it by default, so `OTEL_LOGS_EXPORTER=otlp` on its own ships zero logs. You have to point your default log destination at the channel.-The simplest path: set `LOG_CHANNEL=otlp` in your `.env` to send every log directly to Traceway.--If you'd rather keep file logs and ship a copy to Traceway, switch your default channel to a stack:+On Laravel 11 and newer this is env-only, no file edit needed. The shipped `stack` channel is defined as `explode(',', (string) env('LOG_STACK', 'single'))`, so adding `otlp` to `LOG_STACK` is enough:```bash# .envLOG_CHANNEL=stack+LOG_STACK=single,otlp```+That keeps writing `storage/logs/laravel.log` and ships a copy to Traceway. Drop `single` from the list if you do not want local files, or skip the stack entirely with `LOG_CHANNEL=otlp` to send everything straight to Traceway.++If your app has a published `config/logging.php` that hardcodes the stack's channel list instead of reading `LOG_STACK`, add `otlp` to that array:+```php// config/logging.php'channels' => ['stack' => ['driver' => 'stack',- 'channels' => ['single', 'otlp'],+ 'channels' => ['single', 'otlp'], // <- add 'otlp' here'ignore_exceptions' => false,],- 'single' => [- 'driver' => 'single',- 'path' => storage_path('logs/laravel.log'),- 'level' => env('LOG_LEVEL', 'debug'),- ],-- // The 'otlp' channel is auto-injected — but you can override it here if- // you want to change the level threshold or the handler config.- // 'otlp' => [- // 'driver' => 'monolog',- // 'handler' => \Keepsuit\LaravelOpenTelemetry\Support\OpenTelemetryMonologHandler::class,- // 'level' => 'info',- // ],+ // ... leave the other channels alone ...],```-The stack fans every log out to both the local file (`single`) and Traceway (`otlp`). Drop the `single` channel if you do not want local files.+Note that hardcoding the array makes `LOG_STACK` dead, so prefer the env-only form above.++### Changing the Level Threshold++The auto-injected `otlp` channel logs at `debug`. To reduce volume in production, define the channel yourself in `config/logging.php` and raise the level. A channel you define wins over the injected one:++```php+// config/logging.php+'channels' => [+ // ...+ 'otlp' => [+ 'driver' => 'monolog',+ 'handler' => \Keepsuit\LaravelOpenTelemetry\Support\OpenTelemetryMonologHandler::class,+ 'level' => 'info',+ ],+],+```-> The default `otlp` channel level is `debug`. To reduce volume in production, override the channel as shown in the commented block above and raise the level (`info`, `warning`, …). Only records at or above the threshold are forwarded.+Only records at or above the threshold are forwarded.## Emit Logs With Trace Context@@ -100,12 +107,15 @@ Open the endpoint's trace in the Traceway dashboard and the **Logs** tab will sh## Inject `trace_id` Into Non-OTLP Loggers-If you also write logs to `single` / `daily` files (or any non-OTLP channel), the package can inject the active trace id into Laravel's log context so it shows up alongside the message. This is enabled by default for instrumentation-started spans. If you start a root span manually with `Tracer::newSpan(...)->start()`, call:+If you also write logs to `single` / `daily` files (or any non-OTLP channel), the package can inject the active trace id into Laravel's log context so it shows up alongside the message. This is enabled by default for instrumentation-started spans. If you start a root span manually with `Tracer::newSpan(...)->start()`, call `Tracer::updateLogContext()`:```phpuse Keepsuit\LaravelOpenTelemetry\Facades\Tracer;+use OpenTelemetry\API\Trace\SpanKind;-$span = Tracer::newSpan('cron.cleanup')->start();+$span = Tracer::newSpan('cron.cleanup')+ ->setSpanKind(SpanKind::KIND_CONSUMER) // required when this is the root span+ ->start();$scope = $span->activate();Tracer::updateLogContext(); // makes 'trace_id' available to subsequent Log:: calls@@ -120,6 +130,8 @@ try {The context field name is configurable via `opentelemetry.logs.trace_id_field` (default: `trace_id`).+> **Set the span kind when the span is a trace root.** `Tracer::newSpan()` produces an `INTERNAL` span, and Traceway discards a root `INTERNAL` span that has no HTTP attributes and no `console.command`. The span, its children, and the trace linkage for every log emitted inside it are all lost, silently. `setSpanKind(SpanKind::KIND_CONSUMER)` makes it a Task instead. Spans started inside an existing request, job or command are children, so they need nothing extra. See [Tasks](/client/otel/laravel/tasks#scheduled-closures-and-other-background-work).+## Logger Facade (Alternative)For code paths where you want to send a record straight to Traceway without involving Monolog's pipeline, the package also exposes a `Logger` facade. It's particularly handy in low-level contexts where the standard `Log::` facade isn't available:@@ -150,10 +162,12 @@ Laravel uses Monolog under the hood, so its levels are mapped to OpenTelemetry s| `notice` | 10 | INFO || `warning` | 13 | WARN || `error` | 17 | ERROR |-| `critical` | 18 | ERROR |+| `critical` | 18 | FATAL || `alert` | 19 | ERROR || `emergency` | 21 | FATAL |+Traceway resolves the label from `severityText` first and only falls back to `severityNumber`. That is why `critical` renders as FATAL: the text `CRITICAL` maps straight to FATAL, while severity number 18 on its own would land in the ERROR band.+Only records at or above the channel's configured `level` threshold are forwarded.## Test Your Integration@@ -178,6 +192,6 @@ Visit `/log-test`, then open **Logs** in the dashboard. You should see records a## Next Steps-- [Exceptions](/client/laravel/exceptions) — record caught exceptions with context-- [Metrics](/client/laravel/metrics) — custom counters, histograms, and gauges-- [OTel Logs reference](/client/otel/logs) — full OTLP logs mapping and severity details+- [Exceptions](/client/otel/laravel/exceptions): record caught exceptions with context+- [Metrics](/client/otel/laravel/metrics): custom counters, histograms, and gauges+- [OTel Logs reference](/client/otel/logs): full OTLP logs mapping and severity detailsdiff --git a/docs/pages/client/laravel/metrics.mdx b/docs/pages/client/otel/laravel/metrics.mdxsimilarity index 72%rename from docs/pages/client/laravel/metrics.mdxrename to docs/pages/client/otel/laravel/metrics.mdxindex 1410104e..0ff9b657 100644--- a/docs/pages/client/laravel/metrics.mdx+++ b/docs/pages/client/otel/laravel/metrics.mdx@@ -1,10 +1,10 @@# Metrics-The OTel auto-instrumentation captures default request, DB, queue, and outbound-HTTP metrics automatically (e.g., `http.server.request.duration`, `http.client.request.duration`, `db.client.operation.duration`). To track custom application metrics, use the `Meter` facade.+The OTel auto-instrumentation captures default request, DB, Redis, and outbound-HTTP metrics automatically (`http.server.request.duration`, `http.client.request.duration`, `db.client.operation.duration`). Queued jobs are traced as spans but emit no metrics, so build your own queue counters with the `Meter` facade if you want them. The same facade covers any other custom application metric.## The `Meter` Facade-`keepsuit/laravel-opentelemetry` ships a `Meter` facade that gives you direct access to all OTel instrument types. Instruments are cached by name, so calling `Meter::counter('orders.created')` twice returns the same counter — safe to call from anywhere.+`keepsuit/laravel-opentelemetry` ships a `Meter` facade that gives you direct access to all OTel instrument types. Instruments are cached by name, so calling `Meter::counter('orders.created')` twice returns the same counter. It is safe to call from anywhere.```phpuse Keepsuit\LaravelOpenTelemetry\Facades\Meter;@@ -21,7 +21,7 @@ $counter->add(1);| `Meter::upDownCounter(...)` | Counter that can go up or down || `Meter::observableCounter(...)` | Counter whose value is read via callback at export time || `Meter::observableUpDownCounter(...)` | Up/down counter read via callback |-| `Meter::gauge(...)` | Synchronous gauge — record a value at a point in time |+| `Meter::gauge(...)` | Synchronous gauge: record a value at a point in time || `Meter::observableGauge(...)` | Gauge read via callback at export time || `Meter::histogram($name, $unit, $description)` | Distribution of values (durations, sizes) |@@ -66,22 +66,38 @@ Gauges track point-in-time values that can go up or down (queue depth, active co```phpuse Keepsuit\LaravelOpenTelemetry\Facades\Meter;-// Synchronous — record a value when you know it+// Synchronous: record a value when you know it$gauge = Meter::gauge('cache.size_mb', 'MB', 'Cache memory usage');$gauge->record(42.3);-// Observable — value is read by a callback at export time+// Observable: value is read by a callback at export timeMeter::observableGauge('queue.depth', 'items', 'Pending jobs in the default queue')->observe(function ($observer) {$observer->observe(\Queue::size('default'));});```+> **Register observable instruments once, at boot.** The observable instruments take a callback that the SDK invokes at export time. Registering one per request re-adds the callback every iteration in long-lived processes (Octane, Horizon, `queue:work`). Put them in a service provider's `boot()`:+>+> ```php+> // app/Providers/AppServiceProvider.php+> use Keepsuit\LaravelOpenTelemetry\Facades\Meter;+>+> public function boot(): void+> {+> Meter::observableGauge('queue.depth', 'items', 'Pending jobs in the default queue')+> ->observe(fn ($observer) => $observer->observe(\Queue::size('default')));+> }+> ```+>+> The synchronous instruments (`counter`, `upDownCounter`, `gauge`, `histogram`) are cached by name and safe to call from anywhere, including controllers and jobs.+## Batch Observation-When several observable instruments share an expensive data source, use `batchObserve` so the source is only consulted once per export cycle:+When several observable instruments share an expensive data source, use `batchObserve` so the source is only consulted once per export cycle. Like the other observable instruments, register this once at boot:```php+// app/Providers/AppServiceProvider.php, inside boot()use Keepsuit\LaravelOpenTelemetry\Facades\Meter;use OpenTelemetry\API\Metrics\ObserverInterface;@@ -89,7 +105,10 @@ Meter::batchObserve([Meter::observableCounter('usage', description: 'count of items used'),Meter::observableGauge('pressure', description: 'force per unit area'),], function (ObserverInterface $usageObserver, ObserverInterface $pressureObserver): void {- [$usage, $pressure] = expensive_system_call();+ // Swap this line for your own lookup. The point of batchObserve is that it+ // runs once per export cycle instead of once per instrument.+ [$usage, $pressure] = [\Queue::size('default'), memory_get_usage(true) / 1048576];+$usageObserver->observe($usage);$pressureObserver->observe($pressure);});@@ -148,6 +167,7 @@ public function fetchExternalData(string $endpoint): array### Resource Metrics```php+// app/Providers/AppServiceProvider.php, inside boot()use Keepsuit\LaravelOpenTelemetry\Facades\Meter;Meter::observableGauge('redis.connected_clients', 'connections', 'Active Redis connections')diff --git a/docs/pages/client/laravel/spans.mdx b/docs/pages/client/otel/laravel/spans.mdxsimilarity index 75%rename from docs/pages/client/laravel/spans.mdxrename to docs/pages/client/otel/laravel/spans.mdxindex 46e7734a..fc329dd0 100644--- a/docs/pages/client/laravel/spans.mdx+++ b/docs/pages/client/otel/laravel/spans.mdx@@ -30,7 +30,7 @@ class OrderService$total = Tracer::newSpan('order.total')->measure(fn () => $this->calculateTotal());```-> **Exception handling inside `measure`:** if the callback throws, `measure` calls `$span->recordException($e)` and rethrows. It does **not** call `setStatus(STATUS_ERROR)` — that means the exception event is attached to the span but the span's status code stays `UNSET`. Traceway's Issues feed still picks up the exception event, but if you want the span to be marked as failed in the trace view, set the status yourself:+> **Exception handling inside `measure`:** if the callback throws, `measure` calls `$span->recordException($e)` and rethrows. It does **not** call `setStatus(STATUS_ERROR)`. The exception event is attached to the span, but the span's status code stays `UNSET`. Traceway's Issues feed still picks up the exception event, but if you want the span to be marked as failed in the trace view, set the status yourself:>> ```php> use OpenTelemetry\API\Trace\StatusCode;@@ -45,7 +45,7 @@ $total = Tracer::newSpan('order.total')->measure(fn () => $this->calculateTotal(> });> ```>-> Or if the exception goes through Laravel's exception handler (i.e., bubbles up out of `measure`), the package's auto-registered `reportable()` callback sets the status on the still-active span before `measure`'s `finally` ends it — so you usually don't need to do this manually.+> Or if the exception goes through Laravel's exception handler (i.e., bubbles up out of `measure`), the package's auto-registered `reportable()` callback sets the status on the still-active span before `measure`'s `finally` ends it, so you usually don't need to do this manually.### With Attributes@@ -95,6 +95,19 @@ try {Always end spans in a `finally` block so they are closed even if an exception is thrown.+> **If the span is the root of its trace, set the span kind.** The snippet above is safe inside an HTTP request, a queue job or an instrumented Artisan command, where the span is a child of something Traceway already records. Started on its own (a cron closure, a hand-rolled background loop, a non-instrumented command) it is a root `INTERNAL` span, and Traceway discards root `INTERNAL` spans that carry no HTTP attributes and no `console.command`. The span, every child under it, and the trace linkage for any logs emitted inside are dropped with no error.+>+> ```php+> use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;+> use OpenTelemetry\API\Trace\SpanKind;+>+> $span = Tracer::newSpan('long-running-job')+> ->setSpanKind(SpanKind::KIND_CONSUMER)+> ->start();+> ```+>+> `setSpanKind()` takes any `SpanKind::KIND_*` constant and is available on the builder returned by `Tracer::newSpan()`. A `CONSUMER` root lands in **Tasks**. See [Tasks](/client/otel/laravel/tasks#scheduled-closures-and-other-background-work).+## Other Tracer UtilitiesThe `Tracer` facade exposes a few helpers that come in handy when working with custom spans:@@ -105,12 +118,14 @@ use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;Tracer::traceId(); // the active trace id (string)Tracer::activeSpan(); // the currently-active SpanInterfaceTracer::activeScope(); // the currently-active ScopeInterface-Tracer::currentContext(); // the OTel Context object — for advanced use+Tracer::currentContext(); // the OTel Context object, for advanced useTracer::propagationHeaders(); // headers to inject into outbound requests so downstream services join the traceTracer::extractContextFromPropagationHeaders($headers); // build a Context from inbound headersTracer::updateLogContext(); // inject the active trace id into Laravel's log context (only needed if you start the root span manually)```+The builder returned by `Tracer::newSpan()` also has `setSpanKind()`, `setParent()`, `setAttribute()`, `setAttributes()` and `setStartTimestamp()`, each of which returns the builder so calls chain before `start()` or `measure()`.+## Adding AttributesAttach metadata to spans for filtering and debugging:diff --git a/docs/pages/client/otel/laravel/tasks.mdx b/docs/pages/client/otel/laravel/tasks.mdxnew file mode 100644index 00000000..3fc07f2e--- /dev/null+++ b/docs/pages/client/otel/laravel/tasks.mdx@@ -0,0 +1,249 @@+# Tasks++The `QueueInstrumentation` automatically traces Laravel queue jobs. When a job is dispatched, a `PRODUCER` span is created; when the worker picks it up, a `CONSUMER` span runs the handler. W3C trace context is propagated through the job payload so the consumer span links back to the dispatch trace. The full lifecycle is visible as one continuous trace in Traceway, and each consumed job appears as a **task** with duration, status, and any child spans.++## Setup++`QueueInstrumentation` is enabled by default. If it's not already in your `config/opentelemetry.php`:++```php+use Keepsuit\LaravelOpenTelemetry\Instrumentation;++return [+ 'instrumentation' => [+ // ...+ Instrumentation\QueueInstrumentation::class => env('OTEL_INSTRUMENTATION_QUEUE', true),+ ],+];+```++That's it. Every queued job is now traced automatically. The same setup applies whether you use `database`, `redis`, `sqs`, or `beanstalkd` drivers.++A Task only appears once a worker actually consumes the job. Dispatching alone gives you the `send default` producer span under the request that dispatched it, nothing more. Run `php artisan queue:work` (or Horizon) and the `process default` Task shows up.++Add one env var before you start that worker:++```bash+OTEL_WORKER_MODE_FLUSH_AFTER_EACH_ITERATION=true+```++A long-running worker batches its spans and has no timer for them. Without this flag the job you just ran stays inside the worker until a later job runs or the worker stops, so a single test job looks like it was lost. See [Why a One-Off Command Shows Nothing](#why-a-one-off-command-shows-nothing) below.++With `QUEUE_CONNECTION=sync` the job runs inline inside the request instead, so its `process sync` Task is recorded as part of that request's trace rather than as a standalone background task. No worker and no flush flag are involved in that case.++## What Gets Captured++For each queued job the instrumentation creates two spans:++- **Dispatch span**: name `send {queue}` (e.g., `send default`), `PRODUCER` kind, ended when Laravel finishes pushing the payload onto the queue.+- **Process span**: name `process {queue}` (e.g., `process default`), `CONSUMER` kind, ended when the handler returns or fails. On `JobFailed` the span records the exception and is marked `STATUS_ERROR`.++Both spans carry these attributes (using OTel messaging semantic conventions):++| Attribute | Value |+|---|---|+| `messaging.system` | The queue driver: `redis`, `database`, `sqs`, `beanstalkd`, … |+| `messaging.operation.type` | `send` (producer) or `process` (consumer) |+| `messaging.message.id` | Laravel's job UUID |+| `messaging.destination.name` | The queue name |+| `messaging.message.envelope.size` | Payload size in bytes |+| `messaging.message.job_name` | The job class name (e.g., `App\Jobs\SendOrderEmail`) |+| `messaging.message.attempts` | Current attempt number |+| `messaging.message.max_tries` / `max_exceptions` / `retry_until` / `timeout` | Job configuration |++W3C trace context is also injected into the job payload, so the `process` span is linked to the trace where the job was originally dispatched, even if dispatch and execution happen on different services. Any spans you create inside `handle()` automatically nest under the `process` span.++> **Task naming.** The consumer span is named after the *queue*, not the job, so every job running on `default` groups into one Traceway Task row called `process default`. The job class is carried as the `messaging.message.job_name` attribute. If you want one Task row per job class, rename the span at the top of `handle()`:+>+> ```php+> use OpenTelemetry\API\Trace\Span;+>+> public function handle(): void+> {+> Span::getCurrent()->updateName(static::class);+>+> // ... business logic ...+> }+> ```++## Adding Attributes in Handlers++Since the instrumentation activates the consumer span before your `handle()` runs, use `Span::getCurrent()` to add custom attributes:++```php+namespace App\Jobs;++use Illuminate\Contracts\Queue\ShouldQueue;+use Illuminate\Foundation\Queue\Queueable;+use OpenTelemetry\API\Trace\Span;++class SendOrderEmail implements ShouldQueue+{+ use Queueable;++ public function __construct(+ public readonly int $orderId,+ public readonly string $recipient,+ ) {}++ public function handle(): void+ {+ $span = Span::getCurrent();+ $span->setAttribute('order.id', $this->orderId);+ $span->setAttribute('email.to', $this->recipient);++ // ... business logic ...+ }+}+```++Dispatch it the usual way:++```php+SendOrderEmail::dispatch($order->id, $order->email);+```++> **Use the right `Queueable`.** Laravel 11 and newer generate jobs with `Illuminate\Foundation\Queue\Queueable`, which bundles `Dispatchable`, `InteractsWithQueue`, `QueueableByBus` and `SerializesModels`. The older `Illuminate\Bus\Queueable` does not provide the static `dispatch()` method, so a job built on it fails at runtime with `Call to undefined method App\Jobs\SendOrderEmail::dispatch()`.++## Child Spans in Handlers++For jobs with distinct sub-operations, use the `Tracer` facade to create child spans. They automatically nest under the task's consumer span:++```php+namespace App\Jobs;++use Illuminate\Contracts\Queue\ShouldQueue;+use Illuminate\Foundation\Queue\Queueable;+use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;+use OpenTelemetry\API\Trace\Span;++class ProcessPayment implements ShouldQueue+{+ use Queueable;++ public function __construct(+ public readonly int $paymentId,+ public readonly float $amount,+ ) {}++ public function handle(): void+ {+ Tracer::newSpan('validate.payment')->measure(function () {+ Span::getCurrent()->setAttribute('payment.id', $this->paymentId);+ $this->validate();+ });++ Tracer::newSpan('charge.gateway')->measure(function () {+ Span::getCurrent()->setAttribute('payment.amount', $this->amount);+ $this->chargeGateway();+ });+ }+}+```++## Console Commands++Console commands are **not traced by default**. Only commands you opt-in are wrapped in a span. Add them to the `commands` array of `ConsoleInstrumentation`:++```php+// config/opentelemetry.php+use Keepsuit\LaravelOpenTelemetry\Instrumentation;+use App\Console\Commands\CleanupOrphans;++return [+ 'instrumentation' => [+ // ...+ Instrumentation\ConsoleInstrumentation::class => [+ 'enabled' => env('OTEL_INSTRUMENTATION_CONSOLE', true),+ 'commands' => [+ CleanupOrphans::class, // by class name+ 'app:rebuild-search-index', // by signature+ 'reports:*', // wildcard: every command whose name starts with "reports:"+ ],+ ],+ ],+];+```++A trailing `*` is the only way to trace a whole command namespace. Everything before the `*` is matched as a prefix against the command name.++Each listed command is opened as a root span at start and ended on exit, with `console.command` set to the command name and every argument and option recorded as `console.argument.*` / `console.option.*` attributes. Inside `handle()`, use `Span::getCurrent()` to add your own:++```php+namespace App\Console\Commands;++use Illuminate\Console\Command;+use OpenTelemetry\API\Trace\Span;++class CleanupOrphans extends Command+{+ protected $signature = 'cleanup:orphans';++ public function handle(): int+ {+ Span::getCurrent()->setAttribute('cleanup.dry_run', false);++ // ... do work ...++ return self::SUCCESS;+ }+}+```++Unlisted commands run without a span, so `Span::getCurrent()` inside them returns a no-op span. Attribute and event calls are silently dropped, which is safe but not useful.++## Scheduled Closures and Other Background Work++`ConsoleInstrumentation` only wraps Artisan *commands* you list. A scheduled closure (`Schedule::call(...)`) is not a command, and neither is a hand-rolled background loop, so nothing traces them for you.++Starting a span yourself is not enough on its own. `Tracer::newSpan()` creates an `INTERNAL` span, and Traceway records a root span only when it is an HTTP request, a `CONSUMER` span, or carries a `console.command` attribute. A root `INTERNAL` span is discarded on ingest along with every child span under it, with no error anywhere. Any logs emitted inside it still arrive, but they point at a trace that does not exist.++Mark the root span as `CONSUMER` and the work lands in **Tasks**:++```php+// routes/console.php+use Illuminate\Support\Facades\Schedule;+use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;+use OpenTelemetry\API\Trace\Span;+use OpenTelemetry\API\Trace\SpanKind;++Schedule::call(function () {+ $span = Tracer::newSpan('cron.prune-sessions')+ ->setSpanKind(SpanKind::KIND_CONSUMER)+ ->start();+ $scope = $span->activate();++ Tracer::updateLogContext(); // so Log:: calls carry the trace id++ try {+ Span::getCurrent()->setAttribute('cron.dry_run', false);++ // child spans nest under the task+ Tracer::newSpan('prune.expired')->measure(function () {+ // ... the actual work ...+ });+ } finally {+ $scope->detach();+ $span->end();+ }+})->hourly();+```++The alternative is to move the work into a real Artisan command and list it in `ConsoleInstrumentation`. Then the package sets `console.command` for you and the root span is recorded as a Task without any span-kind handling.++Only the **root** span of a trace needs this. Spans you create inside an already-traced request, job or command are children, and children are recorded whatever their kind.++## Why a One-Off Command Shows Nothing++Spans are batched and flushed when the process shuts down, so a short `php artisan …` run reports a second or two after it exits. Give it a moment before concluding the data was dropped. Two things genuinely break the flush:++- Killing the process (`Ctrl-C`, `kill -9`, a container `SIGKILL`). The pending batch goes with it.+- Long-running workers (Octane, Horizon, `queue:work`). Spans there are batched with no timer, so a finished job's spans wait for the next job or for a clean worker shutdown. An idle worker holds them indefinitely.++For low-traffic workers, and whenever you are testing the integration, flush after every job:++```bash+OTEL_WORKER_MODE_FLUSH_AFTER_EACH_ITERATION=true+```++See [Flushing](/client/otel/laravel#flushing-short-lived-processes-and-long-running-workers) in the Quick Start for the full picture.diff --git a/docs/pages/client/otel/logs.mdx b/docs/pages/client/otel/logs.mdxindex 2b6350ae..7d898d24 100644--- a/docs/pages/client/otel/logs.mdx+++ b/docs/pages/client/otel/logs.mdx@@ -1,5 +1,7 @@# Logs+Logs are their own OTLP signal. If you have only configured a trace exporter, no logs are reaching Traceway yet. You need two things: a log record processor pointed at `/v1/logs`, and a bridge from your app's logging library into the OTel logs API. Both are shown below.+Traceway ingests OpenTelemetry logs via the OTLP/HTTP protocol at `POST /api/otel/v1/logs`. Logs emitted inside an active span context are automatically linked to the trace and span that produced them.## Log Record Mapping@@ -13,13 +15,15 @@ Each OTLP `LogRecord` maps to a Traceway log as follows:| `SpanId` (8 bytes) | Span ID | Hex-encoded || `SeverityNumber` (1-24) | Severity Number | Also used to compute Severity Text when text is empty || `SeverityText` | Severity Text | Uppercased; when empty, derived from `SeverityNumber` (see below) |-| `Body` | Body | Strings stored as-is; arrays and maps are JSON-encoded |+| `Body` | Body | Strings as-is; numbers, booleans and bytes stringified; arrays and maps JSON-encoded; a missing body stores an empty string || `Resource.service.name` | Service | Required for the Logs page service filter |-| Resource attributes | Resource Attributes | Stringified; array and map values are dropped at attribute level |+| Resource attributes | Resource Attributes | Strings, numbers and booleans stringified; array, map and bytes values are dropped || Scope name / version | Scope Name / Version | || Scope attributes | Scope Attributes | Same stringification rules as resource attributes || Log record attributes | Log Attributes | Same stringification rules |+The attribute rule applies to span attributes and metric tags too, since all three signals share the same conversion. If you need list or object data to survive, JSON-encode it into a string attribute yourself.+## Severity FallbackWhen an SDK sends a `SeverityNumber` but leaves `SeverityText` blank, Traceway computes the level:@@ -33,37 +37,46 @@ When an SDK sends a `SeverityNumber` but leaves `SeverityText` blank, Traceway c| 17-20 | ERROR || 21+ | FATAL |-A severity number of `0` is treated as unset.+A severity number of `0` means unset, and Traceway stores an **empty** severity text for it. Those rows still show under the default **All levels**, but they carry no level badge and they disappear the moment you pick any level, because the filter compares severity numbers. Always send a `severityNumber`.++If you send a `severityText`, it wins and is uppercased exactly as given, so `"warning"` is stored as `WARNING`. That does not affect filtering. The Logs page level picker and `traceway logs query --min-severity` both compare the severity **number**, never the text. What matters is that the number is right.## Linking Logs to Traces-OTel Logs SDKs read the active span from the context you pass into `Emit` / `logger.log` / equivalent. When a log is emitted inside a request handler, background job, or child span, its `trace_id` and `span_id` are populated automatically — no plumbing on your side.+OTel Logs SDKs read the active span from the context you pass into `Emit` / `logger.log` / equivalent. When a log is emitted inside a request handler, background job, or child span, its `trace_id` and `span_id` are populated automatically. There is no plumbing on your side.++In the Traceway dashboard, the Endpoint / Task / AI Trace detail page carries a **Logs** card. It has a **This Trace** tab listing the logs whose `trace_id` matches this run, and, when the run is part of a distributed trace, an **All Distributed Traces** tab that pulls in logs from the linked runs in other services. Both tabs search a ±1 hour window around the run and show up to 100 rows.-In the Traceway dashboard, the trace detail page shows a **Logs** tab listing every log for that trace, plus an **All Distributed Traces** tab that pulls in logs from linked traces across services.+The match is on the OTel `trace_id`, and only a **root** span's run is keyed by it. A run promoted from a non-root span, which is what a cross-service inbound hop produces, is keyed by its span id instead. Its Logs card comes up empty even though the logs were recorded correctly. Find those logs on the **Logs** page: set the search dropdown to **Trace ID** first, then paste the trace id in. The dropdown defaults to **Message**, which searches the log body and returns nothing for a trace id.## Example: Node.js+```bash+npm install @opentelemetry/sdk-node @opentelemetry/exporter-logs-otlp-http \+ @opentelemetry/sdk-logs @opentelemetry/resources @opentelemetry/api-logs+```+```typescriptimport { NodeSDK } from "@opentelemetry/sdk-node";import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";-import { Resource } from "@opentelemetry/resources";+import { resourceFromAttributes } from "@opentelemetry/resources";import { logs, SeverityNumber } from "@opentelemetry/api-logs";const sdk = new NodeSDK({- resource: new Resource({+ resource: resourceFromAttributes({"service.name": "my-service","service.version": "1.0.0",}),logRecordProcessors: [- new BatchLogRecordProcessor(- new OTLPLogExporter({+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({url: "https://your-traceway-instance.com/api/otel/v1/logs",headers: {Authorization: "Bearer your-project-token",},}),- ),+ }),],});@@ -77,20 +90,69 @@ logger.emit({body: "order received",attributes: { "order.id": "ord_123" },});++await sdk.shutdown(); // short-lived script: flush before exiting```+Keep that final `sdk.shutdown()` in a script that exits right after emitting. The batch processor holds records for up to a second before its first export, so a process that ends sooner sends nothing and reports no error. A long-running service does not need it.++Two details that fail silently if you get them wrong:++- `BatchLogRecordProcessor` takes an **options object**, `{ exporter }`. Passing the exporter positionally, `new BatchLogRecordProcessor(exporter)`, fails to compile in TypeScript with `error TS2741: Property 'exporter' is missing`. In plain JavaScript it constructs without complaint and then exports nothing, and you will not see an error unless you turn on OTel diag logging.+- `resourceFromAttributes` replaced `new Resource(...)`, which was removed in `@opentelemetry/resources` 2.x. The old form throws at import time.++**Do not create a second `NodeSDK`.** If you already have one for traces, add `logRecordProcessors` to that same instance rather than starting another. The [overview page](/client/otel#quick-start-direct-sdk-export) shows one SDK wiring all three signals.+When called inside an OTel-instrumented HTTP handler (or any code that runs under an active span), the emitted log automatically carries the active `trace_id` and `span_id`.-## Example: Python+## Example: Python (zero-code)++`opentelemetry-bootstrap -a install` installs `opentelemetry-instrumentation-logging`, which attaches the OTel handler to the **root** logger. Every `logger.info(...)` in your code, your framework, and your libraries is then forwarded and linked to the active span, with no bridge to write yourself:++```bash+export OTEL_LOGS_EXPORTER=otlp+export OTEL_PYTHON_LOG_CORRELATION=true+export OTEL_PYTHON_LOG_LEVEL=info+``````pythonimport logging-from opentelemetry import trace-from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler-from opentelemetry.sdk._logs.export import BatchLogRecordProcessor++logger = logging.getLogger(__name__)++[email protected]("/users/{user_id}")+def get_user(user_id: str):+ logger.info("looking up user %s", user_id)+ return {"id": user_id}+```++> **`OTEL_PYTHON_LOG_CORRELATION=true` is what makes INFO records arrive.** Python's root logger defaults to `WARNING`, so `logger.info(...)` is filtered out before the OTel handler ever runs, and only WARN and above reach Traceway. That variable calls `logging.basicConfig()`, which sets the root level and restores a console handler, and it stamps `otelTraceID` / `otelSpanID` onto every record. `OTEL_PYTHON_LOG_LEVEL` alone does nothing: it is only read alongside the correlation variable, where it defaults to `info`. The equivalent in code is `logging.basicConfig(level=logging.INFO)` early in your app.++Attaching the OTel handler also **replaces the root handler list**, so without the correlation variable your log lines stop appearing on stdout and go only to Traceway. On a host whose runbook is `docker logs`, that reads as the app having stopped logging.++Do not set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`. The bridge is on by default on current versions, and that variable switches it to the SDK's deprecated handler, which then logs a duplicate-handler warning that is itself ingested as a log record.++The rest of the zero-code setup is in the [Python guide](/client/otel/python).++## Example: Python with OTel SDK++To wire the bridge by hand, install the handler package alongside the SDK and the exporter:++```bash+pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http \+ opentelemetry-instrumentation-logging+```++```python+import logging++from opentelemetry._logs import set_logger_providerfrom opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter+from opentelemetry.instrumentation.logging.handler import LoggingHandler+from opentelemetry.sdk._logs import LoggerProvider+from opentelemetry.sdk._logs.export import BatchLogRecordProcessorfrom opentelemetry.sdk.resources import Resource-from opentelemetry._logs import set_logger_providerresource = Resource.create({"service.name": "my-python-service",@@ -106,24 +168,36 @@ exporter = OTLPLogExporter()provider.add_log_record_processor(BatchLogRecordProcessor(exporter))-handler = LoggingHandler(level=logging.NOTSET, logger_provider=provider)-logging.getLogger().addHandler(handler)-logging.getLogger().setLevel(logging.INFO)+logging.getLogger().addHandler(LoggingHandler(logger_provider=provider))+logging.getLogger().setLevel(logging.INFO) # root defaults to WARNINGlogging.info("order received", extra={"order.id": "ord_123"})++provider.shutdown() # short-lived script: flush before exiting```-The standard `LoggingHandler` bridges Python's `logging` module to OTel. Logs emitted inside a traced request automatically pick up the current trace and span IDs.+`LoggingHandler` is the bridge: it forwards Python's standard `logging` module into OTel. Without it, `logging.info(...)` writes to stdout and nothing is exported. Logs emitted inside a traced request automatically pick up the current trace and span IDs.++Import the handler from `opentelemetry.instrumentation.logging.handler`. The same class in `opentelemetry.sdk._logs` still works but is deprecated, warns on construction, and is slated for removal. Keep the `setLevel(logging.INFO)` line: the root logger's `WARNING` default drops INFO records before the handler sees them here too. A long-running service can skip the final `shutdown()`, since the batch processor exports on its own interval.## Example: Go+```bash+go get go.opentelemetry.io/otel \+ go.opentelemetry.io/otel/log \+ go.opentelemetry.io/otel/sdk/log \+ go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp+```+```goimport ("context""time"+ "go.opentelemetry.io/otel/attribute""go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"otellog "go.opentelemetry.io/otel/log"+ "go.opentelemetry.io/otel/log/global"sdklog "go.opentelemetry.io/otel/sdk/log""go.opentelemetry.io/otel/sdk/resource"semconv "go.opentelemetry.io/otel/semconv/v1.26.0"@@ -150,6 +224,7 @@ func initLogs(ctx context.Context) (*sdklog.LoggerProvider, error) {sdklog.WithExportInterval(2*time.Second))),sdklog.WithResource(res),)+ global.SetLoggerProvider(lp)return lp, nil}@@ -159,18 +234,62 @@ func logInfo(ctx context.Context, logger otellog.Logger, msg string) {rec.SetTimestamp(time.Now())rec.SetSeverity(otellog.SeverityInfo)rec.SetSeverityText("INFO")- rec.SetBody(otellog.StringValue(msg))+ rec.SetBody(attribute.StringValue(msg))logger.Emit(ctx, rec)}```+`Record.SetBody` takes an `attribute.Value`, so the body value comes from `go.opentelemetry.io/otel/attribute`. There is no `log.StringValue`.+When `ctx` carries an active span (for example, the request context inside a Gin or net/http handler instrumented with OTel), the log automatically carries that span's trace and span IDs.+If you log with `log/slog` rather than the OTel API directly, install the bridge instead of writing `Record` values by hand:++```bash+go get go.opentelemetry.io/contrib/bridges/otelslog+```++```go+import "go.opentelemetry.io/contrib/bridges/otelslog"++logger := otelslog.NewLogger("my-service")+logger.InfoContext(ctx, "order received", "order.id", "ord_123")+```++Pass the request `ctx` on every call. That is what carries the trace and span ids.++## Example: Java (Agent)++The Java agent bridges Logback, Log4j2 and `java.util.logging` to OTLP for you. There is no code to write and no appender to register. Logs export by default, since `otel.logs.exporter` is `otlp`, so simply do not disable them:++```bash+java \+ -javaagent:opentelemetry-javaagent.jar \+ -Dotel.service.name=my-spring-app \+ -Dotel.exporter.otlp.protocol=http/protobuf \+ -Dotel.exporter.otlp.endpoint=https://<your-instance>/api/otel \+ -Dotel.exporter.otlp.headers="Authorization=Bearer <project_token>" \+ -jar target/my-app.jar+```++Every `log.info(...)` inside a traced request is exported with that request's trace and span ids, so it lands on the endpoint's Logs card with no extra wiring. If you previously set `-Dotel.logs.exporter=none` to send traces only, removing that flag is the whole change.+## Quick Start: OTel CollectorIf you route telemetry through a Collector, add a `logs` pipeline that targets the same OTLP/HTTP endpoint:```yaml+receivers:+ otlp:+ protocols:+ grpc:+ endpoint: 0.0.0.0:4317+ http:+ endpoint: 0.0.0.0:4318++processors:+ batch:+exporters:otlphttp:endpoint: "https://your-traceway-instance.com/api/otel"@@ -180,15 +299,23 @@ exporters:service:pipelines:traces:+ receivers: [otlp]+ processors: [batch]exporters: [otlphttp]metrics:+ receivers: [otlp]+ processors: [batch]exporters: [otlphttp]logs:+ receivers: [otlp]+ processors: [batch]exporters: [otlphttp]```+Every pipeline must declare at least one receiver, or the Collector fails validation with `must have at least one receiver` and never starts. The `otlphttp` exporter appends `/v1/logs` to the base `endpoint`, so point it at `/api/otel` and nothing else.+## Next Steps-- [Traces](/client/otel/traces) — how OTel spans map to Traceway concepts-- [Metrics](/client/otel/metrics) — supported metric types and histogram handling-- [Overview](/client/otel) — protocol details and authentication+- [Traces](/client/otel/traces): how OTel spans map to Traceway concepts+- [Metrics](/client/otel/metrics): supported metric types and histogram handling+- [Overview](/client/otel): protocol details, authentication, and a "nothing is showing up" checklistdiff --git a/docs/pages/client/otel/metrics.mdx b/docs/pages/client/otel/metrics.mdxindex 5ed02795..1a41b04f 100644--- a/docs/pages/client/otel/metrics.mdx+++ b/docs/pages/client/otel/metrics.mdx@@ -2,6 +2,8 @@Traceway ingests OpenTelemetry metrics via the OTLP/HTTP protocol at `POST /api/otel/v1/metrics`.+Metrics are their own OTLP signal. A trace exporter does not send them, so the SDK needs a metric reader of its own. See [Three signals, three exporters](/client/otel#three-signals-three-exporters).+## Supported Metric Types| OTel Type | Supported | Traceway Handling |@@ -9,8 +11,10 @@ Traceway ingests OpenTelemetry metrics via the OTLP/HTTP protocol at `POST /api/| Gauge | Yes | Stored as-is || Sum | Yes | Stored as-is || Histogram | Yes | Computes average and count (see below) |-| ExponentialHistogram | No | — |-| Summary | No | — |+| ExponentialHistogram | No | Dropped |+| Summary | No | Dropped |++> **ExponentialHistogram and Summary are dropped without an error.** Traceway answers `200 OK` and stores nothing. The metric never shows up in dashboards or in metric discovery, so the only symptom is an empty widget. If a metric is missing, check its aggregation first. Switch the SDK's histogram aggregation to explicit-bucket (`Aggregation.Explicit`, `explicit_bucket_histogram`, or `EXPLICIT_BUCKET_HISTOGRAM` for the Java agent), or convert in a Collector before exporting. Summary is a Prometheus-only legacy type; export the same data as a Gauge or a Histogram instead.## Histogram Handling@@ -22,17 +26,39 @@ Histograms are converted into two separate metric records:| Count | `{name}.count` | Total number of observations |For example, a histogram named `http.request.duration` produces:-- `http.request.duration.avg` — the average request duration-- `http.request.duration.count` — the total number of requests observed+- `http.request.duration.avg`: the average request duration+- `http.request.duration.count`: the total number of requests observed++`{name}.count` is registered as a counter with unit `count`. `{name}.avg` keeps the histogram's own unit and is registered as a gauge.-Histogram bucket boundaries are not stored.+`{name}.avg` is only emitted for data points that have both a non-zero `count` and a `sum`. A delta histogram with no observations in an interval, or a histogram exported without a sum, produces `{name}.count` alone. An average chart then shows a gap for those intervals rather than a zero.++Histogram bucket boundaries are not stored, so percentiles cannot be computed from a Histogram. Use the Endpoints page, which keeps raw request durations, for P50/P95/P99.## Resource Attributes-The `service.name` resource attribute is used as the Server Name for all metrics in the resource.+`service.name` is copied onto every metric point as the `server_name` tag.++Some receivers put the identity of the thing being measured on the **Resource** rather than on the data point. The hostmetrics process scraper emits one `ResourceMetrics` block per process, and docker_stats, kubeletstats and the postgresql receiver do the same for containers, pods, nodes and databases. Without lifting those, every `process.cpu.utilization` point would look identical. Traceway copies this fixed list onto each point's tags so you can group and filter by them:++| Attribute | Typical source |+|-----------|----------------|+| `process.pid`, `process.executable.name`, `process.command_line`, `process.owner` | hostmetrics process scraper |+| `container.name`, `container.image.name` | docker_stats receiver |+| `k8s.pod.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.deployment.name`, `k8s.container.name` | kubeletstats and k8s receivers |+| `postgresql.database.name` | postgresql receiver |++This is a fixed allowlist, not a passthrough, because each distinct tag value multiplies the number of stored series. Notably `host.name` and `deployment.environment` are **not** lifted, so you cannot group by them. If you need to slice by something outside the list, set it as a data-point attribute in your instrumentation, or add it with the Collector's `attributes` processor, rather than as a resource attribute.++As with all attributes, only strings, numbers and booleans survive. Array and map values are dropped.## Example: Node.js Metric Export+```bash+npm install @opentelemetry/sdk-node @opentelemetry/exporter-metrics-otlp-http \+ @opentelemetry/sdk-metrics @opentelemetry/api+```+```typescriptimport { NodeSDK } from "@opentelemetry/sdk-node";import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";@@ -40,22 +66,45 @@ import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";const sdk = new NodeSDK({serviceName: "my-service",- metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({- url: "https://your-traceway-instance.com/api/otel/v1/metrics",- headers: {- Authorization: "Bearer your-project-token",- },+ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({+ url: "https://your-traceway-instance.com/api/otel/v1/metrics",+ headers: {+ Authorization: "Bearer your-project-token",+ },+ }),+ exportIntervalMillis: 30000,}),- exportIntervalMillis: 30000,- }),+ ],});sdk.start();```+Use `metricReaders` (an array). The singular `metricReader` option still works but is deprecated. If you already have a `NodeSDK` for traces, add `metricReaders` to that instance instead of starting a second SDK. The [overview page](/client/otel#quick-start-direct-sdk-export) shows one SDK wiring all three signals.++Recording a custom metric then looks like this:++```typescript+import { metrics } from "@opentelemetry/api";++const meter = metrics.getMeter("shop");+const ordersPlaced = meter.createCounter("orders.placed", {+ description: "Orders successfully placed",+});++ordersPlaced.add(1, { channel: "web" });+```++The counter is exported on the next 30-second interval and appears in metric discovery as `orders.placed`, with `channel` as a tag you can group by.+## Example: Python Metric Export+```bash+pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http+```+```pythonfrom opentelemetry import metricsfrom opentelemetry.sdk.metrics import MeterProvider@@ -73,9 +122,48 @@ exporter = OTLPMetricExporter(reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30000)provider = MeterProvider(resource=resource, metric_readers=[reader])metrics.set_meter_provider(provider)++meter = metrics.get_meter("shop")++orders = meter.create_counter("orders.placed", unit="1", description="orders placed")+orders.add(1, {"channel": "web"})++checkout = meter.create_histogram("checkout.latency", unit="ms")+checkout.record(12.5, {"channel": "web"})++# Short-lived script: flush before exiting, or the batch never leaves.+provider.force_flush()+provider.shutdown()```+The provider on its own exports nothing. An instrument has to be created and a value recorded, and in a script that exits quickly the flush has to happen before the interpreter tears down. A long-running service can drop the last two lines, since the reader exports on its own interval.++`checkout.latency` arrives as `checkout.latency.avg` and `checkout.latency.count`, per the histogram split above.++> **The SDK's default export interval is 60 seconds.** With `export_interval_millis` left out, or on the zero-code `opentelemetry-instrument` path, a working pipeline still shows nothing for the first minute, which reads as failure. Set `export_interval_millis=10000`, or `OTEL_METRIC_EXPORT_INTERVAL=10000` on the env-var path, while you are verifying the integration.++The zero-code path is simpler: under `opentelemetry-instrument` the meter provider is already configured, so drop everything above and start at `metrics.get_meter(...)`. See the [Python guide](/client/otel/python).++## Example: Java (Agent)++The Java agent exports JVM runtime metrics (heap, GC, thread counts) and HTTP server metrics by default, because `otel.metrics.exporter` is `otlp`. There is nothing to add:++```bash+java \+ -javaagent:opentelemetry-javaagent.jar \+ -Dotel.service.name=my-spring-app \+ -Dotel.exporter.otlp.protocol=http/protobuf \+ -Dotel.exporter.otlp.endpoint=https://<your-instance>/api/otel \+ -Dotel.exporter.otlp.headers="Authorization=Bearer <project_token>" \+ -jar target/my-app.jar+```++Remove any `-Dotel.metrics.exporter=none` flag and the metrics appear in metric discovery, ready to chart.++Leave `otel.exporter.otlp.metrics.default.histogram.aggregation` at its default `EXPLICIT_BUCKET_HISTOGRAM`. Setting it to `BASE2_EXPONENTIAL_BUCKET_HISTOGRAM` makes Traceway drop every histogram silently, which takes `http.server.request.duration` and the JVM latency histograms off your dashboards.+## Next Steps-- [Traces](/client/otel/traces) — how OTel spans map to Traceway concepts-- [Overview](/client/otel) — protocol details and authentication+- [Traces](/client/otel/traces): how OTel spans map to Traceway concepts+- [Logs](/client/otel/logs): export logs and link them to your traces+- [Overview](/client/otel): protocol details, authentication, and a "nothing is showing up" checklistdiff --git a/docs/pages/client/nestjs/_meta.json b/docs/pages/client/otel/nestjs/_meta.jsonsimilarity index 100%rename from docs/pages/client/nestjs/_meta.jsonrename to docs/pages/client/otel/nestjs/_meta.jsondiff --git a/docs/pages/client/otel/nestjs/custom-instrumentation.mdx b/docs/pages/client/otel/nestjs/custom-instrumentation.mdxnew file mode 100644index 00000000..13893341--- /dev/null+++ b/docs/pages/client/otel/nestjs/custom-instrumentation.mdx@@ -0,0 +1,232 @@+# Custom Instrumentation++Add manual spans, custom decorators, and explicit exception capture to your NestJS services using the OpenTelemetry API.++## Setup++This page assumes you already followed the [Quick Start](/client/otel/nestjs), so the SDK is running and `@opentelemetry/api` is installed. If it is not:++```bash+npm install @opentelemetry/api+```++## Manual Spans in Services++Create spans for operations that aren't auto-instrumented. A span started inside a request handler nests under that request's endpoint on its own, with no context to pass around:++```typescript+import { Injectable } from "@nestjs/common";+import { trace, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-nestjs-app");++@Injectable()+export class UsersService {+ async findAll() {+ return tracer.startActiveSpan("db.users.findAll", async (span) => {+ try {+ const users = await this.userRepository.find();+ span.setAttribute("user.count", users.length);+ return users;+ } catch (error: any) {+ span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ span.recordException(error);+ throw error;+ } finally {+ span.end();+ }+ });+ }+}+```++`this.userRepository` is your own repository. Swap in whatever your service already calls.++## Custom @Span Decorator++Create a reusable decorator that wraps methods in OTel spans:++```typescript+import { trace, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-nestjs-app");++export function Span(name?: string): MethodDecorator {+ return (target, propertyKey, descriptor: PropertyDescriptor) => {+ const originalMethod = descriptor.value;+ const spanName = name || `${target.constructor.name}.${String(propertyKey)}`;++ descriptor.value = function (...args: any[]) {+ return tracer.startActiveSpan(spanName, async (span) => {+ try {+ const result = await originalMethod.apply(this, args);+ span.setStatus({ code: SpanStatusCode.OK });+ return result;+ } catch (error: any) {+ span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ span.recordException(error);+ throw error;+ } finally {+ span.end();+ }+ });+ };++ return descriptor;+ };+}+```++Use it on `async` service methods:++```typescript+import { Injectable } from "@nestjs/common";+import { Span } from "./span.decorator";++@Injectable()+export class OrdersService {+ @Span("orders.process")+ async processOrder(orderId: string) {+ // Span is automatically created and ended+ await this.validate(orderId);+ await this.charge(orderId);+ await this.ship(orderId);+ }++ @Span()+ async validate(orderId: string) {+ // Span name defaults to "OrdersService.validate"+ }+}+```++> **This decorator makes every method async.** It always returns the promise from `startActiveSpan`, so putting `@Span()` on a synchronous method changes its return type from `T` to `Promise<T>`. A method that used to return `5` now returns `Promise { <pending> }`, and TypeScript does not warn you, because `descriptor: PropertyDescriptor` erases the signature. Only decorate methods that already return a promise.++### A decorator that also handles synchronous methods++If you want one decorator you can put on anything, branch on the result instead of awaiting it:++```typescript+import { trace, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-nestjs-app");++export function Span(name?: string): MethodDecorator {+ return (target, propertyKey, descriptor: PropertyDescriptor) => {+ const originalMethod = descriptor.value;+ const spanName = name || `${target.constructor.name}.${String(propertyKey)}`;++ descriptor.value = function (...args: any[]) {+ return tracer.startActiveSpan(spanName, (span) => {+ let result: any;++ try {+ result = originalMethod.apply(this, args);+ } catch (error: any) {+ span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ span.recordException(error);+ span.end();+ throw error;+ }++ if (result instanceof Promise) {+ return result.then(+ (value: any) => {+ span.setStatus({ code: SpanStatusCode.OK });+ span.end();+ return value;+ },+ (error: any) => {+ span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ span.recordException(error);+ span.end();+ throw error;+ },+ );+ }++ span.setStatus({ code: SpanStatusCode.OK });+ span.end();+ return result;+ });+ };++ return descriptor;+ };+}+```++A synchronous method keeps returning its plain value, an `async` method still returns a promise, and a throw still propagates in both cases.++## Exception Capture++Record exceptions on the active span:++```typescript+import { Injectable } from "@nestjs/common";+import { trace, SpanStatusCode } from "@opentelemetry/api";++@Injectable()+export class PaymentService {+ async charge(amount: number) {+ const span = trace.getActiveSpan();++ try {+ await this.gateway.charge(amount);+ } catch (error: any) {+ if (span) {+ span.recordException(error);+ span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ }+ throw error;+ }+ }+}+```++Exceptions recorded as span events appear as **Issues** in Traceway with full stack traces.++You rarely need this for uncaught errors. `@opentelemetry/instrumentation-nestjs-core` already records anything thrown out of a controller handler. Use `recordException` when you swallow an error and still want to see it, or when the failure happens outside a request, such as in a background job.++> **A thrown error shows a count of 2.** `instrumentation-nestjs-core` wraps every handler in two spans, one with `nestjs.type=request_context` and one with `nestjs.type=handler`, and both record the exception. That is one request, not two. `HttpException` subclasses like `NotFoundException` are recorded too, so intentional 404s also show up under Issues.++## Span Attributes++Add context to spans for richer trace data:++```typescript+const span = trace.getActiveSpan();+if (span) {+ span.setAttribute("user.id", userId);+ span.setAttribute("order.total", orderTotal);+ span.setAttributes({+ "payment.method": "credit_card",+ "payment.currency": "USD",+ });+}+```++## What Gets Auto-Instrumented vs Manual++| Operation | Auto-Instrumented? | Notes |+|-----------|-------------------|-------|+| HTTP requests (incoming) | Yes | Via `instrumentation-http` + `instrumentation-express` |+| `http.route` grouping | Yes on Express | Fastify needs `@opentelemetry/instrumentation-fastify`, which is not bundled |+| Thrown exceptions | Yes | Recorded as Issues with stack traces, two events per throw |+| Outgoing `fetch()` calls | Yes | Via `instrumentation-undici` |+| Prisma queries | Yes | Via `@prisma/instrumentation` (install separately) |+| Database queries (`pg`, `mysql2`, `mongodb`) | Yes | CJS packages, patched by `require-in-the-middle` |+| Cache (`redis`, `ioredis`) | Yes | CJS packages, patched automatically |+| NestJS handlers | Yes | Via `instrumentation-nestjs-core` |+| Custom business logic | No | Use `tracer.startActiveSpan()` or the `@Span()` decorator |+| SQLite (`better-sqlite3`) | No | Use manual spans |+| Background jobs and cron | No | Needs a root span with `SpanKind.CONSUMER`, see [Background Jobs](/client/otel/nestjs#background-jobs) |+| Logs | No | Needs a bridged logger, see [Logs](/client/otel/nestjs#logs) |++## Next Steps++- [Quick Start](/client/otel/nestjs): setup, logs, background jobs, and custom metrics+- [Node.js Traces](/client/otel/nodejs/traces): more on manual spans and context propagation+- [OTel Metrics](/client/otel/metrics): how OTel instruments map to Traceway metrics+- [OTel Logs](/client/otel/logs): shipping logs and linking them to traces+- [OTel Trace Mapping](/client/otel/traces): how OTel spans map to Traceway conceptsdiff --git a/docs/pages/client/otel/nestjs/index.mdx b/docs/pages/client/otel/nestjs/index.mdxnew file mode 100644index 00000000..f9e0ef48--- /dev/null+++ b/docs/pages/client/otel/nestjs/index.mdx@@ -0,0 +1,553 @@+# NestJS (OpenTelemetry)++Instrument your NestJS application with OpenTelemetry and send traces, metrics, and logs to Traceway. There is no Traceway-specific NestJS module to install. OTel auto-instrumentation does the work.++## Installation++```bash+npm install @opentelemetry/sdk-node \+ @opentelemetry/auto-instrumentations-node \+ @opentelemetry/exporter-trace-otlp-http \+ @opentelemetry/exporter-metrics-otlp-http \+ @opentelemetry/exporter-logs-otlp-http \+ @opentelemetry/sdk-metrics \+ @opentelemetry/sdk-logs \+ @opentelemetry/resources \+ @opentelemetry/instrumentation-express \+ @opentelemetry/api \+ @opentelemetry/api-logs+```++Install every package on that list, even the ones you never import directly. Some of them are pulled in as dependencies of `@opentelemetry/sdk-node`, so on npm the imports happen to resolve anyway. Under pnpm or Yarn PnP they do not, and the versions drift.++If you use Prisma, also install its OTel instrumentation for automatic query tracing:++```bash+npm install @prisma/instrumentation+```++## Setup++### 1. Create `src/instrumentation.ts`++Put the file inside `src/`, not at the project root. A `.ts` file at the root widens the TypeScript `rootDir`, and `nest build` then emits `dist/src/main.js` instead of `dist/main.js`. That silently breaks the scaffolded `start:prod` script.++```typescript+import { NodeSDK } from "@opentelemetry/sdk-node";+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";+import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";+import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";+import { ExpressLayerType } from "@opentelemetry/instrumentation-express";+import { resourceFromAttributes } from "@opentelemetry/resources";++const TRACEWAY_URL = "https://your-traceway-instance.com";+const TRACEWAY_TOKEN = "your-project-token";+const headers = { Authorization: `Bearer ${TRACEWAY_TOKEN}` };++const sdk = new NodeSDK({+ resource: resourceFromAttributes({+ "service.name": "my-nestjs-app",+ "service.version": "1.0.0",+ }),++ traceExporter: new OTLPTraceExporter({+ url: `${TRACEWAY_URL}/api/otel/v1/traces`,+ headers,+ }),++ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({+ url: `${TRACEWAY_URL}/api/otel/v1/metrics`,+ headers,+ }),+ exportIntervalMillis: 30_000,+ }),+ ],++ logRecordProcessors: [+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({+ url: `${TRACEWAY_URL}/api/otel/v1/logs`,+ headers,+ }),+ }),+ ],++ instrumentations: [+ getNodeAutoInstrumentations({+ "@opentelemetry/instrumentation-fs": { enabled: false },+ "@opentelemetry/instrumentation-net": { enabled: false },+ "@opentelemetry/instrumentation-dns": { enabled: false },+ // NestJS 11 runs on Express 5, which routes through the `router` package.+ // This instrumentation duplicates instrumentation-express and emits+ // "middleware - patched" spans that can surface as junk endpoint rows.+ "@opentelemetry/instrumentation-router": { enabled: false },+ "@opentelemetry/instrumentation-express": {+ ignoreLayersType: [ExpressLayerType.MIDDLEWARE],+ },+ }),+ ],+});++sdk.start();++for (const signal of ["SIGTERM", "SIGINT"] as const) {+ process.on(signal, () => {+ sdk.shutdown().finally(() => process.exit(0));+ });+}+```++Three details in that file are easy to get wrong:++> **Use `resourceFromAttributes`, not `new Resource(...)`.** The `Resource` class was removed in `@opentelemetry/resources` 2.x, which is the version `@opentelemetry/sdk-node` installs today. Older guides that still show `new Resource({ ... })` fail to compile with `error TS2693: 'Resource' only refers to a type, but is being used as a value here`.++> **Pass the log exporter as `{ exporter }`.** `new BatchLogRecordProcessor({ exporter: ... })` is the shape current `@opentelemetry/sdk-logs` expects. The older positional form, `new BatchLogRecordProcessor(exporter)`, fails to compile with `error TS2345: Property 'exporter' is missing`.++> **Configure `getNodeAutoInstrumentations()`.** Called bare, it also turns on `instrumentation-router`, which duplicates `instrumentation-express` on Express 5 and adds 8 to 11 child spans per request. Those extra spans carry `http.route` but no HTTP method, and when a batch flush separates them from their parent Traceway reads them as roots and files them as junk endpoint rows named `middleware - patched` with status code 0. The config above removes them.++`sdk.shutdown()` flushes buffered spans, logs, and metrics. Without it a container restart drops the last few seconds of telemetry.++### 2. Import it first in `src/main.ts`++```typescript+// MUST be the first import: it starts the OTel SDK before @nestjs/core+// (and therefore express and http) is loaded, so auto-instrumentation can patch them.+import './instrumentation';++import { NestFactory } from '@nestjs/core';+import { AppModule } from './app.module';++async function bootstrap() {+ const app = await NestFactory.create(AppModule);+ await app.listen(process.env.PORT ?? 3000);+}+bootstrap();+```++The import has to be on the first line. Auto-instrumentation works by patching modules as they are loaded, so anything loaded before the SDK starts is never traced.++> If you use Prisma, add `import { PrismaInstrumentation } from "@prisma/instrumentation"` and put `new PrismaInstrumentation()` in the `instrumentations` array next to `getNodeAutoInstrumentations()`.++## Running Your App++Nothing changes. The side-effect import means every script the NestJS CLI scaffolded keeps working as it is:++```bash+npm run start # nest start+npm run start:dev # nest start --watch+npm run build && npm run start:prod+```++Do not set `NODE_OPTIONS="--require ./instrumentation.js"`. `NODE_OPTIONS` applies to the `nest` CLI process itself, which runs before anything is compiled, so `nest start` dies with `Cannot find module`.++If you would rather preload the file with `--require` instead of importing it, add scripts under new names so the scaffolded ones survive:++```json+{+ "scripts": {+ "start:otel-require": "node --require ts-node/register --require ./src/instrumentation.ts src/main.ts",+ "start:prod:otel-require": "node --require ./dist/instrumentation.js dist/main.js"+ }+}+```++Note the paths: `./src/instrumentation.ts` in development, `./dist/instrumentation.js` after a build. With the `import './instrumentation'` line in `main.ts` you do not need either script.++## Test Your Integration++Add two routes to a controller:++```typescript+import { Controller, Get, Param } from "@nestjs/common";++@Controller("api")+export class AppController {+ @Get("users/:id")+ getUser(@Param("id") id: string) {+ return { id, name: `User ${id}` };+ }++ @Get("test-error")+ testError() {+ throw new Error("Test error from NestJS");+ }+}+```++Start the app and send a few requests:++```bash+curl http://localhost:3000/api/users/1+curl http://localhost:3000/api/users/2+curl http://localhost:3000/api/users/3+curl -i http://localhost:3000/api/test-error+```++The last one returns `HTTP/1.1 500 Internal Server Error`. Telemetry is batched, so wait about 10 seconds, or stop the app with Ctrl-C to flush it right away. Then check the dashboard:++- **Endpoints** shows one row, `GET /api/users/:id`, with a count of 3. Three rows with literal ids means `http.route` is missing. See [Endpoint Grouping](#endpoint-grouping) below.+- **Endpoints** shows `GET /api/test-error` with status code 500.+- **Issues** shows `Error: Test error from NestJS` with the full stack trace, pointing at the line in `app.controller.ts` that threw.++## Endpoint Grouping++`@opentelemetry/instrumentation-express` sets `http.route` on every request it handles, and the `http` instrumentation copies it onto the root span. So on the default Express adapter, grouping works with no extra code. `GET /api/users/1` and `GET /api/users/2` land in one `GET /api/users/:id` row.++A request that matched no route returns 404 and carries no usable `http.route`. Traceway groups all of those into one endpoint named `UNMATCHED`, which keeps scanner traffic and typos out of your endpoint list. A route you defined that returns 404 on purpose keeps its own name.++### If you use Fastify++`@nestjs/platform-fastify` needs one more package. `@opentelemetry/instrumentation-fastify` is **not** part of `@opentelemetry/auto-instrumentations-node`, and without it `http.route` is never set, so every path parameter becomes its own endpoint row.++```bash+npm install @opentelemetry/instrumentation-fastify+```++Then add the import at the top of `src/instrumentation.ts` and replace its `instrumentations` array:++```typescript+import { FastifyInstrumentation } from "@opentelemetry/instrumentation-fastify";++ instrumentations: [+ getNodeAutoInstrumentations({+ "@opentelemetry/instrumentation-fs": { enabled: false },+ "@opentelemetry/instrumentation-net": { enabled: false },+ "@opentelemetry/instrumentation-dns": { enabled: false },+ }),+ new FastifyInstrumentation(),+ ],+```++The export is `FastifyInstrumentation`. `FastifyOtelInstrumentation` is a different package and that import fails to compile.++The `instrumentation-router` and `instrumentation-express` entries only matter on the Express adapter, so drop them from a Fastify config.++## Errors and Issues++Nest's default exception filter turns an uncaught `Error` into a 500 response. `@opentelemetry/instrumentation-nestjs-core` records the exception on the handler span before that happens, so you get the Issue with a stack trace *and* an endpoint row correctly marked 500, with no extra code. Adding your own global `@Catch()` filter does not turn this off.++Two behaviours to expect:++- **Each throw produces two exception events.** `instrumentation-nestjs-core` wraps every handler in two spans, one with `nestjs.type=request_context` and one with `nestjs.type=handler`, and both call `recordException`. One failed request shows a count of 2 on the issue.+- **`HttpException` subclasses are captured too.** `throw new NotFoundException("no such thing")` returns a 404 as intended and still shows up under Issues. A global `@Catch()` filter runs after the span already recorded the exception, so it does not help. If you use exceptions for ordinary control flow, catch them inside the handler and build the response yourself, or expect 4xx noise in the feed.++Issue grouping is based on the stack trace, so the same error gets a different hash in development and in production. `ts-node` frames point at `src/*.ts`, compiled frames point at `dist/*.js`. Do not be surprised by two issues for what is really one bug.++To record an error without throwing, put it on the active span:++```typescript+import { Injectable } from "@nestjs/common";+import { trace, SpanStatusCode } from "@opentelemetry/api";++@Injectable()+export class PaymentService {+ async charge(amount: number) {+ try {+ await this.gateway.charge(amount);+ } catch (error: any) {+ const span = trace.getActiveSpan();+ if (span) {+ span.recordException(error);+ span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ }+ return { ok: false };+ }+ }+}+```++## Logs++The `logRecordProcessors` block in `src/instrumentation.ts` already ships logs. What is left is picking a logger that OTel can read.++> **Nest's built-in `Logger` is not bridged.** `new Logger("AppController").log("...")` writes to stdout and never reaches Traceway. This is the most common reason a log pipeline looks broken when it is fine.++### Option 1: winston++```bash+npm install winston @opentelemetry/winston-transport+```++`@opentelemetry/winston-transport` is required and is **not** a dependency of `@opentelemetry/auto-instrumentations-node`. Without it, the winston instrumentation still injects `trace_id` and `span_id` into your console output but quietly stops there, logging only `@opentelemetry/winston-transport is not available, log records will not be automatically sent.`++```typescript+// src/logger.ts+import * as winston from 'winston';++export const logger = winston.createLogger({+ level: 'info',+ format: winston.format.json(),+ transports: [new winston.transports.Console()],+});+```++```typescript+// in a controller or service+import { logger } from './logger';++logger.info('fetching user', { userId: id });+```++Logs emitted inside a request handler pick up that request's `trace_id` and `span_id` on their own. There is no context to pass around.++### Option 2: keep Nest's `Logger`, backed by winston++If you want to keep injecting `Logger` the NestJS way, route it through winston with `nest-winston`. Records written with `this.logger.log(...)` then reach Traceway with the right trace id.++```bash+npm install winston nest-winston @opentelemetry/winston-transport+```++```typescript+// src/app.module.ts+import { Module } from '@nestjs/common';+import { WinstonModule } from 'nest-winston';+import * as winston from 'winston';+import { AppController } from './app.controller';++@Module({+ imports: [+ WinstonModule.forRoot({+ level: 'info',+ format: winston.format.json(),+ transports: [new winston.transports.Console()],+ }),+ ],+ controllers: [AppController],+})+export class AppModule {}+```++```typescript+// src/main.ts+import './instrumentation';++import { NestFactory } from '@nestjs/core';+import { WINSTON_MODULE_NEST_PROVIDER } from 'nest-winston';+import { AppModule } from './app.module';++async function bootstrap() {+ const app = await NestFactory.create(AppModule, { bufferLogs: true });+ app.useLogger(app.get(WINSTON_MODULE_NEST_PROVIDER));+ await app.listen(process.env.PORT ?? 3000);+}+bootstrap();+```++### Option 3: the OTel Logs API directly++No logging library needed:++```typescript+import { logs, SeverityNumber } from '@opentelemetry/api-logs';++const otelLogger = logs.getLogger('my-nestjs-app');++otelLogger.emit({+ severityNumber: SeverityNumber.INFO,+ severityText: 'INFO',+ body: `handling user ${id}`,+ attributes: { 'user.id': id },+});+```++## Background Jobs++Traceway's **Tasks** page is filled by spans with `SpanKind.CONSUMER`. Auto-instrumentation never creates those, so `@nestjs/schedule` cron methods, BullMQ processors, and plain `setInterval` work stay invisible until you wrap them yourself:++```typescript+import { Injectable } from '@nestjs/common';+import { trace, context, ROOT_CONTEXT, SpanKind, SpanStatusCode } from '@opentelemetry/api';++const tracer = trace.getTracer('my-nestjs-app');++@Injectable()+export class ReportsService {+ async runNightlyReport() {+ // ROOT_CONTEXT detaches the job from whatever request happened to trigger it,+ // so it gets its own trace instead of hanging off an HTTP endpoint.+ return context.with(ROOT_CONTEXT, () =>+ tracer.startActiveSpan(+ 'nightly-report',+ { kind: SpanKind.CONSUMER, attributes: { 'messaging.system': 'cron' } },+ async (span) => {+ try {+ const rows = await this.buildReport();+ span.setAttribute('rows.processed', rows.length);+ span.setStatus({ code: SpanStatusCode.OK });+ } catch (error) {+ span.recordException(error as Error);+ span.setStatus({ code: SpanStatusCode.ERROR });+ throw error;+ } finally {+ span.end();+ }+ },+ ),+ );+ }++ // Replace this with your real work.+ private async buildReport(): Promise<unknown[]> {+ return [];+ }+}+```++The span name becomes the task name in Traceway. Any `span.recordException(...)` inside the job also shows up under Issues, attributed to that task.++> **Always set the kind.** A root span left at the default `SpanKind.INTERNAL` with no HTTP attributes matches nothing Traceway stores, so it is dropped on arrival: no endpoint, no task, no span row. Logs you emitted inside it still arrive, pointing at a trace that does not exist. `kind: SpanKind.CONSUMER` is the whole fix.++## Custom Metrics++The `metricReaders` entry in `src/instrumentation.ts` exports anything you record through the OTel metrics API:++```typescript+import { Body, Controller, Post } from '@nestjs/common';+import { metrics } from '@opentelemetry/api';+import { OrdersService } from './orders.service';++const meter = metrics.getMeter('my-nestjs-app');++const ordersCreated = meter.createCounter('orders.created', {+ description: 'Orders successfully created',+});++const checkoutLatency = meter.createHistogram('checkout.latency', {+ description: 'Checkout handler latency',+ unit: 'ms',+});++@Controller('orders')+export class OrdersController {+ constructor(private readonly ordersService: OrdersService) {}++ @Post()+ async create(@Body() dto: { plan: string }) {+ const started = Date.now();+ const order = await this.ordersService.create(dto);+ ordersCreated.add(1, { plan: dto.plan });+ checkoutLatency.record(Date.now() - started, { plan: dto.plan });+ return order;+ }+}+```++`OrdersService` there is your own service. The two lines that matter are `ordersCreated.add(...)` and `checkoutLatency.record(...)`.++Tag keys become filterable dimensions in Traceway.++| OTel instrument | In Traceway |+|-----------------|-------------|+| Counter, UpDownCounter | Stored as is, for example `orders.created` |+| Gauge | Stored as is |+| Histogram | Split into two metrics, `checkout.latency.avg` and `checkout.latency.count` |+| ExponentialHistogram, Summary | Dropped. Use a plain histogram instead |++With `exportIntervalMillis: 30_000`, a brand new metric can take up to a minute to appear.++## What Gets Captured++| Layer | What's Captured |+|-------|-----------------|+| HTTP (Express or Fastify) | Incoming requests with method, route, status code, duration |+| `http.route` | Parameterized routes such as `GET /api/users/:id`, set automatically |+| NestJS handlers | Controller handler spans via `instrumentation-nestjs-core` |+| Exceptions | Thrown errors recorded as Issues with stack traces |+| Metrics | Node.js runtime metrics, HTTP server duration, plus anything you record yourself |+| Logs | Only through a bridged logger. See [Logs](#logs) |+| Background jobs | Only spans you create with `SpanKind.CONSUMER`. See [Background Jobs](#background-jobs) |+| Prisma (`@prisma/instrumentation`) | Query spans with operation name, model, duration |+| Database (`pg`, `mysql2`, `mongodb`) | Query spans with SQL statements |+| Cache (`redis`, `ioredis`) | Cache operation spans |+| Outgoing `fetch()` | HTTP client spans, via `instrumentation-undici` |+| Outgoing HTTP (`axios`) | HTTP client spans, via `instrumentation-http` |++### How Traceway Classifies Your Spans++| Span shape | Where it lands |+|-----------|----------------|+| Root span with `http.route` or `url.path` | **Endpoints**, named `METHOD /route` |+| Any span with `kind: CONSUMER` | **Tasks**, named after the span |+| Any non-root span | **Spans**, in the trace waterfall |+| An `exception` event on any span | **Issues**, with the stack trace |+| A root span with no HTTP attributes and a kind other than CONSUMER | Dropped. Nothing is stored |++A single `GET /api/users/:id` gives you a trace like this:++```+GET /api/users/:id ← Endpoint (root SERVER span)+ ├─ request handler - /api/users/:id+ ├─ AppController.getUser ← @nestjs/core request context+ ├─ getUser ← @nestjs/core handler+ ├─ users.findOne ← your own span+ └─ GET ← outgoing fetch+```++### Prisma Auto-Instrumentation++With `@prisma/instrumentation` registered in the SDK, every Prisma query creates child spans on its own:++```typescript+// Just use Prisma normally. Spans are created automatically.+const users = await this.prismaService.user.findMany();+const user = await this.prismaService.user.create({ data: { name, email } });+```++The resulting trace in Traceway:++```+GET /api/users ← Endpoint+ └─ prisma:client:query ← Span (findMany on User)+```++## Environment Variable Alternative++To configure everything from the environment, strip `resource`, `traceExporter`, `metricReaders`, and `logRecordProcessors` out of the SDK. Options set in code win over `OTEL_EXPORTER_OTLP_*`, so leaving them in gives you a confusing half override where the environment sets the service name and the code keeps sending to the hardcoded URL.++Keep the `getNodeAutoInstrumentations()` config exactly as it is. Only the exporter and resource settings move to the environment. Calling it bare here puts `instrumentation-router` and the Express middleware spans back, which is the noise the setup above removes.++```typescript+import { NodeSDK } from "@opentelemetry/sdk-node";+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";+import { ExpressLayerType } from "@opentelemetry/instrumentation-express";++const sdk = new NodeSDK({+ instrumentations: [+ getNodeAutoInstrumentations({+ "@opentelemetry/instrumentation-fs": { enabled: false },+ "@opentelemetry/instrumentation-net": { enabled: false },+ "@opentelemetry/instrumentation-dns": { enabled: false },+ "@opentelemetry/instrumentation-router": { enabled: false },+ "@opentelemetry/instrumentation-express": {+ ignoreLayersType: [ExpressLayerType.MIDDLEWARE],+ },+ }),+ ],+});++sdk.start();+```++```bash+export OTEL_SERVICE_NAME="my-nestjs-app"+export OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0"+export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+```++`OTEL_EXPORTER_OTLP_ENDPOINT` is the `/api/otel` base. The SDK appends `/v1/traces`, `/v1/metrics`, and `/v1/logs` itself. There is no `OTEL_SERVICE_VERSION` variable, which is why the version goes in `OTEL_RESOURCE_ATTRIBUTES`.++## Next Steps++- [OpenTelemetry Overview](/client/otel): endpoint, authentication, and project setup+- [Custom Instrumentation](/client/otel/nestjs/custom-instrumentation): manual spans, decorators, and exception capture+- [Node.js OTel Guide](/client/otel/nodejs): general Node.js setup details+- [OTel Trace Mapping](/client/otel/traces): how OTel spans map to Traceway concepts+- [OTel Logs](/client/otel/logs): shipping logs and linking them to traces+- [OTel Metrics](/client/otel/metrics): how OTel instruments map to Traceway metricsdiff --git a/docs/pages/client/nextjs/_meta.json b/docs/pages/client/otel/nextjs/_meta.jsonsimilarity index 100%rename from docs/pages/client/nextjs/_meta.jsonrename to docs/pages/client/otel/nextjs/_meta.jsondiff --git a/docs/pages/client/otel/nextjs/index.mdx b/docs/pages/client/otel/nextjs/index.mdxnew file mode 100644index 00000000..0419eff0--- /dev/null+++ b/docs/pages/client/otel/nextjs/index.mdx@@ -0,0 +1,667 @@+# Next.js++Traceway instruments Next.js on two sides:++- **Frontend (Browser)**: uncaught errors, render errors, and session recording via [`@tracewayapp/react`](https://www.npmjs.com/package/@tracewayapp/react).+- **Backend (Server)**: request traces, database spans, and server-side exceptions via OpenTelemetry.++The two work side by side on the same project. Most apps will want both.++---++## Quick Start++Minimum configuration to get errors flowing in. Drop these in, replace the connection string, and you're done.++**Frontend.** Install `@tracewayapp/react`, wrap your app in `TracewayProvider`:++```bash+npm install @tracewayapp/react+```++```tsx+// app/traceway-provider.tsx+"use client";++import { TracewayProvider } from "@tracewayapp/react";++export function TracewayClientProvider({ children }: { children: React.ReactNode }) {+ return (+ <TracewayProvider connectionString="your-token@https://traceway.example.com/api/report">+ {children}+ </TracewayProvider>+ );+}+```++```tsx+// app/layout.tsx+import { TracewayClientProvider } from "./traceway-provider";++export default function RootLayout({ children }: { children: React.ReactNode }) {+ return (+ <html lang="en">+ <body>+ <TracewayClientProvider>{children}</TracewayClientProvider>+ </body>+ </html>+ );+}+```++**Backend.** Install the OTel Node SDK and add a single `instrumentation.ts`. This installs all three signals at once: traces, metrics, and logs.++```bash+npm install @opentelemetry/sdk-node \+ @opentelemetry/auto-instrumentations-node \+ @opentelemetry/exporter-trace-otlp-http \+ @opentelemetry/exporter-metrics-otlp-http \+ @opentelemetry/exporter-logs-otlp-http \+ @opentelemetry/sdk-metrics \+ @opentelemetry/sdk-logs \+ @opentelemetry/resources \+ @opentelemetry/api \+ @opentelemetry/api-logs+```++```typescript+// instrumentation.ts (project root)+export async function register() {+ if (process.env.NEXT_RUNTIME !== "nodejs") return;++ const { NodeSDK } = await import("@opentelemetry/sdk-node");+ const { OTLPTraceExporter } = await import("@opentelemetry/exporter-trace-otlp-http");+ const { OTLPMetricExporter } = await import("@opentelemetry/exporter-metrics-otlp-http");+ const { OTLPLogExporter } = await import("@opentelemetry/exporter-logs-otlp-http");+ const { PeriodicExportingMetricReader } = await import("@opentelemetry/sdk-metrics");+ const { BatchLogRecordProcessor } = await import("@opentelemetry/sdk-logs");+ const { getNodeAutoInstrumentations } = await import("@opentelemetry/auto-instrumentations-node");+ const { resourceFromAttributes } = await import("@opentelemetry/resources");++ const base = "https://your-traceway-instance.com/api/otel";+ const headers = { Authorization: "Bearer your-project-token" };++ new NodeSDK({+ resource: resourceFromAttributes({+ "service.name": "my-nextjs-app",+ "service.version": "1.0.0",+ }),+ traceExporter: new OTLPTraceExporter({ url: `${base}/v1/traces`, headers }),+ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({ url: `${base}/v1/metrics`, headers }),+ exportIntervalMillis: 30_000,+ }),+ ],+ logRecordProcessors: [+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({ url: `${base}/v1/logs`, headers }),+ }),+ ],+ instrumentations: [getNodeAutoInstrumentations()],+ }).start();+}+```++Three details in that file are easy to get wrong:++- `resourceFromAttributes({ ... })` is the current API. `@opentelemetry/resources` 2.x removed the `Resource` class, so `new Resource({ ... })` throws `TypeError: Resource is not a constructor` and takes the Next.js dev server down with it.+- `new BatchLogRecordProcessor({ exporter })` takes an options object. Passing the exporter directly (`new BatchLogRecordProcessor(exporter)`) fails to compile in TypeScript with `error TS2345`. In plain JavaScript it starts and then ships zero logs.+- `metricReaders` and `logRecordProcessors` are arrays. The older singular `metricReader` and `logRecordProcessor` fields still work but are deprecated.++The `process.env.NEXT_RUNTIME !== "nodejs"` guard is required. Next.js runs `instrumentation.ts` in both the Node.js and Edge runtimes, and the OTel Node SDK only works in Node.js. The dynamic `import()` calls keep OTel out of the Edge bundle.++> Next.js < 15: also set `experimental.instrumentationHook: true` in `next.config.js`. Not needed for 15+.++That's the whole base setup. Restart the dev server, make a few requests, and confirm they landed with the checklist in [Verify it worked](#verify-it-worked). The rest of this page is for the cases where the defaults aren't enough.++---++## Frontend (Browser)++Captures browser errors, render errors, and session recordings. Works with both App Router and Pages Router.++### Pages Router setup++If you're still on `pages/`, mount the provider directly in `_app.tsx`:++```tsx+// pages/_app.tsx+import type { AppProps } from "next/app";+import { TracewayProvider } from "@tracewayapp/react";++export default function App({ Component, pageProps }: AppProps) {+ return (+ <TracewayProvider connectionString="your-token@https://traceway.example.com/api/report">+ <Component {...pageProps} />+ </TracewayProvider>+ );+}+```++### What you get by default++Defaults match the rest of the JS SDK family (no flags needed):++| Option | Default | Purpose |+|---|---|---|+| `sessionRecording` | `true` | rrweb recorder runs; segments ride along with each exception |+| `recordAllSessions` | `false` | Only exception-bound clips upload (no always-on recording) |+| `captureLogs` | `true` | `console.{log,info,warn,error}` mirrored into the rolling buffer |+| `captureNetwork` | `true` | `fetch` and XHR calls recorded as network actions |+| `captureNavigation` | `true` | History API push/replace/pop recorded as navigation actions |++The `connectionString` uses the standard `{token}@{apiUrl}` format from your Traceway project page.++`TracewayProvider` also acts as a React error boundary. Render-time exceptions are captured and re-thrown so your app behaves exactly as without Traceway. No separate `<TracewayErrorBoundary>` is required.++### Capture errors manually++Inside any client component, use the `useTraceway` hook:++```tsx+"use client";++import { useTraceway } from "@tracewayapp/react";++export function CheckoutButton() {+ const { captureException } = useTraceway();++ async function handleClick() {+ try {+ await placeOrder();+ } catch (e) {+ captureException(e as Error);+ }+ }++ return <button onClick={handleClick}>Place order</button>;+}+```++### Test the frontend integration++Throw during render, so the throw goes through the error boundary described above. A `throw` inside an `onClick` handler does not reach a React error boundary, it reaches the global `window.onerror` handler instead, which is a different path.++```tsx+"use client";++import { useState } from "react";++export function TestErrorButton() {+ const [boom, setBoom] = useState(false);+ if (boom) throw new Error("Test error from Next.js client");+ return <button onClick={() => setBoom(true)}>Send test error</button>;+}+```++The error should appear in your Traceway dashboard under **Issues** within a few seconds, with the session recording attached.++---++## Backend (Server)++Captures server-side traces, database spans, and exceptions thrown in Server Components, route handlers, and `getServerSideProps`. The frontend SDK can see none of these.++### Verify it worked++Start the app, make a few requests, then wait about 15 seconds for the batch exporters to flush.++1. **Endpoints.** Hit a dynamic route with three different ids (`/api/users/1`, `/api/users/2`, `/api/users/3`). You should see one row, `GET /api/users/[id]`, with count 3. Three separate rows mean the request span is not coming from Next.js (see [Do not preload the SDK](#do-not-preload-the-sdk)).+2. **Spans.** Open that endpoint. You should see Next.js's own children (`resolve page components`, `executing api route (app) /api/users/[id]`, `start response`) plus any database or `fetch` spans.+3. **Issues.** Add a route that throws. The response is HTTP 500, the endpoint row shows 500, and the Issue carries a stack trace.+4. **Logs.** Emit a log inside a handler and open the request's trace. The log shows up on it.+5. **Metrics.** `nodejs.eventloop.utilization`, `v8js.memory.heap.used` and `v8js.gc.duration` arrive on their own once a metric reader is configured. Outgoing calls also produce `http.client.request.duration`.++There is no `http.server.request.duration`. Next.js emits the incoming request span from its own tracer, so `@opentelemetry/instrumentation-http` never runs on incoming requests and never records its server metrics. Request duration and status code live on the endpoint row instead.++If nothing arrives at all, check that `instrumentation.ts` is being loaded. Older Next.js versions printed `Compiling instrumentation Node.js ...` at startup, but Next.js 16 with Turbopack prints nothing, so do not rely on that line. Put a `console.log("otel register")` as the first statement inside `register()` and restart. If it never prints, the file is not being picked up (wrong location, or the hook is off on Next.js < 15).++If the file loads and data still does not show up, turn on OTel's own diagnostics. Exporters are silent about a bad URL or a 401 by default:++```bash+OTEL_LOG_LEVEL=debug npm run dev+```++### Endpoint grouping++Nothing to do. Next.js runs its own tracer and emits the request span (`next.span_type: BaseServer.handleRequest`) with `http.route` already set to the route pattern, so `/api/users/1` and `/api/users/2` both land on one endpoint, `GET /api/users/[id]`. A plain route handler is enough:++```typescript+// app/api/users/[id]/route.ts+const users: Record<string, { id: number; name: string }> = {+ "1": { id: 1, name: "Ada" },+ "2": { id: 2, name: "Grace" },+ "3": { id: 3, name: "Alan" },+};++export async function GET(+ req: Request,+ { params }: { params: Promise<{ id: string }> }+) {+ const { id } = await params;+ const user = users[id];+ if (!user) {+ return Response.json({ error: "User not found" }, { status: 404 });+ }+ return Response.json(user);+}+```++Swap the map for your real data source. The grouping does not depend on it. A 404 from this handler also stays on the `GET /api/users/[id]` row, because Next.js still sets `http.route` when the route matched but the record did not exist.++Do not wrap handlers in a helper that sets `http.route` by hand. Traceway names the endpoint from the request span, and `trace.getActiveSpan()` inside a handler is not that span, it is the inner `executing api route (app) ...` span. Anything you set there shows up on that child span in the trace, never on the endpoint row:++```typescript+import { trace } from "@opentelemetry/api";++export async function GET() {+ // lands on the "executing api route (app) /api/orders" span, useful for debugging+ trace.getActiveSpan()?.setAttribute("order.source", "web");+ return Response.json({ ok: true });+}+```++### Errors and exceptions++Throw and let it bubble. Next.js records the exception on its own span, so an uncaught error in a route handler already produces an HTTP 500 response, an endpoint row with status 500, and an Issue with a full stack trace:++```typescript+// app/api/test-error/route.ts+export async function GET() {+ throw new Error("Test error from Next.js server");+}+```++Do not catch the error, call `span.recordException(error)`, and re-throw. Next.js has already recorded it, so the same error is reported twice and every issue count, error rate and alert threshold is doubled. One request through a hand-wrapped handler produces an Issue with a count of 2.++If you catch an error and handle it, so it never bubbles out of the handler, then record it yourself. That is the one case where `recordException` belongs in your code:++```typescript+import { trace, SpanStatusCode } from "@opentelemetry/api";++export async function POST() {+ const span = trace.getActiveSpan();+ try {+ await processPayment();+ } catch (error) {+ span?.recordException(error as Error);+ span?.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });+ return Response.json({ error: "Payment failed" }, { status: 500 });+ }+ return Response.json({ status: "ok" });+}+```++### Logs++Logs are a separate OTLP pipeline. The Quick Start `instrumentation.ts` already configures it. Create one shared module for the logger:++```typescript+// lib/otel.ts+import { logs } from "@opentelemetry/api-logs";++export const logger = logs.getLogger("my-nextjs-app");+```++Then emit from any route handler, Server Component, or server action:++```typescript+// app/api/orders/route.ts+import { SeverityNumber } from "@opentelemetry/api-logs";+import { logger } from "@/lib/otel";++export async function GET() {+ logger.emit({+ severityNumber: SeverityNumber.INFO,+ severityText: "INFO",+ body: "listing orders",+ attributes: { "user.id": "u-123" },+ });+ return Response.json({ ok: true });+}+```++You never pass a trace id. The logs API reads the active span from context, so a log emitted during a request carries that request's `trace_id` and `span_id` and appears on the request's trace in Traceway, next to its spans.++### Background tasks++Work that isn't an HTTP request, such as a cron route, a queue consumer, or a webhook processor, should open a span with `SpanKind.CONSUMER`. Traceway lists those under **Tasks** instead of **Endpoints**, using the span name as the task name:++```typescript+// app/api/cron/nightly/route.ts+import { trace, SpanKind } from "@opentelemetry/api";++const tracer = trace.getTracer("my-nextjs-app");++async function generateReport() {+ // your job goes here+}++export async function GET() {+ await tracer.startActiveSpan(+ "nightly-report",+ { kind: SpanKind.CONSUMER, root: true },+ async (span) => {+ try {+ span.setAttribute("job.queue", "reports");+ await generateReport();+ } finally {+ span.end();+ }+ }+ );+ return Response.json({ ok: true });+}+```++`root: true` detaches the job from the triggering HTTP request so it gets its own trace, which is what you want for a cron endpoint. Drop it when the work really is part of the request.++The `kind` is the part that matters. A root span with the default kind (`INTERNAL`) and no HTTP attributes is dropped on ingest: no Task, no Endpoint, no span row, and no error to tell you why. If a job never shows up under Tasks, this is the first thing to check.++### Packages Next.js bundles++OTel patches CommonJS packages at `require()` time. Next.js bundles server dependencies, which hides them from that hook. Next.js already excludes `pg`, `mongodb`, `@prisma/client` and `better-sqlite3`, so those are instrumented with no extra config. Anything else, including `redis`, `ioredis` and `mysql2`, has to be listed:++```typescript+// next.config.ts+import type { NextConfig } from "next";++const nextConfig: NextConfig = {+ serverExternalPackages: ["ioredis", "redis", "mysql2"],+};++export default nextConfig;+```++If a database or cache client produces no child spans, this is almost always why.++Also check the client version against what the instrumentation supports. `@opentelemetry/instrumentation-ioredis` supports `ioredis >=2.0.0 <6`, so a fresh `npm install ioredis` (v6) yields no spans no matter how `next.config.ts` is set up.++### Do not preload the SDK++Generic OTel Node guides tell you to preload the SDK:++```bash+# don't do this with Next.js+NODE_OPTIONS='--require ./otel.js' next start+```++On Next.js that makes `@opentelemetry/instrumentation-http` create the request root span before Next.js's own tracer runs. That span only knows the literal URL (`url.path: /api/users/42`), never the route pattern, so it wins over Next.js's span and your endpoints fragment into one row per id. Wrapping handlers does not repair it, because a handler can only reach the innermost active span.++Use `instrumentation.ts` with the `register()` hook, as shown in the Quick Start.++### What gets captured++With the setup above, OTel captures the following automatically:++| Layer | What's Captured |+|-------|-----------------|+| HTTP server | Incoming requests with route pattern, method, status code, duration |+| Route handler errors | Uncaught throws recorded as Issues, with a 500 on the endpoint |+| Prisma (`@prisma/instrumentation`) | Query spans with operation name, model, duration |+| Database (`pg`, `mongodb`) | Query spans, Next.js externalizes these packages by default |+| Cache (`redis`, `ioredis`), `mysql2` | Query spans, but only after adding them to `serverExternalPackages` |+| Outgoing `fetch()` | Outgoing HTTP request spans (auto via `instrumentation-undici`) |+| Node.js runtime metrics | Event loop, heap, and GC. No HTTP server histogram, see [Verify it worked](#verify-it-worked) |++Root SERVER spans become **Endpoints** in Traceway (e.g., `GET /api/users/[id]`), CONSUMER spans become **Tasks**, child spans become **Spans**, and exception events become **Issues**.++#### Prisma auto-instrumentation++With `@prisma/instrumentation` registered, every Prisma query creates child spans (no manual wrapping needed):++```typescript+const users = await prisma.user.findMany();+const user = await prisma.user.create({ data: { name, email } });+```++Resulting trace:++```+GET /api/users ← Endpoint+ └─ prisma:client:query ← Span (findMany on User)+```++To register it, add the instrumentation to the Quick Start file:++```bash+npm install @prisma/instrumentation+```++```typescript+// inside register(), next to the other imports+const { PrismaInstrumentation } = await import("@prisma/instrumentation");++// then in the NodeSDK options+instrumentations: [getNodeAutoInstrumentations(), new PrismaInstrumentation()],+```++#### Outgoing fetch() auto-instrumentation++Outgoing `fetch()` calls are traced by `@opentelemetry/instrumentation-undici`:++```typescript+export async function GET() {+ const res = await fetch("https://api.example.com/data");+ const data = await res.json();+ return Response.json(data);+}+```++Resulting trace:++```+GET /api/external ← Endpoint+ └─ GET ← Span (https://api.example.com/data)+```++### Manual spans++For operations that aren't auto-instrumented (SQLite via `better-sqlite3`, Server Component rendering, custom business logic), open a span by hand with `@opentelemetry/api`:++```typescript+import { trace, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-nextjs-app");++export async function GET(+ req: Request,+ { params }: { params: Promise<{ id: string }> }+) {+ const { id } = await params;++ return tracer.startActiveSpan("fetch-order", async (span) => {+ try {+ span.setAttribute("order.id", id);+ const order = await db.orders.findById(id);++ if (!order) {+ span.setStatus({ code: SpanStatusCode.ERROR, message: "Order not found" });+ return Response.json({ error: "Not found" }, { status: 404 });+ }++ return Response.json(order);+ } finally {+ span.end();+ }+ });+}+```++Ending the span in `finally` is the safe shape. If the body throws, the span still closes and the error still bubbles up to Next.js, which reports it once.++#### Server Components++Wrap Server Component bodies in a span to see their render time in the trace:++```typescript+import { trace } from "@opentelemetry/api";++const tracer = trace.getTracer("my-nextjs-app");++export default async function UsersPage() {+ return tracer.startActiveSpan("UsersPage.render", async (span) => {+ try {+ const users = await fetchUsers();+ span.setAttribute("user.count", users.length);+ return (+ <ul>+ {users.map((user) => (<li key={user.id}>{user.name}</li>))}+ </ul>+ );+ } finally {+ span.end();+ }+ });+}+```++Server Component spans appear as **Spans** in Traceway, nested under the parent HTTP request trace.++#### Nested spans++Spans created inside an active span are automatically linked as children:++```typescript+async function processPayment(orderId: string) {+ return tracer.startActiveSpan("process-payment", async (span) => {+ try {+ await validateCard(orderId);+ await chargeAmount(orderId);+ span.setStatus({ code: SpanStatusCode.OK });+ } catch (error) {+ span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });+ throw error;+ } finally {+ span.end();+ }+ });+}+```++### Custom metrics++The Quick Start already exports Node.js runtime metrics. To add your own, create a meter once and reuse it. Put it in the same `lib/otel.ts` as the logger:++```typescript+// lib/otel.ts (alongside the logger above)+import { metrics } from "@opentelemetry/api";++export const meter = metrics.getMeter("my-nextjs-app");+export const ordersProcessed = meter.createCounter("orders.processed", {+ description: "Orders processed",+});+```++```typescript+// app/api/orders/route.ts+import { ordersProcessed } from "@/lib/otel";++async function placeOrder() {+ // your order logic goes here+}++export async function POST() {+ await placeOrder();+ ordersProcessed.add(1, { plan: "pro" });+ return Response.json({ ok: true });+}+```++The counter shows up in Traceway under the metric name you gave it, and the attributes become tags you can group by in a widget.++### Enabling the Instrumentation Hook (Next.js 13.4–14.x)++For Next.js versions before 15, enable the instrumentation hook in `next.config.js`:++```javascript+/** @type {import('next').NextConfig} */+const nextConfig = {+ experimental: {+ instrumentationHook: true,+ },+};++module.exports = nextConfig;+```++Not needed for Next.js 15+, where the hook is stable by default.++### Readable stack traces++By default the frames in an Issue point into Next.js's build output rather than your code:++```+Error: Test error from Next.js server+ .next/server/chunks/[root-of-the-server]__0og452-._.js:1:983+```++Next.js installs its own `Error.prepareStackTrace`, which bypasses Node's source map support, so the stack recorded by OTel keeps the bundled frames. Reset it at the top of `register()` to get Node's default formatter back:++```typescript+// instrumentation.ts (project root)+export async function register() {+ if (process.env.NEXT_RUNTIME !== "nodejs") return;++ (Error as unknown as { prepareStackTrace?: unknown }).prepareStackTrace = undefined;++ // ...the rest of the SDK setup+}+```++That is enough in development, where Next.js already runs the server with `--enable-source-maps`. Frames then read `app/api/orders/route.ts:12:9`.++A production build needs two more things. Node has to be told to use the source maps, and the maps have to exist:++```bash+NODE_OPTIONS='--enable-source-maps' next start+```++```typescript+// next.config.ts, only if the build does not already emit .next/server/chunks/*.js.map+const nextConfig: NextConfig = {+ experimental: { serverSourceMaps: true },+};+```++Next.js 16 with Turbopack writes those `.map` files on its own, so the config change is usually unnecessary. Check for them after `next build` before adding it.++### Auto vs manual reference++| Operation | Auto-instrumented? | Notes |+|-----------|-------------------|-------|+| HTTP requests (incoming) | Yes | Next.js's own tracer sets `http.route` and the status code |+| Uncaught route handler errors | Yes | Reported once, with a 500 on the endpoint |+| Outgoing `fetch()` calls | Yes | Via `instrumentation-undici` |+| Prisma queries | Yes | Via `@prisma/instrumentation` |+| Database queries (`pg`, `mongodb`) | Yes | Next.js externalizes these, so `require-in-the-middle` sees them |+| Cache (`redis`, `ioredis`), `mysql2` | Only with config | Add them to `serverExternalPackages` |+| SQLite (`better-sqlite3`) | No | Use manual spans |+| Server Component rendering | No | Use `tracer.startActiveSpan()` |+| Background jobs / cron | No | Use `tracer.startActiveSpan()` with `SpanKind.CONSUMER` |+| Custom business logic | No | Use `tracer.startActiveSpan()` |++### Environment variable alternative++Instead of hardcoding the endpoint and token, use standard OTel environment variables:++```bash+export OTEL_SERVICE_NAME="my-nextjs-app"+export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+```++With these set, your `instrumentation.ts` can omit the exporter URLs, the headers, and the `resource` block. The SDK picks all of them up automatically.++---++## Next Steps++- [React SDK Reference](/client/react): provider options, hooks, advanced usage+- [Sessions](/learn/sessions): how session recording is stored and replayed+- [Node.js OTel Guide](/client/otel/nodejs): general Node.js setup details+- [OTel Overview](/client/otel): protocol details and authentication+- [OTel Trace Mapping](/client/otel/traces): how OTel spans map to Traceway concepts+- [OTel Logs](/client/otel/logs): log record fields and severity mapping+- [OTel Metrics](/client/otel/metrics): metric types and how they are storeddiff --git a/docs/pages/client/node-sdk/_meta.json b/docs/pages/client/otel/nodejs/_meta.jsonsimilarity index 80%rename from docs/pages/client/node-sdk/_meta.jsonrename to docs/pages/client/otel/nodejs/_meta.jsonindex 772acc0f..595095f6 100644--- a/docs/pages/client/node-sdk/_meta.json+++ b/docs/pages/client/otel/nodejs/_meta.json@@ -1,5 +1,6 @@{"index": "Quick Start","traces": "Traces",+ "logs": "Logs","metrics": "Metrics"}diff --git a/docs/pages/client/otel/nodejs/index.mdx b/docs/pages/client/otel/nodejs/index.mdxnew file mode 100644index 00000000..8b2a9a20--- /dev/null+++ b/docs/pages/client/otel/nodejs/index.mdx@@ -0,0 +1,358 @@+# Node.js (OpenTelemetry)++Instrument any Node.js backend (Express, Fastify, Koa, Hono, NestJS, or plain HTTP) with OpenTelemetry and export traces, metrics, and logs to Traceway.++## Requirements++- Node 20.6 or newer. The ESM loader hook below uses `module.register()`, which landed in Node 18.19 / 20.6.+- Node 22.18 or newer if you want to run `.ts` files directly. Node strips the types itself, so you do not need `tsx` or `ts-node`.++## Installation++Install every package the snippets on this page import. Most of them also arrive as transitive dependencies of `@opentelemetry/sdk-node`, but relying on that breaks under pnpm's strict `node_modules` and Yarn PnP, and it leaves the versions unpinned:++```bash+npm install @opentelemetry/sdk-node \+ @opentelemetry/auto-instrumentations-node \+ @opentelemetry/instrumentation \+ @opentelemetry/api \+ @opentelemetry/api-logs \+ @opentelemetry/resources \+ @opentelemetry/sdk-metrics \+ @opentelemetry/sdk-logs \+ @opentelemetry/exporter-trace-otlp-http \+ @opentelemetry/exporter-metrics-otlp-http \+ @opentelemetry/exporter-logs-otlp-http+```++`@opentelemetry/instrumentation` is what provides the ESM loader hook. With npm it resolves today only as a hoisted transitive dependency of `sdk-node`, which breaks under pnpm's strict `node_modules` and Yarn PnP and leaves the version unpinned, so declare it directly.++## Setup++Create an `instrumentation.ts` (or `.js`) file at the root of your project. This file must be loaded **before** your application code.++```typescript+// instrumentation.ts+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);++import { NodeSDK } from "@opentelemetry/sdk-node";+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";+import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";+import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";+import { resourceFromAttributes } from "@opentelemetry/resources";++const sdk = new NodeSDK({+ resource: resourceFromAttributes({+ "service.name": "my-service",+ "service.version": "1.0.0",+ }),++ traceExporter: new OTLPTraceExporter({+ url: "https://your-traceway-instance.com/api/otel/v1/traces",+ headers: { Authorization: "Bearer your-project-token" },+ }),++ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({+ url: "https://your-traceway-instance.com/api/otel/v1/metrics",+ headers: { Authorization: "Bearer your-project-token" },+ }),+ exportIntervalMillis: 30_000,+ }),+ ],++ logRecordProcessors: [+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({+ url: "https://your-traceway-instance.com/api/otel/v1/logs",+ headers: { Authorization: "Bearer your-project-token" },+ }),+ scheduledDelayMillis: 2000,+ }),+ ],++ instrumentations: [getNodeAutoInstrumentations()],+});++sdk.start();++for (const signal of ["SIGTERM", "SIGINT"]) {+ process.on(signal, () => {+ sdk.shutdown().finally(() => process.exit(0));+ });+}+```++Four details in that file are easy to get wrong:++- The first two lines register the ESM loader hook. They must come before every other import. See [ESM: Register the Loader Hook](#esm-register-the-loader-hook) for what breaks without them.+- `resourceFromAttributes({...})` replaced the old `new Resource({...})`. The `Resource` class was removed in `@opentelemetry/resources` 2.x, so `import { Resource }` now crashes on startup with `SyntaxError: Named export 'Resource' not found`.+- `metricReaders` takes an array. The singular `metricReader` option still works but is deprecated.+- `BatchLogRecordProcessor` takes a **single options object** with an `exporter` key. Passing the exporter as a positional argument fails to compile in TypeScript with `error TS2345`. In plain JavaScript it runs and ships zero logs with no error printed.++The shutdown handler flushes the pending trace, metric, and log batches when the process stops. Without it you lose everything still sitting in a batch, which is usually why "I started the app, hit a route, pressed Ctrl-C, and saw nothing".++> **Using CommonJS?** Write the same file with `require(...)` and delete the two `register(...)` lines. `import.meta.url` is not valid in a CommonJS file, and `require` is patched directly so the hook is not needed.++## Running Your App++Running `.ts` or `.js` files as ESM needs `"type": "module"` in your `package.json`. A fresh `npm init -y` writes `"type": "commonjs"`, and the ESM file fails under it with `SyntaxError: Cannot use import statement outside a module`:++```json+{+ "type": "module",+ "scripts": {+ "start": "node --import ./instrumentation.ts server.ts"+ }+}+```++Load the instrumentation file before everything else using Node's `--require` (CommonJS) or `--import` (ESM) flag:++```bash+# CommonJS+node --require ./instrumentation.js server.js++# ESM with .mjs files, works whatever "type" says+node --import ./instrumentation.mjs server.mjs++# ESM with .ts files, needs "type": "module"+# Node 22.18+ strips the types natively, no tsx or ts-node+node --import ./instrumentation.ts server.ts+```++Or set it via environment variable:++```bash+export NODE_OPTIONS="--require ./instrumentation.js"+node server.js+```++## ESM: Register the Loader Hook++OpenTelemetry patches libraries as they load. Under CommonJS it hooks `require` and this happens on its own. Under ESM, Node needs a loader hook registered before anything else is imported. Without it your web framework is never patched, and the HTTP spans that reach Traceway carry no route information.++These two lines, at the very top of `instrumentation.ts`, are the fix:++```typescript+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);+```++On Node 22.15 and newer this prints `DeprecationWarning: module.register() is deprecated. Use module.registerHooks() instead.` at startup. Ignore it. `registerHooks()` takes hook functions, not a module path, so it cannot load `hook.mjs`, and `register()` is still the supported way to install it.++### How to tell it is missing++Nothing errors. Requests still reach Traceway, statuses are correct, exceptions still land in Issues. The tell is on the **Endpoints** page: instead of one row per route you get one row per URL.++| | Without the hook | With the hook |+|---|---|---|+| Endpoint rows for 3 requests to `/api/users/101`, `/202`, `/303` | three rows, `GET /api/users/101`, `GET /api/users/202`, `GET /api/users/303` | one row, `GET /api/users/:id` with count 3 |+| SERVER span name | `GET` | `GET /api/users/:id` |+| SERVER span attributes | `url.path` only | `url.path` **and** `http.route` |++Traceway names an endpoint from `http.route` and falls back to `url.path` when it is missing, so an unpatched framework turns every path parameter value into its own endpoint.++### Confirming which libraries were patched++Run once with OTel diagnostics turned on, at the top of `instrumentation.ts`:++```typescript+import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";+diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);+```++You should see an `Applying instrumentation patch` line for `@opentelemetry/instrumentation-express` (Express 4) or `@opentelemetry/instrumentation-router` (Express 5). If the only patch line is for `http`, the hook is not registered.++## Environment Variable Alternative++Instead of hardcoding the endpoint and token, you can use standard OTel environment variables:++```bash+export OTEL_SERVICE_NAME="my-service"+export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-traceway-instance.com/api/otel"+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+```++The SDK appends `/v1/traces`, `/v1/metrics`, and `/v1/logs` to the base endpoint itself, which matches Traceway's ingest paths. The space inside `Bearer your-project-token` needs no escaping.++With these set, construct the exporters with no arguments and drop the `resource` block:++```typescript+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);++import { NodeSDK } from "@opentelemetry/sdk-node";+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";++const sdk = new NodeSDK({+ traceExporter: new OTLPTraceExporter(),+ instrumentations: [getNodeAutoInstrumentations()],+});++sdk.start();+```++The environment variables only supply the endpoint, the headers, and the service name. Metrics and logs still need their `metricReaders` and `logRecordProcessors` entries, with the exporters constructed the same way, no arguments.++## What Gets Auto-Instrumented++The `@opentelemetry/auto-instrumentations-node` package automatically instruments:++| Library | What's Captured |+|---------|-----------------|+| `http` / `https` | Incoming and outgoing HTTP requests |+| `express` | Route-level spans with `http.route` (Express 4) |+| `router` | Route handler spans and route exception events (Express 5) |+| `fastify` | Route-level spans |+| `koa` | Middleware and route spans |+| `hono` | HTTP request spans (via `http` instrumentation) |+| `undici` / global `fetch` | Outgoing request spans |+| `pg` / `mysql2` / `mongodb` | Database query spans |+| `redis` / `ioredis` | Cache operation spans |+| `grpc` | gRPC call spans |+| `pino` / `winston` | Log records, see [Logs](/client/otel/nodejs/logs) |+| `runtime-node` | Node runtime metrics such as `nodejs.eventloop.delay.p50` |++> **Two instrumentations ship disabled.** `@opentelemetry/instrumentation-fs` and `@opentelemetry/instrumentation-host-metrics` are in the package but excluded by default. Turn one on explicitly if you need it:+>+> ```typescript+> instrumentations: [+> getNodeAutoInstrumentations({+> "@opentelemetry/instrumentation-fs": { enabled: true },+> }),+> ],+> ```+>+> `fs` spans are very high volume. Expect a much noisier waterfall.++Root SERVER spans become **Endpoints** in Traceway, CONSUMER spans become **Tasks**, child spans become **Spans**, and exception events become **Issues**. See [Traces](/client/otel/nodejs/traces) for the full mapping.++## Framework Quick Start++These snippets assume the framework itself is installed (`npm install express`, `npm install fastify`, and so on). None of them import anything from OpenTelemetry.++### Express++```typescript+import express from "express";++const app = express();++app.get("/api/users/:id", (req, res) => {+ res.json({ id: req.params.id });+});++app.listen(3000);+```++No extra code needed. Auto-instrumentation captures all requests, routes, and database calls, as long as the loader hook is registered for ESM.++### Fastify++```typescript+import Fastify from "fastify";++const app = Fastify();++app.get("/api/users", async () => {+ return { users: [] };+});++app.listen({ port: 3000 });+```++### Hono++See the dedicated [Hono guide](/client/otel/hono) for multi-runtime setup.++```typescript+import { serve } from "@hono/node-server";+import { Hono } from "hono";++const app = new Hono();++app.get("/api/users", (c) => c.json({ users: [] }));++serve({ fetch: app.fetch, port: 3000 });+```++## Background Jobs++Cron jobs and queue workers show up under **Tasks**. The signal Traceway looks for is the span kind, so start the job's span with `kind: SpanKind.CONSUMER`:++```typescript+import { trace, SpanKind } from "@opentelemetry/api";++const tracer = trace.getTracer("my-service");++async function runNightlyReport() {+ await tracer.startActiveSpan(+ "nightly-report",+ { kind: SpanKind.CONSUMER },+ async (span) => {+ await buildReport();+ span.end();+ },+ );+}+```++> **A plain root span is dropped.** A root `INTERNAL` or `PRODUCER` span with no HTTP attributes and no `console.command` attribute produces no Endpoint, no Task, and no Span row. It is discarded on ingest with no error. Wrapping a job in `tracer.startActiveSpan("my-job", ...)` at the top level of a script gets you silence, not a Task.++[Traces](/client/otel/nodejs/traces) has the full example with error handling and the console-command variant.++## Test Your Integration++Add a route that throws an error:++```typescript+app.get("/test-error", () => {+ throw new Error("Test error from Traceway integration");+});+```++Visit `/test-error`. The request returns HTTP 500 (Express's default error handler) and the error appears under **Issues** with a full stack trace. You do not need error middleware for this.++The exception event is recorded by the framework instrumentation, not by the HTTP one. Keep `@opentelemetry/instrumentation-express` (Express 4) or `@opentelemetry/instrumentation-router` (Express 5) enabled. If you narrow `getNodeAutoInstrumentations()` down to HTTP only, the span still ends with an error status but carries no exception event, so no Issue is created.++If you catch the error in your own handler, record it explicitly:++```typescript+import type { NextFunction, Request, Response } from "express";+import { trace, SpanStatusCode } from "@opentelemetry/api";++app.use((err: Error, req: Request, res: Response, next: NextFunction) => {+ const span = trace.getActiveSpan();+ if (span) {+ span.recordException(err);+ span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });+ }+ res.status(500).json({ error: "Internal Server Error" });+});+```++Express only treats a middleware as an error handler when it declares all four parameters, so keep `next` in the signature even though it is unused.++## Verify It Worked++Telemetry is batched, so give it about 10 seconds, or stop the app and let the shutdown handler flush. Then check these four things in the dashboard:++1. **Endpoints.** Hit a parametrized route three times with different values, for example `/api/users/101`, `/api/users/202`, `/api/users/303`. You should see **one** row, `GET /api/users/:id`, with a count of 3. Three separate rows means the ESM loader hook is missing.+2. **Issues.** `/test-error` shows up as an Issue with a stack trace, and its Endpoints row reports status 500.+3. **Logs on the trace.** Open the endpoint's trace detail page and switch to the **Logs** tab. A log emitted inside the handler is listed there, because it carries the same trace id. See [Logs](/client/otel/nodejs/logs) for the pipeline.+4. **Tasks.** A span started with `kind: SpanKind.CONSUMER` shows up under Tasks by its span name.++## Next Steps++- [Traces](/client/otel/nodejs/traces): manual spans, background jobs, exception recording, context propagation+- [Logs](/client/otel/nodejs/logs): ship application logs and link them to traces+- [Metrics](/client/otel/nodejs/metrics): custom counters, histograms, and gauges+- [OTel Overview](/client/otel): protocol details and authentication+- [OTel Trace Mapping](/client/otel/traces): how OTel spans map to Traceway conceptsdiff --git a/docs/pages/client/otel/nodejs/logs.mdx b/docs/pages/client/otel/nodejs/logs.mdxnew file mode 100644index 00000000..f6b4eda2--- /dev/null+++ b/docs/pages/client/otel/nodejs/logs.mdx@@ -0,0 +1,132 @@+# Logs++Ship your application logs to Traceway over OTLP/HTTP. A log emitted inside a request handler is stamped with the active trace and span id automatically, so it shows up on the trace detail page next to the request that produced it.++## Install++```bash+npm install @opentelemetry/api-logs \+ @opentelemetry/sdk-logs \+ @opentelemetry/exporter-logs-otlp-http+```++If you followed the [Quick Start](/client/otel/nodejs), these are already installed.++## Setup++Add a `logRecordProcessors` entry to the `NodeSDK` you created in the [Quick Start](/client/otel/nodejs):++```typescript+// instrumentation.ts+import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";+import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs";++const sdk = new NodeSDK({+ // ...resource, traceExporter, metricReaders, instrumentations++ logRecordProcessors: [+ new BatchLogRecordProcessor({+ exporter: new OTLPLogExporter({+ url: "https://your-traceway-instance.com/api/otel/v1/logs",+ headers: { Authorization: "Bearer your-project-token" },+ }),+ scheduledDelayMillis: 2000,+ }),+ ],+});++sdk.start();+```++> **`BatchLogRecordProcessor` takes one options object.** The `exporter` goes on an `exporter` key. Passing it positionally, `new BatchLogRecordProcessor(new OTLPLogExporter({...}))`, fails to compile in TypeScript with `error TS2345: Property 'exporter' is missing`. In plain JavaScript it runs and exports nothing: the processor reads `options.exporter`, gets `undefined`, and throws inside its own export loop where you never see it. Traces and metrics keep flowing, so it looks like Traceway is dropping your logs.++## Emitting Logs++```typescript+import { logs, SeverityNumber } from "@opentelemetry/api-logs";++const logger = logs.getLogger("app");++app.get("/api/users/:id", (req, res) => {+ logger.emit({+ severityNumber: SeverityNumber.INFO,+ severityText: "INFO",+ body: `fetching user ${req.params.id}`,+ attributes: { "user.id": req.params.id },+ });++ res.json({ id: req.params.id });+});+```++Called inside a request handler, the log picks up the active span on its own. You do not pass a context. In the dashboard, open the endpoint's trace and switch to the **Logs** tab to see it attached to that request.++Use dotted attribute keys (`user.id`, `order.id`) to match OTel semantic conventions. Attribute values are stored as strings.++## Bridging Pino or Winston++If you already log with Pino or Winston, keep doing that. Once the pipeline above is configured, the auto-instrumentation forwards those records to the same OTLP exporter and stamps the active `trace_id` and `span_id` on each one. Pino works as is, Winston needs one extra package.++**Pino works with no extra packages.** `@opentelemetry/instrumentation-pino` ships in `@opentelemetry/auto-instrumentations-node` and is enabled by default, so every `logger.info(...)` is both printed locally and sent to Traceway.++```typescript+import pino from "pino";++const logger = pino();++app.get("/api/orders/:id", (req, res) => {+ logger.info({ "order.id": req.params.id }, "order lookup");+ res.json({ ok: true });+});+```++**Winston needs one more package.** Without it, `@opentelemetry/instrumentation-winston` only injects `trace_id` and `span_id` into your local output. The records are never sent to Traceway, and the reason is only printed with OTel diagnostics on:++```bash+npm install @opentelemetry/winston-transport+```++The instrumentation adds its own transport when your logger is created, so you do not register anything yourself. With the package installed, `logger.info("...")` reaches Traceway with the trace id attached.++## Severity Levels++`SeverityNumber` values map to Traceway's severity labels as follows:++| SeverityNumber | Traceway Severity |+| -------------- | ----------------- |+| 1-4 | TRACE |+| 5-8 | DEBUG |+| 9-12 | INFO |+| 13-16 | WARN |+| 17-20 | ERROR |+| 21+ | FATAL |++If you set `severityText` yourself it is used as-is (uppercased). See [Log Record Mapping](/client/otel/logs) for every field Traceway reads.++## Verify It Worked++1. Hit a route that logs, then wait a couple of seconds for the batch, or stop the app so the shutdown handler flushes.+2. Open **Logs** in the dashboard. The record should be listed with your `service.name` as the service.+3. Open the same request on **Endpoints**, go to its trace, and switch to the **Logs** tab. The log is there because it carries the request's trace id. If the Logs tab is empty but the Logs page has the record, the log was emitted outside the request's span.++## Troubleshooting++Log export failures are reported only through OTel's diagnostic channel. If nothing arrives, turn it on temporarily at the top of `instrumentation.ts`:++```typescript+import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";+diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);+```++Common causes, in the order worth checking:++- `BatchLogRecordProcessor` called with a positional exporter (see the warning above).+- The URL is missing the `/v1/logs` suffix. The full path is `https://your-traceway-instance.com/api/otel/v1/logs`.+- The process exited before the batch was sent. Add the `SIGTERM` handler from the [Quick Start](/client/otel/nodejs).+- Winston records missing while Pino records arrive: install `@opentelemetry/winston-transport`.++## Next Steps++- [Traces](/client/otel/nodejs/traces): manual spans, background jobs, exception recording+- [Metrics](/client/otel/nodejs/metrics): custom counters, histograms, and gauges+- [Log Record Mapping](/client/otel/logs): how OTLP log fields map to Traceway fieldsdiff --git a/docs/pages/client/otel/nodejs/metrics.mdx b/docs/pages/client/otel/nodejs/metrics.mdxnew file mode 100644index 00000000..ace79113--- /dev/null+++ b/docs/pages/client/otel/nodejs/metrics.mdx@@ -0,0 +1,129 @@+# Metrics++Export custom application metrics from your Node.js app to Traceway using the OpenTelemetry Metrics API.++## Setup++Metrics export is configured in your `instrumentation.ts` via the `PeriodicExportingMetricReader`. If you followed the [Quick Start](/client/otel/nodejs), metrics are already enabled.++```typescript+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";++// In your NodeSDK config:+metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({+ url: "https://your-traceway-instance.com/api/otel/v1/metrics",+ headers: { Authorization: "Bearer your-project-token" },+ }),+ exportIntervalMillis: 30_000,+ }),+],+```++The option is `metricReaders` and it takes an array. The singular `metricReader` still works but is deprecated. Keep `exportIntervalMillis` at or above the export timeout (30 seconds by default), otherwise the SDK clamps the timeout and warns about it on startup.++Values are pushed on that interval, so a fresh counter takes up to 30 seconds to appear in the dashboard. Stopping the app flushes early if you added the `SIGTERM` handler from the Quick Start.++## Creating Custom Metrics++Use `@opentelemetry/api` to create meters and instruments. Install it as a direct dependency (`npm install @opentelemetry/api`) rather than relying on it being hoisted from the SDK:++```typescript+import { metrics } from "@opentelemetry/api";++const meter = metrics.getMeter("my-service");+```++Create the meter and its instruments once at module scope, not per request.++## Metric Types++### Counter++Monotonically increasing value. Use for counts of events:++```typescript+const requestCounter = meter.createCounter("app.requests.total", {+ description: "Total number of requests",+});++// Increment+requestCounter.add(1, { endpoint: "/api/users", method: "GET" });+```++### UpDownCounter++Value that can increase or decrease. Use for gauges like active connections:++```typescript+const activeConnections = meter.createUpDownCounter("app.connections.active", {+ description: "Number of active connections",+});++activeConnections.add(1); // connection opened+activeConnections.add(-1); // connection closed+```++### Histogram++Distribution of values. Use for durations and sizes:++```typescript+const requestDuration = meter.createHistogram("app.request.duration", {+ description: "Request duration in milliseconds",+ unit: "ms",+});++const start = performance.now();+await handleRequest();+requestDuration.record(performance.now() - start, {+ endpoint: "/api/users",+});+```++### Observable Gauge++Asynchronously observed value. Use for system metrics:++```typescript+meter.createObservableGauge("app.memory.heap_used", {+ description: "Heap memory usage in bytes",+ unit: "bytes",+}).addCallback((result) => {+ result.observe(process.memoryUsage().heapUsed);+});+```++## How Metrics Appear in Traceway++| OTel Type | Traceway Handling |+|-----------|-------------------|+| Gauge | Stored as-is |+| Sum (Counter) | Stored as-is |+| Histogram | Converted to `{name}.avg` and `{name}.count` |++A histogram named `app.request.duration` therefore shows up in the dashboard as two metrics, `app.request.duration.avg` (carrying your `unit`) and `app.request.duration.count`. Search for the base name and you will find neither.++See [OTel Metrics](/client/otel/metrics) for full details on histogram handling and supported types.++## Metrics You Get for Free++`@opentelemetry/auto-instrumentations-node` enables `@opentelemetry/instrumentation-runtime-node` by default, so these arrive without any code once a metric reader is configured:++| Metric | What it measures |+|--------|------------------|+| `nodejs.eventloop.delay.p50` / `.p90` / `.p99` / `.mean` / `.max` | Event loop lag |+| `nodejs.eventloop.utilization` | Fraction of time the loop was busy |+| `http.server.request.duration.avg` / `.count` | Server request duration, from the HTTP instrumentation |+| `v8js.memory.heap.used` | Heap bytes in use, from the V8 runtime metrics |+| `v8js.gc.duration.avg` / `.count` | Garbage collection pauses |++Host-level metrics (CPU, memory, disk) are **not** included. `@opentelemetry/instrumentation-host-metrics` ships disabled, and the [Traceway OTel Agent](/learn/otel-agent) is the usual way to collect them.++## Next Steps++- [Traces](/client/otel/nodejs/traces): manual spans, background jobs, exception recording+- [Logs](/client/otel/nodejs/logs): ship application logs and link them to traces+- [Quick Start](/client/otel/nodejs): setup and installationdiff --git a/docs/pages/client/otel/nodejs/traces.mdx b/docs/pages/client/otel/nodejs/traces.mdxnew file mode 100644index 00000000..05ae9377--- /dev/null+++ b/docs/pages/client/otel/nodejs/traces.mdx@@ -0,0 +1,179 @@+# Traces++OpenTelemetry auto-instrumentation handles most tracing automatically. This page covers manual span creation and exception recording for cases where you need more control.++## How Spans Map to Traceway++Spans are classified by kind and attributes, not by whether they are a root span:++| OTel Span | Traceway Concept |+|-----------|-----------------|+| SERVER or INTERNAL span with HTTP attributes, when it is a root or its parent is not in the same export batch | **Endpoint** (e.g., `GET /api/users`) |+| `SpanKind = CONSUMER`, root or child | **Task** |+| Root INTERNAL span with a `console.command` attribute | **Task** |+| Any span with `gen_ai.*` attributes | **AI Trace** |+| Any other span with a parent | **Span** |+| Event named `"exception"` on any span | **Issue** |++> **A root span that matches none of these rows is dropped.** A root `INTERNAL` or `PRODUCER` span with no HTTP attributes and no `console.command` attribute produces no Endpoint, no Task, and no Span row. Ingest discards it and returns success, so there is nothing to debug. If the span carries an exception event, the Issue is still created, but the span itself is gone. To trace a cron job or a queue worker, set `kind: SpanKind.CONSUMER`.++See [OTel Trace Mapping](/client/otel/traces) for full details on attributes and conventions.++## Manual Span Creation++Use `@opentelemetry/api` to create custom spans for operations that aren't auto-instrumented:++```typescript+import { trace, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-service");++async function processOrder(orderId: string) {+ return tracer.startActiveSpan("process-order", async (span) => {+ try {+ span.setAttribute("order.id", orderId);++ await validateOrder(orderId);+ await chargePayment(orderId);+ await sendConfirmation(orderId);++ span.setStatus({ code: SpanStatusCode.OK });+ } catch (error) {+ const err = error instanceof Error ? error : new Error(String(error));+ span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });+ span.recordException(err);+ throw error;+ } finally {+ span.end();+ }+ });+}+```++TypeScript types a caught value as `unknown`, so narrow it with `error instanceof Error` before reading `.message` or passing it to `recordException`. Skipping that step still runs under Node's type stripping but fails `tsc` in strict mode.++Called inside a request handler, this span joins that request's waterfall. Called at the top level of a worker or a script, where it has no parent, it needs `kind: SpanKind.CONSUMER` to survive ingest. See [Background Jobs and Tasks](#background-jobs-and-tasks).++## Nested Spans++Spans created inside an active span are automatically linked as children:++```typescript+async function chargePayment(orderId: string) {+ return tracer.startActiveSpan("charge-payment", async (span) => {+ try {+ const result = await paymentGateway.charge(orderId);+ span.setAttribute("payment.status", result.status);+ return result;+ } finally {+ span.end();+ }+ });+}+```++## Background Jobs and Tasks++Cron jobs, queue consumers, and scheduled work show up under **Tasks** in Traceway. The signal is the span kind: use `SpanKind.CONSUMER`. The task name is the span name.++```typescript+import { trace, SpanKind, SpanStatusCode } from "@opentelemetry/api";++const tracer = trace.getTracer("my-service");++async function runNightlyReport() {+ await tracer.startActiveSpan(+ "nightly-report",+ { kind: SpanKind.CONSUMER },+ async (span) => {+ try {+ span.setAttribute("messaging.system", "cron");+ await buildReport();+ span.setStatus({ code: SpanStatusCode.OK });+ } catch (error) {+ const err = error instanceof Error ? error : new Error(String(error));+ span.recordException(err);+ span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });+ throw error;+ } finally {+ span.end();+ }+ },+ );+}+```++A CONSUMER span becomes a Task whether it is a root span or a child of a request. A job dispatched inside an HTTP handler still gets its own Task row, linked back to the request through the distributed trace.++> **Do not use a plain root span for a job.** A root `INTERNAL` or `PRODUCER` span with no HTTP attributes and no `console.command` attribute is dropped on ingest. `tracer.startActiveSpan("my-job", async (span) => { ... })` at the top level of a worker script produces no Task and no Span, and nothing reports the loss. Pass `{ kind: SpanKind.CONSUMER }`.++For a CLI or console command, the other accepted signal is a `console.command` attribute on a root INTERNAL span:++```typescript+const span = tracer.startSpan("app:send-digest", { kind: SpanKind.INTERNAL });+span.setAttribute("console.command", "app:send-digest");+// ... run the command ...+span.end();+```++Short-lived job processes exit before the batch exporter flushes, so call `sdk.shutdown()` (or the `SIGTERM` handler from the [Quick Start](/client/otel/nodejs)) before the process ends, otherwise the Task never leaves the machine.++## Recording Exceptions++Exceptions recorded as span events appear as **Issues** in Traceway:++```typescript+import { trace, SpanStatusCode } from "@opentelemetry/api";++function riskyOperation() {+ const span = trace.getActiveSpan();++ try {+ doSomethingDangerous();+ } catch (error) {+ const err = error instanceof Error ? error : new Error(String(error));+ if (span) {+ span.recordException(err);+ span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });+ }+ throw error;+ }+}+```++The `recordException` call adds an event with `exception.type`, `exception.message`, and `exception.stacktrace` attributes. Traceway normalizes and hashes the stack trace to group identical errors into a single Issue.++## Span Attributes++Add attributes to enrich your spans with context:++```typescript+span.setAttribute("user.id", userId);+span.setAttribute("db.statement", "SELECT * FROM users");+span.setAttributes({+ "order.id": orderId,+ "order.total": 99.99,+ "order.items": 3,+});+```++## Context Propagation++OTel automatically propagates trace context across async operations. For manual propagation across service boundaries, use the `W3C Trace Context` headers:++```typescript+import { propagation, context } from "@opentelemetry/api";++// Inject trace context into outgoing request headers+const headers = {};+propagation.inject(context.active(), headers);++// The headers now contain `traceparent` and optionally `tracestate`+await fetch("https://other-service.com/api", { headers });+```++## Next Steps++- [Logs](/client/otel/nodejs/logs): ship application logs and link them to traces+- [Metrics](/client/otel/nodejs/metrics): custom counters, histograms, and gauges+- [Quick Start](/client/otel/nodejs): setup and installationdiff --git a/docs/pages/client/otel/python/_meta.json b/docs/pages/client/otel/python/_meta.jsonnew file mode 100644index 00000000..8393d864--- /dev/null+++ b/docs/pages/client/otel/python/_meta.json@@ -0,0 +1,3 @@+{+ "index": "Quick Start"+}diff --git a/docs/pages/client/otel/python/index.mdx b/docs/pages/client/otel/python/index.mdxnew file mode 100644index 00000000..c85a50d1--- /dev/null+++ b/docs/pages/client/otel/python/index.mdx@@ -0,0 +1,255 @@+# Python (OpenTelemetry)++Instrument FastAPI, Flask, or any WSGI/ASGI Python application with OpenTelemetry and export traces, metrics, and logs to Traceway.++There is no application code to write. The OTel agent patches your framework at interpreter startup, so endpoints, status codes, child spans, exceptions, logs, and HTTP metrics all arrive from an unmodified app.++Django uses this same path but needs two extra steps of its own. See the [Django guide](/client/otel/django).++## Requirements++- Python 3.10 or newer. Current `opentelemetry-sdk` releases declare `requires-python >= 3.10`.+- A Traceway project created with framework **OpenTelemetry**, and that project's token.++## Install++```bash+pip install opentelemetry-distro opentelemetry-exporter-otlp+opentelemetry-bootstrap -a install+```++`opentelemetry-bootstrap` reads the packages already installed in the environment and installs the matching instrumentation for each one: FastAPI, Flask, Starlette, `requests`, `httpx`, `urllib3`, `sqlite3`, `psycopg`, and the standard `logging` module, among others. It only sees what is installed when you run it, so run it again after you add a dependency.++Install both packages. `opentelemetry-sdk` on its own records spans and throws them away, because it ships no OTLP exporter and no `opentelemetry-instrument` command.++## Configure++Set these in the **process environment**: the container `env:` block, a systemd `Environment=` line, a Procfile, or `export` in the shell that starts the server. `opentelemetry-instrument` reads them before your application is imported, so a `.env` file that your settings module loads at import time is too late.++```bash+export OTEL_SERVICE_NAME=my-python-service+export OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0"+export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+export OTEL_TRACES_EXPORTER=otlp+export OTEL_METRICS_EXPORTER=otlp+export OTEL_LOGS_EXPORTER=otlp++# Python only. Without these, logger.info(...) never reaches Traceway.+export OTEL_PYTHON_LOG_CORRELATION=true+export OTEL_PYTHON_LOG_LEVEL=info+```++> **`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` is not optional.** With `OTEL_TRACES_EXPORTER=otlp` and no protocol set, the Python SDK resolves the exporter to **gRPC**, and Traceway has no gRPC listener. The app starts, serves traffic, and exits 0 while every export goes into a void. The gRPC exporter does log the failure (`StatusCode.UNAVAILABLE ... Expected SETTINGS frame as the first frame`), but you only see it once something has put a console handler back on the root logger, which is what `OTEL_PYTHON_LOG_CORRELATION=true` below does. Without that variable the failure is silent.++Three more things about that block:++- `OTEL_EXPORTER_OTLP_ENDPOINT` takes the **base** URL. The SDK appends `/v1/traces`, `/v1/metrics`, and `/v1/logs` itself, which is exactly how Traceway's ingest paths are laid out. A full signal path here produces `/v1/traces/v1/traces`.+- `OTEL_SERVICE_NAME` becomes the **Server Name** on every endpoint, task, and issue.+- There is no `OTEL_SERVICE_VERSION` variable. Version travels in `OTEL_RESOURCE_ATTRIBUTES`, and without it every endpoint row carries an empty App Version, so release comparison does not work.++## Run++Put `opentelemetry-instrument` in front of whatever starts your process.++```bash+# FastAPI, Starlette, any ASGI app+opentelemetry-instrument uvicorn app:app --port 8000++# Flask development server+opentelemetry-instrument flask --app wsgi run --port 8000++# Any WSGI server, same shape+opentelemetry-instrument gunicorn wsgi:app --bind 0.0.0.0:8000++# A worker, cron job, or one-shot script+opentelemetry-instrument python worker.py+```++Starting the server directly, without the `opentelemetry-instrument` prefix, sends nothing at all. That is the single most common reason a Python integration stays silent.++For short-lived scripts, the last batch flushes on exit either way. `opentelemetry-instrument` registers a shutdown hook, and a hand-built provider registers its own `atexit` hook by default. Only an abrupt end such as `os._exit` or `SIGKILL` loses the batch.++## What Arrives With No Application Code++| Traceway | Where it comes from |+|----------|---------------------|+| **Endpoints**, grouped by route pattern | `opentelemetry-instrumentation-fastapi` / `-flask` / `-starlette` set `http.route` from the matched route |+| Status codes, including 500 on an unhandled exception | The ASGI and WSGI instrumentation |+| Child **Spans** for database and outgoing HTTP calls | `-sqlite3`, `-psycopg`, `-requests`, `-httpx`, `-urllib3` |+| **Issues** with the full Python traceback | Unhandled exceptions, recorded as `exception` span events |+| **Logs**, linked to the trace that produced them | `-logging` bridges the standard `logging` module |+| HTTP **Metrics** | `http.server.duration`, `http.server.active_requests`, `http.server.response.size`, `http.client.duration` |++## Verify Endpoint Grouping First++Hit one parametrized route three times with different ids, then open **Endpoints**. You must see one row with a count of 3, not three rows.++| Framework | Route in your code | Endpoint row |+|-----------|--------------------|--------------|+| FastAPI | `@app.get("/users/{user_id}")` | `GET /users/{user_id}` |+| Flask | `@app.get("/orders/<order_id>")` | `GET /orders/<order_id>` |++The pattern syntax is the framework's own, and Traceway stores whatever `http.route` carried. Those two rows look different and both are correct.++Three rows with literal ids in them (`GET /users/101`, `GET /users/102`) means `http.route` is not being set, so Traceway fell back to the concrete `url.path`. Check that the app really started under `opentelemetry-instrument` and that `opentelemetry-bootstrap -a install` ran after your web framework was installed.++## Exceptions++An unhandled exception needs no configuration. FastAPI, Starlette, and Flask all respond `500` on their own, the ASGI/WSGI instrumentation records the exception event, and Traceway shows the Issue with the full traceback and the endpoint row with status 500.++```python[email protected]("/boom")+def boom():+ raise ValueError("exploded on purpose")+```++Only catch-and-handle paths need code. `record_exception` fills in `exception.type`, `exception.message`, and `exception.stacktrace` for you, and `set_status` takes a `Status` object rather than a dict:++```python+from opentelemetry import trace+from opentelemetry.trace import Status, StatusCode++span = trace.get_current_span()+try:+ charge_card()+except PaymentError as error:+ span.record_exception(error)+ span.set_status(Status(StatusCode.ERROR, str(error)))+ raise+```++If you swallow the error and still return `200`, the endpoint row is recorded as a success and the Issue arrives with no failing request beside it. Return `500` whenever you record an exception.++## Logs++`opentelemetry-bootstrap -a install` installs `opentelemetry-instrumentation-logging`, which attaches the OTel handler to the **root** logger. Every `logger.info(...)` and `logger.error(...)` in your code, your framework, and your libraries is then forwarded to Traceway and stamped with the active trace and span ids.++Two Python behaviours decide what you actually see:++1. **Python's root logger defaults to `WARNING`.** A logger that inherits that level drops its own INFO records before the OTel handler ever runs, so only WARN, ERROR, and CRITICAL reach Traceway. Since most application logging is `logger.info` on an inheriting logger, the practical result is "my logs do not show up" with no error anywhere. Libraries that set their own level, such as Werkzeug at INFO, are unaffected and keep arriving.+2. **The OTel handler becomes the root logger's only handler.** The root logger normally has none, so Python falls back to printing WARNING and above on stderr. Adding the OTel handler ends that fallback: your records go to Traceway and stop appearing on stdout. On a server whose runbook is `docker logs`, that reads as the app having stopped logging.++`OTEL_PYTHON_LOG_CORRELATION=true` fixes both. It calls `logging.basicConfig()`, which restores a console handler and sets the root level, and it stamps `otelTraceID` / `otelSpanID` onto every record so your own log lines carry the trace id too:++```bash+export OTEL_PYTHON_LOG_CORRELATION=true+export OTEL_PYTHON_LOG_LEVEL=info+```++`OTEL_PYTHON_LOG_LEVEL` on its own does nothing. It is only read inside the correlation branch, so without `OTEL_PYTHON_LOG_CORRELATION=true` the root logger stays at `WARNING`. With correlation on and no level set, the level defaults to `info`. Setting both is the readable form.++The equivalent in code, if you would rather not use the variables, is `logging.basicConfig(level=logging.INFO)` early in your app.++With both set, a handler like this lands in Traceway and on the endpoint's **Logs** card:++```python+import logging++logger = logging.getLogger(__name__)++[email protected]("/users/{user_id}")+def get_user(user_id: str):+ logger.info("looking up user %s", user_id)+ return {"id": user_id}+```++Structured attributes go through `extra`, and dotted keys match the OTel semantic conventions:++```python+logger.info("order received", extra={"order.id": "ord_123"})+```++> **Do not set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`.** On current versions the bridge is on by default, and that variable switches it to the SDK's deprecated handler. The instrumentation then logs a duplicate-handler warning, which is itself ingested as a log record.++## Background Tasks++A cron job or queue worker becomes a **Task** only when its root span has `SpanKind.CONSUMER`. A root span with the default `INTERNAL` kind and no HTTP attributes is discarded on ingest, and Traceway still answers `200 OK`, so a plain `start_as_current_span("my-job")` gets you silence.++In Python the kind is a keyword argument, not an options object:++```python+from opentelemetry import trace+from opentelemetry.trace import SpanKind, Status, StatusCode++tracer = trace.get_tracer("my-app")+++def run_scheduled_job():+ with tracer.start_as_current_span(+ "cleanup-expired-sessions", kind=SpanKind.CONSUMER+ ) as span:+ try:+ do_work()+ span.set_status(Status(StatusCode.OK))+ except Exception as error:+ span.record_exception(error)+ span.set_status(Status(StatusCode.ERROR, str(error)))+ raise+```++Run it as `opentelemetry-instrument python worker.py`. The span name is the task name and the grouping key, so keep it stable and put job ids, batch sizes, and timestamps in span attributes instead.++## Custom Metrics++Under `opentelemetry-instrument` the meter provider is already configured, so ask the global API for a meter and record:++```python+from opentelemetry import metrics++meter = metrics.get_meter("my-app")++sessions_cleaned = meter.create_counter(+ "app.sessions_cleaned",+ unit="1",+ description="expired sessions removed by the cleanup job",+)+sessions_cleaned.add(7, {"job": "cleanup-expired-sessions"})++checkout_latency = meter.create_histogram("app.checkout.latency", unit="ms")+checkout_latency.record(12.5, {"tier": "pro"})+```++The counter appears in metric discovery as `app.sessions_cleaned`, with `job` as a tag you can group by. A histogram arrives split in two, `app.checkout.latency.avg` and `app.checkout.latency.count`, which is how Traceway stores every histogram. See [Metrics](/client/otel/metrics).++> **Metrics export every 60 seconds by default.** That is the SDK default for `OTEL_METRIC_EXPORT_INTERVAL`, so a working pipeline still looks empty for the first minute. Set `OTEL_METRIC_EXPORT_INTERVAL=10000` while you are verifying the integration.++## Child Spans and Database Calls++Database and outgoing HTTP calls appear as child spans on the endpoint's waterfall, named after the SQL statement or the request. One detail costs people an afternoon:++> **The DB-API instrumentation traces cursor calls.** `connection.cursor().execute(...)` produces a child span named after the SQL. The `connection.execute(...)` shortcut bypasses the traced cursor proxy and produces no span at all. If your queries are missing from the waterfall while your outgoing HTTP calls are there, that is usually why.++For business logic that no instrumentation covers, start your own child span. It stays attached to the request because it runs inside the active context:++```python+from opentelemetry import trace++tracer = trace.get_tracer("my-app")++with tracer.start_as_current_span("rebuild-price-index"):+ rebuild()+```++## Verify It Worked++Telemetry is batched, so give traces about 10 seconds and metrics up to a minute. Set the dashboard time picker to the last 15 minutes, then check:++1. **Endpoints.** Three requests to a parametrized route give one row with count 3. Three rows means `http.route` is missing.+2. **Issues.** A route that raises shows up with its Python traceback, and its endpoint row reports status 500.+3. **Spans.** Open that endpoint and confirm your database and outgoing HTTP calls are in the waterfall.+4. **Logs.** A `logger.info(...)` from inside a handler appears on the endpoint's **Logs** card, on the **This Trace** tab. If only WARN and above are there, `OTEL_PYTHON_LOG_CORRELATION` is not set.+5. **Tasks.** A job run under `opentelemetry-instrument` with `kind=SpanKind.CONSUMER` appears by its span name.+6. **Metrics.** `http.server.duration.avg` and your custom metric names appear in metric discovery.++Nothing at all in any of the six, with the app running and no error printed? Check `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` first. Then set `OTEL_PYTHON_LOG_CORRELATION=true` and restart. Exporter failures are logged at WARNING and ERROR, but the OTel handler is the only handler on the root logger until something calls `logging.basicConfig()`, so a broken exporter swallows its own error report. Correlation restores the console handler and the failures appear on stderr. `OTEL_LOG_LEVEL` does not help here: opentelemetry-python declares the variable but no released package reads it.++## Next Steps++- [Traces](/client/otel/traces): how OTel spans map to Endpoints, Tasks, and Issues+- [Logs](/client/otel/logs): log record mapping and severity handling+- [Metrics](/client/otel/metrics): supported metric types and histogram handling+- [Django](/client/otel/django): the two extra steps Django needs+- [Overview](/client/otel): protocol details, authentication, and a "nothing is showing up" checklistdiff --git a/docs/pages/client/symfony/_meta.json b/docs/pages/client/otel/symfony/_meta.jsonsimilarity index 100%rename from docs/pages/client/symfony/_meta.jsonrename to docs/pages/client/otel/symfony/_meta.jsondiff --git a/docs/pages/client/otel/symfony/exceptions.mdx b/docs/pages/client/otel/symfony/exceptions.mdxnew file mode 100644index 00000000..16021e3e--- /dev/null+++ b/docs/pages/client/otel/symfony/exceptions.mdx@@ -0,0 +1,160 @@+# Exceptions++The bundle automatically captures unhandled exceptions and records them on the request span. Anything that escapes your controller becomes an Issue with a full PHP stack trace, and the response is a 500. You do not have to do anything for that case.++This page is about the other case: errors you catch and handle, but still want to see in Traceway.++## Recording an Exception on the Current Span++When you catch an error but want it reported, record it as an event on the active span:++```php+use OpenTelemetry\API\Trace\Span;+use OpenTelemetry\API\Trace\StatusCode;++try {+ $this->paymentGateway->charge($order);+} catch (\Throwable $e) {+ $span = Span::getCurrent();+ $span->recordException($e);+ $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());++ throw $e;+}+```++`recordException` adds an `exception` event to the span carrying the type, message, and stack trace. Traceway turns those events into Issues, grouped by a hash of the normalized stack trace.++`Span::getCurrent()` returns whatever span is active right now. Inside a controller that is the request span, inside a Messenger handler it is the task span, and inside a `TracingInterface` callback it is your own span. There is no separate API per context.++Swallowing the exception (not re-throwing) still produces the Issue, but the request will report the status code you actually return. Re-throw when you want the endpoint marked as failed too.++## Adding Context++Put context on the **span**, with `setAttribute()`:++```php+use OpenTelemetry\API\Trace\Span;+use OpenTelemetry\API\Trace\StatusCode;++$span = Span::getCurrent();++$span->setAttribute('user.id', $userId);+$span->setAttribute('order.id', $orderId);++try {+ $this->processOrder($orderId);+} catch (\Throwable $e) {+ $span->setAttribute('order.status', 'failed');+ $span->setAttribute('retry.count', $retryCount);++ $span->recordException($e);+ $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());+ throw $e;+}+```++The Issue in the dashboard now carries `user.id`, `order.id`, `order.status`, and `retry.count` alongside the automatic HTTP attributes.++> **Do not use the second argument of `recordException()` for this.** Those attributes live on the OpenTelemetry exception *event*. Traceway builds an Issue's attribute map from the owning **span**, so event attributes never reach the dashboard. Anything you want to read or filter on has to go through `setAttribute()`.++Attribute values must be scalars or arrays of scalars. Cast objects and enums yourself, for example `$order->status->value`.++## Capturing Exceptions in Services++For exceptions in services outside of a controller, get the current span the same way:++```php+namespace App\Service;++use OpenTelemetry\API\Trace\Span;+use OpenTelemetry\API\Trace\StatusCode;++class PaymentService+{+ public function processPayment(string $customerId, float $amount): void+ {+ $span = Span::getCurrent();++ try {+ $this->gateway->charge($customerId, $amount);+ } catch (\Throwable $e) {+ $span->setAttribute('customer.id', $customerId);+ $span->setAttribute('payment.amount', $amount);++ $span->recordException($e);+ $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());+ throw $e;+ }+ }+}+```++Note that this attaches the attributes to whatever span is active, which in a controller-driven request is the request span. If you want the failure isolated to its own span, wrap the work with `TracingInterface` first. See [Spans](/client/otel/symfony/spans).++## Custom Error Types++Custom exception classes work the same way. The Issue is titled with the class name and the message:++```php+namespace App\Exception;++class InsufficientFundsException extends \RuntimeException+{+ public function __construct(+ public readonly string $accountId,+ public readonly float $requested,+ public readonly float $available,+ ) {+ parent::__construct("Insufficient funds: requested {$requested}, available {$available}");+ }+}+```++```php+use App\Exception\InsufficientFundsException;+use OpenTelemetry\API\Trace\Span;+use OpenTelemetry\API\Trace\StatusCode;++try {+ $this->withdraw($accountId, $amount);+} catch (InsufficientFundsException $e) {+ $span = Span::getCurrent();++ $span->setAttribute('account.id', $e->accountId);+ $span->setAttribute('amount.requested', $e->requested);+ $span->setAttribute('amount.available', $e->available);++ $span->recordException($e);+ $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());+}+```++## How Issues Are Grouped++Traceway hashes the normalized stack trace, not the message. Values that change between runs (ids, addresses, absolute paths, and any run of 5 or more digits, which covers epoch timestamps) are stripped before hashing, so the same failure from different users lands on one Issue.++That is also why message-only differences do not split a group, as long as the exception class name has no backslash in it. `RuntimeException: requested 50, available 10` and `RuntimeException: requested 900, available 10` are one Issue raised twice, which is what you want. Put the varying numbers in span attributes so you can still see them per occurrence.++**Namespaced classes are the exception.** Traceway strips the message only when the part before the first colon looks like a type name (letters, digits, dots and underscores). `App\Exception\InsufficientFundsException` contains backslashes, so it is not recognised, the message stays in the hash, and every distinct message becomes its own Issue. Keep the message text constant for those classes and put the varying values in span attributes:++```php+namespace App\Exception;++class InsufficientFundsException extends \RuntimeException+{+ public function __construct(+ public readonly int $requested,+ public readonly int $available,+ ) {+ parent::__construct('Insufficient funds');+ }+}+```++Then record the numbers on the span, where they stay visible per occurrence without splitting the group:++```php+$span->setAttribute('order.requested', $e->requested);+$span->setAttribute('order.available', $e->available);+```diff --git a/docs/pages/client/otel/symfony/index.mdx b/docs/pages/client/otel/symfony/index.mdxnew file mode 100644index 00000000..0557c38a--- /dev/null+++ b/docs/pages/client/otel/symfony/index.mdx@@ -0,0 +1,335 @@+# Symfony++The `traceway/opentelemetry-symfony` bundle automatically instruments your Symfony application and exports traces, metrics, and logs to Traceway's OTLP endpoints. No manual instrumentation is needed.++## Prerequisites++- PHP 8.1+ (no PECL extension needed, the bundle is pure PHP)+- Symfony 6.4 LTS, 7.x, or 8.x+- Composer+- A Traceway project (create one in the dashboard)++> **Note:** the upstream `open-telemetry/opentelemetry-auto-*` instrumentations require the `ext-opentelemetry` PECL extension. This bundle does not, so it installs on managed hosts where you cannot build extensions. `ext-protobuf` is only worth adding if you switch the exporter to `http/protobuf`.++## Step 1: Install Packages++```bash+composer require traceway/opentelemetry-symfony open-telemetry/exporter-otlp php-http/guzzle7-adapter+```++The bundle pulls in `open-telemetry/sdk` for you. `open-telemetry/exporter-otlp` and `php-http/guzzle7-adapter` are the OTLP/HTTP exporter and its HTTP client, and both are needed at runtime to actually ship data to Traceway.++## Step 2: Register the Bundle++Flex does not do this for you on a stock Symfony app. The bundle's recipe lives in `symfony/recipes-contrib`, and new apps ship with `extra.symfony.allow-contrib` set to `false`, so `composer require` prints `IGNORING traceway/opentelemetry-symfony` and registers nothing. Add it to `config/bundles.php` yourself:++```php+// config/bundles.php+return [+ // ...+ Traceway\OpenTelemetryBundle\OpenTelemetryBundle::class => ['all' => true],+];+```++The bundle is what produces the spans. Without it registered, the SDK starts but nothing instruments your app, and Traceway stays empty.++## Step 3: Configure Environment Variables++Add the following to your `.env` file, replacing the endpoint and token with your project values:++```bash+OTEL_SERVICE_NAME=my-symfony-app+OTEL_TRACES_EXPORTER=otlp+OTEL_METRICS_EXPORTER=otlp+OTEL_LOGS_EXPORTER=otlp+OTEL_EXPORTER_OTLP_PROTOCOL=http/json+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+```++Your Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.++> **Do not put `OTEL_PHP_AUTOLOAD_ENABLED` in `.env`.** OpenTelemetry PHP decides whether to start during Composer autoload, which happens before Symfony's Dotenv reads `.env`. The variable is always read too late there. Worse, once Dotenv has loaded it the bundle's own fallback (Step 4) sees it as already set and skips starting the SDK, so every signal silently becomes a no-op. Step 4 shows the two places the flag actually works.++> **Note:** `http/json` needs no extra extensions and is the easiest starting point. `http/protobuf` also works, but the pure-PHP protobuf encoder is much slower. Install `ext-protobuf` (`pecl install protobuf`) if you switch to it in production. `grpc` is not included out of the box and additionally needs `ext-grpc` plus `open-telemetry/transport-grpc`.++## Step 4: Turn the SDK On++The instrumentation bundle and the OpenTelemetry SDK are two different things. Step 2 wired the bundle. This step starts the SDK that actually exports data. Pick one of the two options below.++### Option A (recommended): let the bundle start it++Create `config/packages/open_telemetry.yaml`:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ sdk:+ autoload_enabled: true+```++The bundle sets `OTEL_PHP_AUTOLOAD_ENABLED` during its `boot()` and loads the SDK's autoload file itself. This works everywhere, including `symfony server:start` and `php -S`, and it works with Symfony Secrets. It is the only option that does not need access to the server config.++### Option B: set a real process environment variable++If you control the process environment, set the flag there instead and skip the YAML key:++```ini+; php-fpm pool config+env[OTEL_PHP_AUTOLOAD_ENABLED] = true+```++```dockerfile+ENV OTEL_PHP_AUTOLOAD_ENABLED=true+```++```apache+SetEnv OTEL_PHP_AUTOLOAD_ENABLED true+```++Pick one option, not both.++### Leave `public/index.php` alone++You never call `SdkAutoloader::autoload()` by hand, and you never edit the front controller. `open-telemetry/sdk` registers its `_autoload.php` as a Composer `files` autoload entry, so it runs on every request already. The stock Symfony front controller is correct as generated:++```php+<?php+// public/index.php (leave this file exactly as Symfony generated it)++use App\Kernel;++require_once dirname(__DIR__).'/vendor/autoload_runtime.php';++return function (array $context) {+ return new Kernel($context['APP_ENV'], (bool) $context['APP_DEBUG']);+};+```++Replacing `vendor/autoload_runtime.php` with `vendor/autoload.php` drops `symfony/runtime`, which is what boots Dotenv. Your `.env` then never loads, so both the OTEL variables and every `%env(...)%` in your config break. A typical app starts throwing `Environment variable not found: "DATABASE_URL"`.++## Step 5: Verify++```bash+bin/console cache:clear+bin/console traceway:doctor+```++A healthy setup looks like this:++```+Traceway Doctor+═══════════════++Runtime+ ○ protocol is http/json; ext-protobuf not required+ ✓ ext-opentelemetry not loaded (no conflict risk)+ ○ protocol is http/json; gRPC transport not required++SDK configuration+ ✓ OTEL_SERVICE_NAME = "my-symfony-app"+ ✓ OTEL_TRACES_EXPORTER = otlp+ ✓ OTEL_EXPORTER_OTLP_ENDPOINT = https://your-traceway-instance.com/api/otel+ ✓ OTEL_EXPORTER_OTLP_PROTOCOL = http/json+ ✓ OTEL_TRACES_SAMPLER unset (defaults to parentbased_always_on)+ ✓ TracerProvider is TracerProvider++Bundle configuration+ ○ propagator=w3c, id_generator=default; X-Ray not configured+ ✓ Messenger tracing enabled and symfony/messenger is installed+ ✓ Log export wired (LoggerProvider: LoggerProvider)++Connectivity+ ✓ OTLP endpoint reachable (HTTP 404, 6ms)++Results: 10 ok, 0 warning, 0 error, 3 skipped, 0 info+```++Your counts will differ. The Messenger and Connectivity lines above need `symfony/messenger` and `symfony/http-client` to be installed. On an app without them you get `⚠ traces.messenger.enabled is true but symfony/messenger is not installed` and a skipped reachability probe. Neither affects HTTP tracing.++The line that matters most is `TracerProvider`. If you see this instead:++```+ ✗ TracerProvider is NoopTracerProvider — spans are silently dropped+```++the SDK never started. Nothing in your code will be exported, and there will be no error anywhere to tell you. Go back to Step 4 and make sure exactly one of the two options is in place, and that `OTEL_PHP_AUTOLOAD_ENABLED` is not in `.env`.++`bin/console traceway:doctor --format=json --skip-network` prints a stable JSON envelope, so you can gate CI on it.++### Exclude `cache:clear` from console tracing++In the `dev` environment `bin/console cache:clear` prints a red block like this:++```+In DebugScope.php line 79:++ User Notice: Scope: missing call to Scope::detach() for scope #608, created+ at OpenTelemetry.Context.Context.activate(Context.php:82)+```++The cache is still cleared and the command still exits `0`. The notice comes from OpenTelemetry's debug scope tracking, which only runs when `kernel.debug` is on, and `cache:clear` rebuilds the kernel underneath the console span. Add `cache:clear` to the excluded command list and it goes away:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ traces:+ console:+ excluded_commands: ['messenger:consume', 'messenger:consume-messages', 'cache:clear']+```++You lose nothing by excluding it. `cache:clear` is a build step, not a job worth tracking on the Tasks page.++## Complete Working Configuration++Everything above in one place. This is the full set of files needed for endpoints, issues, spans, tasks, logs, and metrics.++```bash+composer require traceway/opentelemetry-symfony open-telemetry/exporter-otlp php-http/guzzle7-adapter+composer require symfony/monolog-bundle # only if you want logs+```++```php+// config/bundles.php (add this yourself, Flex skips the contrib recipe)+return [+ // ...+ Traceway\OpenTelemetryBundle\OpenTelemetryBundle::class => ['all' => true],+];+```++```bash+# .env (note that OTEL_PHP_AUTOLOAD_ENABLED is deliberately not here)+OTEL_SERVICE_NAME=my-symfony-app+OTEL_TRACES_EXPORTER=otlp+OTEL_METRICS_EXPORTER=otlp+OTEL_LOGS_EXPORTER=otlp+OTEL_EXPORTER_OTLP_PROTOCOL=http/json+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+```++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ sdk:+ autoload_enabled: true # starts the SDK; Dotenv runs too late to do it+ traces:+ excluded_paths: ['/_profiler', '/_wdt', '/health']+ console:+ excluded_commands: ['messenger:consume', 'messenger:consume-messages', 'cache:clear']+ messenger:+ root_spans: true # each consumed message becomes its own Task+ metrics:+ enabled: true+ http_server:+ enabled: true+ logs:+ export:+ enabled: true+ level: debug # raise to info or warning in production+```++Leave `public/index.php` exactly as Symfony generated it. Then run `bin/console cache:clear` and `bin/console traceway:doctor`.++## What Gets Captured++Once configured, the bundle automatically captures:++- **Endpoints**: every HTTP request, grouped by route template (`GET /users/{id}`, not `GET /users/42`)+- **Status codes**: the real response code, including 500s from unhandled exceptions+- **Exceptions**: unhandled errors become Issues with a full PHP stack trace (see [Exceptions](/client/otel/symfony/exceptions))+- **Client IP, body size, user agent**: standard HTTP attributes+- **Doctrine queries**: child spans per query if `doctrine/dbal` is installed (see [Spans](/client/otel/symfony/spans))+- **Twig renders, cache operations, outgoing HttpClient calls, Mailer sends**: child spans, each individually toggleable+- **Console commands**: every `bin/console` run lands on the Tasks page (see [Tasks](/client/otel/symfony/tasks))+- **Messenger jobs**: dispatched and consumed messages (see [Tasks](/client/otel/symfony/tasks))+- **Monolog records**: trace id and span id injected into every log record, plus opt-in forwarding to Traceway (see [Logs](/client/otel/symfony/logs))++Unrouted requests such as 404s group under `UNMATCHED` rather than leaking the raw URL into your endpoint list.++## Configuration Reference++Every option is optional. The keys below are the v2.0+ nested schema. The old flat keys (`excluded_paths`, `messenger_root_spans`, `log_export_enabled`, and friends) still load but emit a deprecation and are removed in v4.0.++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ sdk:+ autoload_enabled: true # start the SDK from bundle config+ resource_attributes:+ service.version: '%env(APP_VERSION)%' # becomes the App Version column+ traces:+ excluded_paths: # matched as path prefixes+ - /_profiler+ - /_wdt+ - /health+ record_client_ip: true # default true, set false for GDPR+ error_status_threshold: 500 # minimum 500, a value of 400-499 fails at boot+ console:+ enabled: true+ excluded_commands: ['messenger:consume', 'messenger:consume-messages']+ messenger:+ root_spans: false # true = each consumed message is its own trace+ doctrine:+ record_statements: false # true records the SQL text on the span+ http_client:+ enabled: true # OTLP endpoint is auto-excluded+ cache:+ enabled: true+ twig:+ enabled: true+ mailer:+ enabled: true+ record_subject: false # subjects can contain personal data+ metrics:+ enabled: false # master switch, off by default+ http_server:+ enabled: false+ logs:+ correlation:+ enabled: true # trace_id / span_id in every log record+ export:+ enabled: false # forward Monolog records to Traceway+ level: debug+```++Setting `service.version` is worth doing early. It fills Traceway's **App Version** column, which is how you tell one release apart from the next when an Issue starts spiking.++The on/off switches above are read at compile time to decide which services get wired, so they must be plain `true` or `false` values. Use separate `config/packages/dev/` and `config/packages/prod/` files to vary them per environment. `%env(...)%` placeholders do work inside `sdk.resource_attributes` and `sdk.exporter_otlp_headers`, because those are only read at boot.++## Test Your Integration++Add a test route to verify data is flowing:++```php+// src/Controller/TestController.php+namespace App\Controller;++use Symfony\Component\HttpFoundation\Response;+use Symfony\Component\Routing\Attribute\Route;++class TestController+{+ #[Route('/testing', name: 'testing')]+ public function index(): Response+ {+ throw new \RuntimeException("Test error from Traceway integration");+ }+}+```++Visit `/testing` in your browser. Within a few seconds you should see two things in the dashboard:++- **Endpoints** shows `GET /testing` with status code 500+- **Issues** shows `RuntimeException: Test error from Traceway integration` with a PHP stack trace pointing at `TestController.php`++Both come from the same request. Symfony's error listener turns the throwable into a 500 response, and the bundle records the exception on the request span, so the endpoint row and the Issue agree.++## Next Steps++- [Exceptions](/client/otel/symfony/exceptions): manually capture caught exceptions with context+- [Spans](/client/otel/symfony/spans): create custom spans to measure sub-operations+- [Tasks](/client/otel/symfony/tasks): trace Symfony Messenger jobs and console commands+- [Metrics](/client/otel/symfony/metrics): track custom counters, histograms, and gauges+- [Logs](/client/otel/symfony/logs): forward Monolog / PSR-3 logs to Traceway+- [OpenTelemetry docs](/client/otel): how OTel traces, metrics, and logs map to Tracewaydiff --git a/docs/pages/client/otel/symfony/logs.mdx b/docs/pages/client/otel/symfony/logs.mdxnew file mode 100644index 00000000..733ed4c6--- /dev/null+++ b/docs/pages/client/otel/symfony/logs.mdx@@ -0,0 +1,217 @@+# Logs++Forward your Symfony application logs to Traceway over the OpenTelemetry logs endpoint. The bundle ships its own Monolog handler, so all of your existing `$logger->info(...)` / `$logger->error(...)` calls keep working, and every record is linked to the trace and span that was active when you logged it.++## Install Monolog++```bash+composer require symfony/monolog-bundle+```++> Symfony's PSR-3 logger comes from `symfony/monolog-bundle`. The `symfony/skeleton` starter does not include it, so install it first. Without it `$this->logger->info(...)` goes nowhere, `config/packages/monolog.yaml` is ignored, and the bundle refuses to boot with log export turned on.++You do not need `open-telemetry/opentelemetry-logger-monolog`. The bundle has its own handler.++## Enable Log Export++Two lines of YAML, and no changes to `config/packages/monolog.yaml`:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ logs:+ export:+ enabled: true+ level: debug # raise to info or warning in production+```++The bundle registers its handler into Monolog's handler stack for you, on every channel. `level` is the minimum Monolog level that gets forwarded. Start at `debug` while you verify the pipeline, then raise it once records are arriving.++Make sure the logs exporter is on in `.env` too. This is already in the [Quick Start](/client/otel/symfony) env block:++```bash+OTEL_LOGS_EXPORTER=otlp+```++`OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL. The logs exporter appends `/v1/logs` itself, so there is no extra endpoint to configure.++Run `bin/console cache:clear` and then `bin/console traceway:doctor`. A working setup reports:++```+ ✓ Log export wired (LoggerProvider: LoggerProvider)+```++If it reports `logs.export.enabled is false`, the setting has not taken effect. Check the file path, the indentation, and that you actually cleared the cache.++## Filter Out the `event` Channel++The bundle's handler listens on every channel, including the `event` channel that Symfony's own default `monolog.yaml` excludes from all of its handlers with `channels: ["!event"]`. That channel carries one `Notified event "{event}" to listener "{listener}".` record per listener per request. If `symfony/stopwatch` is installed, which most apps get through `doctrine/doctrine-bundle` or the debug pack, a single request that logs 4 records of its own ships 26 records to Traceway.++Apply the same exclusion to the bundle's handler. Its Monolog handler key is `opentelemetry`:++```yaml+# config/packages/monolog.yaml+monolog:+ handlers:+ opentelemetry:+ channels: ["!event"]+```++Do not add `type` or `id` here. The bundle already set them, and this block only merges the channel filter on top. After `bin/console cache:clear`, the same request ships 5 records: your 4 plus Symfony's `Matched route`.++`channels` takes an exclude list (`["!event", "!doctrine"]`) or an include list (`["app"]`). Use it to drop any other chatty channel the same way.++## Emit Logs With Trace Context++Inject Symfony's standard `Psr\Log\LoggerInterface` as usual. Trace context attaches automatically:++```php+// src/Controller/OrderController.php+namespace App\Controller;++use Psr\Log\LoggerInterface;+use Symfony\Component\HttpFoundation\JsonResponse;+use Symfony\Component\HttpFoundation\Response;+use Symfony\Component\Routing\Attribute\Route;++class OrderController+{+ public function __construct(+ private readonly LoggerInterface $logger,+ ) {}++ #[Route('/orders', methods: ['POST'])]+ public function create(): Response+ {+ $this->logger->info('order received', ['order.id' => 'ord_123']);++ try {+ // ... business logic ...+ $this->logger->info('order processed', ['order.id' => 'ord_123']);+ return new JsonResponse(['status' => 'ok']);+ } catch (\Throwable $e) {+ $this->logger->error('order failed', [+ 'order.id' => 'ord_123',+ 'exception' => $e,+ ]);+ throw $e;+ }+ }+}+```++Because the controller runs inside the instrumented request span, every log emitted here carries that span's `trace_id` and `span_id`. Open the endpoint's trace in the dashboard and the **Logs** tab shows these records attached to it.++The same is true for logs emitted inside Messenger handlers, console commands, and any other code that runs under an active span.++Two things about the context array:++- Each key becomes a log attribute you can filter on. `['order.id' => 'ord_123']` is stored as the attribute `order.id`.+- A key named `exception` holding a `Throwable` is special. Pass the object, not `$e->getMessage()`, and the record gains `exception.type`, `exception.message`, and `exception.stacktrace` attributes.++Logging an exception this way does not create an Issue. Issues come from exceptions recorded on spans, which is covered in [Exceptions](/client/otel/symfony/exceptions). Do both when you want the error searchable in Logs and tracked as an Issue.++The Monolog channel becomes the log's scope name, so records from the `doctrine` or `security` channels are distinguishable from your own `app` records.++## Trace Correlation Without Export++Log export is one half of the story. The bundle also injects `trace_id`, `span_id`, and `trace_flags` into every Monolog record, which is what makes your local `var/log/dev.log` (or your existing Elastic/Loki pipeline) correlate with Traceway traces. That part is on by default and independent of export:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ logs:+ correlation:+ enabled: true # default+```++## Severity Mapping++Monolog levels map to OpenTelemetry severity numbers like this, and Traceway stores the level name it receives:++| Monolog Level | OTLP Severity Number | Shown in Traceway |+| ------------- | -------------------- | ----------------- |+| `DEBUG` | 5 | DEBUG |+| `INFO` | 9 | INFO |+| `NOTICE` | 10 | NOTICE |+| `WARNING` | 13 | WARNING |+| `ERROR` | 17 | ERROR |+| `CRITICAL` | 18 | CRITICAL |+| `ALERT` | 19 | ALERT |+| `EMERGENCY` | 21 | EMERGENCY |++The severity **filter** in the dashboard buckets by number, not by name. `WARN+` starts at 13 and so matches WARNING. `ERROR+` starts at 17 and so matches ERROR, CRITICAL, ALERT, and EMERGENCY.++Only records at or above the configured `level` are forwarded.++## Test Your Integration++Add a route that logs at multiple levels, hit it once, and check the **Logs** page in the dashboard:++```php+// src/Controller/LogTestController.php+namespace App\Controller;++use Psr\Log\LoggerInterface;+use Symfony\Component\HttpFoundation\JsonResponse;+use Symfony\Component\HttpFoundation\Response;+use Symfony\Component\Routing\Attribute\Route;++class LogTestController+{+ public function __construct(+ private readonly LoggerInterface $logger,+ ) {}++ #[Route('/log-test', name: 'log_test')]+ public function test(): Response+ {+ $this->logger->debug('debug sample');+ $this->logger->info('info sample', ['request.id' => 'req_1']);+ $this->logger->warning('warning sample');+ $this->logger->error('error sample', ['order.id' => 'ord_1']);+ return new JsonResponse(['emitted' => 4]);+ }+}+```++Visit `/log-test`, then open **Logs** in the dashboard. With `level: debug` you should see all four records within a few seconds, each carrying the trace id of that request. At `level: info` the `debug sample` record is filtered out before it reaches the handler, so you get three.++Symfony also logs on your behalf, so expect a framework record such as `Matched route "log_test".` on the same trace. If you see twenty or more extra `Notified event` records instead, apply the channel filter from [Filter Out the `event` Channel](#filter-out-the-event-channel) above.++## Using the Contrib Handler Instead++The community package `open-telemetry/opentelemetry-logger-monolog` also works, but it needs more care in Symfony and there is no benefit over the bundle's handler. Use one or the other, never both, or every record is sent twice. If you wire it anyway, note two things.++First, nothing registers `OpenTelemetry\API\Logs\LoggerProviderInterface` as a service, so you have to add the factory yourself or the container will not compile:++```yaml+# config/packages/monolog.yaml+monolog:+ handlers:+ traceway:+ type: service+ id: OpenTelemetry\Contrib\Logs\Monolog\Handler+ level: info+ channels: ~++services:+ OpenTelemetry\API\Logs\LoggerProviderInterface:+ factory: ['OpenTelemetry\API\Globals', 'loggerProvider']++ OpenTelemetry\Contrib\Logs\Monolog\Handler:+ arguments:+ $loggerProvider: '@OpenTelemetry\API\Logs\LoggerProviderInterface'+ $level: !php/const Monolog\Logger::INFO+ $bubble: true+```++Second, that handler resolves the logger provider when the container builds the `logger` service, which happens inside `FrameworkBundle::boot()`. That runs before any bundle can start the SDK, and OpenTelemetry PHP caches its providers on first access. So with this handler you **must** use Option B from Step 4 of the [Quick Start](/client/otel/symfony) and set `OTEL_PHP_AUTOLOAD_ENABLED=true` as a real process environment variable. With `open_telemetry.sdk.autoload_enabled` the SDK starts too late, and traces, metrics, and logs all silently become no-ops. Run `bin/console traceway:doctor` and confirm it says `TracerProvider is TracerProvider`.++The bundle's own handler resolves the provider lazily, on the first log write, which is why it has no such ordering requirement.++## Next Steps++- [Exceptions](/client/otel/symfony/exceptions): record caught exceptions with context+- [Metrics](/client/otel/symfony/metrics): custom counters, histograms, and gauges+- [OTel Logs reference](/client/otel/logs): full OTLP logs mapping and severity detailsdiff --git a/docs/pages/client/otel/symfony/metrics.mdx b/docs/pages/client/otel/symfony/metrics.mdxnew file mode 100644index 00000000..8407efff--- /dev/null+++ b/docs/pages/client/otel/symfony/metrics.mdx@@ -0,0 +1,220 @@+# Metrics++Metrics are off by default in the bundle. This page covers turning on the automatic ones, and recording your own counters, histograms, gauges, and up/down counters.++Everything here needs `OTEL_METRICS_EXPORTER=otlp` in `.env`. That is already in the [Quick Start](/client/otel/symfony) env block.++## Turn Metrics On++`metrics.enabled` is the master switch. Each subsystem then has its own toggle, so you only pay for what you want:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ metrics:+ enabled: true+ http_server:+ enabled: true+ http_client:+ enabled: true+ messenger:+ enabled: true+ doctrine:+ enabled: true+```++Enabling a subsystem without `metrics.enabled: true` is a configuration error and fails at boot, so you cannot half-configure this by accident.++With `http_server` on, these arrive without any code:++| Metric | Kind | What it measures |+|---|---|---|+| `http.server.request.duration` | Histogram | Request latency, tagged with `http.route` and `http.response.status_code` |+| `http.server.active_requests` | UpDownCounter | Requests currently in flight |+| `http.server.request.body.size` | Histogram | Request body bytes |+| `http.server.response.body.size` | Histogram | Response body bytes |++The other subsystems add `messaging.process.duration` and `messaging.client.consumed.messages` for Messenger, `db.client.operation.duration` for Doctrine, and `http.client.request.duration` for outgoing HttpClient calls.++Traceway splits every histogram into an `.avg` and a `.count` series, so `http.server.request.duration` shows up in the dashboard as `http.server.request.duration.avg` and `http.server.request.duration.count`. See [OTel metrics](/client/otel/metrics#histogram-handling).++## Recording Your Own Metrics++Inject `MeterRegistryInterface`. It hands you cached instruments, so calling `counter('orders.created')` twice returns the same instrument rather than creating a second one:++```php+// src/Service/OrderService.php+namespace App\Service;++use App\Entity\Order;+use Traceway\OpenTelemetryBundle\Metrics\MeterRegistryInterface;++class OrderService+{+ public function __construct(+ private readonly MeterRegistryInterface $metrics,+ ) {}++ public function createOrder(array $items): Order+ {+ $order = $this->doCreateOrder($items);++ $this->metrics->counter('orders.created', 'orders')->add(1, ['plan' => $order->plan]);++ return $order;+ }+}+```++This service only exists when `metrics.enabled: true`. If you leave metrics off, use `Globals::meterProvider()` instead (see the bottom of this page).++### Counter++Counters only go up. Use them for totals: orders placed, emails sent, cache hits.++```php+$orders = $this->metrics->counter('orders.created', 'orders', 'Total orders created');++$orders->add(1);+$orders->add(1, ['plan' => 'pro', 'region' => 'eu']);+```++The array is the attribute set. Each distinct combination becomes its own series, so keep the values low cardinality. `plan` and `region` are fine, a customer id is not.++### Histogram++Histograms track a distribution: durations, payload sizes, item counts per batch.++```php+$duration = $this->metrics->histogram('order.processing_ms', 'ms', 'Order processing time');++$start = hrtime(true);+$this->processOrder($order);+$duration->record((hrtime(true) - $start) / 1e6);+```++### Gauge++Gauges are point-in-time readings that can go up or down. Record the current value whenever you know it:++```php+$this->metrics->gauge('queue.depth', 'items', 'Pending jobs')+ ->record($this->queue->count(), ['queue' => 'async']);+```++### Up/Down Counter++Use an up/down counter when you want to track a value by its changes rather than its absolute reading, such as items currently being processed:++```php+$active = $this->metrics->upDownCounter('jobs.active', 'jobs', 'Jobs in flight');++$active->add(1);+try {+ $this->run($job);+} finally {+ $active->add(-1);+}+```++## Grouping Related Instruments++Rather than looking up instruments by string all over the codebase, declare them once in a small service:++```php+// src/Telemetry/AppMetrics.php+namespace App\Telemetry;++use OpenTelemetry\API\Metrics\CounterInterface;+use OpenTelemetry\API\Metrics\HistogramInterface;+use Traceway\OpenTelemetryBundle\Metrics\MeterRegistryInterface;++class AppMetrics+{+ public readonly CounterInterface $ordersCreated;+ public readonly HistogramInterface $orderProcessingMs;+ public readonly CounterInterface $paymentSuccess;+ public readonly CounterInterface $paymentFailed;++ public function __construct(MeterRegistryInterface $metrics)+ {+ $this->ordersCreated = $metrics->counter('orders.created', 'orders');+ $this->orderProcessingMs = $metrics->histogram('orders.processing_ms', 'ms');+ $this->paymentSuccess = $metrics->counter('payments.success', 'payments');+ $this->paymentFailed = $metrics->counter('payments.failed', 'payments');+ }+}+```++Then inject `AppMetrics` wherever you need it:++```php+// src/Service/PaymentService.php+namespace App\Service;++use App\Entity\Order;+use App\Telemetry\AppMetrics;++class PaymentService+{+ public function __construct(+ private readonly AppMetrics $metrics,+ ) {}++ public function processPayment(Order $order): void+ {+ $start = hrtime(true);++ try {+ $this->gateway->charge($order->total);+ $this->metrics->paymentSuccess->add(1, ['plan' => $order->plan]);+ } catch (\Throwable $e) {+ $this->metrics->paymentFailed->add(1, ['plan' => $order->plan]);+ throw $e;+ } finally {+ $this->metrics->orderProcessingMs->record((hrtime(true) - $start) / 1e6);+ }+ }+}+```++## Naming Conventions++Use dot-separated names, lowercase, with the subject first:++```php+// Good+$metrics->counter('orders.created');+$metrics->histogram('db.query_ms');+$metrics->counter('cache.hits');++// Bad+$metrics->counter('created'); // too vague+$metrics->counter('orderCount'); // inconsistent style+```++Do not prefix names with your service name. Traceway already separates series by service through the OTel resource, which comes from `OTEL_SERVICE_NAME`.++## Without the Bundle's Registry++If you want custom metrics but do not want to turn `metrics.enabled` on, go through the SDK's global meter provider directly. This works whatever the bundle config says, because it never touches the bundle:++```php+use OpenTelemetry\API\Globals;++$meter = Globals::meterProvider()->getMeter('app');++$orders = $meter->createCounter('orders.created', 'orders', 'Total orders created');+$orders->add(1, ['plan' => 'pro']);+```++An observable gauge is also available on this path. Its callback runs at export time, which suits values you can read on demand:++```php+$meter->createObservableGauge('cache.entries', 'items', 'Cache entry count')+ ->observe(function ($observer) {+ $observer->observe($this->cache->getStats()['entries']);+ });+```++Register observable gauges once, in a service constructor, not on every request. Each call adds another callback.diff --git a/docs/pages/client/otel/symfony/spans.mdx b/docs/pages/client/otel/symfony/spans.mdxnew file mode 100644index 00000000..9e8b6c56--- /dev/null+++ b/docs/pages/client/otel/symfony/spans.mdx@@ -0,0 +1,183 @@+# Spans++The bundle creates one span per HTTP request, and that span is what becomes an endpoint row in Traceway. Everything below it in the trace is a child span: database queries, template renders, outgoing HTTP calls, and any work you instrument yourself.++## Spans You Get for Free++Install the relevant Symfony component and the bundle instruments it. A typical request ends up looking like this in the trace view:++```+GET /users/{id} 120ms+├─ db.users.find 12ms your own span+│ └─ SELECT 1ms db.system.name=sqlite+├─ twig.render profile.html.twig 8ms+└─ GET 40ms server.address=api.example.com+```++Outgoing HttpClient spans are named after the method alone. The host lives in the `server.address` attribute, which keeps one busy client from splitting into hundreds of span names.++| Component | Span kind | Needs |+|---|---|---|+| Doctrine DBAL queries and transactions | CLIENT | `doctrine/dbal` |+| Twig template rendering | INTERNAL | `twig/twig` |+| Cache pool get / delete / invalidateTags | INTERNAL | `symfony/cache` |+| Outgoing HttpClient requests | CLIENT | `symfony/http-client` |+| Mailer sends | PRODUCER + CLIENT | `symfony/mailer` |++Each of these has its own toggle under `open_telemetry.traces` if you want to turn one off. See the [configuration reference](/client/otel/symfony#configuration-reference).++Outgoing HttpClient calls also get W3C trace headers injected, so if the service you call is instrumented too, both sides join the same distributed trace. Traceway's own OTLP endpoint is auto-excluded, so exporting telemetry never traces itself.++### Doctrine Queries Show No SQL by Default++Doctrine spans are named after the operation, plus the table when it can be extracted: `SELECT`, `INSERT orders`, `BEGIN`, `COMMIT`. They carry `db.system.name`, `db.namespace`, and `db.operation.name`. Since v3.0 the SQL text itself is **not** recorded, because raw `query()` and `exec()` calls can contain unescaped literals. A span named `SELECT` with no query attached is the expected default, not a broken setup.++Opt in when you want `db.query.text` on the span:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ traces:+ doctrine:+ record_statements: true+```++Prepared statements record placeholders rather than values. `query()` and `exec()` record the raw string, so leave this off in production if your SQL can carry personal data.++## Using TracingInterface++For your own spans, inject the bundle's `TracingInterface`. It handles span creation, activation, error recording, and cleanup, so there is no `finally` block to forget:++```php+use Traceway\OpenTelemetryBundle\TracingInterface;++class OrderService+{+ public function __construct(+ private readonly TracingInterface $tracing,+ ) {}++ public function processOrder(int $orderId): void+ {+ $this->tracing->trace('order.validate', function () use ($orderId) {+ $this->validateOrder($orderId);+ });++ $this->tracing->trace('order.charge', function () use ($orderId) {+ $this->chargePayment($orderId);+ });+ }+}+```++`trace()` returns whatever the callback returns, so you can wrap an expression directly:++```php+$user = $this->tracing->trace('db.users.find', fn () => $this->repository->find($id));+```++If the callback throws, the span records the exception, is marked as an error, and the throwable is re-thrown unchanged.++### With Attributes and Span Kind++```php+use OpenTelemetry\API\Trace\SpanKind;++$result = $this->tracing->trace(+ 'stripe.charge',+ fn () => $this->stripe->charge($amount),+ attributes: ['payment.amount' => $amount, 'payment.currency' => 'usd'],+ kind: SpanKind::KIND_CLIENT,+);+```++### Nested Spans++Child spans automatically attach to whatever span is active:++```php+$this->tracing->trace('order.fulfill', function () {+ $this->tracing->trace('inventory.reserve', fn () => $this->reserve());+ $this->tracing->trace('payment.charge', fn () => $this->charge());+ $this->tracing->trace('email.send', fn () => $this->notify());+});+```++For tests, stub the interface and have `trace()` call the callback directly:++```php+$tracing = $this->createStub(TracingInterface::class);+$tracing->method('trace')->willReturnCallback(fn (string $name, callable $cb) => $cb());+```++## Using the OpenTelemetry API Directly++For span events, manual status codes, or a span whose lifetime does not fit a callback, use the tracer directly:++```php+use OpenTelemetry\API\Globals;++$tracer = Globals::tracerProvider()->getTracer('app');++$span = $tracer->spanBuilder('process-order')->startSpan();+$scope = $span->activate();+try {+ $this->processOrder($orderId);+} finally {+ $scope->detach();+ $span->end();+}+```++Always detach the scope and end the span in a `finally` block. A span that is never ended is never exported, and a scope that is never detached leaves the wrong parent active for everything that runs afterwards.++## Adding Attributes++Attach metadata to spans for filtering and debugging:++```php+$span->setAttribute('db.system', 'mysql');+$span->setAttribute('cache.key', $key);+$span->setAttribute('db.row_count', $rowCount);+```++`Span::getCurrent()` gets you the active span from anywhere, including a controller, a service, or a Messenger handler, without passing it around:++```php+use OpenTelemetry\API\Trace\Span;++Span::getCurrent()->setAttribute('user.id', $userId);+```++Attributes set on the request span show up on the endpoint row, and on any Issue raised during that request. See [Exceptions](/client/otel/symfony/exceptions).++## Recording Errors on Spans++When a span's operation fails, record the exception and set the status:++```php+use OpenTelemetry\API\Trace\StatusCode;++$span = $tracer->spanBuilder('external-api-call')->startSpan();+try {+ $response = $this->httpClient->request('GET', 'https://api.example.com/data');+} catch (\Throwable $e) {+ $span->recordException($e);+ $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());+ throw $e;+} finally {+ $span->end();+}+```++## Span Naming Conventions++Span names group in the dashboard, so they must be low cardinality. Never put an id, an email, or a URL with parameters in a span name. Put those in attributes.++| Good | Bad |+|------|-----|+| `db.users.find` | `query` |+| `cache.sessions.get` | `cache` |+| `stripe.charge` | `api` |+| `s3.upload-image` | `upload` |+| `email.send-welcome` | `send user 4821 an email` |diff --git a/docs/pages/client/otel/symfony/tasks.mdx b/docs/pages/client/otel/symfony/tasks.mdxnew file mode 100644index 00000000..1c51a8dc--- /dev/null+++ b/docs/pages/client/otel/symfony/tasks.mdx@@ -0,0 +1,167 @@+# Tasks++Traceway's **Tasks** page is where background work shows up, as opposed to HTTP requests, which go to Endpoints. The bundle fills it from two sources with no extra code: Symfony Messenger messages and `bin/console` commands.++## Messenger++Nothing extra to configure. Every consumed message is traced as soon as the bundle is installed.++By default the consumer span is parented to the trace that dispatched it, so the request that dispatched the message and the worker that processed it read as one continuous distributed trace. Traceway captures the message as a Task either way, and marks it with a **Non-root** chip when it was triggered from another trace.++If you would rather have each consumed message start its own independent trace, opt in:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ traces:+ messenger:+ root_spans: true+```++This is mostly a presentation choice. With `root_spans: true` the consumer is a root Task with no chip, and the cross-trace link back to the dispatcher is gone. Without it, the consumer is a non-root Task and the dispatcher to consumer relationship stays visible in the distributed-trace view.++### What Gets Captured++For each consumed message, the bundle records:++- **Span name**: `process <MessageClass>`, for example `process EmailSendMessage`. The dispatch side is `send <MessageClass>`. (Before v3.0 these were `EmailSendMessage process` and `EmailSendMessage publish`. If you are searching for an older name and finding nothing, that is why.)+- **Span kind**: `CONSUMER`, which is what tells Traceway this is a Task and not a plain internal span+- **Duration**: total handler execution time+- **Transport**: `messaging.destination.name`, for example `async`, plus `messaging.system` and `messaging.operation.type`+- **Errors**: exceptions are recorded with stack traces, the span is marked failed, and the error shows up under Issues+- **Child spans**: any spans created inside the handler nest under the task span++### Adding Attributes in Handlers++The bundle activates the task span before your handler runs, so `Span::getCurrent()` inside the handler is that task:++```php+use OpenTelemetry\API\Trace\Span;+use Symfony\Component\Messenger\Attribute\AsMessageHandler;++#[AsMessageHandler]+class EmailSendHandler+{+ public function __invoke(EmailSendMessage $message): void+ {+ $span = Span::getCurrent();++ $span->setAttribute('email.to', $message->to);+ $span->setAttribute('email.subject', $message->subject);++ $this->mailer->send($message);+ }+}+```++### Child Spans in Handlers++For handlers with distinct sub-operations, inject `TracingInterface` to create child spans. They automatically nest under the task span:++```php+use OpenTelemetry\API\Trace\SpanKind;+use Symfony\Component\Messenger\Attribute\AsMessageHandler;+use Traceway\OpenTelemetryBundle\TracingInterface;++#[AsMessageHandler]+class PaymentProcessorHandler+{+ public function __construct(+ private readonly TracingInterface $tracing,+ ) {}++ public function __invoke(PaymentProcessorMessage $message): void+ {+ $this->tracing->trace('validate.payment', function () use ($message) {+ $this->validate($message);+ }, ['payment.amount' => $message->amount, 'payment.currency' => $message->currency]);++ $this->tracing->trace('charge.gateway', function () use ($message) {+ $this->chargeGateway($message);+ }, kind: SpanKind::KIND_CLIENT);+ }+}+```++### Dispatch Tracing++The bundle also traces the dispatch side. When your controller or service dispatches a message, a PRODUCER span named `send <MessageClass>` is created as a child of the current request, and W3C trace context is injected into the message envelope so the consumer links back. With `root_spans: true` the consumer starts an independent trace instead. Either way the consumer lands on the Tasks page.++Logs emitted inside a handler carry the task's trace id, so the Logs tab of a Task shows exactly what that job logged. See [Logs](/client/otel/symfony/logs).++## Console Commands++Every `bin/console` run is traced too, and lands on the Tasks page under the command name:++```+Task name Count Avg+app:import-orders 12 1.4s+app:sync-invoices 3 820ms+```++The span carries `console.command`, `process.exit.code`, `process.pid`, and `process.executable.name`. A command that throws produces both a failed Task and an Issue, which makes this the simplest way to monitor cron jobs.++Add your own attributes the same way as anywhere else:++```php+namespace App\Command;++use OpenTelemetry\API\Trace\Span;+use Symfony\Component\Console\Attribute\AsCommand;+use Symfony\Component\Console\Command\Command;+use Symfony\Component\Console\Input\InputInterface;+use Symfony\Component\Console\Output\OutputInterface;++#[AsCommand(name: 'app:import-orders')]+class ImportOrdersCommand extends Command+{+ protected function execute(InputInterface $input, OutputInterface $output): int+ {+ $rows = $this->importer->run();++ Span::getCurrent()->setAttribute('import.rows', $rows);++ return Command::SUCCESS;+ }+}+```++Two commands are excluded by default, because they are long-running workers whose per-message spans are already captured above: `messenger:consume` and `messenger:consume-messages`. Adjust the list if you want to skip noisy commands like `cache:clear`:++```yaml+# config/packages/open_telemetry.yaml+open_telemetry:+ traces:+ console:+ enabled: true+ excluded_commands: ['messenger:consume', 'messenger:consume-messages', 'cache:clear']+```++## Custom Background Work++If your background work is neither Messenger nor a console command, you have to create the span yourself, and it **must** be `SpanKind::KIND_CONSUMER`:++```php+use OpenTelemetry\API\Globals;+use OpenTelemetry\API\Trace\SpanKind;+use OpenTelemetry\Context\Context;++$tracer = Globals::tracerProvider()->getTracer('app');++$span = $tracer->spanBuilder('nightly.reconcile')+ ->setParent(Context::getRoot()) // start a fresh trace, not a child of whatever ran before+ ->setSpanKind(SpanKind::KIND_CONSUMER)+ ->startSpan();+$scope = $span->activate();++try {+ $this->reconcile();+} finally {+ $scope->detach();+ $span->end();+}+```++> **A root span with the default `INTERNAL` kind is dropped.** Traceway classifies a span as a Task when its kind is `CONSUMER`, or when it is a root span carrying `console.command`. A root `INTERNAL` span with no HTTP attributes matches neither, so it is discarded on ingest without any error. If you created a background span and it never appeared anywhere in the dashboard, this is almost always why.++Child spans inside the block do not need any of this. They nest under the consumer span and are shown in its trace regardless of their kind.diff --git a/docs/pages/client/otel/traces.mdx b/docs/pages/client/otel/traces.mdxindex c363b171..f42c3e19 100644--- a/docs/pages/client/otel/traces.mdx+++ b/docs/pages/client/otel/traces.mdx@@ -4,20 +4,34 @@ Traceway ingests OpenTelemetry spans via the OTLP/HTTP protocol and maps them to## Span Mapping-Spans are classified by **kind and attributes**, not by whether they're a root span. Both root and child spans can be promoted to a top-level concept when the signal is there — a queued job processed as a child of the dispatcher's trace still becomes a Task, an LLM call made inside a request handler still becomes an AI Trace.+Spans are classified by **kind and attributes**, not by whether they're a root span. Both root and child spans can be promoted to a top-level concept when the signal is there. A queued job processed as a child of the dispatcher's trace still becomes a Task, and an LLM call made inside a request handler still becomes an AI Trace.-| Condition | Traceway Concept |-|-----------|-----------------|-| `SpanKind = SERVER` / `INTERNAL` with HTTP attributes, and the span is a root (or its parent is in another process — a cross-service inbound hop) | **Endpoint** |-| `SpanKind = CONSUMER` (queue / messenger workers) | **Task** |-| Root `INTERNAL` span with a `console.command` attribute (artisan commands, console runners) | **Task** |-| Any span with `gen_ai.*` attributes (root or child of a request/task) | **AI Trace** |-| Any other span with a parent | **Span** (waterfall row under its nearest enclosing entity) |-| Event named `"exception"` on any span | **Issue** (attributed to the owning entity) |+**The rules below are checked in order, and the first match wins.**-Each Endpoint / Task / AI Trace row carries an `is_root` flag. Non-root rows surface a **Non-root** chip in the dashboard list so you can tell at a glance which runs were triggered by another trace, and a **View distributed trace** link on the detail page jumps you to the full waterfall (endpoint → producer → consumer, request → child LLM call, etc.). The `distributed_trace_id` linking the rows is auto-derived from the OTel `trace_id`, so cross-trace lookup works without any vendor attribute.+| # | Condition | Traceway Concept |+|---|-----------|-----------------|+| 1 | `SpanKind = INTERNAL` carrying `exception.*` attributes | **Issue.** Never promoted to an Endpoint, Task or AI Trace. A child span still gets its ordinary Span row, a root span gets nothing else |+| 2 | `SpanKind = SERVER` / `INTERNAL` with HTTP attributes, and the span is a root, or its parent span is not present in the same `ResourceSpans` block (a cross-service inbound hop) | **Endpoint** |+| 3 | `SpanKind = CONSUMER` (queue / messenger workers) | **Task** |+| 4 | Root `INTERNAL` span with a `console.command` attribute (artisan commands, console runners) | **Task** |+| 5 | Any remaining span with `gen_ai.*` attributes | **AI Trace** |+| 6 | Any other span with a parent | **Span** (waterfall row under its nearest enclosing entity) |+| 7 | A root span matching none of the above | **Dropped.** No row is stored |+| - | Event named `"exception"` on any span | **Issue** (attributed to the owning entity) |-> **Sub-handler spans are not duplicated.** An in-process child SERVER/INTERNAL+HTTP span (e.g. a framework's "handler /path" sub-span) is kept in the Spans waterfall — Endpoints are only promoted for entry points (root or cross-service hop).+Order matters most for `gen_ai.*`, which is checked last. A `CONSUMER` span carrying `gen_ai.*` becomes a Task, and a request span carrying `gen_ai.*` becomes an Endpoint. Neither shows up on AI Traces. Put every model call on its own child span so rule 5 is the one that matches it.++> **The default span kind is `INTERNAL`, and a root `INTERNAL` span with no HTTP attributes is discarded.** If you wrap a cron job or a worker in `tracer.startActiveSpan("my-job")` without setting a kind, nothing appears on Tasks and nothing appears on Spans. Traceway still answers `200 OK`. Set `{ kind: SpanKind.CONSUMER }` for queue and job work, or add a `console.command` attribute for CLI commands. The same applies to root `CLIENT` and `PRODUCER` spans, and to root `SERVER` spans that carry no HTTP attributes.++"HTTP attributes" means at least one of `http.request.method`, `http.method`, `http.route` or `url.path`. Nothing else counts, so a span carrying only `url.full`, `http.target` or `server.address` is not treated as an HTTP entry point.++Rule 1 exists because browser SDKs stamp page context like `url.path` onto their error spans, and those must become Issues rather than endpoints. It applies to `exception.*` set as span **attributes**, not to a recorded `exception` **event**, and it suppresses every promotion, so an `INTERNAL` span with both `exception.*` and `gen_ai.*` attributes yields an Issue and no AI Trace. To record an error on a span you also want promoted, use `recordException` (which writes an event) instead of setting the attributes yourself.++Child spans that carry `db.query.text` (or the legacy `db.statement`) are displayed under the query text instead of the span name, so a database waterfall reads as the statements that ran rather than a column of identical `pg.query` rows.++Each Endpoint / Task / AI Trace row carries an `is_root` flag. Rows triggered by another trace surface a **Non Root** chip in the dashboard list, or a **Mixed** chip when a grouped row holds both root and non-root runs. The detail page renders a **Distributed Trace** card with the full waterfall (endpoint → producer → consumer, request → child LLM call) whenever the trace touches more than one run. The `distributed_trace_id` that links the rows is derived from the OTel `trace_id`, so cross-service lookup works with no vendor attribute. If you need to stitch runs that genuinely carry different trace ids, set the span attribute `traceway.distributed_trace_id` to a shared UUID and it overrides the derived value.++> **Sub-handler spans are not duplicated, as long as parent and child ship together.** An in-process child SERVER/INTERNAL+HTTP span (a framework's "handler /path" sub-span) stays in the Spans waterfall when its parent arrives in the same `ResourceSpans` block. Parents are matched inside one block, not across the whole export, though spans from one service normally share a Resource and land in the same block. Traceway cannot see process boundaries, so it treats "parent missing from this block" as a cross-service hop and promotes the child to its own Endpoint. If you see the same route listed twice with a **Mixed** chip, a batch processor split the parent and the child across two exports. Raise the exporter's `maxExportBatchSize` and `scheduledDelayMillis` so a whole trace ships together, or stop emitting the redundant sub-handler span.## Resource Attributes@@ -35,12 +49,22 @@ For root SERVER spans with HTTP attributes, Traceway extracts the following:| OTel Attribute | Fallback | Traceway Field ||---------------|----------|----------------|| `http.request.method` | `http.method` | HTTP method |-| `http.route` | Span name | Route path |+| `http.route` | `url.path`, then the span name | Route path (`http.route` is ignored unless it starts with `/`) || `http.response.status_code` | `http.status_code` | Status code || `http.response.body.size` | `http.response_content_length` | Response body size || `client.address` | `net.peer.ip` | Client IP |-The endpoint name is constructed as `"METHOD /route"` (e.g., `"GET /api/users"`).+The endpoint name is built as `"METHOD /route"`, for example `"GET /api/users"`.++**A method attribute is required for the route to be used at all.** If neither `http.request.method` nor `http.method` is set, the raw span name becomes the endpoint name and `url.path` is ignored entirely.++Prefer `http.route` (the low-cardinality template, `/users/:id`) over `url.path` (the concrete URL, `/users/8412`). Falling back to `url.path` creates one endpoint row per distinct URL.++### Unmatched 404s++A 404 response whose span carries no real route is stored under the single endpoint name `UNMATCHED` instead of its URL. "No real route" means `http.route` is missing, does not start with `/`, or is a catch-all made only of slashes and wildcards such as `/`, `/*` or `/**`, which is what not-found handlers usually report.++This keeps scanner traffic and typo'd URLs from creating one endpoint row per bogus path. A 404 returned from a concretely matched route, for example `GET /users/:id` for a user that does not exist, keeps its own identity. Only status 404 collapses. Other statuses on a catch-all route keep the catch-all name.### Streaming Responses (SSE / WebSocket)@@ -48,11 +72,13 @@ Long-lived streaming routes are flagged as `is_stream` on ingest so that the das| Signal | OTel attribute ||--------|----------------|-| Server-Sent Events | `http.response.header.content-type` contains `text/event-stream` |+| Server-Sent Events | `http.response.header.content-type` **starts with** `text/event-stream` (`text/event-stream; charset=utf-8` matches) || WebSocket upgrade | `http.response.status_code = 101` |-| Vendor opt-in | `traceway.is_stream = true` (bool) — for protocols the headers don't expose |+| Vendor opt-in | `traceway.is_stream = true` (bool), for protocols the headers don't expose |-There is no OpenTelemetry-standard streaming attribute; Traceway sniffs the captured Content-Type response header. Make sure your OTel instrumentation actually captures `http.response.header.content-type` (some SDKs require explicit opt-in for header capture). Otherwise set `traceway.is_stream = true` directly on the span when starting a stream.+There is no OpenTelemetry-standard streaming attribute, so Traceway sniffs the captured Content-Type response header. Two things trip people up. The attribute key must be exactly lowercase, `http.response.header.content-type`, which is what the OTel HTTP conventions emit. And the match is a prefix, so a comma-joined value like `application/json, text/event-stream` is not detected.++Many SDKs require explicit opt-in before response headers are captured at all. When in doubt, set `traceway.is_stream = true` on the span yourself. It must be a **boolean**. The string `"true"` is ignored.## Exception Events@@ -66,16 +92,33 @@ Spans with events named `"exception"` are extracted as Issues in Traceway. The fExceptions are grouped by a normalized hash of the stack trace, so the same logical error produces the same Issue regardless of runtime differences like memory addresses or user IDs.+Traceway reads exception data from two places: an event named `exception` on the span, which is what `span.recordException(err)` produces, and `exception.type` / `exception.message` / `exception.stacktrace` set directly as span attributes. The event wins when both are present.++The Issue is attributed to the nearest enclosing Endpoint, Task or AI Trace, walking up the parent chain from the span that raised it. An error thrown in a database span therefore shows up on the request that ran the query.++**An exception on a dropped span still becomes an Issue, but an unlinked one.** If you record an error on a root `SpanKind.INTERNAL` span, the span itself is discarded (see Span Mapping) while the Issue is still created. There is then no Endpoint or Task for it to point at, so the occurrence carries only the raw trace id and you lose the request or job context that would tell you what was running. Record exceptions on spans that sit inside a promoted entity, or fix the span kind.+## Example: Node.js with OTel SDK+```bash+npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \+ @opentelemetry/instrumentation @opentelemetry/exporter-trace-otlp-http \+ @opentelemetry/resources+```+```typescript+// telemetry.mjs+// Load it before your app: node --import ./telemetry.mjs server.js+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);+import { NodeSDK } from "@opentelemetry/sdk-node";import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";-import { Resource } from "@opentelemetry/resources";+import { resourceFromAttributes } from "@opentelemetry/resources";const sdk = new NodeSDK({- resource: new Resource({+ resource: resourceFromAttributes({"service.name": "my-service","service.version": "1.0.0",}),@@ -91,16 +134,51 @@ const sdk = new NodeSDK({sdk.start();```-This automatically instruments HTTP servers, database clients, and other libraries. SERVER spans become Endpoints, CONSUMER spans become Tasks, any span with `gen_ai.*` attributes becomes an AI Trace (root or child), and exception events become Issues.+The two `node:module` lines are required for an ESM app (`.mjs`, or `"type": "module"`). Without the loader hook nothing is patched, so `http.route` is never set and your endpoints are named after raw span names. A CommonJS app does not need them. Use `resourceFromAttributes`, not the old `new Resource(...)`, which was removed in `@opentelemetry/resources` 2.x.++This automatically instruments HTTP servers, database clients, and other libraries. SERVER spans become Endpoints, CONSUMER spans become Tasks, a span whose only signal is `gen_ai.*` becomes an AI Trace, and exception events become Issues.++This snippet exports traces only. Metrics and logs each need their own exporter. See the [full three-signal setup](/client/otel#quick-start-direct-sdk-export) on the overview page.++## Example: Python (zero-code)++Most Python apps need no instrumentation code at all. `opentelemetry-instrument` starts the SDK, patches your web framework, and flushes on exit:++```bash+pip install opentelemetry-distro opentelemetry-exporter-otlp+opentelemetry-bootstrap -a install+```++```bash+export OTEL_SERVICE_NAME=my-python-service+export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"+export OTEL_TRACES_EXPORTER=otlp++opentelemetry-instrument uvicorn app:app --port 8000 # FastAPI, any ASGI app+opentelemetry-instrument flask --app wsgi run --port 8000 # Flask+```++`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` is required. Left unset, the Python SDK resolves `otlp` to gRPC, which Traceway does not accept, and the app runs and exits with no error while nothing arrives.++The FastAPI, Flask, Starlette, and Django instrumentations set `http.route` from the matched route pattern in the framework's own syntax (`/users/{user_id}` on FastAPI, `/orders/<order_id>` on Flask), so endpoints arrive grouped. The [Python guide](/client/otel/python) covers logs, metrics, background tasks, and exceptions on this same path.## Example: Python with OTel SDK+To wire the SDK by hand instead, build the provider and create spans with a kind Traceway keeps. A provider on its own records nothing. The SDK registers an `atexit` hook, so a clean exit flushes the last batch even without the explicit `provider.shutdown()` below; only an abrupt end such as `os._exit` or `SIGKILL` loses it.++```bash+pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http+```+```pythonfrom opentelemetry import trace-from opentelemetry.sdk.trace import TracerProvider-from opentelemetry.sdk.trace.export import BatchSpanProcessorfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporterfrom opentelemetry.sdk.resources import Resource+from opentelemetry.sdk.trace import TracerProvider+from opentelemetry.sdk.trace.export import BatchSpanProcessor+from opentelemetry.trace import SpanKind, Status, StatusCoderesource = Resource.create({"service.name": "my-python-service",@@ -108,15 +186,40 @@ resource = Resource.create({})provider = TracerProvider(resource=resource)-exporter = OTLPSpanExporter(+provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="https://your-traceway-instance.com/api/otel/v1/traces",headers={"Authorization": "Bearer your-project-token"},-)-provider.add_span_processor(BatchSpanProcessor(exporter))+)))trace.set_tracer_provider(provider)++tracer = trace.get_tracer("my-app")+++def do_work():+ ... # the job's real work+++with tracer.start_as_current_span(+ "cleanup-expired-sessions", kind=SpanKind.CONSUMER+) as span:+ try:+ with tracer.start_as_current_span("delete-batch"):+ do_work()+ span.set_status(Status(StatusCode.OK))+ except Exception as error:+ span.record_exception(error)+ span.set_status(Status(StatusCode.ERROR, str(error)))+ raise++provider.shutdown()```+That run appears under **Tasks** as `cleanup-expired-sessions`, with `delete-batch` as a child span. In Python the span kind is a keyword argument, `kind=SpanKind.CONSUMER`. Drop it and the root span defaults to `INTERNAL`, which is discarded on ingest.++Python's `Resource.create(...)` is unaffected by the JavaScript `Resource` removal. Only the JS SDK changed.+## Next Steps-- [Metrics](/client/otel/metrics) — export metrics to Traceway via OTel-- [Overview](/client/otel) — protocol details and authentication+- [Metrics](/client/otel/metrics): export metrics to Traceway via OTel+- [Logs](/client/otel/logs): export logs and link them to these traces+- [Overview](/client/otel): protocol details, authentication, and a "nothing is showing up" checklistdiff --git a/docs/pages/client/react-native/index.mdx b/docs/pages/client/react-native/index.mdxindex cb57e9da..c44dfd73 100644--- a/docs/pages/client/react-native/index.mdx+++ b/docs/pages/client/react-native/index.mdx@@ -1,6 +1,6 @@# React Native Quick Start-Integrate Traceway into your React Native or Expo application with the [`@tracewayapp/react-native`](https://www.npmjs.com/package/@tracewayapp/react-native) package. Plain JavaScript — no native modules, no `pod install`, no Gradle changes — works in Expo Go, in bare React Native CLI projects, and in EAS builds.+Integrate Traceway into your React Native or Expo application with the [`@tracewayapp/react-native`](https://www.npmjs.com/package/@tracewayapp/react-native) package. Plain JavaScript (no native modules, no `pod install`, no Gradle changes) that works in Expo Go, in bare React Native CLI projects, and in EAS builds.## Installation@@ -10,7 +10,7 @@ npm install @tracewayapp/react-native## Setup-Wrap your app in `TracewayProvider` from your entry component (typically `App.tsx`). Mounting the provider runs `init(...)` once, which installs RN's `ErrorUtils` global handler, the `fetch` / `XMLHttpRequest` wrappers, and the `console.*` mirror. The provider also catches render-time errors anywhere below it and reports them automatically — no separate error boundary needed:+Wrap your app in `TracewayProvider` from your entry component (typically `App.tsx`). Mounting the provider runs `init(...)` once, which installs RN's `ErrorUtils` global handler, the `fetch` / `XMLHttpRequest` wrappers, and the `console.*` mirror. The provider also catches render-time errors anywhere below it and reports them automatically, so no separate error boundary is needed:```tsximport { TracewayProvider } from "@tracewayapp/react-native";@@ -24,9 +24,9 @@ export default function App() {}```-That's it. Every uncaught throw, unhandled promise rejection, render-time exception, and `fetch` call is captured automatically. If you want to replace a crashed subtree with a custom fallback view, `TracewayErrorBoundary` is still exported for that purpose.+That's it. Every uncaught throw, render-time exception, and `fetch` call is captured automatically. If you want to replace a crashed subtree with a custom fallback view, `TracewayErrorBoundary` is still exported for that purpose, but it is deprecated as of v1.1.0 and will be removed in v2. Do not build new code around it.-> **No screen recording.** Unlike the browser SDK ([`@tracewayapp/frontend`](/client/js-sdk)), this package does not record the screen. The `rrweb` recorder is intentionally absent — nothing in the bundle reaches into a DOM that doesn't exist.+> **No screen recording.** Unlike the browser SDK ([`@tracewayapp/frontend`](/client/js-sdk)), this package does not record the screen. The `rrweb` recorder is intentionally absent: nothing in the bundle reaches into a DOM that doesn't exist.## Capture Errors Manually@@ -34,7 +34,7 @@ That's it. Every uncaught throw, unhandled promise rejection, render-time exceptimport { useTraceway } from "@tracewayapp/react-native";import { captureException, captureMessage, flush } from "@tracewayapp/react-native";-// Inside a component — get scoped helpers from the hook:+// Inside a component, get scoped helpers from the hook:function CheckoutButton() {const { captureException, recordAction } = useTraceway();// ...@@ -64,13 +64,15 @@ All field names mirror the browser, Android, and Flutter SDKs so existing config| `debounceMs` | `number` | `1500` | Milliseconds before flushing batched events || `retryDelayMs` | `number` | `10000` | Retry delay on failed uploads || `ignoreErrors` | `Array<string \| RegExp>` | sensible defaults | Drop exceptions whose stack/message match any pattern |-| `beforeCapture` | `(exception) => boolean` | — | Return `false` to suppress an exception |+| `beforeCapture` | `(exception) => boolean` | none | Return `false` to suppress an exception || `captureLogs` | `boolean` | `true` | Mirror `console.{debug,log,info,warn,error}` into the rolling log buffer || `captureNetwork` | `boolean` | `true` | Record `fetch` / `XMLHttpRequest` calls as network actions || `captureNavigation` | `boolean` | `true` | Record manual `recordNavigation()` calls into the action buffer |-| `captureDeviceInfo` | `boolean` | `true` | Auto-collect `os.name`, `os.version`, screen, locale, runtime engine and attach to every report |+| `captureDeviceInfo` | `boolean` | `true` | Auto-collect `device.locale` and `runtime.engine` and attach them to every report (see [Attributes](#1-device-info-auto-collected)) || `eventsWindowMs` | `number` | `10000` | Rolling window kept in the log/action buffers (ms) || `eventsMaxCount` | `number` | `200` | Hard cap applied independently to logs and actions |+| `distributedTraceHosts` | `Array<string \| RegExp>` | `[]` | Hosts that receive an outgoing `traceway-trace-id` header, so a server-side capture links back to the client request that caused it. React Native has no `window.location.origin` to infer from, so unlike the browser SDK this is opt-in: list your own backend hosts, e.g. `["api.example.com"]`. Each entry is an exact hostname or a `RegExp` tested against the URL host |+| `captureHttpServerErrors` | `boolean` | `false` | Also report every `fetch` / XHR response with `status >= 500` as a synthetic exception, on top of the network action already recorded. 4xx is deliberately excluded (see `DEFAULT_IGNORE_PATTERNS`) |```tsx<TracewayProvider@@ -84,15 +86,41 @@ All field names mirror the browser, Android, and Flutter SDKs so existing config>```-The `DEFAULT_IGNORE_PATTERNS` export contains the built-in `ignoreErrors` list (`Network request failed`, `Failed to fetch`, generic 4xx, etc.) — you can extend it rather than replace it.+The `DEFAULT_IGNORE_PATTERNS` export contains the built-in `ignoreErrors` list (`Network request failed`, `Failed to fetch`, generic 4xx, etc.), so you can extend it rather than replace it.++## Distributed Tracing++By default the SDK injects nothing into outgoing requests. List your backend hosts in `distributedTraceHosts` to turn it on:++```tsx+<TracewayProvider+ connectionString="your-token@https://traceway.example.com/api/report"+ options={{ distributedTraceHosts: ["api.example.com", /\.internal\.example\.com$/] }}+>+ <RootNavigator />+</TracewayProvider>+```++Each matching `fetch` call gets a `traceway-trace-id` header carrying a fresh id. If your backend reports an error while handling that request, both sides carry the same id and you can jump from the mobile exception to the server one. Entries are matched against the URL's host, so include the port when the URL has one (`"localhost:3000"`). `XMLHttpRequest` calls are still recorded as network actions, but they do not get the header, so route traffic you want linked through `fetch`.++`getActiveDistributedTraceId()` returns the id of the instrumented `fetch` currently in flight, and `null` when there is none. Use it to put that same id on work you kick off from inside such a request (a WebSocket handshake, a native module, a client the SDK does not wrap):++```ts+import { DISTRIBUTED_TRACE_HEADER, getActiveDistributedTraceId } from "@tracewayapp/react-native";++const traceId = getActiveDistributedTraceId();+if (traceId) {+ headers[DISTRIBUTED_TRACE_HEADER] = traceId;+}+```## Logs & Actions-Every captured exception ships with the last ~10 seconds of session context, attached to a `sessionRecordings[]` entry on the wire — the same shape every other Traceway SDK uses. Two independent rolling buffers — **logs** and **actions** — are kept in memory, each capped at 200 entries by default.+Every captured exception ships with the last ~10 seconds of session context, attached to a `sessionRecordings[]` entry on the wire, the same shape every other Traceway SDK uses. Two independent rolling buffers (**logs** and **actions**) are kept in memory, each capped at 200 entries by default.### Logs-Every `console.{debug, log, info, warn, error}` call is mirrored into the log buffer. The original `console` output is preserved — the SDK only piggybacks on the call.+Every `console.{debug, log, info, warn, error}` call is mirrored into the log buffer. The original `console` output is preserved; the SDK only piggybacks on the call.Disable with `captureLogs: false`.@@ -100,12 +128,13 @@ Disable with `captureLogs: false`.Three kinds of actions are collected:-- **Network** — every `fetch` and `XMLHttpRequest` call (method, URL, status, duration, byte counts) is recorded automatically. RN polyfills XHR (and `fetch` is implemented on top of it on some platforms) — both paths are covered. Calls to the Traceway endpoint itself are skipped to avoid recursion.+- **Network**: every `fetch` and `XMLHttpRequest` call (method, URL, status, duration, byte counts) is recorded automatically. RN polyfills XHR (and `fetch` is implemented on top of it on some platforms), so both paths are covered. Calls to the Traceway endpoint itself are skipped to avoid recursion.-- **Navigation** — RN has no `window.history` to auto-instrument. Wire `recordNavigation` into whatever navigation library you use:+- **Navigation**: RN has no `window.history` to auto-instrument. Wire `recordNavigation` into whatever navigation library you use:```tsx- import { useNavigationContainerRef } from "@react-navigation/native";+ import { useRef } from "react";+ import { NavigationContainer, useNavigationContainerRef } from "@react-navigation/native";import { recordNavigation } from "@tracewayapp/react-native";const navigationRef = useNavigationContainerRef();@@ -127,7 +156,7 @@ Three kinds of actions are collected:For [Expo Router](https://docs.expo.dev/router/) drop the same call into a `useEffect` keyed on `usePathname()`.-- **Custom** — anything you record explicitly:+- **Custom**: anything you record explicitly:```tsximport { recordAction } from "@tracewayapp/react-native";@@ -137,7 +166,7 @@ Three kinds of actions are collected:### Wire shape-Logs and actions are kept in two separate rolling buffers, each capped at 200 entries / 10 seconds. They ship inside `sessionRecordings[].logs` and `sessionRecordings[].actions` on the wire, with `startedAt` / `endedAt` ISO 8601 timestamps spanning the captured window — the same shape the Flutter and Android SDKs use:+Logs and actions are kept in two separate rolling buffers, each capped at 200 entries / 10 seconds. They ship inside `sessionRecordings[].logs` and `sessionRecordings[].actions` on the wire, with `startedAt` / `endedAt` ISO 8601 timestamps spanning the captured window, the same shape the Flutter and Android SDKs use:```json{@@ -178,20 +207,33 @@ Each channel can be turned off individually via `TracewayProvider`'s `options`:Every captured exception ships with an `attributes` map composed of three layers (per-call wins, then global scope, then device info).-### 1. Device info — auto-collected+### 1. Device info (auto-collected)-Auto-stamped at `init()` from React Native core APIs (no native modules required, works in Expo Go). The keys mirror what the Flutter and Android SDKs emit so the dashboard renders them consistently across platforms:+`init()` stamps these two keys on every report. They need no native module and work in Expo Go:| Key | Source ||---|---|-| `os.name` | `Platform.OS` (`ios` / `android` / `web`) |-| `os.version` | `Platform.Version` |-| `screen.resolution` | `Dimensions.get("screen")` (whole pixels) |-| `screen.density` | `PixelRatio.get()` || `device.locale` | `Intl.DateTimeFormat().resolvedOptions().locale` || `runtime.engine` | `hermes` / `javascriptcore` |-To replace or extend the auto-collected map at runtime (e.g. add info from `expo-device` / `react-native-device-info` you've installed yourself):+`collectSyncDeviceInfo()` also knows how to read `os.name`, `os.version`, `screen.resolution` and `screen.density` from `Platform`, `Dimensions` and `PixelRatio`, but it reaches `react-native` through a global `require`. Metro does not define one (it registers modules on `global.__r`), so in a real app those four keys come back empty. Pass them in yourself if you want them on the dashboard. The key names mirror what the Flutter and Android SDKs emit, so the dashboard renders them consistently across platforms:++```ts+import { setDeviceAttributes, collectSyncDeviceInfo } from "@tracewayapp/react-native";+import { Dimensions, PixelRatio, Platform } from "react-native";++const screen = Dimensions.get("screen");++setDeviceAttributes({+ ...collectSyncDeviceInfo(),+ "os.name": Platform.OS,+ "os.version": String(Platform.Version),+ "screen.resolution": `${Math.round(screen.width)}x${Math.round(screen.height)}`,+ "screen.density": PixelRatio.get().toFixed(1),+});+```++Use the same call to add info from `expo-device` / `react-native-device-info` you've installed yourself:```tsimport { setDeviceAttributes, collectSyncDeviceInfo } from "@tracewayapp/react-native";@@ -206,11 +248,11 @@ setDeviceAttributes({To opt out of device info collection entirely, pass `captureDeviceInfo: false` in `options`.-### 2. Global scope — app-level identifiers+### 2. Global scope (app-level identifiers)Use the global scope for things tied to the **user / tenant / app state**, not the device: user id, organization id, build channel, feature flags, A/B bucket. Two ways to set it.-**Declarative — `<TracewayAttributes>` or the `useTracewayAttributes` hook.** Pass a map; the SDK diffs against the previous map and pushes only the deltas. On unmount, every key the component owned is removed.+**Declarative: `<TracewayAttributes>` or the `useTracewayAttributes` hook.** Pass a map; the SDK diffs against the previous map and pushes only the deltas. On unmount, every key the component owned is removed.```tsximport { TracewayAttributes, useTracewayAttributes } from "@tracewayapp/react-native";@@ -225,9 +267,9 @@ function App() {}```-Both accept `null` / `undefined` as "empty map" — useful while user data loads or after logout.+Both accept `null` / `undefined` as "empty map", which is useful while user data loads or after logout.-**Imperative — `setAttribute` / `setAttributes` / `removeAttribute` / `clearAttributes`.** Use these outside React component trees (auth listeners, background tasks):+**Imperative: `setAttribute` / `setAttributes` / `removeAttribute` / `clearAttributes`.** Use these outside React component trees (auth listeners, background tasks):```tsimport { setAttribute, setAttributes, clearAttributes } from "@tracewayapp/react-native";@@ -240,7 +282,7 @@ clearAttributes();### 3. Per-call attributes-`captureExceptionWithAttributes(error, attrs)` — wins on key collision over both layers above.+`captureExceptionWithAttributes(error, attrs)` wins on key collision over both layers above.### Layering@@ -248,12 +290,74 @@ For each captured exception: `device info < global scope < per-call`.## What Gets Captured Automatically-- **Uncaught throws** on the JS thread via `ErrorUtils.setGlobalHandler` — RN's equivalent of `window.onerror`. The red-box dev overlay still appears; we forward to the previous handler.-- **Unhandled promise rejections** — `Promise.reject(...)` without a `.catch()` reaches the same `ErrorUtils` handler.-- **Render errors** — `<TracewayProvider>` itself acts as an error boundary: exceptions thrown during render or in lifecycle methods are captured and re-thrown so your app behaves exactly as without Traceway.-- **`fetch` calls** — wrapped at `init()`, recorded with method, URL, status, duration, byte counts.-- **`XMLHttpRequest` calls** — RN polyfills XHR (and `fetch` is implemented on top of it on some platforms); both paths are covered.-- **Console output** — `console.{debug,log,info,warn,error}` is mirrored into the log buffer that rides along the next exception.+- **Uncaught throws** on the JS thread via `ErrorUtils.setGlobalHandler`, RN's equivalent of `window.onerror`. The red-box dev overlay still appears; we forward to the previous handler.+- **Unhandled promise rejections are not captured.** React Native hands them to its own `ExceptionsManager` (and only in dev builds), never through `ErrorUtils`, so the SDK never sees them. Catch them yourself and call `captureException`.+- **Render errors**: `<TracewayProvider>` itself acts as an error boundary. Exceptions thrown during render or in lifecycle methods are captured and re-thrown so your app behaves exactly as without Traceway.+- **`fetch` calls**: wrapped at `init()`, recorded with method, URL, status, duration, byte counts.+- **`XMLHttpRequest` calls**: RN polyfills XHR (and `fetch` is implemented on top of it on some platforms); both paths are covered.+- **Console output**: `console.{debug,log,info,warn,error}` is mirrored into the log buffer that rides along the next exception.++## Release Symbolication (source map upload)++A debug build gives you readable stack traces for free. A release build does not: Metro minifies the JavaScript into one bundle, and Hermes compiles that bundle to bytecode. Every production frame lands on the Issues page as an offset into `index.android.bundle` or `main.jsbundle`, with no function names and no original file. Upload the build's source map and Traceway resolves those frames server-side.++Do this on every release. A crash only resolves against the map for the exact build it came from.++### Get the upload token++Symbol uploads use a dedicated **upload token**, not the project token in your connection string. Open the **Connection** page for your project, scroll to **Source Maps**, and click **Generate Upload Token**. Treat it like any other CI secret. Members with the `readonly` role cannot generate one.++### Emit the source map++Android emits the map already. iOS needs one extra line.++**Android.** The React Native Gradle plugin passes `-output-source-map` to Hermes by default, so a release build writes the map with no config change. Only if your `android/app/build.gradle` overrides `hermesFlags` do you need to keep the flag in the list:++```groovy+react {+ hermesFlags = ["-O", "-output-source-map"]+}+```++After `./gradlew assembleRelease` the map is at:++```+android/app/build/generated/sourcemaps/react/release/index.android.bundle.map+```++**iOS.** In Xcode, open your target's **Build Phases**, find **Bundle React Native code and images**, and add this line to the script:++```sh+export SOURCEMAP_FILE="$(pwd)/../main.jsbundle.map"+```++After an archive build the map is at `ios/main.jsbundle.map`.++Both paths moved between React Native versions. If yours is not there, search the build output for `*.bundle.map` or `main.jsbundle.map`.++### Upload it++Traceway matches a map to a frame by filename: the frame says `index.android.bundle`, so the uploaded file must be named `index.android.bundle.map`. `curl -F` sends the basename of the local path, so keep the filename as the build produced it.++```bash+export TRACEWAY_URL="https://traceway.example.com"+export TRACEWAY_SOURCEMAP_TOKEN="your-upload-token"++curl --fail -X POST "$TRACEWAY_URL/api/sourcemaps/upload" \+ -H "Authorization: Bearer $TRACEWAY_SOURCEMAP_TOKEN" \+ -F "files=@android/app/build/generated/sourcemaps/react/release/index.android.bundle.map"+```++A `200` with `{"uploaded":1}` means it landed. Uploads take effect immediately, so exceptions arriving right after a deploy resolve against the new map.++The [`traceway-sourcemaps`](/client/js-sdk/sourcemap-upload) CLI does the same thing and reads the two variables above. Point it at the directory holding the map:++```bash+npm install -D @tracewayapp/sourcemap-upload+npx traceway-sourcemaps --directory android/app/build/generated/sourcemaps/react/release+```++See [JavaScript symbolication](/symbolicator/javascript) for how a frame is matched to its map.## Test Your Integration@@ -288,7 +392,7 @@ Tap the button and check your Traceway dashboard to verify the error appears wit|----------|----------------|------------------|| iOS (RN ≥ 0.72) | Yes | No || Android (RN ≥ 0.72) | Yes | No |-| Expo (SDK 49+) | Yes — works in Expo Go | No |+| Expo (SDK 49+) | Yes (works in Expo Go) | No || Web (via React Native Web) | Yes | No |For native Android-only apps without React Native, use the [Android SDK](/client/android). For Flutter apps, use the [Flutter SDK](/client/flutter). For browser apps, use the [React SDK](/client/react) or [generic JS SDK](/client/js-sdk).diff --git a/docs/pages/client/react/hooks.mdx b/docs/pages/client/react/hooks.mdxindex 8ff780ff..680e70d7 100644--- a/docs/pages/client/react/hooks.mdx+++ b/docs/pages/client/react/hooks.mdx@@ -57,9 +57,9 @@ function UserProfile({ userId }) {setProfile(profile);} catch (error) {captureExceptionWithAttributes(error, {- userId,+ userId: String(userId),action: "loadProfile",- timestamp: Date.now(),+ timestamp: String(Date.now()),});}}@@ -68,6 +68,8 @@ function UserProfile({ userId }) {}```+**Attribute values must be strings.** The ingest endpoint types them as a string map and rejects the whole upload with `400 Bad Request` if any value is a number, boolean, or object. The SDK re-queues a rejected batch and retries it forever, so one number blocks every later exception and message from that page too. Wrap non-string values in `String(...)`. TypeScript catches this for you (the signature is `Record<string, string>`), plain `.jsx` does not.+## captureMessageTrack non-error events:diff --git a/docs/pages/client/react/index.mdx b/docs/pages/client/react/index.mdxindex 0599db12..d3ce7841 100644--- a/docs/pages/client/react/index.mdx+++ b/docs/pages/client/react/index.mdx@@ -10,7 +10,7 @@ npm install @tracewayapp/react## Setup-Wrap your application with `TracewayProvider`. It also acts as an error boundary — render-time exceptions thrown anywhere in the tree are captured and reported automatically, then re-thrown so your app behaves exactly as it would without Traceway installed. Session recording is on by default; the last ~30s of DOM events ship with every captured exception:+Wrap your application with `TracewayProvider`. It also acts as an error boundary: render-time exceptions thrown anywhere in the tree are captured and reported automatically, then re-thrown so your app behaves exactly as it would without Traceway installed. Session recording is on by default; the last ~30s of DOM events ship with every captured exception:```jsximport { TracewayProvider } from "@tracewayapp/react";@@ -26,7 +26,9 @@ function App() {export default App;```-> **Custom fallback UI?** If you want to replace a crashed subtree with a fallback view instead of letting the error propagate, wrap the relevant section in `TracewayErrorBoundary` — it's still exported for that purpose, but is no longer required for capture.+> **Custom fallback UI?** If you want to replace a crashed subtree with a fallback view instead of letting the error propagate, wrap the relevant section in `TracewayErrorBoundary`. It's still exported for that purpose, but is no longer required for capture.++Seeing every dev-mode error twice? That is React StrictMode, which double-invokes the provider's constructor and creates two SDK clients. Production is unaffected. See [A note on StrictMode](/client/react/provider#a-note-on-strictmode).## Capture Errors Manually@@ -84,19 +86,11 @@ Click the button and check your Traceway dashboard to verify the error appears.## Distributed Tracing-The SDK automatically instruments `fetch` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them — no extra configuration needed.--For Axios users, add the interceptor to your instance:+The SDK automatically instruments both `fetch` and `XMLHttpRequest` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them.-```jsx-import axios from "axios";-import { createAxiosInterceptor } from "@tracewayapp/frontend";--const api = axios.create({ baseURL: "/api" });-api.interceptors.request.use(createAxiosInterceptor());-```+Axios needs no setup. It uses `XMLHttpRequest` in the browser, which the SDK already instruments. Do not register `createAxiosInterceptor()` in a browser app: it adds a second ID under the same header name, and the combined value is rejected by the backend.-See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for full details.+Your backend still has to set the incoming header on its server span as the `traceway.distributed_trace_id` attribute, or the two sides never join. See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for the middleware.## Next Steps@@ -104,3 +98,5 @@ See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for full- [Error Boundary](/client/react/error-boundary) - Automatic error capture- [Hooks](/client/react/hooks) - Manual error capture- [Distributed Tracing](/client/js-sdk/distributed-tracing) - Frontend ↔ backend correlation+- [Next.js](/client/otel/nextjs) - Next.js and Remix apps: this SDK covers the browser half, OpenTelemetry covers the server half+- [Project Structure](/learn/projects) - Why the browser half and the server half are two separate Traceway projectsdiff --git a/docs/pages/client/react/provider.mdx b/docs/pages/client/react/provider.mdxindex 5194ebb5..7c3d540e 100644--- a/docs/pages/client/react/provider.mdx+++ b/docs/pages/client/react/provider.mdx@@ -28,7 +28,7 @@ function App() {| Option | Type | Default | Description ||--------|------|---------|-------------|-| `debug` | `boolean` | `false` | Log events to console |+| `debug` | `boolean` | `false` | Log SDK problems to the console: failed uploads, `beforeCapture` throwing, and errors dropped by `ignoreErrors` / `beforeCapture`. Captured events themselves are not logged || `debounceMs` | `number` | `1500` | Batch delay in milliseconds || `retryDelayMs` | `number` | `10000` | Retry delay for failed uploads || `version` | `string` | `undefined` | Your application version |@@ -42,6 +42,7 @@ function App() {| `captureNavigation` | `boolean` | `true` | Record History API push / replace / pop transitions || `eventsWindowMs` | `number` | `10000` (`30000` w/ `recordAllSessions`) | Rolling log/action buffer window || `eventsMaxCount` | `number` | `200` (`600` w/ `recordAllSessions`) | Hard cap on log/action buffer entries |+| `captureHttpServerErrors` | `boolean` | `false` | Report every `fetch` response with status >= 500 as a synthetic exception. 4xx is never included, and it is wired into the `fetch` wrapper only, so `XMLHttpRequest` (including browser Axios) does not trigger it |## Custom Attributes@@ -60,7 +61,7 @@ function App() {}```-Both accept `null` / `undefined` as "empty map" — useful while user data loads or after logout. New object reference with the same content does not trigger SDK calls.+Both accept `null` / `undefined` as "empty map", which is useful while user data loads or after logout. New object reference with the same content does not trigger SDK calls.For imperative use (background workers, init scripts), the same primitives are exported as plain functions:@@ -139,6 +140,20 @@ root.render();```+### A note on StrictMode++`TracewayProvider` initializes the SDK in its constructor, and StrictMode double-invokes constructors in development. Two clients get created, so every uncaught error is reported twice and two batches are uploaded. Production builds do not double-invoke, so live counts are correct. If the doubled counts get in the way while you develop, mount the provider outside `<StrictMode>`:++```jsx+root.render(+ <TracewayProvider connectionString="your-token@...">+ <React.StrictMode>+ <App />+ </React.StrictMode>+ </TracewayProvider>+);+```+## TracewayContextFor advanced use cases, you can access the context directly:diff --git a/docs/pages/client/sdk/_meta.json b/docs/pages/client/sdk/_meta.jsondeleted file mode 100644index 739735de..00000000--- a/docs/pages/client/sdk/_meta.json+++ /dev/null@@ -1,11 +0,0 @@-{- "index": "Overview",- "initialization": "Initialization",- "traces": "Traces",- "spans": "Spans",- "exceptions": "Exceptions",- "messages": "Messages",- "metrics": "Metrics",- "attributes": "Attributes",- "stack-traces": "Stack Traces"-}diff --git a/docs/pages/client/sdk/attributes.mdx b/docs/pages/client/sdk/attributes.mdxdeleted file mode 100644index 36ffaf0e..00000000--- a/docs/pages/client/sdk/attributes.mdx+++ /dev/null@@ -1,151 +0,0 @@-# Attributes--Attributes attach key-value metadata to traces and issues. The `Attributes` type is thread-safe and available through both global configuration and per-request context.--## Attributes API--### SetTag--Set a string attribute:--```go-attributes.SetTag("user_id", "12345")-attributes.SetTag("tenant", "acme-corp")-```--### SetTagJson--Set an attribute with a JSON-serialized value. The value is marshaled to JSON and stored as a string. In the dashboard, JSON values are displayed with formatting.--```go-attributes.SetTagJson("cart_items", []string{"item1", "item2", "item3"})-attributes.SetTagJson("user_prefs", map[string]bool{"notifications": true})-```--Returns an error if JSON marshaling fails.--### GetTag--Retrieve a single attribute value:--```go-value, ok := attributes.GetTag("user_id")-if ok {- fmt.Println("User:", value)-}-```--### GetTags--Get a copy of all attributes as a `map[string]string`:--```go-allTags := attributes.GetTags()-for k, v := range allTags {- fmt.Printf("%s = %s\n", k, v)-}-```--Returns a copy — modifying the returned map does not affect the original attributes.--### Clone--Create an independent copy of the attributes:--```go-copy := attributes.Clone()-copy.SetTag("extra", "value") // Does not affect original-```--### Clear--Remove all attributes:--```go-attributes.Clear()-```--## Global Attributes--### ConfigureAttributes--Set attributes that apply to all traces and issues. Call once at application startup:--```go-traceway.ConfigureAttributes(func(s *traceway.Attributes) {- s.SetTag("environment", "production")- s.SetTag("region", "us-east-1")- s.SetTag("service", "user-api")- s.SetTag("version", os.Getenv("APP_VERSION"))-})-```--Global attributes are inherited by every request's attribute set.--## Context-Based Attributes--### WithAttributes--Create a new context with a cloned and modified attribute set:--```go-ctx = traceway.WithAttributes(ctx, func(a *traceway.Attributes) {- a.SetTag("operation", "checkout")- a.SetTag("step", "payment")-})-```--This clones the attributes from the current context, applies the modifications, and stores the new set in the returned context. The original context's attributes are unchanged.--### GetAttributesFromContext--Retrieve attributes from a Go context:--```go-attributes := traceway.GetAttributesFromContext(ctx)-attributes.SetTag("user_id", userID)-```--If the context has no attributes, returns a clone of the global attributes. If the context is `nil`, also returns a clone of the global attributes.--## Attribute Inheritance--Per-request attributes start as a clone of the global attributes. Request-level values override global values for the same key:--```-Global Attributes Request Attributes Final Tags-───────────────── ────────────────── ──────────-environment: prod + user_id: 123 = environment: prod-region: us-east-1 tenant: acme region: us-east-1-service: user-api service: user-api- user_id: 123- tenant: acme-```--## Usage in Middleware--Add attributes for all requests in a Gin middleware:--```go-func authMiddleware() gin.HandlerFunc {- return func(c *gin.Context) {- user := authenticate(c)- attributes := traceway.GetAttributesFromContext(c.Request.Context())- attributes.SetTag("user_id", user.ID)- attributes.SetTag("organization", user.OrgID)- c.Next()- }-}-```--## Usage in Handlers--Add request-specific attributes:--```go-func createOrderHandler(c *gin.Context) {- attributes := traceway.GetAttributesFromContext(c.Request.Context())- attributes.SetTag("action", "create_order")- attributes.SetTag("item_count", fmt.Sprintf("%d", len(items)))-}-```diff --git a/docs/pages/client/sdk/exceptions.mdx b/docs/pages/client/sdk/exceptions.mdxdeleted file mode 100644index 73ce06ec..00000000--- a/docs/pages/client/sdk/exceptions.mdx+++ /dev/null@@ -1,141 +0,0 @@-# Exceptions--The SDK provides multiple ways to capture exceptions, from simple error capture to context-aware variants that link exceptions to traces.--## CaptureException--Capture an error with an automatically generated stack trace:--```go-if err := doSomething(); err != nil {- traceway.CaptureException(err)-}-```--This creates a standalone issue — not linked to any trace.--## CaptureExceptionWithAttributes--Capture an error with custom attributes and an optional trace ID link:--```go-traceway.CaptureExceptionWithAttributes(- err,- map[string]string{"user_id": "123", "action": "checkout"},- &traceId, // or nil for standalone-)-```--## CaptureExceptionWithContext--Capture an error using the Go context to automatically extract the trace ID, task flag, and attributes:--```go-func handler(c *gin.Context) {- ctx := c.Request.Context()-- if err := doSomething(); err != nil {- traceway.CaptureExceptionWithContext(ctx, err)- c.AbortWithStatus(500)- return- }-}-```--This is the recommended way to capture exceptions within traced requests or tasks — it links the exception to the current trace and includes all in-scope attributes.--## CaptureTraceException--Link an exception to a specific trace by ID:--```go-traceway.CaptureTraceException(traceId, stackTraceString)-```--## CaptureTraceExceptionWithAttributes--Same as above with attributes:--```go-traceway.CaptureTraceExceptionWithAttributes(- traceId,- stackTraceString,- map[string]string{"endpoint": "GET /api/users"},-)-```--## CaptureTaskException--Link an exception to a task trace:--```go-traceway.CaptureTaskException(traceId, stackTraceString)-```--The difference from `CaptureTraceException` is that the issue is marked as belonging to a task rather than an endpoint.--## CaptureTaskExceptionWithAttributes--```go-traceway.CaptureTaskExceptionWithAttributes(- traceId,- stackTraceString,- map[string]string{"task": "report.monthly"},-)-```--## Recover--Recover from panics in goroutines and capture them as issues:--```go-func worker() {- defer traceway.Recover()-- // If this panics, the stack trace is captured- riskyOperation()-}-```--`Recover` captures the panic value and stack trace, then swallows the panic (the goroutine exits cleanly).--## RecoverWithContext--Same as `Recover` but includes attributes from the context:--```go-func worker(ctx context.Context) {- defer traceway.RecoverWithContext(ctx)-- riskyOperation()-}-```--Attributes in the context are attached to the captured issue.--## PanicError--`PanicError` is the error type used when `MeasureTask` re-panics after capturing an exception:--```go-type PanicError struct {- Value interface{} // The original panic value- Stack string // Formatted stack trace-}-```--You can check for it when recovering panics from `MeasureTask`:--```go-defer func() {- if r := recover(); r != nil {- if pe, ok := r.(*traceway.PanicError); ok {- log.Printf("Task panicked: %v", pe.Value)- }- }-}()--traceway.MeasureTask("risky.task", func(ctx context.Context) {- panic("something broke")-})-```diff --git a/docs/pages/client/sdk/index.mdx b/docs/pages/client/sdk/index.mdxdeleted file mode 100644index 7aa34834..00000000--- a/docs/pages/client/sdk/index.mdx+++ /dev/null@@ -1,46 +0,0 @@-# Go Client SDK--The `traceway` package is the core SDK for instrumenting Go applications. It captures traces, exceptions, messages, metrics, and spans.--## Installation--```bash-go get go.tracewayapp.com-```--## Packages--| Package | Import | Purpose |-|---------|--------|---------|-| Core | `go.tracewayapp.com` | All tracing, exception, message, metric, and span APIs |-| Gin Middleware | `go.tracewayapp.com/tracewaygin` | Automatic instrumentation for Gin applications |-| HTTP Middleware | `go.tracewayapp.com/tracewayhttp` | Automatic instrumentation for `net/http` servers |-| DB Instrumentation | `go.tracewayapp.com/tracewaydb` | Automatic span creation for `database/sql` operations |--## When to Use Core vs Middleware--**Use middleware** ([Gin](/client/gin-middleware) or [HTTP](/client/http-middleware)) when your application serves HTTP requests. The middleware handles initialization, trace creation, panic recovery, and data capture automatically.--**Use the core package directly** when:--- Running background tasks or workers (use `MeasureTask` or `StartTrace`)-- Capturing exceptions outside of HTTP context (use `CaptureException`)-- Sending messages (use `CaptureMessage`)-- Adding spans to traced operations (use `StartSpan`)-- Initializing Traceway without middleware (use `Init`)--## Getting Your Project Token--Before using the SDK, you need a project token from your Traceway instance:--1. Register an account on your Traceway dashboard-2. Create an organization-3. Create a project — select your framework (e.g., "Go - Gin")-4. Copy the generated project token (UUID format)--## Next Steps--- [Initialization](/client/sdk/initialization) — Connection string format and all configuration options-- [Traces](/client/sdk/traces) — Recording HTTP requests and background tasks-- [Exceptions](/client/sdk/exceptions) — Capturing errors and panics-- [Spans](/client/sdk/spans) — Timing sub-operationsdiff --git a/docs/pages/client/sdk/initialization.mdx b/docs/pages/client/sdk/initialization.mdxdeleted file mode 100644index 89ed6d2e..00000000--- a/docs/pages/client/sdk/initialization.mdx+++ /dev/null@@ -1,85 +0,0 @@-# Initialization--## Connection String--The SDK uses a connection string to configure the backend endpoint:--```-<project_token>@<api_url>-```--Examples:--```-550e8400-e29b-41d4-a716-446655440000@http://localhost:8082/api/report-your-token@https://traceway.example.com/api/report-```--## Init--Initialize the core client directly when not using middleware:--```go-import traceway "go.tracewayapp.com"--func main() {- err := traceway.Init(- "your-token@http://localhost:8082/api/report",- traceway.WithVersion("1.0.0"),- traceway.WithDebug(true),- )- if err != nil {- log.Fatal(err)- }-- // Your application code...-}-```--`Init` can only be called once. A second call returns an error.--When using middleware (`tracewaygin.New` or `tracewayhttp.New`), initialization is handled automatically — do not call `Init` separately.--## Configuration Options--| Option | Description | Default |-|--------|-------------|---------|-| `WithDebug(bool)` | Enable debug logging — prints captured telemetry to stdout | `false` |-| `WithMaxCollectionFrames(int)` | Maximum frames to buffer in the ring buffer before forcing send | `12` |-| `WithCollectionInterval(time.Duration)` | How often batched telemetry data is sent to the server | `5s` |-| `WithUploadTimeout(time.Duration)` | HTTP timeout for uploading frames to the backend | `2s` |-| `WithMetricsInterval(time.Duration)` | How often system metrics (CPU, memory, goroutines) are collected | `30s` |-| `WithVersion(string)` | Application version shown in the dashboard | `""` |-| `WithServerName(string)` | Server hostname identifier | Auto-detected from `os.Hostname()` |-| `WithSampleRate(float64)` | Percentage of normal traces to record (0.0–1.0) | `1.0` |-| `WithErrorSampleRate(float64)` | Percentage of error traces to record (0.0–1.0) | `1.0` |--## Ring Buffer Batching--The SDK uses a ring buffer system for efficient data collection:--1. **Collection** — Traces, exceptions, and metrics are collected into the current "frame"-2. **Rotation** — After the collection interval (default 5s), the frame is rotated into the send queue-3. **Upload** — Frames in the send queue are compressed (gzip) and sent to the backend-4. **Cleanup** — Successfully uploaded frames are removed from the buffer--If the backend is temporarily unreachable, frames remain in the ring buffer. The buffer holds up to `maxCollectionFrames` (default 12) frames — at the default 5s interval, this means roughly 60 seconds of data is buffered before the oldest frames are dropped.--## Sampling--Sample rates control what percentage of traces are recorded:--```go-traceway.Init(- connectionString,- // Record 10% of normal traces- traceway.WithSampleRate(0.1),- // Record all error traces- traceway.WithErrorSampleRate(1.0),-)-```--A trace is considered an error when:-- A panic was recovered-- The HTTP status code is >= 500-- Gin context errors exist (Gin middleware only)diff --git a/docs/pages/client/sdk/messages.mdx b/docs/pages/client/sdk/messages.mdxdeleted file mode 100644index 180bce97..00000000--- a/docs/pages/client/sdk/messages.mdx+++ /dev/null@@ -1,60 +0,0 @@-# Messages--Messages are explicitly captured strings that appear as issues in the dashboard. Unlike exceptions, messages don't carry automatic stack traces — they record whatever string you provide.--## CaptureMessage--Capture a standalone message:--```go-traceway.CaptureMessage("User exceeded rate limit")-```--This creates an issue with no trace link and no attributes beyond the message itself.--## CaptureMessageWithContext--Capture a message with trace linkage and attributes from the Go context:--```go-func handler(c *gin.Context) {- ctx := c.Request.Context()-- if user.RequestCount > rateLimit {- traceway.CaptureMessageWithContext(ctx, "Rate limit exceeded for user")- }-}-```--This links the message to the current trace and includes all in-scope attributes (user ID, tenant, etc.).--## CaptureMessageAttributes--Capture a message with explicit attributes:--```go-traceway.CaptureMessageAttributes(- "Payment processing delayed",- map[string]string{- "payment_id": paymentID,- "delay_ms": fmt.Sprintf("%d", delayMs),- "provider": "stripe",- },-)-```--This creates a standalone message (no trace link) with the provided attributes.--## When to Use Messages vs Exceptions--Use **exceptions** (`CaptureException`) for actual errors — things that shouldn't happen and need stack traces for debugging.--Use **messages** (`CaptureMessage`) for notable events that aren't errors:--- Rate limit hits-- Feature flag activations-- Degraded service notifications-- Business rule violations-- Audit events--Both appear in the Issues list, but messages are marked distinctly from errors.diff --git a/docs/pages/client/sdk/metrics.mdx b/docs/pages/client/sdk/metrics.mdxdeleted file mode 100644index a6350634..00000000--- a/docs/pages/client/sdk/metrics.mdx+++ /dev/null@@ -1,114 +0,0 @@-# Metrics--Metrics are time-series measurements of your application's health. System metrics are collected automatically; you can also capture custom metrics.--<img src="/dashboards/dashboards-page.png" alt="Metrics on a dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />--## CaptureMetric--Record a metric value:--```go-traceway.CaptureMetric("queue.depth", float64(len(queue)))-traceway.CaptureMetric("cache.hit_rate", hitRate)-traceway.CaptureMetric("active.users", float64(activeUsers))-```--Each call records the metric name, value, and current timestamp.--## Automatic System Metrics--The SDK automatically collects system metrics at the configured interval (default: 30 seconds):--| Metric Constant | Name | Description |-|-----------------|------|-------------|-| `MetricNameCpuUsage` | `cpu.used_pcnt` | CPU usage percentage |-| `MetricNameMemoryUsage` | `mem.used` | Memory usage in MB |-| `MetricNameMemoryTotal` | `mem.total` | Total system memory in MB |-| `MetricNameGoRoutines` | `go.go_routines` | Active goroutine count |-| `MetricNameHeapObjects` | `go.heap_objects` | Objects on the heap |-| `MetricNameNumGC` | `go.num_gc` | Total GC cycles |-| `MetricNameGCPauseTotal` | `go.gc_pause` | Total GC pause time in nanoseconds |--## Custom Metrics--Capture application-specific measurements anywhere in your code:--```go-// Queue monitoring-traceway.CaptureMetric("queue.depth", float64(len(queue)))-traceway.CaptureMetric("queue.processing_rate", itemsPerSecond)--// Cache performance-traceway.CaptureMetric("cache.hit_rate", hits / total)-traceway.CaptureMetric("cache.size", float64(cache.Len()))--// Business metrics-traceway.CaptureMetric("active.users", float64(activeUsers))-traceway.CaptureMetric("orders.per_minute", orderRate)-```--## CaptureMetricWithTags--Attach key-value tags to a metric point for filtering and grouping in the dashboard:--```go-traceway.CaptureMetricWithTags("queue.depth", float64(len(queue)), map[string]string{- "queue_name": "emails",- "priority": "high",-})--traceway.CaptureMetricWithTags("http.external_ms", latency, map[string]string{- "service": "stripe",- "region": "us-east-1",-})--traceway.CaptureMetricWithTags("cache.hit_rate", hitRate, map[string]string{- "cache": "redis",-})-```--## Tags--Tags are key-value pairs attached to individual metric points. In the dashboard, you can:--- **Filter** to specific tag values (e.g. show only `queue_name = emails`)-- **Group by** a tag to split metrics into separate series (e.g. one line per `server_name`)--Common tags:--| Tag | Example |-| -------------- | ------------------------ |-| `server_name` | `"web-1"`, `"web-2"` |-| `region` | `"us-east-1"` |-| `queue_name` | `"emails"`, `"payments"`|-| `environment` | `"production"` |--If you don't need tags, use `CaptureMetric` — it records the same data without any tag metadata.--## What to Look For--- **Memory growth** — Continuous increase suggests a memory leak-- **Goroutine growth** — Unbounded increase indicates goroutine leak-- **GC frequency** — High `num_gc` growth means allocation pressure-- **CPU spikes** — Correlate with specific endpoints or tasks--## Metrics Interval--Control how often system metrics are collected:--```go-traceway.Init(- connectionString,- traceway.WithMetricsInterval(60 * time.Second), // Collect every 60s instead of 30s-)-```--Or via middleware:--```go-tracewaygin.New(- connectionString,- tracewaygin.WithMetricsInterval(60 * time.Second),-)-```diff --git a/docs/pages/client/sdk/spans.mdx b/docs/pages/client/sdk/spans.mdxdeleted file mode 100644index af015272..00000000--- a/docs/pages/client/sdk/spans.mdx+++ /dev/null@@ -1,136 +0,0 @@-# Spans--Spans track timed sub-operations within a trace. Use them to measure where time is spent during request processing or task execution.--## StartSpan--Create a span within an active trace:--```go-func handler(c *gin.Context) {- ctx := c.Request.Context()-- span := traceway.StartSpan(ctx, "db.users.find")- user, err := db.FindUser(userID)- span.End()-- if err != nil {- c.Error(err)- return- }- c.JSON(200, user)-}-```--`StartSpan` requires a context that carries a `TraceContext` (created by middleware or `StartTrace`). If no trace context exists, it returns `nil`.--## The Defer Pattern--Use `defer` to ensure spans are ended even when errors or panics occur:--```go-func fetchData(ctx context.Context) (*Data, error) {- span := traceway.StartSpan(ctx, "api.external.fetch")- defer span.End()-- resp, err := http.Get(url)- if err != nil {- return nil, err- }- return parseResponse(resp)-}-```--`End()` is safe to call on a `nil` span and safe to call multiple times — only the first call records the duration.--## Multiple Spans--Track sequential operations:--```go-func processOrder(ctx context.Context, orderID string) error {- span := traceway.StartSpan(ctx, "db.orders.find")- order, err := db.GetOrder(orderID)- span.End()- if err != nil {- return err- }-- span = traceway.StartSpan(ctx, "logic.inventory.check")- available := checkInventory(order.Items)- span.End()- if !available {- return errors.New("items unavailable")- }-- span = traceway.StartSpan(ctx, "api.stripe.charge")- err = stripe.Charge(order.PaymentMethod, order.Total)- span.End()-- return err-}-```--## Nesting--Spans can be nested by starting child spans before ending the parent:--```go-func handler(c *gin.Context) {- ctx := c.Request.Context()-- parent := traceway.StartSpan(ctx, "process.checkout")-- child := traceway.StartSpan(ctx, "process.checkout.validate")- validate(cart)- child.End()-- child = traceway.StartSpan(ctx, "process.checkout.charge")- charge(payment)- child.End()-- parent.End()-}-```--All spans are attached to the same trace context regardless of nesting depth.--## Spans in Background Tasks--Spans work within `MeasureTask`:--```go-traceway.MeasureTask("report.monthly", func(ctx context.Context) {- span := traceway.StartSpan(ctx, "report.fetch_data")- data := fetchReportData()- span.End()-- span = traceway.StartSpan(ctx, "report.process")- result := processData(data)- span.End()-- span = traceway.StartSpan(ctx, "report.upload")- uploadToS3(result)- span.End()-})-```--## Naming Conventions--Use hierarchical names following `category.resource.action`:--```go-// Database operations-traceway.StartSpan(ctx, "db.users.find")-traceway.StartSpan(ctx, "db.orders.insert")--// External APIs-traceway.StartSpan(ctx, "api.stripe.charge")-traceway.StartSpan(ctx, "api.s3.upload")--// Cache operations-traceway.StartSpan(ctx, "cache.redis.get")--// Business logic-traceway.StartSpan(ctx, "logic.pricing.calculate")-```diff --git a/docs/pages/client/sdk/stack-traces.mdx b/docs/pages/client/sdk/stack-traces.mdxdeleted file mode 100644index 8e4228c3..00000000--- a/docs/pages/client/sdk/stack-traces.mdx+++ /dev/null@@ -1,99 +0,0 @@-# Stack Traces--The SDK provides utilities for creating errors with embedded stack traces and formatting stack trace output.--## NewStackTraceErrorf--Create an error with an embedded stack trace, using `fmt.Errorf`-style formatting:--```go-err := traceway.NewStackTraceErrorf("failed to process order %s: %w", orderID, originalErr)-```--Returns a `*StackTraceError` that implements the `error` interface. The stack trace is captured at the call site.--If the wrapped error (`%w`) already contains a `StackTraceError`, the existing stack frames are preserved instead of capturing new ones. This prevents duplicate stack traces when wrapping errors up the call chain.--## WrapWithStackTrace--Wrap an existing error with a stack trace:--```go-result, err := db.Query(query)-if err != nil {- return traceway.WrapWithStackTrace(err, 0)-}-```--The `skip` parameter controls how many stack frames to skip (0 captures from the call site).--## StackTraceError Type--Both `NewStackTraceErrorf` and `WrapWithStackTrace` return `*StackTraceError`:--```go-type StackTraceError struct {- Err error- Frames []runtime.Frame-}-```--Methods:--| Method | Description |-|--------|-------------|-| `Error() string` | Returns the underlying error's message |-| `Unwrap() error` | Returns the wrapped error (for `errors.Is`/`errors.As`) |-| `GetStackFrames() []runtime.Frame` | Returns the captured stack frames |--`StackTraceError` integrates with Go's error wrapping — `errors.Is` and `errors.As` work through it.--## Why Use StackTraceError--When you pass errors to Gin's `c.Error(err)`, the Gin middleware checks if the error contains a `StackTraceError`. If it does, the embedded stack frames are used for the issue's stack trace instead of capturing at the middleware level.--This gives you accurate stack traces that point to where the error actually originated:--```go-func handler(c *gin.Context) {- user, err := findUser(c.Param("id"))- if err != nil {- // The stack trace will point to findUser, not this handler- c.Error(traceway.NewStackTraceErrorf("user lookup failed: %w", err))- c.AbortWithStatus(500)- return- }- c.JSON(200, user)-}-```--## CaptureStack--Capture the current call stack as `[]runtime.Frame`:--```go-frames := traceway.CaptureStack(0) // 0 = capture from call site-```--The `skip` parameter controls how many frames to skip from the top.--## FormatErrorWithStack--Format an error and stack frames into a human-readable string:--```go-frames := traceway.CaptureStack(0)-formatted := traceway.FormatErrorWithStack(err, frames)-```--Output format:--```-*errors.errorString: connection refused-Dial()- /app/net/dial.go:42-Connect()- /app/db/connect.go:15-handler()- /app/api/handler.go:28-```diff --git a/docs/pages/client/sdk/traces.mdx b/docs/pages/client/sdk/traces.mdxdeleted file mode 100644index 8d0ec918..00000000--- a/docs/pages/client/sdk/traces.mdx+++ /dev/null@@ -1,110 +0,0 @@-# Traces--The core SDK provides functions for creating and recording traces — both for HTTP endpoints and background tasks.--## StartTrace--Creates a new trace context and stores it in the Go context. Use this when you need manual control over trace creation outside of middleware.--```go-ctx := traceway.StartTrace(context.Background())--// The context now carries a TraceContext with a unique ID-// Use this context for spans, exceptions, and attributes-```--## GetTraceFromContext--Retrieves the current trace context from a Go context:--```go-tc := traceway.GetTraceFromContext(ctx)-if tc != nil {- fmt.Println("Trace ID:", tc.Id)-}-```--## CaptureTrace--Records a completed HTTP trace. Typically called by middleware — you rarely need this directly.--```go-traceway.CaptureTrace(- tc, // *TraceContext- "GET /api/users", // endpoint- duration, // time.Duration- startedAt, // time.Time- 200, // statusCode- 1024, // bodySize- "192.168.1.1", // clientIP-)-```--## CaptureTraceWithAttributes--Same as `CaptureTrace` but includes attributes:--```go-traceway.CaptureTraceWithAttributes(- tc,- "GET /api/users",- duration,- startedAt,- 200,- 1024,- "192.168.1.1",- map[string]string{"user_id": "123", "tenant": "acme"},-)-```--## CaptureTask--Records a completed background task:--```go-tc := &traceway.TraceContext{- Id: uuid.NewString(),- IsTask: true,-}--start := time.Now()-// ... do work ...-duration := time.Since(start)--traceway.CaptureTask(- tc,- "report.monthly", // task name- duration,- start,- map[string]string{"report_type": "revenue"},-)-```--## MeasureTask--A convenience wrapper that creates a trace context, measures execution time, captures the task, and handles panics:--```go-traceway.MeasureTask("email.send_batch", func(ctx context.Context) {- // ctx carries a TraceContext and Attributes- // You can add spans and attributes here-- span := traceway.StartSpan(ctx, "email.fetch_recipients")- recipients := fetchRecipients()- span.End()-- span = traceway.StartSpan(ctx, "email.send_all")- sendEmails(recipients)- span.End()-})-```--`MeasureTask` automatically:--- Creates a `TraceContext` with `IsTask: true`-- Creates an `Attributes` instance available via context-- Measures execution duration-- Records the task trace via `CaptureTask`-- Recovers panics — captures the stack trace as an issue, then re-panics--If the function panics, both the task trace and the exception are recorded before the panic propagates.diff --git a/docs/pages/client/svelte/context.mdx b/docs/pages/client/svelte/context.mdxindex 757fda8b..34c9062e 100644--- a/docs/pages/client/svelte/context.mdx+++ b/docs/pages/client/svelte/context.mdx@@ -24,7 +24,7 @@ Must be called during component initialization (not in an event handler):</script>```-**Delayed Initialization:** The SDK initialization (`init()`) is deferred until `onMount`. This means the actual connection to the Traceway server happens after the component mounts, not during the initial script execution. The context functions (`captureException`, etc.) are available immediately, but events captured before mount will be queued and sent once initialization completes.+**Initialization is synchronous:** `setupTraceway` calls the SDK's `init()` immediately, during component setup, not in `onMount`. Everything captured after that point is recorded, including captures in a child component's setup code that runs before `onMount`. There is no pre-init buffer, so a capture made before `setupTraceway` runs is dropped.### Options@@ -37,7 +37,7 @@ Must be called during component initialization (not in an event handler):| Option | Type | Default | Description ||--------|------|---------|-------------|-| `debug` | `boolean` | `false` | Log events to console |+| `debug` | `boolean` | `false` | Log filtered-out exceptions and failed uploads to the console || `debounceMs` | `number` | `1500` | Batch delay in milliseconds || `retryDelayMs` | `number` | `10000` | Retry delay for failed uploads || `version` | `string` | `undefined` | Your application version |@@ -69,6 +69,7 @@ Retrieve capture functions in any child component:| `captureException(error)` | Capture an error with stack trace || `captureExceptionWithAttributes(error, attributes)` | Capture error with metadata || `captureMessage(message)` | Send a custom message |+| `recordAction(category, name, data)` | Record an action into the rolling buffer that ships with the next exception |## Complete Example@@ -78,17 +79,14 @@ Root layout:<!-- src/routes/+layout.svelte --><script>import { setupTraceway } from "@tracewayapp/svelte";- import { browser } from "$app/environment";import { PUBLIC_TRACEWAY_CONNECTION } from "$env/static/public";- if (browser) {- setupTraceway({- connectionString: PUBLIC_TRACEWAY_CONNECTION,- options: {- debug: import.meta.env.DEV,- },- });- }+ setupTraceway({+ connectionString: PUBLIC_TRACEWAY_CONNECTION,+ options: {+ debug: import.meta.env.DEV,+ },+ });</script><slot />@@ -152,7 +150,7 @@ Use `useTracewayAttributes` to bind a reactive map of attributes (`userId`, `ten</script>```-The setter accepts `null` / `undefined` as "empty map" — useful while user data loads or after logout.+The setter accepts `null` / `undefined` as "empty map", which is useful while user data loads or after logout.For imperative use (load functions, hooks), the same primitives are exported as plain functions:@@ -181,41 +179,50 @@ For advanced use cases, you can access the context key directly:## SvelteKit Error Handling-For SvelteKit's error handling, create a custom error hook:+`handleError` in `src/hooks.client.js` is where SvelteKit hands you the real error from a failed `load` or a render crash during client-side navigation. Svelte context is not available in a hook, so import the capture function directly. It is the same function `getTraceway()` returns.```javascript// src/hooks.client.js-import { getTraceway } from "@tracewayapp/svelte";+import { captureExceptionWithAttributes } from "@tracewayapp/svelte";export function handleError({ error, event }) {- // Note: Context may not be available in hooks- // Consider using the core SDK directly here- console.error("Unhandled error:", error);+ captureExceptionWithAttributes(error, {+ route: event.route?.id ?? event.url.pathname,+ });+ return { message: "Something went wrong" };}```-For client-side errors, use `+error.svelte`:+Do not capture from `+error.svelte`. That page only receives `$page.error`, which SvelteKit has already reduced to a plain `{ message }` object. Passing it to `captureException` produces an issue titled `Object: Internal Error` with no stack trace and no original message. Use the page for display and let `handleError` do the reporting.```svelte<!-- src/routes/+error.svelte --><script>import { page } from "$app/stores";- import { getTraceway } from "@tracewayapp/svelte";- import { onMount } from "svelte";-- const { captureException } = getTraceway();-- onMount(() => {- if ($page.error) {- captureException($page.error);- }- });</script><h1>Error: {$page.status}</h1><p>{$page.error?.message}</p>```+### Render Errors (Svelte 5)++Svelte 5's `<svelte:boundary>` catches errors thrown while its children render. Pass `captureSvelteError` as `onerror` to report them:++```svelte+<script>+ import { captureSvelteError } from "@tracewayapp/svelte";+ import Widget from "$lib/components/Widget.svelte";+</script>++<svelte:boundary onerror={captureSvelteError}>+ <Widget />+ {#snippet failed(error)}+ <p>Something went wrong: {error.message}</p>+ {/snippet}+</svelte:boundary>+```+## Environment VariablesFor SvelteKit, use public environment variables:@@ -238,7 +245,7 @@ PUBLIC_TRACEWAY_CONNECTION=your-token@https://traceway.example.com/api/report## Best Practices1. **Initialize early**: Call `setupTraceway` in your root layout-2. **Check for browser**: In SvelteKit, only initialize on the client+2. **Skip the browser guard**: `setupTraceway` must also run during SSR, or `getTraceway()` throws in child components3. **Use attributes**: Add context with `captureExceptionWithAttributes`4. **Handle gracefully**: Show user-friendly errors while capturing detailsdiff --git a/docs/pages/client/svelte/index.mdx b/docs/pages/client/svelte/index.mdxindex 7a353170..888fc820 100644--- a/docs/pages/client/svelte/index.mdx+++ b/docs/pages/client/svelte/index.mdx@@ -10,25 +10,22 @@ npm install @tracewayapp/svelte## Setup-Call `setupTraceway` in your root layout. The `if (browser)` guard keeps the rrweb recorder out of the SSR bundle. Session recording is on by default — the last ~30s of DOM events ship with every captured exception:+Call `setupTraceway` in your root layout. Do not wrap it in an `if (browser)` guard. `setupTraceway` is what puts the capture functions into Svelte's context, so if it is skipped during SSR, every child component that calls `getTraceway()` throws and the page returns a 500. The call is safe on the server: the rrweb recorder and the fetch/XHR instrumentation only start when `window` exists. Session recording is on by default, so the last ~30s of DOM events ship with every captured exception.```svelte<!-- src/routes/+layout.svelte --><script>import { setupTraceway } from "@tracewayapp/svelte";- import { browser } from "$app/environment";- if (browser) {- setupTraceway({- connectionString: "your-token@https://traceway.example.com/api/report",- });- }+ setupTraceway({+ connectionString: "your-token@https://traceway.example.com/api/report",+ });</script><slot />```-For a non-SvelteKit Svelte app, drop the `if (browser)` guard and import nothing from `$app/environment`.+The same layout works in a non-SvelteKit Svelte app. On Svelte 5, write `{@render children()}` instead of `<slot />` and `onclick` instead of `on:click`. The legacy forms below still compile, with a deprecation warning.## Capture Errors in Components@@ -90,19 +87,11 @@ Click the button and check your Traceway dashboard to verify the error appears.## Distributed Tracing-The SDK automatically instruments `fetch` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them — no extra configuration needed.+The SDK automatically instruments both `fetch` and `XMLHttpRequest` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them.-For Axios users, add the interceptor to your instance:--```javascript-import axios from "axios";-import { createAxiosInterceptor } from "@tracewayapp/frontend";--const api = axios.create({ baseURL: "/api" });-api.interceptors.request.use(createAxiosInterceptor());-```+Axios needs no setup. It uses `XMLHttpRequest` in the browser, which the SDK already instruments. Do not register `createAxiosInterceptor()` in a browser app: it adds a second ID under the same header name, and the combined value is rejected by the backend.-See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for full details.+Your backend still has to set the incoming header on its server span as the `traceway.distributed_trace_id` attribute, or the two sides never join. See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for the middleware.## Next Stepsdiff --git a/docs/pages/client/symfony/exceptions.mdx b/docs/pages/client/symfony/exceptions.mdxdeleted file mode 100644index 195851b6..00000000--- a/docs/pages/client/symfony/exceptions.mdx+++ /dev/null@@ -1,110 +0,0 @@-# Exceptions--The bundle automatically captures unhandled exceptions and records them on the request span. To report caught exceptions or add context to errors, use the OpenTelemetry PHP API directly.--## Recording an Exception on the Current Span--When you catch an error but want it reported to Traceway, record it as an event on the active span:--```php-use OpenTelemetry\API\Trace\Span;-use OpenTelemetry\API\Trace\StatusCode;--try {- $this->paymentGateway->charge($order);-} catch (\Throwable $e) {- $span = Span::getCurrent();- $span->recordException($e);- $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());-- throw $e;-}-```--`recordException` adds an `exception` event to the span with the type, message, and stack trace. Traceway extracts these events and creates Issues from them.--## Adding Attributes to Exceptions--Add context by setting span attributes before or after recording the exception:--```php-use OpenTelemetry\API\Trace\Span;-use OpenTelemetry\API\Trace\StatusCode;--$span = Span::getCurrent();--$span->setAttribute('user.id', $userId);-$span->setAttribute('order.id', $orderId);--try {- $this->processOrder($orderId);-} catch (\Throwable $e) {- $span->recordException($e, [- 'order.status' => 'failed',- 'retry.count' => $retryCount,- ]);- $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());- throw $e;-}-```--The additional attributes passed to `recordException` are attached to the exception event itself.--## Capturing Exceptions in Services--For exceptions in services outside of a controller, get the current span from context:--```php-namespace App\Service;--use OpenTelemetry\API\Trace\Span;-use OpenTelemetry\API\Trace\StatusCode;--class PaymentService-{- public function processPayment(string $customerId, float $amount): void- {- $span = Span::getCurrent();-- try {- $this->gateway->charge($customerId, $amount);- } catch (\Throwable $e) {- $span->recordException($e, [- 'customer.id' => $customerId,- 'payment.amount' => $amount,- ]);- $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());- throw $e;- }- }-}-```--## Custom Error Types--Custom exception classes work the same way. The exception type name appears in the Traceway dashboard:--```php-class InsufficientFundsException extends \RuntimeException-{- public function __construct(- public readonly string $accountId,- public readonly float $requested,- public readonly float $available,- ) {- parent::__construct("Insufficient funds: requested {$requested}, available {$available}");- }-}--try {- $this->withdraw($accountId, $amount);-} catch (InsufficientFundsException $e) {- $span = Span::getCurrent();- $span->recordException($e, [- 'account.id' => $e->accountId,- 'amount.requested' => $e->requested,- 'amount.available' => $e->available,- ]);- $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());-}-```diff --git a/docs/pages/client/symfony/index.mdx b/docs/pages/client/symfony/index.mdxdeleted file mode 100644index e3eba6c6..00000000--- a/docs/pages/client/symfony/index.mdx+++ /dev/null@@ -1,129 +0,0 @@-# Symfony--The `traceway/opentelemetry-symfony` bundle automatically instruments your Symfony application and exports traces to Traceway's OTLP endpoints — no manual instrumentation needed.--## Prerequisites--- PHP 8.1+-- Symfony 6.4+-- Composer-- A Traceway project (create one in the dashboard)--## Step 1: Install Packages--```bash-composer require traceway/opentelemetry-symfony open-telemetry/exporter-otlp php-http/guzzle7-adapter-```--## Step 2: Register the Bundle--If Symfony Flex doesn't register it automatically, add it to `config/bundles.php`:--```php-// config/bundles.php-return [- // ...- Traceway\OpenTelemetryBundle\OpenTelemetryBundle::class => ['all' => true],-];-```--## Step 3: Configure Environment Variables--Add the following to your `.env` file, replacing the endpoint and token with your project values:--> **Important:** `OTEL_PHP_AUTOLOAD_ENABLED=true` is required — without it, the OpenTelemetry SDK will not start and no telemetry will be collected.--```bash-OTEL_PHP_AUTOLOAD_ENABLED=true-OTEL_SERVICE_NAME=my-symfony-app-OTEL_TRACES_EXPORTER=otlp-OTEL_METRICS_EXPORTER=otlp-OTEL_LOGS_EXPORTER=otlp-OTEL_EXPORTER_OTLP_PROTOCOL=http/json-OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel-OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--Your Traceway dashboard includes a **Connection** page with a ready-made config snippet and your project token. Go to **Connection** in the sidebar to grab it.--> **Note:** Use `http/json` instead of `http/protobuf`. The protobuf encoder has known deprecation warnings on PHP 8.4+ that can cause noisy logs.--## Step 4: Initialize the SDK--Add the SDK autoloader to your `public/index.php`, after the Composer autoload:--```php-<?php-// public/index.php--use App\Kernel;--require_once dirname(__DIR__) . '/vendor/autoload.php';--\OpenTelemetry\SDK\SdkAutoloader::autoload();--$kernel = new Kernel($_SERVER['APP_ENV'] ?? 'dev', (bool) ($_SERVER['APP_DEBUG'] ?? true));-$request = \Symfony\Component\HttpFoundation\Request::createFromGlobals();-$response = $kernel->handle($request);-$response->send();-$kernel->terminate($request, $response);-```--## What Gets Captured--Once configured, the bundle automatically captures:--- **Endpoints** — every HTTP request with method and route template (e.g., `GET /users/{id}`)-- **Status codes** — 2xx, 4xx, 5xx responses-- **Exceptions** — unhandled errors with stack traces-- **Client IP, body size, user agent** — standard HTTP attributes-- **Doctrine queries** — if `doctrine/dbal` is installed-- **Console commands** — Symfony CLI commands-- **Messenger jobs** — dispatched and consumed messages (see [Tasks](/client/symfony/tasks))--## Configuration (Optional)--Create `config/packages/open_telemetry.yaml` to customize behavior:--```yaml-open_telemetry:- excluded_paths:- - /_profiler- - /_wdt- - /health- record_client_ip: true # default: true (set false for GDPR)- error_status_threshold: 500 # HTTP status >= this marked as error- messenger_root_spans: true # tasks appear as independent jobs in Traceway-```--## Test Your Integration--Add a test route to verify data is flowing:--```php-// src/Controller/TestController.php-namespace App\Controller;--use Symfony\Component\HttpFoundation\Response;-use Symfony\Component\Routing\Attribute\Route;--class TestController-{- #[Route('/testing', name: 'testing')]- public function index(): Response- {- throw new \RuntimeException("Test error from Traceway integration");- }-}-```--Visit `/testing` in your browser, then check the Traceway dashboard — the exception should appear within a few seconds.--## Next Steps--- [Exceptions](/client/symfony/exceptions) — manually capture caught exceptions with context-- [Spans](/client/symfony/spans) — create custom spans to measure sub-operations-- [Tasks](/client/symfony/tasks) — trace Symfony Messenger background jobs-- [Metrics](/client/symfony/metrics) — track custom counters, histograms, and gauges-- [Logs](/client/symfony/logs) — forward Monolog / PSR-3 logs to Traceway-- [OpenTelemetry docs](/client/otel) — how OTel traces, metrics, and logs map to Tracewaydiff --git a/docs/pages/client/symfony/logs.mdx b/docs/pages/client/symfony/logs.mdxdeleted file mode 100644index ad818d2a..00000000--- a/docs/pages/client/symfony/logs.mdx+++ /dev/null@@ -1,161 +0,0 @@-# Logs--Forward your Symfony application logs to Traceway via the OpenTelemetry Logs OTLP endpoint. The recommended path is a Monolog handler that bridges Symfony's standard PSR-3 logger to the OTel Logs SDK — all of your existing `$logger->info(...)` / `$logger->error(...)` calls keep working, and each log is automatically linked to the active trace and span.--## Install the Logs Packages--```bash-composer require \- open-telemetry/opentelemetry-logger-monolog \- open-telemetry/sdk \- open-telemetry/exporter-otlp-```--- `opentelemetry-logger-monolog` — Monolog handler that emits records to the OTel Logs SDK.-- `sdk` and `exporter-otlp` — already installed if you completed the [Quick Start](/client/symfony), listed here for completeness.--## Configure Environment Variables--Extend the env config from the Quick Start to also enable the logs exporter:--```bash-# Existing traces + metrics config from the Quick Start-OTEL_PHP_AUTOLOAD_ENABLED=true-OTEL_SERVICE_NAME=my-symfony-app-OTEL_TRACES_EXPORTER=otlp-OTEL_METRICS_EXPORTER=otlp--# New: enable OTLP logs-OTEL_LOGS_EXPORTER=otlp--OTEL_EXPORTER_OTLP_PROTOCOL=http/json-OTEL_EXPORTER_OTLP_ENDPOINT=https://your-traceway-instance.com/api/otel-OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-project-token"-```--`OTEL_EXPORTER_OTLP_ENDPOINT` is the base URL — the logs exporter appends `/v1/logs` automatically. No extra endpoint config is needed.--## Wire the Monolog Handler--Register the OTel Monolog handler in `config/packages/monolog.yaml` so every channel (app, request, security, doctrine, …) is forwarded to Traceway:--```yaml-# config/packages/monolog.yaml-monolog:- handlers:- traceway:- type: service- id: OpenTelemetry\Contrib\Logs\Monolog\Handler- level: info- channels: ~ # all channels--services:- OpenTelemetry\Contrib\Logs\Monolog\Handler:- arguments:- $loggerProvider: '@OpenTelemetry\API\Logs\LoggerProviderInterface'- $level: !php/const Monolog\Logger::INFO- $bubble: true-```--The `$loggerProvider` argument is resolved by the OTel PHP autoloader — it's available as a service as soon as `OTEL_PHP_AUTOLOAD_ENABLED=true` is set.--Set `level` to the Monolog level threshold you want forwarded. Use `debug` during early development to verify the pipeline, then raise it to `info` or `warning` in production to keep volume under control.--## Emit Logs With Trace Context--Inject Symfony's standard `Psr\Log\LoggerInterface` as usual — the Monolog handler takes care of bridging to OTel, and trace context attaches automatically:--```php-// src/Controller/OrderController.php-namespace App\Controller;--use Psr\Log\LoggerInterface;-use Symfony\Component\HttpFoundation\JsonResponse;-use Symfony\Component\HttpFoundation\Response;-use Symfony\Component\Routing\Attribute\Route;--class OrderController-{- public function __construct(- private readonly LoggerInterface $logger,- ) {}-- #[Route('/orders', methods: ['POST'])]- public function create(): Response- {- $this->logger->info('order received', ['order.id' => 'ord_123']);-- try {- // ... business logic ...- $this->logger->info('order processed', ['order.id' => 'ord_123']);- return new JsonResponse(['status' => 'ok']);- } catch (\Throwable $e) {- $this->logger->error('order failed', [- 'order.id' => 'ord_123',- 'exception' => $e->getMessage(),- ]);- throw $e;- }- }-}-```--Because the controller runs inside the OTel-instrumented request span, every log emitted here carries that span's `trace_id` and `span_id` automatically. Open the endpoint's trace in the Traceway dashboard and the **Logs** tab will show these records attached.--The same is true for logs emitted inside Messenger handlers, Console commands, and any other code that runs under an active span.--## Severity Mapping--Monolog levels are mapped to OpenTelemetry severity numbers (and therefore Traceway's TRACE → FATAL labels) as follows:--| Monolog Level | OTLP Severity Number | Traceway Severity |-| ------------- | -------------------- | ----------------- |-| `DEBUG` | 5 | DEBUG |-| `INFO` | 9 | INFO |-| `NOTICE` | 10 | INFO |-| `WARNING` | 13 | WARN |-| `ERROR` | 17 | ERROR |-| `CRITICAL` | 18 | ERROR |-| `ALERT` | 19 | ERROR |-| `EMERGENCY` | 21 | FATAL |--Only records at or above the handler's configured `level` threshold are forwarded.--## Test Your Integration--Add a route that logs at multiple levels, hit it once, and check the **Logs** page in the Traceway dashboard:--```php-// src/Controller/LogTestController.php-namespace App\Controller;--use Psr\Log\LoggerInterface;-use Symfony\Component\HttpFoundation\JsonResponse;-use Symfony\Component\HttpFoundation\Response;-use Symfony\Component\Routing\Attribute\Route;--class LogTestController-{- public function __construct(- private readonly LoggerInterface $logger,- ) {}-- #[Route('/log-test', name: 'log_test')]- public function test(): Response- {- $this->logger->debug('debug sample');- $this->logger->info('info sample', ['request.id' => 'req_1']);- $this->logger->warning('warning sample');- $this->logger->error('error sample', ['order.id' => 'ord_1']);- return new JsonResponse(['emitted' => 4]);- }-}-```--Visit `/log-test`, then open **Logs** in the dashboard. You should see all four records within a few seconds, each linked to the trace for that request.--## Next Steps--- [Exceptions](/client/symfony/exceptions) — record caught exceptions with context-- [Metrics](/client/symfony/metrics) — custom counters, histograms, and gauges-- [OTel Logs reference](/client/otel/logs) — full OTLP logs mapping and severity detailsdiff --git a/docs/pages/client/symfony/metrics.mdx b/docs/pages/client/symfony/metrics.mdxdeleted file mode 100644index af1c2274..00000000--- a/docs/pages/client/symfony/metrics.mdx+++ /dev/null@@ -1,187 +0,0 @@-# Metrics--The OTel auto-instrumentation captures basic request metrics automatically. To track custom application-specific metrics, use the OpenTelemetry PHP API to create counters, gauges, and histograms.--## Creating a Meter--Get a meter instance from the global meter provider:--```php-use OpenTelemetry\API\Globals;--$meter = Globals::meterProvider()->getMeter('app');-```--## Counter--Counters track cumulative values that only go up (e.g., total orders, requests processed):--```php-$meter = Globals::meterProvider()->getMeter('app');-$orderCounter = $meter->createCounter('orders.created', 'orders', 'Total orders created');--// Increment by 1-$orderCounter->add(1);--// Increment with attributes-$orderCounter->add(1, ['plan' => 'pro', 'region' => 'eu']);-```--## Histogram--Histograms track distributions of values (e.g., response times, payload sizes):--```php-$meter = Globals::meterProvider()->getMeter('app');-$durationHistogram = $meter->createHistogram('order.processing_ms', 'ms', 'Order processing time');--$start = hrtime(true);-$this->processOrder($order);-$durationMs = (hrtime(true) - $start) / 1e6;--$durationHistogram->record($durationMs);-```--Traceway converts histograms into average and count metrics (see [OTel metrics](/client/otel/metrics#histogram-handling)).--## Gauge--Gauges track point-in-time values that can go up or down (e.g., queue depth, active connections):--```php-$meter = Globals::meterProvider()->getMeter('app');--// Observable gauge — value is read by a callback at export time-$meter->createObservableGauge('queue.size', 'items', 'Current queue depth')- ->observe(function ($observer) {- $observer->observe($this->queue->count());- });-```--## Use Cases--### Business Metrics--```php-$meter = Globals::meterProvider()->getMeter('app');-$revenueCounter = $meter->createCounter('payments.revenue', 'usd', 'Total revenue');-$signupCounter = $meter->createCounter('users.signups', 'users', 'Total signups');--// In your payment service-public function processPayment(Order $order): void-{- $this->gateway->charge($order->total);- $revenueCounter->add($order->total, ['plan' => $order->plan]);-}--// In your registration controller-public function register(Request $request): Response-{- $user = $this->userService->create($request);- $signupCounter->add(1);- return new JsonResponse($user, 201);-}-```--### Performance Metrics--```php-$meter = Globals::meterProvider()->getMeter('app');-$apiLatency = $meter->createHistogram('external_api.latency_ms', 'ms', 'External API call duration');--public function fetchExternalData(string $endpoint): array-{- $start = hrtime(true);- try {- return $this->client->request('GET', $endpoint)->toArray();- } finally {- $durationMs = (hrtime(true) - $start) / 1e6;- $apiLatency->record($durationMs, ['endpoint' => $endpoint]);- }-}-```--### Resource Metrics--```php-$meter = Globals::meterProvider()->getMeter('app');--$meter->createObservableGauge('cache.entries', 'items', 'Cache entry count')- ->observe(function ($observer) {- $observer->observe($this->cache->getStats()['entries']);- });--$meter->createObservableGauge('cache.memory_mb', 'MB', 'Cache memory usage')- ->observe(function ($observer) {- $observer->observe($this->cache->getStats()['memory'] / 1024 / 1024);- });-```--## Metric Naming Conventions--Use dot-separated names for organization:--```php-// Good naming-$meter->createCounter('orders.created');-$meter->createHistogram('db.query_ms');-$meter->createCounter('cache.hits');--// Bad naming-$meter->createCounter('created'); // Too vague-$meter->createCounter('orderCount'); // Inconsistent style-```--## Registering Metrics as a Service--Define your meters in a Symfony service for dependency injection:--```php-// src/Telemetry/AppMetrics.php-namespace App\Telemetry;--use OpenTelemetry\API\Globals;-use OpenTelemetry\API\Metrics\CounterInterface;-use OpenTelemetry\API\Metrics\HistogramInterface;--class AppMetrics-{- public readonly CounterInterface $ordersCreated;- public readonly HistogramInterface $orderProcessingMs;- public readonly CounterInterface $paymentSuccess;- public readonly CounterInterface $paymentFailed;-- public function __construct()- {- $meter = Globals::meterProvider()->getMeter('app');-- $this->ordersCreated = $meter->createCounter('orders.created', 'orders');- $this->orderProcessingMs = $meter->createHistogram('orders.processing_ms', 'ms');- $this->paymentSuccess = $meter->createCounter('payments.success', 'payments');- $this->paymentFailed = $meter->createCounter('payments.failed', 'payments');- }-}-```--Then inject it where needed:--```php-class OrderService-{- public function __construct(- private readonly AppMetrics $metrics,- ) {}-- public function createOrder(array $items): Order- {- $start = hrtime(true);- try {- $order = $this->doCreateOrder($items);- $this->metrics->ordersCreated->add(1);- return $order;- } finally {- $this->metrics->orderProcessingMs->record((hrtime(true) - $start) / 1e6);- }- }-}-```diff --git a/docs/pages/client/symfony/spans.mdx b/docs/pages/client/symfony/spans.mdxdeleted file mode 100644index 2a648a5f..00000000--- a/docs/pages/client/symfony/spans.mdx+++ /dev/null@@ -1,118 +0,0 @@-# Spans--The bundle creates spans for HTTP requests automatically. To measure sub-operations like database queries, API calls, or business logic, create spans using the bundle's `TracingInterface` helper or the OpenTelemetry PHP API directly.--## Using TracingInterface--The bundle provides a `TracingInterface` service you can inject into any service or controller. It handles span creation, activation, error recording, and cleanup:--```php-use Traceway\OpenTelemetryBundle\TracingInterface;--class OrderService-{- public function __construct(- private readonly TracingInterface $tracing,- ) {}-- public function processOrder(int $orderId): void- {- $this->tracing->trace('order.validate', function () use ($orderId) {- $this->validateOrder($orderId);- });-- $this->tracing->trace('order.charge', function () use ($orderId) {- $this->chargePayment($orderId);- });- }-}-```--### With Attributes and Span Kind--```php-use OpenTelemetry\API\Trace\SpanKind;--$result = $this->tracing->trace(- 'stripe.charge',- fn () => $this->stripe->charge($amount),- attributes: ['payment.amount' => $amount, 'payment.currency' => 'usd'],- kind: SpanKind::KIND_CLIENT,-);-```--### Nested Spans--Child spans are automatically linked to the active parent span:--```php-$this->tracing->trace('order.fulfill', function () {- $this->tracing->trace('inventory.reserve', fn () => $this->reserve());- $this->tracing->trace('payment.charge', fn () => $this->charge());- $this->tracing->trace('email.send', fn () => $this->notify());-});-```--## Using the OpenTelemetry API Directly--For advanced use cases (span events, manual status codes, lazy attributes), use the tracer directly:--```php-use OpenTelemetry\API\Globals;--$tracer = Globals::tracerProvider()->getTracer('app');--$span = $tracer->spanBuilder('process-order')->startSpan();-$scope = $span->activate();-try {- $this->processOrder($orderId);-} finally {- $scope->detach();- $span->end();-}-```--Always end spans in a `finally` block to ensure they are closed even if an exception is thrown.--## Adding Attributes--Attach metadata to spans for filtering and debugging:--```php-$span = $tracer->spanBuilder('db-query')->startSpan();-$span->setAttribute('db.system', 'mysql');-$span->setAttribute('db.statement', 'SELECT * FROM users WHERE id = ?');-$span->setAttribute('db.row_count', $rowCount);-$span->end();-```--## Recording Errors on Spans--When a span's operation fails, record the exception and set the status:--```php-use OpenTelemetry\API\Trace\StatusCode;--$span = $tracer->spanBuilder('external-api-call')->startSpan();-try {- $response = $this->httpClient->request('GET', 'https://api.example.com/data');-} catch (\Throwable $e) {- $span->recordException($e);- $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());- throw $e;-} finally {- $span->end();-}-```--## Span Naming Conventions--Use descriptive names that indicate the operation type:--| Good | Bad |-|------|-----|-| `db.users.find` | `query` |-| `cache.sessions.get` | `cache` |-| `stripe.charge` | `api` |-| `s3.upload-image` | `upload` |-| `email.send-welcome` | `send` |diff --git a/docs/pages/client/symfony/tasks.mdx b/docs/pages/client/symfony/tasks.mdxdeleted file mode 100644index ed1d0551..00000000--- a/docs/pages/client/symfony/tasks.mdx+++ /dev/null@@ -1,84 +0,0 @@-# Tasks--The bundle automatically traces Symfony Messenger messages. Each consumed message appears as a task in Traceway with its duration, status, and any child spans.--## Setup--Nothing extra to configure — every consumed message is traced automatically as soon as the bundle is installed.--Symfony Messenger's default behavior is to parent the consumer span to the dispatcher's trace (so the request that dispatched the message and the worker that processed it appear as one continuous distributed trace). Traceway captures these as Tasks regardless of whether they're root spans, so the consumed message shows up in the Tasks page either way — flagged with a **Non-root** chip when it was triggered from another trace.--If you'd rather have each consumed message start its own independent trace, opt in:--```yaml filename="config/packages/open_telemetry.yaml"-open_telemetry:- messenger_root_spans: true-```--This is mostly a presentation choice. With `messenger_root_spans: true`, the consumer is a root Task (no chip) and the cross-trace link to the dispatcher is broken. Without it, the consumer is a non-root Task and the dispatcher → consumer relationship is preserved in the distributed-trace view.--## What Gets Captured--For each consumed message, the bundle records:--- **Span name** — the message class name (e.g., `EmailSendMessage process`)-- **Duration** — total handler execution time-- **Transport** — which transport delivered the message (e.g., `async`)-- **Errors** — exceptions are recorded with stack traces and the span is marked as failed-- **Child spans** — any spans created inside the handler nest under the task span--## Adding Attributes in Handlers--Since the bundle activates the task span before your handler runs, use `Span::getCurrent()` to add custom attributes:--```php-use OpenTelemetry\API\Trace\Span;-use Symfony\Component\Messenger\Attribute\AsMessageHandler;--#[AsMessageHandler]-class EmailSendHandler-{- public function __invoke(EmailSendMessage $message): void- {- $span = Span::getCurrent();-- $span->setAttribute('email.to', $message->to);- $span->setAttribute('email.subject', $message->subject);-- $this->mailer->send($message);- }-}-```--## Child Spans in Handlers--For handlers with distinct sub-operations, inject `TracingInterface` to create child spans. They automatically nest under the task's root span:--```php-use OpenTelemetry\API\Trace\SpanKind;-use Symfony\Component\Messenger\Attribute\AsMessageHandler;-use Traceway\OpenTelemetryBundle\TracingInterface;--#[AsMessageHandler]-class PaymentProcessorHandler-{- public function __construct(- private readonly TracingInterface $tracing,- ) {}-- public function __invoke(PaymentProcessorMessage $message): void- {- $this->tracing->trace('validate.payment', function () use ($message) {- $this->validate($message);- }, ['payment.amount' => $message->amount, 'payment.currency' => $message->currency]);-- $this->tracing->trace('charge.gateway', function () use ($message) {- $this->chargeGateway($message);- }, kind: SpanKind::KIND_CLIENT);- }-}-```--## Dispatch Tracing--The bundle also traces the dispatch side — when your controller or service dispatches a message, a PRODUCER span is created. W3C trace context is injected into the message envelope so the consumer span links back to the dispatching trace (unless `messenger_root_spans` is `true`, in which case the consumer creates an independent root span). Either way, Traceway captures the consumer as a Task and surfaces the cross-trace link when one exists.diff --git a/docs/pages/client/vue/composables.mdx b/docs/pages/client/vue/composables.mdxindex 84a6fd7f..4ff97454 100644--- a/docs/pages/client/vue/composables.mdx+++ b/docs/pages/client/vue/composables.mdx@@ -8,7 +8,7 @@ Use the `useTraceway` composable to capture errors and messages in Vue component<script setup>import { useTraceway } from "@tracewayapp/vue";-const { captureException, captureExceptionWithAttributes, captureMessage } = useTraceway();+const { captureException, captureExceptionWithAttributes, captureMessage, recordAction } = useTraceway();</script>```@@ -19,6 +19,7 @@ const { captureException, captureExceptionWithAttributes, captureMessage } = use| `captureException(error)` | Capture an error with stack trace || `captureExceptionWithAttributes(error, attributes)` | Capture an error with metadata || `captureMessage(message)` | Send a custom message |+| `recordAction(category, name, data?)` | Add a custom entry to the session action timeline |## captureException@@ -91,6 +92,26 @@ function handleCheckout() {</template>```+## recordAction++Drop a breadcrumb into the session action timeline. It does not create an issue on its own. It rides along with the next exception captured in the same session, next to the automatic network and navigation entries:++```vue+<script setup>+import { useTraceway } from "@tracewayapp/vue";++const { recordAction } = useTraceway();++function startCheckout() {+ recordAction("checkout", "started", { cartSize: 3 });+}+</script>++<template>+ <button @click="startCheckout">Start Checkout</button>+</template>+```+## Composable PatternCreate reusable async handlers:diff --git a/docs/pages/client/vue/index.mdx b/docs/pages/client/vue/index.mdxindex 6d10767f..5f0b1661 100644--- a/docs/pages/client/vue/index.mdx+++ b/docs/pages/client/vue/index.mdx@@ -10,7 +10,7 @@ npm install @tracewayapp/vue## Setup-Install the Traceway plugin in your Vue application. Session recording is on by default — the last ~30s of DOM events ship with every captured exception:+Install the Traceway plugin in your Vue application. Session recording is on by default; the last ~30s of DOM events ship with every captured exception:```javascriptimport { createApp } from "vue";@@ -86,19 +86,11 @@ Click the button and check your Traceway dashboard to verify the error appears.## Distributed Tracing-The SDK automatically instruments `fetch` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them — no extra configuration needed.+The SDK automatically instruments both `fetch` and `XMLHttpRequest` to propagate a `traceway-trace-id` header on same-origin requests. This links frontend errors to the backend requests that caused them.-For Axios users, add the interceptor to your instance:+Axios needs no setup. It uses `XMLHttpRequest` in the browser, which the SDK already instruments. Do not register `createAxiosInterceptor()` in a browser app: it adds a second ID under the same header name, and the combined value is rejected by the backend.-```javascript-import axios from "axios";-import { createAxiosInterceptor } from "@tracewayapp/frontend";--const api = axios.create({ baseURL: "/api" });-api.interceptors.request.use(createAxiosInterceptor());-```--See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for full details.+Your backend still has to set the incoming header on its server span as the `traceway.distributed_trace_id` attribute, or the two sides never join. See the [Distributed Tracing](/client/js-sdk/distributed-tracing) guide for the middleware.## Next Stepsdiff --git a/docs/pages/client/vue/plugin.mdx b/docs/pages/client/vue/plugin.mdxindex 41a9cb3f..aa7ba0df 100644--- a/docs/pages/client/vue/plugin.mdx+++ b/docs/pages/client/vue/plugin.mdx@@ -91,6 +91,25 @@ app.config.errorHandler = (err, instance, info) => {};```+Vue has a single `app.config.errorHandler` slot, and the plugin assigns it. If your app assigns its own handler after `app.use(createTracewayPlugin(...))`, it replaces Traceway's and component errors stop reaching the dashboard. To keep both, capture the error yourself inside your handler:++```javascript+import { captureException, captureMessage } from "@tracewayapp/vue";++app.use(createTracewayPlugin({+ connectionString: "your-token@https://traceway.example.com/api/report",+}));++app.config.errorHandler = (err, instance, info) => {+ if (err instanceof Error) {+ captureException(err);+ } else {+ captureMessage(String(err));+ }+ myOwnErrorReporter(err, info);+};+```+## Environment-Specific Configuration```javascript@@ -108,12 +127,50 @@ app.use(createTracewayPlugin({## Custom Attributes-Attach app-level identifiers (`userId`, `tenant`, feature flags, etc.) to every session and exception using the imperative scope API. Drive it from `watchEffect` so it tracks reactive state:+Attach app-level identifiers (`userId`, `tenant`, feature flags, etc.) to every session and exception. The `useTracewayAttributes` composable is the Vue way to do it. Pass a getter and it tracks reactive state, pushes only the keys that changed, and removes the keys it owns when the component unmounts:++```vue+<script setup>+import { useTracewayAttributes } from "@tracewayapp/vue";+import { useUser } from "./auth";++const { user, org } = useUser();+useTracewayAttributes(() =>+ user.value && org.value+ ? { userId: user.value.id, tenant: org.value.id }+ : null+);+</script>+```++Return `null` when there is nothing to attach. The composable then removes the keys it had set.++There is a declarative wrapper for the same thing. It renders nothing and removes its keys on unmount:++```vue+<script setup>+import { TracewayAttributes } from "@tracewayapp/vue";+import { useUser } from "./auth";++const { user, org } = useUser();+</script>++<template>+ <TracewayAttributes+ v-if="user && org"+ :attributes="{ userId: user.id, tenant: org.id }"+ />+</template>+```++Treat every key you pass to the composable or the component as owned by it. Do not set the same key with `setAttribute` elsewhere, because the unmount cleanup will remove it.++You can also drive the scope imperatively from `watchEffect`:```vue<script setup>import { watchEffect } from "vue";-import { setAttributes, clearAttributes } from "@tracewayapp/vue";+import { setAttributes, removeAttribute } from "@tracewayapp/vue";import { useUser } from "./auth";const { user, org } = useUser();@@ -121,12 +178,15 @@ watchEffect(() => {if (user.value && org.value) {setAttributes({ userId: user.value.id, tenant: org.value.id });} else {- clearAttributes();+ removeAttribute("userId");+ removeAttribute("tenant");}});</script>```+Use `removeAttribute` per key here rather than `clearAttributes()`. `clearAttributes()` empties the whole global scope, so it also drops keys set by other parts of the app.+The full API (`setAttribute`, `setAttributes`, `removeAttribute`, `clearAttributes`) is re-exported from `@tracewayapp/vue`. Layering on each event: **defaults < global scope < per-call**. See [Sessions](/learn/sessions) for the full attribute model.## Manual Injection@@ -166,7 +226,7 @@ app.use(createTracewayPlugin({For Nuxt 3, create a plugin:```javascript-// plugins/traceway.client.ts+// plugins/traceway.tsimport { createTracewayPlugin } from "@tracewayapp/vue";export default defineNuxtPlugin((nuxtApp) => {@@ -176,6 +236,8 @@ export default defineNuxtPlugin((nuxtApp) => {});```+Name the file `traceway.ts`, not `traceway.client.ts`. A `.client` plugin never runs during SSR, so the injection key is missing on the server and the first component calling `useTraceway()` fails the render with `useTraceway must be used within a Vue app that has installed the Traceway plugin` (HTTP 500). The plugin is safe to run universally: it only installs the browser handlers and the recorder when `window` exists, so the server pass just provides the injection key.+Add to `nuxt.config.ts`:```typescriptdiff --git a/docs/pages/learn/_meta.json b/docs/pages/learn/_meta.jsonindex 5231e9fd..2b2b51c0 100644--- a/docs/pages/learn/_meta.json+++ b/docs/pages/learn/_meta.json@@ -1,6 +1,7 @@{"index": "Overview","data-flow": "Data Flow",+ "projects": "Project Structure","traces": "Traces","ai-tracing": "AI Tracing","issues": "Issues",diff --git a/docs/pages/learn/ai-tracing.mdx b/docs/pages/learn/ai-tracing.mdxindex 4f4e6ccd..5871a222 100644--- a/docs/pages/learn/ai-tracing.mdx+++ b/docs/pages/learn/ai-tracing.mdx@@ -1,16 +1,16 @@# AI Tracing-**AI Tracing** lets you monitor LLM and AI model calls across your application. Every API call to an AI provider — whether it's OpenAI, Anthropic, Cohere, or any model routed through OpenRouter — is captured as an AI trace with full cost, token usage, and conversation data.+**AI Tracing** lets you monitor LLM and AI model calls across your application. Every API call to an AI provider, whether it's OpenAI, Anthropic, Cohere, or any model routed through OpenRouter, is captured as an AI trace with full cost, token usage, and conversation data.## Why AI Observability MattersAI API calls are fundamentally different from traditional HTTP requests:-- **Costs are unpredictable** — A single prompt with a large context window can cost 100x more than average. Without visibility, cost spikes go unnoticed until the invoice arrives.-- **Latency varies wildly** — The same model can respond in 500ms or 30 seconds depending on prompt length, model load, and provider routing. You need P50/P95 breakdowns to understand real-world performance.-- **Token usage drives cost** — Input and output tokens are billed separately at different rates. Cached tokens and reasoning tokens add further complexity. You need per-call granularity.-- **Prompt/completion quality matters** — Seeing the actual conversation helps debug unexpected model behavior, hallucinations, or prompt injection issues.-- **Provider reliability differs** — When routing through multiple providers, you need to know which provider is slower, more expensive, or failing more often.+- **Costs are unpredictable.** A single prompt with a large context window can cost 100x more than average. Without visibility, cost spikes go unnoticed until the invoice arrives.+- **Latency varies wildly.** The same model can respond in 500ms or 30 seconds depending on prompt length, model load, and provider routing. You need P50/P95 breakdowns to understand real-world performance.+- **Token usage drives cost.** Input and output tokens are billed separately at different rates. Cached tokens and reasoning tokens add further complexity. You need per-call granularity.+- **Prompt/completion quality matters.** Seeing the actual conversation helps debug unexpected model behavior, hallucinations, or prompt injection issues.+- **Provider reliability differs.** When routing through multiple providers, you need to know which provider is slower, more expensive, or failing more often.## What Gets Captured@@ -38,15 +38,17 @@ Every AI trace records:## How It Works-AI traces are ingested via the [OpenTelemetry](/client/otel) OTLP/HTTP endpoint. Any span carrying `gen_ai.*` semantic convention attributes is promoted to an AI trace — **regardless of whether it's a root span or a child** of an HTTP request, queue job, or scheduled task. This matters in practice: most LLM calls happen inside a request handler, so the gen_ai span is a child of a SERVER span. Traceway captures both.+AI traces are ingested via the [OpenTelemetry](/client/otel) OTLP/HTTP endpoint. Any span carrying `gen_ai.*` semantic convention attributes is promoted to an AI trace, **regardless of whether it's a root span or a child** of an HTTP request, queue job, or scheduled task. This matters in practice: most LLM calls happen inside a request handler, so the gen_ai span is a child of a SERVER span. Traceway captures both.++One caveat: `gen_ai.*` is checked last. A span that already looks like an endpoint or a task wins first, so a SERVER span with HTTP attributes becomes an Endpoint and a `CONSUMER` span becomes a Task, even when both carry `gen_ai.*`. Neither reaches AI Traces. Put every model call on its own child span and the rule always matches. See [Traces](/client/otel/traces) for the full ordered list.The data flow:-1. **OTLP span arrives** — From an AI provider broadcast (e.g. [OpenRouter](/client/openrouter)) or your own SDK instrumentation. The span contains `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.total_cost`, and other standardized attributes.-2. **Traceway promotes the span by attributes, not by position** — If a span has `gen_ai.*` attributes, it lands in the AI Traces table. Root spans use the OTel trace_id as their id; non-root child spans use their own span_id, so a request handler and the LLM call it made each get their own row without collision.-3. **Conversation content stored separately** — The full prompt and completion are stored in object storage (S3 or local filesystem) to keep the database lean.-4. **Linked across the trace** — `distributed_trace_id` is auto-derived from the OTel trace_id, so a child AI trace and its parent endpoint show up together in the distributed-trace view with no extra configuration.-5. **Dashboard shows aggregated view** — AI traces are grouped by trace name (your agent or workflow identifier), with drill-down to individual calls. Groups containing non-root calls are flagged with a **Non-root** chip.+1. **OTLP span arrives.** From an AI provider broadcast (e.g. [OpenRouter](/client/openrouter)) or your own SDK instrumentation. The span contains `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.total_cost`, and other standardized attributes.+2. **Traceway promotes the span by attributes, not by position.** If a span has `gen_ai.*` attributes, it lands in the AI Traces table. Root spans use the OTel trace_id as their id; non-root child spans use their own span_id, so a request handler and the LLM call it made each get their own row without collision.+3. **Conversation content stored separately.** The full prompt and completion are stored in object storage (S3 or local filesystem) to keep the database lean.+4. **Linked across the trace.** `distributed_trace_id` is auto-derived from the OTel trace_id, so a child AI trace and its parent endpoint show up together in the distributed-trace view with no extra configuration.+5. **Dashboard shows aggregated view.** AI traces are grouped by trace name (your agent or workflow identifier), with drill-down to individual calls. Groups containing non-root calls are flagged with a **Non-root** chip.## Three-Level Drill-Down@@ -54,13 +56,13 @@ AI traces follow the same drill-down pattern as [endpoints](/learn/traces#endpoi### Level 1: Grouped by Trace Name-See all your AI agents/workflows at a glance with aggregate metrics — total calls, P50/P95 duration, total tokens consumed, and total cost.+See all your AI agents/workflows at a glance with aggregate metrics: total calls, P50/P95 duration, total tokens consumed, and total cost.<img src="/ai-traces.png" alt="AI Traces Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />### Level 2: Individual Calls-Click a trace name to see every individual call — timestamp, duration, model used, tokens in/out, cost, and provider.+Click a trace name to see every individual call: timestamp, duration, model used, tokens in/out, cost, and provider.### Level 3: Call Detail@@ -130,12 +132,12 @@ Three notification rule types cover AI spend and content:## Conversation Storage-Conversation content (prompts and completions) can be large — a single prompt with RAG context might be 50KB+. To keep the database fast for aggregation queries, Traceway stores conversation content in object storage (S3 or local filesystem) and reads it on demand when you view a specific trace.+Conversation content (prompts and completions) can be large. A single prompt with RAG context might be 50KB+. To keep the database fast for aggregation queries, Traceway stores conversation content in object storage (S3 or local filesystem) and reads it on demand when you view a specific trace.This means:-- **List views are fast** — No large text fields in aggregation queries-- **Detail views load conversations** — The full input/output is fetched from storage when you drill into a specific call-- **Privacy-friendly** — You can configure your AI provider to exclude prompt/completion data if needed+- **List views are fast**: no large text fields in aggregation queries+- **Detail views load conversations**: the full input/output is fetched from storage when you drill into a specific call+- **Privacy-friendly**: you can configure your AI provider to exclude prompt/completion data if needed## Supported Providersdiff --git a/docs/pages/learn/alerts.mdx b/docs/pages/learn/alerts.mdxindex fc288db4..02f0831d 100644--- a/docs/pages/learn/alerts.mdx+++ b/docs/pages/learn/alerts.mdx@@ -1,6 +1,6 @@# Alerts-Alerts notify you when conditions in your application require attention — errors, latency spikes, metric thresholds, or missing data. Configure channels and rules to get notified through email, Slack, webhooks, GitHub Issues, Pushover, or Telegram, or to page whoever is [on call](/learn/on-call).+Alerts notify you when conditions in your application require attention: errors, latency spikes, metric thresholds, or missing data. Configure channels and rules to get notified through email, Slack, webhooks, GitHub Issues, Pushover, or Telegram, or to page whoever is [on call](/learn/on-call).## How Alerts Work@@ -41,7 +41,7 @@ To send alerts to a Slack channel, you need an **Incoming Webhook URL**:2. Choose **From scratch**, give it a name (e.g. "Traceway Alerts"), and select your workspace3. In the app settings, navigate to **Incoming Webhooks** and toggle it **on**4. Click **Add New Webhook to Workspace**, select the channel you want alerts posted to, and click **Allow**-5. Copy the **Webhook URL** — it will look like `https://hooks.slack.com/services/T.../B.../xxx`+5. Copy the **Webhook URL**. It will look like `https://hooks.slack.com/services/T.../B.../xxx`Then configure the Slack channel in Traceway:@@ -63,7 +63,7 @@ To create GitHub issues from alerts, you need a **Personal Access Token** with i4. Set a descriptive name (e.g. "Traceway Alerts") and choose an expiration5. Under **Repository access**, select **Only select repositories** and choose the repo where issues should be created6. Under **Permissions → Repository permissions**, set **Issues** to **Read and write**-7. Click **Generate token** and copy it — you won't be able to see it again+7. Click **Generate token** and copy it; you won't be able to see it againThen configure the GitHub channel in Traceway:@@ -80,11 +80,11 @@ Then configure the GitHub channel in Traceway:To send alerts to Telegram, you need a **Bot Token** and a **Chat ID**:-1. Open Telegram and start a chat with [@BotFather](https://t.me/BotFather). Send `/newbot` and follow the prompts to give your bot a name and username. BotFather will reply with a **Bot Token** in the form `123456789:ABCdefGhIJKlmNoPQRstuVwxyZ` — copy it.+1. Open Telegram and start a chat with [@BotFather](https://t.me/BotFather). Send `/newbot` and follow the prompts to give your bot a name and username. BotFather will reply with a **Bot Token** in the form `123456789:ABCdefGhIJKlmNoPQRstuVwxyZ`. Copy it.2. Open a chat with your new bot and send it `/start` so it is allowed to message you.3. Create a private channel in Telegram for the alerts.4. Open the channel's settings → **Administrators** → **Add Admin**, search for your bot's username, and add it with permission to post messages.-5. Forward any message from the channel to [@userinfobot](https://t.me/userinfobot) — it will reply with the channel's **Chat ID** (a negative number, e.g. `-1001234567890`). Copy it including the minus sign.+5. Forward any message from the channel to [@userinfobot](https://t.me/userinfobot); it will reply with the channel's **Chat ID** (a negative number, e.g. `-1001234567890`). Copy it including the minus sign.Then configure the Telegram channel in Traceway:@@ -258,7 +258,7 @@ Fires when the percentage of task executions that recorded an error exceeds the### Metric Threshold-Fires when an aggregated [metric](/learn/metrics) value violates the configured comparison over the lookback window. Works for both system metrics (`cpu.used_pcnt`, `mem.used_pcnt`, ...) and custom metrics. If the window contains no data points the rule does not fire; pair it with a No Data rule if you also want to know when the metric stops reporting.+Fires when an aggregated [metric](/learn/metrics) value violates the configured comparison over the lookback window. Works for both system metrics (`cpu.used_pcnt`, `mem.used`, ...) and custom metrics. If the window contains no data points the rule does not fire; pair it with a No Data rule if you also want to know when the metric stops reporting.| Option | Default | Description ||--------|---------|-------------|diff --git a/docs/pages/learn/attributes.mdx b/docs/pages/learn/attributes.mdxindex e6717f22..4862a86c 100644--- a/docs/pages/learn/attributes.mdx+++ b/docs/pages/learn/attributes.mdx@@ -1,6 +1,6 @@# Attributes-Attributes are key-value metadata attached to traces and issues. They provide the context you need to debug problems — who was affected, what they were doing, and in what environment.+Attributes are key-value metadata attached to traces and issues. They provide the context you need to debug problems: who was affected, what they were doing, and in what environment.## Structure@@ -19,8 +19,8 @@ Examples of common attributes:Since all values are strings, there are two ways they get displayed in the dashboard:-- **Plain strings** — Displayed as-is, without any formatting (e.g., `"12345"`, `"production"`)-- **JSON strings** — If the string contains valid JSON, the dashboard renders it in a JSON viewer with syntax highlighting and collapsible sections (e.g., `'{"name": "John", "age": 30}'`)+- **Plain strings**: Displayed as-is, without any formatting (e.g., `"12345"`, `"production"`)+- **JSON strings**: If the string contains valid JSON, the dashboard renders it in a JSON viewer with syntax highlighting and collapsible sections (e.g., `'{"name": "John", "age": 30}'`)This means you can attach structured data to any trace or issue by serializing it to a JSON string.@@ -50,8 +50,8 @@ service: user-api service: user-api## Where Attributes Appear-- **Traces** (endpoints and tasks) — attributes are recorded with the trace-- **Issues** (exceptions and messages) — attributes in scope at capture time are attached to the issue+- **Traces** (endpoints and tasks): attributes are recorded with the trace+- **Issues** (exceptions and messages): attributes in scope at capture time are attached to the issueIn the dashboard, attributes appear in the detail view for endpoints, tasks, and issues.diff --git a/docs/pages/learn/cli.mdx b/docs/pages/learn/cli.mdxindex aeb6b280..2bd89468 100644--- a/docs/pages/learn/cli.mdx+++ b/docs/pages/learn/cli.mdx@@ -2,7 +2,7 @@import { Callout } from "nextra/components";-`traceway` is a command-line client for the Traceway HTTP API — exceptions, logs, endpoints, tasks, traces, and metrics from a terminal. It is designed to be first-class for both humans (with `gh`-style ergonomics) and LLM agents invoking it via shell tools: JSON output when piped, stable error identifiers and exit codes, and nothing that hangs waiting for input. For a quick tour, see the [CLI product page](https://tracewayapp.com/product/cli).+`traceway` is a command-line client for the Traceway HTTP API: exceptions, logs, endpoints, tasks, traces, and metrics from a terminal. It is designed to be first-class for both humans (with `gh`-style ergonomics) and LLM agents invoking it via shell tools: JSON output when piped, stable error identifiers and exit codes, and nothing that hangs waiting for input. For a quick tour, see the [CLI product page](https://tracewayapp.com/product/cli).## Installation@@ -12,11 +12,11 @@ import { Callout } from "nextra/components";curl -fsSL https://cli.tracewayapp.com/install.sh | sh```-The installer detects your OS and architecture, downloads the latest CLI release from GitHub, verifies its sha256 checksum, and installs `traceway` to `/usr/local/bin` (when writable) or `~/.local/bin` — no sudo. Two environment variables tweak it: `TRACEWAY_CLI_VERSION` pins a version (e.g. `1.9.3`) and `TRACEWAY_INSTALL_DIR` overrides the target directory.+The installer detects your OS and architecture, downloads the latest CLI release from GitHub, verifies its sha256 checksum, and installs `traceway` to `/usr/local/bin` (when writable) or `~/.local/bin`, without sudo. Two environment variables tweak it: `TRACEWAY_CLI_VERSION` pins a version (e.g. `1.9.3`) and `TRACEWAY_INSTALL_DIR` overrides the target directory.### Prebuilt binaries-Every release publishes prebuilt binaries at [github.com/tracewayapp/traceway/releases](https://github.com/tracewayapp/traceway/releases) under the matching `CLI vX.Y.Z` tag — the CLI version tracks the backend release. Download the archive for your platform (`traceway_<version>_<os>_<arch>.tar.gz`, or `.zip` on Windows), extract it, and put `traceway` on your `PATH`. This is also the route for Windows, which the install script doesn't cover.+Every release publishes prebuilt binaries at [github.com/tracewayapp/traceway/releases](https://github.com/tracewayapp/traceway/releases) under the matching `CLI vX.Y.Z` tag (the CLI version tracks the backend release). Download the archive for your platform (`traceway_<version>_<os>_<arch>.tar.gz`, or `.zip` on Windows), extract it, and put `traceway` on your `PATH`. This is also the route for Windows, which the install script doesn't cover.### From source@@ -39,7 +39,7 @@ Verify an installed binary with `traceway version` (source builds report `dev`).## Quick start```bash-# 1. log in — prints a URL + short code, you approve in the browser+# 1. log in: prints a URL + short code, you approve in the browsertraceway login --url https://cloud.tracewayapp.com# 2. pick a project (one-time; future calls use it implicitly)@@ -53,7 +53,7 @@ traceway endpoints list --since 1htraceway metrics query --name http.server.duration --aggregation avg --since 1h```-The default `traceway login` runs a browser device flow with an auto-refreshing token, so you stay logged in for up to 90 days of inactivity. Password login (`--password`) and personal access tokens (`--token`) are also supported — see [CLI Authentication](/learn/cli-auth) for all three modes, and for what to use in CI.+The default `traceway login` runs a browser device flow with an auto-refreshing token, so you stay logged in for up to 90 days of inactivity. Password login (`--password`) and personal access tokens (`--token`) are also supported; see [CLI Authentication](/learn/cli-auth) for all three modes, and for what to use in CI.## Commands@@ -81,7 +81,7 @@ The default `traceway login` runs a browser device flow with an auto-refreshingRun `traceway <command> --help` for full per-command flags.<Callout type="info">- The `show` / `occurrence` commands take a UUID plus a **required** timestamp flag (`--recorded-at`, or `--started-at` for `sessions`). Telemetry tables are partitioned by day; the timestamp bounds the query so the lookup prunes partitions instead of scanning all of them. Get the id and its timestamp together — from a dashboard URL's `?t=` param, a notification's `Occurred at`, or a list row's `recordedAt`. Omitting the flag exits 2 (`usage_error`).+ The `show` / `occurrence` commands take a UUID plus a **required** timestamp flag (`--recorded-at`, or `--started-at` for `sessions`). Telemetry tables are partitioned by day; the timestamp bounds the query so the lookup prunes partitions instead of scanning all of them. Get the id and its timestamp together: from a dashboard URL's `?t=` param, a notification's `Occurred at`, or a list row's `recordedAt`. Omitting the flag exits 2 (`usage_error`).</Callout>## Profiles@@ -97,7 +97,7 @@ traceway --profile personal exceptions list # one-off override## Output formats-The `--output` flag picks the format. The default is `table` on a TTY and `json` otherwise — piping always gets machine-readable output.+The `--output` flag picks the format. The default is `table` on a TTY and `json` otherwise, so piping always gets machine-readable output.| Format | Use ||---|---|@@ -113,7 +113,7 @@ traceway exceptions list --output json --fields exceptionHash,count,lastSeen## Errors and exit codes-Every error writes a stable JSON envelope to stderr (in `json` / `yaml` modes — prose in `table` mode):+Every error writes a stable JSON envelope to stderr (in `json` / `yaml` modes; prose in `table` mode):```json{"error":"token_expired","message":"session expired or invalid","hint":"traceway login","exit_code":4}@@ -140,7 +140,7 @@ The `error` field is a stable snake_case identifier that scripts and LLMs can br- Or set `TRACEWAY_ASSUME_YES=1` in the environment.- Or run interactively and answer the `Continue? [y/N]` prompt.-Calling a mutating command from a non-TTY context (script, LLM tool call) without one of the opt-ins fails immediately with `usage_error` (exit 2) — no hung prompts.+Calling a mutating command from a non-TTY context (script, LLM tool call) without one of the opt-ins fails immediately with `usage_error` (exit 2); no hung prompts.## MCP serverdiff --git a/docs/pages/learn/contributing.mdx b/docs/pages/learn/contributing.mdxindex 4e2fb237..60235471 100644--- a/docs/pages/learn/contributing.mdx+++ b/docs/pages/learn/contributing.mdx@@ -2,7 +2,7 @@import { Callout } from "nextra/components";-This guide covers how to contribute to Traceway — from filing an issue to getting a PR merged.+This guide covers how to contribute to Traceway, from filing an issue to getting a PR merged.## Community@@ -14,11 +14,11 @@ The fastest way to ask questions, share ideas, or get help from maintainers is [Search open issues first to avoid duplicates. When you file a bug, include:-- **What happened** — a clear description of the bug-- **What you expected** — what should have happened instead-- **Steps to reproduce** — the smallest sequence of steps that triggers it-- **Environment** — Traceway version, deployment type (SQLite, Docker Compose, all-in-one), OS-- **Logs** — any relevant error messages or stack traces+- **What happened**: a clear description of the bug+- **What you expected**: what should have happened instead+- **Steps to reproduce**: the smallest sequence of steps that triggers it+- **Environment**: Traceway version, deployment type (SQLite, Docker Compose, all-in-one), OS+- **Logs**: any relevant error messages or stack tracesThe more precise the reproduction steps, the faster the fix.@@ -34,8 +34,8 @@ Maintainers add labels after the issue is filed. You don't need to set them your| Directory | Description ||-----------|-------------|-| `backend/` | Go/Gin API server — handles telemetry ingestion, REST API, and database access |-| `frontend/` | SvelteKit 2 dashboard — the web UI users interact with |+| `backend/` | Go/Gin API server: handles telemetry ingestion, REST API, and database access |+| `frontend/` | SvelteKit 2 dashboard: the web UI users interact with || `docs/` | Nextra documentation site (what you're reading now) || `examples/` | Working example apps demonstrating embedded mode || `scripts/` | Helper scripts for testing and CI |@@ -54,7 +54,7 @@ Maintainers add labels after the issue is filed. You don't need to set them your- **Go** 1.25 or later- **Node.js** 18+ and npm- **Docker** (for running the full ClickHouse + PostgreSQL test suite)-- **ClickHouse** and **PostgreSQL** (only if running the standalone server locally — not needed for embedded/SQLite mode)+- **ClickHouse** and **PostgreSQL** (only if running the standalone server locally; not needed for embedded/SQLite mode)## Build Tags@@ -73,7 +73,7 @@ See the [Build Tags](/learn/build-tags) page for more details.### Backend (embedded/SQLite mode)-The fastest way to get going — no databases to install:+The fastest way to get going (no databases to install):```bashcd backend@@ -151,7 +151,7 @@ This builds a Docker image with both databases, applies all migrations, and runs### OTEL trace converter tests-These test the OTLP trace parsing logic using snapshot/golden-file testing. No database required — `convertTraces()` is a pure function.+These test the OTLP trace parsing logic using snapshot/golden-file testing. No database required: `convertTraces()` is a pure function.```bashcd backend@@ -191,13 +191,13 @@ The SDK sends telemetry (traces, exceptions, metrics) as gzipped JSON to the bacTwo separate auth systems:-- **Client auth**: Project bearer tokens — used by the SDK to send telemetry (`Authorization: Bearer <project_token>`)-- **App auth**: JWT tokens — used by the dashboard for user sessions (`Authorization: Bearer <jwt_token>`)+- **Client auth**: Project bearer tokens, used by the SDK to send telemetry (`Authorization: Bearer <project_token>`)+- **App auth**: JWT tokens, used by the dashboard for user sessions (`Authorization: Bearer <jwt_token>`)### Database Split-- **PostgreSQL** (or SQLite in embedded mode): Relational data that gets updated — users, organizations, projects, invitations, notification rules, dashboards and templates-- **ClickHouse** (or SQLite in embedded mode): High-volume append-only telemetry — traces, exceptions, metrics, spans, tasks, session recordings+- **PostgreSQL** (or SQLite in embedded mode): Relational data that gets updated, such as users, organizations, projects, invitations, notification rules, dashboards and templates+- **ClickHouse** (or SQLite in embedded mode): High-volume append-only telemetry, such as traces, exceptions, metrics, spans, tasks, session recordings### Backend Patterns@@ -306,18 +306,18 @@ All pull requests require **at least one approving review** before merging. This### What reviewers look at-- **Correctness** — does the code do what the PR says it does?+- **Correctness**: does the code do what the PR says it does?- **Testing**: are the requirements in "Before submitting" met? Repository changes need the ClickHouse + PostgreSQL Docker tests to pass.-- **Patterns** — does the change follow the conventions in [CLAUDE.md](https://github.com/tracewayapp/traceway/blob/main/CLAUDE.md)? Reviewers will push back on divergence from established patterns.-- **Scope** — is the PR focused? A small, focused PR is always preferred over one that mixes unrelated changes.+- **Patterns**: does the change follow the conventions in [CLAUDE.md](https://github.com/tracewayapp/traceway/blob/main/CLAUDE.md)? Reviewers will push back on divergence from established patterns.+- **Scope**: is the PR focused? A small, focused PR is always preferred over one that mixes unrelated changes.### Merge strategy-PRs are squash-merged, so write a clear PR title and description — they become the commit message on `main`. Don't just say "fix bug"; say what bug, why it happened, and what you changed.+PRs are squash-merged, so write a clear PR title and description; they become the commit message on `main`. Don't just say "fix bug"; say what bug, why it happened, and what you changed.### Review timeline-Maintainers aim to review PRs within a few days. If a PR has been open for more than a week with no response, ping in Discord. If you push new commits after requesting review, leave a comment explaining what changed — don't silently update the branch.+Maintainers aim to review PRs within a few days. If a PR has been open for more than a week with no response, ping in Discord. If you push new commits after requesting review, leave a comment explaining what changed. Don't silently update the branch.## Using AI Toolsdiff --git a/docs/pages/learn/dashboards.mdx b/docs/pages/learn/dashboards.mdxindex 12e2f3f2..4958df58 100644--- a/docs/pages/learn/dashboards.mdx+++ b/docs/pages/learn/dashboards.mdx@@ -34,7 +34,7 @@ Templates are ready-made dashboards you install and then customize freely. The m| Template | Category | Source of the metrics || --- | --- | --- || OTelemetry Server Agent | server | [Traceway OTel Agent](/learn/otel-agent) host CPU, memory, disk and network metrics |-| Go Application | go | Go SDK runtime and process metrics (goroutines, heap, GC, CPU, memory) |+| Go Application | go | Go runtime and process metrics (goroutines, heap, GC, CPU, memory) || Traceway ClickHouse | traceway | A monitored Traceway instance's `traceway.ch.*` metrics || Traceway DuckDB | traceway | A monitored Traceway instance's `traceway.duckdb.*` metrics |diff --git a/docs/pages/learn/data-flow.mdx b/docs/pages/learn/data-flow.mdxindex 3f2c1a6b..2a20c8c6 100644--- a/docs/pages/learn/data-flow.mdx+++ b/docs/pages/learn/data-flow.mdx@@ -24,7 +24,7 @@ Here's how the concepts connect in a typical scheduled/async task:1. **System metrics collected** → CPU, memory, goroutines, and GC stats are captured automatically every 30 seconds (Automatically by the client SDK)2. **Custom metrics captured** → Your code calls `CaptureMetric` or `CaptureMetricWithTags` to record application-specific measurements with optional tags3. **Batched and uploaded** → All metrics are batched, compressed, and sent to Traceway with the next collection frame-4. **Displayed on dashboards** → Metrics appear on configurable, org-shareable dashboards with line charts, area charts, bar charts, single values, gauges, and tables — filterable and groupable by tags+4. **Displayed on dashboards** → Metrics appear on configurable, org-shareable dashboards with line charts, area charts, bar charts, single values, gauges, and tables, filterable and groupable by tags## Ingestion pipelinesdiff --git a/docs/pages/learn/embedded-mode.mdx b/docs/pages/learn/embedded-mode.mdxindex 7a452986..cf45c725 100644--- a/docs/pages/learn/embedded-mode.mdx+++ b/docs/pages/learn/embedded-mode.mdx@@ -2,7 +2,7 @@import { Callout, Steps } from "nextra/components";-Embedded mode runs a full Traceway server inside your Go process — using **SQLite** for all storage (relational data and telemetry). No Docker, no infrastructure — just one `tracewaybackend.Run()` call and your app has local observability.+Embedded mode runs a full Traceway server inside your Go process, using **SQLite** for all storage (relational data and telemetry). No Docker, no infrastructure: just one `tracewaybackend.Run()` call and your app has local observability.This is intended for **local development only**. For production, use the [Docker Compose](/server/docker-compose) or [minimal](/server/minimal) deployment.@@ -10,8 +10,8 @@ Telemetry storage follows your build tags: a plain `go build` embeds SQLite, and<Callout type="info">**Want to see it in action?** Check out the [full working example on- GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel)- — a complete Go app with embedded Traceway and OpenTelemetry instrumentation+ GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel):+ a complete Go app with embedded Traceway and OpenTelemetry instrumentationyou can run in one command.</Callout>@@ -25,6 +25,30 @@ Telemetry storage follows your build tags: a plain `go build` embeds SQLite, andgo get github.com/tracewayapp/traceway/backend```+<Callout type="warning">+ **On backend v1.9.15 and earlier, that command fails.** Those releases declare+ their dependency on the Traceway CLI module with a placeholder version, so+ `go get` stops with:++ ```+ github.com/tracewayapp/traceway/[email protected]: invalid version: unknown revision 000000000000+ ```++ Point that dependency at a real published CLI version in your own `go.mod`:++ ```+ replace github.com/tracewayapp/traceway/cli => github.com/tracewayapp/traceway/cli v1.9.15+ ```++ Then run `go get github.com/tracewayapp/traceway/backend` again. The failed+ attempt leaves your `go.mod` untouched, so there is nothing to clean up first.+ The CLI and the backend are released together, so use the CLI version that+ matches the backend version you want. A `replace` only applies to your own+ module, so it does not affect anything that imports your code.++ Later releases need none of this. Upgrade instead if you can.+</Callout>+### Start Traceway inside your appCall `tracewaybackend.Run()` in a goroutine during startup. It boots an in-memory SQLite server and seeds a default user and project:@@ -68,7 +92,6 @@ package mainimport ("context"- "fmt""time""github.com/gin-gonic/gin"@@ -76,6 +99,7 @@ import ("go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin""go.opentelemetry.io/otel""go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"+ "go.opentelemetry.io/otel/propagation""go.opentelemetry.io/otel/sdk/resource"sdktrace "go.opentelemetry.io/otel/sdk/trace"semconv "go.opentelemetry.io/otel/semconv/v1.26.0"@@ -103,13 +127,16 @@ func main() {}func initTracer() func() {- exporter, _ := otlptracehttp.New(+ exporter, err := otlptracehttp.New(context.Background(),otlptracehttp.WithEndpointURL("http://localhost:8082/api/otel/v1/traces"),otlptracehttp.WithHeaders(map[string]string{"Authorization": "Bearer " + tracewayToken,}),)+ if err != nil {+ panic(err)+ }tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter),@@ -120,6 +147,14 @@ func initTracer() func() {)otel.SetTracerProvider(tp)+ // Go's global propagator is a no-op by default. Without this line the+ // traceparent header is never sent or read, so an outgoing call starts a+ // brand new trace instead of joining the request it came from.+ otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(+ propagation.TraceContext{},+ propagation.Baggage{},+ ))+return func() {ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)defer cancel()@@ -129,8 +164,10 @@ func initTracer() func() {```The key parts:-- **Exporter URL** points to `http://localhost:8082/api/otel/v1/traces` — the embedded server's OTLP endpoint+- **Exporter URL** points to `http://localhost:8082/api/otel/v1/traces`, the embedded server's OTLP endpoint- **Authorization header** uses the same token from `WithDefaultProject`+- **`otel.SetTextMapPropagator`** is required in Go. The global propagator ships as a no-op, so without this call an HTTP call your app makes to another service shows up as its own separate trace instead of a child span+- **`defer shutdown()`** flushes the batch processor on exit. Without it the last few seconds of spans never leave the process### Open the dashboard@@ -138,7 +175,7 @@ Run your app, then open **[http://localhost:8082](http://localhost:8082)** in yoLog in with the credentials from `WithDefaultUser` (`[email protected]` / `admin`).-Hit your app at `http://localhost:8080/ping` a few times — traces will appear in the Traceway dashboard within a few seconds.+Hit your app at `http://localhost:8080/ping` a few times. Traces will appear in the Traceway dashboard within a few seconds.</Steps>@@ -146,17 +183,31 @@ Hit your app at `http://localhost:8080/ping` a few times — traces will appearAll options are passed to `tracewaybackend.Run()`. When any options are provided, embedded mode is activated automatically (pure SQLite).-| Option | Description | Default |-| -------------------------------------------- | -------------------------------------------------------------------------- | ------------------------- |-| `WithPort(port)` | HTTP port for the Traceway server | `8082` |-| `WithServerURL(url)` | Base URL override (used in connection string output) | `http://localhost:{port}` |-| `WithSQLitePath(path)` | SQLite database file path | `:memory:` (in-memory) |-| `WithDefaultUser(email, password)` | Create a default admin user on startup | None |-| `WithDefaultProject(name, framework, token)` | Create a default project with a fixed token (can be called multiple times) | None |+| Option | Description | Default |+| --------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------- |+| `WithPort(port)` | HTTP port for the Traceway server | `8082` |+| `WithServerURL(url)` | Base URL override (used in connection string output) | `http://localhost:{port}` |+| `WithSQLitePath(path)` | SQLite database file path | `:memory:` (in-memory) |+| `WithDefaultUser(email, password)` | Create a default admin user on startup | None |+| `WithDefaultProject(name, framework, token)` | Create a default project with a fixed token (can be called multiple times) | None |+| `DisableLogging()` | Silence the embedded server's startup and request logs so they don't mix with your app's output | Logging enabled |+| `WithDefaultProjectSourceMapToken(name, token)` | Set a fixed source map upload token on a default project. Call it after the matching `WithDefaultProject`, since it looks the project up by name | None |+| `WithMonitoringURL(connectionString)` | Report the embedded server's own telemetry to another Traceway instance. Takes a connection string in the form `<project_token>@<url>/api/report` | None |++If the embedded server's startup lines get in the way of your own output, add `tracewaybackend.DisableLogging()` to the option list:++```go+go tracewaybackend.Run(+ tracewaybackend.WithPort(8082),+ tracewaybackend.DisableLogging(),+ tracewaybackend.WithDefaultUser("[email protected]", "admin"),+ tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),+)+```## Multiple Projects-`WithDefaultProject` can be called multiple times to create separate projects — one for your Go backend, another for a frontend or any other service. Each project gets its own token, keeping telemetry organized in the dashboard:+`WithDefaultProject` can be called multiple times to create separate projects: one for your Go backend, another for a frontend or any other service. Each project gets its own token, keeping telemetry organized in the dashboard:```gogo tracewaybackend.Run(@@ -169,19 +220,39 @@ go tracewaybackend.Run(## Data Persistence-By default, the database is **in-memory** — all data is lost when your application restarts. This is fine for quick debugging sessions, but if you want data to survive restarts, set a file path:+By default, the database is **in-memory**, which means all data is lost when your application restarts. This is fine for quick debugging sessions, but if you want data to survive restarts, set a file path:++Add `WithSQLitePath` to the options you already pass. It does not replace them. Keep the `go` keyword too, otherwise `Run` blocks and your own server never starts.```go-tracewaybackend.Run(+go tracewaybackend.Run(+ tracewaybackend.WithPort(8082),tracewaybackend.WithSQLitePath("./traceway.db"),tracewaybackend.WithDefaultUser("[email protected]", "admin"),tracewaybackend.WithDefaultProject("My App", "opentelemetry", "dev-token"),)```-`WithSQLitePath` stores everything — users, projects, traces, issues, and metrics — in a single SQLite file.+### What ends up on disk++`WithSQLitePath("./traceway.db")` creates **two** databases side by side, not one:++| File | Contents |+| --- | --- |+| `traceway.db` | Relational data: users, organizations, projects, settings |+| `traceway_telemetry.db` | Telemetry: traces, endpoints, issues, logs, metrics |-Seeding is idempotent — if the user from `WithDefaultUser` already exists, it's skipped.+SQLite also writes `-wal` and `-shm` sidecar files for each database. Back up or ignore all of them, not just the file you named. The server also creates a `storage/` directory in the process working directory for session recordings and synthetics screenshots. That directory appears in the default in-memory mode too.++Add all of it to `.gitignore`:++```text+traceway.db*+traceway_telemetry.db*+storage/+```++Seeding is idempotent: if the user from `WithDefaultUser` already exists, it's skipped.## Development-Only Guard@@ -200,3 +271,11 @@ if os.Getenv("ENV") != "production" {## Full ExampleFor a complete working example with error recording and child spans, see the [embedded-backend-otel example on GitHub](https://github.com/tracewayapp/traceway/tree/main/examples/embedded-backend-otel).++## Next Steps++The Quick Start above only sends request traces. Everything else works the same against the embedded server as against a hosted one, so point these guides at `http://localhost:8082` and use your `WithDefaultProject` token:++- [Traces](/client/otel/traces): child spans, recording errors so they land in Issues with a stack trace, and background jobs. A background job span needs `SpanKind.CONSUMER`, a plain root span is silently dropped.+- [Logs](/client/otel/logs): emitting logs that attach to the request's trace.+- [Metrics](/client/otel/metrics): counters, histograms, and how Traceway stores them.diff --git a/docs/pages/learn/healthchecks.mdx b/docs/pages/learn/healthchecks.mdxindex f015561b..2e66478a 100644--- a/docs/pages/learn/healthchecks.mdx+++ b/docs/pages/learn/healthchecks.mdx@@ -41,7 +41,7 @@ If your healthcheck lives somewhere else, add custom paths in the project settin## Configuration-The feature is **enabled by default** for every project. To change it, open the project switcher, edit the project, and use the **Drop healthy healthcheck requests** toggle. The same form accepts custom paths (one per line) and shows the full built-in list.+The feature is **enabled by default** for every project you create in the dashboard or through the API. Projects seeded by embedded mode's `WithDefaultProject` start with it off, so turn it on there if you want probe traffic filtered. To change it, open the project switcher, edit the project, and use the **Drop healthy healthcheck requests** toggle. The same form accepts custom paths (one per line) and shows the full built-in list.Turning the toggle off stores all healthcheck traffic again. The setting takes effect immediately, no restart required.diff --git a/docs/pages/learn/index.mdx b/docs/pages/learn/index.mdxindex 1691381d..eed6cef1 100644--- a/docs/pages/learn/index.mdx+++ b/docs/pages/learn/index.mdx@@ -1,10 +1,10 @@# Architecture Overview-Traceway's mental model is small. There's one underlying primitive — **spans** — and a handful of **materialized views** over those spans that make the dashboard fast.+Traceway's mental model is small. There's one underlying primitive, **spans**, and a handful of **materialized views** over those spans that make the dashboard fast.## The primitive: spans-Every piece of trace data Traceway ingests is an OpenTelemetry span. Spans carry a name, a kind, a duration, attributes, events (exceptions live here), and parent/trace ids. The shape is OTel-native — Traceway speaks OTLP directly and doesn't impose its own trace model on top.+Every piece of trace data Traceway ingests is an OpenTelemetry span. Spans carry a name, a kind, a duration, attributes, events (exceptions live here), and parent/trace ids. The shape is OTel-native: Traceway speaks OTLP directly and doesn't impose its own trace model on top.[Learn more about spans →](/learn/spans)@@ -12,31 +12,31 @@ Every piece of trace data Traceway ingests is an OpenTelemetry span. Spans carryOn ingestion, Traceway classifies each span and writes a row into a dedicated table when the kind or attributes match a rule. These tables are the top-level dashboard concepts:-- **Endpoints** — inbound HTTP entry points (`GET /api/users/:id`).-- **Tasks** — queue consumers, scheduled commands, anything that runs as a "thing that completes".-- **AI Traces** — any span carrying `gen_ai.*` attributes, root or child.-- **Issues** — exception events lifted off whichever entity owned the span.+- **Endpoints**: inbound HTTP entry points (`GET /api/users/:id`).+- **Tasks**: queue consumers, scheduled commands, anything that runs as a "thing that completes".+- **AI Traces**: any span carrying `gen_ai.*` attributes, root or child.+- **Issues**: exception events lifted off whichever entity owned the span.-Each table is indexed and shaped for one job: answering "what's slow?", "what's expensive?", "what's failing?" without scanning the full span store. They're projections, not parallel universes — the underlying span data is the source of truth.+Each table is indexed and shaped for one job: answering "what's slow?", "what's expensive?", "what's failing?" without scanning the full span store. They're projections, not parallel universes: the underlying span data is the source of truth.[Trace classification rules and the views →](/learn/traces) · [Issues →](/learn/issues) · [AI Tracing →](/learn/ai-tracing)## Root and non-root entities-Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root** when the span that produced it was triggered by another trace — a queue worker started by an HTTP request, an LLM call made inside that same handler, a downstream service hop. Non-root rows link back to the originating trace via the auto-derived `distributed_trace_id`, so the distributed-trace view shows the full picture across all the entities that share one OTel trace id.+Endpoints, Tasks, and AI Traces all carry an `is_root` flag. A row is **non-root** when the span that produced it was triggered by another trace: a queue worker started by an HTTP request, an LLM call made inside that same handler, a downstream service hop. Non-root rows link back to the originating trace via the auto-derived `distributed_trace_id`, so the distributed-trace view shows the full picture across all the entities that share one OTel trace id.[More on root vs non-root →](/learn/traces#root-and-non-root-entities)## Enrichments-- **Attributes** — key-value metadata on spans and the entities derived from them. Flow from global → per-request scope. [Details →](/learn/attributes)-- **Sessions** — user-session grouping for browser/mobile clients, separate from traces. [Details →](/learn/sessions)-- **Metrics** — time-series measurements, ingested via OTLP metrics. Independent of the trace pipeline. [Details →](/learn/metrics)-- **Logs** — OTLP logs ingestion, joinable to traces by trace id. [Details →](/learn/logs)+- **Attributes**: key-value metadata on spans and the entities derived from them. Flow from global → per-request scope. [Details →](/learn/attributes)+- **Sessions**: user-session grouping for browser/mobile clients, separate from traces. [Details →](/learn/sessions)+- **Metrics**: time-series measurements, ingested via OTLP metrics. Independent of the trace pipeline. [Details →](/learn/metrics)+- **Logs**: OTLP logs ingestion, joinable to traces by trace id. [Details →](/learn/logs)<img src="/architecture.png" alt="Traceway Architecture" style={{ maxWidth: '100%', height: 'auto' }} />-See [Data Flow](/learn/data-flow) for how these connect during request processing, task execution, and metrics collection.+See [Data Flow](/learn/data-flow) for how these connect during request processing, task execution, and metrics collection, and [Project Structure](/learn/projects) for how to split an application across Traceway projects before you instrument it.## Authenticationdiff --git a/docs/pages/learn/issues.mdx b/docs/pages/learn/issues.mdxindex c9f46cab..a6794f91 100644--- a/docs/pages/learn/issues.mdx+++ b/docs/pages/learn/issues.mdx@@ -1,6 +1,6 @@# Issues-Issues capture problems in your application — exceptions, panics, and explicitly sent messages.+Issues capture problems in your application: exceptions, panics, and explicitly sent messages.<img src="/issues.png" alt="Issues Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />@@ -16,8 +16,8 @@ An issue is created when:Traceway captures two kinds of issues:-- **Errors** — Runtime exceptions with stack traces (panics, captured errors)-- **Messages** — Explicitly captured strings sent through the client SDK+- **Errors**: Runtime exceptions with stack traces (panics, captured errors)+- **Messages**: Explicitly captured strings sent through the client SDKBoth appear in the Issues list with full context and attributes.@@ -52,19 +52,19 @@ Before hashing, Traceway normalizes the stack trace so values that vary between| Normalization | Effect ||---------------|--------|-| Error message | Only the error type is kept (`TimeoutError`, `*net.OpError`). The message text, which often embeds user data, is dropped, including the messages on JVM `Caused by:` lines. |+| Error message | Only the error type is kept (`TimeoutError`, `*net.OpError`). The message text, which often embeds user data, is dropped, including the messages on JVM `Caused by:` lines. The type has to be letters, digits, dots and underscores, so a PHP class written with backslashes (`App\Exception\Foo`) is not recognised and keeps its message in the hash. || Runtime values | Hex addresses, UUIDs, IP addresses and ports, email addresses, goroutine numbers, and long numbers (epoch timestamps, numeric IDs) are replaced with placeholders. || File locations | Absolute paths collapse to `filename:line`, URL origins are stripped from browser frames, and `@v1.2.3` module version suffixes are removed so a dependency bump does not split a group. || JVM frames | Line numbers inside `(File.java:123)` are dropped, and `... 12 more` becomes `... more`. |-| Column numbers | Dropped everywhere except line 1, where they are kept: minified bundles put everything on line 1, and the column is the only thing distinguishing frames there. |-| Resolved function names | Function names produced by the [symbolicator](/symbolicator/javascript) are excluded from the hash, so uploading source maps later improves readability without reshuffling existing groups. |+| Column numbers | Dropped from frames that end in `:line:col`, unless the line number is 1. Minified bundles put everything on line 1, and the column is the only thing distinguishing frames there. Frames that carry the position inside brackets, such as Dart `(main.dart:20:3)`, keep the column. |+| Resolved function names | JavaScript function-name lines are collapsed to `<fn>`, so better JS name resolution alone does not re-bucket an issue. Function and method names in other formats, such as Dart, iOS and JVM frames, are part of the hash. Uploading debug info for the first time also rewrites each frame's file, line and column, so occurrences from before the upload keep their old hash and later ones form a new issue. || Whitespace | Runs of spaces and blank lines are collapsed. |The normalized trace is hashed with SHA-256 and truncated to 16 characters.### What Stays Together, What Splits-The same logical error keeps the same hash across servers, environments, app versions, and users. A new hash (and therefore a new issue and a New Issue alert) appears when the failure itself changes: a different error type, or a different code path with different frames. Line numbers are part of the hash, so a refactor that moves code within a file can legitimately re-group an error after a deploy.+The same logical error keeps the same hash across servers, environments, app versions, and users. A new hash (and therefore a new issue and a New Issue alert) appears when the failure itself changes: a different error type, or a different code path with different frames. Line numbers are part of the hash for most languages, so a refactor that moves code within a file can legitimately re-group an error after a deploy. JVM frames are the exception. Their line numbers are removed, so moving Java, Kotlin or Scala code inside a file keeps the same issue.### Messages@@ -81,13 +81,13 @@ traceway.CaptureMessage(fmt.Sprintf("payment provider unreachable for order %d",## Issue Lifecycle-1. **Active** — When an issue first occurs-2. **Archived** — Marked as resolved by you (hidden from the default view)-3. **Reopened** — An archived issue that occurred again (regression detected)+1. **Active**: When an issue first occurs+2. **Archived**: Marked as resolved by you (hidden from the default view)+3. **Reopened**: An archived issue that occurred again (regression detected)## Attributes on Issues-Issues carry their own attributes — the key-value metadata that was in scope when the exception occurred. If the issue is linked to a trace, it inherits the trace's attributes.+Issues carry their own attributes: the key-value metadata that was in scope when the exception occurred. If the issue is linked to a trace, it inherits the trace's attributes.When error recording flags are enabled in middleware, additional context can be attached (configurable per client library):diff --git a/docs/pages/learn/logs.mdx b/docs/pages/learn/logs.mdxindex e0eddf12..9e096241 100644--- a/docs/pages/learn/logs.mdx+++ b/docs/pages/learn/logs.mdx@@ -44,9 +44,9 @@ The severity dropdown on the Logs page filters by threshold. Picking `WARN+` keeOTel logs carry three layers of attributes, all of which are indexed and searchable in Traceway:-- **Resource attributes** describe the emitting process — e.g. `service.name`, `service.version`, `host.name`, `deployment.environment`. Every log emitted by a given service carries the same resource attributes.-- **Scope attributes** describe the instrumentation library or logger emitting the record — e.g. `logger.name`, scope `version`. Useful for separating framework logs from application logs.-- **Log attributes** are specific to a single log record — e.g. `user.id`, `order.id`, `request.method`. These are the per-event details you add when logging.+- **Resource attributes** describe the emitting process, e.g. `service.name`, `service.version`, `host.name`, `deployment.environment`. Every log emitted by a given service carries the same resource attributes.+- **Scope attributes** describe the instrumentation library or logger emitting the record, e.g. `logger.name`, scope `version`. Useful for separating framework logs from application logs.+- **Log attributes** are specific to a single log record, e.g. `user.id`, `order.id`, `request.method`. These are the per-event details you add when logging.Click any log row to expand it. The expanded view shows the raw body, the linked trace and span IDs, and every attribute from all three layers. Each attribute has a filter toggle, so you can pivot from a single log line to all logs sharing that value in one click.@@ -72,12 +72,12 @@ A few details:The **Logs** page supports filtering and search across all three attribute layers plus the body:-- **Severity threshold** — keep only records at or above a minimum level (e.g. `WARN+` hides INFO and below).-- **Service** — exact match on `service.name`.-- **Trace** — exact match on the hex trace ID, or on a Traceway distributed trace ID (returns logs from every linked trace).-- **Body search** — full-text search over the log message. Single-token queries use a token-bloom index for speed; multi-word or punctuation-heavy queries fall back to substring matching.-- **Attribute filters** — match on any `resource.*`, `scope.*`, or log attribute key, either as equals or not equals.-- **Time range** — pick from presets (24h, 7d, 30d) or a custom window.+- **Severity threshold**: keep only records at or above a minimum level (e.g. `WARN+` hides INFO and below).+- **Service**: exact match on `service.name`.+- **Trace**: exact match on the hex trace ID, or on a Traceway distributed trace ID (returns logs from every linked trace).+- **Body search**: full-text search over the log message. Single-token queries use a token-bloom index for speed; multi-word or punctuation-heavy queries fall back to substring matching.+- **Attribute filters**: match on any `resource.*`, `scope.*`, or log attribute key, either as equals or not equals.+- **Time range**: pick from presets (24h, 7d, 30d) or a custom window.All filters combine. Results load newest first, and a "Load more" button at the end of the list fetches older records in the same query.@@ -113,7 +113,7 @@ When your SDK emits a log inside an active OTel span context, the log record aut- When a trace crosses service boundaries, a **Distributed Traces** tab shows logs from every service that participated.- Jumping from a log to the trace that produced it is a single click in the dashboard.-No extra configuration is required — the standard OTel context propagation carries the IDs to your logger.+No extra configuration is required. The standard OTel context propagation carries the IDs to your logger.## Retention@@ -123,6 +123,6 @@ Logs are stored in ClickHouse with daily partitioning and a 90-day TTL on the tiSee the SDK-specific guides for setup instructions:-- [OpenTelemetry](/client/otel/logs) — direct SDK export or OTel Collector-- [Symfony](/client/symfony/logs) — Monolog handler and the OTel PHP Logs API-- [Cloudflare Workers](/client/cloudflare) — Worker-side OTel logs exporter+- [OpenTelemetry](/client/otel/logs): direct SDK export or OTel Collector+- [Symfony](/client/otel/symfony/logs): Monolog handler and the OTel PHP Logs API+- [Cloudflare Workers](/client/otel/cloudflare): Worker-side OTel logs exporterdiff --git a/docs/pages/learn/metrics.mdx b/docs/pages/learn/metrics.mdxindex a0d5c039..c6882edf 100644--- a/docs/pages/learn/metrics.mdx+++ b/docs/pages/learn/metrics.mdx@@ -15,11 +15,11 @@ Each metric point contains:## Automatic vs Custom Metrics-**Automatic metrics** are collected without any code changes. The Traceway Go and Node.js SDKs capture CPU usage, memory consumption, goroutine count (Go), and GC statistics at a regular interval (default: 30 seconds). For OpenTelemetry, the available automatic metrics depend on your SDK and instrumentation libraries — for example, the Go OTel runtime instrumentation provides goroutine and memory metrics, while the Node.js OTel runtime instrumentation provides event loop and heap metrics.+**Automatic metrics** are collected without any code changes. With OpenTelemetry the set depends on your SDK and instrumentation libraries: the Go OTel runtime instrumentation provides goroutine and memory metrics, the Node.js OTel runtime instrumentation provides event loop and heap metrics. For host-level CPU, memory, disk and network, run the [Traceway OTel Agent](/learn/otel-agent).-**Custom metrics** are captured in your application code using `CaptureMetric` or `CaptureMetricWithTags`. Use these for business metrics (orders per minute), resource metrics (queue depth), or performance metrics (external API latency).+**Custom metrics** are recorded in your application code with an OTel meter (`createCounter`, `createHistogram`, `createObservableGauge`). Use these for business metrics (orders per minute), resource metrics (queue depth), or performance metrics (external API latency).-See the SDK-specific pages for API details: [Go SDK](/client/sdk/metrics) | [Node.js SDK](/client/node-sdk/metrics)+See [OTel Metrics](/client/otel/metrics) for the mapping and per-language examples, and the [Node.js metrics guide](/client/otel/nodejs/metrics) for a full worked setup.## Tags@@ -34,7 +34,7 @@ Common tags include:| `queue_name` | `emails`, `payments` | Per-queue monitoring || `environment` | `production`, `staging` | Environment comparison |-Tags are captured via `CaptureMetricWithTags` in the Go SDK, or as resource attributes in OpenTelemetry.+Tags come from the data-point attributes you pass when recording a value, plus the small allowlist of resource attributes Traceway lifts. See [Resource Attributes](/client/otel/metrics#resource-attributes).## Visualizing Metrics@@ -60,9 +60,9 @@ This keeps queries fast regardless of the time window.## Creating Your First Dashboard1. **Open the Dashboards page** and press <kbd>⌘K</kbd> (or click **Add or find dashboards...**) to install a template, pick up a dashboard from another project, or create a blank one-2. **Add a widget** — Click **Add Metric Widget**, pick a metric from the dropdown, choose an aggregation, and optionally add tag filters or a group-by-3. **Adjust the time range** — Use the preset buttons or select a custom range to see the data window you need-4. **Interact** — Hover over data points for exact values, drag-select a region to zoom in, and use the widget menu to move, edit, or delete widgets+2. **Add a widget**: Click **Add Metric Widget**, pick a metric from the dropdown, choose an aggregation, and optionally add tag filters or a group-by+3. **Adjust the time range**: Use the preset buttons or select a custom range to see the data window you need+4. **Interact**: Hover over data points for exact values, drag-select a region to zoom in, and use the widget menu to move, edit, or delete widgetsSee [Dashboards](/learn/dashboards) for applying dashboards across projects, editing them as JSON, and importing from Grafana.diff --git a/docs/pages/learn/otel-agent.mdx b/docs/pages/learn/otel-agent.mdxindex a82f322b..e6b33fa6 100644--- a/docs/pages/learn/otel-agent.mdx+++ b/docs/pages/learn/otel-agent.mdx@@ -2,7 +2,7 @@import { Callout, Steps } from "nextra/components";-The **Traceway OTel Agent** is a tiny, pre-configured OpenTelemetry Collector that pulls host metrics (CPU, memory, disk, filesystem, network) from a server every 60 seconds and ships them to your Traceway project over OTLP/HTTP. Runs as a background service via `systemd`, `launchd`, or the Windows Service Manager — nothing to write, nothing to tune.+The **Traceway OTel Agent** is a tiny, pre-configured OpenTelemetry Collector that pulls host metrics (CPU, memory, disk, filesystem, network) from a server every 60 seconds and ships them to your Traceway project over OTLP/HTTP. Runs as a background service via `systemd`, `launchd`, or the Windows Service Manager. There is nothing to write, nothing to tune.<Callout type="info">Full source, advanced config, and the contributor guide live at@@ -19,7 +19,7 @@ Open your Traceway dashboard → the project you want metrics to land in → **S### Run the installer on your server-**Linux / macOS** (requires `curl`, `tar`, `sudo` or root):+**Linux / macOS** (requires `curl`, `tar`, `awk`, `mktemp`, either `sha256sum` or `shasum`, and `sudo` or root):```bashcurl -fsSL https://install.tracewayapp.com/install.sh | TRACEWAY_TOKEN=<your-token> bash@@ -34,20 +34,33 @@ iwr -useb https://install.tracewayapp.com/install.ps1 | iex</Steps>-That's it. The installer downloads the agent, verifies its checksum, writes a systemd unit / launchd plist / Windows service, and starts it. Metrics begin arriving in your dashboard within about 60 seconds — the first scrape interval.+That's it. The installer downloads the agent, verifies its checksum, writes a systemd unit / launchd plist / Windows service, and starts it. Metrics begin arriving in your dashboard within about 60 seconds, the first scrape interval.## What gets capturedEvery 60 seconds the agent scrapes:-- **CPU** — utilization (% per core per state), load averages (1m / 5m / 15m).-- **Memory** — usage in bytes, utilization %.-- **Disk** — throughput bytes, IOPS, I/O time per device.-- **Filesystem** — usage bytes and utilization % per mount.-- **Network** — throughput bytes, packets, errors, and open connections per interface.-- **Process** *(opt-in)* — set `TRACEWAY_PROCESS_NAMES` to a list of process names (or `*` for all) to enable per-process CPU time, RSS, and virtual memory.+- **CPU**: utilization (% per core per state), load averages (1m / 5m / 15m).+- **Memory**: usage in bytes, utilization %.+- **Disk**: throughput bytes, IOPS, I/O time per device.+- **Filesystem**: usage bytes and utilization % per mount.+- **Network**: throughput bytes, packets, errors, and open connections per interface.+- **Process** *(opt-in)*: set `TRACEWAY_PROCESS_NAMES` to a list of process names (or `*` for all) to enable per-process CPU time, RSS, and virtual memory.-Every signal is tagged with resource attributes so you can filter in the dashboard — `service.name`, `host.name`, `host.id`, `host.arch`, `os.type`, and on cloud VMs, `cloud.provider`, `cloud.region`, and `cloud.account.id` (EC2 / GCE / Azure are auto-detected).+### How hosts are identified++Metrics are tagged with `server_name`, which comes from `TRACEWAY_SERVICE_NAME` (default: the machine's hostname). Give every host a distinct `TRACEWAY_SERVICE_NAME` or the hosts cannot be told apart. You can also group and filter by the attributes each scraper puts on its own data points, for example `cpu` and `state` on CPU metrics, `device` and `direction` on disk and network metrics, and `device` and `mountpoint` on filesystem metrics.++The agent also attaches resource attributes to the OTLP payload: `service.name`, `host.name`, `os.type`, and on cloud VMs `cloud.provider`, `cloud.region`, and `cloud.account.id` (EC2 / GCE / Azure are auto-detected). Traceway lifts `service.name` onto metric tags, renamed to `server_name`. Every other resource attribute is dropped unless it is on a fixed allowlist. For this agent the allowlist matters only when `TRACEWAY_PROCESS_NAMES` is set: `process.pid`, `process.executable.name`, `process.command_line` and `process.owner` are lifted, which is what keeps the per-process series apart. `host.name`, `os.type` and the `cloud.*` attributes are never stored as tags. See [Resource Attributes](/client/otel/metrics#resource-attributes) for the full allowlist.++## What it opens++The agent binds two loopback-only ports:++- `127.0.0.1:13133` for the health check. The installer polls it to confirm the agent started, and `curl -fsS http://127.0.0.1:13133/` is a quick way to check it is still running.+- `localhost:8888` for the Collector's own internal metrics.++Neither port is reachable from off the host.## Options@@ -56,11 +69,13 @@ Set these env vars alongside `TRACEWAY_TOKEN` before piping into the shell:| Var | Purpose || ------------------------- | ------------------------------------------------------------------------------------------------------ || `TRACEWAY_SERVICE_NAME` | `service.name` label for this host. Default: the machine's hostname. |-| `TRACEWAY_ENDPOINT` | OTLP/HTTP base URL. Override when running against self-hosted Traceway. |+| `TRACEWAY_ENDPOINT` | OTLP/HTTP base URL. Default: `https://cloud.tracewayapp.com/api/otel`. Override when running against self-hosted Traceway. || `TRACEWAY_LOG_PATHS` | Comma-separated file globs to tail as logs (e.g. `/var/log/app/*.log`). Leave unset to ship metrics only. |-| `TRACEWAY_PROCESS_NAMES` | Comma-separated process names (e.g. `myapp,postgres`) or `*` for all. Off by default — enables per-process CPU / memory metrics. |+| `TRACEWAY_PROCESS_NAMES` | Comma-separated process names (e.g. `myapp,postgres`) or `*` for all. Off by default; enables per-process CPU / memory metrics. |+| `TRACEWAY_VERSION` | Pin the agent version, e.g. `v0.5.0`. Default: the version the installer was published with. Works on both installers. |+| `TRACEWAY_RELEASES_URL` | Override the release-archive base URL. Default: GitHub Releases. Accepts `file://` URLs for air-gapped installs. Linux / macOS installer only. |-Example — named host, app logs, self-hosted Traceway:+Example (named host, app logs, self-hosted Traceway):```bashcurl -fsSL https://install.tracewayapp.com/install.sh | \@@ -73,6 +88,8 @@ curl -fsSL https://install.tracewayapp.com/install.sh | \Re-running the installer with the same env vars upgrades in place.+Note that it also overwrites `config.yaml` with the release default. If you edit that file by hand, pin `TRACEWAY_VERSION` first so an upgrade cannot swap the config under you unnoticed.+## Uninstall**Linux / macOS:**@@ -92,4 +109,4 @@ Stops and removes the service, binary, and config directory. Your Traceway proje----For deeper configuration — tuning the scrape interval, writing filelog operators, air-gapped and manual installs, signed releases, or contributing support for additional auth modes — see the agent repo: [github.com/tracewayapp/traceway-otel-agent](https://github.com/tracewayapp/traceway-otel-agent).+For deeper configuration (tuning the scrape interval, writing filelog operators, air-gapped and manual installs, signed releases, or contributing support for additional auth modes), see the agent repo: [github.com/tracewayapp/traceway-otel-agent](https://github.com/tracewayapp/traceway-otel-agent).diff --git a/docs/pages/learn/projects.mdx b/docs/pages/learn/projects.mdxnew file mode 100644index 00000000..b82b91e7--- /dev/null+++ b/docs/pages/learn/projects.mdx@@ -0,0 +1,141 @@+# Project Structure++A Traceway project is an observability boundary. It controls which telemetry shares dashboards and distributed traces, which framework-specific setup appears on the **Connection** page, and which runtime and artifact-upload credentials are used.++Choose projects around application runtime boundaries, not folders or processes.++## Recommended Structure++| Application part | Traceway project | Framework | What it records |+|---|---|---|---|+| Backend system | One backend project | **OpenTelemetry** | Endpoints, spans, issues, background tasks, AI traces, logs, application metrics, and host metrics |+| Browser application | Separate project per deployed browser app | React, Svelte, Vue.js, or jQuery | Browser errors, web vitals, session replay, distributed-trace linkage, and source-mapped stacks |+| Mobile application | Separate project per independently released app | Flutter, React Native, Android, or iOS | Mobile errors and crashes, replay where supported, and symbolicated stacks |+| Full-stack JavaScript application | Backend and browser projects | OpenTelemetry + browser framework | Server telemetry stays in the backend project; browser telemetry stays in the browser project |++Do not send browser or mobile telemetry to the backend project. Those runtimes use different SDKs, build artifacts, and dashboard behavior.++The framework picker offers exactly nine options, and every backend takes the same one:++| Group | Options |+|---|---|+| Backend | OpenTelemetry |+| Browser | React, Svelte, Vue.js, jQuery |+| Mobile | Flutter, React Native, Android, iOS |++There is no Gin, Django, Laravel, or Hono entry, because the language and web framework are chosen later on the project's **Connection** page, which then shows the exact install and exporter setup. A meta-framework picks the framework it renders with: Next.js and Remix select **React**, SvelteKit selects **Svelte**, Nuxt selects **Vue.js**.++## Keep Backend Signals Together++The main backend project should receive all server-side signals:++- HTTP endpoints and their child spans+- Queue consumers, scheduled work, and CLI tasks+- AI and LLM spans+- Exceptions and OTel logs+- Application and runtime metrics+- Host CPU, memory, disk, filesystem, network, and process metrics from the [Traceway OTel Agent](/learn/otel-agent)++APIs, workers, schedulers, and the host agent use the same backend project token. Give each process or host a stable `service.name` so it remains filterable inside the project:++```text+checkout-api+checkout-worker+checkout-scheduler+checkout-prod-host-1+```++Create separate backend projects only when the services are separate products or require different ownership, access control, compliance, or data isolation. A different deployment process alone is not a reason to split them.++## Browser and Mobile Boundaries++Create one project for each independently deployed browser application. Its framework selection drives the browser SDK setup on the Connection page, and its dedicated upload token keeps source maps scoped to that application.++Create one project for each independently released mobile application. A Flutter product that ships to both Android and iOS normally uses one Flutter project. Separate native Android and iOS applications use separate projects and credentials.++A full-stack framework such as Next.js, SvelteKit, or Remix needs two projects only when its server runs meaningful production code:++- OpenTelemetry project for API routes, server rendering, workers, tasks, AI calls, and server metrics+- Browser-framework project for code running in the user's browser++For Next.js, the server half is covered by the [Next.js OpenTelemetry guide](/client/otel/nextjs) and the browser half by the [React SDK](/client/react). Each half uses its own project token.++A static export with an external API needs only the browser project; the external API belongs to its backend project.++## Create the Projects++For each application boundary:++1. Open your Traceway dashboard.+2. Open the project selector in the header and select **Add Project**.+3. Choose the organization.+4. Enter a name that identifies the product and runtime, such as `Acme Backend`, `Acme Web`, or `Acme Mobile`.+5. Select the framework:+ - **OpenTelemetry** for the backend, regardless of language or HTTP framework; it is the only backend option, and the preselected default+ - **React**, **Svelte**, **Vue.js**, or **jQuery** for a browser project+ - **Flutter**, **React Native**, **Android**, or **iOS** for a mobile project+6. Select **New Project**.+7. Save the project token and open **Go to Connection** for the tailored setup. For a backend project, the Connection page asks for your language and web framework there and then shows the matching install commands.++Repeat until every deployed browser app and independently released mobile app has its own project. Do not reuse the backend token for them.++## Credentials++| Credential | Purpose | Where it belongs |+|---|---|---|+| Backend project token | Authenticates OTLP traces, metrics, and logs | Backend deployment secret |+| Browser connection string | Authenticates browser SDK reports | Public build-time browser configuration; never reuse the backend token |+| Mobile connection string | Authenticates mobile SDK reports | Mobile build configuration; never reuse another project's token |+| Upload token | Uploads source maps, Flutter symbols, iOS dSYMs, or Android R8 mappings | CI secret; never embed it in the application |++Generate an upload token from the project's **Connection** page under **Source Maps** or **Symbol Upload**. Upload tokens are project-specific. Regenerating one invalidates the previous value, so update the corresponding CI secret immediately.++Use component-specific environment variables for runtime credentials so a monorepo cannot accidentally cross-wire projects:++```text+TRACEWAY_BACKEND_TOKEN+PUBLIC_TRACEWAY_WEB_CONNECTION_STRING+TRACEWAY_MOBILE_CONNECTION_STRING+```++Browser frameworks require their public prefix: for example `VITE_` in Vite, `PUBLIC_` in SvelteKit, and `NEXT_PUBLIC_` in Next.js.++Upload tokens are the exception: each uploader reads one fixed variable name.++| Uploader | Token variable |+|---|---|+| `traceway-sourcemaps` (JavaScript source maps) | `TRACEWAY_SOURCEMAP_TOKEN` |+| `dart run traceway:upload_symbols` (Flutter) | `TRACEWAY_UPLOAD_TOKEN` |+| iOS dSYM upload | `TRACEWAY_UPLOAD_TOKEN` |+| `com.tracewayapp.symbols` Gradle plugin (Android) | `TRACEWAY_UPLOAD_TOKEN` |++All of them also read `TRACEWAY_URL`. Name the CI secret exactly what the uploader reads, or keep a per-component name and pass it explicitly:++```bash+traceway-sourcemaps --url "$TRACEWAY_URL" --token "$TRACEWAY_WEB_UPLOAD_TOKEN" --directory ./dist+```++If one repository releases two mobile apps, their tokens collide on `TRACEWAY_UPLOAD_TOKEN`. Scope the secret per CI job, or pass `--token` per build. Upload tokens never use a public prefix and never ship inside the application.++## Example Monorepo++For a repository containing a Go API, a Svelte dashboard, one shared worker binary, and a Flutter application, create three projects:++| Project | Framework | Reporters |+|---|---|---|+| `Acme Backend` | OpenTelemetry | Go API, worker, scheduler, AI model calls, OTel Agent |+| `Acme Web` | Svelte | Svelte browser SDK and source-map upload job |+| `Acme Mobile` | Flutter | Flutter SDK and symbol-upload job when release builds are obfuscated |++The API and worker use different `service.name` values but the same backend project token. This keeps an endpoint, its queued task, its AI calls, and the affected server metrics in the same project.++## Next Steps++- [OpenTelemetry integration](/client/otel)+- [OTel span mapping](/client/otel/traces)+- [Traceway OTel Agent](/learn/otel-agent)+- [JavaScript source maps](/client/js-sdk/sourcemap-upload)+- [Flutter](/client/flutter)+- [React Native](/client/react-native)+- [Android](/client/android)+- [iOS](/client/ios)diff --git a/docs/pages/learn/sessions.mdx b/docs/pages/learn/sessions.mdxindex 23f41795..441318b8 100644--- a/docs/pages/learn/sessions.mdx+++ b/docs/pages/learn/sessions.mdx@@ -1,18 +1,18 @@# Sessions-A **session** is a full recording of one user's interactions with a browser app — typically the lifetime of an open tab. Where an [Issue](/learn/issues) shows you a single moment that something broke, a session shows you the entire journey: what the user clicked, navigated to, scrolled past, and saw on screen, with the same console logs and network calls layered on top.+A **session** is a full recording of one user's interactions with a browser app, typically the lifetime of an open tab. Where an [Issue](/learn/issues) shows you a single moment that something broke, a session shows you the entire journey: what the user clicked, navigated to, scrolled past, and saw on screen, with the same console logs and network calls layered on top.Sessions are a browser-only concept. They're emitted by the JavaScript SDK and its framework wrappers (React, Vue, Svelte, jQuery) when **always-on session recording** is enabled. Backend and mobile SDKs do not produce sessions.## Sessions vs. Exception Clips-Traceway's browser SDK has always shipped a short rrweb clip alongside every captured exception — the rolling window of events from just before the error fired. That clip stays on the issue detail page; it answers "what did the page look like the moment things went wrong?".+Traceway's browser SDK has always shipped a short rrweb clip alongside every captured exception: the rolling window of events from just before the error fired. That clip stays on the issue detail page; it answers "what did the page look like the moment things went wrong?".A session is the larger envelope around those moments:| | Exception Clip | Session ||---|---|---|-| Default behaviour | On (rolling window flushed with each captured exception) | Off — opt in with `recordAllSessions: true` |+| Default behaviour | On (rolling window flushed with each captured exception) | Off (opt in with `recordAllSessions: true`) || Scope | Last ~10 s before an exception | Entire tab lifetime, capped at 60 min || Storage | One file per exception | One file per ~30 s segment, many per session || When uploaded | At exception capture | Continuously, every segment rotation |@@ -22,18 +22,18 @@ Both can coexist: an exception fired during an always-on session gets its inline## What a Session Records-The recording uses [rrweb](https://github.com/rrweb-io/rrweb) to capture DOM mutations, input events, mouse movement, and viewport changes. Replay is a faithful reconstruction — text, layout, scrolling, hover states, even mouse trails. Privacy markers (`rr-mask`, `rr-block`, `rr-ignore`) work the same way they do for exception clips. See [Session Replay in the JS SDK guide](/client/js-sdk/session-replay#privacy--masking) for the full masking reference.+The recording uses [rrweb](https://github.com/rrweb-io/rrweb) to capture DOM mutations, input events, mouse movement, and viewport changes. Replay is a faithful reconstruction: text, layout, scrolling, hover states, even mouse trails. Privacy markers (`rr-mask`, `rr-block`, `rr-ignore`) work the same way they do for exception clips. See [Session Replay in the JS SDK guide](/client/js-sdk/session-replay#privacy--masking) for the full masking reference.Alongside the rrweb stream, each segment carries:-- **Logs** — every `console.{debug,log,info,warn,error}` call recorded during the segment window.-- **Actions** — `fetch` / `XMLHttpRequest` calls, History API navigations, and any `recordAction(...)` breadcrumbs.+- **Logs**: every `console.{debug,log,info,warn,error}` call recorded during the segment window.+- **Actions**: `fetch` / `XMLHttpRequest` calls, History API navigations, and any `recordAction(...)` breadcrumbs.The dashboard's session detail page renders these as separate tabs below the player, scrub-synced to the replay timeline.## Session Lifecycle-A session begins automatically when the SDK initializes with `recordAllSessions: true` (or when the page is restored from the bfcache after a back/forward navigation — a fresh `sessionId` is generated each time).+A session begins automatically when the SDK initializes with `recordAllSessions: true` (or when the page is restored from the bfcache after a back/forward navigation; a fresh `sessionId` is generated each time).It ends on the first of:@@ -49,26 +49,26 @@ If a session ends without an explicit close (e.g. the browser is force-quit befoEvery session ships with a context map. Three layers contribute:-1. **Auto-collected defaults** — the SDK stamps `url`, `path`, `referrer`, `userAgent`, `language`, `platform`, `viewport`, `screen`, and `timezone` from `window` / `navigator`. The backend additionally stamps `client.ip` from the request.-2. **Global scope** — anything set via `setAttribute("userId", "u_42")` / `setAttributes({ tenant: "acme" })` on the SDK. These flow through every subsequent session, exception, and segment update.-3. **Per-call attributes** — `captureExceptionWithAttributes(err, { … })` for exceptions only.+1. **Auto-collected defaults**: the SDK stamps `url`, `path`, `referrer`, `userAgent`, `language`, `platform`, `viewport`, `screen`, and `timezone` from `window` / `navigator`. The backend additionally stamps `client.ip` from the request.+2. **Global scope**: anything set via `setAttribute("userId", "u_42")` / `setAttributes({ tenant: "acme" })` on the SDK. These flow through every subsequent session, exception, and segment update.+3. **Per-call attributes**: `captureExceptionWithAttributes(err, { … })` for exceptions only.Layering: defaults → global scope → per-call. Higher layers win on key collision.See the framework-specific guides for the API:- [JS SDK initialization](/client/js-sdk/initialization)-- [React](/client/react/provider) — `setAttribute*` plus the `<TracewayAttributes>` component / `useTracewayAttributes` hook for declarative reactive scope-- [Svelte](/client/svelte/context) — `setAttribute*` plus the `useTracewayAttributes` factory-- [Vue](/client/vue/plugin) — `setAttribute*` (drive from `watchEffect`)-- [React Native](/client/react-native) — same as React, layered above device info+- [React](/client/react/provider): `setAttribute*` plus the `<TracewayAttributes>` component / `useTracewayAttributes` hook for declarative reactive scope+- [Svelte](/client/svelte/context): `setAttribute*` plus the `useTracewayAttributes` factory+- [Vue](/client/vue/plugin): `setAttribute*` (drive from `watchEffect`)+- [React Native](/client/react-native): same as React, layered above device info## Filtering SessionsThe Sessions page in the dashboard accepts:-- **Search by session ID** — paste a UUID for an exact match.-- **Attribute filters** — add `key=value` filters (e.g. `userId=u_42`, `tenant=acme`, `experiment.cart_redesign=variant_b`). Filters AND together; values are exact match.+- **Search by session ID**: paste a UUID for an exact match.+- **Attribute filters**: add `key=value` filters (e.g. `userId=u_42`, `tenant=acme`, `experiment.cart_redesign=variant_b`). Filters AND together; values are exact match.Attribute filters are backed by a Map column with bloom-filter indexes on keys and values, so they stay fast even as the table grows.@@ -83,16 +83,16 @@ This stitching is independent of whether the exception was caught by your code o## Cost and Storage-Always-on session recording is more data than exception-only capture — every active tab uploads continuously, ~30 s segments, until the user leaves. Plan accordingly:+Always-on session recording is more data than exception-only capture: every active tab uploads continuously, ~30 s segments, until the user leaves. Plan accordingly:- Default segment duration is 30 s. This is a 3× reduction in S3 reads compared to a 10 s cadence; tune via `sessionRecordingSegmentDuration` (ms) if your traffic profile differs.- Logs and actions are drained from the rolling buffer at each segment rotation, so a single segment carries the activity it spans (no double-counting across segments).- Recordings older than the platform retention window are pruned automatically.-For most apps, recording a sample of sessions (or only premium-tier users, etc.) is the right balance. Set `recordAllSessions: false` to turn it off entirely — the SDK falls back to exception-only clips with no per-segment uploads.+For most apps, recording a sample of sessions (or only premium-tier users, etc.) is the right balance. Set `recordAllSessions: false` to turn it off entirely. The SDK falls back to exception-only clips with no per-segment uploads.## Next Steps-- [Session Replay in the JS SDK](/client/js-sdk/session-replay) — privacy masking, segment rotation, and the wire format.-- [Issues](/learn/issues) — how exceptions are grouped and what an issue page shows.-- [Attributes](/learn/attributes) — the broader attribute model that backs sessions, exceptions, traces, and tasks.+- [Session Replay in the JS SDK](/client/js-sdk/session-replay): privacy masking, segment rotation, and the wire format.+- [Issues](/learn/issues): how exceptions are grouped and what an issue page shows.+- [Attributes](/learn/attributes): the broader attribute model that backs sessions, exceptions, traces, and tasks.diff --git a/docs/pages/learn/spans.mdx b/docs/pages/learn/spans.mdxindex 8fad2a42..33063934 100644--- a/docs/pages/learn/spans.mdx+++ b/docs/pages/learn/spans.mdx@@ -1,32 +1,32 @@# Spans-Spans are the underlying primitive Traceway stores. Everything else — Endpoints, Tasks, AI Traces, Issues — is a [materialized view](/learn/traces#spans-are-the-primitive) derived from the spans you ingest. The shape is OpenTelemetry-native, so your existing OTel instrumentation works without translation.+Spans are the underlying primitive Traceway stores. Everything else (Endpoints, Tasks, AI Traces, Issues) is a [materialized view](/learn/traces#spans-are-the-primitive) derived from the spans you ingest. The shape is OpenTelemetry-native, so your existing OTel instrumentation works without translation.## What a span captures| Field | Description || --- | --- || `span_id` | 8-byte OTel span id, stored as the low 8 bytes of a UUID |-| `trace_id` | The id of the owning entity (endpoint, task, or ai_trace) — see "Owning entity" below |+| `trace_id` | The id of the owning entity (endpoint, task, or ai_trace); see "Owning entity" below || `parent_span_id` | The span's direct parent within the trace, if any || `name` | Descriptive label (`db.users.find`, `POST https://api.stripe.com/v1/charges`, etc.) || `kind` | One of OTel's `SERVER`, `CLIENT`, `INTERNAL`, `PRODUCER`, `CONSUMER` || `start_time` / `duration` | When it started and how long it ran || `attributes` | Arbitrary key-value metadata (HTTP, DB, AI, custom) |-| `events` | Timestamped events on the span — `"exception"` events become [Issues](/learn/issues) |+| `events` | Timestamped events on the span; `"exception"` events become [Issues](/learn/issues) |## Owning entity, not OTel trace id-Each row in the `spans` table carries a `trace_id` column — but in Traceway it points to the **owning entity** (an endpoint, task, or ai_trace id), not the raw OTel trace id. This is what makes the waterfall view cheap:+Each row in the `spans` table carries a `trace_id` column, but in Traceway it points to the **owning entity** (an endpoint, task, or ai_trace id), not the raw OTel trace id. This is what makes the waterfall view cheap:- "Show me the children of this endpoint" is a single indexed read on `spans.trace_id = <endpoint.id>`.- The full cross-process trace is still reachable via the entity's `distributed_trace_id`, which equals the OTel trace id.-Re-rooting happens at ingestion. When a non-root entity (e.g. a CONSUMER task) is promoted, its descendant spans get the task's id as their `trace_id`. Sibling spans under the originating endpoint keep the endpoint's id. So a single OTel trace that produced an endpoint + a task ends up with two clean waterfalls — one rooted at the endpoint, one at the task — joinable through `distributed_trace_id`.+Re-rooting happens at ingestion. When a non-root entity (e.g. a CONSUMER task) is promoted, its descendant spans get the task's id as their `trace_id`. Sibling spans under the originating endpoint keep the endpoint's id. So a single OTel trace that produced an endpoint + a task ends up with two clean waterfalls (one rooted at the endpoint, one at the task), joinable through `distributed_trace_id`.## What spans are good for-Spans tell you **where the time went**. If an endpoint takes 500ms and 450ms of that is in a `db.users.find` span, you know what to optimize. The dashboard's drill-down — Endpoints / Tasks / AI Traces → instance → waterfall — bottoms out in spans.+Spans tell you **where the time went**. If an endpoint takes 500ms and 450ms of that is in a `db.users.find` span, you know what to optimize. The dashboard's drill-down (Endpoints / Tasks / AI Traces → instance → waterfall) bottoms out in spans.Common things to wrap in a span:@@ -49,7 +49,7 @@ cache.redis.getlogic.pricing.calculate```-`category.resource.action` is a useful default — pick something and stay consistent.+`category.resource.action` is a useful default. Pick something and stay consistent.## Nestingdiff --git a/docs/pages/learn/sso.mdx b/docs/pages/learn/sso.mdxindex fabc8cd2..5608fde9 100644--- a/docs/pages/learn/sso.mdx+++ b/docs/pages/learn/sso.mdx@@ -1,6 +1,6 @@# Single Sign-On (SSO)-Traceway supports signing in with **Google**, **GitHub**, and any **standards-compliant OIDC provider** (Authentik, Keycloak, Dex, Okta, etc.) in addition to email + password. SSO is optional — if you don't configure a provider, only its `Continue with …` button is hidden, the rest of the dashboard works exactly the same.+Traceway supports signing in with **Google**, **GitHub**, and any **standards-compliant OIDC provider** (Authentik, Keycloak, Dex, Okta, etc.) in addition to email + password. SSO is optional: if you don't configure a provider, only its `Continue with …` button is hidden, the rest of the dashboard works exactly the same.When at least one provider is configured, the buttons appear on the login and register pages, the backend exposes `GET /api/auth/providers`, and `GET /api/auth/start/{provider}` kicks off the OAuth flow.@@ -20,17 +20,17 @@ Browser ──► /api/auth/start/githubSign-in matching, in order:-1. **Existing OAuth user** — looked up by `(provider, provider_user_id)`. Logs in.-2. **Existing email user** — same email already registered with a password. The OAuth identity is linked to that account; future logins via either method work.-3. **New user** — created on the fly. On a brand-new self-hosted instance the user lands at `/finish-setup` to create the organization and first project.+1. **Existing OAuth user**: looked up by `(provider, provider_user_id)`. Logs in.+2. **Existing email user**: same email already registered with a password. The OAuth identity is linked to that account; future logins via either method work.+3. **New user**: created on the fly. On a brand-new self-hosted instance the user lands at `/finish-setup` to create the organization and first project.-In **self-hosted mode** (`CLOUD_MODE` unset or not `true`), once any organization exists, OAuth signups for unknown emails are blocked with `invite_required` — they have to be invited to the org first, just like password signups. In cloud mode (`CLOUD_MODE=true`), OAuth signups always create an account and prompt the user through `/finish-setup`.+In **self-hosted mode** (`CLOUD_MODE` unset or not `true`), once any organization exists, OAuth signups for unknown emails are blocked with `invite_required`. They have to be invited to the org first, just like password signups. In cloud mode (`CLOUD_MODE=true`), OAuth signups always create an account and prompt the user through `/finish-setup`.-The OIDC provider can bypass this restriction with `OIDC_AUTO_CREATE_USERS=true` — see below.+The OIDC provider can bypass this restriction with `OIDC_AUTO_CREATE_USERS=true` (see below).## Environment variables-Set these on the backend container/process. All provider pairs are independent — configure one, several, or none.+Set these on the backend container/process. All provider pairs are independent: configure one, several, or none.| Variable | Required when | Description || ------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |@@ -58,15 +58,15 @@ The cookie used during the round-trip is automatically marked `Secure` when `APP## Setting up GitHub-GitHub calls these OAuth Apps (not GitHub Apps — that's a different page).+GitHub calls these OAuth Apps (not GitHub Apps; that's a different page).1. Sign in to GitHub. Open **Settings → Developer settings → OAuth Apps → New OAuth App**, or for an organization-owned app, your org's **Settings → Developer settings → OAuth Apps → New OAuth App**.2. Fill the form:- - **Application name** — whatever shows on the consent screen, e.g. `Traceway`.- - **Homepage URL** — your `APP_BASE_URL`, e.g. `https://traceway.example.com`.- - **Authorization callback URL** — `https://traceway.example.com/api/auth/callback/github`. Must match exactly, no trailing slash.+ - **Application name**: whatever shows on the consent screen, e.g. `Traceway`.+ - **Homepage URL**: your `APP_BASE_URL`, e.g. `https://traceway.example.com`.+ - **Authorization callback URL**: `https://traceway.example.com/api/auth/callback/github`. Must match exactly, no trailing slash.3. Click **Register application**.-4. On the app page, click **Generate a new client secret**. Copy the **Client ID** and the secret immediately — GitHub only shows the secret once.+4. On the app page, click **Generate a new client secret**. Copy the **Client ID** and the secret immediately. GitHub only shows the secret once.5. Set the env vars on the backend:```@@ -79,19 +79,19 @@ GitHub calls these OAuth Apps (not GitHub Apps — that's a different page).For local development, register a second OAuth App with `http://localhost:5173/api/auth/callback/github` (the Vite dev server proxies `/api` to the backend on `:8082`).-The SDK requests the `user:email` scope, so the user's primary email needs to be verified on GitHub — if it isn't, the callback redirects back to `/login?error=oauth_no_email`.+The SDK requests the `user:email` scope, so the user's primary email needs to be verified on GitHub. If it isn't, the callback redirects back to `/login?error=oauth_no_email`.## Setting up Google1. In the [Google Cloud Console](https://console.cloud.google.com/), pick or create a project.2. Open **APIs & Services → OAuth consent screen**:- - **User type** — choose **External** unless you're a Workspace organization that wants the app limited to your domain (then **Internal**).+ - **User type**: choose **External** unless you're a Workspace organization that wants the app limited to your domain (then **Internal**).- Fill in app name, support email, and developer contact email.- Add your domain under **Authorized domains** (e.g. `example.com`).3. **Create Credentials → OAuth client ID**:- - **Application type** — **Web application**.- - **Authorized JavaScript origins** — `${yourappurl}` (and `http://localhost:5173` for dev).- - **Authorized redirect URIs** — `${yourappurl}/api/auth/callback/google` (and `http://localhost:5173/api/auth/callback/google` for dev). Must match exactly.+ - **Application type**: **Web application**.+ - **Authorized JavaScript origins**: `${yourappurl}` (and `http://localhost:5173` for dev).+ - **Authorized redirect URIs**: `${yourappurl}/api/auth/callback/google` (and `http://localhost:5173/api/auth/callback/google` for dev). Must match exactly.4. Copy the **Client ID** and **Client secret** from the dialog.5. Set the env vars on the backend:@@ -112,12 +112,12 @@ Any provider that implements OpenID Connect works: Authentik, Keycloak, Dex, Okt1. In the Authentik admin UI, go to **Applications → Providers → Create**.2. Choose **OAuth2/OpenID Connect Provider**.3. Configure it:- - **Name** — e.g. `traceway`.- - **Client type** — **Confidential**.- - **Client ID / Client Secret** — Authentik generates these; copy them.- - **Redirect URIs** — add exactly: `{APP_BASE_URL}/api/auth/callback/oidc`+ - **Name**: e.g. `traceway`.+ - **Client type**: **Confidential**.+ - **Client ID / Client Secret**: Authentik generates these; copy them.+ - **Redirect URIs**: add exactly `{APP_BASE_URL}/api/auth/callback/oidc`(e.g. `https://traceway.example.com/api/auth/callback/oidc`). No trailing slash.- - **Scopes** — ensure `openid`, `email`, and `profile` are selected.+ - **Scopes**: ensure `openid`, `email`, and `profile` are selected.4. Go to **Applications → Create** and link the provider you just created.### Finding the discovery URL@@ -134,7 +134,7 @@ For example:https://auth.example.com/application/o/traceway/.well-known/openid-configuration```-Open it in a browser to confirm it returns a JSON document — this is the URL you set as `OIDC_DISCOVERY_URL`.+Open it in a browser to confirm it returns a JSON document. This is the URL you set as `OIDC_DISCOVERY_URL`.### Environment variables@@ -157,12 +157,12 @@ Keycloak is a popular open-source identity provider. The setup follows the same1. In the Keycloak admin console, select your realm (or create one).2. Go to **Clients → Create client**.3. Configure it:- - **Client type** — **OpenID Connect**.- - **Client ID** — e.g. `traceway`.- - **Client authentication** — **On** (makes it a confidential client).+ - **Client type**: **OpenID Connect**.+ - **Client ID**: e.g. `traceway`.+ - **Client authentication**: **On** (makes it a confidential client).4. On the **Settings** tab:- - **Valid redirect URIs** — add `{APP_BASE_URL}/api/auth/callback/oidc` (e.g. `https://traceway.example.com/api/auth/callback/oidc`).- - **Web origins** — add your `APP_BASE_URL`.+ - **Valid redirect URIs**: add `{APP_BASE_URL}/api/auth/callback/oidc` (e.g. `https://traceway.example.com/api/auth/callback/oidc`).+ - **Web origins**: add your `APP_BASE_URL`.5. On the **Credentials** tab, copy the **Client secret**.### Discovery URL@@ -203,9 +203,9 @@ OIDC_AUTO_CREATE_USERS=trueWith this flag, any user who authenticates successfully via OIDC is automatically:- Given a Traceway account (if they don't have one yet), and-- Added to a Traceway organization with the `user` role (or the role resolved by `OIDC_ROLE_MAP` if configured — see [Role mapping](#role-mapping)).+- Added to a Traceway organization with the `user` role (or the role resolved by `OIDC_ROLE_MAP` if configured; see [Role mapping](#role-mapping)).-This applies on every sign-in, not just the first — so existing accounts that were created before this flag was set will also be joined to the organization on their next login.+This applies on every sign-in, not just the first, so existing accounts that were created before this flag was set will also be joined to the organization on their next login.On a **single-organization** instance this is all you need. No additional configuration is required.@@ -213,7 +213,7 @@ Do not enable this on a public-facing instance with an external OIDC provider un## Role mapping-By default, OIDC users are added to organizations with the `user` role. Role mapping lets you automatically assign Traceway roles based on claims in the OIDC token — useful for enterprise IAM systems like Keycloak where group or role membership is already managed centrally.+By default, OIDC users are added to organizations with the `user` role. Role mapping lets you automatically assign Traceway roles based on claims in the OIDC token. This is useful for enterprise IAM systems like Keycloak where group or role membership is already managed centrally.Two env vars control this:@@ -222,11 +222,11 @@ Two env vars control this:| `OIDC_ROLE_CLAIM` | `realm_access.roles` | Dot-notation path to the claim holding the role value(s). Supports nested objects and string arrays. || `OIDC_ROLE_MAP` | `{"traceway-admin":"admin","traceway-ro":"readonly"}` | JSON object mapping claim values to Traceway roles (`admin`, `user`, or `readonly`). |-Role mapping is applied on **every login**, not just the first — so changing a user's role in your IdP takes effect on their next sign-in.+Role mapping is applied on **every login**, not just the first, so changing a user's role in your IdP takes effect on their next sign-in.-**Priority order:** When a user has multiple claim values that match different roles, the highest-privilege match wins: `admin` > `user` > `readonly`. The `owner` role is never assigned automatically — it can only be set manually by an existing owner.+**Priority order:** When a user has multiple claim values that match different roles, the highest-privilege match wins: `admin` > `user` > `readonly`. The `owner` role is never assigned automatically. It can only be set manually by an existing owner.-### Keycloak example — realm roles+### Keycloak example: realm rolesKeycloak includes realm-level roles in the token under `realm_access.roles`:@@ -247,7 +247,7 @@ OIDC_ROLE_MAP={"traceway-admin":"admin","traceway-user":"user","traceway-readonlCreate matching realm roles in Keycloak (**Realm Settings → Roles → Create role**) and assign them to users or groups.-### Generic example — groups claim+### Generic example: groups claimMany providers expose group membership as a top-level `groups` claim:@@ -281,27 +281,27 @@ Authentik doesn't include role or group claims by default. Add them via a scope## Routing users to organizations-> This section only applies to **multi-organization** instances. If you have one organization, `OIDC_AUTO_CREATE_USERS=true` is sufficient — users are automatically added to it.+> This section only applies to **multi-organization** instances. If you have one organization, `OIDC_AUTO_CREATE_USERS=true` is sufficient: users are automatically added to it.By default, auto-created OIDC users are added to the first organization in the instance. To route users to a specific organization on a multi-org instance, use a custom claim in the ID token:-**Step 1** — Set `OIDC_ORG_CLAIM` to the name of the claim that will carry the organization name:+**Step 1**: Set `OIDC_ORG_CLAIM` to the name of the claim that will carry the organization name:```OIDC_ORG_CLAIM=traceway_org```-**Step 2** — Set `OIDC_EXTRA_SCOPES` to the scope that exposes the claim (so Traceway requests it from the provider):+**Step 2**: Set `OIDC_EXTRA_SCOPES` to the scope that exposes the claim (so Traceway requests it from the provider):```OIDC_EXTRA_SCOPES=traceway```-**Step 3** — Configure your OIDC provider to include the claim in the token (see below).+**Step 3**: Configure your OIDC provider to include the claim in the token (see below).When a user signs in, Traceway reads the `traceway_org` claim from the ID token and looks up the organization by name. If the claim is absent or the name doesn't match any organization, it falls back to the first organization.-The organization name in the claim must exactly match the name in Traceway — it is case-sensitive.+The organization name in the claim must exactly match the name in Traceway. It is case-sensitive.### Adding the claim in Authentik@@ -314,7 +314,7 @@ The organization name in the claim must exactly match the name in Traceway — iYou can make this dynamic per-user using Authentik's expression variables (e.g. `request.user.ak_groups.all()`).4. Go to your Traceway OIDC provider → **Advanced protocol settings → Scopes** and add `traceway` to the list alongside `openid email profile`.-This works identically with any OIDC-compliant provider — Keycloak, Okta, Dex, and others all support custom claims via their own claim/attribute mapping configuration.+This works identically with any OIDC-compliant provider: Keycloak, Okta, Dex, and others all support custom claims via their own claim/attribute mapping configuration.## Providers without OIDC discovery@@ -328,7 +328,7 @@ OIDC_TOKEN_URL=https://provider.example.com/oauth/tokenOIDC_USER_INFO_URL=https://provider.example.com/oauth/userinfo```-All three manual URLs must be set together — you cannot mix them with `OIDC_DISCOVERY_URL`. If `OIDC_DISCOVERY_URL` is set, it takes precedence and the manual URLs are ignored.+All three manual URLs must be set together. You cannot mix them with `OIDC_DISCOVERY_URL`. If `OIDC_DISCOVERY_URL` is set, it takes precedence and the manual URLs are ignored.## Forcing SSO-only login@@ -340,7 +340,7 @@ DISABLE_PASSWORD_LOGIN=trueWith this flag:- The email/password form is hidden on the login and register pages.-- `POST /api/login` and `POST /api/register` return `403 Forbidden` — existing scripts or integrations that use password auth will fail explicitly rather than silently.+- `POST /api/login` and `POST /api/register` return `403 Forbidden`, so existing scripts or integrations that use password auth will fail explicitly rather than silently.- The "or" divider between SSO buttons and the password form is also removed.On a self-hosted instance with no organizations yet, the first user is created and onboarded through the SSO flow (via `/finish-setup`) rather than the register page. Pair this with `OIDC_AUTO_CREATE_USERS=true` so the first login automatically provisions the account without needing an invitation.@@ -356,14 +356,14 @@ curl https://traceway.example.com/api/auth/providers`passwordLoginEnabled` is `false` when `DISABLE_PASSWORD_LOGIN=true`.-If the `oidc` entry is missing but you set the env vars, check the backend logs — a failed discovery URL fetch is logged at startup. Common causes:+If the `oidc` entry is missing but you set the env vars, check the backend logs: a failed discovery URL fetch is logged at startup. Common causes:- The discovery URL is wrong or the OIDC provider is not reachable from the backend container at startup.- Only some of the required OIDC vars are set. Either `OIDC_DISCOVERY_URL` or all three of `OIDC_AUTH_URL` + `OIDC_TOKEN_URL` + `OIDC_USER_INFO_URL` must be provided, together with `OIDC_CLIENT_ID` and `OIDC_CLIENT_SECRET`.If the array is empty for all providers, the backend didn't see your env vars. Common causes:-- The container was started before the secrets were set — recreate, don't just restart.+- The container was started before the secrets were set. Recreate, don't just restart.- The variables were set in a different shell than the one that launched `go run` / `docker run`.Once `/api/auth/providers` reports the providers, the buttons appear automatically on the login and register pages.@@ -381,7 +381,7 @@ The OAuth callback redirects back to `/login?error=<code>` when something goes w## Security notes- The callback hands the JWT back via the URL fragment (`#token=…`), so it never reaches the backend logs or a TLS-terminating proxy as a query string.-- The transient cookie used by gothic during the round-trip is `HttpOnly`, `SameSite=Lax`, and 10-minute lifetime. It only carries the OAuth state and PKCE verifier — never a long-lived session.+- The transient cookie used by gothic during the round-trip is `HttpOnly`, `SameSite=Lax`, and 10-minute lifetime. It only carries the OAuth state and PKCE verifier, never a long-lived session.- `OAUTH_SESSION_SECRET` rotates the cookie signing key. Rotating it invalidates any in-flight OAuth round-trips but does not affect issued JWTs (those are signed with `JWT_SECRET`).-- An OAuth-only user has an empty password hash. Password login for them always fails — `CheckPassword` short-circuits on empty hashes.-- Role mapping (`OIDC_ROLE_MAP`) never elevates a user to `owner` — that role can only be assigned manually by an existing owner.+- An OAuth-only user has an empty password hash. Password login for them always fails: `CheckPassword` short-circuits on empty hashes.+- Role mapping (`OIDC_ROLE_MAP`) never elevates a user to `owner`; that role can only be assigned manually by an existing owner.diff --git a/docs/pages/learn/traces.mdx b/docs/pages/learn/traces.mdxindex 45766ee6..f394b2f9 100644--- a/docs/pages/learn/traces.mdx+++ b/docs/pages/learn/traces.mdx@@ -1,10 +1,10 @@# Traces-A **trace** is the lifeline of one logical operation as it moves through your system — a request, a queued job, a scheduled command, an LLM call. Traceway models traces the same way OpenTelemetry does: a tree of **spans** identified by a shared trace id.+A **trace** is the lifeline of one logical operation as it moves through your system: a request, a queued job, a scheduled command, an LLM call. Traceway models traces the same way OpenTelemetry does: a tree of **spans** identified by a shared trace id.## Spans are the primitive-Spans are the ground truth. Every other concept in Traceway — Endpoints, Tasks, AI Traces, Issues — is **derived from spans on ingestion**, then stored in a dedicated table so the dashboard can answer "what's slow?" or "what's expensive?" in milliseconds without scanning every span.+Spans are the ground truth. Every other concept in Traceway (Endpoints, Tasks, AI Traces, Issues) is **derived from spans on ingestion**, then stored in a dedicated table so the dashboard can answer "what's slow?" or "what's expensive?" in milliseconds without scanning every span.```OTLP span batch@@ -25,13 +25,13 @@ OTLP span batchYou can think of `endpoints`, `tasks`, and `ai_traces` as **cached projections** of the spans table. They exist for one reason: aggregations like "P95 of every endpoint over 7 days" or "total LLM cost grouped by trace_name" need to be cheap. Scanning the full spans table to compute those every dashboard load would not be cheap.The trade-off this design makes:-- Top-level dashboards (Endpoints, Tasks, AI Traces) are **fast** — they read a small, pre-aggregated table.+- Top-level dashboards (Endpoints, Tasks, AI Traces) are **fast**: they read a small, pre-aggregated table.- The detail/waterfall view falls back to `spans` for the full timeline of children under any entity.- Promotion happens **at write time**, not at query time, so the rules are baked in once per span.## What gets promoted-A span lands in a top-level table when its kind or attributes match a rule. The classifier doesn't care whether the span is a root or a child — it cares about what the span represents.+A span lands in a top-level table when its kind or attributes match a rule. The classifier doesn't care whether the span is a root or a child. It cares about what the span represents.| Condition on the OTel span | Promoted to ||---|---|@@ -42,14 +42,14 @@ A span lands in a top-level table when its kind or attributes match a rule. The| `"exception"` event on any span | **Issue** (attributed to the owning entity above) || Anything else with a parent | **Span** in the waterfall, re-rooted to its nearest enclosing entity |-In-process sub-handler spans (e.g. a framework's "handler /path" INTERNAL+HTTP child) stay in the spans table — endpoints are only promoted for actual entry points, so frameworks don't silently double-emit.+In-process sub-handler spans (e.g. a framework's "handler /path" INTERNAL+HTTP child) stay in the spans table. Endpoints are only promoted for actual entry points, so frameworks don't silently double-emit.## Root and non-root entitiesEach Endpoint / Task / AI Trace row carries an `is_root` flag.-- **Root** — the span had no parent in any process. This is the typical case: an inbound HTTP request, a standalone cron run, a fresh LLM call.-- **Non-root** — the span was triggered by another trace. Examples:+- **Root**: the span had no parent in any process. This is the typical case: an inbound HTTP request, a standalone cron run, a fresh LLM call.+- **Non-root**: the span was triggered by another trace. Examples:- A queue worker's CONSUMER span parented to the dispatcher's PRODUCER span via the trace context serialized into the job payload.- A child gen_ai span made inside a request handler.- A cross-service inbound HTTP hop where the upstream service propagated `traceparent`.@@ -64,18 +64,18 @@ The id rules avoid collisions when a root and a non-root entity share the same tEvery Endpoint / Task / AI Trace stores a **`distributed_trace_id`** equal to the OTel trace id, auto-derived on ingestion. So:-- A request that dispatches a queue job and the worker that processes it share the same `distributed_trace_id`. They show up as two rows — the endpoint and the task — and a single page (`/distributed-traces/<id>`) renders them side by side with their spans.+- A request that dispatches a queue job and the worker that processes it share the same `distributed_trace_id`. They show up as two rows (the endpoint and the task), and a single page (`/distributed-traces/<id>`) renders them side by side with their spans.- A request handler that calls an LLM produces an endpoint row and an ai_trace row with the same `distributed_trace_id`. Same cross-trace page.- Cross-service HTTP hops (each service exports its own SERVER span) all share the trace id. Each hop appears as its own endpoint row.-You don't need to set any vendor attribute for this to work — OTLP's native trace_id is enough. (The Go SDK's manual `distributedTraceId` field continues to work for legacy `/api/report` flows and overrides the auto-derived value when set.)+You don't need to set any vendor attribute for this to work, OTLP's native trace_id is enough. If you do need to stitch runs that genuinely carry different trace ids, the span attribute `traceway.distributed_trace_id` overrides the derived value.## What each entity storesThe shape mirrors what's useful for the dashboard's aggregations on that entity.### Endpoints-Identified by HTTP method + route template (`GET /api/users/:id`). Ranked by **Impact** — a composite of Apdex, error rate, P99, client-error rate, and volume-weighted error rate. Streaming responses (SSE / WebSocket) are flagged and excluded from latency-based components.+Identified by HTTP method + route template (`GET /api/users/:id`). Ranked by **Impact**, a composite of Apdex, error rate, P99, client-error rate, and volume-weighted error rate. Streaming responses (SSE / WebSocket) are flagged and excluded from latency-based components.#### Unmatched requests (404s)@@ -98,15 +98,15 @@ Identified by `trace.name` (your agent or workflow). Carries model, provider, in## Lifecycle and sampling-1. A span is created in your app (by a framework's auto-instrumentation, the Traceway SDK, or manual OTel calls).+1. A span is created in your app (by a framework's auto-instrumentation or by manual OTel calls).2. Child spans, attributes, and events accumulate against it.-3. The trace is exported via OTLP (or `/api/report` for the legacy Go SDK).+3. The trace is exported via OTLP.4. Traceway classifies each span, inserts into the matching entity table, and re-roots the remaining children into `spans` under their nearest enclosing entity.5. Exceptions recorded as `"exception"` events become Issues attributed to the owning entity.Sampling:-- **Sample rate** — percentage of normal traces recorded (default: 100%).-- **Error sample rate** — percentage of error traces recorded (default: 100%). Keep this at 100% to catch every failure.+- **Sample rate**: percentage of normal traces recorded (default: 100%).+- **Error sample rate**: percentage of error traces recorded (default: 100%). Keep this at 100% to catch every failure.<img src="/endpoints.png" alt="Endpoints Dashboard" style={{ maxWidth: '100%', height: 'auto', marginTop: '16px', borderRadius: '8px', border: '1px solid #e5e7eb' }} />diff --git a/docs/pages/protocol/index.md b/docs/pages/protocol/index.mdindex e648af5b..f8ab855c 100644--- a/docs/pages/protocol/index.md+++ b/docs/pages/protocol/index.md@@ -50,7 +50,7 @@ Parse by splitting on the first `@`:### Gzip Requirement-The request body **should** be gzip-compressed JSON for the regular sync path. The backend's gzip middleware decompresses when `Content-Encoding: gzip` is present and passes the body through unchanged when it is absent. Browser SDKs use the uncompressed path on the page-unload (`pagehide`) flush so the request can dispatch synchronously inside the unload handler — `CompressionStream` is async and would yield control before the request leaves. Servers MUST therefore accept both compressed and uncompressed JSON on `/api/report`.+The request body **should** be gzip-compressed JSON for the regular sync path. The backend's gzip middleware decompresses when `Content-Encoding: gzip` is present and passes the body through unchanged when it is absent. Browser SDKs use the uncompressed path on the page-unload (`pagehide`) flush so the request can dispatch synchronously inside the unload handler; `CompressionStream` is async and would yield control before the request leaves. Servers MUST therefore accept both compressed and uncompressed JSON on `/api/report`.## Request Payload@@ -136,9 +136,9 @@ A trace represents either an **endpoint** (HTTP request) or a **task** (backgrou**UUIDs** are v4, formatted as `"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"`.-**`attributes` and `spans`** use `omitempty` in Go — they are omitted from JSON when null/empty. Your SDK may omit them or send `null`/`{}` and `[]`.+**`attributes` and `spans`** use `omitempty` in Go, which means they are omitted from JSON when null/empty. Your SDK may omit them or send `null`/`{}` and `[]`.-**`isTask`** uses `omitempty` — it is omitted when `false`. The backend treats a missing `isTask` as `false`.+**`isTask`** uses `omitempty`, so it is omitted when `false`. The backend treats a missing `isTask` as `false`.### Endpoints vs. Tasks@@ -199,7 +199,7 @@ An exception record represents either an **error/panic** or an explicitly **capt### Notes- **`traceId`** is a nullable string pointer. When there is no linked trace, send `null` or omit the field entirely.-- **`isTask`** uses `omitempty` — omitted when `false`.+- **`isTask`** uses `omitempty` (omitted when `false`).- **`isMessage=false`** → error or panic with a stack trace. The backend normalizes and hashes the stack trace for grouping.- **`isMessage=true`** → an explicitly captured message (e.g., via `CaptureMessage`). The full `stackTrace` string is hashed as-is for grouping.@@ -278,7 +278,7 @@ The Go client collects these system metrics automatically:### Custom Metrics-SDKs can send any `name`/`value` pair. There is no registry — any string name is accepted.+SDKs can send any `name`/`value` pair. There is no registry: any string name is accepted.```json{ "name": "queue.length", "value": 42.0, "recordedAt": "2025-01-15T10:30:00Z" }@@ -286,9 +286,9 @@ SDKs can send any `name`/`value` pair. There is no registry — any string name## Sessions-A **session** represents one user's continuous interaction with a browser app — typically the lifetime of an open tab. Sessions are emitted only by browser SDKs that have **always-on session recording** enabled (`recordAllSessions: true`); back-end and mobile SDKs omit the `sessions` and `sessionRecordings` fields entirely.+A **session** represents one user's continuous interaction with a browser app, typically the lifetime of an open tab. Sessions are emitted only by browser SDKs that have **always-on session recording** enabled (`recordAllSessions: true`); back-end and mobile SDKs omit the `sessions` and `sessionRecordings` fields entirely.-The session row is upserted on the backend (keyed by `id`), so the same session can be emitted multiple times during its lifetime — each emission overwrites mutable fields (`endedAt`, `duration`, `attributes`). The shape is identical for the open, mid-session refresh, and close payloads; only the populated fields differ.+The session row is upserted on the backend (keyed by `id`), so the same session can be emitted multiple times during its lifetime; each emission overwrites mutable fields (`endedAt`, `duration`, `attributes`). The shape is identical for the open, mid-session refresh, and close payloads; only the populated fields differ.### Session Object@@ -323,13 +323,13 @@ The session row is upserted on the backend (keyed by `id`), so the same sessionA new session begins on:1. SDK init when `recordAllSessions: true` and a browser context is available.-2. Page restore from bfcache (`pageshow` with `event.persisted === true`) — a fresh `id` is generated and the previous session row is left as-is.+2. Page restore from bfcache (`pageshow` with `event.persisted === true`). A fresh `id` is generated and the previous session row is left as-is.A session ends on:-1. **Inactivity timeout** — 15 min since the last DOM event observed by rrweb (or the last `markActivity()` call).-2. **Maximum duration** — 60 min from `startedAt`.-3. **Page unload** — `pagehide` triggers a final flush and end.+1. **Inactivity timeout**: 15 min since the last DOM event observed by rrweb (or the last `markActivity()` call).+2. **Maximum duration**: 60 min from `startedAt`.+3. **Page unload**: `pagehide` triggers a final flush and end.The closing payload is sent via `fetch(..., { keepalive: true })` with raw JSON (no gzip) so it dispatches synchronously inside the `pagehide` handler.@@ -352,7 +352,7 @@ Browser SDKs stamp these on every session at open and close (caller-provided key## Session Recordings-Session recordings carry the actual replay data. Each recording is **one segment** — typically ~30 s of rrweb events. A session is reassembled at view time by ordering all of its recordings by `segmentIndex`.+Session recordings carry the actual replay data. Each recording is **one segment**, typically ~30 s of rrweb events. A session is reassembled at view time by ordering all of its recordings by `segmentIndex`.A recording can be **session-bound** (always-on path), **exception-bound** (legacy, when an exception is captured), or both at once when an exception fires inside an always-on segment.@@ -382,7 +382,7 @@ A recording can be **session-bound** (always-on path), **exception-bound** (lega| StartedAt | `startedAt` | `string` | No | RFC 3339 timestamp of the first event in the segment || EndedAt | `endedAt` | `string` | No | RFC 3339 timestamp of the last event in the segment |-The backend stores each recording as a row in `session_recordings` with the events bytes written to S3 (or a configured object store). At least one of `exceptionId` or `sessionId` must be set — recordings with neither linkage are dropped.+The backend stores each recording as a row in `session_recordings` with the events bytes written to S3 (or a configured object store). At least one of `exceptionId` or `sessionId` must be set; recordings with neither linkage are dropped.## Exception ↔ Session Linking@@ -405,7 +405,7 @@ The legacy `sessionRecordingId` field still works for exception-bound clips. Bot## Batching and Collection Strategy-This section describes the recommended client-side implementation, based on the Go SDK.+This section describes the recommended client-side implementation for anyone writing a client against this protocol directly. Most applications should use OpenTelemetry instead, which handles all of it. See [OpenTelemetry](/client/otel).### Collection Interval@@ -469,7 +469,7 @@ When a trace is not sampled, do not send the trace or any associated exceptions.| Status | Meaning | Action ||--------|---------|--------|| `200` | Success | Remove sent frames from the send queue |-| `400` | Missing `Content-Encoding: gzip` or malformed JSON | Fix the request — do not retry as-is |+| `400` | Missing `Content-Encoding: gzip` or malformed JSON | Fix the request; do not retry as-is || `401` | Invalid or expired token | Check the project token configuration || `429` | Rate limited | Back off and retry on the next interval || Any other | Server error or network issue | Retry on the next interval |diff --git a/docs/pages/server/capacity-monitoring.mdx b/docs/pages/server/capacity-monitoring.mdxindex 8e843225..d44575ed 100644--- a/docs/pages/server/capacity-monitoring.mdx+++ b/docs/pages/server/capacity-monitoring.mdx@@ -1,6 +1,6 @@# Capacity & Self-Monitoring-Traceway's backend can monitor itself with the same Go SDK your applications use. When enabled, the server continuously reports ingestion pressure, ClickHouse health, cache sizes, and process resource usage into a regular Traceway project — so you can dashboard them, and more importantly, alert on them before you hit a wall.+Traceway's backend can monitor itself. When enabled, the server continuously reports ingestion pressure, ClickHouse health, cache sizes, and process resource usage into a regular Traceway project, so you can dashboard them, and more importantly, alert on them before you hit a wall.This page lists every self-monitoring metric, recommended alert thresholds, and a decision tree for the three scaling questions that matter in production: **do I scale ClickHouse, do I scale the backend server, or do I put Kafka in front of ingestion?**@@ -16,7 +16,7 @@ MONITORING_TRACEWAY_URL="<project_token>@https://your-traceway-host/api/report"That single variable turns everything on:- Every HTTP request to the backend is captured as a transaction (so endpoint latency/error-rate alert rules work on `POST /api/report` out of the box).-- The Go SDK reports runtime metrics every 30s: `cpu.used_pcnt`, `mem.used`, `mem.total`, `go.go_routines`, `go.heap_objects`, `go.num_gc`, `go.gc_pause`.+- Runtime metrics are reported every 30s: `cpu.used_pcnt`, `mem.used`, `mem.total`, `go.go_routines`, `go.heap_objects`, `go.num_gc`, `go.gc_pause`.- The backend reporters described below start emitting `traceway.*` metrics.Note: `mem.used` is Go **heap** memory only. The real OOM-relevant number is `traceway.proc.rss_mb` (resident set size, Linux only).@@ -30,7 +30,7 @@ Emitted once per `/api/report` and OTLP request, tagged with `signal` and `table| Metric | Meaning ||--------|---------|| `traceway.ingest.batch.convert_ms` | Time converting the client payload into rows (backend CPU work) |-| `traceway.ingest.batch.insert_ms` | Time writing the batch into ClickHouse (inside the request — clients wait on this) |+| `traceway.ingest.batch.insert_ms` | Time writing the batch into ClickHouse (inside the request, so clients wait on this) || `traceway.ingest.batch.size` | Rows in the batch || `traceway.ingest.batch.bytes` | Decompressed payload size in bytes || `traceway.ingest.rate_limited` | Fired when an org hits its ingest quota (tagged `org_id`) |@@ -69,7 +69,7 @@ Not emitted in SQLite mode.| `traceway.recordings.queue_depth` | Segments waiting in the upload queue (cap default 2048) || `traceway.recordings.in_flight` | Workers mid-upload || `traceway.recordings.uploaded.delta` | Segments uploaded since the last tick |-| `traceway.recordings.dropped.delta` | Segments dropped (queue full) since the last tick — data loss |+| `traceway.recordings.dropped.delta` | Segments dropped (queue full) since the last tick, meaning data loss || `traceway.recordings.failed.delta` | Upload/insert failures since the last tick |### Source map cache (every 30s)@@ -117,12 +117,12 @@ Sizing guidance:Create rules in the monitoring project under **Notifications** using the **Metric threshold** rule type. A rule is defined by: metric name, aggregation (`avg`, `max`, `min`, `sum`, `p95`, `p99`), operator (`gt`, `gte`, `lt`, `lte`, `eq`), threshold value, and lookback window in minutes. Rules are evaluated every 60 seconds.-**Tag caveat:** metric threshold rules aggregate across *all* tag combinations of a metric name. That is what you want for the metrics on this page — `max` on `traceway.ch.parts.max_per_partition` gives you the worst partition across every table, and `max` on `traceway.ch.disk.used_pcnt` gives you the fullest disk. Just avoid `avg` on per-table/per-disk metrics, where mixing series would dilute the signal.+**Tag caveat:** metric threshold rules aggregate across *all* tag combinations of a metric name. That is what you want for the metrics on this page: `max` on `traceway.ch.parts.max_per_partition` gives you the worst partition across every table, and `max` on `traceway.ch.disk.used_pcnt` gives you the fullest disk. Just avoid `avg` on per-table/per-disk metrics, where mixing series would dilute the signal.Two more rule types complete the picture:-- **Endpoint P95 threshold** on `POST /api/report` — client-visible ingest latency (the backend's own HTTP requests are transactions in the monitoring project).-- **No data** with data type `metrics` — fires when the backend stops self-reporting entirely (crash, network partition).+- **Endpoint P95 threshold** on `POST /api/report`: client-visible ingest latency (the backend's own HTTP requests are transactions in the monitoring project).+- **No data** with data type `metrics`, which fires when the backend stops self-reporting entirely (crash, network partition).## Recommended starting alerts@@ -130,15 +130,15 @@ Two more rule types complete the picture:|--------|-----|----|-----------|----------|---------|---------|| `traceway.ch.failed_inserts.delta` | max | gt | 0 | 5 | critical | Inserts are failing right now || `traceway.ch.rejected_inserts.delta` | max | gt | 0 | 5 | critical | ClickHouse is rejecting inserts (3000 parts/partition hit) |-| `traceway.ch.inserted_rows.delta` | max | lte | 0 | 10 | critical | Ingestion stalled — nothing written for 10 minutes |+| `traceway.ch.inserted_rows.delta` | max | lte | 0 | 10 | critical | Ingestion stalled: nothing written for 10 minutes || `traceway.ch.disk.used_pcnt` | max | gt | 90 | 5 | critical | ClickHouse disk nearly full || `traceway.ch.parts.max_per_partition` | max | gt | 2000 | 5 | critical | Approaching the insert-rejection threshold |-| `traceway.ingest.batch.insert_ms` | p95 | gt | 500 | 10 | warning | ClickHouse write latency rising — clients feel this directly |+| `traceway.ingest.batch.insert_ms` | p95 | gt | 500 | 10 | warning | ClickHouse write latency rising; clients feel this directly || `traceway.ch.delayed_inserts.delta` | max | gt | 0 | 10 | warning | ClickHouse is throttling inserts (merge back-pressure) || `traceway.ch.parts.max_per_partition` | max | gt | 700 | 10 | warning | Approaching the insert-delay threshold (1000) || `traceway.ch.disk.used_pcnt` | max | gt | 80 | 15 | warning | ClickHouse disk filling up |-| `traceway.ch.pool.open` | avg | gte | 15 | 10 | warning | Connection pool saturated — check `insert_ms` next |-| `traceway.recordings.dropped.delta` | max | gt | 0 | 5 | warning | Recording segments being dropped — ingestion overload |+| `traceway.ch.pool.open` | avg | gte | 15 | 10 | warning | Connection pool saturated; check `insert_ms` next |+| `traceway.recordings.dropped.delta` | max | gt | 0 | 5 | warning | Recording segments being dropped (ingestion overload) || `traceway.sourcemap.load_failures.delta` | max | gt | 0 | 10 | warning | Source map loads failing (storage errors/timeouts); traces stored unsymbolicated || `cpu.used_pcnt` | avg | gt | 80 | 10 | warning | Backend CPU-bound || `traceway.proc.rss_mb` | avg | gt | 80% of host RAM | 10 | warning | Memory pressure / OOM risk (e.g. 800 on a 1 GB box) |@@ -148,7 +148,7 @@ Plus the two non-metric rules: **Endpoint P95 threshold** on `POST /api/report`## Scale ClickHouse, scale the backend, or add Kafka?-Ingestion in Traceway is synchronous: each `/api/report` request converts the payload and writes it to ClickHouse before responding. That makes the diagnosis straightforward — the metrics tell you which side of that pipeline is saturated, and whether the problem is sustained or bursty.+Ingestion in Traceway is synchronous: each `/api/report` request converts the payload and writes it to ClickHouse before responding. That makes the diagnosis straightforward: the metrics tell you which side of that pipeline is saturated, and whether the problem is sustained or bursty.### Scale ClickHouse when the pressure is sustained and CH-side@@ -159,12 +159,12 @@ ClickHouse can't merge or store as fast as you're writing. Signals, in rough ord- `traceway.ch.disk.used_pcnt` rising, `traceway.ch.memory.tracking_bytes` near the host's RAM- `traceway.ingest.batch.insert_ms` p95 high **while the ClickHouse host's CPU is also high**-Fix: more CPU/RAM/disk for ClickHouse, faster disks, or sharding. Kafka won't help here — a buffer in front of a database that's saturated on average just delays the inevitable.+Fix: more CPU/RAM/disk for ClickHouse, faster disks, or sharding. Kafka won't help here: a buffer in front of a database that's saturated on average just delays the inevitable.### Scale the backend server when the Go process is the bottleneck-- `cpu.used_pcnt` avg > 80% together with high `traceway.ingest.batch.convert_ms` — payload conversion (JSON/OTLP decoding, hashing) is CPU work that scales with traffic-- `traceway.proc.rss_mb` approaching host RAM — before resizing, tune the source map cache variables above; they're the largest tunable consumer+- `cpu.used_pcnt` avg > 80% together with high `traceway.ingest.batch.convert_ms`; payload conversion (JSON/OTLP decoding, hashing) is CPU work that scales with traffic+- `traceway.proc.rss_mb` approaching host RAM: before resizing, tune the source map cache variables above; they're the largest tunable consumer- `go.go_routines` growing without boundFix: a bigger box, or multiple backend instances behind a load balancer (the backend is stateless apart from in-memory caches).@@ -175,7 +175,7 @@ Because inserts happen inside the request, every ClickHouse hiccup or burst beco- `traceway.ingest.batch.insert_ms` p95 spiking while ClickHouse CPU, disk, and merges look calm on average- `traceway.ch.pool.open` pinned at 15 (pool saturated) during peaks-- `traceway.recordings.dropped.delta` > 0 — the one buffered path is already overflowing during bursts+- `traceway.recordings.dropped.delta` > 0; the one buffered path is already overflowing during bursts- P95 on `POST /api/report` rising even though total `traceway.ch.inserted_rows.delta` is well within what ClickHouse handles at steady stateThat pattern means ClickHouse can absorb your *average* write rate but not your *peaks*, and the synchronous path turns peaks into client errors and data loss. A queue (Kafka) between the HTTP handlers and ClickHouse decouples client latency from insert latency, absorbs spikes, and enables larger, better-shaped batches.diff --git a/docs/pages/server/docker-compose.mdx b/docs/pages/server/docker-compose.mdxindex 17018932..d7781e21 100644--- a/docs/pages/server/docker-compose.mdx+++ b/docs/pages/server/docker-compose.mdx@@ -4,7 +4,7 @@ Runs Traceway, ClickHouse, and PostgreSQL as separate services using the [Minima## Quick Start-No build required — use the pre-built image from GitHub Container Registry with this `docker-compose.yml`:+No build required. Use the pre-built image from GitHub Container Registry with this `docker-compose.yml`:```yamlservices:diff --git a/docs/pages/server/local-setup.mdx b/docs/pages/server/local-setup.mdxindex 8ee288c3..125e5f80 100644--- a/docs/pages/server/local-setup.mdx+++ b/docs/pages/server/local-setup.mdx@@ -126,25 +126,16 @@ The project token is used by the SDK to send telemetry data.## Connecting Your Application-Once you have a project token, you can connect your Go application:+Once you have a project token, point your application's OpenTelemetry exporter at the local instance. The endpoint is `http://localhost:8082/api/otel` and the token goes on the `Authorization` header:-```go-tracewaygin "go.tracewayapp.com/tracewaygin"--func main() {- router := gin.Default()-- // Replace with your project token- router.Use(tracewaygin.New(- "your_project_token@http://localhost:8082/api/report",- ))-- router.GET("/", handler)- router.Run(":8080")-}+```bash+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8082/api/otel+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your_project_token"+export OTEL_SERVICE_NAME=my-service```-See the [SDK Reference](/client/sdk) for detailed SDK usage.+See [OpenTelemetry](/client/otel) for the full setup, including the per-framework guides.## Troubleshootingdiff --git a/docs/pages/server/sqlite.mdx b/docs/pages/server/sqlite.mdxindex c3fc4ab9..2a21dfd6 100644--- a/docs/pages/server/sqlite.mdx+++ b/docs/pages/server/sqlite.mdx@@ -4,7 +4,7 @@ Smallest possible Traceway deployment: a single Alpine container that uses SQLitIf your dashboards slow down as telemetry grows, the [DuckDB deployment](/server/duckdb) keeps this exact single-container setup but swaps the telemetry store for a columnar engine with far more read headroom.-All persistent state — both SQLite database files and blob storage — lives under `/data`. Bind-mount that single folder to keep your data across restarts.+All persistent state (both SQLite database files and blob storage) lives under `/data`. Bind-mount that single folder to keep your data across restarts.## Quick Start@@ -54,7 +54,7 @@ docker run -d --name traceway \For non-AWS providers add `-e S3_ENDPOINT="https://s3.example.com"`. If `S3_ACCESS_KEY` and `S3_SECRET_KEY` are omitted, the AWS SDK falls back to the default credential chain (IAM role, environment, shared config).-If `STORAGE_TYPE` is unset (or set to `local`), blobs are written to `/data/storage`, the same volume as the SQLite files — no extra mounts needed.+If `STORAGE_TYPE` is unset (or set to `local`), blobs are written to `/data/storage`, the same volume as the SQLite files. No extra mounts are needed.## Configuration@@ -137,7 +137,7 @@ docker stop traceway && docker rm traceway## Deploying to Railway-[Railway](https://railway.app) builds and runs the image from your GitHub repository, mounts a persistent volume at `/data`, and gives you a public HTTPS URL — a great fit for the SQLite deployment.+[Railway](https://railway.app) builds and runs the image from your GitHub repository, mounts a persistent volume at `/data`, and gives you a public HTTPS URL, a great fit for the SQLite deployment.**1. Connect the repo.** Push the project to GitHub, then in Railway choose **New Project → Deploy from GitHub repo** and pick the Traceway repository.diff --git a/docs/pages/symbolicator/android.mdx b/docs/pages/symbolicator/android.mdxindex db86aa69..baeeb697 100644--- a/docs/pages/symbolicator/android.mdx+++ b/docs/pages/symbolicator/android.mdx@@ -43,6 +43,7 @@ Then apply it to your app module:```kotlinplugins {+ id("com.android.application") // must already be appliedid("com.tracewayapp.symbols") version "1.0.1"}@@ -61,6 +62,8 @@ traceway {}```+Order matters in the `plugins { }` block. Gradle applies entries top to bottom, and `com.tracewayapp.symbols` looks for the Android extension the moment it is applied, so it has to come **after** `com.android.application` (or `com.android.library`). Declared above it, the build fails at configuration time with `com.tracewayapp.symbols must be applied after an Android application/library plugin`. The app module already has the Android plugin line, so add the Traceway line under it rather than pasting a new block above it.+`uploadToken` is your project's **upload token** from the dashboard, the same one used for JavaScript source maps and iOS dSYMs. It is **not** the runtime token in your SDK connection string. Using the connection-string token here is rejected with a 401.With `autoUpload = true` the upload runs automatically after `assembleRelease`. The default is `false` (so a release build never depends on backend availability); upload explicitly instead, which is the better fit for CI:diff --git a/docs/pages/symbolicator/index.mdx b/docs/pages/symbolicator/index.mdxindex 18fd3109..3828ac32 100644--- a/docs/pages/symbolicator/index.mdx+++ b/docs/pages/symbolicator/index.mdx@@ -67,10 +67,10 @@ Raw artifacts are fetched on demand and never kept on local disk by the engine.## Where to go next-- **[JavaScript](/symbolicator/javascript)** — source maps and bundle scope analysis.-- **[Dart](/symbolicator/dart)** — stripped Flutter traces and DWARF.-- **[iOS](/symbolicator/ios)** — stripped Swift/ObjC release crashes and dSYM/DWARF.-- **[Android](/symbolicator/android)** — obfuscated R8 release crashes and `mapping.txt` retrace.-- **[Performance](/symbolicator/performance)** — memory vs disk cache, sizing, benchmarks, and the metrics to watch.-- **[OpenTelemetry](/symbolicator/opentelemetry)** — running the engine as a collector processor.-- **[Library](/symbolicator/library)** — calling the engine from your own Go code.+- **[JavaScript](/symbolicator/javascript)**: source maps and bundle scope analysis.+- **[Dart](/symbolicator/dart)**: stripped Flutter traces and DWARF.+- **[iOS](/symbolicator/ios)**: stripped Swift/ObjC release crashes and dSYM/DWARF.+- **[Android](/symbolicator/android)**: obfuscated R8 release crashes and `mapping.txt` retrace.+- **[Performance](/symbolicator/performance)**: memory vs disk cache, sizing, benchmarks, and the metrics to watch.+- **[OpenTelemetry](/symbolicator/opentelemetry)**: running the engine as a collector processor.+- **[Library](/symbolicator/library)**: calling the engine from your own Go code.diff --git a/docs/pages/symbolicator/ios.mdx b/docs/pages/symbolicator/ios.mdxindex 1d1f4251..a281e69d 100644--- a/docs/pages/symbolicator/ios.mdx+++ b/docs/pages/symbolicator/ios.mdx@@ -72,7 +72,7 @@ Unlike the Dart upload, iOS needs no `arch` or `debug_id` form fields. The backeiossymbols/{projectId}/{uuid}.dsym```-Because the key is content-addressed, releases never collide and re-uploading the same build is idempotent. The response reports how many uploaded files were accepted — a fat dSYM counts once, even though it stores one entry per slice:+Because the key is content-addressed, releases never collide and re-uploading the same build is idempotent. The response reports how many uploaded files were accepted. A fat dSYM counts once, even though it stores one entry per slice:```json{ "uploaded": 2 }@@ -98,7 +98,7 @@ The parser pulls three things from each frame:A trace that is already symbolic, or carries no recognizable iOS frames, passes through untouched.-When crafting traces by hand (a custom sender, CI tooling), include the `*** …` separator line between the error message and the frames, as the Traceway iOS SDK does. The symbolicated output preserves everything above the first `***` line as the error preamble — without a separator, the entire raw trace (frame lines included) is treated as preamble and carried into the resolved output verbatim, duplicating the frames.+When crafting traces by hand (a custom sender, CI tooling), include the `*** …` separator line between the error message and the frames, as the Traceway iOS SDK does. The symbolicated output preserves everything above the first `***` line as the error preamble. Without a separator, the entire raw trace (frame lines included) is treated as preamble and carried into the resolved output verbatim, duplicating the frames.## How an offset resolvesdiff --git a/docs/public/_redirects b/docs/public/_redirectsnew file mode 100644index 00000000..4a5afd4c--- /dev/null+++ b/docs/public/_redirects@@ -0,0 +1,44 @@+# OTel-based backend guides moved under /client/otel when the backend docs were+# consolidated onto one OpenTelemetry path. Keep these until the old URLs stop+# showing up in referrers.+/client/node-sdk /client/otel/nodejs 301+# node-sdk subpages with no 1:1 twin under /client/otel/nodejs fall back to the+# framework index (first match wins, so these must precede the splat).+/client/node-sdk/exceptions /client/otel/nodejs 301+/client/node-sdk/initialization /client/otel/nodejs 301+/client/node-sdk/spans /client/otel/nodejs 301+/client/node-sdk/tasks /client/otel/nodejs 301+/client/node-sdk/* /client/otel/nodejs/:splat 301+/client/nestjs /client/otel/nestjs 301+# Retired nestjs subpages (decorators/filter/middleware/module/service) fall+# back to the framework index; only custom-instrumentation still exists.+/client/nestjs/custom-instrumentation /client/otel/nestjs/custom-instrumentation 301+/client/nestjs/* /client/otel/nestjs 301+# nextjs has no subpages under /client/otel, so everything lands on its index.+/client/nextjs /client/otel/nextjs 301+/client/nextjs/* /client/otel/nextjs 301+/client/hono /client/otel/hono 301+/client/hono/* /client/otel/hono/:splat 301+/client/cloudflare /client/otel/cloudflare 301+/client/cloudflare/* /client/otel/cloudflare/:splat 301+/client/symfony /client/otel/symfony 301+/client/symfony/* /client/otel/symfony/:splat 301+/client/laravel /client/otel/laravel 301+/client/laravel/* /client/otel/laravel/:splat 301+/client/django /client/otel/django 301+/client/django/* /client/otel/django/:splat 301++# The native Go SDK was retired. OpenTelemetry is the one Go integration path,+# so every Go SDK page now lands on the OpenTelemetry overview.+/client/sdk /client/otel 301+/client/sdk/* /client/otel 301+/client/gin-middleware /client/otel 301+/client/gin-middleware/* /client/otel 301+/client/chi-middleware /client/otel 301+/client/chi-middleware/* /client/otel 301+/client/fiber-middleware /client/otel 301+/client/fiber-middleware/* /client/otel 301+/client/fasthttp-middleware /client/otel 301+/client/fasthttp-middleware/* /client/otel 301+/client/http-middleware /client/otel 301+/client/http-middleware/* /client/otel 301diff --git a/docs/styles/custom.css b/docs/styles/custom.cssindex 0dfe164b..b2acfdbc 100644--- a/docs/styles/custom.css+++ b/docs/styles/custom.css@@ -1,43 +1,47 @@/* =========================================================================- Traceway Docs: dark-only design tokens ported from website design-guidelines- (styles.css). Override Nextra's default theme colors and apply mono-native- typography.+ Traceway Docs: dark-only design tokens matching the dashboard's dark mode+ (frontend/src/routes/layout.css). Override Nextra's default theme colors+ and apply mono-native typography.========================================================================= */:root,:is(html[class~="dark"]) {- /* Ink & surfaces */- --ink-0: #05070c;- --ink-1: #0a0d14;- --ink-2: #0f131c;- --ink-3: #151a24;- --ink-4: #1d2430;- --ink-5: #283041;- --hair: rgba(255, 255, 255, 0.06);- --hair-2: rgba(255, 255, 255, 0.1);-- /* Text */- --fg-0: #f4f6fb;- --fg-1: #c9d0dd;- --fg-2: #8a93a6;- --fg-3: #5a6374;-- /* Accent: Electric Indigo */- --a1: #7c5cff;- --a2: #00d4ff;- --a3: #ff4d8d;- --a4: #ffd166;- --ok: #22e0a8;- --crit: #ff5a5f;-- /* Nextra theme overrides */- --nextra-primary-hue: 253deg;- --nextra-primary-saturation: 100%;- --nextra-bg: 5, 7, 12; /* ink-0 triplet */-- /* Radius + layout */- --radius: 6px;- --radius-lg: 10px;+ /* Ink & surfaces: pure black page ground (website style), dashboard navy+ --ink-* scale for raised surfaces */+ --ink-0: #000;+ --ink-1: #0f1220;+ --ink-2: #151a2c;+ --ink-3: #1b2138;+ --ink-4: #252c47;+ --ink-5: #313a5b;+ --hair: rgba(255, 255, 255, 0.08);+ --hair-2: rgba(255, 255, 255, 0.12);++ /* Text (dashboard --fg-* scale) */+ --fg-0: #f4f6fc;+ --fg-1: #cbd2e4;+ --fg-2: #929bb4;+ --fg-3: #5f6780;++ /* Accent: dashboard blue family */+ --blue: oklch(0.623 0.214 259.815);+ --blue-bright: oklch(0.707 0.165 254.624);+ --blue-deep: oklch(0.546 0.245 262.881);+ --a1: var(--blue-deep);+ --a2: var(--blue-bright);+ --a3: oklch(0.656 0.241 354.308); /* pink */+ --a4: oklch(0.769 0.188 70.08); /* amber */+ --ok: #22c08a;+ --crit: #f43f5e;++ /* Nextra theme overrides: blue-deep ≈ hsl(221, 83%, 53%) */+ --nextra-primary-hue: 221deg;+ --nextra-primary-saturation: 83%;+ --nextra-bg: 0, 0, 0; /* ink-0 triplet */++ /* Radius + layout (dashboard: 0.75rem base, 8px controls) */+ --radius: 8px;+ --radius-lg: 12px;--max: 1240px;/* Font aliases mirroring website var names@@ -77,8 +81,23 @@ body {font-family: var(--f-body);}+/* Dashboard's ambient radial glows (blue top-right, cyan left), fixed to the+ viewport so long pages don't shift the gradient while scrolling. */body {- background: var(--ink-0) !important;+ background:+ radial-gradient(+ 1100px 600px at 88% -12%,+ oklch(0.623 0.214 259.815 / 10%),+ transparent 60%+ )+ fixed,+ radial-gradient(+ 800px 500px at -8% 15%,+ oklch(0.715 0.143 215.221 / 5%),+ transparent 60%+ )+ fixed,+ var(--ink-0) !important;}main h1,@@ -159,7 +178,7 @@ main a,main a:hover,.nextra-content a:hover {- color: var(--a1) !important;+ color: color-mix(in oklab, var(--a2) 70%, white) !important;}/* =========================================================================@@ -170,8 +189,23 @@ main a:hover,border-bottom: 1px solid var(--hair);}+/* Same fixed-attachment gradient as body so the opaque navbar blends into the+ page's top-right glow instead of cutting it off. */html[class~="dark"] .nextra-nav-container-blur.nextra-nav-container-blur {- background: #000 !important;+ background:+ radial-gradient(+ 1100px 600px at 88% -12%,+ oklch(0.623 0.214 259.815 / 10%),+ transparent 60%+ )+ fixed,+ radial-gradient(+ 800px 500px at -8% 15%,+ oklch(0.715 0.143 215.221 / 5%),+ transparent 60%+ )+ fixed,+ var(--ink-0) !important;-webkit-backdrop-filter: none !important;backdrop-filter: none !important;box-shadow: none !important;@@ -182,6 +216,16 @@ html[class~="dark"] .nextra-nav-container-blur.nextra-nav-container-blur {border-right: 1px solid var(--hair);}+/* Nextra masks the sidebar with a 20px top fade (scrolling items dissolve+ near the top edge). Our first item is a filled blue pill, so the fade cuts+ visibly through it and reads as a stray gradient. Drop the mask; scrolled+ items simply clip at the edge instead. */+.nextra-sidebar-container,+.nextra-sidebar-container > .nextra-scrollbar {+ -webkit-mask-image: none !important;+ mask-image: none !important;+}+/* Sidebar links */.nextra-sidebar-container a,.nextra-sidebar-container button {@@ -196,12 +240,17 @@ html[class~="dark"] .nextra-nav-container-blur.nextra-nav-container-blur {background: var(--ink-2) !important;}-.nextra-sidebar-container a.nx-bg-primary-100,-.nextra-sidebar-container .nx-text-primary-600,-.nextra-sidebar-container a[aria-current="page"] {+.nextra-sidebar-container .nx-text-primary-600 {color: var(--fg-0) !important;background: transparent !important;- border-color: var(--fg-0) !important;+}++/* Active page: filled blue pill, matching the dashboard sidebar. */+.nextra-sidebar-container a.nx-bg-primary-100,+.nextra-sidebar-container a[aria-current="page"] {+ color: #fff !important;+ background: var(--blue-deep) !important;+ border-color: transparent !important;}/* Expanded top-level folder containing the active page: Nextra fills it with the@@ -223,12 +272,19 @@ nav.nextra-toc .nx-text-primary-600 {color: var(--a2) !important;}-/* Footer */-.nextra-footer,-footer.nextra-footer {+/* Footer: Nextra 2's footer carries no .nextra-footer class. It ships as+ footer.nx-bg-gray-100 with dark:nx-bg-neutral-900 (#171717). Let the black+ page show through and dim its separator + text to our tokens. */+footer.nx-bg-gray-100 {background: transparent !important;- border-top: 1px solid var(--hair);- color: var(--fg-3);+}++footer.nx-bg-gray-100 hr {+ border-color: var(--hair) !important;+}++footer.nx-bg-gray-100 .nx-text-gray-600 {+ color: var(--fg-3) !important;}/* =========================================================================@@ -288,8 +344,11 @@ main td,main blockquote,.nextra-content blockquote {background: color-mix(in oklab, var(--a2) 8%, transparent);- border-left: 3px solid var(--a2);+ border-left: 2px solid color-mix(in oklab, var(--a2) 70%, transparent) !important;+ border-radius: 0 8px 8px 0;color: var(--fg-1);+ font-style: normal !important;+ padding: 12px 18px !important;}/* Nextra callouts */@@ -506,11 +565,11 @@ main hr,.btn-accent {background: var(--a1);- color: #0a0612 !important;+ color: #fff !important;}.btn-accent:hover {- background: color-mix(in oklab, var(--a1) 88%, white);- color: #0a0612 !important;+ background: color-mix(in oklab, var(--a1) 85%, white);+ color: #fff !important;}.btn-ghost {@@ -656,27 +715,111 @@ li:has(> .sdk-selector) > span:first-child {font-family: var(--font-mono), ui-monospace, monospace;}+/* =========================================================================+ Agent skills strip (/client)+ ========================================================================= */++.agent-skills {+ display: flex;+ align-items: center;+ justify-content: space-between;+ gap: 20px;+ margin-top: 28px;+ padding: 12px 16px;+ border: 1px solid var(--hair);+ border-radius: 10px;+ background: var(--ink-1);+}++.agent-skills-text {+ margin: 0;+ font-size: 13.5px;+ line-height: 1.5;+ color: var(--fg-2) !important;+ min-width: 0;+}++.agent-skills-cmd {+ flex: none;+ display: flex;+ align-items: center;+ gap: 10px;+ padding: 10px 12px;+ font-family: var(--f-mono);+ font-size: 13px;+ color: var(--fg-0);+ background: var(--ink-0);+ border: 1px solid var(--hair-2);+ border-radius: 8px;+ cursor: pointer;+ transition: border-color 0.15s;+}++.agent-skills-cmd:hover {+ border-color: color-mix(in oklab, var(--a2) 40%, var(--hair-2));+}++.agent-skills-cmd:focus-visible {+ outline: 2px solid color-mix(in oklab, var(--a2) 60%, transparent);+ outline-offset: 2px;+}++.agent-skills-prompt {+ color: var(--a2);+ user-select: none;+}++.agent-skills-cmd-icon {+ color: var(--fg-3);+ transition: color 0.15s;+}++.agent-skills-cmd:hover .agent-skills-cmd-icon {+ color: var(--fg-1);+}++.agent-skills-cmd-icon--ok {+ color: var(--ok);+}++.agent-skills-cmd:hover .agent-skills-cmd-icon--ok {+ color: var(--ok);+}++@media (max-width: 860px) {+ .agent-skills {+ flex-direction: column;+ align-items: flex-start;+ }++ .agent-skills-cmd {+ max-width: 100%;+ }++ .agent-skills-cmd-text {+ overflow: hidden;+ text-overflow: ellipsis;+ white-space: nowrap;+ }+}+/* =========================================================================Framework Picker (card grid on /client)========================================================================= */.framework-picker {- margin-top: 24px;+ margin-top: 40px;}-.framework-picker-heading {- font-size: 20px;+/* Doubled class specificity so this beats the global+ `.nextra-content h2 { !important }` heading sizes. */+.framework-picker-heading.framework-picker-heading {+ font-size: 28px !important;+ line-height: 1.2 !important;+ letter-spacing: -0.02em !important;font-weight: 600;- margin: 0 0 8px 0;+ margin: 0 0 20px 0;color: var(--fg-0);- font-family: var(--f-body);- letter-spacing: -0.01em;-}--.framework-picker-subheading {- font-size: 15px;- color: var(--fg-2);- margin: 0 0 24px 0;}.framework-picker-grid {@@ -702,17 +845,22 @@ li:has(> .sdk-selector) > span:first-child {background: var(--ink-1);cursor: pointer;transition:- border-color 0.2s,- background 0.2s;+ border-color 0.15s,+ background 0.15s;text-align: left;font-family: inherit;}.framework-picker-card:hover {- border-color: var(--hair-2);+ border-color: color-mix(in oklab, var(--a2) 30%, transparent);background: var(--ink-2);}+.framework-picker-card:focus-visible {+ outline: 2px solid color-mix(in oklab, var(--a2) 60%, transparent);+ outline-offset: 2px;+}+.framework-picker-icon {width: 28px;height: 28px;@@ -878,5 +1026,5 @@ nav.nextra-toc .nx-sticky.nx-bottom-0,.nextra-sidebar-container .nx-sticky.nx-bottom-0 {background: var(--ink-0) !important;box-shadow: none !important;- border-top: 1px solid rgba(255, 255, 255, 0.06) !important;+ border-top: 1px solid var(--hair) !important;}diff --git a/docs/theme.config.jsx b/docs/theme.config.jsxindex 892d2d6f..2f6c4180 100644--- a/docs/theme.config.jsx+++ b/docs/theme.config.jsx@@ -3,16 +3,10 @@ import SdkSelector from "./components/SdkSelector";import HiddenItem from "./components/HiddenItem";import { useSdk } from "./components/SdkContext";+// Keyed by the first path segment after /client/. Every OTel-based backend guide+// now lives under /client/otel/*, so the whole backend story is one `otel` entry.const SDK_VISIBILITY = {- "gin-middleware": "go-gin",- "chi-middleware": "go-chi",- "fiber-middleware": "go-fiber",- "fasthttp-middleware": "go-fasthttp",- "http-middleware": "go-http",- sdk: "go-",- "node-sdk": "js-node",- nestjs: "js-nestjs",- hono: "js-hono",+ otel: "otel",react: "js-react","react-native": "react-native",vue: "js-vue",@@ -20,12 +14,6 @@ const SDK_VISIBILITY = {jquery: "js-jquery","js-sdk": ["js-react", "js-vue", "js-svelte", "js-jquery", "js-generic"],openrouter: "openrouter",- otel: "otel",- cloudflare: "cloudflare",- nextjs: "js-nextjs",- symfony: "php-symfony",- laravel: "php-laravel",- django: "python-django",flutter: "flutter",android: "android",ios: "ios",@@ -64,16 +52,16 @@ export default {name="description"content="Traceway - Error tracking and monitoring platform"/>- <meta name="theme-color" content="#05070c" />+ <meta name="theme-color" content="#000000" /><meta name="color-scheme" content="dark" /><link rel="icon" type="image/svg+xml" href="/favicon.svg" /><link rel="alternate icon" href="/favicon.ico" sizes="any" /><link rel="apple-touch-icon" href="/apple-touch-icon.png" /></>),- // Electric Indigo (#7c5cff) ≈ hsl(253, 100%, 68%)- primaryHue: 253,- primarySaturation: 100,+ // Dashboard blue (oklch(0.546 0.245 262.881) ≈ #2563eb) ≈ hsl(221, 83%, 53%)+ primaryHue: 221,+ primarySaturation: 83,darkMode: false,nextThemes: {defaultTheme: "dark",diff --git a/frontend/src/lib/components/add-project-modal.svelte b/frontend/src/lib/components/add-project-modal.svelteindex 98a4b0cb..d6b2e625 100644--- a/frontend/src/lib/components/add-project-modal.svelte+++ b/frontend/src/lib/components/add-project-modal.svelte@@ -21,7 +21,7 @@let { open, onOpenChange, onProjectCreated }: Props = $props();let projectName = $state('');- let selectedFramework = $state<Framework>('gin');+ let selectedFramework = $state<Framework>('opentelemetry');let loading = $state(false);let error = $state('');let createdProject = $state<ProjectWithToken | null>(null);@@ -68,7 +68,7 @@function handleClose() {projectName = '';- selectedFramework = 'gin';+ selectedFramework = 'opentelemetry';error = '';createdProject = null;selectedOrgId = null;diff --git a/frontend/src/lib/components/framework-combobox.svelte b/frontend/src/lib/components/framework-combobox.svelteindex e0c0c01b..a2c84d53 100644--- a/frontend/src/lib/components/framework-combobox.svelte+++ b/frontend/src/lib/components/framework-combobox.svelte@@ -20,27 +20,18 @@let triggerWrapperRef = $state<HTMLDivElement | null>(null);let contentRef = $state<HTMLElement | null>(null);+ const GROUP_ORDER = ['Backend', 'Browser', 'Mobile'] as const;+const frameworks = [- { value: 'gin', label: 'Gin', description: 'Fast HTTP web framework', group: 'Go' },- { value: 'fiber', label: 'Fiber', description: 'Express-inspired framework', group: 'Go' },- { value: 'chi', label: 'Chi', description: 'Lightweight router', group: 'Go' },- { value: 'fasthttp', label: 'FastHTTP', description: 'High-performance HTTP', group: 'Go' },- { value: 'stdlib', label: 'Standard Library', description: 'net/http package', group: 'Go' },- { value: 'custom', label: 'Custom', description: 'Other / manual setup', group: 'Go' },- { value: 'react', label: 'React', description: 'Frontend UI library', group: 'JavaScript' },- { value: 'react-native', label: 'React Native', description: 'React Native and Expo apps', group: 'JavaScript' },- { value: 'svelte', label: 'Svelte', description: 'Frontend compiler framework', group: 'JavaScript' },- { value: 'vuejs', label: 'Vue.js', description: 'Progressive frontend framework', group: 'JavaScript' },- { value: 'jquery', label: 'jQuery', description: 'Legacy-friendly AJAX and DOM library', group: 'JavaScript' },- { value: 'symfony', label: 'Symfony', description: 'PHP full-stack framework (OTLP)', group: 'PHP' },- { value: 'laravel', label: 'Laravel', description: 'PHP full-stack framework (OTLP)', group: 'PHP' },- { value: 'django', label: 'Django', description: 'Python full-stack framework (OTLP)', group: 'Python' },- { value: 'hono', label: 'Hono', description: 'Lightweight multi-runtime framework (OTLP)', group: 'Backend' },- { value: 'cloudflare', label: 'Cloudflare', description: 'Cloudflare Workers (OTLP Traces)', group: 'Backend' },- { value: 'opentelemetry', label: 'OpenTelemetry', description: 'Vendor-neutral observability', group: 'Backend' },- { value: 'flutter', label: 'Flutter', description: 'Mobile UI framework', group: 'Mobile' },- { value: 'android', label: 'Android', description: 'Native Android (Kotlin/Java)', group: 'Mobile' },- { value: 'ios', label: 'iOS', description: 'Native iOS (Swift/Objective-C)', group: 'Mobile' },+ { value: 'opentelemetry', label: 'OpenTelemetry', description: 'Any backend: Go, Node.js, Python, PHP, Java, .NET, Ruby', group: 'Backend', keywords: 'otel opentelemetry go golang gin fiber chi fasthttp echo node nodejs express nestjs fastify koa hono cloudflare workers python django flask fastapi php symfony laravel slim java spring dotnet .net csharp ruby rails api server backend' },+ { value: 'react', label: 'React', description: 'React apps, including Next.js and Remix', group: 'Browser', keywords: 'react nextjs next.js remix browser frontend spa' },+ { value: 'svelte', label: 'Svelte', description: 'Svelte and SvelteKit apps', group: 'Browser', keywords: 'svelte sveltekit browser frontend' },+ { value: 'vuejs', label: 'Vue.js', description: 'Vue and Nuxt apps', group: 'Browser', keywords: 'vue vuejs nuxt browser frontend' },+ { value: 'jquery', label: 'jQuery', description: 'Legacy-friendly AJAX and DOM library', group: 'Browser', keywords: 'jquery browser frontend legacy' },+ { value: 'flutter', label: 'Flutter', description: 'Flutter apps on Android and iOS', group: 'Mobile', keywords: 'flutter dart mobile android ios' },+ { value: 'react-native', label: 'React Native', description: 'React Native and Expo apps', group: 'Mobile', keywords: 'react native expo mobile' },+ { value: 'android', label: 'Android', description: 'Native Android (Kotlin/Java)', group: 'Mobile', keywords: 'android kotlin java mobile native' },+ { value: 'ios', label: 'iOS', description: 'Native iOS (Swift/Objective-C)', group: 'Mobile', keywords: 'ios swift swiftui objective-c apple mobile native' },] as const;const selectedFrameworkLabel = $derived(getFrameworkLabel(value));@@ -53,16 +44,16 @@: frameworks.filter(f =>f.label.toLowerCase().includes(query) ||f.description.toLowerCase().includes(query) ||- f.group.toLowerCase().includes(query)+ f.group.toLowerCase().includes(query) ||+ f.keywords.includes(query)));- const goFrameworks = $derived(filteredFrameworks.filter(f => f.group === 'Go'));- const jsFrameworks = $derived(filteredFrameworks.filter(f => f.group === 'JavaScript'));- const phpFrameworks = $derived(filteredFrameworks.filter(f => f.group === 'PHP'));- const pythonFrameworks = $derived(filteredFrameworks.filter(f => f.group === 'Python'));- const backendFrameworks = $derived(filteredFrameworks.filter(f => f.group === 'Backend'));- const mobileFrameworks = $derived(filteredFrameworks.filter(f => f.group === 'Mobile'));+ const groups = $derived(+ GROUP_ORDER+ .map(name => ({ name, items: filteredFrameworks.filter(f => f.group === name) }))+ .filter(g => g.items.length > 0)+ );function handleOpenChange(isOpen: boolean) {open = isOpen;@@ -128,140 +119,10 @@{#if filteredFrameworks.length === 0}<div class="py-4 text-center text-sm text-muted-foreground">No frameworks found</div>{:else}- {#if goFrameworks.length > 0}- <Combobox.Group>- <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">Go</Combobox.GroupHeading>- {#each goFrameworks as fw}- <Combobox.Item- value={fw.value}- class={cn(- "data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground relative flex w-full cursor-default items-center gap-2 rounded-sm py-1.5 ps-2 pe-8 text-sm outline-hidden select-none data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"- )}- >- {#snippet children({ selected })}- <div class="flex items-center gap-2">- <FrameworkIcon framework={fw.value} />- <div class="flex flex-col">- <span class="font-medium">{fw.label}</span>- <span class="text-xs text-muted-foreground">{fw.description}</span>- </div>- </div>- {#if selected}- <Check class="absolute end-2 size-4" />- {/if}- {/snippet}- </Combobox.Item>- {/each}- </Combobox.Group>- {/if}- {#if jsFrameworks.length > 0}- <Combobox.Group>- <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">JavaScript</Combobox.GroupHeading>- {#each jsFrameworks as fw}- <Combobox.Item- value={fw.value}- class={cn(- "data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground relative flex w-full cursor-default items-center gap-2 rounded-sm py-1.5 ps-2 pe-8 text-sm outline-hidden select-none data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"- )}- >- {#snippet children({ selected })}- <div class="flex items-center gap-2">- <FrameworkIcon framework={fw.value} />- <div class="flex flex-col">- <span class="font-medium">{fw.label}</span>- <span class="text-xs text-muted-foreground">{fw.description}</span>- </div>- </div>- {#if selected}- <Check class="absolute end-2 size-4" />- {/if}- {/snippet}- </Combobox.Item>- {/each}- </Combobox.Group>- {/if}- {#if phpFrameworks.length > 0}- <Combobox.Group>- <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">PHP</Combobox.GroupHeading>- {#each phpFrameworks as fw}- <Combobox.Item- value={fw.value}- class={cn(- "data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground relative flex w-full cursor-default items-center gap-2 rounded-sm py-1.5 ps-2 pe-8 text-sm outline-hidden select-none data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"- )}- >- {#snippet children({ selected })}- <div class="flex items-center gap-2">- <FrameworkIcon framework={fw.value} />- <div class="flex flex-col">- <span class="font-medium">{fw.label}</span>- <span class="text-xs text-muted-foreground">{fw.description}</span>- </div>- </div>- {#if selected}- <Check class="absolute end-2 size-4" />- {/if}- {/snippet}- </Combobox.Item>- {/each}- </Combobox.Group>- {/if}- {#if pythonFrameworks.length > 0}- <Combobox.Group>- <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">Python</Combobox.GroupHeading>- {#each pythonFrameworks as fw}- <Combobox.Item- value={fw.value}- class={cn(- "data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground relative flex w-full cursor-default items-center gap-2 rounded-sm py-1.5 ps-2 pe-8 text-sm outline-hidden select-none data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"- )}- >- {#snippet children({ selected })}- <div class="flex items-center gap-2">- <FrameworkIcon framework={fw.value} />- <div class="flex flex-col">- <span class="font-medium">{fw.label}</span>- <span class="text-xs text-muted-foreground">{fw.description}</span>- </div>- </div>- {#if selected}- <Check class="absolute end-2 size-4" />- {/if}- {/snippet}- </Combobox.Item>- {/each}- </Combobox.Group>- {/if}- {#if backendFrameworks.length > 0}- <Combobox.Group>- <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">Backend</Combobox.GroupHeading>- {#each backendFrameworks as fw}- <Combobox.Item- value={fw.value}- class={cn(- "data-[highlighted]:bg-accent data-[highlighted]:text-accent-foreground [&_svg:not([class*='text-'])]:text-muted-foreground relative flex w-full cursor-default items-center gap-2 rounded-sm py-1.5 ps-2 pe-8 text-sm outline-hidden select-none data-[disabled]:pointer-events-none data-[disabled]:opacity-50 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4"- )}- >- {#snippet children({ selected })}- <div class="flex items-center gap-2">- <FrameworkIcon framework={fw.value} />- <div class="flex flex-col">- <span class="font-medium">{fw.label}</span>- <span class="text-xs text-muted-foreground">{fw.description}</span>- </div>- </div>- {#if selected}- <Check class="absolute end-2 size-4" />- {/if}- {/snippet}- </Combobox.Item>- {/each}- </Combobox.Group>- {/if}- {#if mobileFrameworks.length > 0}+ {#each groups as group (group.name)}<Combobox.Group>- <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">Mobile</Combobox.GroupHeading>- {#each mobileFrameworks as fw}+ <Combobox.GroupHeading class="px-2 py-1.5 text-xs font-semibold text-muted-foreground">{group.name}</Combobox.GroupHeading>+ {#each group.items as fw (fw.value)}<Combobox.Itemvalue={fw.value}class={cn(@@ -283,7 +144,7 @@</Combobox.Item>{/each}</Combobox.Group>- {/if}+ {/each}{/if}</Combobox.Viewport><SelectScrollDownButton />diff --git a/frontend/src/routes/finish-setup/+page.svelte b/frontend/src/routes/finish-setup/+page.svelteindex 31d7e9d1..1abeef5d 100644--- a/frontend/src/routes/finish-setup/+page.svelte+++ b/frontend/src/routes/finish-setup/+page.svelte@@ -15,7 +15,7 @@import FrameworkCombobox from '$lib/components/framework-combobox.svelte';import { consumeSsoReturnTo, safeLocalPath } from '$lib/utils/navigation';- const DEFAULT_FRAMEWORK: Framework = 'gin';+ const DEFAULT_FRAMEWORK: Framework = 'opentelemetry';let organizationName = $state('');let timezone = $state(Intl.DateTimeFormat().resolvedOptions().timeZone);diff --git a/frontend/src/routes/register/+page.svelte b/frontend/src/routes/register/+page.svelteindex 13e7b553..69e8c459 100644--- a/frontend/src/routes/register/+page.svelte+++ b/frontend/src/routes/register/+page.svelte@@ -17,7 +17,7 @@import TurnstileWidget from '$lib/components/turnstile-widget.svelte';import OauthButtons from '$lib/components/oauth-buttons.svelte';- const DEFAULT_FRAMEWORK: Framework = 'gin';+ const DEFAULT_FRAMEWORK: Framework = 'opentelemetry';function parseFrameworkParam(value: string | null): Framework {if (!value) return DEFAULT_FRAMEWORK;diff --git a/skills/traceway-setup/SKILL.md b/skills/traceway-setup/SKILL.mdindex 11807088..26314bf3 100644--- a/skills/traceway-setup/SKILL.md+++ b/skills/traceway-setup/SKILL.md@@ -1,86 +1,152 @@---name: traceway-setup-description: Connect a project to a Traceway instance so it reports endpoints, spans, errors, background tasks, AI traces, and metrics. Backends use OpenTelemetry over OTLP/HTTP, frontends and mobile apps (including native iOS/Swift) use the Traceway SDKs, and host metrics use the Traceway OTel Agent. AI agents and chatbots additionally get conversation grouping, per-user analytics, and tool-call instrumentation (see ai-agent.md). Use when the user wants to add Traceway (or OpenTelemetry tracing that exports to Traceway) to a backend, frontend, full-stack, mobile, iOS/Swift, or AI agent/chatbot project. Accepts a project token and instance URL, e.g. "/traceway-setup with token abc123".+description: Analyze a repository's backend, browser, mobile, AI, background-work, existing-observability, build, and deployment architecture; propose and explain the correct Traceway project topology; guide the user through creating projects and credentials in the Traceway dashboard; then instrument and verify every selected component. Every backend uses OpenTelemetry over OTLP/HTTP regardless of language or framework, keeping endpoints, tasks, AI traces, logs, application metrics, and host metrics in one backend project. Browser frontends and independently released mobile apps use separate Traceway projects with the Traceway SDKs, plus source map or symbol uploads where applicable. Use when the user wants to plan, add, migrate, or complete Traceway or OpenTelemetry monitoring for a backend, frontend, full-stack, mobile, iOS/Swift, or AI agent/chatbot repository. Accepts a project token and instance URL in the invocation, e.g. "/traceway-setup with token abc123 and url https://traceway.example.com".----# Set Up Traceway in a Project+# Set Up Traceway-Connect an existing project to a Traceway instance so it reports endpoints, spans, errors, background tasks, AI traces, and metrics.+Analyze the application first, agree on its Traceway project structure with the user, help them create that structure in the dashboard, and only then integrate and verify it.-## Step 0: Gather Connection Info+## Operating Contract-| Value | Example | Where to find it |+- Inspect before asking for credentials or changing code.+- Explain what was detected, what is already tracked, and how each production component should report to Traceway.+- Present a proposed project map and wait for the user to confirm it before implementation.+- Never print token values found in files or the environment. Report only that a value exists and the variable name that carries it.+- Do not require the user to paste secrets into chat. Implement against environment variables and let the user set real values in their local, deployment, or CI secret store. Real values are required only for live verification.+- Never commit tokens or filled connection strings.++**Every backend integrates with OpenTelemetry.** There is one backend path, not one per language or web framework: Go, Node, Python, PHP, Java, .NET, Ruby, and everything else export OTLP/HTTP to `<instance>/api/otel/*`. The Traceway project is created with framework **OpenTelemetry**, which is the only backend option in the dashboard's framework picker. The native Traceway Go SDK is a deliberate exception used only when the user explicitly asks for it.++## Fast Path++When all of the following hold, skip Steps 2 and 3 and go straight to the integration steps:++- the repository has exactly one deployable component, and+- the user supplied (or the environment already carries) an instance URL and a project token, and+- the token's project already matches that component.++Still run Step 1, because it decides *how* to instrument. Collapse "Propose and Confirm the Project Map" to a single confirmation line ("This is a Go API; it reports to your existing `<name>` project over OpenTelemetry") and skip the dashboard walkthrough entirely. The full map-and-dashboard ceremony exists for repositories with more than one deployable component, or where no project exists yet. Do not make a user who handed you a token sit through a project-map review for a single service.++## Step 1: Analyze the Architecture++Before changing anything, build a picture of what is deployed and what is already instrumented:++1. **Frameworks and languages**: detect them from `package.json`, `go.mod`, `composer.json`, `requirements.txt`/`pyproject.toml`, `pubspec.yaml`, `build.gradle`(`.kts`), `Package.swift`, `*.xcodeproj`/`*.xcworkspace`, `Podfile`, and source extensions. For iOS/Apple targets, note whether the sources are Swift or Objective-C. That choice picks the path in "Frontend and Mobile" (Swift gets the native SDK; Objective-C-only has none and falls back to OTel).+2. **Deployable components and entry points**: inventory APIs, SSR servers, browser apps, workers, schedulers, CLIs, and mobile applications. Treat code organization as evidence, not proof that something is deployed.+3. **Existing observability**: find OpenTelemetry configuration, Traceway SDKs, Sentry, Datadog, New Relic, Honeycomb, logging exporters, tracing middleware, source map or symbol upload steps, and observability environment-variable names. Explain whether Traceway will replace, coexist with, or extend each integration. Never display discovered credential values.+4. **Production role of JS meta-frameworks** (Next.js, SvelteKit, Remix, Nuxt): never assume full-stack.+ - **Frontend-only signals**: `output: 'export'` in `next.config.*` (static export, so there is no server in production); no API routes (`app/api/**/route.*`, `pages/api/*`) or only trivial ones; no `'use server'` server actions; `rewrites`/proxy config or a `NEXT_PUBLIC_API_URL`-style env var pointing the browser at an external API; a separate backend service in this repo, another repo, or another language; static hosting in the deploy config (S3/CloudFront, GitHub Pages, nginx serving `out/`).+ - **Server signals**: API routes with real logic, `'use server'` server actions, database clients imported in server code, SSR reading its own data layer, `next start` or standalone output in the deploy config.+ - **Mixed or unclear**: ask how the application is deployed and whether the framework server does production work worth tracking.++ Frontend-only means integrate ONLY the browser side with the frontend SDK; do NOT add OTel to, or otherwise instrument, the framework's server side. A separate backend is its own component with its own Traceway project.+5. **Background work**: find cron jobs, queue consumers, schedulers, CLI commands, and long-running workers. Record whether libraries already emit `CONSUMER` spans.+6. **AI/LLM usage**: check dependencies for `openai`, `@anthropic-ai/sdk`, `anthropic`, `langchain` / `@langchain/*`, `ai` (Vercel AI SDK), `litellm`, `google-generativeai`, `cohere`, `openrouter`, and agent frameworks (`langgraph`, `crewai`, `autogen` / `ag2`, `pydantic-ai`, `@openai/agents`, `@mastra/*`, `llamaindex` / `llama-index`, `semantic-kernel`). Then decide whether this is a **conversational product** (chatbot, assistant, agent) rather than one-shot LLM calls: look for chat or streaming-chat routes (`/chat`, `/completions`, SSE or websocket handlers feeding an LLM), message-history persistence (a `messages`/`conversations` table or thread ids), tool/function-calling definitions passed to the model, or any agent framework above. If conversational or tool-calling, read `ai-agent.md` in this skill directory before proposing the AI integration.+7. **Browser build and release flow**: identify the bundler, output directory, source-map settings, deploy pipeline, and existing artifact uploads.+8. **Mobile build and release flow**: determine whether Flutter is obfuscated, Android uses R8/minification, iOS produces dSYMs, and whether multiple platform directories are one cross-platform product or independently released apps.+9. **Deployment and host metrics**: inspect Dockerfiles, Compose, Kubernetes, Helm, Terraform, Ansible, cloud-init, PaaS configs, and CI/CD. Determine whether the backend runs on a host where the Traceway OTel Agent is applicable.++Present the findings before asking questions:++| Component | Repository evidence | Production role | Current tracking | Proposed Traceway project | Integration | Credentials needed |+|---|---|---|---|---|---|---|+| `apps/api` | Go module + Gin routes | Backend API | OTel / none / vendor | `Product Backend` | OTel | Runtime token |+| `apps/web` | Svelte + Vite | Browser app | Existing SDK / none | `Product Web` | Traceway Svelte SDK | Runtime + upload token |++Use actual paths and findings. Include unresolved deployment assumptions explicitly.++## Step 2: Propose and Confirm the Project Map++Explain these default boundaries:++| Application boundary | Traceway project | What belongs in it ||---|---|---|-| **Instance URL** | `https://traceway.example.com` | The URL of the Traceway dashboard |-| **Project token** | `abc123...` | Traceway dashboard -> Connection page |-| **Upload token** (optional; frontend source maps, obfuscated Flutter symbols, iOS dSYMs, and Android R8 mappings) | `def456...` | Traceway dashboard -> Connection page -> Source Maps / Symbol Upload |+| **Backend system** | One project with framework **OpenTelemetry** | API endpoints, child spans, issues, background tasks, AI traces, logs, application/runtime metrics, and host metrics. APIs, workers, schedulers, and the host agent share the project token; distinguish them with stable `service.name` values. |+| **Browser frontend** | One separate project per deployed browser application, using **React**, **Svelte**, **Vue.js**, or **jQuery** | Browser errors, web vitals, session replay, distributed-trace linkage, source maps, and browser release metadata. |+| **Mobile app** | One separate project per independently released application, using **Flutter**, **React Native**, **Android**, or **iOS** | Mobile errors/crashes, replay where supported, and build symbols or mappings. A single Flutter product targeting Android and iOS normally uses one project; separate native apps use separate projects. |+| **Full-stack JS app** | Two projects when its server runs in production | Server-side work goes to the OpenTelemetry backend project; browser-side work goes to the browser project. |++The dashboard's framework picker offers exactly these nine options: OpenTelemetry for every backend, React / Svelte / Vue.js / jQuery for browsers, and Flutter / React Native / Android / iOS for mobile. There is no Gin, Django, Laravel, Next.js, or Remix entry, and none is needed. A Next.js or Remix browser project selects **React**, and its server side selects **OpenTelemetry** like any other backend. The framework-specific setup for a backend lives on the project's Connection page, which asks for the language and web framework after the project exists.-Instance URL and project token may be provided in the invocation (e.g. `/traceway-setup with token abc123 and url https://traceway.example.com`). If either is missing:+Do not create one project per backend process by default. Keeping the API, workers, AI calls, and server metrics together preserves their operational context and distributed traces. Split backend projects only for a real product, ownership, access-control, compliance, or data-isolation boundary.-1. Check for existing `TRACEWAY_URL` / `TRACEWAY_TOKEN` environment variables or `.env` entries in the project.-2. Still missing: ask the user whether they already have a Traceway account.- - **Yes**: ask them to open their Traceway dashboard, go to the project's **Connection** page, and paste the instance URL and project token here. If no project exists yet for this app, have them create one first, picking the framework that matches this codebase.- - **No**: send them to the register page to create an account: https://cloud.tracewayapp.com/register (or `https://<their-instance>/register` if they are self-hosting). After registering and creating a project, the Connection page shows the token; ask them to paste the URL and token here.+Ask the user to confirm only what the repository cannot establish safely:-**One Traceway project per part of the application.** Each separately deployed part gets its own Traceway project and token: a browser frontend is one project, its backend API is another (OTel) project, a mobile app a third. Do not funnel two parts through one token; mixing browser and server telemetry in one project muddles the dashboards and the framework-specific connection instructions.+1. Which detected components are deployed and should be tracked?+2. Traceway Cloud or self-hosted, and which organization should own the projects?+3. Are the proposed names and project boundaries correct?+4. Does the backend run on a VM/host, Kubernetes, or serverless/PaaS, and should host metrics be collected?+5. Do mobile directories represent one cross-platform product or separate released apps?-**Analyze every token you receive.** A token is opaque, so establish what it represents: ask the user (or have them check the Connection page) which Traceway project it is from and which framework was picked when that project was created. That framework tells you which part of the codebase the token is for (`react`/`vuejs`/`svelte` is the browser side, `opentelemetry`/`express`/`gin`/... is a backend service, `nextjs` is settled by the production-role analysis in Step 1). If Step 1 finds more instrumentable parts than you have tokens, ask the user to create one Traceway project per missing part (picking the matching framework) and paste each token here; never stretch one token across parts.+Do not modify code until the user confirms this map.-Do not proceed without real values. Never invent placeholder values in committed code; wire everything through environment variables.+## Step 3: Guide Dashboard Project Creation++Read `dashboard-project-setup.md` and give the user a tailored, ordered checklist containing exactly the projects from the confirmed map. Explain why each project exists, which framework to select, which signals it will receive, and which credentials it needs. Include the relevant Traceway documentation links.++For every credential, define an environment variable rather than asking the user to paste the value. Runtime credentials take component-specific names (`TRACEWAY_BACKEND_TOKEN`, `PUBLIC_TRACEWAY_WEB_CONNECTION_STRING`, `TRACEWAY_MOBILE_CONNECTION_STRING`), respecting the framework's public-environment prefix rules.++**Upload tokens are different: the uploaders read fixed variable names.** `traceway-sourcemaps` reads `TRACEWAY_SOURCEMAP_TOKEN`, and `dart run traceway:upload_symbols` and the iOS dSYM script read `TRACEWAY_UPLOAD_TOKEN` (all three also read `TRACEWAY_URL`). Either name the CI secret exactly that, or keep a component-specific secret and pass it explicitly:++```bash+traceway-sourcemaps --url "$TRACEWAY_URL" --token "$TRACEWAY_WEB_UPLOAD_TOKEN" --directory ./dist+```++Inventing a name like `TRACEWAY_WEB_UPLOAD_TOKEN` and then calling the uploader with no `--token` fails at release time, not at setup time, so make the choice explicit in the CI step.++Proceed when the user confirms the required projects exist and the variables will be available. Existing correctly mapped Traceway projects may be reused.## Integration Paths-Pick the path by project type, and pick the project type from the Step 1 analysis, never from the framework name alone (a Next.js repo can be a full-stack app or just the frontend of a separate backend). Per path, this is not negotiable per framework; it is how Traceway is designed to receive data:+Pick the path by project type, and pick the project type from the "Analyze the Architecture" findings, never from the framework name alone (a Next.js repo can be a full-stack app or just the frontend of a separate backend). Per path, this is not negotiable per framework; it is how Traceway is designed to receive data:| Project type | Path ||---|---|-| **Backend** (any language) | OpenTelemetry, exporting OTLP/HTTP to `<instance>/api/otel/v1/*`. Always, including Go. The native Traceway Go SDK is used only when the user explicitly asks for it. |+| **Backend** (any language) | OpenTelemetry, exporting OTLP/HTTP to `<instance>/api/otel/v1/*`. Always, including Go. || **Frontend** (browser SPA, or a JS meta-framework running frontend-only in production) | Traceway `@tracewayapp/<framework>` SDK + bundler plugin + source map upload (see "Frontend and Mobile" below). |-| **Full-stack JS** (Next.js, SvelteKit, Remix, actually serving its API/SSR in production per Step 1) | BOTH sides, each under its own Traceway project: server side via OpenTelemetry AND browser side via the frontend SDK. |+| **Full-stack JS** (Next.js, SvelteKit, Remix, actually serving its API/SSR in production per "Analyze the Architecture") | BOTH sides, each under its own Traceway project: server side via OpenTelemetry AND browser side via the frontend SDK. || **Mobile** (Flutter, React Native, Android, native Swift iOS) | The Traceway platform SDK. Never OTel. Sole exception: a non-Swift iOS/Apple app has no native SDK, so it uses an OTel library (e.g. Honeycomb) exporting to Traceway like a backend (see "Frontend and Mobile"). |-Two hard rules apply to every backend integration:+The rules every backend integration must satisfy, and the table of what each span becomes, are in Step 4 where they are applied.-1. **Endpoints MUST arrive parametrized.** `http.route` must be set to the route pattern (`/api/users/:id`), never the concrete URL. Traceway uses the value as-is; when it is missing it falls back to `url.path`, and the Endpoints page explodes into one row per unique URL.-2. **Background work MUST use `SpanKind.CONSUMER`.** A root span with the default `INTERNAL` kind and no HTTP attributes is silently dropped (exceptions recorded on it still reach Issues, but the task run itself is lost).+## Step 4: Backend OTel Setup-### How Traceway classifies spans+The same shape in every language:-| OTel Span | Condition | Traceway Concept |-|---|---|---|-| Root span (or span whose parent lives in another service) | `SpanKind = SERVER` or `INTERNAL` with HTTP attributes | **Endpoint** |-| Any span | `SpanKind = CONSUMER` | **Task** |-| Root span | `SpanKind = INTERNAL` with a `console.command` attribute | **Task** (CLI command) |-| Any span | Has any `gen_ai.*` attribute | **AI Trace** |-| Non-root span | Has a parent span ID, matched none of the above | **Span** (child) |-| Exception | `exception` event (or `exception.*` attributes) on any span | **Issue** |-| Root span | Matched none of the above | **Dropped** (exceptions recorded on it still become Issues, unlinked) |+1. Install the language's OpenTelemetry SDK plus the auto-instrumentation for the web framework and database clients.+2. Point the OTLP/HTTP exporter at Traceway with the project token as a Bearer header.+3. Set `service.name` (becomes the Server Name in Traceway) and `service.version` (enables release comparison) on the resource.+4. Verify endpoint grouping before anything else.-For the exact classification rules, endpoint naming, metric conversion, and all the quirks, read `data-model.md` in this skill directory. It is the authoritative reference.+### Three rules that are not negotiable-## Step 1: Analyze the Architecture+1. **Endpoints MUST arrive parametrized.** `http.route` must be the route pattern (`/api/users/:id`), never the concrete URL. Traceway uses the value as-is, but only when it starts with `/`: a route *name* like `app_user_show` is discarded exactly like a missing value. With no usable route it falls back to `url.path`, and the Endpoints page explodes into one row per unique URL.+2. **Background work MUST use `SpanKind.CONSUMER`.** A root span with the default `INTERNAL` kind and no HTTP attributes is silently dropped (exceptions recorded on it still reach Issues, but the task run itself is lost).+3. **The exporter MUST use an OpenTelemetry project's token.** If OTLP arrives with a browser project's token (framework React, Svelte, Vue.js, jQuery, or React Native), Traceway keeps only the Issues and discards every endpoint, task, child span, and AI trace. The symptom is "errors appear but the Endpoints page is empty", and the cause is almost always two projects whose tokens got swapped.-Before changing anything, build a picture of what needs instrumenting:+### How Traceway classifies spans-1. **Frameworks and languages**: detect them by reading `package.json` (Node.js), `go.mod` (Go), `composer.json` (PHP), `requirements.txt`/`pyproject.toml` (Python), `pubspec.yaml` (Flutter), `build.gradle` (Android), `Package.swift` / `*.xcodeproj` / `*.xcworkspace` / `Podfile` (iOS/Swift; note whether the sources are Swift or Objective-C, which picks the path in "Frontend and Mobile"), or asking the user.-2. **Production role of JS meta-frameworks** (Next.js, SvelteKit, Remix, Nuxt): never assume full-stack; determine how the app actually runs in production before picking a path.- - **Frontend-only signals** (Next.js): `output: 'export'` in `next.config.*` (static export, there is no server in production); no API routes (`app/api/**/route.*`, `pages/api/*`) or only trivial ones; no `'use server'` server actions; `rewrites`/proxy config or a `NEXT_PUBLIC_API_URL`-style env var pointing the browser at an external API; a separate backend service in this repo, another repo, or another language; static hosting in the deploy config (S3/CloudFront, GitHub Pages, nginx serving `out/`).- - **Full-stack signals**: API routes with real logic, server actions, database clients imported in server code, SSR reading its own data layer, `next start` or standalone output in the deploy config.- - **Mixed or unclear**: ask the user how the app is deployed in production and whether the framework's server side does anything worth tracking.- Frontend-only means integrate ONLY the browser side with the frontend SDK (React for Next.js); do not add OTel or instrument the framework's server side. The separate backend is its own part with its own Traceway project and its own Step 2 integration.-3. **Services and entry points**: in a monorepo, list each deployable service and its entry point. Each part that should report to Traceway gets its own integration and its own project token, per the one-project-per-part rule in Step 0; ask the user for the missing tokens instead of reusing one across parts.-4. **Background work**: find cron jobs, queue consumers, schedulers, CLI commands, and long-running workers. These must be instrumented as Tasks (Step 3).-5. **AI/LLM usage**: check dependencies for `openai`, `@anthropic-ai/sdk`, `anthropic`, `langchain` / `@langchain/*`, `ai` (Vercel AI SDK), `litellm`, `google-generativeai`, `cohere`, `openrouter`, and agent frameworks (`langgraph`, `crewai`, `autogen` / `ag2`, `pydantic-ai`, `@openai/agents`, `@mastra/*`, `llamaindex` / `llama-index`, `semantic-kernel`). If any are present, Step 4 applies. Then determine whether this is a **conversational product** (chatbot, assistant, agent) rather than one-shot LLM calls: look for chat endpoints or streaming chat routes (`/chat`, `/completions`, SSE or websocket handlers feeding an LLM), message-history persistence (a `messages`/`conversations` table or thread ids), tool/function-calling definitions passed to the model, or any agent framework above. If conversational, `ai-agent.md` in this skill directory governs Step 4 — it adds conversation grouping, per-user attribution, tool calls, and sub-agents.-6. **Deployment signals**: note Dockerfiles, `docker-compose.yml`, Kubernetes manifests, Helm charts, deploy/provisioning scripts, `fly.toml`, `vercel.json`, Procfiles. You will use these in Step 5 to set up server metrics.+The rules are evaluated in this order and the first match wins.-## Step 2: Backend OTel Setup+| # | Span | Condition | Becomes |+|---|---|---|---|+| 1 | Any span | `SpanKind = INTERNAL` **and** carries `exception.*` attributes | **Issue.** Never promoted, even with HTTP or `gen_ai.*` attributes present. A child span keeps its ordinary Span row, a root span gets only the Issue |+| 2 | Root span, or a span whose parent is not in the same export batch | `SpanKind = SERVER` or `INTERNAL` with HTTP attributes | **Endpoint** |+| 3 | Any span | `SpanKind = CONSUMER` | **Task** |+| 4 | Root span | `SpanKind = INTERNAL` with a `console.command` attribute | **Task** (CLI command) |+| 5 | Any span | Has any `gen_ai.*` attribute | **AI Trace** |+| 6 | Non-root span | Has a parent span id | **Span** (child) |+| 7 | Root span | Nothing above matched | **Dropped** (exceptions recorded on it still become Issues, unlinked) |-The same shape in every language:+Because the order is fixed, a `CONSUMER` span carrying `gen_ai.*` is a Task, and a `SERVER` request span carrying `gen_ai.*` is an Endpoint. Put the model call on its own child span (Step 6).-1. Install the language's OpenTelemetry SDK plus the auto-instrumentation for the web framework and database clients.-2. Point the OTLP/HTTP exporter at Traceway with the project token as a Bearer header.-3. Set `service.name` (becomes the Server Name in Traceway) and `service.version` (enables release comparison) on the resource.-4. Verify endpoint grouping (the hard rule above) before anything else.+Two more rules that do not depend on span kind:++- An `exception` event, or `exception.*` attributes, on any span produces an **Issue**, whatever the span itself became.+- An endpoint that returns **404** with no real route matched (`http.route` missing, or a catch-all like `/`, `/*`) is renamed to the literal endpoint `UNMATCHED`, so bot scans and typo'd URLs collapse into one row. A concrete matched route returning 404 keeps its own name.++For the exact rules, endpoint naming, metric conversion, and the remaining quirks, read `data-model.md` in this skill directory. It is the authoritative reference.### Exporter configuration@@ -88,23 +154,40 @@ Where the SDK supports the standard env vars, prefer them; they work identically```bashOTEL_SERVICE_NAME=my-service+OTEL_RESOURCE_ATTRIBUTES=service.version=1.2.3 # there is no OTEL_SERVICE_VERSION variableOTEL_EXPORTER_OTLP_ENDPOINT=https://<instance>/api/otel-OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf # required, several SDKs default to gRPCOTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <project-token>"OTEL_TRACES_EXPORTER=otlpOTEL_METRICS_EXPORTER=otlpOTEL_LOGS_EXPORTER=otlp```-SDKs append `/v1/traces`, `/v1/metrics`, `/v1/logs` to the endpoint automatically. When configuring in code instead, the full URLs are `https://<instance>/api/otel/v1/traces` (and `/v1/metrics`, `/v1/logs`) with header `Authorization: Bearer <project-token>`.+Two of those lines are load-bearing in a way that fails silently:++- **`OTEL_EXPORTER_OTLP_PROTOCOL` is not optional.** Left unset, the Python SDK and the Java agent resolve `otlp` to **gRPC**, and Traceway has no gRPC listener. The app starts, serves traffic, exits 0, prints no warning, and every export is lost. (PHP is the one exception to the value: it needs `http/json`.)+- **There is no `OTEL_SERVICE_VERSION`.** `service.version` only reaches Traceway through `OTEL_RESOURCE_ATTRIBUTES`. Without it, every endpoint row comes back with an empty App Version and release comparison does nothing.++SDKs append `/v1/traces`, `/v1/metrics`, `/v1/logs` to the endpoint automatically, so set the **base** URL only. A full signal path in `OTEL_EXPORTER_OTLP_ENDPOINT` produces `/v1/traces/v1/traces`, and the signal-specific variables (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`) are used verbatim with nothing appended. When configuring in code instead, the full URLs are `https://<instance>/api/otel/v1/traces` (and `/v1/metrics`, `/v1/logs`) with header `Authorization: Bearer <project-token>`.-Constraints: OTLP/HTTP only (protobuf or JSON), OTLP/gRPC is NOT supported; gzip is fine; max request body 10 MB.+Constraints: OTLP/HTTP only (protobuf or JSON). OTLP/gRPC is not supported, there is no listener on port 4317. `Content-Encoding: gzip` is fine. The body is read up to 10 MB and the rest is truncated, so an oversized batch answers `400 failed to unmarshal` rather than `413`. A wrong or missing token answers `401`, and any other path answers `404`. All three are invisible from the application: most SDKs log exporter failures at debug level only, so when nothing arrives, turn the SDK's own diagnostic logging on first.### Node.js example-Create `instrumentation.js` at the project root and load it before the app (`node --import ./instrumentation.js server.js`):+```bash+npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \+ @opentelemetry/instrumentation @opentelemetry/api @opentelemetry/sdk-metrics \+ @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http+```++Declare `@opentelemetry/api` yourself even though it also arrives transitively. Relying on hoisting breaks under pnpm's strict `node_modules` and under Yarn PnP.++Create `instrumentation.mjs` at the project root and load it before the app: `node --import ./instrumentation.mjs server.js`. Use the `.mjs` extension. A `.js` file holding `import` syntax fails to load in a CommonJS project, which is what most projects still are.```javascript+import { register } from "node:module";+register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);+import { NodeSDK } from "@opentelemetry/sdk-node";import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";@@ -112,33 +195,88 @@ import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";const url = process.env.TRACEWAY_URL;-const headers = { Authorization: `Bearer ${process.env.TRACEWAY_TOKEN}` };+const headers = { Authorization: `Bearer ${process.env.TRACEWAY_BACKEND_TOKEN}` };const sdk = new NodeSDK({+ serviceName: process.env.OTEL_SERVICE_NAME ?? "my-service",traceExporter: new OTLPTraceExporter({ url: `${url}/api/otel/v1/traces`, headers }),- metricReader: new PeriodicExportingMetricReader({- exporter: new OTLPMetricExporter({ url: `${url}/api/otel/v1/metrics`, headers }),- exportIntervalMillis: 30_000,- }),+ metricReaders: [+ new PeriodicExportingMetricReader({+ exporter: new OTLPMetricExporter({ url: `${url}/api/otel/v1/metrics`, headers }),+ exportIntervalMillis: 30_000,+ }),+ ],instrumentations: [getNodeAutoInstrumentations()],});sdk.start();```-Auto-instrumentation covers Express routes (sets `http.route`), status codes, errors, and CJS database clients (`pg`, `mysql2`, `mongodb`, `ioredis`). SQLite and custom business logic need manual `tracer.startActiveSpan()` child spans.+Three parts of that snippet are load-bearing:++- **The two `register` lines install the ESM loader hook.** Without them an ESM app (`"type": "module"`, `import express from "express"`) still gets HTTP spans but no `http.route`, so every URL becomes its own endpoint row. A CommonJS app (`require("express")`) is patched without the hook, and the lines are harmless there, so keep them either way.+- **`serviceName`** becomes the Server Name in Traceway. Leave it out and every span reports `unknown_service:node`. For release comparison also set `OTEL_RESOURCE_ATTRIBUTES=service.version=1.2.3`.+- **`metricReaders` is a list.** The older singular `metricReader` option is deprecated.++Auto-instrumentation covers Express routes (sets `http.route`), status codes, errors, and database clients (`pg`, `mysql2`, `mongodb`, `ioredis`). SQLite and custom business logic need manual `tracer.startActiveSpan()` child spans.### Per-language notes-- **Go**: use the contrib middleware for the framework: `otelgin`, `otelchi` (with `otelchi.WithChiRoutes(r)`), `otelfiber`; they set `http.route` from the route pattern. For stdlib `net/http`, current `otelhttp` derives `http.route` from Go 1.23+ `ServeMux` method+pattern routes (`GET /api/users/{id}`); for other routers set `http.route` manually on the active span. Exporter: `otlptracehttp.WithEndpointURL(...)` + `WithHeaders`.-- **Python**: `opentelemetry-distro` + `opentelemetry-bootstrap -a install`, run under `opentelemetry-instrument` with the env vars above. Django/Flask/FastAPI instrumentations set `http.route`.-- **PHP**: Symfony and Laravel via their OTel packages and `OTEL_*` env vars. Symfony caveat: the stock auto-instrumentation sets `http.route` to the route NAME, which Traceway ignores; use the Traceway Symfony integration (`composer require traceway/opentelemetry-symfony`). See `data-model.md`.+- **Go**: use the framework's OTel middleware. `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin`, `github.com/riandyrn/otelchi` (a community package, and pass `otelchi.WithChiRoutes(r)` or it reports raw URLs), `github.com/gofiber/contrib/otelfiber/v2`. All three set `http.route` from the matched route pattern. Exporter: `otlptracehttp.WithEndpointURL(...)` + `WithHeaders`. Two Go-only traps:+ - **Set the propagator.** Go's global propagator is a no-op by default, so `traceparent` is never sent or read and cross-service traces never link. One line, right after `otel.SetTracerProvider(tp)`:++ ```go+ otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(+ propagation.TraceContext{}, propagation.Baggage{},+ ))+ ```++ - **stdlib `net/http` does not report its route.** `otelhttp` reads the route from `r.Pattern`, which `ServeMux` fills in only after it has matched, so the usual top-level `otelhttp.NewHandler(mux, "server")` records no `http.route` at all and every URL becomes its own endpoint row. Either wrap each registered handler, which puts the span start inside the mux:++ ```go+ mux.Handle("GET /api/users/{id}", otelhttp.NewHandler(usersHandler, "users"))+ ```++ or keep the single top-level wrap and stamp the route yourself:++ ```go+ // imports: strings, go.opentelemetry.io/otel/trace,+ // semconv "go.opentelemetry.io/otel/semconv/v1.26.0"+ func withRoute(mux *http.ServeMux) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if _, pattern := mux.Handler(r); pattern != "" {+ if i := strings.IndexByte(pattern, '/'); i >= 0 {+ route := pattern[i:] // drop the "GET " prefix, http.route must start with "/"+ trace.SpanFromContext(r.Context()).SetAttributes(semconv.HTTPRoute(route))+ }+ }+ mux.ServeHTTP(w, r)+ })+ }++ handler := otelhttp.NewHandler(withRoute(mux), "server")+ ```++- **Python**: `pip install opentelemetry-distro opentelemetry-exporter-otlp`, then `opentelemetry-bootstrap -a install`, then run the app under `opentelemetry-instrument` with the env vars above (`opentelemetry-instrument uvicorn app:app`, `opentelemetry-instrument gunicorn wsgi:app`, `opentelemetry-instrument python worker.py`). Starting the server without that prefix sends nothing. FastAPI and Flask instrumentation sets `http.route` correctly with zero application code, in the framework's own syntax (`GET /users/{user_id}`, `GET /orders/<order_id>`), and both frameworks already answer `500` on an unhandled exception, so the endpoint status and the Issue line up without extra work. Three Python-only gotchas:+ - **Logs need two more variables.** `OTEL_LOGS_EXPORTER=otlp` attaches the OTel handler to the **root** logger, whose level defaults to `WARNING`, so every `logger.info(...)` is filtered out before the bridge sees it and only WARN and above reach Traceway. Attaching the handler also replaces the root handler list, so the app's own records stop appearing on stdout. Add both:++ ```bash+ OTEL_PYTHON_LOG_CORRELATION=true # restores console output, stamps otelTraceID/otelSpanID+ OTEL_PYTHON_LOG_LEVEL=info # only read alongside the line above+ ```++ `OTEL_PYTHON_LOG_LEVEL` on its own does nothing. Do **not** set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true`; on current versions the bridge is already on and that variable switches it to the SDK's deprecated handler.+ - **Metrics take a minute.** The SDK default for `OTEL_METRIC_EXPORT_INTERVAL` is 60000 ms, so a correct pipeline looks empty while you are checking it. Set `OTEL_METRIC_EXPORT_INTERVAL=10000` during verification.+ - **Django needs two extra things**: export `DJANGO_SETTINGS_MODULE` or `opentelemetry-instrument` fails to start Django at all, and add the route middleware from the Django guide, because `opentelemetry-instrumentation-django` reports `http.route` as Django's own pattern (`api/users/<int:user_id>/`), which has no leading `/` and is therefore discarded.++ Guides: https://docs.tracewayapp.com/client/otel/python and https://docs.tracewayapp.com/client/otel/django+- **PHP**: Laravel via `composer require keepsuit/laravel-opentelemetry open-telemetry/exporter-otlp php-http/guzzle7-adapter`; Symfony via `composer require traceway/opentelemetry-symfony open-telemetry/exporter-otlp php-http/guzzle7-adapter` (the stock Symfony auto-instrumentation sets `http.route` to the route NAME, which Traceway discards; see `data-model.md`). PHP does not take `OTEL_PHP_AUTOLOAD_ENABLED` as a plain env var next to the block above. Laravel needs nothing extra, the keepsuit service provider starts the SDK. Symfony starts the SDK through `open_telemetry.sdk.autoload_enabled: true` in `config/packages/open_telemetry.yaml`, or through a real process environment variable set in php-fpm, the Dockerfile, or Apache. Never put `OTEL_PHP_AUTOLOAD_ENABLED` in `.env`: Dotenv reads it after Composer autoload, the bundle then skips starting the SDK, and every signal silently becomes a no-op. PHP also prefers `OTEL_EXPORTER_OTLP_PROTOCOL=http/json`, though `http/protobuf` works too and is only slower without `ext-protobuf`.- **Java / .NET / anything else**: the standard OTel agent or SDK with the env vars above works as-is.- Full per-framework docs: https://docs.tracewayapp.com/client/otel### Verify endpoint grouping (do this first)-Hit a parametrized route a few times with different IDs and check the Traceway Endpoints page: you must see ONE row (`GET /api/users/:id`), not one row per ID. If you see raw IDs, the instrumentation is not setting `http.route`; fix that before continuing. Last resort, set it manually in a middleware that knows the matched route pattern:+Hit a parametrized route a few times with different IDs and check the Traceway Endpoints page: you must see ONE row (`GET /api/users/:id`), not one row per ID. If you see raw IDs, the instrumentation is not setting `http.route`; fix that before continuing. Last resort, set it manually in a middleware that knows the matched route pattern (the value must start with `/`, or it is discarded):```javascriptimport { trace } from "@opentelemetry/api";@@ -160,15 +298,51 @@ span?.setStatus({ code: SpanStatusCode.ERROR, message: error.message });(Go: `span.RecordError(err, trace.WithStackTrace(true))`; the stack trace option is what produces the `exception.stacktrace` attribute.)-**Return HTTP 500 when an exception happens.** A request that throws or panics must respond with `500` and set the span status to `ERROR` — never let it fall through to a `200`/`2xx`. The common trap is a recover/panic middleware that reports the exception but lets the response writer keep its default `200`: Traceway then records the endpoint transaction as a *success*, so the Endpoints page looks healthy while Issues fills up, and the exception becomes an unlinked island instead of correlating to a failed request (and, with distributed tracing, to the frontend call that triggered it). Always pair "an exception was recorded" with "respond `500` and set span status `ERROR`". Genuine client errors that are NOT bugs — validation `422`, auth `401`, not-found `404` — keep their real 4xx status and should not be recorded as exceptions in the first place.+**Return HTTP 500 when an exception happens.** A request that throws or panics must respond with `500` and set the span status to `ERROR`. Never let it fall through to a `200`/`2xx`. The common trap is a recover/panic middleware that reports the exception but lets the response writer keep its default `200`: Traceway then records the endpoint transaction as a *success*, so the Endpoints page looks healthy while Issues fills up, and the exception becomes an unlinked island instead of correlating to a failed request (and, with distributed tracing, to the frontend call that triggered it). Always pair "an exception was recorded" with "respond `500` and set span status `ERROR`". Genuine client errors that are NOT bugs (validation `422`, auth `401`, not-found `404`) keep their real 4xx status and should not be recorded as exceptions in the first place.++Go has a sharper version of the same trap. `otelgin` records the status only after `c.Next()` returns, which never happens during a panic unwind, so an unrecovered panic leaves the endpoint row at status **0** even though Gin's own recovery already returned 500 to the client, and the Issue arrives with a title but no stack trace. Register a recovery middleware **after** `otelgin.Middleware` so it runs inside the server span:++```go+// imports: fmt, net/http, go.opentelemetry.io/otel/codes, go.opentelemetry.io/otel/trace+func otelRecovery() gin.HandlerFunc {+ return func(c *gin.Context) {+ defer func() {+ if r := recover(); r != nil {+ err, ok := r.(error)+ if !ok {+ err = fmt.Errorf("%v", r)+ }+ span := trace.SpanFromContext(c.Request.Context())+ span.RecordError(err, trace.WithStackTrace(true))+ span.SetStatus(codes.Error, err.Error())+ c.AbortWithStatus(http.StatusInternalServerError)+ }+ }()+ c.Next()+ }+}++router.Use(otelgin.Middleware(serviceName))+router.Use(otelRecovery())+```++### Logs++`OTEL_LOGS_EXPORTER=otlp` wires the exporter only. The application's own log calls still need a bridge, or the Logs page stays empty while traces and metrics look perfectly healthy.++- **Node**: add `@opentelemetry/winston-transport`, or the Pino or Bunyan instrumentation, so each record picks up `trace_id` and `span_id`.+- **Python, PHP, Java, .NET**: `opentelemetry-instrument` and the language agents bridge the standard logger automatically.+- **Go**: build an `sdklog.LoggerProvider` with `otlploghttp.WithEndpointURL(...)`, register it with `global.SetLoggerProvider(lp)`, and `defer lp.Shutdown(ctx)` or the last batch is lost on exit.-### If the user explicitly asks for the Traceway Go SDK+Emit from the **request** context (`c.Request.Context()` in Gin, `r.Context()` in net/http, the active context in Node). A line emitted from a background context carries no trace id and cannot be opened from the endpoint it belongs to. Traceway reads `severity_text` (or derives it from `severity_number`), the body, `service.name`, and the trace and span ids.-Only on explicit request, instead of OTel: `go get go.tracewayapp.com/tracewaygin` (or `tracewaychi`, `tracewayfiber`, `tracewayfasthttp`, `tracewayhttp`), add the middleware with connection string `<project-token>@https://<instance>/api/report`. Trade-off: the Go SDK natively emits the built-in system metric names (`cpu.used_pcnt`, `mem.used`, `go.*`) that populate the dashboard's built-in charts; on the OTel path those charts stay empty and host metrics come from the Traceway OTel Agent (Step 5).+### Go uses OpenTelemetry-## Step 3: Background Tasks (Boundaries and Labeling)+There is no separate Traceway Go SDK. It was retired, and Go instruments with `opentelemetry-go` exactly like every other backend: `go get go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`, then the same env vars as the block above. Host CPU and memory come from the Traceway OTel Agent (see "Deployment and Server Metrics"), not from the app.-If Step 1 found background work, instrument it as Tasks. The rules:+## Step 5: Background Tasks (Boundaries and Labeling)++If "Analyze the Architecture" found background work, instrument it as Tasks. The rules:- **Boundary: one Task = one execution of a unit of background work.** A whole cron job run is one task. One queue message or job is one task. One CLI command invocation is one task. Per-item work inside a run (each email in a batch, each row in an import) is a child span of the task span, never a separate `CONSUMER` span.- **Do not double-wrap.** If a library's auto-instrumentation already emits `CONSUMER` spans (Kafka, RabbitMQ, Symfony Messenger consumers), wrapping them again creates duplicate Task entries.@@ -189,8 +363,9 @@ async function runScheduledJob() {await doWork();span.setStatus({ code: SpanStatusCode.OK });} catch (error) {- span.recordException(error);- span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });+ const err = error instanceof Error ? error : new Error(String(error));+ span.recordException(err);+ span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });throw error;} finally {span.end();@@ -202,13 +377,13 @@ async function runScheduledJob() {(Go: `tracer.Start(ctx, "cleanup-expired-sessions", trace.WithSpanKind(trace.SpanKindConsumer))`.)-## Step 4: AI Traces+## Step 6: AI Traces-If Step 1 found AI/LLM dependencies, instrument the model calls. Any span carrying at least one `gen_ai.*` attribute is promoted to an **AI Trace**; since calls usually happen inside a request or task, the span is naturally a child and stays linked to its Endpoint/Task by trace ID.+If "Analyze the Architecture" found AI/LLM dependencies, instrument the model calls. A span becomes an **AI Trace** when it carries at least one `gen_ai.*` attribute *and* matched none of the earlier classification rules. In practice that means: keep `gen_ai.*` off the HTTP route span (it becomes an Endpoint), keep it off `CONSUMER` spans (they become Tasks), and give the model call its own child span with `SpanKind.CLIENT` rather than `INTERNAL`, so a failed call that records `exception.*` attributes still produces its AI Trace. Since calls happen inside a request or task, that child span stays linked to its Endpoint or Task by trace ID.Boundaries: **one span per model call** (one provider API request = one AI Trace row). A multi-step agent run is multiple spans sharing a stable `trace.name` (same labeling discipline as task names: no IDs or timestamps). For streaming, end the span when the stream finishes.-**Conversational product?** If Step 1 flagged a chatbot, assistant, or tool-calling agent, follow `ai-agent.md` in this skill directory instead of stopping at the table below — it covers `gen_ai.conversation.id` (multi-turn grouping), what value belongs in `user.id` (a stable end-customer id, never a session id), tool-call payload shapes, and sub-agents. Without those, the Conversations and Users tabs stay empty.+**Conversational product?** If "Analyze the Architecture" flagged a chatbot, assistant, or tool-calling agent, follow `ai-agent.md` in this skill directory instead of stopping at the table below. It covers `gen_ai.conversation.id` (multi-turn grouping), what value belongs in `user.id` (a stable end-customer id, never a session id), tool-call payload shapes, and sub-agents. Without those the Users tab stays empty, and the Conversations tab fills with one throwaway single-turn conversation per request, because Traceway falls back to the OTel trace id. That is worse than empty: it looks like the feature is working.Attributes Traceway reads (all optional, set what is available):@@ -226,16 +401,18 @@ Attributes Traceway reads (all optional, set what is available):| `gen_ai.response.finish_reason` | Why generation stopped || `gen_ai.prompt` / `gen_ai.completion` | Conversation content, shown on the trace detail page (skip if content must not leave the app) |-Conversation content is also read from `trace.input`/`trace.output` (or `span.input`/`span.output`) when `gen_ai.prompt`/`gen_ai.completion` are absent, and missing `total_tokens`/`total_cost` are computed from the input + output values.+Conversation content is also read from `trace.input`/`trace.output` (or `span.input`/`span.output`) when `gen_ai.prompt`/`gen_ai.completion` are absent, and missing `total_tokens`/`total_cost` are computed from the input + output values. Token counts must be integer attributes and costs must be double or integer attributes. A number sent as a string is silently dropped and stored as 0, with no error on the export.```typescript-return tracer.startActiveSpan("chat-completion", async (span) => {+import { SpanKind } from "@opentelemetry/api";++return tracer.startActiveSpan("chat-completion", { kind: SpanKind.CLIENT }, async (span) => {const response = await openai.chat.completions.create({ model: "gpt-4o", messages });span.setAttributes({"gen_ai.system": "openai","gen_ai.request.model": "gpt-4o",- "gen_ai.usage.input_tokens": response.usage.prompt_tokens,- "gen_ai.usage.output_tokens": response.usage.completion_tokens,+ "gen_ai.usage.input_tokens": response.usage?.prompt_tokens ?? 0,+ "gen_ai.usage.output_tokens": response.usage?.completion_tokens ?? 0,"trace.name": "support-agent",});span.end();@@ -245,7 +422,7 @@ return tracer.startActiveSpan("chat-completion", async (span) => {Zero-code path for OpenRouter users: in OpenRouter Settings -> Observability, add an OpenTelemetry Collector destination pointing at `https://<instance>/api/otel/v1/traces` with header `{"Authorization": "Bearer <project-token>"}`. Docs: https://docs.tracewayapp.com/client/openrouter-## Frontend and Mobile+## Step 7: Frontend and MobileFrontend and mobile projects do NOT use OTel; they use the Traceway SDKs reporting to `/api/report` with connection string `<project-token>@https://<instance>/api/report`.@@ -253,22 +430,22 @@ Frontend and mobile projects do NOT use OTel; they use the Traceway SDKs reporti1. **SDK**: `npm install @tracewayapp/react` (or `vue`, `svelte`, `jquery`, `frontend` for plain JS) and initialize with the connection string (React: wrap the app in `<TracewayProvider connectionString="...">`). Captures errors, web vitals, and session replay.2. **Bundler plugin**: `npm install -D @tracewayapp/bundler-plugin`, then add `tracewayDebugIds()` from `@tracewayapp/bundler-plugin/vite` (or `/rollup`, or `TracewayDebugIdsWebpackPlugin` from `/webpack`) to the bundler config, with source maps enabled (`build.sourcemap: true` / `devtool: "source-map"`).-3. **Source map upload**: `npm install -D @tracewayapp/sourcemap-upload`, then run `traceway-sourcemaps --url <instance> --token <source-map-upload-token> --directory ./dist` as a postbuild or CI step (env vars: `TRACEWAY_URL`, `TRACEWAY_SOURCEMAP_TOKEN`). The upload token comes from Step 0 and is a CI secret, never committed.+3. **Source map upload**: `npm install -D @tracewayapp/sourcemap-upload`, then run `traceway-sourcemaps --url <instance> --token <source-map-upload-token> --directory ./dist` as a postbuild or CI step (env vars: `TRACEWAY_URL`, `TRACEWAY_SOURCEMAP_TOKEN`). The upload token comes from the browser project and is a CI secret, never committed.For the per-framework init code (plain JS, React, Vue, Svelte/SvelteKit, jQuery), the shared SDK options, error filtering, custom attributes, distributed tracing, and the full debug-ID + source map pipeline, read `frontend-js.md` in this skill directory. Online docs: https://docs.tracewayapp.com/client/react (or `vue`, `svelte`, `jquery`, `js-sdk`).-**Full-stack JS** (Next.js, SvelteKit, Remix): only when Step 1 confirmed the framework's server actually serves the app in production. A frontend-only deployment (static export, or the API lives in a separate backend) gets just the browser pieces above and no server-side instrumentation; the separate backend follows Step 2 as its own part. When it is genuinely full-stack, integrate both sides, each under its own Traceway project (Step 0): server side follows Step 2 with the backend project's token (verify `http.route` grouping; set it manually in a server hook where the auto-instrumentation does not know the router), browser side follows the three pieces above with the frontend project's token.+**Full-stack JS** (Next.js, SvelteKit, Remix): only when "Analyze the Architecture" confirmed the framework's server actually serves the app in production. A frontend-only deployment gets just the browser pieces above; a separate backend follows "Backend OTel Setup". When it is genuinely full-stack, integrate both sides under the two confirmed projects: server side follows "Backend OTel Setup" with the backend project's token, and browser side follows the three pieces above with the frontend project's token.**Mobile**, always the platform SDK, never OTel:-- **Flutter**: `flutter pub add traceway`, then `Traceway.run(connectionString: '<token>@https://<instance>/api/report', child: MyApp())`. Then check whether the release build is obfuscated (`--obfuscate --split-debug-info`): if it is, production crash stack traces arrive obfuscated and stay unreadable until the build's `.symbols` files are uploaded, so ask the user for the upload token (Step 0) and wire up the symbol upload. For options, platform permissions, the navigator observer, screen recording, privacy masking, the obfuscation check and symbol upload, and the Flutter web caveat, read `flutter.md` in this skill directory. Docs: https://docs.tracewayapp.com/client/flutter+- **Flutter**: `flutter pub add traceway`, then `Traceway.run(connectionString: '<token>@https://<instance>/api/report', child: MyApp())`. Then check whether the release build is obfuscated (`--obfuscate --split-debug-info`): if it is, production crash stack traces arrive obfuscated and stay unreadable until the build's `.symbols` files are uploaded, so use the mobile project's upload token and wire up the symbol upload. For options, platform permissions, the navigator observer, screen recording, privacy masking, the obfuscation check and symbol upload, and the Flutter web caveat, read `flutter.md` in this skill directory. Docs: https://docs.tracewayapp.com/client/flutter- **React Native**: `npm install @tracewayapp/react-native`, wrap the app in `TracewayProvider`. Docs: https://docs.tracewayapp.com/client/react-native-- **Android** (native Kotlin/Java): add `implementation("com.tracewayapp:traceway:1.0.1")` from Maven Central and call `Traceway.init(application = this, connectionString = "<token>@https://<instance>/api/report", options = TracewayOptions(version = "1.0.0"))` from `Application.onCreate()` (register the `Application` class and add the `INTERNET` permission in the manifest). It captures every uncaught Java/Kotlin exception on every thread plus manual `Traceway.captureException(...)`; errors and crashes only, no session replay. Release builds run R8, so production crashes arrive with renamed classes and rewritten line numbers and stay unreadable until the build's `mapping.txt` is uploaded: apply the `com.tracewayapp.symbols` Gradle plugin (`version "1.0.1"`, resolved from `mavenCentral()`), which injects `BuildConfig.TRACEWAY_PROGUARD_UUID` and uploads the mapping with the upload token (Step 0); pass that UUID into `TracewayOptions(proguardUuid = ...)` so each crash matches its mapping. For init code, options, the manifest, and the full Gradle plugin setup, read `android.md` in this skill directory. Docs: https://docs.tracewayapp.com/client/android-- **iOS / Swift** (native SwiftUI or UIKit): add the Traceway iOS SDK via Swift Package Manager (`https://github.com/tracewayapp/traceway-ios.git`) and call `Traceway.start(connectionString: "<token>@https://<instance>/api/report", options: TracewayOptions(version: "1.0.0"))` as early as possible. It captures uncaught `NSException`s and fatal signals (hard crashes upload on the next launch) plus manual `Traceway.capture(...)`; it reports errors and crashes only (no session replay). Release crashes arrive as bare addresses until the build's dSYMs are uploaded, so set up dSYM upload with the upload token (Step 0). For init code, options, the debugger caveat, and dSYM upload, read `ios.md` in this skill directory. **If the app is NOT a Swift app** (Objective-C only, a cross-platform stack with no Traceway mobile SDK, or a team standardized on OpenTelemetry), there is no native SDK: use an OTel distribution like Honeycomb with its exporter pointed at `<instance>/api/otel` and a `Authorization: Bearer <project-token>` header, exactly like a backend (Step 2). The non-Swift path is also in `ios.md`.+- **Android** (native Kotlin/Java): add `implementation("com.tracewayapp:traceway:1.0.1")` from Maven Central and call `Traceway.init(application = this, connectionString = "<token>@https://<instance>/api/report", options = TracewayOptions(version = "1.0.0"))` from `Application.onCreate()` (register the `Application` class in the manifest; no permission entry is needed, the AAR's own manifest declares `INTERNET` and `ACCESS_NETWORK_STATE` and the merger folds them in). It captures every uncaught Java/Kotlin exception on every thread plus manual `Traceway.captureException(...)`; errors and crashes only, no session replay. Release builds run R8, so production crashes arrive with renamed classes and rewritten line numbers and stay unreadable until the build's `mapping.txt` is uploaded: apply the `com.tracewayapp.symbols` Gradle plugin (`version "1.0.1"`, resolved from `mavenCentral()`), which injects `BuildConfig.TRACEWAY_PROGUARD_UUID` and uploads the mapping with the mobile project's upload token; pass that UUID into `TracewayOptions(proguardUuid = ...)` so each crash matches its mapping. For init code, options, the manifest, and the full Gradle plugin setup, read `android.md` in this skill directory. Docs: https://docs.tracewayapp.com/client/android+- **iOS / Swift** (native SwiftUI or UIKit): add the Traceway iOS SDK via Swift Package Manager (`https://github.com/tracewayapp/traceway-ios.git`) and call `Traceway.start(connectionString: "<token>@https://<instance>/api/report", options: TracewayOptions(version: "1.0.0"))` as early as possible. It captures uncaught `NSException`s and fatal signals (hard crashes upload on the next launch) plus manual `Traceway.capture(...)`; it reports errors and crashes only (no session replay). Release crashes arrive as bare addresses until the build's dSYMs are uploaded, so set up dSYM upload with the mobile project's upload token. For init code, options, the debugger caveat, and dSYM upload, read `ios.md` in this skill directory. **If the app is NOT a Swift app** (Objective-C only, a cross-platform stack with no Traceway mobile SDK, or a team standardized on OpenTelemetry), there is no native SDK: use an OTel distribution like Honeycomb with its exporter pointed at `<instance>/api/otel` and a `Authorization: Bearer <project-token>` header, exactly like a backend (see "Backend OTel Setup"). The non-Swift path is also in `ios.md`.-## Step 5: Deployment and Server Metrics+## Step 8: Deployment and Server Metrics-After the code-side integration is in place, ask the user two questions (pre-fill your best guess from the deployment signals found in Step 1 and let them confirm):+Use the deployment and host-metrics choice confirmed in "Propose and Confirm the Project Map". If it remains unresolved, ask before changing deployment configuration:1. **How is this project deployed?** Docker on a VM / directly on a VM or bare metal / Kubernetes / serverless or PaaS.2. **Do you want server (host) metrics tracked in Traceway?** CPU, memory, disk, filesystem, network of the machine running the app.@@ -276,11 +453,11 @@ After the code-side integration is in place, ask the user two questions (pre-fil| Deployment | Wants host metrics | What to do ||---|---|---|| Docker on a VM, or directly on a VM/host | Yes | Install the **Traceway OTel Agent** on the host (below). For Docker deploys this is the default; the agent goes on the host, not in a container. |-| Kubernetes | Any | Agent not applicable (host service, no Docker image or K8s manifests by design). In-process app metrics still flow via the OTLP metrics exporter from Step 2. |+| Kubernetes | Any | Agent not applicable (host service, no Docker image or K8s manifests by design). In-process app metrics still flow via the OTLP metrics exporter from "Backend OTel Setup". || Serverless / PaaS | Any | No host to install on; skip. || Anything | No | Skip. |-The agent is a tiny pre-configured OTel Collector that scrapes host metrics every 60s and ships them with the project token. Install on the host (Linux systemd / macOS launchd; PowerShell installer exists for Windows):+The agent is a tiny pre-configured OTel Collector that scrapes host metrics every 60s. It MUST use the same backend project token as the API, workers, tasks, and AI traces. Install it on the host (Linux systemd / macOS launchd; PowerShell installer exists for Windows):```bashcurl -fsSL https://install.tracewayapp.com/install.sh | \@@ -295,21 +472,37 @@ curl -fsSL https://install.tracewayapp.com/install.sh | \- Re-running the installer upgrades in place, so the command is safe to keep in provisioning scripts.- If the repo has host provisioning or deploy scripts (cloud-init, Ansible, Terraform `user_data`, `deploy.sh`), add the command there with the token referenced from a secret. Otherwise hand the operator the filled-in one-liner; do not modify the repo.-Metrics arrive within ~60s under their hostmetrics names (`system.cpu.utilization`, `system.memory.usage`, ...), tagged with `service.name` and `host.name`. They are chartable via custom widgets; they do NOT populate the built-in CPU/memory charts (those read the Go SDK's exact names). Agent repo: https://github.com/tracewayapp/traceway-otel-agent--## Step 6: Verify--1. Start the app and hit a few endpoints (or trigger an error on purpose).-2. Check the Traceway dashboard:- - **Endpoints page**: routes appear grouped by pattern (e.g. `GET /api/users/:id`), not by literal URL, and status codes are non-zero.- - **Issues page**: thrown errors appear with stack traces. For frontend projects, stack traces are symbolicated to original files and lines; for obfuscated Flutter builds, they resolve once the build's `.symbols` files have been uploaded; for native iOS Release crashes, they resolve to symbol names and `file:line` once the build's dSYMs have been uploaded.- - **Endpoint detail -> Spans tab**: database queries and outgoing calls appear as children.- - **Tasks page**: after triggering each background job once, it appears under one stable name.- - **AI Traces page** (if Step 4 applied): model calls appear with token counts and cost.- - **Metrics** (if the agent was installed): host metrics arrive within about 60 seconds.-3. If the `traceway` CLI is installed and authenticated, verify from the terminal instead:+Metrics arrive within ~60s under their hostmetrics names (`system.cpu.utilization`, `system.memory.usage`, ...). The host is identified by the `server_name` tag, which comes from `TRACEWAY_SERVICE_NAME`. Resource attributes such as `host.name` are not stored as tags at all, so give every host a distinct `TRACEWAY_SERVICE_NAME` or the hosts cannot be told apart. The metrics do NOT populate the built-in CPU/memory charts (those read the Go SDK's exact names). Install the **OTelemetry Server Agent** dashboard template instead: open the command palette with Cmd K, search for it, and it installs a dashboard already wired to `system.cpu.utilization`, `system.memory.*`, `system.filesystem.*`, `system.disk.*` and `system.network.*`. Custom widgets cover anything the template misses. Agent repo: https://github.com/tracewayapp/traceway-otel-agent++## Step 9: Verify++Every dashboard page named below is at `<instance>/<page>`: `/endpoints`, `/issues`, `/tasks`, `/ai-traces`, `/logs`, `/dashboards`. Set the time picker to the last 15 minutes before reading any of them. Verify each project independently and finish with a project-to-component summary.++1. **Backend project**+ - Run `curl <app>/api/users/1`, then `/2`, then `/3`, and open `/endpoints`. Exactly one row, `GET /api/users/:id`, with non-zero status codes. Three rows means `http.route` is not being set. Fix that before checking anything else.+ - Two results that look broken but are not: a request to a URL matching no route arrives as `UNMATCHED`, not as the URL you typed; and if the project has "drop healthy healthchecks" enabled, successful `/health` requests are discarded on ingest, so never use a healthcheck route as the smoke test.+ - Trigger a handled or test exception, then open `/issues`. The exception is listed with its endpoint or task context, and that request's endpoint row shows status 500.+ - Open one endpoint row: database queries and outgoing HTTP calls appear as child spans.+ - Run each background job once, then open `/tasks`. One row per job, under a stable name with no ids or timestamps in it.+ - If logs were wired, emit one INFO and one ERROR line inside a request, then open `/logs`. Both carry the same trace id as that endpoint.+ - If "AI Traces" applied, make one model call, then open `/ai-traces`. The call is listed with model, tokens, and cost where available.+ - Open `/dashboards` and confirm application/runtime metrics arrive. If the OTel Agent was installed, host metrics reach this same backend project within about 60 seconds, each host under its own `server_name` tag.+2. **Browser project**+ - Trigger a test browser error and confirm it appears in this project, not the backend project.+ - Confirm the stack resolves to original source files and lines after the matching source maps and bundles are uploaded.+ - If distributed tracing was configured, confirm a browser issue or request links to its backend trace.+3. **Each mobile project**+ - Trigger the platform's safe test error or crash flow and confirm it appears only in the matching mobile project.+ - For obfuscated Flutter, minified Android, or Release iOS builds, confirm the matching symbols, R8 mapping, or dSYM resolves the production stack.+4. **Credential and topology audit**+ - Confirm browser and mobile integrations do not use the backend token, and that the OTLP exporter does not use a browser project's token (rule 3 in Step 4).+ - Confirm APIs, workers, schedulers, AI calls, and host metrics use the backend project unless the user approved an isolation boundary.+ - Report any verification that could not run because a local, deployment, or CI variable is not yet populated.+5. If the `traceway` CLI is installed, run `traceway login`, select the backend project with `traceway projects use <project-id>` (or pass `--project <project-id>` on every call), then:```bashtraceway endpoints list --since 15mtraceway exceptions list --since 15m+ traceway logs query --since 15mtraceway metrics query --name system.cpu.utilization --since 15m```+ Tasks and AI traces have no `list` subcommand, only `show <id>`, so check those two on `/tasks` and `/ai-traces` in the dashboard.diff --git a/skills/traceway-setup/ai-agent.md b/skills/traceway-setup/ai-agent.mdindex fdb67c43..3d1fd7d3 100644--- a/skills/traceway-setup/ai-agent.md+++ b/skills/traceway-setup/ai-agent.md@@ -1,6 +1,69 @@# Instrumenting an AI Agent or Chatbot for Traceway-Read this file when Step 1 detected a conversational AI product: a chatbot, a multi-turn assistant, a tool-calling agent, or anything built on an agent framework. The base rules from SKILL.md Step 4 still apply (one span per model call, `gen_ai.*` attributes promote the span to an AI Trace). This file adds what conversational products need on top, so Traceway's Conversations and Users analytics light up: conversation grouping, per-user attribution, tool calls, and sub-agents.+Read this file when "Analyze the Architecture" detected a conversational AI product: a chatbot, a multi-turn assistant, a tool-calling agent, or anything built on an agent framework. The base rules from the "AI Traces" section of SKILL.md still apply (one span per model call, and a `gen_ai.*` attribute promotes that span to an AI Trace). This file adds what conversational products need on top, so Traceway's Conversations and Users analytics light up: conversation grouping, per-user attribution, tool calls, and sub-agents.++## Before you write anything: three preconditions++Get these wrong and the export still returns 200 while the dashboard stays empty. Check all three first.++### 1. Your instrumentation must emit `gen_ai.*`++Traceway promotes a span to an AI Trace only when it carries an attribute literally prefixed `gen_ai.`. Agent-framework auto-instrumentation is split on this:++| Instrumentation | Emits | Works out of the box? |+|---|---|---|+| OpenLLMetry / Traceloop (`traceloop-sdk`, `@traceloop/node-server-sdk`) | `gen_ai.*` including `gen_ai.prompt` / `gen_ai.completion` | Yes |+| OpenLIT (`openlit`) | `gen_ai.*` | Yes |+| Vercel AI SDK `experimental_telemetry` | `gen_ai.*` for model and tokens, `ai.*` for content | Partly, see precondition 2 |+| OpenInference / Arize (`openinference-instrumentation-langchain`, `-llama-index`, `-crewai`, `-openai`) | `openinference.span.kind`, `llm.model_name`, `llm.token_count.*`, `input.value`, `output.value`, and **no `gen_ai.*`** | **No** |+| Langfuse / LangSmith OTel exporters | vendor-prefixed attributes | **No** |++An OpenInference-style span is not merely missing tokens. It is a root INTERNAL span that matches no promotion rule, so it is dropped outright: no AI trace, no span row, nothing on any page, while every OTLP export answers 200. If the project uses one of the "No" rows, pick one of these: switch to OpenLLMetry or OpenLIT, add a span processor that copies the vendor attributes onto their `gen_ai.*` equivalents, or instrument the model calls by hand with the skeleton below.++### 2. Content must land on the attribute names Traceway reads++Traceway reads the prompt from `gen_ai.prompt`, falling back to `trace.input`, then `span.input`. It reads the completion from `gen_ai.completion`, falling back to `trace.output`, then `span.output`. It does **not** read the newer OTel semconv names `gen_ai.input.messages` / `gen_ai.output.messages`, nor the Vercel AI SDK's `ai.prompt.messages` / `ai.response.text`.++Content on an unread name is stored as an inert attribute. The symptom is confusing: tokens and cost look perfectly healthy while the chat timeline is blank, tool calls are 0, and content flagging can never fire. If your SDK emits the semconv or `ai.*` names, copy them across with a span processor that runs before the exporting one:++```typescript+import type { ReadableSpan, SpanProcessor } from "@opentelemetry/sdk-trace-base";++class CopyGenAiContent implements SpanProcessor {+ onStart() {}+ onEnd(span: ReadableSpan) {+ const attrs = span.attributes as Record<string, unknown>;+ const copy = (target: string, ...sources: string[]) => {+ if (attrs[target] !== undefined) return;+ for (const source of sources) {+ const value = attrs[source];+ if (value === undefined) continue;+ attrs[target] = typeof value === "string" ? value : JSON.stringify(value);+ return;+ }+ };+ copy("gen_ai.prompt", "gen_ai.input.messages", "ai.prompt.messages");+ copy("gen_ai.completion", "gen_ai.output.messages", "ai.response.text");+ }+ async forceFlush() {}+ async shutdown() {}+}++// Order matters: this must come before the batch processor that exports.+new NodeSDK({+ spanProcessors: [new CopyGenAiContent(), new BatchSpanProcessor(exporter)],+});+```++### 3. `gen_ai.*` belongs on the model-call span and nowhere else++The `gen_ai.*` rule is the last one Traceway evaluates, so three earlier rules silently win the span:++- A root `SERVER` or `INTERNAL` span with any HTTP attribute (`http.route`, `http.request.method`, `url.path`) becomes an **Endpoint**. Never decorate the `/api/chat` route span with `gen_ai.conversation.id` or `user.id`, or the model call disappears from AI Traces.+- A `CONSUMER` span becomes a **Task**. An LLM call inside a queue worker needs its own child span, it cannot ride on the consumer span.+- An `INTERNAL` span carrying `exception.*` attributes is dropped entirely. The Issue is still recorded, the AI trace is not.++Use `SpanKind.CLIENT` for model calls, as the skeleton below does. That is not decoration, it is what keeps a failed call's AI trace alive.## What you get when this is wired correctly@@ -9,16 +72,20 @@ The AI Traces section of the dashboard has three tabs. Per-call spans alone only| Tab | Needs | Shows ||---|---|---|| **Traces** | any `gen_ai.*` span | per-agent call counts, latency, tokens, cost |-| **Conversations** | `gen_ai.conversation.id` | turns, cost, tokens, tool calls per conversation; flagged-content badges; P95 outlier highlighting; tool/model/user filters |-| **Users** | `gen_ai.conversation.id` + `user.id` | per-customer analytics: conversation count, median/avg/min conversation length, cost per conversation, flagged conversations |+| **Conversations** | `gen_ai.conversation.id` for real grouping | turns, cost, tokens, tool calls per conversation; flagged-content badges; P95 outlier highlighting; tool/model/user filters |+| **Users** | `user.id` | per-customer analytics: conversation count, median/avg/min conversation length, cost per conversation, flagged conversations |++The Conversations tab is never literally empty. When `gen_ai.conversation.id` is absent Traceway falls back to the OTel trace id, so every HTTP request becomes its own one-turn "conversation". That is the failure mode to look for: a long list of UUID-named, one-turn rows rather than a blank page.## Conversation identityTraceway resolves a conversation id per call, in this order:-1. `gen_ai.conversation.id` span attribute — set this explicitly; it is the reliable path.+1. `gen_ai.conversation.id` span attribute. Set this explicitly, it is the reliable path.2. `session.id` span or resource attribute.-3. The OTel trace id — a single agent run inside one request groups automatically, but turns arriving in later requests will NOT join it.+3. The OTel trace id, stored in dashed-UUID form (trace `331b1aca5ef21647145c51e9dc39c601` becomes conversation `331b1aca-5ef2-1647-145c-51e9dc39c601`). Every AI span in one trace groups together, so a single agent run inside one request groups automatically, but turns arriving in later requests will NOT join it.++An empty-string `gen_ai.conversation.id` counts as absent and falls through to the next level, so a nullish chat id degrades quietly to per-request grouping instead of failing loudly.Use the id your app already has for the chat session: the chat/thread id from your database, or the session id the frontend holds. Stable for the life of the conversation, different between conversations. Every model call in the agent loop (including retries and sub-agent calls) must carry the same value.@@ -26,7 +93,7 @@ Use the id your app already has for the chat session: the chat/thread id from yo`user.id` is the customer dimension for the Users tab and the user filter on Conversations. Set it on every model-call span.-**Good values** — a stable identifier for the end user of YOUR product:+**Good values** are stable identifiers for the end user of YOUR product:- your internal account/user id (best),- the tenant or organization id for B2B products where the account is the unit you care about,@@ -34,75 +101,162 @@ Use the id your app already has for the chat session: the chat/thread id from yo**Bad values:**-- session ids, request ids, or anything random per conversation — that is `gen_ai.conversation.id`'s job; using it here makes every user look like a new user and destroys the per-customer medians,+- session ids, request ids, or anything random per conversation (that is `gen_ai.conversation.id`'s job; using it here makes every user look like a new user and destroys the per-customer medians),- device ids that churn across reinstalls,-- leaving it unset — the Users tab stays empty and flagged conversations cannot be attributed.+- leaving it unset (the Users tab stays empty and flagged conversations cannot be attributed).-**Privacy**: the value is stored on every AI trace row and is searchable and filterable by anyone with project access. If PII must not enter telemetry, use your internal id or a hash of it — the analytics only need stability, not readability.+**Privacy**: the value is stored on every AI trace row and is searchable and filterable by anyone with project access. If PII must not enter telemetry, use your internal id or a hash of it. The analytics only need stability, not readability.## Tool calls-Traceway parses tool calls out of the completion payload (`gen_ai.completion`), so the shape matters. Any of these work:+Traceway parses tool calls out of the resolved completion payload: `gen_ai.completion`, else `trace.output`, else `span.output`. The exact nesting is what gets matched, and a near miss silently yields zero tool calls. Exactly these three shapes work:-- OpenAI chat completions: `{"choices":[{"message":{"tool_calls":[{"function":{"name":...,"arguments":...}}]}}]}`-- Anthropic messages: `{"content":[{"type":"tool_use","name":...,"input":...}]}`-- OTel GenAI output messages: `[{"role":"assistant","parts":[{"type":"tool_call","name":...}]}]`+- **OpenAI chat completions**: `{"choices":[{"message":{"tool_calls":[{"function":{"name":"get_weather","arguments":"{...}"}}]}}]}`. The `choices[].message` wrapper is required. A bare `{"tool_calls":[...]}` parses to zero.+- **Anthropic messages**: `{"content":[{"type":"tool_use","name":"get_weather","input":{...}}]}`. The `content` object wrapper is required. A top-level `[{"type":"tool_use",...}]` array parses to zero.+- **OTel GenAI output messages**: `[{"role":"assistant","parts":[{"type":"tool_call","name":"get_weather"}]}]`. The `name` must sit on the part itself; nesting it under a `tool_call` object gives you a count with no name. This shape feeds tool counts and filters, but the chat timeline cannot render it, so prefer one of the two provider shapes if you want the conversation view.-Set `gen_ai.completion` to the raw provider response (or the shapes above) and tool names/counts appear on the conversation rows, become filterable, and render in the chat view with arguments and results.+The simplest correct choice is to serialize the provider's raw response: `JSON.stringify(response)` for Anthropic, `JSON.stringify({ choices: response.choices })` for OpenAI. Then tool names and counts appear on the conversation rows, become filterable, and render in the chat view with arguments and results.Two rules for the tool executions themselves:-1. **Wrap each tool execution in a plain child span** (name it `tool <name>`, attributes like `tool.name`, `tool.args`) so it shows in the trace waterfall.-2. **Do NOT put `gen_ai.*` attributes on tool-execution spans.** Any `gen_ai.*` attribute promotes the span to its own AI Trace row, which inflates the conversation's turn count. The single exception is a span that represents a standalone tool-execution step in the OTel semconv style (`gen_ai.operation.name = "execute_tool"` + `gen_ai.tool.name`) — Traceway counts that as a tool call, but the completion-payload path above is preferred.+1. **Wrap each tool execution in a plain child span of the enclosing request or task span, not of the model-call span.** Name it `tool <name>` with attributes like `tool.name` and `tool.args`. Traceway re-roots every unpromoted child span onto its nearest promoted ancestor, and AI Traces have no waterfall view, so a tool span parented to the model call is stored and then rendered nowhere.++ ```+ POST /api/chat SERVER -> Endpoint (has a Spans tab)+ ├── chat gpt-4o CLIENT -> AI Trace (turn 1)+ ├── tool lookup_order INTERNAL -> Span, shown on the Endpoint's Spans tab OK+ └── db SELECT orders CLIENT -> Span, shown on the Endpoint's Spans tab OK++ POST /api/chat SERVER -> Endpoint+ └── chat gpt-4o CLIENT -> AI Trace+ └── tool lookup_order INTERNAL -> re-rooted onto the AI Trace, rendered nowhere BAD+ ```++ In practice: end the model-call span before you run the tools, or start the tool span from the request context instead of from inside `startActiveSpan` of the model call. The AI Trace still links back to its Endpoint through the shared distributed trace id either way.++2. **Do NOT put `gen_ai.*` attributes on tool-execution spans.** Any `gen_ai.*` attribute, including `gen_ai.operation.name = "execute_tool"`, promotes the span to its own AI Trace row. If it also carries the conversation id it becomes an extra *turn*, and if the model span already reported that tool in its completion payload the tool is counted twice: one model call plus one `execute_tool` span in the same conversation reports `turns: 2, toolCallCount: 2`. Use `execute_tool` plus `gen_ai.tool.name` only when you have no completion payload at all and accept one AI-trace row per tool invocation. Otherwise keep tool spans free of `gen_ai.*` and let the completion payload carry the names.## Sub-agents-When the main agent delegates to sub-agents (a researcher, a planner, a summarizer), give each sub-agent's model calls their own `trace.name` ("Research Sub-Agent") while keeping the SAME `gen_ai.conversation.id` and `user.id`. Each agent then appears separately on the Traces tab with its own latency/cost profile, while the Conversations tab shows the whole delegation as one conversation, with the sub-agent turns interleaved in the chat timeline.+When the main agent delegates to sub-agents (a researcher, a planner, a summarizer), give each sub-agent's model calls their own `trace.name` ("Research Sub-Agent") while keeping the SAME `gen_ai.conversation.id` and `user.id`. Each agent then appears separately on the Traces tab with its own latency and cost profile, while the Conversations tab aggregates the whole delegation as one conversation: turns ordered by time, models and tool names unioned across agents.++The single merged chat timeline needs one more thing: each turn's `gen_ai.prompt` must be a superset of the previous turn's message list, with the same role order. A sub-agent that sends its own system prompt or a compacted history breaks that match, and the conversation page falls back to rendering each turn as its own input/output block. That is cosmetic. Turn counts, costs and tool names are unaffected.## Putting it together (agent loop skeleton)```typescript+import { SpanKind, trace } from "@opentelemetry/api";++const tracer = trace.getTracer("agent");+const MODEL = "gpt-4o";++// Traceway has no server-side price list. You compute cost, in USD per token.+const PRICES: Record<string, { in: number; out: number }> = {+ "gpt-4o": { in: 2.5 / 1_000_000, out: 10 / 1_000_000 },+};+async function agentTurn(conversationId: string, userId: string, messages: Message[]) {- return tracer.startActiveSpan(`chat ${MODEL}`, { kind: SpanKind.CLIENT }, async (span) => {+ // CLIENT, not INTERNAL: an INTERNAL span carrying exception.* attributes is+ // dropped, so a failed call would lose its AI trace.+ const span = tracer.startSpan(`chat ${MODEL}`, { kind: SpanKind.CLIENT });+ try {const response = await llm.chat({ model: MODEL, messages, tools });+ const usage = response.usage;+ const price = PRICES[MODEL] ?? { in: 0, out: 0 };span.setAttributes({- "trace.name": "Support Chat Agent", // sub-agents use their own name here- "gen_ai.conversation.id": conversationId, // same for every call in the conversation- "user.id": userId, // stable end-customer id+ "trace.name": "Support Chat Agent", // sub-agents use their own name here+ "gen_ai.conversation.id": conversationId, // same for every call in the conversation+ "user.id": userId, // stable end-customer id"gen_ai.system": "openai",+ "gen_ai.operation.name": "chat","gen_ai.request.model": MODEL,"gen_ai.response.model": response.model,- "gen_ai.usage.input_tokens": response.usage.prompt_tokens,- "gen_ai.usage.output_tokens": response.usage.completion_tokens,+ "gen_ai.usage.input_tokens": usage.prompt_tokens, // integers, never strings+ "gen_ai.usage.output_tokens": usage.completion_tokens,+ "gen_ai.usage.input_cost": usage.prompt_tokens * price.in, // doubles+ "gen_ai.usage.output_cost": usage.completion_tokens * price.out,"gen_ai.response.finish_reason": response.choices[0].finish_reason,"gen_ai.prompt": JSON.stringify({ messages }),"gen_ai.completion": JSON.stringify({ choices: response.choices }),});- span.end();return response;- });+ } finally {+ // End before running the tools, so tool spans attach to the request span+ // rather than to this one.+ span.end();+ }}```-Each iteration of the tool-call loop is its own span/call — that is what "turns" counts. Resending the full message history in `gen_ai.prompt` each turn is expected; the conversation view deduplicates it.+Each iteration of the tool-call loop is its own span and call. That is what "turns" counts. Resending the full message history in `gen_ai.prompt` each turn is expected, the conversation view deduplicates it by role-sequence prefix. Include the assistant tool-call message and the `role: "tool"` result message in the resent history, because that is where the chat view gets tool results from.++**Cost is yours to compute.** Traceway stores whatever you send and has no model price list. Without `gen_ai.usage.input_cost` / `.output_cost` (or `.total_cost` directly) every cost figure stays at 0.00: the Conversations cost column, the Users cost-per-conversation and total cost, the P95 outlier highlighting, and the `ai_trace_cost` / `ai_conversation_cost` notification rules. Keep a small per-model price map in the app and multiply by the token counts. Send them as numeric OTLP attributes, integers for tokens and doubles for costs. A number sent as a JSON string is silently dropped and stored as 0, with no error on the export.++## What lights up what++When a dashboard column is blank, work backwards from here.++| Dashboard column | Attribute that fills it | Blank means |+|---|---|---|+| Traces tab grouping name | `trace.name`, else the span name | never blank |+| Model / Response model | `gen_ai.request.model` / `gen_ai.response.model` | attribute unset |+| Provider | `gen_ai.system`, else `gen_ai.provider.name` | neither set |+| Tokens | `gen_ai.usage.input_tokens` / `.output_tokens` (integers only); `total_tokens` defaults to their sum | unset, or sent as a string |+| Cached / Reasoning tokens | `gen_ai.usage.input_tokens.cached` / `gen_ai.usage.output_tokens.reasoning` | unset |+| Cost | `gen_ai.usage.input_cost` / `.output_cost` (doubles); `total_cost` defaults to their sum | you did not compute pricing |+| Finish reason | `gen_ai.response.finish_reason`, else `gen_ai.response.finish_reasons` (array, joined with commas) | unset |+| Conversation id | `gen_ai.conversation.id`, then `session.id` (span), then `session.id` (resource), then the trace id as a UUID | never blank |+| User | `user.id` | unset, and the Users tab stays empty |+| Tool count / names | tool calls parsed from `gen_ai.completion`, then `trace.output`, then `span.output`; else `gen_ai.operation.name=execute_tool` plus `gen_ai.tool.name` | payload not in one of the three shapes above |+| Chat timeline | `gen_ai.prompt`, then `trace.input`, then `span.input`, plus the completion chain above. The prompt must parse to `{"messages":[...]}` or to a bare array of messages | content on attribute names Traceway does not read, or a prompt in some other JSON shape |+| Flagged badge | matched at ingest against the project's packs and custom terms, over that same resolved input/output | no content stored, or no term matched |+| Everything else you set | kept in the trace's attribute bag, minus the standard keys above | n/a |## OpenRouter zero-code pathIf the app calls OpenRouter and uses the Broadcast/Observability feature instead of its own spans, the same analytics work through request-level fields:-- Pass `user` in the request body (up to 128 chars) — the end-customer id, same guidance as `user.id` above. Without it, calls attribute to the OpenRouter account itself, which makes the Users tab useless.-- Pass `session_id` in the request body (or the `x-session-id` header) for conversation grouping.-- Optional `trace` metadata (`trace_name`) names the agent.+- Pass `user` in the request body (up to 128 chars), the end-customer id, same guidance as `user.id` above. Without it, calls attribute to the OpenRouter account itself, which makes the Users tab useless.+- Pass `session_id` in the request body (up to 256 chars, also accepted as the `x-session-id` header) for conversation grouping. OpenRouter exports it as the OTLP `session.id` attribute, which is Traceway's second-level conversation key. If the app also emits its own spans, use the SAME value for `gen_ai.conversation.id` there, or one chat splits into two conversations.+- Optional `trace` metadata names the agent: `trace_name` for the root, `generation_name` for the individual call.++Full reference: https://docs.tracewayapp.com/client/openrouter## Content flagging (no code required)-Prompt and completion text is scanned at ingestion against per-project profanity language packs (en/de/es/fr/it/pt/sr) plus custom terms. Configure from the Conversations page's **Flagged terms** button. Nothing to instrument — it works as soon as `gen_ai.prompt`/`gen_ai.completion` are populated. Mention this to the user; custom terms (competitor names, compliance phrases) are usually worth setting up on day one.+Prompt and completion text is scanned at ingest against built-in profanity packs (en/de/es/fr/it/pt/sr) plus per-project custom terms. Nothing to instrument, it works as soon as `gen_ai.prompt` / `gen_ai.completion` are populated (or the `trace.input` / `trace.output` fallbacks). Configure it from the Conversations page's **Flagged terms** button, which opens the project settings AI tab and needs write access on the project.++Four things to tell the user:++- **Only `en` is on by default, and the pack list replaces it.** Selecting `es` turns English off. Select every language your users actually write in. Deselecting all leaves custom terms only.+- **Flags are computed at ingest and never backfilled.** Adding a term affects future calls only, so set custom terms up front.+- **Matching is whole-token and case-insensitive.** `acmecorp` matches "AcmeCorp" and "AcmeCorp-adjacent" but not "AcmeCorporation". Multi-word terms work as phrases. Terms are stored lowercased.+- **Limits:** 200 custom terms, 100 characters each, at most 20 distinct terms reported per call, and only the first 256 KB of each text is scanned.++Custom terms (competitor names, refund and cancellation phrases, compliance triggers) are usually worth setting up on day one.++## Payload size and retention++Each call's resolved input and output are written as one blob per AI trace to object storage (S3, or `STORAGE_PATH` on disk) under `ai-traces/<projectId>/<traceId>.json`. The row itself holds only the metrics. Because agents resend the whole history every turn, a 30-turn conversation stores roughly 30 copies of that history. Budget for it, and consider trimming long system prompts or retrieved context out of `gen_ai.prompt` if volume matters more than fidelity.++Read-side limits worth knowing: a conversation renders at most 1000 turns, and only the first 200 of them get their stored prompt and completion blobs attached, so beyond that the turns still show metrics but no chat body. AI trace rows age out with the project's telemetry retention (`SQLITE_RETENTION_DAYS` / `DUCKDB_RETENTION_DAYS`, default 30 days; no TTL on ClickHouse), while the payload blobs are not on that cleanup path.++## What this unlocks beyond the dashboard++Once conversation ids, costs and content are flowing, three event-driven notification rule types become usable (Settings, then Notification rules):++- **`ai_trace_cost`** fires when a single call exceeds a cost threshold. Needs `gen_ai.usage.*_cost`.+- **`ai_conversation_cost`** fires when one conversation's rolling 24h cost crosses a threshold. Needs `gen_ai.conversation.id` and the cost attributes. Without a real conversation id the trace-id fallback makes this meaningless, since every request looks like a fresh conversation.+- **`ai_flagged_content`** fires on a flagged-term match, optionally filtered to specific terms. Needs the content attributes populated and the terms configured.++Mention these when you finish the integration. A runaway-cost alert on a customer-facing agent is usually worth setting up the same day.## VerifyAfter wiring, run one real multi-turn chat with a tool call, then check in the dashboard (or via the API):-1. **AI Traces -> Conversations**: the conversation appears, turn count matches the number of model calls, tool names are listed.-2. **AI Traces -> Users**: the test user appears with 1 conversation.-3. Open the conversation: the chat timeline renders with tool calls inline; each turn links to its call.-4. If any of these are empty, the corresponding attribute is missing — conversation id (tab 1 empty), user id (tab 2 empty), or completion payload shape (tool columns empty).+1. **AI Traces, Traces tab**: your `trace.name` appears with the right call count. Nothing here at all means no span carries a `gen_ai.*` attribute. Re-check precondition 1, and check you did not hang the attributes on the HTTP route span.+2. **AI Traces, Conversations tab**: exactly ONE row for the chat, turn count matching the number of model calls, tool names listed, non-zero cost. A list of UUID-named one-turn rows means `gen_ai.conversation.id` is missing and Traceway fell back to the trace id. Cost 0.00 means no `gen_ai.usage.*_cost` attributes. Blank tool columns mean the completion payload is not in one of the three exact shapes above.+3. **AI Traces, Users tab**: the test user appears with 1 conversation. Empty means `user.id` is unset.+4. Open the conversation: the chat timeline renders with tool calls inline, and each turn links to its call. Turns present but the chat body blank means the content landed on attribute names Traceway does not read (`gen_ai.input.messages`, `ai.prompt.messages`, and so on). Confirm with the Raw JSON toggle on that page.+5. Open the Endpoint that wrapped the chat request: the tool-execution spans appear on its Spans tab. Missing means they were parented to the model-call span instead of the request span.diff --git a/skills/traceway-setup/android.md b/skills/traceway-setup/android.mdindex b28df7f4..603c67f6 100644--- a/skills/traceway-setup/android.md+++ b/skills/traceway-setup/android.md@@ -37,17 +37,18 @@ class MyApp : Application() {}```-Register the `Application` class and add the `INTERNET` permission in `AndroidManifest.xml`, or reports cannot leave the device:+Register the `Application` class in `AndroidManifest.xml`, or `onCreate()` never runs and the SDK is never initialized:```xml<manifest xmlns:android="http://schemas.android.com/apk/res/android">- <uses-permission android:name="android.permission.INTERNET" /><application android:name=".MyApp" ...>...</application></manifest>```+No permission entry is needed. The SDK's own manifest inside the AAR already declares `INTERNET` and `ACCESS_NETWORK_STATE`, and the Android manifest merger folds both into the app. Re-declaring `INTERNET` yourself is harmless if the app already has it.+Wire the token through a build setting or `BuildConfig` field, not a committed literal. The connection string format is `<project-token>@https://<instance>/api/report`, identical to the other mobile SDKs.## Manual capture@@ -77,7 +78,7 @@ The last ~10 seconds of logs and actions ride along with each captured exceptionA release build with `minifyEnabled true` runs R8, which renames classes and methods and rewrites the line-number table, so production crash traces arrive obfuscated and stay unreadable until Traceway retraces them server-side against the build's `mapping.txt` (the Android equivalent of a JavaScript source map). Set this up whenever the app ships a minified release.-**Token.** Symbol uploads authenticate with the dedicated **upload token** (Connection page > Source Maps / Symbol Upload), NOT the project token from the connection string. Get it from Step 0; it is a CI secret, never committed. `readonly` members cannot generate one. Using the connection-string token here is rejected with a 401.+**Token.** Symbol uploads authenticate with the dedicated **upload token** (Connection page > Source Maps / Symbol Upload), NOT the project token from the connection string. It is a CI secret, never committed. `readonly` members cannot generate one. Using the connection-string token here is rejected with a 401.**Apply the plugin.** It is published to Maven Central, so add `mavenCentral()` to the `pluginManagement` repositories in `settings.gradle.kts`:@@ -94,6 +95,7 @@ Then apply it to the app module and point it at the instance:```kotlinplugins {+ id("com.android.application") // must already be appliedid("com.tracewayapp.symbols") version "1.0.1"}@@ -113,6 +115,8 @@ traceway {}```+Order matters in the `plugins { }` block. Gradle applies entries top to bottom, and `com.tracewayapp.symbols` looks for the Android extension the moment it is applied, so it has to come **after** `com.android.application` (or `com.android.library`). Declared above it, the build fails at configuration time with `com.tracewayapp.symbols must be applied after an Android application/library plugin`. The app module already has the Android plugin line, so add the Traceway line under it rather than pasting a new block above it.+**Wire the UUID into the SDK.** The plugin embeds a ProGuard UUID into `BuildConfig.TRACEWAY_PROGUARD_UUID` and names the uploaded mapping `<uuid>.txt`. The UUID is derived from the module path, variant, and app version (`versionName` + `versionCode`), so **bump the version each release** or a new upload overwrites the previous one (or set `proguardUuid` explicitly). Pass the injected value to the SDK so each reported crash carries the matching UUID:```kotlin@@ -132,7 +136,7 @@ Traceway.init(./gradlew assembleRelease uploadReleaseTracewaySymbols```-With `autoUpload = true` the upload runs automatically after `assembleRelease`. Keep the upload token in a CI secret — the `System.getenv` wiring in the `traceway { ... }` block above is what feeds it to the plugin (there is no built-in environment lookup):+With `autoUpload = true` the upload runs automatically after `assembleRelease`. Keep the upload token in a CI secret. The `System.getenv` wiring in the `traceway { ... }` block above is what feeds it to the plugin (there is no built-in environment lookup):```bashTRACEWAY_UPLOAD_TOKEN=<upload-token> ./gradlew assembleRelease uploadReleaseTracewaySymbolsdiff --git a/skills/traceway-setup/dashboard-project-setup.md b/skills/traceway-setup/dashboard-project-setup.mdnew file mode 100644index 00000000..f5685693--- /dev/null+++ b/skills/traceway-setup/dashboard-project-setup.md@@ -0,0 +1,134 @@+# Create the Traceway Project Structure++Use this guide only after "Analyze the Architecture" produced a repository inventory and "Propose and Confirm the Project Map" was confirmed by the user. Tailor every instruction to the detected components; do not hand the user a generic list of every supported platform.++## 1. Establish the Dashboard++Ask whether the user uses Traceway Cloud or a self-hosted instance only if the repository and conversation do not establish it.++- Traceway Cloud dashboard: https://cloud.tracewayapp.com+- Traceway Cloud registration: https://cloud.tracewayapp.com/register+- Self-hosted dashboard: `https://<their-instance>`++If the user does not have an account, direct them to registration and wait for them to finish account and organization setup. Do not ask for dashboard credentials.++## 2. Present the Creation Checklist++Before the user creates anything, show the exact planned structure:++| Order | Project name | Framework to select | Repository component | Signals | Credentials |+|---|---|---|---|---|---|+| 1 | `Product Backend` | OpenTelemetry | API, workers, scheduler, AI calls, backend hosts | Endpoints, tasks, AI traces, issues, logs, application and host metrics | Runtime token |+| 2 | `Product Web` | Detected browser framework | Browser application | Browser issues, web vitals, replay, source-mapped stacks | Runtime token + upload token |+| 3 | `Product Mobile` | Detected mobile framework | Independently released mobile app | Mobile issues/crashes, replay where supported, symbolicated stacks | Runtime token + upload token when symbols are needed |++The picker offers nine frameworks and no more: **OpenTelemetry**; **React**, **Svelte**, **Vue.js**, **jQuery**; **Flutter**, **React Native**, **Android**, **iOS**. Never tell the user to look for Gin, Django, Laravel, Symfony, Hono, Next.js, or Remix. Those entries do not exist, because every backend is OpenTelemetry and the browser half of a meta-framework is React.++Replace the examples with the confirmed names and omit rows that do not apply. For multiple browser or mobile applications, include one row per independently deployed or released application.++Explain the boundary in one paragraph: the backend project intentionally combines APIs, workers, tasks, AI calls, and server metrics so their telemetry and distributed traces stay together; browser and mobile runtimes use separate projects because they use platform SDKs, platform-specific dashboard behavior, and their own build artifacts.++## 3. Create Each Project++For each checklist row, walk the user through the current dashboard UI:++1. Open the Traceway dashboard.+2. Open the project selector in the header.+3. Select **Add Project**.+4. Choose the organization from the confirmed plan. The picker only appears when the user belongs to more than one organization they can write to; otherwise the organization is shown as fixed text and there is nothing to select.+5. Enter the proposed project name.+6. Select the framework from the checklist.+7. Select **New Project**.+8. Save the project token under the planned environment-variable name.+9. Select **Go to Connection** to review the tailored integration details.++Framework selection rules:++- Always select **OpenTelemetry** for a backend project, regardless of language or web framework. Go, Node, Python, PHP, Java, .NET, and Ruby backends all take this one option; the language and framework are chosen later on the Connection page, which then shows the tailored install and exporter setup.+- Select the browser framework for a browser project: **React**, **Svelte**, **Vue.js**, or **jQuery**. A meta-framework picks the framework it renders with: Next.js and Remix select **React** (Next.js browser setup is `@tracewayapp/react`), SvelteKit selects **Svelte**, Nuxt selects **Vue.js**.+- A browser application built on none of those four (vanilla JS, Angular, Solid, Preact, Astro, Web Components) still gets a **browser** project, not an OpenTelemetry one, because it uses `@tracewayapp/frontend` and the browser ingest path. Pick the closest listed browser value; it only decides which snippet the Connection page renders. Follow the plain-JS setup at https://docs.tracewayapp.com/client/js-sdk instead of that snippet.+- Select **Flutter**, **React Native**, **Android**, or **iOS** for a mobile project.+- A full-stack JS application with a production server gets an OpenTelemetry backend project and a separate browser project.++The backend enforces that last split; it is not just a recommendation. A project whose framework is React, Svelte, Vue.js, jQuery, or React Native discards every endpoint, task, AI trace, and span that arrives over OTLP. The request still answers `200`, so a server exporter pointed at a browser project's token looks healthy while its Endpoints page stays empty forever. Server telemetry must use the OpenTelemetry project's token.++## 4. Plan Credentials Without Exposing Them++Do not ask the user to paste tokens into chat. Ask them to set the values in their secret manager, deployment environment, local untracked environment file, or CI settings and then confirm that the variables exist.++| Credential | Used by | Handling |+|---|---|---|+| Project/runtime token | Backend OTLP exporter | Secret; store in the backend deployment environment. |+| Browser connection string | Browser SDK | Build-time public configuration; never reuse the backend token. Follow the framework's public environment-variable prefix. |+| Mobile connection string | Mobile SDK | Build configuration; never reuse the backend or browser token. |+| Upload token | Frontend source maps and mobile symbols/mappings | CI secret; generate per project from **Connection** -> **Source Maps** or **Symbol Upload**. Never embed it in the application. |++Runtime credentials take explicit component names. Adapt these to the repository's conventions:++```text+TRACEWAY_URL+TRACEWAY_BACKEND_TOKEN+PUBLIC_TRACEWAY_WEB_CONNECTION_STRING+TRACEWAY_MOBILE_CONNECTION_STRING+```++For Vite use a `VITE_` public prefix, for SvelteKit use `PUBLIC_`, and for Next.js use `NEXT_PUBLIC_`.++**Upload tokens do not get invented names.** Each uploader reads one fixed variable, so a component-specific name only works when it is passed through explicitly:++| Uploader | Reads | Also reads |+|---|---|---|+| `traceway-sourcemaps` (JS source maps) | `TRACEWAY_SOURCEMAP_TOKEN` | `TRACEWAY_URL` |+| `dart run traceway:upload_symbols` (Flutter) | `TRACEWAY_UPLOAD_TOKEN` | `TRACEWAY_URL` |+| iOS dSYM upload script | `TRACEWAY_UPLOAD_TOKEN` | `TRACEWAY_URL` |+| `com.tracewayapp.symbols` Gradle plugin (Android R8) | nothing, the plugin has no environment lookup | nothing. Wire both in the `traceway { }` block: `uploadToken = System.getenv("TRACEWAY_UPLOAD_TOKEN") ?: ""` and `url = "https://<instance>"` |++Name the CI secret exactly what the uploader reads, or keep a per-component secret and pass it on the command line (`--token "$TRACEWAY_WEB_UPLOAD_TOKEN"`, or `uploadToken = System.getenv("...")` in Gradle). A mismatch here fails at release time, not at setup time. Upload tokens never use a public prefix and never ship inside the application.++When one repository releases two mobile apps, the two upload tokens collide on `TRACEWAY_UPLOAD_TOKEN`; scope them per job in CI, or pass `--token` explicitly per build.++For an existing token, establish its project name and selected framework from the dashboard before mapping it. A token is opaque; never infer its platform from its value. Reuse it only when the project matches the confirmed row.++## 5. Offer the Relevant Documentation++Include only links that match the detected architecture:++Every backend guide lives under the OpenTelemetry section, so link the overview plus the one language page that matches:++| Integration | Documentation |+|---|---|+| Project structure | https://docs.tracewayapp.com/learn/projects |+| Backend OpenTelemetry (start here) | https://docs.tracewayapp.com/client/otel |+| Node.js | https://docs.tracewayapp.com/client/otel/nodejs |+| NestJS | https://docs.tracewayapp.com/client/otel/nestjs |+| Next.js (server side) | https://docs.tracewayapp.com/client/otel/nextjs |+| Hono | https://docs.tracewayapp.com/client/otel/hono |+| Cloudflare Workers | https://docs.tracewayapp.com/client/otel/cloudflare |+| Symfony | https://docs.tracewayapp.com/client/otel/symfony |+| Laravel | https://docs.tracewayapp.com/client/otel/laravel |+| Django | https://docs.tracewayapp.com/client/otel/django |+| Host metrics agent | https://docs.tracewayapp.com/learn/otel-agent |+| JavaScript source maps | https://docs.tracewayapp.com/client/js-sdk/sourcemap-upload |+| Plain JS and other browser frameworks | https://docs.tracewayapp.com/client/js-sdk |+| React | https://docs.tracewayapp.com/client/react |+| Svelte | https://docs.tracewayapp.com/client/svelte |+| Vue | https://docs.tracewayapp.com/client/vue |+| jQuery | https://docs.tracewayapp.com/client/jquery |+| Flutter | https://docs.tracewayapp.com/client/flutter |+| React Native | https://docs.tracewayapp.com/client/react-native |+| Android | https://docs.tracewayapp.com/client/android |+| iOS | https://docs.tracewayapp.com/client/ios |+| OpenRouter | https://docs.tracewayapp.com/client/openrouter |++## 6. Completion Gate++Before implementation, confirm:++- Every selected component has exactly one matching project.+- The backend project's framework is OpenTelemetry.+- Browser and mobile projects do not reuse the backend token.+- APIs, workers, schedulers, AI calls, and the host agent are mapped to the same backend token unless the user confirmed a real isolation boundary.+- Every frontend or symbolicated mobile build has its own upload token stored as a CI secret, under the name its uploader reads or passed explicitly with `--token`.+- Environment-variable names are agreed, even if live values are not available in the current shell.++Do not block code changes merely because live values are unavailable. Implement configuration through the agreed variables, then tell the user exactly which variables must be populated before live verification.diff --git a/skills/traceway-setup/data-model.md b/skills/traceway-setup/data-model.mdindex bb570f87..54127739 100644--- a/skills/traceway-setup/data-model.md+++ b/skills/traceway-setup/data-model.md@@ -9,10 +9,13 @@ This is the framework-agnostic reference for connecting any OpenTelemetry projec| Traces | `POST https://<instance>/api/otel/v1/traces` || Metrics | `POST https://<instance>/api/otel/v1/metrics` || Logs | `POST https://<instance>/api/otel/v1/logs` |+| Profiles | `POST https://<instance>/api/otel/v1development/profiles` (OTLP profiles, the development signal) || Auth | `Authorization: Bearer <project-token>` header on every request |-| Encoding | OTLP/HTTP — both `application/x-protobuf` and `application/json` are accepted. OTLP/gRPC is NOT supported. |+| Encoding | OTLP/HTTP: both `application/x-protobuf` and `application/json` are accepted. OTLP/gRPC is NOT supported. || Compression | `Content-Encoding: gzip` is supported (exporters default to no compression; enabling `compression: gzip` is fine). Max request body: 10 MB. |+Anything whose `Content-Type` is not `application/x-protobuf` or `application/protobuf` is parsed as OTLP/JSON, so a missing `Content-Type` header still works for JSON exporters. When ingest is saturated the endpoints answer `503` with a `Retry-After` header. That is a back-pressure signal to retry, not a misconfiguration.+## What the Dashboard ShowsTraceway turns OTel spans into five distinct concepts, each with its own dashboard page:@@ -29,16 +32,20 @@ Traceway turns OTel spans into five distinct concepts, each with its own dashboaFor every incoming span, Traceway applies these rules in order:-1. **Endpoint** — `SpanKind` is `SERVER` or `INTERNAL`, the span has at least one HTTP attribute (`http.request.method`, `http.method`, `http.route`, or `url.path`), AND it is either a root span (no parent) or its parent span is not present in the same export batch (cross-process tracing, e.g. behind a proxy that injects `traceparent`). One carve-out: an `INTERNAL` span carrying `exception.*` attributes is never promoted to an Endpoint, even with HTTP attributes present — browser SDKs (e.g. Honeycomb's) stamp page context like `url.path` onto their error spans, and those must become Issues, not endpoints.-2. **Task** — `SpanKind` is `CONSUMER`. This applies to ANY consumer span, root or not.-3. **Task** — a root `INTERNAL` span with a `console.command` attribute (CLI command instrumentation, e.g. Laravel/Symfony console).-4. **AI Trace** — the span has any attribute starting with `gen_ai.`.-5. **Child span** — anything else with a parent: stored as a generic span, attached to the nearest promoted ancestor (Endpoint/Task/AI Trace) by walking up the parent chain.-6. **Dropped** — anything else that is a root span. A root span that matches none of the rules above is silently discarded: no entity row, no span row. One exception: if the span carries an exception signal (an `exception` event or `exception.*` attributes), the exception is still captured as an Issue, linked by the raw OTel trace ID — only the span itself is lost.+1. **Endpoint**: `SpanKind` is `SERVER` or `INTERNAL`, the span has at least one HTTP attribute (`http.request.method`, `http.method`, `http.route`, or `url.path`), AND it is either a root span (no parent) or its parent span is not present in the same export batch (cross-process tracing, e.g. behind a proxy that injects `traceparent`). See the carve-out below.+2. **Task**: `SpanKind` is `CONSUMER`. This applies to ANY consumer span, root or not.+3. **Task**: a root `INTERNAL` span with a `console.command` attribute (CLI command instrumentation, e.g. Laravel/Symfony console).+4. **AI Trace**: the span has any attribute starting with `gen_ai.`.+5. **Child span**: anything else with a parent, stored as a generic span and attached to the nearest promoted ancestor (Endpoint/Task/AI Trace) by walking up the parent chain.+6. **Dropped**: anything else that is a root span. A root span that matches none of the rules above is silently discarded: no entity row, no span row. One exception: if the span carries an exception signal (an `exception` event or `exception.*` attributes), the exception is still captured as an Issue, linked by the raw OTel trace ID. Only the span itself is lost.++One carve-out is checked before all six rules: an `INTERNAL` span carrying `exception.*` **attributes** (`exception.type`, `exception.message` or `exception.stacktrace` set as span attributes, not as an `exception` event) is never promoted to anything. Not an Endpoint, not an AI Trace, not a `console.command` Task. The reason is browser SDKs (e.g. Honeycomb's), which stamp page context like `url.path` onto their zero-duration error spans, and those must become Issues, not endpoints. The consequence beyond browsers: do not set `exception.*` attributes on an `INTERNAL` span you also want promoted, or you lose the entity row and keep only the Issue. An agent that stamps `exception.type` onto its own `INTERNAL` `gen_ai.*` span loses the AI Trace row this way and gets only an Issue. Record the error as an `exception` event instead (`span.recordException(error)`), which does not suppress promotion.++A note on "the same export batch": Traceway resolves parents inside one `ResourceSpans` block, not across the whole request. Spans from one service normally share a Resource and land in one block, so this is invisible. But if an exporter splits a trace across Resource blocks, the children in the other block are treated as cross-process. They link by raw OTel trace ID, and a `SERVER`/`INTERNAL` child carrying HTTP attributes is promoted to a second Endpoint.The consequences of rule 6 are the most common integration bug: a custom root span created with `tracer.startActiveSpan("my-job")` and default `SpanKind.INTERNAL` produces nothing on the Endpoints or Tasks pages (errors recorded on it do still reach Issues, but with no Endpoint/Task to link back to). Background work must use `SpanKind.CONSUMER` (see Tasks below).-One project-level gate sits above all of these rules: a project whose framework is a frontend type (react, vue, svelte, ...) never promotes OTel spans to Endpoints/Tasks/AI Traces — only exceptions are extracted. Backend OTel exporters must point at a backend-framework project token.+One project-level gate sits above all of these rules: a project whose framework is one of the five browser SDK types (`react`, `svelte`, `vuejs`, `jquery`, `react-native`) never promotes OTel spans to Endpoints/Tasks/AI Traces, and stores no child span rows either. Only exceptions are extracted. Server-side JS frameworks (`nextjs`, `nestjs`, `express`, `remix`) and the mobile frameworks (`ios`, `android`, `flutter`) are NOT gated. Backend OTel exporters must point at a backend-framework project token.## Endpoints: Name and Route Parameters@@ -52,9 +59,9 @@ e.g. `GET /api/users/:id`. The pieces come from span attributes, with these exac- **Method**: `http.request.method` (current semconv), falling back to `http.method` (old semconv).- **Route**: `http.route`, falling back to `url.path`. A `http.route` value that does not start with `/` is ignored entirely (treated as absent).-- If the method is present but no route can be resolved, the span name is used as the route. If neither is present, the raw span name becomes the endpoint name.+- If the method is present but no route can be resolved, the span name is used as the route. If the method is missing, the route is discarded too and the raw span name alone becomes the endpoint name, even when `http.route` is set correctly. A method attribute is required for the route to be used at all.-### The cardinality quirk — `http.route` is mandatory+### The cardinality quirk: `http.route` is mandatory`http.route` must contain the **route pattern with parameter placeholders**, never the concrete URL:@@ -64,9 +71,13 @@ correct: http.route = /api/users/{id} -> one endpoint: "GET /api/users/{idwrong: http.route missing, url.path only -> thousands of endpoints: "GET /api/users/1", "GET /api/users/2", ...```-The placeholder style (`:id`, `{id}`, `<id>`) does not matter — Traceway uses the route string as-is. What matters is that the string is **identical for every request hitting that route**. If `http.route` is missing, Traceway falls back to `url.path`, which contains real IDs, and the Endpoints page explodes into one row per unique URL. Express, Hono (via `@hono/otel`), and Django instrumentations set `http.route` to the path template automatically — but verify it in the dashboard after setup; it's the #1 thing to check.+The placeholder style (`:id`, `{id}`, `<id>`) does not matter. Traceway uses the route string as-is. What matters is that the string is **identical for every request hitting that route**. If `http.route` is missing, Traceway falls back to `url.path`, which contains real IDs, and the Endpoints page explodes into one row per unique URL. Express and Hono (via `@hono/otel`) set `http.route` to the path template automatically, but verify it in the dashboard after setup; it's the #1 thing to check.++A route that does not start with `/` is discarded outright, as if the attribute were absent. Two real instrumentations trip over this rule:++Symfony caveat: the stock `open-telemetry/opentelemetry-auto-symfony` package sets `http.route` to the Symfony route *name* (e.g. `app_user_show`), not a path template. Traceway ignores it and falls back to `url.path`, so endpoint grouping for raw Symfony auto-instrumentation degrades to literal paths. Use the Traceway Symfony integration (see the Symfony guide) instead.-Symfony caveat: the stock `open-telemetry/opentelemetry-auto-symfony` package sets `http.route` to the Symfony route *name* (e.g. `app_user_show`), not a path template. Traceway deliberately ignores `http.route` values that don't start with `/` and falls back to `url.path` — so endpoint grouping for raw Symfony auto-instrumentation degrades to literal paths. Use the Traceway Symfony integration (see the Symfony guide) instead.+Django caveat: `opentelemetry-instrumentation-django` copies `http.route` straight from Django's URL pattern, which never has a leading slash (e.g. `api/users/<int:user_id>/`). Traceway discards it, and the default Django instrumentation emits no `url.path` either (it is still on the old HTTP conventions, which use `http.target` and `http.url`). With no method-and-route pair to build from, the endpoint name falls back to the span name, which already starts with the method. Rows then arrive as `GET GET api/users/<int:user_id>/`, one per concrete URL, and never group. See the Django guide for the fix.### Other endpoint attributes Traceway reads@@ -76,13 +87,16 @@ Symfony caveat: the stock `open-telemetry/opentelemetry-auto-symfony` package se| `http.response.body.size` | `http.response_content_length` | Response size || `client.address` | `net.peer.ip` | Client IP |+Note that `url.path` is the **only** route fallback Traceway reads. An instrumentation still on the old HTTP semantic conventions emits `http.target` and `http.url` instead of `url.path`, so it has no usable fallback at all: if its `http.route` is missing or unusable, there is nothing left to build a route from.+Quirks:-- **404 responses are renamed to `UNMATCHED`.** Any endpoint span with status 404 is grouped under a single `UNMATCHED` endpoint regardless of its route — bot scans and typo'd URLs don't pollute the endpoint list. Don't be surprised when a legitimate 404-returning route doesn't appear under its own name.-- **Missing status code becomes `0`.** Without `http.response.status_code`, error tracking and Apdex for that endpoint are meaningless. Make sure the instrumentation sets it.-- **Streaming endpoints** (long-lived responses that would otherwise look like terrible P99s) are detected when: status is `101` (WebSocket upgrade), the captured response header `http.response.header.content-type` contains `text/event-stream` (SSE), or the vendor attribute `traceway.is_stream` (boolean) is `true`. OTel clients don't capture response headers by default — for SSE endpoints either enable capture of `content-type` or set `traceway.is_stream` manually. Streaming endpoints keep their request count and error rate, but latency percentiles and Apdex are zeroed (connection lifetime is not request latency).-- **Health-check endpoints are filtered at ingestion** according to the project's health-check configuration, so `GET /health`-style routes don't pollute the endpoint list or stats.-- **Apdex thresholds are hardcoded, and they differ by page.** The Endpoints list buckets requests as satisfied up to 750 ms, tolerating up to 1.5 s, bad above that or on any 5xx — both boundaries shifted upward by the endpoint's slow-endpoint offset when one is configured (Endpoints page -> slow endpoint threshold). The endpoint detail page computes its displayed Apdex score with 500 ms / 2 s boundaries instead (no offset), and the apdex-drop notification rule uses flat 750 ms / 1.5 s. None of the thresholds are configurable.+- **404s collapse to `UNMATCHED` only when no real route matched.** A span with a proper `http.route` (starts with `/`, and is not a catch-all made only of slashes and wildcards like `/`, `/*`, `/**`) keeps its own endpoint name even on a 404, because a deliberate "not found" response on a real route is worth seeing under that route. The rewrite to a single `UNMATCHED` row happens when `http.route` is missing or unusable (only `url.path` available), or when the framework reported a catch-all route, which is what Express middleware, Spring resource handlers and not-found handlers emit for unmatched requests. That is what keeps bot scans and typo'd URLs out of the endpoint list.+- **Missing status code becomes `0`, and so does a wrongly typed one.** Traceway reads `http.response.status_code` only when it is an OTLP **integer** value. A status sent as a double or as a string is ignored and the endpoint is stored with status `0`, even though the attribute still shows "200" on the row. The same strictness applies everywhere: `http.route`, `http.request.method` and `db.statement` are read only as strings, and `traceway.is_stream` only as a real boolean (the string `"true"` does nothing). Standard OTel SDKs get the types right; hand-rolled OTLP/JSON exporters are where this goes wrong. Without a usable status code, error tracking and Apdex for that endpoint are meaningless.+- **Array- and map-valued attributes are dropped from the stored attribute map.** Only string, int, double and bool attribute values are kept (non-strings are stringified). An attribute whose value is an array or a nested key/value list never appears on the endpoint, task, span or exception row. Flatten it into scalar attributes if you need it in the dashboard.+- **Streaming endpoints** (long-lived responses that would otherwise look like terrible P99s) are detected when: status is `101` (WebSocket upgrade), the vendor attribute `traceway.is_stream` is a real boolean `true`, or the captured response header attribute (exactly the lowercase key `http.response.header.content-type`) has a value that *starts with* `text/event-stream`. So `text/event-stream; charset=utf-8` matches, but a combined `application/json, text/event-stream` does not. OTel clients don't capture response headers by default, and some instrumentations normalize the key differently (`Content-Type`, `content_type`), which Traceway will not read. For SSE endpoints, setting `traceway.is_stream` manually is the reliable path. Streaming endpoints keep their request count and error rate, but latency percentiles and Apdex are zeroed (connection lifetime is not request latency).+- **Successful health checks are dropped at ingestion, and the rule is broader than `/health`.** Projects are created with this on (`drop_healthy_healthchecks`, toggleable in project settings). An endpoint span is discarded, together with its child spans, when all of these hold: the method is `GET` or `HEAD`, the status code is below 400, and the path matches one of the built-in names (`/health`, `/healthz`, `/healthcheck`, `/health-check`, `/health_check`, `/ping`, `/livez`, `/readyz`, `/live`, `/ready`, `/alive`, `/up`, `/heartbeat`, `/status`, `/ht`, `/actuator/health`), starts with `/actuator/health/`, **ends with `/health`**, or matches a custom pattern configured on the project (`*` allowed at either end). Trailing slashes and case are ignored. Two consequences worth checking: a real application route named `/status` or `/up` will never appear on the Endpoints page, and a health check that fails (4xx/5xx) or that recorded an exception is always kept, which is what keeps outages visible.+- **Apdex thresholds are hardcoded, and they differ by page.** The Endpoints list buckets requests as satisfied up to 750 ms, tolerating up to 1.5 s, bad above that or on any 5xx. Both boundaries shift upward by the endpoint's slow-endpoint offset when one is configured (Endpoints page -> slow endpoint threshold). The endpoint detail page computes its displayed Apdex score with 500 ms / 2 s boundaries instead (no offset), and the apdex-drop notification rule uses flat 750 ms / 1.5 s. None of the thresholds are configurable.## Tasks: Scheduled Jobs, Cron, Queue Consumers@@ -115,19 +129,21 @@ async function runScheduledJob() {Quirks:-- **The span name IS the task name, and Traceway groups tasks by name.** Use a stable identifier like `cleanup-expired-sessions` or `process-email-queue`. Never put job IDs, timestamps, or user IDs in the span name — each unique name becomes a separate task group.-- **Every `CONSUMER` span becomes a Task, even non-root ones.** If a queue library's auto-instrumentation already emits `CONSUMER` spans (e.g. Kafka/RabbitMQ consumers, Symfony Messenger), do NOT wrap them in another `CONSUMER` span — you'd get duplicate Task entries.-- A root span with `SpanKind.INTERNAL` (the default!) is **dropped silently** — this is the most common reason "my cron job doesn't show up". The kind must be `CONSUMER`. (An exception recorded on the dropped span still reaches Issues, but the task run itself — name, duration, history — is lost.)+- **The span name IS the task name, and Traceway groups tasks by name.** Use a stable identifier like `cleanup-expired-sessions` or `process-email-queue`. Never put job IDs, timestamps, or user IDs in the span name. Each unique name becomes a separate task group.+- **Every `CONSUMER` span becomes a Task, even non-root ones.** If a queue library's auto-instrumentation already emits `CONSUMER` spans (e.g. Kafka/RabbitMQ consumers, Symfony Messenger), do NOT wrap them in another `CONSUMER` span. You'd get duplicate Task entries.+- A root span with `SpanKind.INTERNAL` (the default!) is **dropped silently**. This is the most common reason "my cron job doesn't show up". The kind must be `CONSUMER`. (An exception recorded on the dropped span still reaches Issues, but the task run itself is lost: name, duration, history.)+- **Tasks are timestamped when they finish, endpoints when they start.** A task run lands in the time bucket of its end time, so a nightly job triggered at 01:00 and finishing at 03:00 shows up at 03:00. Widen the time range when a long-running job seems missing.- Per-job context (job ID, batch size) belongs in span **attributes**, where it shows on the task detail page without affecting grouping.## Database Queries and Child Spans-Work inside an endpoint or task — DB queries, cache hits, outgoing HTTP calls, custom business logic — should be **child spans** of the entity's root span. They render in the waterfall on the Endpoint/Task detail page.+Work inside an endpoint or task (DB queries, cache hits, outgoing HTTP calls, custom business logic) should be **child spans** of the entity's root span. They render in the waterfall on the Endpoint/Task detail page.How Traceway handles them:-- **The span name is replaced by the SQL text when present**: if the span has `db.query.text` (current semconv) or `db.statement` (old semconv), that value is displayed instead of the span name. So a `pg.query` span carrying `db.statement = "SELECT * FROM users WHERE id = $1"` shows the query itself in the waterfall — this is what makes auto-instrumented DB clients (`pg`, `mysql2`, `mongodb`, `ioredis`, Prisma via `@prisma/instrumentation`) useful out of the box.-- **Child spans attach to the nearest promoted ancestor.** The parent chain is walked until an Endpoint/Task/AI Trace is found. This means DB spans must be created **inside the active context** of the request/task span (`startActiveSpan`, or the framework middleware's context). A DB span created outside any active context becomes a root `INTERNAL` span — and is dropped.+- **The span name is replaced by the SQL text when present**: if the span has `db.query.text` (current semconv) or `db.statement` (old semconv), that value is displayed instead of the span name. So a `pg.query` span carrying `db.statement = "SELECT * FROM users WHERE id = $1"` shows the query itself in the waterfall. This is what makes auto-instrumented DB clients (`pg`, `mysql2`, `mongodb`, `ioredis`, Prisma via `@prisma/instrumentation`) useful out of the box.+- **Child spans attach to the nearest promoted ancestor.** The parent chain is walked until an Endpoint/Task/AI Trace is found. This means DB spans must be created **inside the active context** of the request/task span (`startActiveSpan`, or the framework middleware's context). A DB span created outside any active context becomes a root `INTERNAL` span, which is dropped.+- **A promoted span is never also a child span.** Once a span becomes an Endpoint, Task or AI Trace it leaves the waterfall of whatever contains it, and its own children re-root under it. So a `CONSUMER` span created inside a request handler (an inline queue publish, a Symfony Messenger sync dispatch) disappears from that endpoint's Spans tab and shows up on the Tasks page instead, with its sub-spans attached to the Task. If you want the work to stay in the request waterfall, use a plain child span (default `INTERNAL`), not `CONSUMER`.- Spans whose parent lives in another process/batch and never resolves to a promoted entity are kept but linked by the raw OTel trace ID instead.- For databases without auto-instrumentation (e.g. SQLite), create a manual child span and set `db.system` + `db.statement` attributes so it renders like the auto-instrumented ones.@@ -135,23 +151,33 @@ How Traceway handles them:Errors become **Issues** when recorded as a span **event named `exception`** (`span.recordException(error)` does exactly this) with the standard attributes:-- `exception.type` — error class, e.g. `TypeError`-- `exception.message` — error message-- `exception.stacktrace` — full stack trace text+- `exception.type`: error class, e.g. `TypeError`+- `exception.message`: error message+- `exception.stacktrace`: full stack trace text++Setting the same `exception.*` keys as plain span **attributes** (instead of an event) also works. Traceway builds an Issue from the attributes when no `exception` event is present on the span.++Every span event named `exception` produces its own Issue occurrence, so a span that records three exceptions writes three occurrences. The attribute path is used only when the span has no `exception` event at all.++The stack-trace text that gets hashed is picked in this order:++1. Honeycomb-style structured frames, when `exception.structured_stacktrace.urls` is present (with `.functions` / `.lines` / `.columns`). Used by some browser and edge SDKs.+2. `exception.stacktrace`.+3. Android/JVM structured frames, when `exception.structured_stacktrace.classes` is present (with `.methods` / `.source_files` / `.lines`).-Setting the same `exception.*` keys as plain span **attributes** (instead of an event) also works — Traceway builds an Issue from the attributes when no `exception` event is present on the span.+An `exception.type: exception.message` header is prepended unless the stack trace already starts with the type, which JVM agents do.Quirks:-- **Exceptions survive span-dropping, but their context doesn't.** An exception recorded on an unpromoted root span (see classification rule 6) still becomes an Issue, but there is no Endpoint/Task for it to link to — the occurrence carries only the raw trace ID. Record exceptions on spans that live inside an Endpoint/Task, or fix the span kind, so Issues stay correlated with the request or job that raised them.-- Traceway also understands Honeycomb-style structured stack traces (`exception.structured_stacktrace.urls` / `.functions` / `.lines` / `.columns` array attributes) — used by some browser/edge SDKs.-- **Grouping is automatic.** Before hashing (SHA-256, truncated to 16 hex chars), Traceway normalizes the stack trace: error messages are stripped (only the error type is kept, including on `Caused by:` lines), JS function-name lines collapse to `<fn>`, absolute paths reduce to `filename:line`, dependency version suffixes (`@v1.2.3`) are removed, and runtime values are replaced with placeholders — hex addresses -> `<hex>`, UUIDs -> `<uuid>`, numbers of 5+ digits -> `<id>`, emails -> `<email>`, IPs -> `<ip>`, goroutine IDs -> `goroutine <n>`. JVM frames additionally lose their line numbers (`(Foo.java:123)` -> `(Foo.java)`) and `... N more` becomes `... more`. The same logical error therefore groups into one Issue even when runtime values differ — don't try to pre-group errors client-side.-- **JS projects get source-map symbolication.** Triggered when the `telemetry.sdk.language` resource attribute is a JS value (`nodejs`, `webjs`, `javascript`, `typescript` — OTel SDKs set this automatically), or when the instrumentation scope name is npm-scoped (`@scope/pkg`) or `next.js`. Minified frames are resolved server-side **before** grouping, so uploading source maps improves grouping too.-- **Source maps are matched by filename (and debug ID when embedded), not by version.** Upload via `POST /api/sourcemaps/upload` (multipart `files` field, `.map`/`.js`/`.cjs`/`.mjs`, max 50 MB per file) using a token from `POST /api/projects/source-map-token`. A frame referencing `app.min.js` resolves against the uploaded `app.min.js.map` by name — keep bundle filenames unique per release (content hashes do this) or rely on debug IDs.+- **Exceptions survive span-dropping, but their context doesn't.** An exception recorded on an unpromoted root span (see classification rule 6) still becomes an Issue, but there is no Endpoint/Task for it to link to. The occurrence carries only the raw trace ID. Such an occurrence is stored with trace type `task`, which is why orphaned errors look task-shaped in the UI. Record exceptions on spans that live inside an Endpoint/Task, or fix the span kind, so Issues stay correlated with the request or job that raised them.+- **`exception.stacktrace` is stripped from every stored attribute map.** The raw blob lives in the Issue's stack trace field only, never duplicated onto the endpoint, task, span or exception rows. Don't re-add it under another key. For JS and Android telemetry the exception's attribute map additionally carries `telemetry.sdk.language`, copied down from the Resource.+- **Grouping is automatic.** Before hashing (SHA-256, truncated to 16 hex chars), Traceway normalizes the stack trace: error messages are stripped (only the error type is kept, including on `Caused by:` lines), JS function-name lines collapse to `<fn>`, URL origins are removed so the same bundle served from different hosts or CDNs groups together, absolute paths reduce to `filename:line`, column numbers are dropped from frames on line 2 or later (the column survives only on line 1, where minified bundles put everything and it is the only thing telling frames apart), dependency version suffixes (`@v1.2.3`) are removed, and runtime values are replaced with placeholders: hex addresses -> `<hex>`, UUIDs -> `<uuid>`, standalone numbers of 5+ digits -> `<id>` (never a number preceded by `:`, so `handler.go:123456` line numbers stay significant), emails -> `<email>`, IPs -> `<ip>`, goroutine IDs -> `goroutine <n>`. JVM frames additionally lose their line numbers (`(Foo.java:123)` -> `(Foo.java)`, same for `.kt` and `.scala`) and `... N more` becomes `... more`. The same logical error therefore groups into one Issue even when runtime values differ, so don't try to pre-group errors client-side.+- **JS projects get source-map symbolication, once the project has a source-map token.** Symbolication is triggered when the `telemetry.sdk.language` resource attribute is a JS value (`nodejs`, `webjs`, `javascript` or `typescript`, all set automatically by OTel SDKs), or when the instrumentation scope name is npm-scoped (`@scope/pkg`) or `next.js`. It is a silent no-op until the project has a source-map token: the trace is canonicalized and stored unresolved, with no error and no warning, and uploaded maps are never consulted. Only iOS projects get a token automatically, so for an OpenTelemetry project generate one with `POST /api/projects/source-map-token` first. Minified frames are resolved server-side **before** grouping, so uploading source maps improves grouping too.+- **Source maps are matched by filename (and debug ID when embedded), not by version.** Upload via `POST /api/sourcemaps/upload` (multipart `files` field, `.map`/`.js`/`.cjs`/`.mjs`, max 50 MB per file) using a token from `POST /api/projects/source-map-token`. A frame referencing `app.min.js` resolves against the uploaded `app.min.js.map` by name. Keep bundle filenames unique per release (content hashes do this) or rely on debug IDs.## Metrics-OTLP metrics sent to `/api/otel/v1/metrics` are stored under their **original names** — there is no renaming or namespacing. Every incoming metric is auto-registered (name, unit, type) in the metric registry, becomes discoverable in the dashboard's metric explorer, and can be charted in custom widgets or queried via `traceway metrics query --name <name>`.+OTLP metrics sent to `/api/otel/v1/metrics` are stored under their **original names**. There is no renaming or namespacing. Every incoming metric is auto-registered (name, unit, type) in the metric registry, becomes discoverable in the dashboard's metric explorer, and can be charted in custom widgets or queried via `traceway metrics query --name <name>`.Conversion rules per OTLP metric type:@@ -160,19 +186,19 @@ Conversion rules per OTLP metric type:| Gauge | `<name>`, value as-is | `gauge` || Sum (monotonic) | `<name>`, value as-is | `counter` || Sum (non-monotonic) | `<name>`, value as-is | `gauge` |-| Histogram | TWO series: `<name>.avg` (sum/count per export) and `<name>.count` | `gauge` + `counter` |-| ExponentialHistogram, Summary | **silently dropped** | — |+| Histogram | `<name>.count` always; `<name>.avg` (sum/count per export) only when the data point has a non-zero count and a sum | `gauge` for `.avg` (keeps the source unit) + `counter` for `.count` (unit forced to `count`) |+| ExponentialHistogram, Summary | **silently dropped** | - |Quirks:-- **Histograms lose their buckets.** Only the average and count survive — you cannot compute true percentiles from an OTLP histogram in Traceway.-- **No percentile aggregations on metrics at all.** Metric queries support `avg`, `sum`, `count`, `min`, and `max` only; anything else silently falls back to `avg`. Latency percentiles (P50/P95/P99) exist for Endpoints and Tasks because those are computed from raw span durations — if you need percentiles on a measurement, model it as a span, not a metric.-- **Tags come from data-point attributes only.** Resource attributes are NOT copied onto metric points, with two exceptions: `service.name` becomes the `server_name` tag, and the process-scraper allowlist (`process.pid`, `process.executable.name`, `process.command_line`, `process.owner`) is lifted so hostmetrics per-process series stay distinguishable. Any per-series dimension must be a data-point attribute, and keep its cardinality low.-- **Raw metric points expire after 7 days** (ClickHouse TTL). Queries over wider ranges read pre-aggregated rollups instead — 1-minute rollups are kept 30 days, 1-hour rollups 1 year, 1-day rollups indefinitely — so older metrics lose granularity rather than disappearing.+- **Histograms lose their buckets.** Only the average and count survive. You cannot compute true percentiles from an OTLP histogram in Traceway. An idle export interval (count 0) writes only the `.count` series, so `<name>.avg` has gaps rather than zeros.+- **No percentile aggregations on metrics at all.** Metric queries support `avg`, `sum`, `count`, `min`, `max` and `last` only; anything else silently falls back to `avg`. `last` is special: it always reads the raw table, so it only works inside the 7-day raw retention window. Latency percentiles (P50/P95/P99) exist for Endpoints and Tasks because those are computed from raw span durations. If you need percentiles on a measurement, model it as a span, not a metric.+- **Tags come from data-point attributes plus a small resource allowlist.** Resource attributes are not copied onto metric points, with these exceptions. `service.name` becomes the `server_name` tag (and overwrites a data-point attribute of that name). And this fixed allowlist is lifted so per-process, per-container and per-pod series stay distinguishable when the receiver puts identity on the Resource: `process.pid`, `process.executable.name`, `process.command_line`, `process.owner`, `container.name`, `container.image.name`, `k8s.pod.name`, `k8s.namespace.name`, `k8s.node.name`, `k8s.deployment.name`, `k8s.container.name`, `postgresql.database.name`. Every other resource attribute (`host.name`, `deployment.environment`, and so on) is dropped. A data-point attribute wins over the resource value when both use the same key. Any other per-series dimension must be a data-point attribute, and keep its cardinality low.+- **Raw metric points expire after 7 days** (ClickHouse TTL). Queries over wider ranges read pre-aggregated rollups instead (1-minute rollups are kept 30 days, 1-hour rollups 1 year, 1-day rollups indefinitely), so older metrics lose granularity rather than disappearing.### Built-in metric names-The dashboard's built-in system charts (Metrics page, default widget suggestions) read these **exact hardcoded names** — they are emitted natively by the Traceway Go SDK:+The dashboard's built-in system charts (Dashboards page templates, default widget suggestions) read these **exact hardcoded names**:| Name | Meaning | Unit ||---|---|---|@@ -184,30 +210,32 @@ The dashboard's built-in system charts (Metrics page, default widget suggestions| `go.num_gc` | Total GC cycles | count || `go.gc_pause` | Last GC pause | nanoseconds |-An OTel project's metrics (e.g. hostmetrics' `system.cpu.utilization`) do NOT populate those built-in charts — they appear as custom metrics under their own names, chartable via custom widgets. To fill the built-in CPU/memory charts from an OTel pipeline, the metrics must be emitted under the exact names and units above.+An OTel project's metrics (e.g. hostmetrics' `system.cpu.utilization`) do NOT populate those built-in charts. They appear as custom metrics under their own names, chartable via custom widgets. To fill the built-in CPU/memory charts from an OTel pipeline, the metrics must be emitted under the exact names and units above.## Logs-OTLP logs sent to `/api/otel/v1/logs` appear on the **Logs** page. From each LogRecord, Traceway reads: timestamp (falling back to observed timestamp), severity number and text, body, trace ID and span ID, and three separate attribute maps — resource, scope, and log-record attributes — each independently filterable.+OTLP logs sent to `/api/otel/v1/logs` appear on the **Logs** page. From each LogRecord, Traceway reads: timestamp (falling back to `observedTimeUnixNano` when `timeUnixNano` is unset), severity number and text, body, trace ID and span ID (stored as hex), and three separate attribute maps (resource, scope, and log-record attributes), each independently filterable.-What the Logs page (and `traceway logs query`) can filter by: minimum severity, service name (from the `service.name` resource attribute), trace ID, free-text body search, and attribute key/value filters scoped to resource, scope, or log attributes.+What the Logs page (and `traceway logs query`) can filter by: minimum severity, service name (from the `service.name` resource attribute), trace ID, span ID, instrumentation scope name, free-text body search, and attribute key/value filters scoped to resource, scope, or log attributes.Quirks:-- **Emit logs inside the active span context.** The trace ID on a log record is what links it to the Endpoint/Task that produced it — OTel log bridges do this automatically when a span is active.-- **Body search over ranges longer than 24 hours requires at least one additional filter** (service, severity, trace, or attribute) — unbounded full-text scans are rejected.+- **Emit logs inside the active span context.** The trace ID on a log record is what links it to the Endpoint/Task that produced it. OTel log bridges do this automatically when a span is active.+- **Severity text is derived and uppercased.** `severityText` is stored upper-cased. When the record carries none, Traceway derives it from the severity number: `0` gives an empty string, `1-4` TRACE, `5-8` DEBUG, `9-12` INFO, `13-16` WARN, `17-20` ERROR, `21` and above FATAL. Those buckets do not match the OTel names one for one (WARN2 through WARN4 all become WARN), so filter by severity *number* when you need precision.+- **Non-string bodies are coerced to text.** Numbers, booleans and bytes are stringified; arrays and key/value maps are JSON-encoded into the body string. A structured log body is therefore searchable as its JSON text.+- **Body search over ranges longer than 24 hours requires at least one additional filter** (service, severity, trace, or attribute). Unbounded full-text scans are rejected.- A JS `exception.stacktrace` attribute on a log record is source-map symbolicated, same as span exceptions.-- Logs are retained for 30 days (database TTL). Spans, endpoints, tasks, and exceptions have no fixed expiry on ClickHouse deployments; metrics roll up with their own TTLs (see Metrics above).+- Logs are retained for 90 days on ClickHouse deployments (table TTL). On self-hosted SQLite/DuckDB deployments a retention worker prunes them, defaulting to 30 days and configurable. Spans, endpoints, tasks, and exceptions have no fixed expiry on ClickHouse deployments; metrics roll up with their own TTLs (see Metrics above).## Resource Attributes-Set these once on the OTel `Resource` — they tag everything the service exports:+Set these once on the OTel `Resource`. They tag everything the service exports:| Resource attribute | Shown in Traceway as | Notes ||---|---|---|| `service.name` | Server Name | Set via `OTEL_SERVICE_NAME` or SDK config. Distinguishes instances/services within a project. || `service.version` | App Version | Enables release-over-release comparison. On Cloudflare Workers, derived from `cloudflare.script_version.id` automatically. |-| `telemetry.sdk.language` | — | Drives JS symbolication; set automatically by every OTel SDK. |+| `telemetry.sdk.language` | - | Drives JS symbolication; set automatically by every OTel SDK. |## Vendor Extension Attributes@@ -216,17 +244,17 @@ Traceway recognizes these non-standard span attributes:| Attribute | Type | Purpose ||---|---|---|| `traceway.is_stream` | boolean | Mark an endpoint span as streaming (SSE/long-poll) so its duration is excluded from latency percentiles. |-| `traceway.distributed_trace_id` | string (UUID) | Override the distributed trace ID used to link entities across services. Rarely needed — the OTel trace ID is used by default. |+| `traceway.distributed_trace_id` | string (UUID) | Override the distributed trace ID used to link entities across services. Rarely needed. The OTel trace ID is used by default. |## Verification ChecklistAfter wiring up any project, confirm in the dashboard (or via `traceway` CLI):-1. **Endpoints page** — routes are grouped by pattern (`GET /api/users/:id`), NOT one row per concrete URL. If you see raw IDs, `http.route` is not being set.-2. **Endpoints page** — status codes are non-zero.-3. **Tasks page** — each scheduled job appears under one stable name after triggering it. (Dashboard only — the CLI has no `tasks` command.)-4. **Issues page** — a deliberately thrown error shows up with a readable stack trace.-5. **Endpoint detail -> Spans tab** — DB queries appear as children showing the SQL text.+1. **Endpoints page**: routes are grouped by pattern (`GET /api/users/:id`), NOT one row per concrete URL. If you see raw IDs, `http.route` is not being set.+2. **Endpoints page**: status codes are non-zero.+3. **Tasks page**: each scheduled job appears under one stable name after triggering it. (Dashboard only for the grouped list. The CLI has `traceway tasks show <taskId> --recorded-at <ts>` for a single run, but no task list command.)+4. **Issues page**: a deliberately thrown error shows up with a readable stack trace.+5. **Endpoint detail -> Spans tab**: DB queries appear as children showing the SQL text.With the `traceway` CLI (`--since` takes relative ranges like `1h`, `24h`, `7d`):diff --git a/skills/traceway-setup/flutter.md b/skills/traceway-setup/flutter.mdindex daaee627..7e02eb35 100644--- a/skills/traceway-setup/flutter.md+++ b/skills/traceway-setup/flutter.md@@ -125,7 +125,7 @@ A plain `flutter build --release` keeps enough symbol information that crash sta**First, check whether the build is obfuscated.** Look at how release artifacts are produced (the `flutter build` invocations in CI under `.github/`, in `fastlane/`, in a `Makefile`, or in shell scripts) and ask the user how they build for release. If neither `--obfuscate` nor `--split-debug-info` is passed, stack traces are already readable and symbol upload is unnecessary; skip the rest of this section. If obfuscation is (or will be) used, set up the upload.-**Then ask for the upload token.** Symbol uploads authenticate with the dedicated upload token (Connection page > Symbol Upload, the same token shown under Source Maps), NOT the project token from the connection string. Get it from Step 0; it is a CI secret, never committed. `readonly` members cannot generate one.+**Then use the project's upload token.** Symbol uploads authenticate with the dedicated upload token (Connection page > Symbol Upload, the same token shown under Source Maps), NOT the project token from the connection string. It is a CI secret, never committed. `readonly` members cannot generate one.The `traceway` package ships an uploader (`dart run traceway:upload_symbols`) that finds the `.symbols` files, derives each one's architecture and debug ID, and posts them. Configure it once, then run it on every release.@@ -147,9 +147,24 @@ flutter build apk --release --obfuscate --split-debug-info=build/symbolsTRACEWAY_UPLOAD_TOKEN=<upload-token> dart run traceway:upload_symbols```-The uploader auto-discovers `build/symbols` and uploads every architecture in one run, so no flags are required. Each value resolves from a CLI flag first (`--token`, `--url`, `--symbols-dir`), then an env var (`TRACEWAY_UPLOAD_TOKEN`, `TRACEWAY_URL`), then the `pubspec.yaml` block. Pass `--dry-run` to preview. Symbols are unique to each build, so upload on every release; a crash only resolves against the exact build it came from.+The uploader auto-discovers `build/symbols` and uploads every architecture in one run, so no flags are required. Each value resolves from a CLI flag first (`--token`, `--url`, `--symbols-dir`), then an env var (`TRACEWAY_UPLOAD_TOKEN`, `TRACEWAY_URL`), then the `pubspec.yaml` block. Pass `--dry-run` to preview, but note that it still requires a resolved token and URL: without them it exits with `No upload token. Set TRACEWAY_UPLOAD_TOKEN, pass --token, or add traceway.upload_token to pubspec.yaml`. Symbols are unique to each build, so upload on every release; a crash only resolves against the exact build it came from.-The build command above produces an Android APK; other targets (`appbundle`, `ios`, `macos`) emit their own `.symbols` files into the same directory, all picked up by a single uploader run. On Android the debug ID is read straight from each file's build-id note. iOS and macOS `.symbols` carry no such note, so the uploader reads the Mach-O UUID from the built `.app` (auto-discovered under `build/`, or point at it with `--app build/macos/Build/Products/Release/YourApp.app`); run the uploader on the same Mac that produced the build. There is no hand-written upload path for Apple builds because that UUID can only be read from the compiled app.+The build command above produces an Android APK; other targets (`appbundle`, `ios`, `macos`) emit their own `.symbols` files into the same directory, all picked up by a single uploader run. On Android the debug ID is read straight from each file's build-id note. iOS and macOS `.symbols` carry no such note, so the uploader reads the Mach-O UUID from the built `.app` (auto-discovered under `build/`, or point at it with `--app build/macos/Build/Products/Release/YourApp.app`); run the uploader on the same Mac that produced the build.++The uploader exists because that UUID has to be read out of the compiled app, which is easiest on the build machine. It is not the only path. If a pipeline cannot run the Dart uploader, read the UUID yourself and pass it as a `debug_id` field:++```bash+dwarfdump --uuid build/macos/Build/Products/Release/YourApp.app/Contents/Frameworks/App.framework/App++curl -fsS -H "Authorization: Bearer <upload-token>" \+ -F "files=@build/symbols/app.darwin-arm64.symbols" \+ -F "debug_id=<uuid-from-dwarfdump>" \+ https://<instance>/api/symbols/upload+```++Case and dashes are normalized server-side, so paste the UUID exactly as `dwarfdump` prints it. The architecture is read from the `.symbols` filename; pass `-F "arch=arm64"` too if the file has been renamed.++**Known issue, verified 2026-08-17.** This curl body is byte-identical to what the Dart uploader sends, and on macOS both currently accept the upload (`2/2 uploaded`, `debug_id` matching the runtime build id) while obfuscated release traces still come back unresolved. The curl path inherits the bug, so it is not a workaround for it. Until that is fixed, keep a copy of the `.symbols` files with each release so traces can be resolved offline.Wire the upload into the release pipeline right after the build step (CI, fastlane, or a build script), with the token from a secret:diff --git a/skills/traceway-setup/frontend-js.md b/skills/traceway-setup/frontend-js.mdindex 6e0481fa..e94f73bc 100644--- a/skills/traceway-setup/frontend-js.md+++ b/skills/traceway-setup/frontend-js.md@@ -75,6 +75,8 @@ function MyComponent() {Options go on the provider: `<TracewayProvider connectionString="..." options={{ debug: true, version: "1.0.0" }}>`.+`TracewayProvider` initializes the SDK in its constructor, and React StrictMode double-invokes constructors in development. The Vite and CRA scaffolds both wrap the app in `<StrictMode>` by default, so in dev you get two clients and every uncaught error is reported twice. Production builds do not double-invoke, so treat inflated dev counts as expected, or mount the provider outside `<StrictMode>` while developing.+## Vue 3 (`@tracewayapp/vue`)The plugin installs a global error handler for uncaught errors:@@ -148,7 +150,7 @@ Via CDN (exposes `window.TracewayJQuery`):</script>```-Beyond the standard capture, the jQuery SDK hooks `$(document).ajaxError()` to capture failed `$.ajax()` calls (URL, method, status, message) and injects trace headers into `$.ajax()` (XHR) as well as `fetch`.+Beyond the standard capture, the jQuery SDK adds exactly one thing: it hooks `$(document).ajaxError()` to capture failed `$.ajax()` calls (URL, method, status, message). Trace-header injection is not jQuery-specific. `init()` in the core SDK patches both `fetch` and `XMLHttpRequest`, so it is already active in every JS SDK.## Options (shared by all JS SDKs)@@ -166,6 +168,9 @@ Beyond the standard capture, the jQuery SDK hooks `$(document).ajaxError()` to c| `captureLogs` | `boolean` | `true` | Mirror `console.*` calls into the rolling log buffer || `captureNetwork` | `boolean` | `true` | Record `fetch` / XHR calls as network actions || `captureNavigation` | `boolean` | `true` | Record History API transitions |+| `captureHttpServerErrors` | `boolean` | `false` | Report every `fetch` response with status >= 500 as a synthetic exception. 4xx is never included, and it is wired into the `fetch` wrapper only, so XHR and browser Axios do not trigger it |+| `eventsWindowMs` | `number` | `10000` (`30000` with `recordAllSessions`) | Rolling window the log and action buffers retain |+| `eventsMaxCount` | `number` | `200` (`600` with `recordAllSessions`) | Hard cap applied independently to the log buffer and the action buffer |### Error filtering@@ -198,24 +203,51 @@ clearAttributes();### Distributed tracing-The SDK instruments `fetch` (and XHR in the jQuery SDK) to propagate a `traceway-trace-id` header on same-origin requests, and the backend echoes the same id back on the response. Errors the SDK captures on its own (uncaught exceptions, unhandled rejections, `captureHttpServerErrors`) link to the originating backend request automatically. For Axios, register the interceptor:+`init()` patches both `fetch` and `XMLHttpRequest`, so every same-origin request carries a `traceway-trace-id` header and sets the SDK's active trace id. That covers `$.ajax()` and browser Axios too, because both go through XHR. Errors the SDK captures on its own (uncaught exceptions, unhandled rejections, `captureHttpServerErrors`) pick up the active id and link to the originating backend request automatically.++**Do not register `createAxiosInterceptor()` in a browser app.** Axios already goes through the instrumented `XMLHttpRequest`. The interceptor sets a *second*, different id under the same header name, XHR concatenates duplicate headers, and the backend receives `"<uuid1>, <uuid2>"`. That value fails to parse as a UUID and is dropped without any error, so adding the interceptor breaks a link that already worked.++#### Backend side++The browser id only reaches Traceway if the backend puts it on its server span. Traceway reads the span attribute `traceway.distributed_trace_id` and uses it as the row's distributed trace id, overriding the one derived from the OTel trace id. That override is what joins the browser exception and the backend endpoint into one trace. The value must be a bare UUID; anything else is ignored silently.++Only the Symfony bundle sets the attribute for you. Node, NestJS, Next.js, Hono, Cloudflare, Laravel and Django install vanilla OpenTelemetry, which has never heard of the header, so add one middleware:```javascript-import { createAxiosInterceptor } from "@tracewayapp/frontend";+import { trace } from "@opentelemetry/api";-api.interceptors.request.use(createAxiosInterceptor());+app.use((req, res, next) => {+ const id = req.headers["traceway-trace-id"];+ if (id) {+ trace.getActiveSpan()?.setAttribute("traceway.distributed_trace_id", id);+ res.setHeader("traceway-trace-id", id);+ }+ next();+});```-**Manual captures in a `fetch`/API wrapper MUST pass the trace id explicitly — this is the #1 reason frontend issues show up unlinked to the backend.** The SDK holds the active distributed-trace id only *for the duration of the request* and clears it the instant the fetch settles. A `captureException` call in your `catch` or `if (!res.ok)` branch runs *after* `await fetch`, by which point the active id is already `null`, so the exception is stored with `distributedTraceId: null` and never connects to the backend trace — even though the request header was sent correctly. Read the id the backend echoed on the response and pass it through as the third argument:+The `setAttribute` line is what creates the link. The `setHeader` line is optional, and only matters for the manual-capture pattern below. When the frontend and backend are on different origins, also send `Access-Control-Expose-Headers: traceway-trace-id` or the browser cannot read the echoed value.++Verify with **View distributed trace** on the backend endpoint's detail page. The browser exception should show up as a second node.++#### Manual captures after `await fetch`++**Manual captures in a `fetch`/API wrapper MUST pass the trace id explicitly. This is the #1 reason frontend issues show up unlinked to the backend.** The SDK holds the active distributed-trace id only *for the duration of the request* and clears it the instant the fetch settles. A `captureException` call in your `catch` or `if (!res.ok)` branch runs *after* `await fetch`, by which point the active id is already `null`, so the exception is stored with `distributedTraceId: null` and never connects to the backend trace, even though the request header was sent correctly.++Read the id off the SDK synchronously, after starting the request and before awaiting it, then pass it as the third argument. This needs nothing from the backend:```javascript-import { captureExceptionWithAttributes } from "@tracewayapp/frontend";+import {+ captureExceptionWithAttributes,+ getActiveDistributedTraceId,+} from "@tracewayapp/frontend";async function request(path, options) {- const res = await fetch(path, options);+ const pending = fetch(path, options);+ const distributedTraceId = getActiveDistributedTraceId() || undefined;+ const res = await pending;if (!res.ok) {const err = new Error(`Request to ${path} failed (${res.status})`);- const distributedTraceId = res.headers.get("traceway-trace-id") || undefined;captureExceptionWithAttributes(err,{ path, method: options?.method || "GET", status: String(res.status) },@@ -227,7 +259,9 @@ async function request(path, options) {}```-The same applies to the React `useTraceway().captureExceptionWithAttributes`, the Vue/Svelte equivalents, and any other place you capture by hand after awaiting a request. When the frontend and backend are on different origins, the backend must also add `traceway-trace-id` to `Access-Control-Expose-Headers` or the browser cannot read it off the response.+No other code can run between the `fetch` call and the `getActiveDistributedTraceId` call, so the id always belongs to this request. If the backend echoes the header (the `res.setHeader` line above), `res.headers.get("traceway-trace-id")` gives you the same value after the await.++The same applies to the React `useTraceway().captureExceptionWithAttributes`, the Vue/Svelte equivalents, and any other place you capture by hand after awaiting a request. Every framework package re-exports `getActiveDistributedTraceId` and `captureExceptionWithAttributes`, so import them from the package you installed.## Bundler Plugin (debug IDs)@@ -267,6 +301,8 @@ esbuild is not covered; bundles processed by Sentry's esbuild plugin still workUploads authenticate with a dedicated upload token (Connection page > Source Maps > Generate Upload Token), NOT the project token. It is a CI secret; never commit it. `readonly` members cannot generate one.+First make the build actually emit maps, or there is nothing to upload. Vite needs `build: { sourcemap: true }`, Rollup needs `output: { sourcemap: true }`, webpack needs `devtool: "source-map"`. Vite's default is `false`, so a stock `vite build` produces no `.map` files at all, and `traceway-sourcemaps` then prints `No .map files found` and exits 0. A `postbuild` hook reports success while uploading nothing.+```bashnpm install -D @tracewayapp/sourcemap-uploaddiff --git a/skills/traceway-setup/ios.md b/skills/traceway-setup/ios.mdindex 242a331d..9045d6a6 100644--- a/skills/traceway-setup/ios.md+++ b/skills/traceway-setup/ios.md@@ -126,7 +126,7 @@ Hard crashes are intercepted with POSIX signal handlers. When the **Xcode debuggRelease crash reports arrive as bare instruction addresses against each loaded binary image. They stay unreadable until Traceway resolves them server-side against the build's **dSYM** debug symbols (the Apple equivalent of JavaScript source maps). Upload the dSYMs on every release build; a crash only resolves against the exact build it came from.-**Token.** Symbol uploads authenticate with the dedicated **upload token** (Connection page > Source Maps / Symbol Upload), NOT the project token from the connection string. Get it from Step 0; it is a CI secret, never committed. `readonly` members cannot generate one. The upload endpoint is `https://<instance>/api/symbols/upload`.+**Token.** Symbol uploads authenticate with the dedicated **upload token** (Connection page > Source Maps / Symbol Upload), NOT the project token from the connection string. It is a CI secret, never committed. `readonly` members cannot generate one. The upload endpoint is `https://<instance>/api/symbols/upload`.**Produce dSYMs.** The build must emit dSYMs: set `DEBUG_INFORMATION_FORMAT = dwarf-with-dsym` for the Release configuration (this is already the default for Archive builds). Without it there are no dSYMs to upload.@@ -168,7 +168,7 @@ Self-hosted instances must have blob storage (S3 or a persistent volume) configu## Non-Swift apps: OpenTelemetry into Traceway-For an iOS/Apple app that is not a Swift app the SDK targets (Objective-C only, a cross-platform stack with no Traceway mobile SDK, or a codebase already standardized on OpenTelemetry), there is no native Traceway SDK. Use an OpenTelemetry distribution such as **Honeycomb's** and repoint its OTLP exporter at Traceway. This is the same OTLP/HTTP path the backend uses (Step 2), so all the rules in `data-model.md` apply.+For an iOS/Apple app that is not a Swift app the SDK targets (Objective-C only, a cross-platform stack with no Traceway mobile SDK, or a codebase already standardized on OpenTelemetry), there is no native Traceway SDK. Use an OpenTelemetry distribution such as **Honeycomb's** and repoint its OTLP exporter at Traceway. This is the same OTLP/HTTP path the backend uses (see "Backend OTel Setup" in SKILL.md), so all the rules in `data-model.md` apply.A Honeycomb-style distro is configured with an API key plus an endpoint. Override both so the data lands in Traceway instead of Honeycomb:@@ -176,18 +176,28 @@ A Honeycomb-style distro is configured with an API key plus an endpoint. Overrid- **Auth header**: `Authorization: Bearer <project-token>`. Traceway authenticates on this Bearer header, not on Honeycomb's `x-honeycomb-team`, so set the `Authorization` header explicitly (Honeycomb's own API-key header is ignored by Traceway and harmless).- **`service.name`** / **`service.version`**: set them; they become the Server Name and enable release comparison.+Both `build()` and `Honeycomb.configure(options:)` are `throws`, so the whole thing has to sit inside a `do` / `catch` (or a throwing function). Call it once, as early as possible:+```swiftimport Honeycomb-let options = try HoneycombOptions.Builder()- .setServiceName("my-ios-app")- .setServiceVersion("1.0.0")- .setApiEndpoint("https://<instance>/api/otel")- .setHeaders(["Authorization": "Bearer <project-token>"])- .build()-Honeycomb.configure(options: options)+do {+ let options = try HoneycombOptions.Builder()+ .setServiceName("my-ios-app")+ .setServiceVersion("1.0.0")+ .setAPIEndpoint("https://<instance>/api/otel")+ .setHeaders(["Authorization": "Bearer <project-token>"])+ .build()+ try Honeycomb.configure(options: options)+} catch {+ NSLog("Honeycomb configure failed: \(error)")+}```+The method is `setAPIEndpoint`, with a capital `API`, matching `setAPIKey`. Swift is case-sensitive, so `setApiEndpoint` does not compile. Do not drop the line if it fails: without it the exporter falls back to Honeycomb's own default endpoint and the telemetry never reaches Traceway.++No Honeycomb API key is needed. `build()` only demands one when the resolved endpoint is a Honeycomb host, and `https://<instance>/api/otel` is not.+(Field and builder names vary by distro version; the three things that matter are endpoint, the `Authorization: Bearer` header, and service name. A plain OpenTelemetry-Swift setup with an `OTLPHTTPExporter` pointed at the same endpoint and header is equivalent.)Constraints are the backend OTLP rules: OTLP/HTTP only (protobuf or JSON), OTLP/gRPC is NOT supported, gzip is fine, max request body 10 MB.diff --git a/skills/traceway/SKILL.md b/skills/traceway/SKILL.mdindex 21d39610..7dfeae19 100644--- a/skills/traceway/SKILL.md+++ b/skills/traceway/SKILL.md@@ -27,6 +27,7 @@ Drive a Traceway instance from the terminal with the `traceway` CLI. The first w- **Time windows**: always bound queries, default `--since 1h` for "now" questions, `--since 24h` otherwise. `--since` accepts `s`, `m`, `h`, lowercase `Nd` (no `1w`, no `7d2h`). Absolute windows via `--from` / `--to` (RFC3339).- **Exit codes**: 0 ok, 1 generic/API, 2 usage, 3 connection, 4 auth, 5 not found, 6 rate limited, 7 server 5xx. Errors emit `{"error":"<stable_id>","message":"...","hint":"...","exit_code":N}` on stderr; branch on the `error` field.- On exit code 4 (auth), do not run `traceway login` yourself; switch to the Login flow and let the user enter credentials.+- **Never guess a fix**: if the telemetry does not explain the failure, do not change behaviour on a hunch. Add the instrumentation that would explain it next time, and say what is still unknown. See "When the evidence is not enough" in the Debug flow.## Resolving Dashboard URLs@@ -281,7 +282,42 @@ The CLI also accepts `p50|p95|p99`, but the server has no quantile aggregation f3. Form a hypothesis that explains the targeted occurrence's full observation set (its message, affected endpoint, timing, volume) — not just the first stack frame. When a URL/hash anchored you to the last occurrence (step 2), explain *that* error; do not stretch one hypothesis to cover sibling clusters that merely share the hash.4. Propose or implement the fix per the user's instruction, scoped to the targeted occurrence's failure path.-### 5. Report and clean up+### 5. When the evidence is not enough, instrument instead of guessing++Before writing a fix, answer one question honestly: **does the telemetry actually explain this failure?** You have enough when you can name the failing line, the input that reached it, and why that input was wrong. A stack trace on its own is usually not enough. Neither is a plausible story that the data neither confirms nor contradicts.++If you cannot explain it, do not ship a speculative fix. A guess that changes behaviour hides the symptom, destroys the next reproduction, and costs another incident cycle. Instead, add the instrumentation that would answer the open question, and let the next occurrence do the diagnosing.++Pick the signal that closes the specific gap:++| What you could not determine | What to add |+|---|---|+| Which branch ran, or what the inputs were | A log at the decision point with the deciding values as attributes. Use ERROR severity on the failure path so `--min-severity 17` finds it |+| Where the time went, or which step failed | Child spans around each suspect step, opened inside the request's active context |+| Who or what it happened to (user, tenant, release, payload shape) | Attributes on the request span, or attributes passed to `recordException` |+| How often it happens, or under which conditions | A counter or histogram metric, tagged with the low-cardinality dimension you want to slice by |+| Where it broke across a service boundary | Trace context propagation on the call, plus a span or log on both sides |+| Whether the code path runs at all | A log or counter at entry, so absence of data becomes evidence |++Rules for what you add:++- Emit it inside the active span of the request or task. That is what stamps the trace id on the log and makes it show up on the trace.+- Put identifiers in attributes, never in the message text. Messages are stripped before grouping, and attributes are filterable.+- Keep cardinality low. User ids and request ids belong in span attributes, never in a metric tag or a span name.+- Never log secrets, tokens, or personal data.+- Ship it as permanent instrumentation, not a temporary debug print. Something you remove tomorrow answers nothing.+- If the area has no instrumentation at all, set it up first with the `traceway-setup` skill rather than bolting on one log line.++Then close the loop with the user. State what you could not determine, what you added, and the exact command to run when it fires again:++```bash+traceway logs query --trace-id <id> --output json+traceway exceptions show <hash> --output json | jq '.occurrences | last | .attributes'+```++Adding instrumentation is a legitimate outcome of a debug session, not a failure to fix. Report it as the deliverable it is.++### 6. Report and clean upSummarize: symptom, evidence (exception hashes, log excerpts, metric anomalies), root cause, fix. Include `traceway exceptions show <hash>` references so the user can verify. After a fix is deployed and verified, archive only when the user asks:@@ -302,10 +338,12 @@ The loop, using the read commands documented in this skill:3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <distributedTraceId> --recorded-at <t>` for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.-6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --aggregation max`, then `mem.used_pcnt`, `go.gc_pause`. A spike that lines up with the latency onset points at saturation, not a code bug.+6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --aggregation max`, then `mem.used`, `go.gc_pause`. A spike that lines up with the latency onset points at saturation, not a code bug.7. **Correlate with code and deploys.** With the onset time from step 3, check what shipped then: `git log --since '<onset - 30m>' --until '<onset + 30m>'` or the deploy history. A jump at a release is a regression; a gradual ramp with no deploy is data growth (N+1, missing index, no pagination).8. **Report.** The bottleneck, the evidence (endpoint percentiles, the onset time, the dominating span, any correlated metric anomaly), the root cause mapped to a checklist item, and a concrete fix, with the `endpoints show` / `traces show` references so the user can verify. If the endpoint is marked slow, state whether the latency exceeded the accepted offset.+If the waterfall does not localize the cost, stop before proposing a fix. An unexplained gap, one fat span covering the whole handler, or a tail you cannot reproduce all mean the same thing: the trace is too coarse to act on. Add child spans around the suspect steps inside that handler, then wait for the next slow request to point at the real one. See "When the evidence is not enough" in the Debug flow.+## Flow: QueryFor free-form requests ("what's broken in prod?", "is /api/checkout slow?", "show errors for service X"), use the read commands directly. A request specifically about latency or slowness routes to the Performance flow above.diff --git a/skills/traceway/performance.md b/skills/traceway/performance.mdindex 445ef733..0d146d8d 100644--- a/skills/traceway/performance.md+++ b/skills/traceway/performance.md@@ -26,7 +26,7 @@ All commands are read-only. Never archive or mutate anything during a performanc4. **Separate code from saturation.** A slow endpoint can be a code bug or an overloaded host. Pull infrastructure and runtime metrics over the same window:```bashtraceway metrics query --name system.cpu.utilization --aggregation max --since 24h- traceway metrics query --name mem.used_pcnt --aggregation max --since 24h+ traceway metrics query --name mem.used --aggregation max --since 24htraceway metrics query --name go.gc_pause --aggregation max --since 24h```A metric spike that lines up with the latency rise points at saturation; flat metrics under rising latency point at a code or query problem.@@ -120,7 +120,7 @@ Each row: the suspect, and how to confirm it from Traceway telemetry.| **Synchronous work that belongs in a background task** | A long handler span runs before the response returns, for work the user does not wait on (emails, exports, image processing). It should be a `CaptureTask`. || **CPU-bound serialization** | A compute gap between IO spans (large JSON encode/decode); `system.cpu.utilization` up during the request. || **Inefficient algorithm / O(n^2) hot path** | Latency scales superlinearly with input size across traces; no IO to blame in the waterfall. |-| **Unbounded in-memory work / large allocations** | `mem.used_pcnt` and `go.num_gc` climb with p99; large compute spans. |+| **Unbounded in-memory work / large allocations** | `mem.used` and `go.num_gc` climb with p99; large compute spans. || **Excessive logging / telemetry in the hot path** | Many log lines per request in `logs query`; logging spans visible in the waterfall. |### Network, IO, and external dependencies@@ -144,12 +144,12 @@ Each row: the suspect, and how to confirm it from Traceway telemetry.| Suspect | How to confirm in Traceway ||---|---|| **CPU saturation** | `system.cpu.utilization` (max) spike lines up with the latency rise across many endpoints at once. |-| **Memory pressure / GC pauses** | `mem.used_pcnt`, `go.gc_pause`, `go.num_gc` correlate with p99; pauses visible as gaps in the waterfall. |+| **Memory pressure / GC pauses** | `mem.used`, `go.gc_pause`, `go.num_gc` correlate with p99; pauses visible as gaps in the waterfall. || **Disk / IO bottleneck** | `system.disk.*` or IO-wait metrics up; slow IO spans with no CPU cost. || **Undersized / noisy-neighbor instance** | Broad latency rise with no single span culprit; infra metrics elevated with no code change. |## Notes on the signals-- Host and runtime metrics live under `system.*` (Traceway OTel Agent) and the SDK runtime names (`go.gc_pause`, `mem.used_pcnt`, `go.go_routines`). The server has no quantile aggregation for metric points: `p50/p95/p99` on `metrics query` silently return `avg`, so never present those as percentiles. Real latency percentiles come only from `endpoints list`, computed from raw request durations.+- Host and runtime metrics live under `system.*` (Traceway OTel Agent) and the SDK runtime names (`go.gc_pause`, `mem.used`, `go.go_routines`). The server has no quantile aggregation for metric points: `p50/p95/p99` on `metrics query` silently return `avg`, so never present those as percentiles. Real latency percentiles come only from `endpoints list`, computed from raw request durations.- OTLP histogram metrics are stored as two series, `<name>.avg` and `<name>.count`. A bogus metric name returns an empty `series: {}` cleanly, so probing names is safe.- When the waterfall shows a gap with no span, that gap is the finding: it is time the request spent waiting (for a connection, a lock, the scheduler), not time spent doing work.diff --git a/website/app/blog/page.tsx b/website/app/blog/page.tsxindex 8ac88350..9ec6a606 100644--- a/website/app/blog/page.tsx+++ b/website/app/blog/page.tsx@@ -15,7 +15,7 @@ export default function BlogIndex() {const posts = getPostsByCategory("engineering");return (- <main className="relative">+ <main className="relative blog-scope"><section className="wrap pt-6 pb-24"><div className="prose max-w-[960px]"><Eyebrow>Blog</Eyebrow>diff --git a/website/app/globals.css b/website/app/globals.cssindex 5426b80d..655ed62f 100644--- a/website/app/globals.css+++ b/website/app/globals.css@@ -35,6 +35,12 @@--ok: #22e0a8;--crit: #ff5a5f;+ /* Dashboard blue family, shared with the docs restyle. Used for links+ site-wide and as the blog band's accent. */+ --blue: oklch(0.623 0.214 259.815);+ --blue-bright: oklch(0.707 0.165 254.624);+ --blue-deep: oklch(0.546 0.245 262.881);+/* Fonts (populated by next/font in layout.tsx) */--font-display: "JetBrains Mono", "IBM Plex Mono", ui-monospace, monospace;--font-mono: "IBM Plex Mono", "JetBrains Mono", ui-monospace, monospace;@@ -119,6 +125,9 @@--color-a2: var(--a2);--color-a3: var(--a3);--color-a4: var(--a4);+ --color-blue: var(--blue);+ --color-blue-bright: var(--blue-bright);+ --color-blue-deep: var(--blue-deep);--color-ok: var(--ok);--color-crit: var(--crit);--color-hair: var(--hair);@@ -394,11 +403,35 @@font-size: 13px;}+ /* Blog band: local token remap like .band-light. The blog renders on the+ dashboard's navy surface scale with the blue accent family (matching the+ docs restyle); the rest of the site keeps the original palette. */+ .blog-scope {+ --ink-1: #0f1220;+ --ink-2: #151a2c;+ --ink-3: #1b2138;+ --ink-4: #252c47;+ --ink-5: #313a5b;+ --hair: rgba(255, 255, 255, 0.08);+ --hair-2: rgba(255, 255, 255, 0.12);+ --a1: var(--blue-deep);+ --a2: var(--blue-bright);+ }++ /* Subscribe button inside the blog band: dashboard primary blue. */+ .blog-scope .btn-accent {+ background: var(--blue-deep);+ color: #fff;+ }+ .blog-scope .btn-accent:hover {+ background: color-mix(in oklab, var(--blue-deep) 85%, white);+ }+/* Blog post cards */.blog-card {display: block;border: 1px solid var(--hair);- background: color-mix(in oklab, var(--ink-2) 70%, transparent);+ background: var(--ink-2);border-radius: var(--radius-lg);padding: 20px 22px;text-decoration: none;@@ -474,7 +507,7 @@}.blog-panel {border: 1px solid var(--hair-2);- background: color-mix(in oklab, var(--ink-2) 55%, transparent);+ background: var(--ink-1);border-radius: var(--radius-xl);padding: clamp(24px, 4vw, 44px);box-shadow: 0 30px 60px -40px rgba(0, 0, 0, 0.6);@@ -1054,7 +1087,7 @@margin: 6px 0;}.prose a {- color: var(--a2);+ color: var(--blue-bright);text-decoration: underline;text-underline-offset: 3px;}diff --git a/website/app/product/agent-skills/page.tsx b/website/app/product/agent-skills/page.tsxindex 0bc0d39b..1dd3b16a 100644--- a/website/app/product/agent-skills/page.tsx+++ b/website/app/product/agent-skills/page.tsx@@ -77,7 +77,7 @@ export default function AgentSkillsPage() {href={GITHUB_URL}target="_blank"rel="noopener noreferrer"- className="text-fg-2 hover:text-a2 transition-colors"+ className="text-fg-2 hover:text-blue-bright transition-colors">View on GitHub →</Link>@@ -323,7 +323,7 @@ export default function AgentSkillsPage() {prompts, and resources.</p></div>- <span className="inline-flex shrink-0 items-center gap-1.5 font-mono text-[0.8125rem] text-a2 transition-transform group-hover:translate-x-0.5">+ <span className="inline-flex shrink-0 items-center gap-1.5 font-mono text-[0.8125rem] text-blue-bright transition-transform group-hover:translate-x-0.5">Explore the MCP server<ArrowRight className="h-4 w-4" /></span>@@ -403,7 +403,7 @@ export default function AgentSkillsPage() {gives the agent live tools.</p><p>- <Link href="/product/mcp" className="text-a2 hover:underline">+ <Link href="/product/mcp" className="text-blue-bright hover:underline">See the MCP server →</Link></p>diff --git a/website/app/product/cli/page.tsx b/website/app/product/cli/page.tsxindex b681f91d..d573faed 100644--- a/website/app/product/cli/page.tsx+++ b/website/app/product/cli/page.tsx@@ -51,7 +51,7 @@ export default function CliPage() {href="https://docs.tracewayapp.com/learn/cli"target="_blank"rel="noopener noreferrer"- className="text-fg-2 hover:text-a2 transition-colors"+ className="text-fg-2 hover:text-blue-bright transition-colors">Read the CLI docs →</Link>@@ -276,7 +276,7 @@ export default function CliPage() {login and current project.</p></div>- <span className="inline-flex shrink-0 items-center gap-1.5 font-mono text-[0.8125rem] text-a2 transition-transform group-hover:translate-x-0.5">+ <span className="inline-flex shrink-0 items-center gap-1.5 font-mono text-[0.8125rem] text-blue-bright transition-transform group-hover:translate-x-0.5">Explore the MCP server<ArrowRight className="h-4 w-4" /></span>diff --git a/website/app/product/mcp/page.tsx b/website/app/product/mcp/page.tsxindex 156083ed..c77d5dee 100644--- a/website/app/product/mcp/page.tsx+++ b/website/app/product/mcp/page.tsx@@ -92,7 +92,7 @@ export default function McpPage() {href="https://docs.tracewayapp.com/learn/mcp"target="_blank"rel="noopener noreferrer"- className="text-fg-2 hover:text-a2 transition-colors"+ className="text-fg-2 hover:text-blue-bright transition-colors">Read the MCP docs →</Link>@@ -191,7 +191,7 @@ export default function McpPage() {</p><p>New to the CLI? Installation, usage, and login live on the{" "}- <Link href="/product/cli" className="text-a2 hover:underline">+ <Link href="/product/cli" className="text-blue-bright hover:underline">CLI page →</Link></p>@@ -328,7 +328,7 @@ export default function McpPage() {Prefer skills over MCP tools?{" "}<Linkhref="/product/agent-skills"- className="text-a2 hover:underline"+ className="text-blue-bright hover:underline">See Agent Skills →</Link>@@ -378,7 +378,7 @@ export default function McpPage() {<p><Linkhref="/product/agent-skills"- className="text-a2 hover:underline"+ className="text-blue-bright hover:underline">See Agent Skills →</Link>diff --git a/website/app/releases/page.tsx b/website/app/releases/page.tsxindex 89a5024a..877b79f8 100644--- a/website/app/releases/page.tsx+++ b/website/app/releases/page.tsx@@ -13,7 +13,7 @@ export default function ReleasesIndex() {const posts = getPostsByCategory("release");return (- <main className="relative">+ <main className="relative blog-scope"><section className="wrap pt-6 pb-24"><div className="prose max-w-[960px]"><Eyebrow>Blog</Eyebrow>diff --git a/website/components/blog-article.tsx b/website/components/blog-article.tsxindex 9ce838b5..263664df 100644--- a/website/components/blog-article.tsx+++ b/website/components/blog-article.tsx@@ -29,7 +29,7 @@ export function BlogArticle({showSubscribe: boolean;}) {return (- <main className="relative">+ <main className="relative blog-scope"><section className="wrap pt-6 pb-24"><div className="blog-article"><Link