The only tool you need to know what is happening and how to fix it.
devtesting-embedded
Embedded Traceway backend + OTel-instrumented Go service + jQuery frontend, for local development and feature testing.
Quick Start
go run .
Then:
- App: http://localhost:8080/cdn (no build step) or http://localhost:8080 (requires
cd frontend && npm install && npm run buildfirst) - AI chat: http://localhost:8080/chat (needs
ORCAROUTER_API_KEY,CHEAPER_INFERENCE_API_KEYorOPENROUTER_API_KEYin.env) - Dashboard: http://localhost:8082 — login
[email protected]/admin
AI chat (conversations, tool calls, sub-agents)
/chat is a no-build chat UI backed by an OpenAI-compatible gateway, built to exercise the AI conversation analytics end to end. Configuration lives in .env (gitignored):
# OrcaRouter (https://www.orcarouter.ai) — preferred; routing, fallback, guardrails
ORCAROUTER_API_KEY=sk-orca-...
ORCAROUTER_MODEL=orcarouter/auto
# Cheaper Inference (https://cheaperinference.com)
# CHEAPER_INFERENCE_API_KEY=ci_live_...
# CHEAPER_INFERENCE_MODEL=gpt-5.4-mini
# OpenRouter — fallback provider
# OPENROUTER_API_KEY=sk-or-...
# OPENROUTER_MODEL=anthropic/claude-sonnet-5
When ORCAROUTER_API_KEY is set the chat routes through OrcaRouter (https://api.orcarouter.ai/v1). When CHEAPER_INFERENCE_API_KEY is set instead, it routes through Cheaper Inference (https://api.cheaperinference.com/v1). Otherwise it falls back to OpenRouter.
How it maps to Traceway:
- Every LLM call emits a
gen_ai.*span into theBackend APIproject, with real token counts and cost from the gateway's usage accounting. - The chat session id becomes
gen_ai.conversation.id, so each browser conversation shows up on the Conversations tab. The persona picker setsuser.idfor the Users tab. - The main Support Chat Agent has fake tools (
get_weather,lookup_order,get_server_time) plus two delegation tools that run sub-agents in the same conversation: Research Sub-Agent (withsearch_knowledge_base) and Math Sub-Agent (with a realcalculateevaluator). Each agent is a separate trace name on the AI Traces tab. - Tool executions run in plain child spans (no
gen_ai.*attributes) so they appear in the trace waterfall without inflating conversation turn counts; the tool calls themselves are parsed from the completion payloads.
Suggestion buttons in the UI trigger each tool and sub-agent. To test content flagging, swear at the bot or add custom terms in the project settings AI tab. The API is also curl-able:
curl -s http://localhost:8080/api/ai/chat -H 'Content-Type: application/json' -d '{
"sessionId": "conv-1", "userId": "[email protected]",
"messages": [{"role": "user", "content": "Weather in Paris, and ask the math agent for 12*(3+4)"}]
}'
What runs
- Traceway backend, embedded on port 8082, SQLite storage in
./storage/. - Go app on port 8080, instrumented with OpenTelemetry:
- Traces via OTLP/HTTP →
http://localhost:8082/api/otel/v1/traces - Logs via OTLP/HTTP →
http://localhost:8082/api/otel/v1/logs - Two OTel providers in the same process to exercise the distributed-logs flow:
backend-serviceandworker-service, both reporting to theBackend APIproject.
- Traces via OTLP/HTTP →
- jQuery frontend pages (
/and/cdn) — unchanged; useful for exercising the jQuery SDK independently.
Testing the logs feature
Five endpoints exercise different parts of the logs feature end-to-end:
| Endpoint | What it exercises |
|---|---|
GET /api/test-error |
Error log + exception recording on the root span |
GET /api/test-success |
DEBUG + INFO logs on a simple successful request |
GET /api/test-log-levels |
One log at each severity (TRACE/DEBUG/INFO/WARN/ERROR/FATAL) — validates SeverityBadge rendering |
GET /api/test-spans-with-logs |
Nested child spans (auth.verify → db.query → cache.lookup) with logs emitted inside each and attributes on every child span. Exercises parent_span_id capture, the span tree + attribute popover on the trace detail page, and span-chip linking |
GET /api/test-distributed-logs |
Two services (backend-service + worker-service) emit logs under a shared W3C trace ID. Both services appear in the trace log panel |
Trigger them all at once:
for path in test-error test-success test-log-levels test-spans-with-logs test-distributed-logs; do
curl -s "http://localhost:8080/api/$path" > /dev/null
done
Then in the dashboard (logged in, Backend API project selected):
- Logs list — open
/logs. You should see six severity badges and every request's logs listed with a Message, Service (backend-serviceorworker-service), and trace ID. The severity dropdown and search (by Message / Service / Trace ID) should all work. - Trace detail + chip linking — open
/endpoints, click one of the test endpoints, click a row. Under the Spans card you should see a Logs card listing that request's logs, each chipped with the span that emitted it. For/api/test-spans-with-logsspecifically, verify logs emitted insidedb.querychip asdb.query, logs fromcache.lookupchip ascache.lookup, and logs from handler code (between the child-span blocks) chip as the endpoint name. - Distributed logs button — on a trace from
/api/test-distributed-logs, the Logs card shows aLoad logs from other tracesbutton. Clicking it loads logs from theworker-servicetrace under a divider. - Span tree + attributes — on a trace from
/api/test-spans-with-logs, the Spans card renders a tree:auth.verifyanddb.queryat the top level,cache.lookupindented underdb.querywith a chevron ondb.queryto collapse it. Clicking a span name opens a popover listing its attributes (e.g.db.system,cache.hit) with per-value copy buttons. parent_span_id+attributes— verify in SQLite:
Non-root spans should show a non-nullsqlite3 storage/traceway_telemetry.db "SELECT substr(id,1,8), name, substr(parent_span_id,1,8), attributes FROM spans ORDER BY start_time DESC LIMIT 10;"parent_span_idmatching their parent span'sidand a JSONattributescolumn.
Starting from a clean slate
To re-run migrations from empty (useful after any of these migrations ship: 0045_create_log_records, 0046_add_parent_span_id_to_spans, 0047_add_span_id_to_endpoints, 0048_add_span_id_to_tasks, or their SQLite counterparts 0003_create_log_records, 0004_add_parent_span_id_to_spans, 0005_add_span_id_to_root_traces):
rm -f storage/traceway.db* storage/traceway_telemetry.db*
The backend will re-create the schema on next start.
Pages (legacy jQuery setup — unchanged)
| URL | Description | Build required? |
|---|---|---|
| http://localhost:8080 | Node-built version — @tracewayapp/jquery bundled via Vite into static/app.js |
Yes (npm install && npm run build in frontend/) |
| http://localhost:8080/cdn | CDN version — loads @tracewayapp/jquery IIFE from jsdelivr |
No |
| http://localhost:8082 | Traceway dashboard | No |