The only tool you need to know what is happening and how to fix it.
Performance debugging
Authoritative reference for the Performance flow. Approach the investigation as a senior software engineer with deep performance-debugging experience: do not guess at fixes, quantify first, localize the cost to a specific span, confirm the cause against the checklist below, and only then propose a change. Every suspect here is paired with the Traceway signal that confirms it, so the checklist is a lookup table from "what the waterfall shows" to "what is actually wrong".
All commands are read-only. Never archive or mutate anything during a performance investigation. For exact CLI syntax, the mandatory --recorded-at / --started-at timestamp rules, and dashboard-URL resolution, the parent SKILL.md is authoritative (over MCP: the traceway://knowledge/timestamps and traceway://knowledge/url-resolution resources).
The investigation loop
Quantify and localize. Find the slow endpoint(s) and read the shape of the latency:
traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json \ | jq '.data[]? | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), impact}'The rows are under
data.p50Duration,p95Duration,p99DurationandavgDurationare nanoseconds, hence the/ 1e6. Read p50 vs p99 before anything else (see "Reading the latency shape").Get a representative slow trace. Drill into one slow request to see where the time goes.
endpoints requestslists the endpoint's requests, slowest first; take anidand itsrecordedAttogether, then open it:traceway endpoints requests "GET /api/checkout" --since 24h --page-size 5 --output json \ | jq '.data[] | {id, recordedAt, duration_ms: (.duration / 1e6), statusCode, traceId}' traceway endpoints show <id> --recorded-at <recordedAt> # span waterfall for one request traceway traces show <traceId> --recorded-at <recordedAt> # cross-service timelineOpen a typical slow request and the slowest one: if they differ, the tail has its own cause.
traces showis the highest-value call when more than one service is involved: it stitches the whole logical request together end to end.Find the long pole and match it to the checklist. In the waterfall, identify the single span (or repeated group of spans) that dominates the duration. Classify it: is it a database call, an external/upstream call, in-process compute, or a gap before work starts (which means queueing, lock wait, or connection-pool exhaustion, not the work itself)? Then look it up in the checklist.
Separate code from saturation. A slow endpoint can be a code bug or an overloaded host. Pull infrastructure and runtime metrics over the same window:
traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min --group-by host.name --since 24h traceway metrics query --name system.memory.utilization --tag state=used --aggregation max --group-by host.name --since 24h traceway metrics query --name jvm.gc.duration.avg --aggregation max --since 24hCPU and memory host metrics are split by a
statetag, so always filter or group by it.system.cpu.utilizationhas one series per state (user, system, idle, wait, steal, ...) whose fractions add up to 1: an unfiltered query averages the states and reads as a flat fraction whatever the load. The busy share is 1 minusstate=idle, and the minimum idle in a bucket is its peak load. Runtime metrics use the OpenTelemetry name for the language:jvm.memory.usedandjvm.gc.duration.avg(JVM),go.memory.usedandgo.memory.gc.goal(Go),v8js.memory.heap.used(Node.js),process.runtime.cpython.memory(Python). Projects on the native protocol reportcpu.used_pcntandmem.used/mem.totalinstead. A metric spike that lines up with the latency rise points at saturation; flat metrics under rising latency point at a code or query problem.Correlate with code and deploys. Open the slow path in the source and read it. If p95 climbed at a specific time, check what shipped then (
git log --since ... --until ...or deploy history). Latency that jumped at a release is a regression; latency that crept up without one is usually data growth.Report. State the bottleneck, the evidence (endpoint percentiles, the dominating span, any correlated metric), the root cause mapped to a specific checklist item, and a concrete fix. Include the
endpoints show/traces showreferences so the user can verify.
Reading the latency shape
The relationship between percentiles, load, and time tells you which family of causes to look at before you open a single trace:
- p50 already high (the median request is slow): every request pays the cost, so it is systemic. Suspect an inefficient query, a missing index, an O(n^2) hot path, or synchronous work that should be backgrounded.
- p99 much greater than p50 (median fine, tail slow): a minority of requests stall. Suspect contention, connection-pool exhaustion, GC pauses, retries, cache stampedes, or a flaky upstream dependency.
- Latency that grew over weeks with no deploy: data-volume driven. Suspect Select N+1, a missing index, or an unbounded result set, all of which scale with table size.
- Latency that jumped at a deploy: a regression. Correlate
firstSeen/ the climb time withgit log. - Latency that tracks request volume: saturation or contention. Confirm with infra metrics and pool-wait gaps in the waterfall.
Account for user-marked slow endpoints
Some endpoints are expected to be slow: a CSV export, a heavy report, a third-party sync. An operator can mark such an endpoint slow with an allowance (offsetMs) and a reason. Check before you treat latency as a problem:
traceway endpoints slow "GET /api/reports/export" --output json # {offsetMs, reason}
offsetMs 0(or "MARKED: no"): default thresholds apply; judge latency normally.offsetMs > 0: the operator accepted that much latency. The apdex and impact thresholds are shifted up by the offset server-side, so the endpoint'simpact/impactReasonalready discount it. Only latency beyondoffsetMs, or a fresh jump above the prior baseline, is a real regression.
Two things to keep straight:
- Raw percentiles are not offset-adjusted. The p50/p95/p99 from
endpoints listandendpoints chartcome from raw durations; the offset only moves the apdex/impact thresholds. A marked-slow endpoint can show a high p95 that is entirely expected, so compare p95 againstoffsetMs, not against zero. - p95/p99 alert rules are not offset-aware.
endpoint_p95_threshold/endpoint_p99_thresholdnotifications fire on raw latency regardless of marking, so a marked-slow endpoint can still page on latency its operators accepted.apdex_dropandimpact_score_*rules are offset-aware. (See "Performance notification" inSKILL.md.)
When you report, state the p95 and the allowance together: "p95 is 2.1s but this endpoint is marked slow with a 2s allowance (reason: nightly export), so it is within baseline", or "p95 is 4s against a 2s allowance, so it regressed past the accepted offset".
Pinpointing when the slowness started
Narrowing the window is how you turn "it is slow" into "it got slow at 14:10, right after deploy X". Do not investigate the default window blindly: adjust it down until it brackets the onset, then correlate that moment with deploys and metrics.
1. Latency over time: endpoints chart. This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a {timestamp, endpoint, value} series in milliseconds. Each timestamp is the start of its bucket. Read down one endpoint's buckets for the first one whose value has jumped: the onset lies inside that bucket, between its timestamp and its timestamp plus the interval, not in the bucket before it. Use --metric-type p95 (or p99/p50/total_time) and a coarse --interval-minutes over a wide window first to find that bucket. A coarse bucket only brackets the onset: with 60-minute buckets its timestamp can be 59 minutes early, with 6-hour buckets almost 6 hours. Before you report an onset time, re-run over just that bucket with an interval of 15 minutes or less, and report the start of the first elevated fine bucket.
traceway endpoints chart --metric-type p95 --since 7d --interval-minutes 60 --output json \
| jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'
# the 60-minute bucket starting 14:00 stepped up: sharpen it in 5-minute buckets
traceway endpoints chart --metric-type p95 --from 2026-06-25T14:00:00Z --to 2026-06-25T15:00:00Z --interval-minutes 5 --output json \
| jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'
Caveat: it always returns the top 5 endpoints by that metric. If your suspect route is not in the top 5 it is folded into "Other"; fall back to method 3 for it.
2. Infra and runtime over time: metrics query --interval-minutes. For the cause behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is {results: [{name, unit, series}]}, one result per query, and series maps each group to its list of {timestamp, value} points (a single __all__ group when ungrouped); read down the points for the first bucket where the value has stepped up. Timestamps are bucket starts here too, so sharpen that bucket the same way before you report a time.
traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min --since 7d --interval-minutes 60 --output json \
| jq '.results[0].series | to_entries[] | .value[] | {t: .timestamp, busy: (1 - .value)}'
A flat line that steps up at the same bucket the latency jumped is the saturation signal; flat metrics under a latency step point at code.
3. A specific non-top-5 endpoint: window bisection. When the route you care about is in "Other", reconstruct its onset from endpoints list (the only source of true p50/p95/p99, computed from raw durations) by comparing the same endpoint's percentiles across adjacent windows with --from/--to. Start wide, split, keep the half where p95 is elevated, recurse until the bracket is tight:
# is the regression in the first or second half of the suspect day?
traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T00:00:00Z --to 2026-06-25T12:00:00Z \
--output json | jq '.data[0] | {p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), count}'
traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T12:00:00Z --to 2026-06-26T00:00:00Z \
--output json | jq '.data[0] | {p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), count}'
Keep bisecting until the elevated window is 15 minutes or less, then report its start. Watch count alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.
4. Cross-check and correlate. Once the methods agree on an onset time:
- A latency jump that coincides with a metric inflection (CPU or GC stepping up at the same bucket) points at saturation.
- A latency jump with flat infra metrics points at a code or query regression:
git log --since '<onset - 30m>' --until '<onset + 30m>'or the deploy history for what shipped then. - A latency curve that ramps gradually over days with no single step is data growth (N+1, missing index, unbounded result set), not a deploy.
Then pass the narrowed --from/--to back into the by-id detail lookups so --recorded-at lands inside the slow window and the trace you pull is actually from the slow period, not a fast one before the regression.
Bottleneck checklist
Each row: the suspect, and how to confirm it from Traceway telemetry.
Database and data access
| Suspect | How to confirm in Traceway |
|---|---|
| Select N+1 | Many repeated, near-identical short query spans inside one request in the span waterfall; total DB time dominates while each call is tiny. |
| Missing or unused index | One query span is the long pole; its duration grows with table size; same query slow across many traces. |
Over-fetching (SELECT * / no projection) |
Large query-result spans, high memory; latency tracks row width/count rather than logic. |
| No pagination / unbounded result set | p50 has crept up over time with data growth; one query returns more rows every week. |
| Expensive JOIN or aggregation | A single query span dominates the trace consistently, independent of load. |
| Connection-pool exhaustion | A gap before the first DB span (time waiting to acquire a connection); p99 rises with concurrency while p50 stays flat. |
| Lock / transaction contention | DB spans queue behind each other; tail latency appears under write load; correlated WARN/ERROR locks in logs query. |
| Missing query/result caching | The same query runs on every request; high DB-call count per endpoint with low cardinality of inputs. |
Application and compute
| Suspect | How to confirm in Traceway |
|---|---|
| Synchronous work that belongs in a background task | A long handler span runs before the response returns, for work the user does not wait on (emails, exports, image processing). It should be a CaptureTask. |
| CPU-bound serialization | A compute gap between IO spans (large JSON encode/decode); busy CPU (1 minus system.cpu.utilization with state=idle) up during the request. |
| Inefficient algorithm / O(n^2) hot path | Latency scales superlinearly with input size across traces; no IO to blame in the waterfall. |
| Unbounded in-memory work / large allocations | The runtime heap (jvm.memory.used, go.memory.used, v8js.memory.heap.used) and GC activity climb with p99; large compute spans. |
| Excessive logging / telemetry in the hot path | Many log lines per request in logs query; logging spans visible in the waterfall. |
Network, IO, and external dependencies
| Suspect | How to confirm in Traceway |
|---|---|
| Chatty IPC / microservice fan-out | traces show reveals many sequential cross-service hops, each adding round-trip latency. |
| Slow third-party / upstream API | An external span dominates the trace; p99 spikes correlate with the provider; not your code. |
| Sequential calls that could run in parallel | A staircase waterfall: each span starts only when the previous ends, though they are independent. |
| No timeouts / aggressive retries | Long-tail p99; retry bursts in logs query; the same downstream span repeated within one trace. |
| No connection reuse (DNS/TLS per call) | Many short external spans each paying a handshake; per-request connection setup cost. |
| Large request/response payloads | Transfer-dominated spans; latency correlates with payload size. |
Caching and delivery
| Suspect | How to confirm in Traceway |
|---|---|
| No cache / low hit ratio | DB or upstream is hit on every request; cache-hit metric (if reported) is low. |
| Cache stampede / thundering herd | Latency spikes at cache-expiry boundaries; a correlated burst of identical traces. |
| Cold start / JIT / connection warmup | First-request-after-deploy outliers; latency normalizes after warmup; firstSeen-correlated. |
Infrastructure and saturation
| Suspect | How to confirm in Traceway |
|---|---|
| CPU saturation | Busy CPU (1 minus system.cpu.utilization with state=idle) spikes in line with the latency rise across many endpoints at once. |
| Memory pressure / GC pauses | system.memory.utilization (state=used), the runtime heap and GC time (jvm.gc.duration.avg, go.memory.gc.goal) correlate with p99; pauses visible as gaps in the waterfall. |
| Disk / IO bottleneck | system.disk.* or IO-wait metrics up; slow IO spans with no CPU cost. |
| Undersized / noisy-neighbor instance | Broad latency rise with no single span culprit; infra metrics elevated with no code change. |
Notes on the signals
- Host metrics live under
system.*(OpenTelemetry hostmetrics, sent by the Traceway OTel Agent) and are split by tags such asstate; runtime metrics use the OpenTelemetry names of the language (jvm.*,go.*,v8js.*,process.runtime.*). The server has no quantile aggregation for metric points:p50/p95/p99onmetrics querysilently returnavg, so never present those as percentiles. Real latency percentiles come only fromendpoints list, computed from raw request durations. - OTLP histogram metrics are stored as two series,
<name>.avgand<name>.count. A bogus metric name returns an emptyseries: {}cleanly, so probing names is safe. - When the waterfall shows a gap with no span, that gap is the finding: it is time the request spent waiting (for a connection, a lock, the scheduler), not time spent doing work.