The only tool you need to know what is happening and how to fix it.
Flow: Performance
/traceway perf <endpoint or symptom>, or any request about slowness, high latency, p95/p99, or "why is X slow". Approach it as a senior performance engineer: quantify first, localize the cost to a span, confirm the cause against the checklist, separate code from saturation, then root cause. Reads only; never archive or mutate.
For the full methodology and the bottleneck checklist (Select N+1, missing index, chatty IPC, connection-pool exhaustion, GC pauses, and ~20 more, each tied to the Traceway signal that confirms it), read performance.md in this skill directory (over MCP: the traceway://knowledge/performance resource). It is the authoritative reference.
The loop, using the read commands documented in this skill:
- Quantify and localize.
traceway endpoints list --since 24h --order-by p95(orimpact). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note thatimpact/impactReasonare offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not. - Check the accepted baseline. Before calling an endpoint slow, run
traceway endpoints slow "<endpoint>". A non-zerooffsetMsmeans an operator accepted that much latency (with areason), so only latency beyond the offset is a real regression;offsetMs 0means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown. - Pinpoint when it started: adjust the window down. Do not investigate the default window blindly; narrow it until it brackets the onset.
traceway endpoints chart --metric-type p95 --interval-minutes <n>returns latency over time for the top endpoints ({timestamp, endpoint, value}in ms): read down one endpoint's buckets for the first one where p95 has jumped. Timestamps are bucket starts, so the onset is inside that bucket, not before it: re-run just that bucket with--from/--toand an interval of 15 minutes or less before you report the time. Confirm the cause withtraceway metrics query --name <metric> --interval-minutes <n>(the infra/runtime curve). For a specific route not in the top 5, bisecttraceway endpoints list --search <name> --from <a> --to <b>over adjacent windows. Seeperformance.md, "Pinpointing when the slowness started". - Get a representative slow trace.
traceway endpoints show <endpointId> --recorded-at <t>for one request's span waterfall, ortraceway traces show <traceId> --recorded-at <t>(thetraceIdthatendpoints showprints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole. - Match the long pole to the checklist. Is the dominant span a database call, an external call, in-process compute, or a gap before work starts (queueing / lock wait / pool exhaustion)? Look it up in
performance.md. - Separate code from saturation. Pull infra/runtime metrics over the same window:
traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min(busy is 1 minus idle; the metric is split bystate, so never read it unfiltered), thensystem.memory.utilizationwithstate=usedand the runtime's heap and GC metrics (jvm.*,go.*,v8js.*). A spike that lines up with the latency onset points at saturation, not a code bug. - Correlate with code and deploys. With the onset time from step 3, check what shipped then:
git log --since '<onset - 30m>' --until '<onset + 30m>'or the deploy history. A jump at a release is a regression; a gradual ramp with no deploy is data growth (N+1, missing index, no pagination). - Report. The bottleneck, the evidence (endpoint percentiles, the onset time, the dominating span, any correlated metric anomaly), the root cause mapped to a checklist item, and a concrete fix, with the
endpoints show/traces showreferences so the user can verify. If the endpoint is marked slow, state whether the latency exceeded the accepted offset.
If the waterfall does not localize the cost, stop before proposing a fix. An unexplained gap, one fat span covering the whole handler, or a tail you cannot reproduce all mean the same thing: the trace is too coarse to act on. Add child spans around the suspect steps inside that handler, then wait for the next slow request to point at the real one. See "When the evidence is not enough" in the Debug flow.