The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
More improvements to evals
narrow-adventure3b00b08318cd
diff --git a/cli/cmd/traceway/endpoints.go b/cli/cmd/traceway/endpoints.goindex da9c17ee..7184286d 100644--- a/cli/cmd/traceway/endpoints.go+++ b/cli/cmd/traceway/endpoints.go@@ -185,7 +185,9 @@ func newEndpointsChartCmd() *cobra.Command {Long: `Show endpoint latency as a time series, the curve behind /endpoints in thedashboard. The server returns the top 5 endpoints ranked by --metric-type plusan "Other" bucket, each bucketed by --interval-minutes, so you can read down the-buckets to find when a latency jump started.+buckets to find when a latency jump started. Each timestamp is the start of its+bucket: the jump happened inside the first bucket that shows it. Re-run that+bucket with --from/--to and a finer --interval-minutes to date it closely.This always returns the top endpoints by the chosen metric; it does not filterto one route. To track a specific endpoint that is not in the top 5, bisectdiff --git a/cli/pkg/mcpserver/knowledge/performance-flow.md b/cli/pkg/mcpserver/knowledge/performance-flow.mdindex 2c5b0c7f..8426cca0 100644--- a/cli/pkg/mcpserver/knowledge/performance-flow.md+++ b/cli/pkg/mcpserver/knowledge/performance-flow.md@@ -8,7 +8,7 @@ The loop, using the read commands documented in this skill:1. **Quantify and localize.** `traceway endpoints list --since 24h --order-by p95` (or `impact`). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note that `impact`/`impactReason` are offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not.2. **Check the accepted baseline.** Before calling an endpoint slow, run `traceway endpoints slow "<endpoint>"`. A non-zero `offsetMs` means an operator accepted that much latency (with a `reason`), so only latency *beyond* the offset is a real regression; `offsetMs 0` means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown.-3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".+3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the first one where p95 has jumped. Timestamps are bucket starts, so the onset is inside that bucket, not before it: re-run just that bucket with `--from`/`--to` and an interval of 15 minutes or less before you report the time. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <traceId> --recorded-at <t>` (the `traceId` that `endpoints show` prints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min` (busy is 1 minus idle; the metric is split by `state`, so never read it unfiltered), then `system.memory.utilization` with `state=used` and the runtime's heap and GC metrics (`jvm.*`, `go.*`, `v8js.*`). A spike that lines up with the latency onset points at saturation, not a code bug.diff --git a/cli/pkg/mcpserver/knowledge/performance.md b/cli/pkg/mcpserver/knowledge/performance.mdindex d7012bf4..c52150b3 100644--- a/cli/pkg/mcpserver/knowledge/performance.md+++ b/cli/pkg/mcpserver/knowledge/performance.md@@ -71,14 +71,17 @@ When you report, state the p95 and the allowance together: "p95 is 2.1s but thisNarrowing the window is how you turn "it is slow" into "it got slow at 14:10, right after deploy X". Do not investigate the default window blindly: adjust it down until it brackets the onset, then correlate that moment with deploys and metrics.-**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Read down one endpoint's buckets for the step where its value jumps: that bucket dates the onset. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first, then re-run the suspect span with a finer interval to sharpen.+**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Each `timestamp` is the **start** of its bucket. Read down one endpoint's buckets for the first one whose value has jumped: the onset lies inside that bucket, between its timestamp and its timestamp plus the interval, not in the bucket before it. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first to find that bucket. A coarse bucket only brackets the onset: with 60-minute buckets its timestamp can be 59 minutes early, with 6-hour buckets almost 6 hours. Before you report an onset time, re-run over just that bucket with an interval of 15 minutes or less, and report the start of the first elevated fine bucket.```bashtraceway endpoints chart --metric-type p95 --since 7d --interval-minutes 60 --output json \| jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'+# the 60-minute bucket starting 14:00 stepped up: sharpen it in 5-minute buckets+traceway endpoints chart --metric-type p95 --from 2026-06-25T14:00:00Z --to 2026-06-25T15:00:00Z --interval-minutes 5 --output json \+ | jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'```Caveat: it always returns the *top 5* endpoints by that metric. If your suspect route is not in the top 5 it is folded into "Other"; fall back to method 3 for it.-**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the bucket where the value steps up.+**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the first bucket where the value has stepped up. Timestamps are bucket starts here too, so sharpen that bucket the same way before you report a time.```bashtraceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min --since 7d --interval-minutes 60 --output json \| jq '.results[0].series | to_entries[] | .value[] | {t: .timestamp, busy: (1 - .value)}'@@ -93,7 +96,7 @@ traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T00:00:00Ztraceway endpoints list --search "GET /api/checkout" --from 2026-06-25T12:00:00Z --to 2026-06-26T00:00:00Z \--output json | jq '.data[0] | {p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), count}'```-A few bisection steps localize the jump to an hour or less. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.+Keep bisecting until the elevated window is 15 minutes or less, then report its start. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.**4. Cross-check and correlate.** Once the methods agree on an onset time:- A latency jump that coincides with a metric inflection (CPU or GC stepping up at the same bucket) points at saturation.diff --git a/cli/pkg/mcpserver/tools.go b/cli/pkg/mcpserver/tools.goindex 2eb3ff2f..4278094e 100644--- a/cli/pkg/mcpserver/tools.go+++ b/cli/pkg/mcpserver/tools.go@@ -69,7 +69,7 @@ func (s *server) addTools(srv *mcp.Server) {mcp.AddTool(srv, &mcp.Tool{Name: "endpoints_chart",OutputSchema: outSchema[client.EndpointChartResponse](),- Description: "Endpoint latency bucketed over time: the top 5 endpoints ranked by metric_type (plus an Other bucket), each as a {timestamp, endpoint, value} series in milliseconds. Use it to pinpoint WHEN latency changed: read down one endpoint's buckets for the step. A non-top-5 endpoint folds into Other; bisect list_endpoints over narrowing from/to windows for it instead.",+ Description: "Endpoint latency bucketed over time: the top 5 endpoints ranked by metric_type (plus an Other bucket), each as a {timestamp, endpoint, value} series in milliseconds. Use it to pinpoint WHEN latency changed: read down one endpoint's buckets for the first one that jumped. Each timestamp is the start of its bucket, so the change happened inside that bucket; re-run it with from/to and an interval_minutes of 15 or less before reporting the time. A non-top-5 endpoint folds into Other; bisect list_endpoints over narrowing from/to windows for it instead.",Annotations: readOnly(),}, s.endpointsChart)diff --git a/cli/test/evals/internal/env/build.go b/cli/test/evals/internal/env/build.goindex e503e4ff..1b7cc21e 100644--- a/cli/test/evals/internal/env/build.go+++ b/cli/test/evals/internal/env/build.go@@ -9,6 +9,7 @@ import ("path/filepath""strings""sync"+ "syscall")// Binaries are the executables a run needs.@@ -46,6 +47,14 @@ func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {if err := os.MkdirAll(dir, 0o755); err != nil {return nil, err}+ // buildMu only covers this process, and go test ./... runs one process+ // per package: without this lock each package that boots a backend+ // compiles it at the same time, several GB of compiler memory at once.+ unlock, err := lockFile(filepath.Join(cacheDir, "bin", ".build.lock"))+ if err != nil {+ return nil, err+ }+ defer unlock()steps := []struct {out, dir stringargs []string@@ -58,8 +67,8 @@ func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {if _, err := os.Stat(s.out); err == nil {continue}- // Build to a private path and rename, so concurrent harness- // processes never see (or execute) a half-written binary.+ // Build to a private path and rename, so a build killed halfway+ // never leaves a binary the Stat above would accept.tmp := fmt.Sprintf("%s.tmp-%d", s.out, os.Getpid())args := append([]string{}, s.args...)for i := range args {@@ -81,6 +90,21 @@ func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {return b, nil}+// lockFile takes an exclusive flock on path, waiting for any other holder.+// The kernel drops the lock when its holder exits, so a killed build never+// leaves it stuck.+func lockFile(path string) (func(), error) {+ f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)+ if err != nil {+ return nil, err+ }+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {+ f.Close()+ return nil, fmt.Errorf("lock %s: %w", path, err)+ }+ return func() { f.Close() }, nil+}+// TreeHash fingerprints the backend and CLI sources: the committed trees,// the uncommitted diff, and untracked files. The evals module itself is// excluded so editing a case never rebuilds the binaries.diff --git a/cli/test/evals/internal/env/build_test.go b/cli/test/evals/internal/env/build_test.gonew file mode 100644index 00000000..9f9cb1be--- /dev/null+++ b/cli/test/evals/internal/env/build_test.go@@ -0,0 +1,39 @@+package env++import (+ "path/filepath"+ "testing"+ "time"+)++// TestLockFileExcludes holds the build lock and checks that a second holder+// waits for it. flock locks belong to the open file, so two opens in one+// process exclude each other the same way two processes do.+func TestLockFileExcludes(t *testing.T) {+ path := filepath.Join(t.TempDir(), ".build.lock")+ unlock, err := lockFile(path)+ if err != nil {+ t.Fatal(err)+ }+ got := make(chan struct{})+ go func() {+ defer close(got)+ second, err := lockFile(path)+ if err != nil {+ t.Error(err)+ return+ }+ second()+ }()+ select {+ case <-got:+ t.Fatal("a second holder took the lock while the first still held it")+ case <-time.After(200 * time.Millisecond):+ }+ unlock()+ select {+ case <-got:+ case <-time.After(5 * time.Second):+ t.Fatal("the lock was not handed over after unlock")+ }+}diff --git a/cli/test/evals/internal/seed/dsl.go b/cli/test/evals/internal/seed/dsl.goindex c260e1fd..c11d5680 100644--- a/cli/test/evals/internal/seed/dsl.go+++ b/cli/test/evals/internal/seed/dsl.go@@ -44,6 +44,8 @@ type Scenario struct {// SpanBudget caps the spans sent with full detail (default 20000).SpanBudget intafterFlush []func() error+ // rows maps each endpoint row's id to the request that holds its span.+ rows map[string]*Request}// Stats summarizes what a seed sent.@@ -380,6 +382,7 @@ type Request struct {Status intDetail boolforced bool+ listed bool // detail only because a listing shows it (listedRequests): no access log lineProgress float64ParentRef *Span // for downstream SERVER spans: the CLIENT span calling it}diff --git a/cli/test/evals/internal/seed/flush.go b/cli/test/evals/internal/seed/flush.goindex 3f2affc8..b56ca730 100644--- a/cli/test/evals/internal/seed/flush.go+++ b/cli/test/evals/internal/seed/flush.go@@ -40,6 +40,7 @@ func (s *Scenario) Flush() error {all = append(all, e.Requests()...)}}+ listed := s.listedRequests(all)keys := make([]string, 0, len(s.lazy))for k := range s.lazy {keys = append(keys, k)@@ -49,6 +50,11 @@ func (s *Scenario) Flush() error {s.exposed[k] = s.lazy[k]()}s.enforceBudget(all)+ for _, r := range listed {+ if !r.Detail {+ r.Detail, r.listed = true, true+ }+ }sort.SliceStable(all, func(i, j int) bool { return all[i].Start.Before(all[j].Start) })batches := map[*Project]*spanBatch{}@@ -181,6 +187,124 @@ func serverSpans(sp *Span) int {return n}+// How many requests from the top of a request listing get their full span+// tree. Every trailing window keeps its listedSlowest slowest (the default+// order) and its listedOthers fastest and newest; every UTC day, for+// --from/--to bisection, its listedPerDay slowest. Verify checks half of the+// trailing ones, which leaves room for the requests that age out of a window+// between the seed and the agent's query.+const (+ listedSlowest = 10+ listedOthers = 4+ listedPerDay = 3+)++var listingWindows = []time.Duration{time.Hour, 6 * time.Hour, 24 * time.Hour, 3 * 24 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour}++// listingDrift is where a listing's window may end relative to s.Now: before+// it in a scenario that plays out in real time (its telemetry runs ahead of+// the clock), after it by up to a run's length. The hour is taken at each+// end, since a few minutes move the top of a busy hour; longer windows barely+// move.+var listingDrift = []time.Duration{-10 * time.Minute, -5 * time.Minute, 0, 5 * time.Minute, 10 * time.Minute, 20 * time.Minute}++type listedRow struct {+ r *Request+ start time.Time+ dur time.Duration+}++// listedRequests returns the requests a request listing (endpoints requests,+// list_endpoint_requests) shows first, which Flush sends with their full span+// tree. The random detail sample almost never holds the slowest requests, so+// an agent that opened one used to find no spans at all. Flush adds them after+// enforceBudget, without an access log line, and this reads no random+// numbers, so the sample and everything else generated stay the same.+func (s *Scenario) listedRequests(all []*Request) []*Request {+ type key struct {+ p *Project+ name string+ }+ groups := map[key][]listedRow{}+ // Only endpoints with traffic of their own: one reached only through+ // Call is seen in its caller's waterfall, and covering its listing would+ // send most of the callers' trees.+ listable := map[key]bool{}+ for _, svc := range s.services {+ for _, e := range svc.endpoints {+ if len(e.shapes) > 0 {+ listable[key{svc.p, e.Name()}] = true+ }+ }+ }+ s.rows = map[string]*Request{}+ for _, r := range all {+ var walk func(sp *Span, svc *Service, abs time.Time)+ walk = func(sp *Span, svc *Service, abs time.Time) {+ if sp.Service != nil {+ svc = sp.Service+ }+ start := abs.Add(sp.Start)+ if sp.Kind == KindServer {+ k := key{svc.p, sp.Name}+ groups[k] = append(groups[k], listedRow{r, start, sp.Dur})+ s.rows[EntityID(svc.p.ID, r.TraceID, sp.SpanID)] = r+ }+ for _, c := range sp.Children {+ walk(c, svc, start)+ }+ }+ walk(r.Root, r.e.svc, r.Start)+ }+ slowest := func(a, b listedRow) bool { return a.dur > b.dur }+ fastest := func(a, b listedRow) bool { return a.dur < b.dur }+ newest := func(a, b listedRow) bool { return a.start.After(b.start) }+ picked := map[*Request]bool{}+ var listed []*Request+ keep := func(rows []listedRow, from, to time.Time, n int, less func(a, b listedRow) bool) {+ var in []listedRow+ for _, x := range rows {+ if !x.start.Before(from) && x.start.Before(to) {+ in = append(in, x)+ }+ }+ sort.SliceStable(in, func(i, j int) bool { return less(in[i], in[j]) })+ for _, x := range in[:min(n, len(in))] {+ if !picked[x.r] {+ picked[x.r] = true+ listed = append(listed, x.r)+ }+ }+ }+ for k, rows := range groups {+ if !listable[k] {+ continue+ }+ for _, d := range listingWindows {+ ends := []time.Duration{0}+ if d == time.Hour {+ ends = listingDrift+ }+ for _, off := range ends {+ to := s.Now.Add(off)+ keep(rows, to.Add(-d), to, listedSlowest, slowest)+ keep(rows, to.Add(-d), to, listedOthers, fastest)+ keep(rows, to.Add(-d), to, listedOthers, newest)+ }+ }+ earliest := rows[0].start+ for _, x := range rows {+ if x.start.Before(earliest) {+ earliest = x.start+ }+ }+ for day := earliest.UTC().Truncate(24 * time.Hour); day.Before(s.Now); day = day.Add(24 * time.Hour) {+ keep(rows, day, day.Add(24*time.Hour), listedPerDay, slowest)+ }+ }+ return listed+}+// enforceBudget keeps the detailed span volume under SpanBudget by dropping// sampled (not forced) detail, oldest first.func (s *Scenario) enforceBudget(all []*Request) {diff --git a/cli/test/evals/internal/seed/traces.go b/cli/test/evals/internal/seed/traces.goindex 38aa5ace..c8a93bad 100644--- a/cli/test/evals/internal/seed/traces.go+++ b/cli/test/evals/internal/seed/traces.go@@ -230,7 +230,7 @@ func (s *Scenario) emit(r *Request, batches map[*Project]*spanBatch, logs map[*S}}walk(r.Root, nil, rootSvc, r.Host, r.Start, r.Detail)- if r.Detail && !r.e.Task && !r.e.SkipAccessLog {+ if r.Detail && !r.listed && !r.e.Task && !r.e.SkipAccessLog {// The access log line a request logger writes on completion,// correlated with the trace like any OTel log bridge does.end := r.Start.Add(r.Root.Dur)diff --git a/cli/test/evals/internal/seed/verify.go b/cli/test/evals/internal/seed/verify.goindex b35a81a2..c75682b1 100644--- a/cli/test/evals/internal/seed/verify.go+++ b/cli/test/evals/internal/seed/verify.go@@ -64,6 +64,70 @@ func (s *Scenario) Verify(timeout time.Duration) error {}time.Sleep(250 * time.Millisecond)}+ if err := s.verifyListings(w.p); err != nil {+ return fmt.Errorf("verify %s: %w", w.p.Name, err)+ }+ }+ return nil+}++// verifyListings checks the first page of every endpoint's request listing,+// in the windows and orders agents ask for: each request on it must have been+// sent with its span tree (see listedRequests).+func (s *Scenario) verifyListings(p *Project) error {+ now := time.Now()+ orders := []struct {+ by, dir string+ n int+ }{{"duration", "desc", listedSlowest / 2}, {"duration", "asc", listedOthers / 2}, {"recorded_at", "desc", listedOthers / 2}}+ for _, svc := range s.services {+ if svc.p != p {+ continue+ }+ for _, e := range svc.endpoints {+ if len(e.shapes) == 0 {+ continue+ }+ q := env.ProjectQuery(p.ID)+ q.Set("endpoint", e.Name())+ for _, win := range []time.Duration{time.Hour, 24 * time.Hour, 7 * 24 * time.Hour} {+ for _, o := range orders {+ res, err := s.Sink.Admin(http.MethodPost, "/api/endpoints/endpoint", q, map[string]any{+ "fromDate": now.Add(-win).Format(time.RFC3339), "toDate": now.Format(time.RFC3339),+ "orderBy": o.by, "sortDirection": o.dir, "pagination": map[string]int{"page": 1, "pageSize": o.n},+ })+ if err != nil {+ return err+ }+ if err := res.Err("list requests"); err != nil {+ return err+ }+ var out struct {+ Data []struct {+ ID string `json:"id"`+ } `json:"data"`+ }+ if err := json.Unmarshal(res.Body, &out); err != nil {+ return err+ }+ known := 0+ for _, d := range out.Data {+ r, ok := s.rows[d.ID]+ if !ok {+ continue+ }+ known+++ if !r.Detail {+ return fmt.Errorf("%s requests over the last %s by %s %s list %s, which was sent without its span tree", e.Name(), win, o.by, o.dir, d.ID)+ }+ }+ // Ids that never match would make this check pass on anything.+ if len(out.Data) > 0 && known == 0 {+ return fmt.Errorf("%s requests over the last %s: none of the %d listed ids is a seeded request", e.Name(), win, len(out.Data))+ }+ }+ }+ }}return nil}diff --git a/skills/traceway/SKILL.md b/skills/traceway/SKILL.mdindex df481ec3..af945d2d 100644--- a/skills/traceway/SKILL.md+++ b/skills/traceway/SKILL.md@@ -337,7 +337,7 @@ The loop, using the read commands documented in this skill:1. **Quantify and localize.** `traceway endpoints list --since 24h --order-by p95` (or `impact`). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note that `impact`/`impactReason` are offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not.2. **Check the accepted baseline.** Before calling an endpoint slow, run `traceway endpoints slow "<endpoint>"`. A non-zero `offsetMs` means an operator accepted that much latency (with a `reason`), so only latency *beyond* the offset is a real regression; `offsetMs 0` means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown.-3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".+3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the first one where p95 has jumped. Timestamps are bucket starts, so the onset is inside that bucket, not before it: re-run just that bucket with `--from`/`--to` and an interval of 15 minutes or less before you report the time. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <traceId> --recorded-at <t>` (the `traceId` that `endpoints show` prints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min` (busy is 1 minus idle; the metric is split by `state`, so never read it unfiltered), then `system.memory.utilization` with `state=used` and the runtime's heap and GC metrics (`jvm.*`, `go.*`, `v8js.*`). A spike that lines up with the latency onset points at saturation, not a code bug.diff --git a/skills/traceway/performance.md b/skills/traceway/performance.mdindex fe89e34b..ccaafb29 100644--- a/skills/traceway/performance.md+++ b/skills/traceway/performance.md@@ -72,14 +72,17 @@ When you report, state the p95 and the allowance together: "p95 is 2.1s but thisNarrowing the window is how you turn "it is slow" into "it got slow at 14:10, right after deploy X". Do not investigate the default window blindly: adjust it down until it brackets the onset, then correlate that moment with deploys and metrics.-**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Read down one endpoint's buckets for the step where its value jumps: that bucket dates the onset. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first, then re-run the suspect span with a finer interval to sharpen.+**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Each `timestamp` is the **start** of its bucket. Read down one endpoint's buckets for the first one whose value has jumped: the onset lies inside that bucket, between its timestamp and its timestamp plus the interval, not in the bucket before it. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first to find that bucket. A coarse bucket only brackets the onset: with 60-minute buckets its timestamp can be 59 minutes early, with 6-hour buckets almost 6 hours. Before you report an onset time, re-run over just that bucket with an interval of 15 minutes or less, and report the start of the first elevated fine bucket.```bashtraceway endpoints chart --metric-type p95 --since 7d --interval-minutes 60 --output json \| jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'+# the 60-minute bucket starting 14:00 stepped up: sharpen it in 5-minute buckets+traceway endpoints chart --metric-type p95 --from 2026-06-25T14:00:00Z --to 2026-06-25T15:00:00Z --interval-minutes 5 --output json \+ | jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'```Caveat: it always returns the *top 5* endpoints by that metric. If your suspect route is not in the top 5 it is folded into "Other"; fall back to method 3 for it.-**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the bucket where the value steps up.+**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the first bucket where the value has stepped up. Timestamps are bucket starts here too, so sharpen that bucket the same way before you report a time.```bashtraceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min --since 7d --interval-minutes 60 --output json \| jq '.results[0].series | to_entries[] | .value[] | {t: .timestamp, busy: (1 - .value)}'@@ -94,7 +97,7 @@ traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T00:00:00Ztraceway endpoints list --search "GET /api/checkout" --from 2026-06-25T12:00:00Z --to 2026-06-26T00:00:00Z \--output json | jq '.data[0] | {p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), count}'```-A few bisection steps localize the jump to an hour or less. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.+Keep bisecting until the elevated window is 15 minutes or less, then report its start. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.**4. Cross-check and correlate.** Once the methods agree on an onset time:- A latency jump that coincides with a metric inflection (CPU or GC stepping up at the same bucket) points at saturation.