Skip to content

The only tool you need to know what is happening and how to fix it.

More improvements to evals

narrow-adventure3b00b08318cd
diff --git a/cli/cmd/traceway/endpoints.go b/cli/cmd/traceway/endpoints.go
index da9c17ee..7184286d 100644
--- a/cli/cmd/traceway/endpoints.go
+++ b/cli/cmd/traceway/endpoints.go
@@ -185,7 +185,9 @@ func newEndpointsChartCmd() *cobra.Command {
Long: `Show endpoint latency as a time series, the curve behind /endpoints in the
dashboard. The server returns the top 5 endpoints ranked by --metric-type plus
an "Other" bucket, each bucketed by --interval-minutes, so you can read down the
-buckets to find when a latency jump started.
+buckets to find when a latency jump started. Each timestamp is the start of its
+bucket: the jump happened inside the first bucket that shows it. Re-run that
+bucket with --from/--to and a finer --interval-minutes to date it closely.
This always returns the top endpoints by the chosen metric; it does not filter
to one route. To track a specific endpoint that is not in the top 5, bisect
diff --git a/cli/pkg/mcpserver/knowledge/performance-flow.md b/cli/pkg/mcpserver/knowledge/performance-flow.md
index 2c5b0c7f..8426cca0 100644
--- a/cli/pkg/mcpserver/knowledge/performance-flow.md
+++ b/cli/pkg/mcpserver/knowledge/performance-flow.md
@@ -8,7 +8,7 @@ The loop, using the read commands documented in this skill:
1. **Quantify and localize.** `traceway endpoints list --since 24h --order-by p95` (or `impact`). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note that `impact`/`impactReason` are offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not.
2. **Check the accepted baseline.** Before calling an endpoint slow, run `traceway endpoints slow "<endpoint>"`. A non-zero `offsetMs` means an operator accepted that much latency (with a `reason`), so only latency *beyond* the offset is a real regression; `offsetMs 0` means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown.
-3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".
+3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the first one where p95 has jumped. Timestamps are bucket starts, so the onset is inside that bucket, not before it: re-run just that bucket with `--from`/`--to` and an interval of 15 minutes or less before you report the time. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".
4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <traceId> --recorded-at <t>` (the `traceId` that `endpoints show` prints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.
5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.
6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min` (busy is 1 minus idle; the metric is split by `state`, so never read it unfiltered), then `system.memory.utilization` with `state=used` and the runtime's heap and GC metrics (`jvm.*`, `go.*`, `v8js.*`). A spike that lines up with the latency onset points at saturation, not a code bug.
diff --git a/cli/pkg/mcpserver/knowledge/performance.md b/cli/pkg/mcpserver/knowledge/performance.md
index d7012bf4..c52150b3 100644
--- a/cli/pkg/mcpserver/knowledge/performance.md
+++ b/cli/pkg/mcpserver/knowledge/performance.md
@@ -71,14 +71,17 @@ When you report, state the p95 and the allowance together: "p95 is 2.1s but this
Narrowing the window is how you turn "it is slow" into "it got slow at 14:10, right after deploy X". Do not investigate the default window blindly: adjust it down until it brackets the onset, then correlate that moment with deploys and metrics.
-**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Read down one endpoint's buckets for the step where its value jumps: that bucket dates the onset. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first, then re-run the suspect span with a finer interval to sharpen.
+**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Each `timestamp` is the **start** of its bucket. Read down one endpoint's buckets for the first one whose value has jumped: the onset lies inside that bucket, between its timestamp and its timestamp plus the interval, not in the bucket before it. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first to find that bucket. A coarse bucket only brackets the onset: with 60-minute buckets its timestamp can be 59 minutes early, with 6-hour buckets almost 6 hours. Before you report an onset time, re-run over just that bucket with an interval of 15 minutes or less, and report the start of the first elevated fine bucket.
```bash
traceway endpoints chart --metric-type p95 --since 7d --interval-minutes 60 --output json \
| jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'
+# the 60-minute bucket starting 14:00 stepped up: sharpen it in 5-minute buckets
+traceway endpoints chart --metric-type p95 --from 2026-06-25T14:00:00Z --to 2026-06-25T15:00:00Z --interval-minutes 5 --output json \
+ | jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'
```
Caveat: it always returns the *top 5* endpoints by that metric. If your suspect route is not in the top 5 it is folded into "Other"; fall back to method 3 for it.
-**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the bucket where the value steps up.
+**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the first bucket where the value has stepped up. Timestamps are bucket starts here too, so sharpen that bucket the same way before you report a time.
```bash
traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min --since 7d --interval-minutes 60 --output json \
| jq '.results[0].series | to_entries[] | .value[] | {t: .timestamp, busy: (1 - .value)}'
@@ -93,7 +96,7 @@ traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T00:00:00Z
traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T12:00:00Z --to 2026-06-26T00:00:00Z \
--output json | jq '.data[0] | {p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), count}'
```
-A few bisection steps localize the jump to an hour or less. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.
+Keep bisecting until the elevated window is 15 minutes or less, then report its start. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.
**4. Cross-check and correlate.** Once the methods agree on an onset time:
- A latency jump that coincides with a metric inflection (CPU or GC stepping up at the same bucket) points at saturation.
diff --git a/cli/pkg/mcpserver/tools.go b/cli/pkg/mcpserver/tools.go
index 2eb3ff2f..4278094e 100644
--- a/cli/pkg/mcpserver/tools.go
+++ b/cli/pkg/mcpserver/tools.go
@@ -69,7 +69,7 @@ func (s *server) addTools(srv *mcp.Server) {
mcp.AddTool(srv, &mcp.Tool{
Name: "endpoints_chart",
OutputSchema: outSchema[client.EndpointChartResponse](),
- Description: "Endpoint latency bucketed over time: the top 5 endpoints ranked by metric_type (plus an Other bucket), each as a {timestamp, endpoint, value} series in milliseconds. Use it to pinpoint WHEN latency changed: read down one endpoint's buckets for the step. A non-top-5 endpoint folds into Other; bisect list_endpoints over narrowing from/to windows for it instead.",
+ Description: "Endpoint latency bucketed over time: the top 5 endpoints ranked by metric_type (plus an Other bucket), each as a {timestamp, endpoint, value} series in milliseconds. Use it to pinpoint WHEN latency changed: read down one endpoint's buckets for the first one that jumped. Each timestamp is the start of its bucket, so the change happened inside that bucket; re-run it with from/to and an interval_minutes of 15 or less before reporting the time. A non-top-5 endpoint folds into Other; bisect list_endpoints over narrowing from/to windows for it instead.",
Annotations: readOnly(),
}, s.endpointsChart)
diff --git a/cli/test/evals/internal/env/build.go b/cli/test/evals/internal/env/build.go
index e503e4ff..1b7cc21e 100644
--- a/cli/test/evals/internal/env/build.go
+++ b/cli/test/evals/internal/env/build.go
@@ -9,6 +9,7 @@ import (
"path/filepath"
"strings"
"sync"
+ "syscall"
)
// Binaries are the executables a run needs.
@@ -46,6 +47,14 @@ func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, err
}
+ // buildMu only covers this process, and go test ./... runs one process
+ // per package: without this lock each package that boots a backend
+ // compiles it at the same time, several GB of compiler memory at once.
+ unlock, err := lockFile(filepath.Join(cacheDir, "bin", ".build.lock"))
+ if err != nil {
+ return nil, err
+ }
+ defer unlock()
steps := []struct {
out, dir string
args []string
@@ -58,8 +67,8 @@ func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {
if _, err := os.Stat(s.out); err == nil {
continue
}
- // Build to a private path and rename, so concurrent harness
- // processes never see (or execute) a half-written binary.
+ // Build to a private path and rename, so a build killed halfway
+ // never leaves a binary the Stat above would accept.
tmp := fmt.Sprintf("%s.tmp-%d", s.out, os.Getpid())
args := append([]string{}, s.args...)
for i := range args {
@@ -81,6 +90,21 @@ func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {
return b, nil
}
+// lockFile takes an exclusive flock on path, waiting for any other holder.
+// The kernel drops the lock when its holder exits, so a killed build never
+// leaves it stuck.
+func lockFile(path string) (func(), error) {
+ f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)
+ if err != nil {
+ return nil, err
+ }
+ if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {
+ f.Close()
+ return nil, fmt.Errorf("lock %s: %w", path, err)
+ }
+ return func() { f.Close() }, nil
+}
+
// TreeHash fingerprints the backend and CLI sources: the committed trees,
// the uncommitted diff, and untracked files. The evals module itself is
// excluded so editing a case never rebuilds the binaries.
diff --git a/cli/test/evals/internal/env/build_test.go b/cli/test/evals/internal/env/build_test.go
new file mode 100644
index 00000000..9f9cb1be
--- /dev/null
+++ b/cli/test/evals/internal/env/build_test.go
@@ -0,0 +1,39 @@
+package env
+
+import (
+ "path/filepath"
+ "testing"
+ "time"
+)
+
+// TestLockFileExcludes holds the build lock and checks that a second holder
+// waits for it. flock locks belong to the open file, so two opens in one
+// process exclude each other the same way two processes do.
+func TestLockFileExcludes(t *testing.T) {
+ path := filepath.Join(t.TempDir(), ".build.lock")
+ unlock, err := lockFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ got := make(chan struct{})
+ go func() {
+ defer close(got)
+ second, err := lockFile(path)
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ second()
+ }()
+ select {
+ case <-got:
+ t.Fatal("a second holder took the lock while the first still held it")
+ case <-time.After(200 * time.Millisecond):
+ }
+ unlock()
+ select {
+ case <-got:
+ case <-time.After(5 * time.Second):
+ t.Fatal("the lock was not handed over after unlock")
+ }
+}
diff --git a/cli/test/evals/internal/seed/dsl.go b/cli/test/evals/internal/seed/dsl.go
index c260e1fd..c11d5680 100644
--- a/cli/test/evals/internal/seed/dsl.go
+++ b/cli/test/evals/internal/seed/dsl.go
@@ -44,6 +44,8 @@ type Scenario struct {
// SpanBudget caps the spans sent with full detail (default 20000).
SpanBudget int
afterFlush []func() error
+ // rows maps each endpoint row's id to the request that holds its span.
+ rows map[string]*Request
}
// Stats summarizes what a seed sent.
@@ -380,6 +382,7 @@ type Request struct {
Status int
Detail bool
forced bool
+ listed bool // detail only because a listing shows it (listedRequests): no access log line
Progress float64
ParentRef *Span // for downstream SERVER spans: the CLIENT span calling it
}
diff --git a/cli/test/evals/internal/seed/flush.go b/cli/test/evals/internal/seed/flush.go
index 3f2affc8..b56ca730 100644
--- a/cli/test/evals/internal/seed/flush.go
+++ b/cli/test/evals/internal/seed/flush.go
@@ -40,6 +40,7 @@ func (s *Scenario) Flush() error {
all = append(all, e.Requests()...)
}
}
+ listed := s.listedRequests(all)
keys := make([]string, 0, len(s.lazy))
for k := range s.lazy {
keys = append(keys, k)
@@ -49,6 +50,11 @@ func (s *Scenario) Flush() error {
s.exposed[k] = s.lazy[k]()
}
s.enforceBudget(all)
+ for _, r := range listed {
+ if !r.Detail {
+ r.Detail, r.listed = true, true
+ }
+ }
sort.SliceStable(all, func(i, j int) bool { return all[i].Start.Before(all[j].Start) })
batches := map[*Project]*spanBatch{}
@@ -181,6 +187,124 @@ func serverSpans(sp *Span) int {
return n
}
+// How many requests from the top of a request listing get their full span
+// tree. Every trailing window keeps its listedSlowest slowest (the default
+// order) and its listedOthers fastest and newest; every UTC day, for
+// --from/--to bisection, its listedPerDay slowest. Verify checks half of the
+// trailing ones, which leaves room for the requests that age out of a window
+// between the seed and the agent's query.
+const (
+ listedSlowest = 10
+ listedOthers = 4
+ listedPerDay = 3
+)
+
+var listingWindows = []time.Duration{time.Hour, 6 * time.Hour, 24 * time.Hour, 3 * 24 * time.Hour, 7 * 24 * time.Hour, 30 * 24 * time.Hour}
+
+// listingDrift is where a listing's window may end relative to s.Now: before
+// it in a scenario that plays out in real time (its telemetry runs ahead of
+// the clock), after it by up to a run's length. The hour is taken at each
+// end, since a few minutes move the top of a busy hour; longer windows barely
+// move.
+var listingDrift = []time.Duration{-10 * time.Minute, -5 * time.Minute, 0, 5 * time.Minute, 10 * time.Minute, 20 * time.Minute}
+
+type listedRow struct {
+ r *Request
+ start time.Time
+ dur time.Duration
+}
+
+// listedRequests returns the requests a request listing (endpoints requests,
+// list_endpoint_requests) shows first, which Flush sends with their full span
+// tree. The random detail sample almost never holds the slowest requests, so
+// an agent that opened one used to find no spans at all. Flush adds them after
+// enforceBudget, without an access log line, and this reads no random
+// numbers, so the sample and everything else generated stay the same.
+func (s *Scenario) listedRequests(all []*Request) []*Request {
+ type key struct {
+ p *Project
+ name string
+ }
+ groups := map[key][]listedRow{}
+ // Only endpoints with traffic of their own: one reached only through
+ // Call is seen in its caller's waterfall, and covering its listing would
+ // send most of the callers' trees.
+ listable := map[key]bool{}
+ for _, svc := range s.services {
+ for _, e := range svc.endpoints {
+ if len(e.shapes) > 0 {
+ listable[key{svc.p, e.Name()}] = true
+ }
+ }
+ }
+ s.rows = map[string]*Request{}
+ for _, r := range all {
+ var walk func(sp *Span, svc *Service, abs time.Time)
+ walk = func(sp *Span, svc *Service, abs time.Time) {
+ if sp.Service != nil {
+ svc = sp.Service
+ }
+ start := abs.Add(sp.Start)
+ if sp.Kind == KindServer {
+ k := key{svc.p, sp.Name}
+ groups[k] = append(groups[k], listedRow{r, start, sp.Dur})
+ s.rows[EntityID(svc.p.ID, r.TraceID, sp.SpanID)] = r
+ }
+ for _, c := range sp.Children {
+ walk(c, svc, start)
+ }
+ }
+ walk(r.Root, r.e.svc, r.Start)
+ }
+ slowest := func(a, b listedRow) bool { return a.dur > b.dur }
+ fastest := func(a, b listedRow) bool { return a.dur < b.dur }
+ newest := func(a, b listedRow) bool { return a.start.After(b.start) }
+ picked := map[*Request]bool{}
+ var listed []*Request
+ keep := func(rows []listedRow, from, to time.Time, n int, less func(a, b listedRow) bool) {
+ var in []listedRow
+ for _, x := range rows {
+ if !x.start.Before(from) && x.start.Before(to) {
+ in = append(in, x)
+ }
+ }
+ sort.SliceStable(in, func(i, j int) bool { return less(in[i], in[j]) })
+ for _, x := range in[:min(n, len(in))] {
+ if !picked[x.r] {
+ picked[x.r] = true
+ listed = append(listed, x.r)
+ }
+ }
+ }
+ for k, rows := range groups {
+ if !listable[k] {
+ continue
+ }
+ for _, d := range listingWindows {
+ ends := []time.Duration{0}
+ if d == time.Hour {
+ ends = listingDrift
+ }
+ for _, off := range ends {
+ to := s.Now.Add(off)
+ keep(rows, to.Add(-d), to, listedSlowest, slowest)
+ keep(rows, to.Add(-d), to, listedOthers, fastest)
+ keep(rows, to.Add(-d), to, listedOthers, newest)
+ }
+ }
+ earliest := rows[0].start
+ for _, x := range rows {
+ if x.start.Before(earliest) {
+ earliest = x.start
+ }
+ }
+ for day := earliest.UTC().Truncate(24 * time.Hour); day.Before(s.Now); day = day.Add(24 * time.Hour) {
+ keep(rows, day, day.Add(24*time.Hour), listedPerDay, slowest)
+ }
+ }
+ return listed
+}
+
// enforceBudget keeps the detailed span volume under SpanBudget by dropping
// sampled (not forced) detail, oldest first.
func (s *Scenario) enforceBudget(all []*Request) {
diff --git a/cli/test/evals/internal/seed/traces.go b/cli/test/evals/internal/seed/traces.go
index 38aa5ace..c8a93bad 100644
--- a/cli/test/evals/internal/seed/traces.go
+++ b/cli/test/evals/internal/seed/traces.go
@@ -230,7 +230,7 @@ func (s *Scenario) emit(r *Request, batches map[*Project]*spanBatch, logs map[*S
}
}
walk(r.Root, nil, rootSvc, r.Host, r.Start, r.Detail)
- if r.Detail && !r.e.Task && !r.e.SkipAccessLog {
+ if r.Detail && !r.listed && !r.e.Task && !r.e.SkipAccessLog {
// The access log line a request logger writes on completion,
// correlated with the trace like any OTel log bridge does.
end := r.Start.Add(r.Root.Dur)
diff --git a/cli/test/evals/internal/seed/verify.go b/cli/test/evals/internal/seed/verify.go
index b35a81a2..c75682b1 100644
--- a/cli/test/evals/internal/seed/verify.go
+++ b/cli/test/evals/internal/seed/verify.go
@@ -64,6 +64,70 @@ func (s *Scenario) Verify(timeout time.Duration) error {
}
time.Sleep(250 * time.Millisecond)
}
+ if err := s.verifyListings(w.p); err != nil {
+ return fmt.Errorf("verify %s: %w", w.p.Name, err)
+ }
+ }
+ return nil
+}
+
+// verifyListings checks the first page of every endpoint's request listing,
+// in the windows and orders agents ask for: each request on it must have been
+// sent with its span tree (see listedRequests).
+func (s *Scenario) verifyListings(p *Project) error {
+ now := time.Now()
+ orders := []struct {
+ by, dir string
+ n int
+ }{{"duration", "desc", listedSlowest / 2}, {"duration", "asc", listedOthers / 2}, {"recorded_at", "desc", listedOthers / 2}}
+ for _, svc := range s.services {
+ if svc.p != p {
+ continue
+ }
+ for _, e := range svc.endpoints {
+ if len(e.shapes) == 0 {
+ continue
+ }
+ q := env.ProjectQuery(p.ID)
+ q.Set("endpoint", e.Name())
+ for _, win := range []time.Duration{time.Hour, 24 * time.Hour, 7 * 24 * time.Hour} {
+ for _, o := range orders {
+ res, err := s.Sink.Admin(http.MethodPost, "/api/endpoints/endpoint", q, map[string]any{
+ "fromDate": now.Add(-win).Format(time.RFC3339), "toDate": now.Format(time.RFC3339),
+ "orderBy": o.by, "sortDirection": o.dir, "pagination": map[string]int{"page": 1, "pageSize": o.n},
+ })
+ if err != nil {
+ return err
+ }
+ if err := res.Err("list requests"); err != nil {
+ return err
+ }
+ var out struct {
+ Data []struct {
+ ID string `json:"id"`
+ } `json:"data"`
+ }
+ if err := json.Unmarshal(res.Body, &out); err != nil {
+ return err
+ }
+ known := 0
+ for _, d := range out.Data {
+ r, ok := s.rows[d.ID]
+ if !ok {
+ continue
+ }
+ known++
+ if !r.Detail {
+ return fmt.Errorf("%s requests over the last %s by %s %s list %s, which was sent without its span tree", e.Name(), win, o.by, o.dir, d.ID)
+ }
+ }
+ // Ids that never match would make this check pass on anything.
+ if len(out.Data) > 0 && known == 0 {
+ return fmt.Errorf("%s requests over the last %s: none of the %d listed ids is a seeded request", e.Name(), win, len(out.Data))
+ }
+ }
+ }
+ }
}
return nil
}
diff --git a/skills/traceway/SKILL.md b/skills/traceway/SKILL.md
index df481ec3..af945d2d 100644
--- a/skills/traceway/SKILL.md
+++ b/skills/traceway/SKILL.md
@@ -337,7 +337,7 @@ The loop, using the read commands documented in this skill:
1. **Quantify and localize.** `traceway endpoints list --since 24h --order-by p95` (or `impact`). Read the latency shape first: p50 already high means every request pays it (systemic: query, index, algorithm); p99 much greater than p50 means a tail (contention, pool exhaustion, GC, retries, a flaky dependency). Note that `impact`/`impactReason` are offset-adjusted for marked-slow endpoints but the raw p50/p95/p99 are not.
2. **Check the accepted baseline.** Before calling an endpoint slow, run `traceway endpoints slow "<endpoint>"`. A non-zero `offsetMs` means an operator accepted that much latency (with a `reason`), so only latency *beyond* the offset is a real regression; `offsetMs 0` means default thresholds apply. Skip this only for a clearly systemic, multi-endpoint slowdown.
-3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the step where p95 jumps. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".
+3. **Pinpoint when it started: adjust the window down.** Do not investigate the default window blindly; narrow it until it brackets the onset. `traceway endpoints chart --metric-type p95 --interval-minutes <n>` returns latency over time for the top endpoints (`{timestamp, endpoint, value}` in ms): read down one endpoint's buckets for the first one where p95 has jumped. Timestamps are bucket starts, so the onset is inside that bucket, not before it: re-run just that bucket with `--from`/`--to` and an interval of 15 minutes or less before you report the time. Confirm the cause with `traceway metrics query --name <metric> --interval-minutes <n>` (the infra/runtime curve). For a specific route not in the top 5, bisect `traceway endpoints list --search <name> --from <a> --to <b>` over adjacent windows. See `performance.md`, "Pinpointing when the slowness started".
4. **Get a representative slow trace.** `traceway endpoints show <endpointId> --recorded-at <t>` for one request's span waterfall, or `traceway traces show <traceId> --recorded-at <t>` (the `traceId` that `endpoints show` prints) for the cross-service timeline. Take the trace from inside the slow window you just found; find the long pole.
5. **Match the long pole to the checklist.** Is the dominant span a database call, an external call, in-process compute, or a *gap* before work starts (queueing / lock wait / pool exhaustion)? Look it up in `performance.md`.
6. **Separate code from saturation.** Pull infra/runtime metrics over the same window: `traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min` (busy is 1 minus idle; the metric is split by `state`, so never read it unfiltered), then `system.memory.utilization` with `state=used` and the runtime's heap and GC metrics (`jvm.*`, `go.*`, `v8js.*`). A spike that lines up with the latency onset points at saturation, not a code bug.
diff --git a/skills/traceway/performance.md b/skills/traceway/performance.md
index fe89e34b..ccaafb29 100644
--- a/skills/traceway/performance.md
+++ b/skills/traceway/performance.md
@@ -72,14 +72,17 @@ When you report, state the p95 and the allowance together: "p95 is 2.1s but this
Narrowing the window is how you turn "it is slow" into "it got slow at 14:10, right after deploy X". Do not investigate the default window blindly: adjust it down until it brackets the onset, then correlate that moment with deploys and metrics.
-**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Read down one endpoint's buckets for the step where its value jumps: that bucket dates the onset. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first, then re-run the suspect span with a finer interval to sharpen.
+**1. Latency over time: `endpoints chart`.** This is the direct curve. It buckets latency over the window and returns the top 5 endpoints ranked by the chosen metric (plus an "Other" bucket), each as a `{timestamp, endpoint, value}` series in milliseconds. Each `timestamp` is the **start** of its bucket. Read down one endpoint's buckets for the first one whose value has jumped: the onset lies inside that bucket, between its timestamp and its timestamp plus the interval, not in the bucket before it. Use `--metric-type p95` (or `p99`/`p50`/`total_time`) and a coarse `--interval-minutes` over a wide window first to find that bucket. A coarse bucket only brackets the onset: with 60-minute buckets its timestamp can be 59 minutes early, with 6-hour buckets almost 6 hours. Before you report an onset time, re-run over just that bucket with an interval of 15 minutes or less, and report the start of the first elevated fine bucket.
```bash
traceway endpoints chart --metric-type p95 --since 7d --interval-minutes 60 --output json \
| jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'
+# the 60-minute bucket starting 14:00 stepped up: sharpen it in 5-minute buckets
+traceway endpoints chart --metric-type p95 --from 2026-06-25T14:00:00Z --to 2026-06-25T15:00:00Z --interval-minutes 5 --output json \
+ | jq '.series[] | select(.endpoint == "GET /api/checkout") | {t: .timestamp, p95_ms: .value}'
```
Caveat: it always returns the *top 5* endpoints by that metric. If your suspect route is not in the top 5 it is folded into "Other"; fall back to method 3 for it.
-**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the bucket where the value steps up.
+**2. Infra and runtime over time: `metrics query --interval-minutes`.** For the *cause* behind a latency step (saturation, GC, throughput), bucket any metric the same way. The JSON is `{results: [{name, unit, series}]}`, one result per query, and `series` maps each group to its list of `{timestamp, value}` points (a single `__all__` group when ungrouped); read down the points for the first bucket where the value has stepped up. Timestamps are bucket starts here too, so sharpen that bucket the same way before you report a time.
```bash
traceway metrics query --name system.cpu.utilization --tag state=idle --aggregation min --since 7d --interval-minutes 60 --output json \
| jq '.results[0].series | to_entries[] | .value[] | {t: .timestamp, busy: (1 - .value)}'
@@ -94,7 +97,7 @@ traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T00:00:00Z
traceway endpoints list --search "GET /api/checkout" --from 2026-06-25T12:00:00Z --to 2026-06-26T00:00:00Z \
--output json | jq '.data[0] | {p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6), count}'
```
-A few bisection steps localize the jump to an hour or less. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.
+Keep bisecting until the elevated window is 15 minutes or less, then report its start. Watch `count` alongside p95: if the slow half also carries far more traffic, suspect load, not a code change.
**4. Cross-check and correlate.** Once the methods agree on an onset time:
- A latency jump that coincides with a metric inflection (CPU or GC stepping up at the same bucket) points at saturation.