Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

dx(platform): add a shared Sentry perf-compare recipe for performance agents (#15461)

Nicholas Tindleb2d370e6ac79
diff --git a/.claude/skills/perf-compare/SKILL.md b/.claude/skills/perf-compare/SKILL.md
new file mode 100644
index 0000000000..42f41e6036
--- /dev/null
+++ b/.claude/skills/perf-compare/SKILL.md
@@ -0,0 +1,243 @@
+---
+name: perf-compare
+description: "Shared measurement recipe for AutoGPT performance work. Compares a baseline and a candidate in Sentry (two releases, two environments such as preview pr-<n> deploys, or two time windows) per transaction: count, p50/p95/p99, failure rate, and a trace-clustered bootstrap verdict of faster / slower / no detectable change / not enough data, with exit codes agents can branch on. Works for span duration, web vitals (measurements.lcp) or any numeric span field. TRIGGER when an agent needs to show a change made the backend or frontend faster or slower, compare a preview against a control or dev, check a release after merge on dev or prod, or check for regressions."
+user-invocable: true
+argument-hint: "[what to compare, e.g. 'builder release OLD vs NEW on prod' or 'pr-123 vs control pr-124 on /api/library/agents']"
+metadata:
+ author: autogpt-team
+ version: "1.0.0"
+---
+
+# perf-compare: one way to measure performance
+
+Every agent that tunes speed or stability measures with this recipe, so results are comparable across agents and noise is not reported as a win. Sentry is the store: org `significant-gravitas`, API `https://us.sentry.io`, tracing at 100%. `scripts/perf_compare.py` (Python 3.8+, standard library only) runs the queries and the statistics.
+
+```bash
+S=$(git rev-parse --show-toplevel)/.claude/skills/perf-compare/scripts/perf_compare.py
+python3 "$S" --help
+```
+
+## When to use it
+
+- Before claiming a change made something faster or slower, on a preview or after merge.
+- After merge, to check a win holds on dev and then prod (release against release).
+- To check a change did not regress other transactions (run without `--transaction`).
+- For any numeric field: `span.duration` (default), web vitals on builder pageloads (`measurements.lcp`), LLM token counts (`gen_ai.usage.input_tokens`), or custom attributes (`tags[name,number]`).
+
+It is not for debugging one slow request (open the trace) or for error triage.
+
+## Token
+
+The script reads `SENTRY_AUTH_TOKEN` from the environment and nowhere else. It needs a read-only token with exactly `org:read`, `project:read` and `event:read`: either a user auth token (Settings > Account > API > Auth Tokens) or an internal integration with Read on Organization, Project and Issue & Event. Organization auth tokens (`sntrys_...`) carry only `org:ci` and get HTTP 403 on these endpoints. A missing token exits with code 5 and these instructions. `--print-queries` needs no token.
+
+## Where things are in Sentry
+
+| Project | What it holds | Environments | `release` |
+|---|---|---|---|
+| `autogpt-server` | Backend: FastAPI routes (`/api/library/agents`, `/api/chat/sessions/{session_id}/stream`) and internal RPC services (`/update_node_execution_status`) | `app:prod-behave:cloud`, `app:dev-behave:cloud` | git sha once the backend sets it; check first |
+| `builder` | Frontend (Next.js on Vercel): `http.server` routes (`GET /api/proxy/[...path]`), `middleware GET`, `serverAction/*`, browser `pageload` and `navigation` with web vitals | `app:prod-behave:cloud` is prod; `app:preview-behave:cloud` is dev **and** every Vercel preview | git sha |
+| `autogpt-server-previews`, `builder-previews` | `!deploy` previews | `pr-<n>` | git sha |
+
+Check a selector has data before you measure:
+
+```text
+search_events projectSlug=<project> dataset=spans query="is_transaction:true"
+ fields=["environment","release","count()"] sort="-count()" period="7d"
+```
+
+If `release` comes back null for a project, select by time window instead of release. If `autogpt-server-previews` has no `pr-<n>` events, that preview's backend is reporting into `autogpt-server` as `app:dev-behave:cloud`, mixed with dev traffic, and cannot be measured separately.
+
+## Choosing baseline and candidate
+
+Both sides take `KEY=VALUE` pairs: `project=`, `env=`, `release=`, `query=` (extra Sentry filter), and time as `window=START/END` (repeatable), `start=`/`end=`, or `period=24h`. Shared defaults come from `--project`, `--env`, `--query` and `--period`/`--start`/`--end`/`--window`. A side's own `project=`, `env=` or `query=` wins over the shared value (`env=` with nothing after it clears it). Any time key on a side (`window=`, `start=`/`end=` or `period=`) replaces all of the shared time flags for that side. Times are UTC ISO 8601; without a time anywhere, both sides use the last 7 days.
+
+**1. After merge: release against release (dev or prod).** Same project and environment, previous release against new release, in windows of equal length that cover the same hours of the day: the 24 hours before the deploy against the 24 hours after, or 7 days against 7 days for quiet routes. Always filter by release when there is one. Old releases keep sending events for days (open tabs, Vercel skew protection), so a time window alone mixes releases.
+
+```bash
+python3 "$S" --project builder --env app:prod-behave:cloud \
+ --baseline release=OLD_SHA window=2026-10-10T14:00:00Z/2026-10-11T14:00:00Z \
+ --candidate release=NEW_SHA window=2026-10-11T14:00:00Z/2026-10-12T14:00:00Z
+```
+
+**2. On a preview: candidate preview against a control preview.** Deploy the change with `!deploy`, and deploy a control (the dev commit the change branches from, with no changes) the same way. Each gets its own namespace and an empty branch database, so the two are alike. Drive the same load at both, either together in one window or interleaved A B B A (control, candidate, candidate, control; about 10 minutes each) so drift cancels:
+
+```bash
+python3 "$S" --project autogpt-server-previews --transaction '/api/store/agents' \
+ --baseline env=pr-CONTROL window=T0/T1 window=T3/T4 \
+ --candidate env=pr-CANDIDATE window=T1/T2 window=T2/T3
+```
+
+Measuring the candidate preview against `app:dev-behave:cloud` at dev HEAD in the same window is cheaper but weaker: dev has a full database, real traffic and different pods. Use it as a rough read, not as proof.
+
+**3. No release tag: before against after.** Equal windows either side of the deploy. Other deploys inside those windows also change things, so say so in your report.
+
+**4. A/A check before trusting a win.** Compare a selector with itself over two windows, for example the same release yesterday and today. It should say `no detectable change`. If it does not, your windows differ too much (hour of day, weekday, traffic mix), and so will your real comparison.
+
+## How much data
+
+- The script gives no p50 or p95 verdict unless each side has `--min-count` samples (default 200) **from** `--min-traces` distinct traces (default 30). At 200, p95 rests on 10 values above it. p99 is shown but never judged. The verdict cell says which minimum was missed. The failure-rate test works from Sentry's event counts instead, and needs `--min-count` events per side.
+- It samples up to `--sample-cap` values per side per transaction (default 1000, at most 20000), paging 100 at a time. A cap below `--min-count` is refused.
+- For a claim about one transaction, name it with `--transaction` and raise `--sample-cap` to 2000–5000. Naming it also shrinks the multiple-comparison correction (see below).
+- Correlated spans count once: the bootstrap resamples whole traces, and the trace minimum stops a few busy traces (a page polling an endpoint) from passing as a big sample. The table shows samples and distinct traces. The effective sample is closer to the trace count than the span count. In A/A tests on clustered data, resampling spans one at a time called 38–44% of identical pairs a change, against 2–5% when resampling by trace.
+- Drive at least 1000 requests per side per transaction you care about. Wait a few minutes after load stops before measuring, so Sentry has the last events.
+
+## Driving load on a preview
+
+A preview's backend is `https://pr-<N>-server.agpt.co` (REST under `/api`, health at `/health`), and its frontend is `https://autogpt-pr-<N>.vercel.app`.
+
+The k6 tests are in `autogpt_platform/backend/load-tests/`. `K6_ENVIRONMENT=PREVIEW` with `PR_NUMBER=<N>` targets a preview.
+
+- **No token needed:** `tests/marketplace/public-access-test.js` (store agents, search, creators).
+
+ ```bash
+ cd autogpt_platform/backend/load-tests
+ k6 run --env K6_ENVIRONMENT=PREVIEW --env PR_NUMBER=<N> \
+ --env VUS=5 --env DURATION=10m --env RAMP_UP=30s --env RAMP_DOWN=30s \
+ tests/marketplace/public-access-test.js
+ ```
+
+- **Token needed:** `tests/api/core-api-test.js`, `tests/api/graph-execution-test.js`, `tests/marketplace/library-access-test.js`, `tests/comprehensive/platform-journey-test.js`. These read `configs/pre-authenticated-tokens.js`, which is gitignored; copy it from `configs/pre-authenticated-tokens.example.js`.
+ - The backend verifies Better Auth tokens (ES256, audience `authenticated`) against `<frontend>/api/auth/jwks` and rejects HS256. `generate-tokens.js` signs in against dev's Supabase, so its tokens do not work.
+ - Mint a token for a preview's seeded account instead. The `preview-*@previews.agpt.co` accounts use the `PREVIEW_ACCOUNTS_PASSWORD` secret, which is in the team Bitwarden. Tokens last one hour.
+ - Export the password first. The recipe below reads it from the environment and sends it on stdin, so it never appears on a command line other users can see. The cookie jar is a private temporary file.
+
+ ```bash
+ F=https://autogpt-pr-<N>.vercel.app
+ jar=$(mktemp)
+ python3 -c 'import json, os; print(json.dumps({"email": "[email protected]", "password": os.environ["PREVIEW_ACCOUNTS_PASSWORD"]}))' |
+ curl -sS -c "$jar" -H "Origin: $F" -H 'Content-Type: application/json' \
+ --data-binary @- "$F/api/auth/sign-in/email" > /dev/null
+ curl -sS -b "$jar" "$F/api/auth/token" # {"token": "..."}
+ rm -f "$jar"
+ ```
+
+- **Previews start with an empty database.** Store and library lists return nothing, so costs that grow with data (N+1 queries, scans) do not show. Seed data through the API first, or confirm on dev and prod after merge.
+- None of the k6 tests open copilot chats. To measure copilot turns, run chats against the preview yourself, or measure on dev and prod traffic.
+- The repository's evals (`backend/copilot/eval`, `backend/copilot/capabilities/eval`, `poetry run expert-style-eval`, `poetry run brain-dump-eval`) are offline quality and cost harnesses. They send no traffic to a deploy.
+- After merge, real traffic on dev and prod is the load.
+
+## Reading the verdict
+
+The markdown output (default) or `--json` reports two things for each transaction.
+
+- **Sentry aggregates over every matching event:** count, p50, p95, p99 and failure rate. These are exact for the windows.
+- **A verdict for p50 and p95:**
+ - The interval is a bootstrap CI of candidate minus baseline from the raw samples, resampling whole traces.
+ - It is Bonferroni-adjusted so that every interval in the run holds together at `--confidence` (95%). Two transactions means four intervals at 98.75% each. The plain 95% interval is in the JSON as `ci`.
+ - Resamples are raised automatically until each tail of the adjusted interval holds 20 draws, up to 50000. For example, 4000 becomes 8000 for five transactions. At the 50000 cap a run still goes ahead with as few as 10 draws per tail. A run that would leave fewer than 10, because too many transactions are compared, is refused before it starts (exit 2). The JSON reports the number used as `settings.resamples`.
+ - `faster` or `slower` needs the adjusted interval to exclude zero, and **both** the sample change and Sentry's all-events change to be at least `--min-effect` (5%) in that direction. Anything else is `no detectable change`. Below `--min-count` samples or `--min-traces` traces it is `not enough data`.
+ - The transaction takes its worst percentile: `slower` if either is slower, else `faster` if either is faster.
+- **Failure rate:** Fisher's exact test on the failure counts behind Sentry's `failure_rate()`. It is exact because a normal approximation is wrong with a handful of failures: 0 of 1000 against 4 of 1000 passes a z-test, but Fisher gives p = 0.12.
+ - The significance level is Bonferroni-adjusted over the K transactions that have `--min-count` events on both sides. That is one test per transaction, not two.
+ - `higher` needs significance and a rise of at least `--min-failure-delta` (0.1 percentage points). It counts as a regression even when the transaction got faster, because requests that fail fast look like a speedup.
+ - The test treats every event as independent, and `--min-traces` does not apply to it. When failures come from a few traces (one broken page retrying), it overstates significance. Check `sample_traces` and the failing traces in Sentry before acting on a failure-rate result alone.
+
+`no detectable change` means "not distinguishable from noise with this much data". It does not mean "no change". Check the interval widths and trace counts before concluding anything.
+
+**A win counts** only when all of these hold:
+
+- you named the transaction with `--transaction` before measuring;
+- the verdict is `faster`;
+- the failure rate is not `higher`;
+- it holds again on a second, independent window or run.
+
+A run over the top transactions is for spotting regressions and leads, not for claiming wins.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | faster: a compared transaction got faster and nothing regressed |
+| 1 | slower or more failures: at least one compared transaction regressed |
+| 2 | usage error: bad arguments (including a `--region-url` that is not https, `--timeout` ≤ 0, or `--sample-cap` below `--min-count`), unknown project, or Sentry rejected the query (HTTP 400) |
+| 3 | no detectable change |
+| 4 | not enough data: no compared transaction met `--min-count` and `--min-traces` on both sides |
+| 5 | auth: `SENTRY_AUTH_TOKEN` missing, or Sentry rejected it (401/403) |
+| 6 | Sentry API or network error after retries, or any unexpected error; never read as a regression |
+
+With `--json`, the fields to branch on are:
+
+- `overall.verdict`, `overall.exit_code` and `overall.reasons`;
+- per transaction in `transactions[]`:
+ - `verdict`, `note` (why there is not enough data), `regression`;
+ - `p50`/`p95`: `baseline`, `candidate`, `diff`, `rel`, `sentry_rel`, `ci`, `ci_adjusted`, `verdict`;
+ - `failure_rate`: `baseline`, `candidate`, `diff`, `p_value`, `verdict`;
+ - `baseline`/`candidate`: `count`, `p50`, `p95`, `p99`, `failure_rate`, `sample_n`, `sample_traces`;
+- `settings`, holding the values actually used: `sample_cap`, `resamples`, `comparisons`, `failure_comparisons`, `adjusted_confidence`, `min_count`, `min_traces`, `min_effect`, `seed`.
+
+Errors in `--json` mode print `{"error": {"type", "message"}, "exit_code"}`, where `type` is `usage`, `auth`, `api` or `internal`.
+
+## The exact queries
+
+All requests are `GET https://us.sentry.io/api/0/organizations/significant-gravitas/events/` with `dataset=spans` and `project=<numeric id>`. The id comes from `GET /api/0/projects/significant-gravitas/<slug>/`. `start` and `end` (UTC) span the side's windows; the query's timestamp filter pins each window exactly. `F` is `--field`.
+
+1. **Aggregate per transaction**, one request per side:
+ - `field=transaction`, `count()`, `p50(F)`, `p95(F)`, `p99(F)`, `failure_rate()`
+ - `sort=-count()`, `per_page=100`
+2. **Raw sample**, per side per compared transaction:
+ - `field=F`, `trace`, `id`
+ - `sort=id`, `per_page=100`, paged with the `cursor` from the `Link` header (`rel="next"; results="true"`) until `--sample-cap` rows.
+ - Span ids are random, so the first N rows by `id` are a uniform random sample of the windows. Sorting by `-timestamp` instead returns the latest burst, often a few traces.
+
+Both use this query string:
+
+```text
+is_transaction:true environment:"ENV" release:"SHA" [has:F] [(EXTRA QUERY)]
+ [transaction:"T" | (transaction:"A" OR transaction:"B")]
+ timestamp:>=2026-10-11T14:00:00Z timestamp:<2026-10-12T14:00:00Z
+```
+
+Several windows become `(timestamp:>=S1 timestamp:<E1 OR timestamp:>=S2 timestamp:<E2)`. `has:F` appears for any field other than `span.duration`.
+
+`python3 "$S" ... --print-queries` prints the exact REST URL and MCP arguments for an invocation, without a token.
+
+**With only the Sentry MCP**, use `search_events` with `dataset: "spans"`, or `search_traces` without `dataset`:
+
+- Pass `projectSlug`, the same `query`, `fields` and `sort`.
+- Pass a `period` long enough to cover the windows, such as `"50h"`; the timestamp filter does the exact cut. The argument is `period`, not `statsPeriod`, and an unknown key silently falls back to 14 days.
+- Pass `limit` ≤ 100.
+- Confirm the "Executed Search" block in the reply shows your query unchanged.
+- The MCP cannot page. For more than 100 samples, split each window into sub-windows, get `count()` for each, and take the first rows by `id` from each in proportion to its count.
+
+## Field names checked against live data
+
+Checked through the Sentry MCP on `builder` and `autogpt-server`.
+
+- **Filters:**
+ - `is_transaction:true` and `transaction:"…"`.
+ - `environment:"…"`: quote it, the values contain colons.
+ - `release:"<sha>"`.
+ - `span.op:` with `pageload`, `navigation`, `http.server`, `http.server.middleware` or `function.server_action`.
+ - `span.status:[…]` and `!span.status:[…]`.
+ - `has:measurements.lcp` and `has:tags[gen_ai.usage.input_tokens,number]`.
+ - `timestamp:>=…Z timestamp:<…Z` and parenthesised `OR` groups.
+- **Aggregates:**
+ - `count()`.
+ - `p50()`, `p75()`, `p95()` and `p99()`, of `span.duration` (ms), `measurements.lcp`, `measurements.fcp`, `measurements.ttfb`, `measurements.cls`, `gen_ai.usage.input_tokens` or `tags[gen_ai.usage.input_tokens,number]`.
+ - `failure_rate()`: the share of spans whose `span.status` is not `ok`, `cancelled` or `unknown`. It reconciles exactly with `count()` and `!span.status:[ok,cancelled,unknown]`.
+- **Sample fields:** `span.duration`, `trace`, `id`, `timestamp`, `release`. Sorting works by `id`, `-id` and `-timestamp`.
+- **`span.status` values seen:** `ok`, `internal_error`, `resource_exhausted`, `cancelled`, `deadline_exceeded`, `invalid_argument`, `not_found`, `already_exists`.
+- **Rejected:** wildcards on `id` such as `id:0*`; span ids must be full 16-character hex.
+
+## Pitfalls
+
+- **Environment names collide.** Builder's dev deploy and every Vercel preview share `app:preview-behave:cloud`. Tell them apart with `release`.
+- **Proxy routes measure the backend.** `GET /api/proxy/[...path]` mostly measures the backend behind it, not the frontend release, and a few polling pages contribute most of its spans.
+- **Middleware durations are quantized** to about 1 ms. A one-step move in p95 reads as +100%; look at the absolute change.
+- **Long-lived transactions measure session length.** `/ws` and `/api/chat/sessions/{session_id}/stream` last minutes to an hour, not request latency. Compare per-turn or per-node transactions instead.
+- **Weekday and hour of day move latency.** Saturday-to-Sunday against Sunday-to-Monday is not a fair pair for a release comparison. Use full days, the same hours, and the A/A check.
+- **Do not rerun until it says faster.** Repeating runs, or trying many transactions and reporting the best, turns noise into wins. Decide the transaction, windows and settings first, then run once and replicate once.
+
+## Tests
+
+```bash
+python3 -m unittest discover -s .claude/skills/perf-compare/scripts
+```
+
+The tests run offline in a few seconds, against the synthetic Sentry responses in `scripts/fixtures/`, served by a fake transport or by a local `http.server`. They cover:
+
+- query building, time parsing and option validation;
+- cursor pagination, retries, and real-socket errors;
+- the percentile, bootstrap and Fisher maths with fixed seeds;
+- seeded A/A calibration on independent and trace-clustered data;
+- the verdict rules, every exit code, and the missing-token error.
diff --git a/.claude/skills/perf-compare/scripts/fixtures/aggregate_baseline.json b/.claude/skills/perf-compare/scripts/fixtures/aggregate_baseline.json
new file mode 100644
index 0000000000..83c1c86ec8
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/aggregate_baseline.json
@@ -0,0 +1,10 @@
+{
+ "headers": {},
+ "body": {
+ "data": [
+ {"transaction": "GET /api/library/agents", "count()": 1500, "p50(span.duration)": 120.4, "p95(span.duration)": 197.9, "p99(span.duration)": 241.7, "failure_rate()": 0.002},
+ {"transaction": "GET /api/store/agents", "count()": 40, "p50(span.duration)": 310.2, "p95(span.duration)": 640.0, "p99(span.duration)": 702.5, "failure_rate()": 0.0}
+ ],
+ "meta": {"fields": {"transaction": "string", "count()": "integer", "p50(span.duration)": "duration", "p95(span.duration)": "duration", "p99(span.duration)": "duration", "failure_rate()": "percentage"}, "units": {"transaction": null, "count()": null, "p50(span.duration)": "millisecond", "p95(span.duration)": "millisecond", "p99(span.duration)": "millisecond", "failure_rate()": null}, "dataset": "spans"}
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/aggregate_candidate.json b/.claude/skills/perf-compare/scripts/fixtures/aggregate_candidate.json
new file mode 100644
index 0000000000..066a5dafbc
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/aggregate_candidate.json
@@ -0,0 +1,10 @@
+{
+ "headers": {},
+ "body": {
+ "data": [
+ {"transaction": "GET /api/library/agents", "count()": 1450, "p50(span.duration)": 150.6, "p95(span.duration)": 246.8, "p99(span.duration)": 301.9, "failure_rate()": 0.029655172413793104},
+ {"transaction": "GET /api/store/agents", "count()": 35, "p50(span.duration)": 305.0, "p95(span.duration)": 655.1, "p99(span.duration)": 690.3, "failure_rate()": 0.0}
+ ],
+ "meta": {"fields": {"transaction": "string", "count()": "integer", "p50(span.duration)": "duration", "p95(span.duration)": "duration", "p99(span.duration)": "duration", "failure_rate()": "percentage"}, "units": {"transaction": null, "count()": null, "p50(span.duration)": "millisecond", "p95(span.duration)": "millisecond", "p99(span.duration)": "millisecond", "failure_rate()": null}, "dataset": "spans"}
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/error_400.json b/.claude/skills/perf-compare/scripts/fixtures/error_400.json
new file mode 100644
index 0000000000..09f6ac1605
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/error_400.json
@@ -0,0 +1,6 @@
+{
+ "headers": {},
+ "body": {
+ "detail": "Invalid query. Field p50(nope) is not supported."
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/project_builder.json b/.claude/skills/perf-compare/scripts/fixtures/project_builder.json
new file mode 100644
index 0000000000..09274d2862
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/project_builder.json
@@ -0,0 +1,8 @@
+{
+ "headers": {},
+ "body": {
+ "id": "1000001",
+ "slug": "builder",
+ "name": "builder"
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/samples_baseline_page1.json b/.claude/skills/perf-compare/scripts/fixtures/samples_baseline_page1.json
new file mode 100644
index 0000000000..4f43a7bac6
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/samples_baseline_page1.json
@@ -0,0 +1,108 @@
+{
+ "headers": {"link": "<https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:0:1>; rel=\"previous\"; results=\"false\"; cursor=\"0:0:1\", <https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:100:0>; rel=\"next\"; results=\"true\"; cursor=\"0:100:0\""},
+ "body": {
+ "data": [
+ {"span.duration": 117.071, "trace": "6d4f6185578715bbd26944ff770e4b94", "id": "00c49b5539ac5ba7", "project.name": "builder"},
+ {"span.duration": 136.218, "trace": "5eb4c269b873ac7a00edb9f7796bfbc2", "id": "00caf6d6f1f6af08", "project.name": "builder"},
+ {"span.duration": 105.563, "trace": "bbda5f05cb39676b9852e160d8020527", "id": "0575870032264fa2", "project.name": "builder"},
+ {"span.duration": 94.111, "trace": "0335d8a1483bba4ee1a9a3a1bcbbe842", "id": "074c45cf807a9f1b", "project.name": "builder"},
+ {"span.duration": 184.458, "trace": "1ce75fc538e29e602225b0dde9bb53f3", "id": "0b7c056ebc875e5b", "project.name": "builder"},
+ {"span.duration": 181.194, "trace": "80f7bec391a97c0de4f91904a170587c", "id": "0c2aa24c4913e4f3", "project.name": "builder"},
+ {"span.duration": 119.537, "trace": "5398003680e7e3b35183ef8333c4774e", "id": "0cd1c1bac7adac1a", "project.name": "builder"},
+ {"span.duration": 116.422, "trace": "c56f24b1c71b106e934d263b5ba0837b", "id": "0f328549c488e00a", "project.name": "builder"},
+ {"span.duration": 233.748, "trace": "e481a65c2011bef2c328a72c5e5b7751", "id": "1018f134a069e3fa", "project.name": "builder"},
+ {"span.duration": 119.456, "trace": "dfe0d5a0cf318656b3e6f0bade65c3b1", "id": "102ddb8379c7ce65", "project.name": "builder"},
+ {"span.duration": 140.917, "trace": "058765a6ca7cff00d796c25410335b40", "id": "1212b62c37663112", "project.name": "builder"},
+ {"span.duration": 122.265, "trace": "013cb790fef33ef2c3ff57de13628bef", "id": "127f6c31d175a632", "project.name": "builder"},
+ {"span.duration": 124.053, "trace": "ff1125cf5ec72ba694165beaecba0afa", "id": "1448c828b4136d3b", "project.name": "builder"},
+ {"span.duration": 59.878, "trace": "b69c64773031f6725480dc3932677172", "id": "1659a2e50add1274", "project.name": "builder"},
+ {"span.duration": 134.113, "trace": "8acd947a1b5a41eafe6ab7233a007b22", "id": "17bd5c2d6a9a5f04", "project.name": "builder"},
+ {"span.duration": 97.441, "trace": "09c30065f846d34530325fed10a47b85", "id": "1832b6ec017c1e17", "project.name": "builder"},
+ {"span.duration": 105.768, "trace": "cb3593871c15d694c1957f8db0391173", "id": "1a6b2dc782bdeae1", "project.name": "builder"},
+ {"span.duration": 110.417, "trace": "dd51983ebf7c99c18fa6eb9eb2b67d8b", "id": "1abd1d97aaf35f3b", "project.name": "builder"},
+ {"span.duration": 139.924, "trace": "6a2c8773e130f7eb19731662b5e803b6", "id": "1ba4168160adb592", "project.name": "builder"},
+ {"span.duration": 139.777, "trace": "ca8b6f3a6a9421cc1c93016f1c4261e5", "id": "1d30b49895d1a0d1", "project.name": "builder"},
+ {"span.duration": 196.824, "trace": "a6aef1515e22e00fd2d741d7a9fdc10a", "id": "1d67a0031dffb3ca", "project.name": "builder"},
+ {"span.duration": 100.632, "trace": "de82eb31f96288b6d8eacf314914bc78", "id": "1ef02216ef29a543", "project.name": "builder"},
+ {"span.duration": 107.684, "trace": "c35526f7eaed46725a2a7b860dcd6c8a", "id": "1f8b46287cced904", "project.name": "builder"},
+ {"span.duration": 101.572, "trace": "a8902073fec8df4f50947aaeb26c57d2", "id": "1fa5d328263dfe57", "project.name": "builder"},
+ {"span.duration": 154.615, "trace": "a5204642bbdb4a78f19e8b8480f3b47c", "id": "20431658b4550b7e", "project.name": "builder"},
+ {"span.duration": 104.151, "trace": "99724caf4941d4072014b3ce107f80e2", "id": "22f828767efc2f91", "project.name": "builder"},
+ {"span.duration": 114.404, "trace": "647c1ac49726e45dac31b3629fb0f26f", "id": "264f879130b64915", "project.name": "builder"},
+ {"span.duration": 130.058, "trace": "0335d8a1483bba4ee1a9a3a1bcbbe842", "id": "26d1195d24734e07", "project.name": "builder"},
+ {"span.duration": 161.048, "trace": "dfe0d5a0cf318656b3e6f0bade65c3b1", "id": "26f74bde94fb78c8", "project.name": "builder"},
+ {"span.duration": 197.697, "trace": "6d4f6185578715bbd26944ff770e4b94", "id": "2a83fdf6a0b29872", "project.name": "builder"},
+ {"span.duration": 123.107, "trace": "ccd71ba82f4dee6a63c59620e6686900", "id": "2b6d08b5ab9315bd", "project.name": "builder"},
+ {"span.duration": 126.404, "trace": "a798b1310582d67fae1983cb936a9882", "id": "2cb5da875953507b", "project.name": "builder"},
+ {"span.duration": 123.39, "trace": "51b6a36e33a4180fd14add2d7bc4d8b9", "id": "2e0a3cfe53b17041", "project.name": "builder"},
+ {"span.duration": 206.187, "trace": "e3a34bff2aaf438c6b8068dc5d44036c", "id": "2e162aaef6076bc3", "project.name": "builder"},
+ {"span.duration": 211.455, "trace": "0ac414f5c500bd6cdaf5ac6860aa8a5f", "id": "2f14d2d9d0243c83", "project.name": "builder"},
+ {"span.duration": 100.852, "trace": "a6aef1515e22e00fd2d741d7a9fdc10a", "id": "2fc3f3c3fd03f91d", "project.name": "builder"},
+ {"span.duration": 105.27, "trace": "dc843565f6ef306e13d6975bb3f25948", "id": "31167628828f5809", "project.name": "builder"},
+ {"span.duration": 151.984, "trace": "73638acc02d384db001dc5bb4bb84554", "id": "3593fde017d4707b", "project.name": "builder"},
+ {"span.duration": 78.347, "trace": "ac5a0a6e39ebbf65b669972d06263739", "id": "36081d28a0db5065", "project.name": "builder"},
+ {"span.duration": 160.08, "trace": "08d35e59c7a80268422c922202b243f8", "id": "389cd5e3eaa60c73", "project.name": "builder"},
+ {"span.duration": 144.479, "trace": "f856469602d1ba9f20df4875b15b0be2", "id": "3b7ac193fe040727", "project.name": "builder"},
+ {"span.duration": 81.629, "trace": "0bc65bfc54d5f667b388b3f9c6ad0984", "id": "4593dedd634d54a7", "project.name": "builder"},
+ {"span.duration": 146.71, "trace": "c938e019bb8723d39553ccaccfab54d9", "id": "46a2d207dc684477", "project.name": "builder"},
+ {"span.duration": 131.937, "trace": "e3a34bff2aaf438c6b8068dc5d44036c", "id": "46eee21f5c7ff43f", "project.name": "builder"},
+ {"span.duration": 64.725, "trace": "c35526f7eaed46725a2a7b860dcd6c8a", "id": "471948d33296c870", "project.name": "builder"},
+ {"span.duration": 99.225, "trace": "4dba41ecccc3fc1626e53a13043b026c", "id": "48bbf33feff9243a", "project.name": "builder"},
+ {"span.duration": 138.301, "trace": "1fe4f7f505aef9ebdd25b001a3ff416d", "id": "4a3baf69dad8199b", "project.name": "builder"},
+ {"span.duration": 178.94, "trace": "d4d14aa605882ac89cd1997cd896416b", "id": "4ba6e1a02da187e9", "project.name": "builder"},
+ {"span.duration": 112.588, "trace": "c0d7ce0ec037c8703ed27e961b130f4c", "id": "4e8bc562ad69a1b3", "project.name": "builder"},
+ {"span.duration": 140.183, "trace": "ad84173581569969e58b081006f7e3df", "id": "50ac2f86702824c1", "project.name": "builder"},
+ {"span.duration": 82.168, "trace": "10c7ac1ff65255845a94f3489967ea4b", "id": "513214825007e2e7", "project.name": "builder"},
+ {"span.duration": 224.738, "trace": "35fd16ccd6b9ccc6c4ae12725b8efa9b", "id": "555246fa3447a992", "project.name": "builder"},
+ {"span.duration": 107.752, "trace": "ef750110c57513064d6d59291f0cde2e", "id": "5738713a818d8962", "project.name": "builder"},
+ {"span.duration": 155.024, "trace": "f6bce6a0302cb17cdc70808d77b6ad89", "id": "5f84992a0f75ae61", "project.name": "builder"},
+ {"span.duration": 164.607, "trace": "b7f1adbc60926f6967e7893f57fd14c1", "id": "604d115cea325a65", "project.name": "builder"},
+ {"span.duration": 143.772, "trace": "624a8940f1f836f99eee3692f09e2e8c", "id": "62248b483b7ffc05", "project.name": "builder"},
+ {"span.duration": 101.367, "trace": "9ea177e8fec375b3be41d62ef430dd73", "id": "6594888e498e656e", "project.name": "builder"},
+ {"span.duration": 117.251, "trace": "e01fbcd9504bca7a5c59340afef8b0ba", "id": "661449771d833424", "project.name": "builder"},
+ {"span.duration": 121.846, "trace": "6aa04ab22031598926e8019792f4cece", "id": "6788749c1736ebeb", "project.name": "builder"},
+ {"span.duration": 79.396, "trace": "1ce75fc538e29e602225b0dde9bb53f3", "id": "67cba892b3ba4a3a", "project.name": "builder"},
+ {"span.duration": 91.13, "trace": "9ea177e8fec375b3be41d62ef430dd73", "id": "6a2e5a2a038d5a1e", "project.name": "builder"},
+ {"span.duration": 163.838, "trace": "a4c123b1612dd272d1371c17149d4395", "id": "6b3216fdaeeb9757", "project.name": "builder"},
+ {"span.duration": 114.568, "trace": "2770c7173601e1c771d814e0f33545a3", "id": "6d32b32732b89994", "project.name": "builder"},
+ {"span.duration": 124.594, "trace": "4e69f569ca039b645d93b4398d8e9a80", "id": "6d8a0990846b3ba3", "project.name": "builder"},
+ {"span.duration": 135.399, "trace": "2c49d49aee9f4580d08fb6d0ed62279c", "id": "6dbedbc37293edbd", "project.name": "builder"},
+ {"span.duration": 122.797, "trace": "7b7d3703a3ef076b1acdc79d2edf85dd", "id": "6e732bd008f56f49", "project.name": "builder"},
+ {"span.duration": 165.025, "trace": "d4ae56ad7675dbd9956e246a395dfeff", "id": "6f4572bc2c3bdabc", "project.name": "builder"},
+ {"span.duration": 92.923, "trace": "c3bfc5e740e61572b4e3c02eaa7f3b4a", "id": "715e4e48dd74089a", "project.name": "builder"},
+ {"span.duration": 142.362, "trace": "864ec8b45d48730d21e9e233c90cb4f2", "id": "7226249de87a13d9", "project.name": "builder"},
+ {"span.duration": 157.886, "trace": "f506b40928b5b7a767c76fb008f86beb", "id": "737f6a6f0fb23c6f", "project.name": "builder"},
+ {"span.duration": 65.006, "trace": "a8902073fec8df4f50947aaeb26c57d2", "id": "739988b886e75774", "project.name": "builder"},
+ {"span.duration": 145.543, "trace": "7a151dd64b338ec80cc5c0b3aa416607", "id": "73ac7d7a7c198ffe", "project.name": "builder"},
+ {"span.duration": 120.158, "trace": "8acd947a1b5a41eafe6ab7233a007b22", "id": "766bb31ed04d259b", "project.name": "builder"},
+ {"span.duration": 113.637, "trace": "7a151dd64b338ec80cc5c0b3aa416607", "id": "77fa31a2e376e9db", "project.name": "builder"},
+ {"span.duration": 135.879, "trace": "6d4f6185578715bbd26944ff770e4b94", "id": "7a3d54ec6390bf61", "project.name": "builder"},
+ {"span.duration": 140.279, "trace": "80f7bec391a97c0de4f91904a170587c", "id": "7a437ecb4e59b08f", "project.name": "builder"},
+ {"span.duration": 133.313, "trace": "683536c4499d863386ce10cd79e048c0", "id": "7dd7753eda83d7c5", "project.name": "builder"},
+ {"span.duration": 148.532, "trace": "d82ef9b1ad85ffa47837771674fbfb16", "id": "7df61a128b3f4534", "project.name": "builder"},
+ {"span.duration": 184.847, "trace": "d4c21a9dbf49a067e24bdb7ec8375637", "id": "8368f7e732d2e433", "project.name": "builder"},
+ {"span.duration": 85.837, "trace": "d8e27e07c36d29ba78a71cdd24221683", "id": "863fe92f442fd405", "project.name": "builder"},
+ {"span.duration": 76.798, "trace": "7f532f3ab3cc2d0b698d5c7e41ba4ea5", "id": "874ae7689447ab57", "project.name": "builder"},
+ {"span.duration": 81.021, "trace": "c5503b11606e4644e0d4887d6e120a57", "id": "8757563e68d1f0e2", "project.name": "builder"},
+ {"span.duration": 119.566, "trace": "5f1abef543b5dfce8a981a049d7ccc7e", "id": "88d519448fb2fc67", "project.name": "builder"},
+ {"span.duration": 118.445, "trace": "c0d7ce0ec037c8703ed27e961b130f4c", "id": "88deeeea35374646", "project.name": "builder"},
+ {"span.duration": 107.5, "trace": "6d4f6185578715bbd26944ff770e4b94", "id": "89639e35aeeb9521", "project.name": "builder"},
+ {"span.duration": 150.196, "trace": "7429ab7bca1aafb77b4460ecec952499", "id": "8a26259bebd2fa58", "project.name": "builder"},
+ {"span.duration": 134.544, "trace": "496af2fac6b0ff663e73a436ab2d319c", "id": "8a906f526bd62214", "project.name": "builder"},
+ {"span.duration": 93.573, "trace": "59ec646f3a708f4aa5a6d107b0811a7a", "id": "8b9bbcc9370d7154", "project.name": "builder"},
+ {"span.duration": 112.75, "trace": "4742684ee75bb6cc69f67e48eb7c6432", "id": "8c0490c257a632b9", "project.name": "builder"},
+ {"span.duration": 113.181, "trace": "61fcd25491215310a53e5356b6b3dacd", "id": "8e7f05554b1e1e0e", "project.name": "builder"},
+ {"span.duration": 100.177, "trace": "6ece6615d3142f505f7965463e3621d7", "id": "8ed41415e97a498a", "project.name": "builder"},
+ {"span.duration": 99.554, "trace": "31af3176813e02ea68ef786e4d3cea27", "id": "934b484e73cf575d", "project.name": "builder"},
+ {"span.duration": 154.139, "trace": "ef7ab5392e335ce1113d4db2b5b52a0f", "id": "94833734f83ae751", "project.name": "builder"},
+ {"span.duration": 117.021, "trace": "4742684ee75bb6cc69f67e48eb7c6432", "id": "94c9bce4850bbd0e", "project.name": "builder"},
+ {"span.duration": 73.071, "trace": "8f3aef3416f9386bd8773c9d51940ea4", "id": "95bd1d6854575622", "project.name": "builder"},
+ {"span.duration": 65.679, "trace": "ad84173581569969e58b081006f7e3df", "id": "967a64cb14028d51", "project.name": "builder"},
+ {"span.duration": 202.587, "trace": "e3ef5404bf7bac806081598a878e2f26", "id": "9b1ecb19dd8b7c46", "project.name": "builder"},
+ {"span.duration": 100.261, "trace": "b7d0b352ad6074dce1118813830d7193", "id": "9b53182e4e349d98", "project.name": "builder"},
+ {"span.duration": 106.115, "trace": "6098b2cc2bd818319478da6bd0c621de", "id": "9f145fda9988c79f", "project.name": "builder"},
+ {"span.duration": 151.865, "trace": "a4c123b1612dd272d1371c17149d4395", "id": "9fae923d5a4fd12a", "project.name": "builder"}
+ ],
+ "meta": {"fields": {"span.duration": "duration", "trace": "string", "id": "string", "project.name": "string"}, "units": {"span.duration": "millisecond"}, "dataset": "spans"}
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/samples_baseline_page2.json b/.claude/skills/perf-compare/scripts/fixtures/samples_baseline_page2.json
new file mode 100644
index 0000000000..d115108060
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/samples_baseline_page2.json
@@ -0,0 +1,68 @@
+{
+ "headers": {"link": "<https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:0:1>; rel=\"previous\"; results=\"true\"; cursor=\"0:0:1\", <https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:200:0>; rel=\"next\"; results=\"false\"; cursor=\"0:200:0\""},
+ "body": {
+ "data": [
+ {"span.duration": 131.612, "trace": "64c090cea7a24129199532290b5cd33e", "id": "9fec3d7c6afcc831", "project.name": "builder"},
+ {"span.duration": 131.339, "trace": "cad6ba2b0aee0ca923732881584d8c4f", "id": "a2815d2802827283", "project.name": "builder"},
+ {"span.duration": 82.513, "trace": "0587061ce6936714122a40680a06aa0f", "id": "a51d12afc8e00aa1", "project.name": "builder"},
+ {"span.duration": 156.612, "trace": "84fdb0dd13f1c4ff54c4d88273eb3564", "id": "a731d512ff6d964e", "project.name": "builder"},
+ {"span.duration": 107.926, "trace": "c35526f7eaed46725a2a7b860dcd6c8a", "id": "a7f770d9106fd287", "project.name": "builder"},
+ {"span.duration": 93.54, "trace": "91c94c8286793b2b023a60e4e81e11e3", "id": "aa766907508db282", "project.name": "builder"},
+ {"span.duration": 75.13, "trace": "c402364f9572b85a8e48f687ab165c58", "id": "ac5831be38cb8cb4", "project.name": "builder"},
+ {"span.duration": 145.523, "trace": "0ffac7cb2c8a2788fbf742b65b754e51", "id": "acbd3d48c3bb9e28", "project.name": "builder"},
+ {"span.duration": 104.348, "trace": "9ddceb1be0273dbc46dfcea25bab2953", "id": "ad5966d513b1d009", "project.name": "builder"},
+ {"span.duration": 107.838, "trace": "7da8cafe1f6151b9267f9ed212562c49", "id": "ad7312fa1c8be785", "project.name": "builder"},
+ {"span.duration": 161.405, "trace": "a6022136ced620104d159e8489b0ac35", "id": "adab23e5617d2669", "project.name": "builder"},
+ {"span.duration": 162.814, "trace": "123a7178b5bd85ee5042d74833c27041", "id": "b29ae696fa4bb784", "project.name": "builder"},
+ {"span.duration": 134.335, "trace": "13dce20c4fd32f640d0032634f087e51", "id": "b429fe8110102c99", "project.name": "builder"},
+ {"span.duration": 102.589, "trace": "ba80622598514f31c827129084bb54b8", "id": "bb53759c0767cb7f", "project.name": "builder"},
+ {"span.duration": 143.184, "trace": "fb6d28c587db821f6a0efa5ea7d26dc4", "id": "bcfb4768314cd2fe", "project.name": "builder"},
+ {"span.duration": 127.499, "trace": "29e7c6be9ff907a76cc0b57aaf896910", "id": "be1ceb374dab4683", "project.name": "builder"},
+ {"span.duration": 122.063, "trace": "1ce680ce2b27c8af6666259bbc471fb3", "id": "be24a0b80316f688", "project.name": "builder"},
+ {"span.duration": 156.041, "trace": "9ea177e8fec375b3be41d62ef430dd73", "id": "be645a80d5282639", "project.name": "builder"},
+ {"span.duration": 83.332, "trace": "c56f24b1c71b106e934d263b5ba0837b", "id": "bf1b3ba3178b6e0e", "project.name": "builder"},
+ {"span.duration": 56.65, "trace": "2770c7173601e1c771d814e0f33545a3", "id": "c0202219ec0605e6", "project.name": "builder"},
+ {"span.duration": 102.942, "trace": "e19cbae530282bd36cb9d21f6be6abf0", "id": "c1c1e21862ab8a18", "project.name": "builder"},
+ {"span.duration": 89.839, "trace": "155a0e9d8f27c7d9cf07255bc509cb3a", "id": "c23db7c6e9b7d180", "project.name": "builder"},
+ {"span.duration": 159.27, "trace": "1c5cb347611a3ce9d97dcbee500fe7ee", "id": "c324bdb2e1142a21", "project.name": "builder"},
+ {"span.duration": 201.55, "trace": "f4de51b20a401549935d49a54e5ec549", "id": "c4a7cb2ae33834aa", "project.name": "builder"},
+ {"span.duration": 108.759, "trace": "5f08b79affd2b49c12a4b0062983475e", "id": "c5296f62e338d74f", "project.name": "builder"},
+ {"span.duration": 131.105, "trace": "27ad909f03fdd9e4a62bce19a285ed73", "id": "c5c8a4b57bc9fa65", "project.name": "builder"},
+ {"span.duration": 72.811, "trace": "54b4667a20f1fa2261bd2b5ff4891e5d", "id": "c9328776e7f1ccac", "project.name": "builder"},
+ {"span.duration": 100.35, "trace": "ad84173581569969e58b081006f7e3df", "id": "c9791e558e08baa7", "project.name": "builder"},
+ {"span.duration": 127.544, "trace": "9ea177e8fec375b3be41d62ef430dd73", "id": "c9cfc4b1d85a6c84", "project.name": "builder"},
+ {"span.duration": 123.848, "trace": "9f34369aad80b891baf90d0d3bf16295", "id": "d06910bf3f5fb859", "project.name": "builder"},
+ {"span.duration": 94.184, "trace": "6b1e5d490340494b35ec2daca1760147", "id": "d301a233f4d05743", "project.name": "builder"},
+ {"span.duration": 111.928, "trace": "0b57048efc48738d444a157d52ed8748", "id": "d31d3092954d2c93", "project.name": "builder"},
+ {"span.duration": 95.862, "trace": "f2b672850882161db80a1e9ad8cdadc4", "id": "d4078c763211caea", "project.name": "builder"},
+ {"span.duration": 110.806, "trace": "da2cec255404e4fb440034d6608697a8", "id": "d41bed440e50454f", "project.name": "builder"},
+ {"span.duration": 165.671, "trace": "2fcdaf171e7156282a2a2d92e7459da3", "id": "d51f35191a136c57", "project.name": "builder"},
+ {"span.duration": 110.713, "trace": "7eafdbea13e284142e192ad24c311943", "id": "d575cdab37e328cf", "project.name": "builder"},
+ {"span.duration": 66.376, "trace": "bfe228f219e9cb0eb53f16947ccf25ec", "id": "d8dbc74254770f58", "project.name": "builder"},
+ {"span.duration": 82.987, "trace": "1de30ca67dbeb4c29d9936dae96f9c23", "id": "d8f8c375d60fcac3", "project.name": "builder"},
+ {"span.duration": 109.495, "trace": "b87113c16fdf5924754ec21ef66b01d4", "id": "da2e055c90eb6f2a", "project.name": "builder"},
+ {"span.duration": 131.78, "trace": "f84d30d3fc4d83cee9b9bcca0fce9594", "id": "dc72aa7a6d0018f9", "project.name": "builder"},
+ {"span.duration": 128.587, "trace": "070796e656984517ea9ca91a291a7457", "id": "e06a3bf9232cdf28", "project.name": "builder"},
+ {"span.duration": 181.181, "trace": "00537e8b3c48d2ae89b9c1ffb013ce94", "id": "e1af408461c58790", "project.name": "builder"},
+ {"span.duration": 157.529, "trace": "ee42ea368b23ff8500f17f4b4ca1b570", "id": "e2e619e469a62c05", "project.name": "builder"},
+ {"span.duration": 207.131, "trace": "b26a22eccdf03eeddf52ecf4076c19ac", "id": "e327203f26e16af1", "project.name": "builder"},
+ {"span.duration": 71.594, "trace": "ff2d3c425c8d99d19bdd0b6cc60d5d32", "id": "e54014c2b54b9552", "project.name": "builder"},
+ {"span.duration": 178.613, "trace": "49701835ea45ac4e8854b47036909a39", "id": "e5e32bc556202c24", "project.name": "builder"},
+ {"span.duration": 122.06, "trace": "33d268f95d09ea9823fa7b3a99b7d87d", "id": "e86440285b86ce53", "project.name": "builder"},
+ {"span.duration": 108.738, "trace": "8acd947a1b5a41eafe6ab7233a007b22", "id": "ec9fc9fab9b32fed", "project.name": "builder"},
+ {"span.duration": 140.536, "trace": "a9df8a1285822184aaf4614dc90792f3", "id": "ee72fd40663e78da", "project.name": "builder"},
+ {"span.duration": 95.868, "trace": "dd51983ebf7c99c18fa6eb9eb2b67d8b", "id": "f14ade9d4a455b81", "project.name": "builder"},
+ {"span.duration": 106.951, "trace": "e01fbcd9504bca7a5c59340afef8b0ba", "id": "f3a8c80bc2b08a9f", "project.name": "builder"},
+ {"span.duration": 101.057, "trace": "2e751989a01749ddb14f71010b93b7d9", "id": "f54074e3248c801b", "project.name": "builder"},
+ {"span.duration": 169.482, "trace": "ff2d3c425c8d99d19bdd0b6cc60d5d32", "id": "f6941fa1c257c6f5", "project.name": "builder"},
+ {"span.duration": 101.23, "trace": "d2cfb8a5f1b461595919cb589f6aec38", "id": "f836ed5a148fd28c", "project.name": "builder"},
+ {"span.duration": 54.453, "trace": "a6022136ced620104d159e8489b0ac35", "id": "fa870d0a7ba07a25", "project.name": "builder"},
+ {"span.duration": 147.624, "trace": "ee42ea368b23ff8500f17f4b4ca1b570", "id": "fbf666f69e87a1d5", "project.name": "builder"},
+ {"span.duration": 87.635, "trace": "496af2fac6b0ff663e73a436ab2d319c", "id": "fe880d8184e66740", "project.name": "builder"},
+ {"span.duration": 241.882, "trace": "624a8940f1f836f99eee3692f09e2e8c", "id": "fec94dbca3a0aac3", "project.name": "builder"},
+ {"span.duration": 102.203, "trace": "c35526f7eaed46725a2a7b860dcd6c8a", "id": "ff02cee737443e21", "project.name": "builder"},
+ {"span.duration": 200.17, "trace": "8a557f78817592ce63dfa1c7ef6853ac", "id": "fff8b3fa5a3bc34f", "project.name": "builder"}
+ ],
+ "meta": {"fields": {"span.duration": "duration", "trace": "string", "id": "string", "project.name": "string"}, "units": {"span.duration": "millisecond"}, "dataset": "spans"}
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/samples_candidate_page1.json b/.claude/skills/perf-compare/scripts/fixtures/samples_candidate_page1.json
new file mode 100644
index 0000000000..4b08290640
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/samples_candidate_page1.json
@@ -0,0 +1,108 @@
+{
+ "headers": {"link": "<https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:0:1>; rel=\"previous\"; results=\"false\"; cursor=\"0:0:1\", <https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:100:0>; rel=\"next\"; results=\"true\"; cursor=\"0:100:0\""},
+ "body": {
+ "data": [
+ {"span.duration": 175.253, "trace": "c3b2930a4e2d7e69dd013e5a87399488", "id": "004480408e0b5fc3", "project.name": "builder"},
+ {"span.duration": 119.659, "trace": "7b2426b2e2266a9f42b887fa598ba642", "id": "01ad4c1c01e797f4", "project.name": "builder"},
+ {"span.duration": 109.803, "trace": "36834045a914d69d357f57ab5b37b0f2", "id": "05e879f5914722bd", "project.name": "builder"},
+ {"span.duration": 145.003, "trace": "d321ccb4a47b722e305a2bd178da950c", "id": "06f156e5d192e0be", "project.name": "builder"},
+ {"span.duration": 287.563, "trace": "0c3baa59159387d83bca57a63d874783", "id": "08bcdbac4ad2e30b", "project.name": "builder"},
+ {"span.duration": 145.583, "trace": "5b53145efa2834725c4711815bbd156d", "id": "0aaf4822ec8ecb04", "project.name": "builder"},
+ {"span.duration": 138.45, "trace": "795984c40a98c5f9dcfce299cc270628", "id": "0bddcc4b5b32109f", "project.name": "builder"},
+ {"span.duration": 192.475, "trace": "e42719c5bd5b0b154038331d235e8c2a", "id": "1367da3c8af5b6b8", "project.name": "builder"},
+ {"span.duration": 127.377, "trace": "08f53e05c253050a4eea4d507bc1534e", "id": "1656a8dfd0f1d36b", "project.name": "builder"},
+ {"span.duration": 121.27, "trace": "7bc4612476c0efecf6c2f708dfc3832c", "id": "1a72f6421f64ee9b", "project.name": "builder"},
+ {"span.duration": 115.578, "trace": "60414b8e8b94bee7a2809a3800ec6004", "id": "1aa7c6fe9cdf3cad", "project.name": "builder"},
+ {"span.duration": 184.189, "trace": "f7b94c5208bc3034f4d9b615b2d43e26", "id": "1b33ad32b29984a8", "project.name": "builder"},
+ {"span.duration": 116.228, "trace": "5d34ebb833fb50beb33a0a67ea64ea27", "id": "1bc648d0bf98cff2", "project.name": "builder"},
+ {"span.duration": 96.236, "trace": "c969e4ecff93edcc0c7e3d54ad0fd2af", "id": "1d97651cf7e25ef3", "project.name": "builder"},
+ {"span.duration": 134.339, "trace": "21be072ad509f2aaea357bbc06e03a09", "id": "201dc067722b1fd2", "project.name": "builder"},
+ {"span.duration": 101.823, "trace": "97cbadca83216255004764cc2b192d39", "id": "206172a7b7694a8e", "project.name": "builder"},
+ {"span.duration": 184.796, "trace": "5cd238cf74b50980ffc38e45c0847492", "id": "20f9fc5ea6dfbf04", "project.name": "builder"},
+ {"span.duration": 127.672, "trace": "cee85df9038b8c559082fdc28124fe5d", "id": "2190f560235e7101", "project.name": "builder"},
+ {"span.duration": 216.703, "trace": "7c4d0c8b923a4ba79e2d1e7463346faf", "id": "228212bbfdfad645", "project.name": "builder"},
+ {"span.duration": 144.569, "trace": "3c25d1406823ee924b2a124b271a79ba", "id": "23bd558c4d237295", "project.name": "builder"},
+ {"span.duration": 121.95, "trace": "8876a64fd7ec9e5afbd110fd5b82bcf6", "id": "266e419984534333", "project.name": "builder"},
+ {"span.duration": 207.121, "trace": "5b5ed93d9ef1fcc1e01f0e2a284cee82", "id": "278aadfff0c87c06", "project.name": "builder"},
+ {"span.duration": 104.407, "trace": "9e2d0738d4bd90a4eac8e63feb2a778c", "id": "28615ba5e0aa3aac", "project.name": "builder"},
+ {"span.duration": 140.276, "trace": "69402c5674bea20e2c193d0eb9e78337", "id": "286ddd43274470f8", "project.name": "builder"},
+ {"span.duration": 202.253, "trace": "97cbadca83216255004764cc2b192d39", "id": "28d584c654035a0a", "project.name": "builder"},
+ {"span.duration": 134.841, "trace": "20877bd605c3b05c03e6f2e364af8656", "id": "299e95c3bc52015b", "project.name": "builder"},
+ {"span.duration": 297.38, "trace": "afc53a7303d3c0ab347945b00482c25c", "id": "29c02cc42755bcb5", "project.name": "builder"},
+ {"span.duration": 120.622, "trace": "238c3e8d8f603c5c9809a6ef04b2f8ae", "id": "2a461d023e8ff699", "project.name": "builder"},
+ {"span.duration": 136.372, "trace": "40cd17ec0555811fdac0db0ef84c67f9", "id": "2be213824f66f111", "project.name": "builder"},
+ {"span.duration": 104.705, "trace": "5df740cd6af35219b146d67add6ca548", "id": "2d97903d27f678e3", "project.name": "builder"},
+ {"span.duration": 136.017, "trace": "5942ad2d8373e590e2984cffbb1d4ab4", "id": "2d9b175fcb5b4a0d", "project.name": "builder"},
+ {"span.duration": 300.487, "trace": "d321ccb4a47b722e305a2bd178da950c", "id": "2f46f11f9b744426", "project.name": "builder"},
+ {"span.duration": 107.745, "trace": "80d81001aa346c60bc6c1a69c67c2c0a", "id": "316593b2136a501e", "project.name": "builder"},
+ {"span.duration": 183.443, "trace": "04c473902739923c07fbbef342ec1f8c", "id": "3172f03d3e0ca6d6", "project.name": "builder"},
+ {"span.duration": 183.831, "trace": "93123b4f42604ac39bab0f61155ade32", "id": "33600bd267ac6099", "project.name": "builder"},
+ {"span.duration": 121.892, "trace": "752f33729b71a68f9296226fc9d20537", "id": "33bf50c542111af8", "project.name": "builder"},
+ {"span.duration": 119.83, "trace": "5a03e339fcf9338f3f808521e4584d1a", "id": "3491c300fbfa80b3", "project.name": "builder"},
+ {"span.duration": 144.949, "trace": "158bb30c842717024f931d096c99604f", "id": "34f8534c6c11aa3f", "project.name": "builder"},
+ {"span.duration": 170.252, "trace": "b74c55d085b644f29cf96f0aa7264c24", "id": "3895e611b96e0ad8", "project.name": "builder"},
+ {"span.duration": 219.256, "trace": "54f9c2e92cefdaf79130bd08b152f101", "id": "3a47bbb276783d06", "project.name": "builder"},
+ {"span.duration": 164.535, "trace": "3b40bded557f761ca99ac964e7ccf57a", "id": "3a8dcb5577fc2728", "project.name": "builder"},
+ {"span.duration": 277.895, "trace": "c9647d819fae638f4033f90f379a5940", "id": "3c33635c50ccb841", "project.name": "builder"},
+ {"span.duration": 98.157, "trace": "a81cff77386f7b2063ceba37f14cb4be", "id": "450fa20436260b6f", "project.name": "builder"},
+ {"span.duration": 168.369, "trace": "9a04411c2df97f2e5b2da03ee46ff005", "id": "45850a928d6739db", "project.name": "builder"},
+ {"span.duration": 140.862, "trace": "f13eb8a336fde340b4cead3cfd4257e6", "id": "460ab25ad947c034", "project.name": "builder"},
+ {"span.duration": 218.134, "trace": "d321ccb4a47b722e305a2bd178da950c", "id": "483325d31ce36eff", "project.name": "builder"},
+ {"span.duration": 84.658, "trace": "fc1675cda78a097dc8fa1f75b55761c7", "id": "48914f2b383413b9", "project.name": "builder"},
+ {"span.duration": 139.418, "trace": "631155a97f32447a6d554e1e9ddbfb59", "id": "48f96c4ff39c11b9", "project.name": "builder"},
+ {"span.duration": 153.3, "trace": "ea2ee98568daedf258aae87aaa666b2e", "id": "4b5bf41319310fb5", "project.name": "builder"},
+ {"span.duration": 105.701, "trace": "b754c4cb656ffe8f9a48f5fcf35062e9", "id": "4cfe8f428439ab59", "project.name": "builder"},
+ {"span.duration": 124.37, "trace": "af485a180c8903d8b057d90f35bc5ae0", "id": "4d354903992d3c27", "project.name": "builder"},
+ {"span.duration": 231.265, "trace": "5d34ebb833fb50beb33a0a67ea64ea27", "id": "4dc1cfe1f0ce2339", "project.name": "builder"},
+ {"span.duration": 147.774, "trace": "23b01b2e362253129bf96ab315b51a3d", "id": "4e5ea245915b1618", "project.name": "builder"},
+ {"span.duration": 116.303, "trace": "dbda8fa1dda321e320514cc5ebef7feb", "id": "4e9461d40c8cb080", "project.name": "builder"},
+ {"span.duration": 144.008, "trace": "36fe9629a9f4eb895c90307d644358ee", "id": "57c9dd050e34be3d", "project.name": "builder"},
+ {"span.duration": 187.647, "trace": "daf91db3d5d0a8e15a06bab2ad006c91", "id": "5859a1c1e3e67a7d", "project.name": "builder"},
+ {"span.duration": 228.571, "trace": "f16c6db28fc197b770d4fa43eb50b6f3", "id": "591f5cd4097be396", "project.name": "builder"},
+ {"span.duration": 148.282, "trace": "63a30faceff303973ca4dd91fbaddca4", "id": "5d6004a14fc51f22", "project.name": "builder"},
+ {"span.duration": 118.126, "trace": "4cd70f4dcc8764fb8cdd015645b8c9e7", "id": "5f91f6e6d505cb19", "project.name": "builder"},
+ {"span.duration": 130.826, "trace": "e605cdaf42d3f53c0b807e849eb38368", "id": "5fce201ddcc2a507", "project.name": "builder"},
+ {"span.duration": 133.522, "trace": "dbda8fa1dda321e320514cc5ebef7feb", "id": "6238491823315b9d", "project.name": "builder"},
+ {"span.duration": 188.124, "trace": "cfc685190324c9be0738ce0a24dc7e34", "id": "63d915e6ea84973a", "project.name": "builder"},
+ {"span.duration": 118.592, "trace": "77a47124672f9c18a9948bad43580c5b", "id": "6502b8aa7cc57a0e", "project.name": "builder"},
+ {"span.duration": 122.8, "trace": "d453abf694b927b709a781d8c9d5c350", "id": "65930ca5d74dedd2", "project.name": "builder"},
+ {"span.duration": 141.07, "trace": "482ea47c0acb933ced0fa600abf3531e", "id": "6721abcaf5f81fe7", "project.name": "builder"},
+ {"span.duration": 195.152, "trace": "0b690af5643766d7188ef280ca44a51e", "id": "684a023410aa7f91", "project.name": "builder"},
+ {"span.duration": 178.098, "trace": "2034861d4fb9d60cdd90d7be82e31e5a", "id": "6a44b8520ae21b14", "project.name": "builder"},
+ {"span.duration": 150.831, "trace": "f56f1b0f8be5e3ce039daedfc03948bb", "id": "6b41529b51f6962f", "project.name": "builder"},
+ {"span.duration": 196.672, "trace": "af485a180c8903d8b057d90f35bc5ae0", "id": "6e12914a89dcb975", "project.name": "builder"},
+ {"span.duration": 261.48, "trace": "d25d6a71ab7b54b7e11febffd8c4062f", "id": "70bc7b9232b5c4a0", "project.name": "builder"},
+ {"span.duration": 127.0, "trace": "08f53e05c253050a4eea4d507bc1534e", "id": "70d6469d3e34e66a", "project.name": "builder"},
+ {"span.duration": 153.851, "trace": "cd72d371eba5d846cb0ac81ad691a7b9", "id": "73226891cb28f590", "project.name": "builder"},
+ {"span.duration": 176.159, "trace": "f4bcd412ec7ca5144391b25e6b17779b", "id": "75c2b5b3c61d2784", "project.name": "builder"},
+ {"span.duration": 90.941, "trace": "317650ab1770ed10587ab0b51af1f7d0", "id": "76efc163f2cea6b2", "project.name": "builder"},
+ {"span.duration": 147.446, "trace": "6e15e395a7a848b75012f28eb6d54730", "id": "77276239bd271cba", "project.name": "builder"},
+ {"span.duration": 234.514, "trace": "6e15e395a7a848b75012f28eb6d54730", "id": "779c077b5d49afa8", "project.name": "builder"},
+ {"span.duration": 122.683, "trace": "ae73ae8cff2667ab37ad640eae0c22e0", "id": "79bdc588ced72d14", "project.name": "builder"},
+ {"span.duration": 106.059, "trace": "20bd8e3c504a41acea638cc23b1629e7", "id": "7aa7a123aef41dd1", "project.name": "builder"},
+ {"span.duration": 144.167, "trace": "8da4d76bd43960971389c01b6a94a6e0", "id": "7f877a6af8b2812f", "project.name": "builder"},
+ {"span.duration": 106.054, "trace": "5ea8d3940318d0d27d2ec4b87fda6960", "id": "7fb4844a20f446d4", "project.name": "builder"},
+ {"span.duration": 178.578, "trace": "1c16c0505d0badb835a8c1c1dc87c6af", "id": "801efcbbdd69bb63", "project.name": "builder"},
+ {"span.duration": 163.709, "trace": "6955398598a56ada30c5eff6392bbd0c", "id": "80a1390c8832eb04", "project.name": "builder"},
+ {"span.duration": 86.831, "trace": "9949937659d8325fc15c3727d54e680a", "id": "81af88b5109697a2", "project.name": "builder"},
+ {"span.duration": 135.631, "trace": "25aa414a5e5fcb4f3207017e1f0e3073", "id": "82432171cedd7c7d", "project.name": "builder"},
+ {"span.duration": 142.176, "trace": "1295da1da104f4e48fc8aa9a065c9529", "id": "84ed46da9186d5e2", "project.name": "builder"},
+ {"span.duration": 248.184, "trace": "55d7a9f7a64b7a90c2bcf24ff5480108", "id": "867a2fe88dbe7eca", "project.name": "builder"},
+ {"span.duration": 153.327, "trace": "54f9c2e92cefdaf79130bd08b152f101", "id": "871abc5a95080194", "project.name": "builder"},
+ {"span.duration": 168.451, "trace": "be391e5ba705850c7306ee99605a0f1c", "id": "8812cac819d53579", "project.name": "builder"},
+ {"span.duration": 168.039, "trace": "73e39aff145e264d924947c840ad1e1b", "id": "888efd00590c0f2c", "project.name": "builder"},
+ {"span.duration": 252.507, "trace": "ac792dba9b936fbc969ccbbd6f2852f8", "id": "89888d7b77e776bf", "project.name": "builder"},
+ {"span.duration": 127.523, "trace": "fd8294362724b9ed5b266478c0a52471", "id": "8c4d02f6a8b68959", "project.name": "builder"},
+ {"span.duration": 97.845, "trace": "a895a73a2efeb23baa9a2d9f79f7473b", "id": "8c6d761fb676778c", "project.name": "builder"},
+ {"span.duration": 179.472, "trace": "97cbadca83216255004764cc2b192d39", "id": "8d4b9e197c2f37f2", "project.name": "builder"},
+ {"span.duration": 197.601, "trace": "3fe0ed41e846de70f66868b920e4527f", "id": "8dd015bf1e5aef97", "project.name": "builder"},
+ {"span.duration": 183.403, "trace": "c9275bb55939663e02e648ae9ae9941e", "id": "8e2c541081377b7c", "project.name": "builder"},
+ {"span.duration": 145.483, "trace": "2fea43bebfc65412964c86301e6aa000", "id": "9016aa2d7036fd3d", "project.name": "builder"},
+ {"span.duration": 186.69, "trace": "d321ccb4a47b722e305a2bd178da950c", "id": "9800950473f987a2", "project.name": "builder"},
+ {"span.duration": 147.091, "trace": "36834045a914d69d357f57ab5b37b0f2", "id": "991be73ce0f3e6c7", "project.name": "builder"},
+ {"span.duration": 169.011, "trace": "54e03aebe000d1d28ac552bcc4e39218", "id": "9a03f1258d564a12", "project.name": "builder"},
+ {"span.duration": 162.644, "trace": "9fc94888c024819393c97de35d8f8838", "id": "9bff51c7bb257285", "project.name": "builder"}
+ ],
+ "meta": {"fields": {"span.duration": "duration", "trace": "string", "id": "string", "project.name": "string"}, "units": {"span.duration": "millisecond"}, "dataset": "spans"}
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/fixtures/samples_candidate_page2.json b/.claude/skills/perf-compare/scripts/fixtures/samples_candidate_page2.json
new file mode 100644
index 0000000000..1bd057e55f
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/fixtures/samples_candidate_page2.json
@@ -0,0 +1,68 @@
+{
+ "headers": {"link": "<https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:0:1>; rel=\"previous\"; results=\"true\"; cursor=\"0:0:1\", <https://us.sentry.io/api/0/organizations/significant-gravitas/events/?cursor=0:200:0>; rel=\"next\"; results=\"false\"; cursor=\"0:200:0\""},
+ "body": {
+ "data": [
+ {"span.duration": 219.415, "trace": "af485a180c8903d8b057d90f35bc5ae0", "id": "9d7aeb863eb21428", "project.name": "builder"},
+ {"span.duration": 209.121, "trace": "b0e190c10c3e05cbd264cf412dfeb3a5", "id": "9e9cf1be6698d98e", "project.name": "builder"},
+ {"span.duration": 101.914, "trace": "1ede839d4acab8917022bcaf165d2432", "id": "9ed033792440ef67", "project.name": "builder"},
+ {"span.duration": 123.643, "trace": "4c5b4bb1a26ffa54cf7c0ab433d36154", "id": "a01f4903e5d98fda", "project.name": "builder"},
+ {"span.duration": 157.064, "trace": "8d73ba64d36f3574c91a1f7a987cb93c", "id": "a16e5f575f5f8c31", "project.name": "builder"},
+ {"span.duration": 96.965, "trace": "5ea8d3940318d0d27d2ec4b87fda6960", "id": "a1f6b8d6a581ae37", "project.name": "builder"},
+ {"span.duration": 109.94, "trace": "ee3b79a4f1a7592fbfbbb19a797ff574", "id": "a74998795904e08c", "project.name": "builder"},
+ {"span.duration": 142.423, "trace": "94a2cb7c9a492e0648e5a79c23727826", "id": "a8a879296a41f815", "project.name": "builder"},
+ {"span.duration": 117.299, "trace": "8415c00eebc71bea9d48d227a6c1e979", "id": "a8d100d16732f45e", "project.name": "builder"},
+ {"span.duration": 112.665, "trace": "0b6d673876546c40c50970340b379f02", "id": "a9bece3772d5e2b5", "project.name": "builder"},
+ {"span.duration": 210.882, "trace": "d46454630da3b29c894911e4aaa6fa38", "id": "ab41a5d4b3fb65e2", "project.name": "builder"},
+ {"span.duration": 241.059, "trace": "5d34ebb833fb50beb33a0a67ea64ea27", "id": "abd787b2e4b8a965", "project.name": "builder"},
+ {"span.duration": 115.099, "trace": "d46454630da3b29c894911e4aaa6fa38", "id": "abecf5e53946d9d6", "project.name": "builder"},
+ {"span.duration": 85.434, "trace": "20891b28b9e77127e913d8d34adbfcff", "id": "acad99fe07282141", "project.name": "builder"},
+ {"span.duration": 147.857, "trace": "5d34ebb833fb50beb33a0a67ea64ea27", "id": "ace62992b4e2d31a", "project.name": "builder"},
+ {"span.duration": 250.499, "trace": "8da4d76bd43960971389c01b6a94a6e0", "id": "affa1cd42554ef63", "project.name": "builder"},
+ {"span.duration": 126.109, "trace": "0d8054e9a0e273f2bf248325ebcf6736", "id": "b086310682d051d0", "project.name": "builder"},
+ {"span.duration": 190.363, "trace": "87aa699b0b206c5cf87264cfac65003e", "id": "b4c9af710dfcfe8b", "project.name": "builder"},
+ {"span.duration": 100.69, "trace": "ae2b896c148044ee1ee2f1dfed571b20", "id": "b4d92dddbfa22d79", "project.name": "builder"},
+ {"span.duration": 104.358, "trace": "826c81e658a2f385088596a1823e2105", "id": "b65d1579ca4e94a4", "project.name": "builder"},
+ {"span.duration": 316.428, "trace": "2b4fa3f8e8677f37382a89a57be207dd", "id": "b9c87ee927ddc958", "project.name": "builder"},
+ {"span.duration": 146.029, "trace": "8d73ba64d36f3574c91a1f7a987cb93c", "id": "c049f149063120e4", "project.name": "builder"},
+ {"span.duration": 181.537, "trace": "23b01b2e362253129bf96ab315b51a3d", "id": "c14b3344930f4aa9", "project.name": "builder"},
+ {"span.duration": 217.915, "trace": "25aa414a5e5fcb4f3207017e1f0e3073", "id": "c2970c1370f6aabc", "project.name": "builder"},
+ {"span.duration": 217.765, "trace": "1cef535a9c3e595bb1c827fe96f22925", "id": "c3355fa76e7c0b93", "project.name": "builder"},
+ {"span.duration": 295.435, "trace": "5e6713f51b58112e9abdaf40d40dacce", "id": "c3ba3a0824f1d09f", "project.name": "builder"},
+ {"span.duration": 187.823, "trace": "65afb91ebf15b22a5f2600c66b4e4478", "id": "c57dd5904e5c765e", "project.name": "builder"},
+ {"span.duration": 107.466, "trace": "0754a99964be73eb370d561eb878b3ee", "id": "c67f0f535371f6ab", "project.name": "builder"},
+ {"span.duration": 147.285, "trace": "8b26d874b42139418ab34cc5b5c64692", "id": "c756e075c00c7747", "project.name": "builder"},
+ {"span.duration": 140.725, "trace": "68c6d0004876d0d256854db3ed7eec1b", "id": "c98be3f3babf450c", "project.name": "builder"},
+ {"span.duration": 157.613, "trace": "99352fd95d2a106fc740410d4e6fcf18", "id": "c9dbe52968ef7d0f", "project.name": "builder"},
+ {"span.duration": 234.584, "trace": "5df740cd6af35219b146d67add6ca548", "id": "ccc3b5519a96c17b", "project.name": "builder"},
+ {"span.duration": 171.592, "trace": "a6a9104dc56ca0dbbe8fee80177553a8", "id": "ccffe41144507699", "project.name": "builder"},
+ {"span.duration": 105.526, "trace": "16b116493f047105a0aba7be458d7664", "id": "cd360588034fc2bf", "project.name": "builder"},
+ {"span.duration": 153.675, "trace": "b9506975778724bad706d8fc6a1e21c2", "id": "ce224afb31ce130b", "project.name": "builder"},
+ {"span.duration": 153.971, "trace": "49b53b1e4a782d49e41d65635f611dd0", "id": "ce9373c17f069d66", "project.name": "builder"},
+ {"span.duration": 137.72, "trace": "65afb91ebf15b22a5f2600c66b4e4478", "id": "cf9208165775ae6a", "project.name": "builder"},
+ {"span.duration": 125.557, "trace": "ad2d45d47407ff41117f4a1ade2f3afc", "id": "d089bc154351a741", "project.name": "builder"},
+ {"span.duration": 149.713, "trace": "afc53a7303d3c0ab347945b00482c25c", "id": "d64d3e33e8ee5d4f", "project.name": "builder"},
+ {"span.duration": 127.484, "trace": "0e9943f8b1468a6bc40988d4d56c901f", "id": "dab5cbd0816135bf", "project.name": "builder"},
+ {"span.duration": 101.381, "trace": "24bd18d6092fa571c98980c6663f4a72", "id": "dbfeb45f37a577d6", "project.name": "builder"},
+ {"span.duration": 132.97, "trace": "8d73ba64d36f3574c91a1f7a987cb93c", "id": "de6e920139072a3b", "project.name": "builder"},
+ {"span.duration": 193.274, "trace": "0754a99964be73eb370d561eb878b3ee", "id": "de874341635fa777", "project.name": "builder"},
+ {"span.duration": 159.626, "trace": "5b9f41d23329085a3a6b74a5b075538c", "id": "df7616c38f3b92b5", "project.name": "builder"},
+ {"span.duration": 144.806, "trace": "58315ee8f84efd67ae013bd2872ab6af", "id": "e16deb29ebb2dd3a", "project.name": "builder"},
+ {"span.duration": 160.296, "trace": "20bd8e3c504a41acea638cc23b1629e7", "id": "e246c401f0ef055a", "project.name": "builder"},
+ {"span.duration": 201.159, "trace": "2806fb75c98a2aa001648850e1c31372", "id": "e31f78485302b4d8", "project.name": "builder"},
+ {"span.duration": 176.187, "trace": "826c81e658a2f385088596a1823e2105", "id": "e495fd57dd90ed62", "project.name": "builder"},
+ {"span.duration": 134.4, "trace": "35fdf24155cb1da03afe2fde938e6b9e", "id": "e838d92b74abc429", "project.name": "builder"},
+ {"span.duration": 154.719, "trace": "54e03aebe000d1d28ac552bcc4e39218", "id": "ec9f0cbb3e5b83fb", "project.name": "builder"},
+ {"span.duration": 160.627, "trace": "e42719c5bd5b0b154038331d235e8c2a", "id": "f19e3f9ca2448210", "project.name": "builder"},
+ {"span.duration": 160.143, "trace": "93123b4f42604ac39bab0f61155ade32", "id": "f4c33e4e2fcbe716", "project.name": "builder"},
+ {"span.duration": 279.188, "trace": "fe4c25b16ecfab662f0900eebd2672ae", "id": "f83325e43a571571", "project.name": "builder"},
+ {"span.duration": 103.612, "trace": "03e854078abd9d822b67d1b7c021cd64", "id": "fa5600ea69243d50", "project.name": "builder"},
+ {"span.duration": 145.054, "trace": "b4861b4afeacc4a93b3f971cae814ac0", "id": "fb12c5ca53770c69", "project.name": "builder"},
+ {"span.duration": 308.244, "trace": "04025e4d3b1bb3a3f57fbbbdb14a117b", "id": "fbf59480e98e99cf", "project.name": "builder"},
+ {"span.duration": 157.356, "trace": "fe8242c44255368af3ed2c356546d6d7", "id": "fc146beb18eba532", "project.name": "builder"},
+ {"span.duration": 272.251, "trace": "5edcec86e12fa92c1efb8681639dce12", "id": "fc8f8764c17879bc", "project.name": "builder"},
+ {"span.duration": 183.286, "trace": "8330724981ee0ff04f6a12763dc24d60", "id": "fc9bb43c93aa3212", "project.name": "builder"},
+ {"span.duration": 153.179, "trace": "8d73ba64d36f3574c91a1f7a987cb93c", "id": "feae8b71d7535b0b", "project.name": "builder"}
+ ],
+ "meta": {"fields": {"span.duration": "duration", "trace": "string", "id": "string", "project.name": "string"}, "units": {"span.duration": "millisecond"}, "dataset": "spans"}
+ }
+}
diff --git a/.claude/skills/perf-compare/scripts/perf_compare.py b/.claude/skills/perf-compare/scripts/perf_compare.py
new file mode 100755
index 0000000000..85f45939e0
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/perf_compare.py
@@ -0,0 +1,1823 @@
+#!/usr/bin/env python3
+"""Compare one numeric span field between a baseline and a candidate in Sentry.
+
+This is the shared measurement recipe for AutoGPT performance work. It reads
+transactions from Sentry's Events API (dataset ``spans``) for two selectors,
+reports count, p50, p95, p99 and failure rate per transaction, then pulls a
+bounded random sample of raw values per side and bootstraps a confidence
+interval for the p50 and p95 difference. The interval decides the verdict.
+``SKILL.md`` one directory up explains when to use it and how to read the
+result.
+
+Standard library only; Python 3.8 or newer.
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import http.client
+import json
+import math
+import os
+import random
+import re
+import sys
+import threading
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass
+from dataclasses import field as dataclass_field
+from dataclasses import replace
+from datetime import datetime, timedelta, timezone
+from typing import Callable, Dict, List, Optional, Sequence, Tuple
+
+DEFAULT_ORG = "significant-gravitas"
+DEFAULT_REGION_URL = "https://us.sentry.io"
+DEFAULT_FIELD = "span.duration"
+DEFAULT_PERIOD = "7d"
+SCHEMA_VERSION = 1
+USER_AGENT = "autogpt-perf-compare/1"
+PAGE_SIZE = 100
+MAX_PAGES = 200
+MAX_SAMPLE_CAP = PAGE_SIZE * MAX_PAGES
+OVERVIEW_ROWS = 15
+RESAMPLE_CAP = 50000
+TARGET_TAIL_DRAWS = 20
+MIN_TAIL_DRAWS = 10
+LOCAL_HOSTS = ("localhost", "127.0.0.1", "::1")
+
+FASTER = "faster"
+SLOWER = "slower"
+NO_CHANGE = "no detectable change"
+NOT_ENOUGH_DATA = "not enough data"
+MORE_FAILURES = "more failures"
+HIGHER = "higher"
+LOWER = "lower"
+
+EXIT_FASTER = 0
+EXIT_SLOWER = 1
+EXIT_USAGE = 2
+EXIT_NO_CHANGE = 3
+EXIT_NOT_ENOUGH_DATA = 4
+EXIT_AUTH = 5
+EXIT_API = 6
+
+EXIT_CODES_HELP = """\
+exit codes:
+ 0 faster: a compared transaction got faster and nothing regressed
+ 1 slower or more failures: at least one compared transaction regressed
+ 2 usage error: bad arguments, unknown project, or Sentry rejected the query
+ 3 no detectable change: nothing moved beyond the noise
+ 4 not enough data: no compared transaction met --min-count on both sides
+ 5 auth: SENTRY_AUTH_TOKEN missing, or Sentry rejected it (401/403)
+ 6 Sentry API or network error after retries, or any unexpected error
+"""
+
+TOKEN_SCOPES = "org:read project:read event:read"
+
+MISSING_TOKEN_MESSAGE = f"""\
+perf-compare: SENTRY_AUTH_TOKEN is not set.
+
+This tool reads Sentry through its REST API and takes the token only from the
+SENTRY_AUTH_TOKEN environment variable; it never looks for tokens on disk.
+Use a read-only token with exactly these scopes:
+
+ {TOKEN_SCOPES}
+
+Either a user auth token (Sentry > Settings > Account > API > Auth Tokens) or
+an internal integration with Read on Organization, Project and Issue & Event.
+Organization auth tokens (sntrys_...) are locked to org:ci and cannot read
+events. Then:
+
+ export SENTRY_AUTH_TOKEN=<token>
+"""
+
+SIDE_KEYS = ("project", "env", "release", "query", "window", "start", "end", "period")
+FIELD_PATTERN = re.compile(r"^[A-Za-z0-9_.\[\],:-]+$")
+PERIOD_PATTERN = re.compile(r"^(\d+)([mhdw])$")
+PERIOD_UNITS = {"m": "minutes", "h": "hours", "d": "days", "w": "weeks"}
+LINK_PATTERN = re.compile(r"<([^>]*)>((?:\s*;\s*[A-Za-z_]+=\"[^\"]*\")*)")
+LINK_ATTR_PATTERN = re.compile(r"([A-Za-z_]+)=\"([^\"]*)\"")
+UNIT_SUFFIX = {
+ "nanosecond": "ns",
+ "microsecond": "us",
+ "millisecond": "ms",
+ "second": "s",
+ "byte": "B",
+}
+
+Transport = Callable[[str, Dict[str, str], float], Tuple[int, Dict[str, str], bytes]]
+
+
+class PerfCompareError(Exception):
+ exit_code = EXIT_API
+ kind = "api"
+
+
+class UsageError(PerfCompareError):
+ exit_code = EXIT_USAGE
+ kind = "usage"
+
+
+class AuthError(PerfCompareError):
+ exit_code = EXIT_AUTH
+ kind = "auth"
+
+
+class SentryApiError(PerfCompareError):
+ exit_code = EXIT_API
+ kind = "api"
+
+
+class NotFoundError(SentryApiError):
+ pass
+
+
+# Time windows ---------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class Window:
+ start: datetime
+ end: datetime
+
+ @property
+ def seconds(self) -> int:
+ return int((self.end - self.start).total_seconds())
+
+ def as_json(self) -> List[str]:
+ return [format_filter_time(self.start), format_filter_time(self.end)]
+
+
+def parse_period(text: str) -> timedelta:
+ match = PERIOD_PATTERN.match(text.strip())
+ if not match or int(match.group(1)) == 0:
+ raise UsageError(
+ f"bad period {text!r}: use a number and m, h, d or w, e.g. 24h or 7d"
+ )
+ return timedelta(**{PERIOD_UNITS[match.group(2)]: int(match.group(1))})
+
+
+def parse_time(text: str) -> datetime:
+ value = text.strip()
+ if value[-1:] in ("Z", "z"):
+ value = value[:-1] + "+00:00"
+ try:
+ parsed = datetime.fromisoformat(value)
+ except ValueError:
+ raise UsageError(
+ f"bad time {text!r}: use ISO 8601 in UTC, e.g. 2026-10-11T14:00:00Z"
+ ) from None
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ return parsed.astimezone(timezone.utc).replace(microsecond=0)
+
+
+def parse_window(text: str) -> Window:
+ parts = text.split("/")
+ if len(parts) != 2:
+ raise UsageError(
+ f"bad window {text!r}: use START/END, "
+ "e.g. 2026-10-11T14:00:00Z/2026-10-11T14:30:00Z"
+ )
+ return Window(parse_time(parts[0]), parse_time(parts[1]))
+
+
+def format_param_time(moment: datetime) -> str:
+ return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S")
+
+
+def format_filter_time(moment: datetime) -> str:
+ return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def check_windows(windows: Sequence[Window], label: str) -> List[Window]:
+ ordered = sorted(windows, key=lambda window: window.start)
+ for window in ordered:
+ if window.end <= window.start:
+ raise UsageError(
+ f"{label}: window {format_filter_time(window.start)}/"
+ f"{format_filter_time(window.end)} does not end after it starts"
+ )
+ for index in range(1, len(ordered)):
+ if ordered[index].start < ordered[index - 1].end:
+ raise UsageError(f"{label}: windows overlap; give each event one window")
+ return ordered
+
+
+def resolve_windows(
+ label: str,
+ windows: Sequence[str],
+ start: Optional[str],
+ end: Optional[str],
+ period: Optional[str],
+ now: datetime,
+) -> Optional[List[Window]]:
+ if not (windows or start or end or period):
+ return None
+ if (windows and (start or end or period)) or (period and (start or end)):
+ raise UsageError(
+ f"{label}: use one of window=, start=/end= or period=, not several"
+ )
+ if windows:
+ resolved = [parse_window(window) for window in windows]
+ elif period:
+ resolved = [Window(now - parse_period(period), now)]
+ else:
+ if not start:
+ raise UsageError(f"{label}: end needs a start")
+ resolved = [Window(parse_time(start), parse_time(end) if end else now)]
+ return check_windows(resolved, label)
+
+
+# Selectors ------------------------------------------------------------------
+
+
+@dataclass
+class Shared:
+ project: Optional[str]
+ env: Optional[str]
+ query: Optional[str]
+ windows: List[Window]
+
+
+@dataclass
+class Side:
+ label: str
+ project: str
+ environment: Optional[str]
+ release: Optional[str]
+ query: Optional[str]
+ windows: List[Window]
+
+ def selector(self) -> Dict[str, object]:
+ return {
+ "project": self.project,
+ "environment": self.environment,
+ "release": self.release,
+ "query": self.query,
+ "windows": [window.as_json() for window in self.windows],
+ }
+
+
+def parse_side(
+ label: str, tokens: Sequence[str], shared: Shared, now: datetime
+) -> Side:
+ values: Dict[str, str] = {}
+ windows: List[str] = []
+ for token in tokens:
+ key, separator, value = token.partition("=")
+ key = key.strip().lower()
+ key = "env" if key == "environment" else key
+ if not separator or key not in SIDE_KEYS:
+ raise UsageError(
+ f"--{label} takes KEY=VALUE pairs with KEY one of "
+ f"{', '.join(SIDE_KEYS)}; got {token!r}"
+ )
+ if key == "window":
+ windows.append(value)
+ continue
+ if key in values:
+ raise UsageError(f"--{label} sets {key}= twice")
+ values[key] = value
+ project = values.get("project") or shared.project
+ if not project:
+ raise UsageError(
+ f"the {label} has no project: pass --project SLUG or "
+ f"--{label} project=SLUG"
+ )
+ side_windows = resolve_windows(
+ label,
+ windows,
+ values.get("start"),
+ values.get("end"),
+ values.get("period"),
+ now,
+ )
+ return Side(
+ label=label,
+ project=project,
+ environment=values.get("env", shared.env) or None,
+ release=values.get("release") or None,
+ query=values.get("query", shared.query) or None,
+ windows=side_windows if side_windows is not None else list(shared.windows),
+ )
+
+
+# Query building -------------------------------------------------------------
+
+
+def quote(value: str) -> str:
+ return '"' + value.replace("\\", "\\\\").replace('"', '\\"') + '"'
+
+
+def window_clause(windows: Sequence[Window]) -> str:
+ terms = [
+ f"timestamp:>={format_filter_time(window.start)} "
+ f"timestamp:<{format_filter_time(window.end)}"
+ for window in windows
+ ]
+ return terms[0] if len(terms) == 1 else "(" + " OR ".join(terms) + ")"
+
+
+def transaction_clause(names: Sequence[str]) -> Optional[str]:
+ terms = [f"transaction:{quote(name)}" for name in names]
+ if not terms:
+ return None
+ return terms[0] if len(terms) == 1 else "(" + " OR ".join(terms) + ")"
+
+
+def build_query(
+ side: Side,
+ field_name: str,
+ transactions: Sequence[str] = (),
+ windows: Optional[Sequence[Window]] = None,
+) -> str:
+ """The Sentry search string for one side; valid for REST and the MCP."""
+ parts = ["is_transaction:true"]
+ if side.environment:
+ parts.append(f"environment:{quote(side.environment)}")
+ if side.release:
+ parts.append(f"release:{quote(side.release)}")
+ if field_name != DEFAULT_FIELD:
+ parts.append(f"has:{field_name}")
+ if side.query:
+ parts.append(f"({side.query})")
+ names = transaction_clause(transactions)
+ if names:
+ parts.append(names)
+ parts.append(window_clause(side.windows if windows is None else windows))
+ return " ".join(parts)
+
+
+def aggregate_fields(field_name: str) -> List[str]:
+ return [
+ "transaction",
+ "count()",
+ f"p50({field_name})",
+ f"p95({field_name})",
+ f"p99({field_name})",
+ "failure_rate()",
+ ]
+
+
+def sample_fields(field_name: str) -> List[str]:
+ return [field_name, "trace", "id"]
+
+
+SAMPLE_SORT = "id"
+
+
+def events_params(
+ project_id: str,
+ query: str,
+ fields: Sequence[str],
+ sort: str,
+ windows: Sequence[Window],
+ per_page: int,
+) -> List[Tuple[str, str]]:
+ params = [("dataset", "spans"), ("project", str(project_id))]
+ params.extend(("field", name) for name in fields)
+ params.extend(
+ [
+ ("query", query),
+ ("sort", sort),
+ ("per_page", str(per_page)),
+ ("start", format_param_time(min(window.start for window in windows))),
+ ("end", format_param_time(max(window.end for window in windows))),
+ ]
+ )
+ return params
+
+
+def mcp_period(windows: Sequence[Window], now: datetime) -> str:
+ earliest = min(window.start for window in windows)
+ return f"{max(1, math.ceil((now - earliest).total_seconds() / 3600)) + 1}h"
+
+
+def mcp_arguments(
+ org: str,
+ region_url: str,
+ side: Side,
+ query: str,
+ fields: Sequence[str],
+ sort: str,
+ limit: int,
+ now: datetime,
+) -> Dict[str, object]:
+ return {
+ "organizationSlug": org,
+ "regionUrl": region_url,
+ "projectSlug": side.project,
+ "dataset": "spans",
+ "query": query,
+ "fields": list(fields),
+ "sort": sort,
+ "period": mcp_period(side.windows, now),
+ "limit": limit,
+ }
+
+
+# Sentry client --------------------------------------------------------------
+
+
+def urllib_transport(
+ url: str, headers: Dict[str, str], timeout: float
+) -> Tuple[int, Dict[str, str], bytes]:
+ request = urllib.request.Request(url, headers=headers)
+ try:
+ with urllib.request.urlopen(request, timeout=timeout) as response:
+ response_headers = {k.lower(): v for k, v in response.headers.items()}
+ return response.status, response_headers, response.read()
+ except urllib.error.HTTPError as error:
+ error_headers = {k.lower(): v for k, v in (error.headers or {}).items()}
+ return error.code, error_headers, error.read()
+
+
+def next_cursor(link_header: str) -> Optional[str]:
+ for match in LINK_PATTERN.finditer(link_header or ""):
+ attributes = dict(LINK_ATTR_PATTERN.findall(match.group(2)))
+ if attributes.get("rel") == "next" and attributes.get("results") == "true":
+ return attributes.get("cursor") or None
+ return None
+
+
+def error_detail(body: bytes) -> str:
+ text = body.decode("utf-8", errors="replace")
+ try:
+ detail = json.loads(text).get("detail")
+ except (ValueError, AttributeError):
+ detail = None
+ return str(detail if detail is not None else text).strip()[:400]
+
+
+class SentryClient:
+ def __init__(
+ self,
+ token: str,
+ org: str = DEFAULT_ORG,
+ region_url: str = DEFAULT_REGION_URL,
+ transport: Optional[Transport] = None,
+ timeout: float = 30.0,
+ retries: int = 3,
+ sleep: Callable[[float], None] = time.sleep,
+ ) -> None:
+ self.token = token
+ self.org = org
+ self.region_url = region_url.rstrip("/")
+ self.transport = transport or urllib_transport
+ self.timeout = timeout
+ self.retries = retries
+ self.sleep = sleep
+ self.requests = 0
+ self._lock = threading.Lock()
+ self._project_ids: Dict[str, str] = {}
+
+ def get(
+ self, path: str, params: Sequence[Tuple[str, str]] = ()
+ ) -> Tuple[object, Dict[str, str]]:
+ url = self.region_url + path
+ if params:
+ url += "?" + urllib.parse.urlencode(list(params))
+ headers = {
+ "Authorization": f"Bearer {self.token}",
+ "Accept": "application/json",
+ "User-Agent": USER_AGENT,
+ }
+ last_problem = ""
+ for attempt in range(self.retries + 1):
+ with self._lock:
+ self.requests += 1
+ retry_after: Optional[str] = None
+ try:
+ status, response_headers, body = self.transport(
+ url, headers, self.timeout
+ )
+ except (OSError, http.client.HTTPException) as error:
+ last_problem = f"network error: {error}"
+ else:
+ if status == 200:
+ try:
+ return json.loads(body.decode("utf-8")), response_headers
+ except ValueError:
+ raise SentryApiError(
+ f"Sentry returned invalid JSON for {path}"
+ ) from None
+ detail = error_detail(body)
+ if status in (401, 403):
+ raise AuthError(
+ f"Sentry rejected SENTRY_AUTH_TOKEN (HTTP {status}: {detail}). "
+ f"The token needs {TOKEN_SCOPES.replace(' ', ', ')}; "
+ "organization auth tokens (sntrys_...) only carry org:ci, so "
+ "use a user auth token or an internal integration token."
+ )
+ if status == 400:
+ raise UsageError(
+ f"Sentry rejected the request (HTTP 400): {detail}"
+ )
+ if status == 404:
+ raise NotFoundError(f"HTTP 404 from {path}: {detail}")
+ if status != 429 and status < 500:
+ raise SentryApiError(f"HTTP {status} from {path}: {detail}")
+ last_problem = f"HTTP {status}: {detail}"
+ retry_after = response_headers.get("retry-after")
+ if attempt < self.retries:
+ self.sleep(backoff_seconds(attempt, retry_after))
+ raise SentryApiError(
+ f"giving up on {path} after {self.retries + 1} attempts: {last_problem}"
+ )
+
+ def project_id(self, project: str) -> str:
+ if project.isdigit():
+ return project
+ if project not in self._project_ids:
+ path = "/api/0/projects/{}/{}/".format(
+ urllib.parse.quote(self.org, safe=""),
+ urllib.parse.quote(project, safe=""),
+ )
+ try:
+ data, _ = self.get(path)
+ except NotFoundError:
+ raise UsageError(
+ f"unknown Sentry project {project!r} in org {self.org!r}"
+ ) from None
+ if not isinstance(data, dict) or "id" not in data:
+ raise SentryApiError(f"unexpected project response for {project!r}")
+ self._project_ids[project] = str(data["id"])
+ return self._project_ids[project]
+
+ def events(
+ self, params: Sequence[Tuple[str, str]]
+ ) -> Tuple[List[Dict[str, object]], Dict[str, object], Dict[str, str]]:
+ path = "/api/0/organizations/{}/events/".format(
+ urllib.parse.quote(self.org, safe="")
+ )
+ data, headers = self.get(path, params)
+ if not isinstance(data, dict) or not isinstance(data.get("data"), list):
+ raise SentryApiError("unexpected events response: no data list")
+ meta = data.get("meta") if isinstance(data.get("meta"), dict) else {}
+ return data["data"], meta, headers
+
+ def paginate(
+ self, params: Sequence[Tuple[str, str]], limit: int
+ ) -> List[Dict[str, object]]:
+ rows: List[Dict[str, object]] = []
+ cursor: Optional[str] = None
+ for _ in range(MAX_PAGES):
+ page_params = list(params) + ([("cursor", cursor)] if cursor else [])
+ page, _, headers = self.events(page_params)
+ rows.extend(page)
+ cursor = next_cursor(headers.get("link", ""))
+ if len(rows) >= limit or not page or not cursor:
+ break
+ return rows[:limit]
+
+
+def backoff_seconds(attempt: int, retry_after: Optional[str]) -> float:
+ if retry_after:
+ try:
+ return min(60.0, max(0.0, float(retry_after)))
+ except ValueError:
+ pass
+ return float(min(30, 2**attempt))
+
+
+# Statistics -----------------------------------------------------------------
+
+
+def percentile(sorted_values: Sequence[float], quantile: float) -> float:
+ """Linear interpolation between closest ranks (numpy's default)."""
+ if not sorted_values:
+ raise ValueError("percentile of an empty sample")
+ if not 0.0 <= quantile <= 1.0:
+ raise ValueError("quantile must be within [0, 1]")
+ position = (len(sorted_values) - 1) * quantile
+ low = math.floor(position)
+ high = math.ceil(position)
+ if low == high:
+ return float(sorted_values[low])
+ return sorted_values[low] + (sorted_values[high] - sorted_values[low]) * (
+ position - low
+ )
+
+
+def _drawer(
+ clusters: Sequence[Sequence[float]], rng: random.Random
+) -> Callable[[], List[float]]:
+ count = len(clusters)
+ if all(len(cluster) == 1 for cluster in clusters):
+ flat = [cluster[0] for cluster in clusters]
+ return lambda: rng.choices(flat, k=count)
+ return lambda: [
+ value for cluster in rng.choices(clusters, k=count) for value in cluster
+ ]
+
+
+def bootstrap_quantile_diffs(
+ baseline: Sequence[Sequence[float]],
+ candidate: Sequence[Sequence[float]],
+ quantiles: Sequence[float],
+ resamples: int,
+ rng: random.Random,
+) -> Dict[float, List[float]]:
+ """Candidate minus baseline quantile, per resample.
+
+ Each side is a list of clusters (all values from one trace). Resampling
+ whole clusters keeps correlated spans of one request together, so the
+ interval does not pretend a page that fired ten calls is ten
+ independent observations.
+ """
+ draw_baseline = _drawer(baseline, rng)
+ draw_candidate = _drawer(candidate, rng)
+ diffs: Dict[float, List[float]] = {quantile: [] for quantile in quantiles}
+ for _ in range(resamples):
+ base_values = sorted(draw_baseline())
+ cand_values = sorted(draw_candidate())
+ for quantile in quantiles:
+ diffs[quantile].append(
+ percentile(cand_values, quantile) - percentile(base_values, quantile)
+ )
+ return diffs
+
+
+def confidence_interval(
+ samples: Sequence[float], confidence: float
+) -> Tuple[float, float]:
+ ordered = sorted(samples)
+ alpha = 1.0 - confidence
+ return percentile(ordered, alpha / 2), percentile(ordered, 1 - alpha / 2)
+
+
+def relative(diff: Optional[float], base: Optional[float]) -> Optional[float]:
+ if diff is None or base is None or base == 0:
+ return None
+ return diff / base
+
+
+# Verdicts -------------------------------------------------------------------
+
+
+@dataclass
+class Options:
+ field: str = DEFAULT_FIELD
+ transactions: List[str] = dataclass_field(default_factory=list)
+ top: int = 5
+ sample_cap: int = 1000
+ min_count: int = 200
+ min_traces: int = 30
+ min_effect: float = 0.05
+ min_failure_delta: float = 0.001
+ confidence: float = 0.95
+ resamples: int = 4000
+ seed: int = 1234
+ concurrency: int = 4
+
+
+def adjusted_confidence(confidence: float, family_size: int) -> float:
+ """Per-interval level that keeps ``family_size`` intervals at ``confidence``.
+
+ Bonferroni: comparing ten percentiles at 95% each would call one of them
+ a change by luck about 40% of the time; this holds the whole set at 95%.
+ """
+ return 1 - (1 - confidence) / max(1, family_size)
+
+
+def tail_share(level: float) -> float:
+ return (1 - level) / 2
+
+
+def required_resamples(level: float) -> int:
+ """Resamples that leave TARGET_TAIL_DRAWS draws in each tail at ``level``."""
+ return min(RESAMPLE_CAP, math.ceil(TARGET_TAIL_DRAWS / tail_share(level) - 1e-9))
+
+
+def effective_resamples(requested: int, level: float) -> int:
+ return max(requested, required_resamples(level))
+
+
+def sample_shortfall(
+ baseline: Sequence[Sequence[float]],
+ candidate: Sequence[Sequence[float]],
+ options: Options,
+) -> Optional[str]:
+ """Why the samples cannot support a verdict, or None when they can.
+
+ Spans from one trace move together, so the bootstrap learns nothing about
+ variation between requests from a side made of a handful of traces, no
+ matter how many spans those traces hold. Both counts have to clear.
+ """
+ spans = (sample_size(baseline), sample_size(candidate))
+ traces = (len(baseline), len(candidate))
+ if min(spans) < options.min_count:
+ return (
+ f"needs {options.min_count} samples per side, has "
+ f"{spans[0]} / {spans[1]}"
+ )
+ if min(traces) < options.min_traces:
+ return (
+ f"needs {options.min_traces} traces per side, has "
+ f"{traces[0]} / {traces[1]}"
+ )
+ return None
+
+
+def quantile_verdict(
+ ci_low: float,
+ ci_high: float,
+ effects: Sequence[Optional[float]],
+ min_effect: float,
+) -> str:
+ """Faster or slower needs the interval to exclude zero and every known
+ relative effect (sample and Sentry's all-events percentile) to clear
+ ``min_effect`` in that direction."""
+ known = [effect for effect in effects if effect is not None]
+ if ci_high < 0 and all(effect <= -min_effect for effect in known):
+ return FASTER
+ if ci_low > 0 and all(effect >= min_effect for effect in known):
+ return SLOWER
+ return NO_CHANGE
+
+
+def combine_verdicts(verdicts: Sequence[str]) -> str:
+ if not verdicts or all(verdict == NOT_ENOUGH_DATA for verdict in verdicts):
+ return NOT_ENOUGH_DATA
+ if SLOWER in verdicts:
+ return SLOWER
+ if FASTER in verdicts:
+ return FASTER
+ return NO_CHANGE
+
+
+QUANTILES = (("p50", 0.5), ("p95", 0.95))
+
+
+def sample_size(clusters: Sequence[Sequence[float]]) -> int:
+ return sum(len(cluster) for cluster in clusters)
+
+
+def compare_quantiles(
+ baseline: Sequence[Sequence[float]],
+ candidate: Sequence[Sequence[float]],
+ rng: random.Random,
+ options: Options,
+ family_size: int = 2,
+ sentry: Tuple[Optional[Dict[str, object]], Optional[Dict[str, object]]] = (
+ None,
+ None,
+ ),
+) -> Dict[str, Dict[str, object]]:
+ """p50 and p95 of each side, the bootstrap intervals, and a verdict each.
+
+ ``sentry`` holds each side's Sentry aggregate (percentiles over every
+ matching event); its relative change has to agree with the sample's.
+ """
+ base_values = sorted(value for cluster in baseline for value in cluster)
+ cand_values = sorted(value for cluster in candidate for value in cluster)
+ enough = sample_shortfall(baseline, candidate, options) is None
+ diffs = (
+ bootstrap_quantile_diffs(
+ baseline,
+ candidate,
+ [quantile for _, quantile in QUANTILES],
+ options.resamples,
+ rng,
+ )
+ if enough
+ else None
+ )
+ level = adjusted_confidence(options.confidence, family_size)
+ result: Dict[str, Dict[str, object]] = {}
+ for name, quantile in QUANTILES:
+ base = percentile(base_values, quantile) if base_values else None
+ cand = percentile(cand_values, quantile) if cand_values else None
+ diff = cand - base if base is not None and cand is not None else None
+ rel = relative(diff, base)
+ sentry_base = sentry[0].get(name) if sentry[0] else None
+ sentry_cand = sentry[1].get(name) if sentry[1] else None
+ sentry_rel = (
+ relative(float(sentry_cand) - float(sentry_base), float(sentry_base))
+ if sentry_base is not None and sentry_cand is not None
+ else None
+ )
+ interval: Optional[Tuple[float, float]] = None
+ adjusted: Optional[Tuple[float, float]] = None
+ verdict = NOT_ENOUGH_DATA
+ if diffs is not None:
+ interval = confidence_interval(diffs[quantile], options.confidence)
+ adjusted = confidence_interval(diffs[quantile], level)
+ verdict = quantile_verdict(
+ adjusted[0], adjusted[1], (rel, sentry_rel), options.min_effect
+ )
+ result[name] = {
+ "baseline": base,
+ "candidate": cand,
+ "diff": diff,
+ "rel": rel,
+ "sentry_rel": sentry_rel,
+ "ci": list(interval) if interval else None,
+ "ci_adjusted": list(adjusted) if adjusted else None,
+ "verdict": verdict,
+ }
+ return result
+
+
+def log_comb(total: int, chosen: int) -> float:
+ return (
+ math.lgamma(total + 1)
+ - math.lgamma(chosen + 1)
+ - math.lgamma(total - chosen + 1)
+ )
+
+
+def fisher_exact_p(
+ base_failures: int, base_total: int, cand_failures: int, cand_total: int
+) -> float:
+ """Two-sided Fisher exact p-value for the 2x2 failure table.
+
+ Sums the hypergeometric probability of every table with the same margins
+ that is no more likely than the observed one. Probabilities are computed
+ in log space; ties use the usual 1e-7 relative tolerance. The walk starts
+ at the most likely table and, since the distribution is unimodal, stops in
+ each direction once tables are e^-40 times less likely than the observed
+ one. Everything further out changes the result by under 1e-9 of itself,
+ so a table with millions of failures still takes milliseconds.
+ """
+ total = base_total + cand_total
+ failures = base_failures + cand_failures
+ low = max(0, cand_total - (total - failures))
+ high = min(failures, cand_total)
+
+ def log_probability(count: int) -> float:
+ return (
+ log_comb(failures, count)
+ + log_comb(total - failures, cand_total - count)
+ - log_comb(total, cand_total)
+ )
+
+ observed = log_probability(cand_failures) + math.log1p(1e-7)
+ cutoff = observed - 40
+ mode = (cand_total + 1) * (failures + 1) // (total + 2)
+ mode = min(high, max(low, mode))
+ p_value = 0.0
+ for counts in (range(mode, low - 1, -1), range(mode + 1, high + 1)):
+ for count in counts:
+ probability = log_probability(count)
+ if probability <= observed:
+ p_value += math.exp(probability)
+ if probability < cutoff:
+ break
+ return min(1.0, p_value)
+
+
+def compare_failure_rates(
+ baseline: Optional[Dict[str, object]],
+ candidate: Optional[Dict[str, object]],
+ options: Options,
+ family_size: int = 1,
+) -> Dict[str, object]:
+ """Fisher's exact test on each side's Sentry ``failure_rate()``.
+
+ Exact rather than a normal approximation, which is wrong with a handful
+ of failures (0 of 1000 against 4 of 1000 would pass a z-test). The
+ significance level is Bonferroni-adjusted over ``family_size`` tests.
+ """
+
+ def split(aggregate: Optional[Dict[str, object]]) -> Optional[Tuple[int, int]]:
+ if not aggregate or aggregate.get("failure_rate") is None:
+ return None
+ total = int(aggregate.get("count") or 0)
+ return total, int(round(float(aggregate["failure_rate"]) * total))
+
+ base, cand = split(baseline), split(candidate)
+ result: Dict[str, object] = {
+ "baseline": baseline.get("failure_rate") if baseline else None,
+ "candidate": candidate.get("failure_rate") if candidate else None,
+ "diff": None,
+ "p_value": None,
+ "verdict": NOT_ENOUGH_DATA,
+ }
+ if base is None or cand is None or min(base[0], cand[0]) < options.min_count:
+ return result
+ base_rate, cand_rate = base[1] / base[0], cand[1] / cand[0]
+ p_value = fisher_exact_p(base[1], base[0], cand[1], cand[0])
+ significant = p_value < (1 - options.confidence) / max(1, family_size)
+ if significant and cand_rate - base_rate >= options.min_failure_delta:
+ verdict = HIGHER
+ elif significant and base_rate - cand_rate >= options.min_failure_delta:
+ verdict = LOWER
+ else:
+ verdict = NO_CHANGE
+ result.update(
+ {
+ "baseline": base_rate,
+ "candidate": cand_rate,
+ "diff": cand_rate - base_rate,
+ "p_value": p_value,
+ "verdict": verdict,
+ }
+ )
+ return result
+
+
+def overall_outcome(results: Sequence[Dict[str, object]]) -> Dict[str, object]:
+ reasons: List[str] = []
+ for result in results:
+ for name, _ in QUANTILES:
+ stat = result[name]
+ if stat["verdict"] in (FASTER, SLOWER):
+ reasons.append(
+ f"{result['transaction']}: {name} {stat['verdict']} "
+ f"(sample {format_rel(stat['rel'])}, "
+ f"Sentry {format_rel(stat['sentry_rel'])})"
+ )
+ failure = result["failure_rate"]
+ if failure["verdict"] in (HIGHER, LOWER):
+ reasons.append(
+ f"{result['transaction']}: failure rate {failure['verdict']} "
+ f"({format_points(failure['diff'])})"
+ )
+ verdicts = [result["verdict"] for result in results]
+ failures = [result["failure_rate"]["verdict"] for result in results]
+ if SLOWER in verdicts:
+ verdict, code = SLOWER, EXIT_SLOWER
+ elif HIGHER in failures:
+ verdict, code = MORE_FAILURES, EXIT_SLOWER
+ elif FASTER in verdicts:
+ verdict, code = FASTER, EXIT_FASTER
+ elif NO_CHANGE in verdicts:
+ verdict, code = NO_CHANGE, EXIT_NO_CHANGE
+ else:
+ verdict, code = NOT_ENOUGH_DATA, EXIT_NOT_ENOUGH_DATA
+ return {"verdict": verdict, "exit_code": code, "reasons": reasons}
+
+
+# Comparison -----------------------------------------------------------------
+
+
+def summarize_aggregate(row: Dict[str, object], field_name: str) -> Dict[str, object]:
+ def number(key: str) -> Optional[float]:
+ value = row.get(key)
+ return float(value) if isinstance(value, (int, float)) else None
+
+ return {
+ "count": int(row.get("count()") or 0),
+ "p50": number(f"p50({field_name})"),
+ "p95": number(f"p95({field_name})"),
+ "p99": number(f"p99({field_name})"),
+ "failure_rate": number("failure_rate()"),
+ }
+
+
+def choose_transactions(
+ baseline: Dict[str, Dict[str, object]],
+ candidate: Dict[str, Dict[str, object]],
+ options: Options,
+) -> List[str]:
+ if options.transactions:
+ return list(dict.fromkeys(options.transactions))
+ shared = [name for name in baseline if name in candidate]
+ shared.sort(
+ key=lambda name: (
+ -min(int(baseline[name]["count"]), int(candidate[name]["count"])),
+ name,
+ )
+ )
+ return shared[: options.top]
+
+
+def to_clusters(
+ rows: Sequence[Dict[str, object]], field_name: str
+) -> List[List[float]]:
+ clusters: Dict[str, List[float]] = {}
+ for index, row in enumerate(rows):
+ value = row.get(field_name)
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ continue
+ if not math.isfinite(value):
+ continue
+ key = str(row.get("trace") or f"row-{index}")
+ clusters.setdefault(key, []).append(float(value))
+ return list(clusters.values())
+
+
+def fetch_sample(
+ client: SentryClient,
+ side: Side,
+ project_id: str,
+ field_name: str,
+ name: str,
+ cap: int,
+) -> List[Dict[str, object]]:
+ """The first ``cap`` matching spans in span-id order.
+
+ Span ids are random, so this is a uniform random sample of the side's
+ windows: it follows the traffic profile Sentry's own percentiles use and
+ is not one burst from one busy page, as "most recent N" would be.
+ """
+ params = events_params(
+ project_id,
+ build_query(side, field_name, [name]),
+ sample_fields(field_name),
+ SAMPLE_SORT,
+ side.windows,
+ min(PAGE_SIZE, cap),
+ )
+ return client.paginate(params, cap)
+
+
+def collect_samples(
+ client: SentryClient,
+ side: Side,
+ project_id: str,
+ options: Options,
+ names: Sequence[str],
+ pool: concurrent.futures.Executor,
+) -> Dict[str, List[List[float]]]:
+ futures = {
+ name: pool.submit(
+ fetch_sample,
+ client,
+ side,
+ project_id,
+ options.field,
+ name,
+ options.sample_cap,
+ )
+ for name in names
+ }
+ return {
+ name: to_clusters(future.result(), options.field)
+ for name, future in futures.items()
+ }
+
+
+def side_summary(
+ aggregate: Optional[Dict[str, object]], clusters: Sequence[Sequence[float]]
+) -> Dict[str, object]:
+ summary = dict(
+ aggregate
+ or {"count": 0, "p50": None, "p95": None, "p99": None, "failure_rate": None}
+ )
+ summary["sample_n"] = sum(len(cluster) for cluster in clusters)
+ summary["sample_traces"] = len(clusters)
+ return summary
+
+
+def compare_transaction(
+ name: str,
+ baseline: Optional[Dict[str, object]],
+ candidate: Optional[Dict[str, object]],
+ base_clusters: Sequence[Sequence[float]],
+ cand_clusters: Sequence[Sequence[float]],
+ options: Options,
+ quantile_family: int = 2,
+ failure_family: int = 1,
+) -> Dict[str, object]:
+ rng = random.Random(f"{options.seed}/{name}")
+ quantiles = compare_quantiles(
+ base_clusters,
+ cand_clusters,
+ rng,
+ options,
+ quantile_family,
+ (baseline, candidate),
+ )
+ failure = compare_failure_rates(baseline, candidate, options, failure_family)
+ verdict = combine_verdicts([stat["verdict"] for stat in quantiles.values()])
+ return {
+ "transaction": name,
+ "baseline": side_summary(baseline, base_clusters),
+ "candidate": side_summary(candidate, cand_clusters),
+ "p50": quantiles["p50"],
+ "p95": quantiles["p95"],
+ "failure_rate": failure,
+ "verdict": verdict,
+ "note": sample_shortfall(base_clusters, cand_clusters, options),
+ "regression": verdict == SLOWER or failure["verdict"] == HIGHER,
+ }
+
+
+def fetch_aggregates(
+ client: SentryClient, side: Side, project_id: str, options: Options
+) -> Tuple[Dict[str, Dict[str, object]], Optional[str]]:
+ params = events_params(
+ project_id,
+ build_query(side, options.field, options.transactions),
+ aggregate_fields(options.field),
+ "-count()",
+ side.windows,
+ PAGE_SIZE,
+ )
+ rows, meta, _ = client.events(params)
+ units = meta.get("units") if isinstance(meta.get("units"), dict) else {}
+ aggregates = {
+ str(row.get("transaction")): summarize_aggregate(row, options.field)
+ for row in rows
+ }
+ return aggregates, units.get(f"p50({options.field})")
+
+
+def eligible_transactions(
+ names: Sequence[str],
+ base_aggregates: Dict[str, Dict[str, object]],
+ cand_aggregates: Dict[str, Dict[str, object]],
+ options: Options,
+) -> List[str]:
+ def count(aggregates: Dict[str, Dict[str, object]], name: str) -> int:
+ return int(aggregates.get(name, {}).get("count") or 0)
+
+ return [
+ name
+ for name in names
+ if min(count(base_aggregates, name), count(cand_aggregates, name))
+ >= options.min_count
+ ]
+
+
+def compare(
+ client: SentryClient, baseline: Side, candidate: Side, options: Options
+) -> Dict[str, object]:
+ project_ids = {
+ side.label: client.project_id(side.project) for side in (baseline, candidate)
+ }
+ base_aggregates, base_unit = fetch_aggregates(
+ client, baseline, project_ids[baseline.label], options
+ )
+ cand_aggregates, cand_unit = fetch_aggregates(
+ client, candidate, project_ids[candidate.label], options
+ )
+ names = choose_transactions(base_aggregates, cand_aggregates, options)
+ eligible = eligible_transactions(names, base_aggregates, cand_aggregates, options)
+ samples: Dict[str, Dict[str, List[List[float]]]] = {}
+ with concurrent.futures.ThreadPoolExecutor(options.concurrency) as pool:
+ for side in (baseline, candidate):
+ samples[side.label] = collect_samples(
+ client, side, project_ids[side.label], options, eligible, pool
+ )
+ return build_report(
+ baseline,
+ candidate,
+ options,
+ project_ids,
+ (base_aggregates, cand_aggregates),
+ (samples[baseline.label], samples[candidate.label]),
+ base_unit or cand_unit,
+ client.requests,
+ )
+
+
+def build_report(
+ baseline: Side,
+ candidate: Side,
+ options: Options,
+ project_ids: Dict[str, str],
+ aggregates: Tuple[Dict[str, Dict[str, object]], Dict[str, Dict[str, object]]],
+ samples: Tuple[Dict[str, List[List[float]]], Dict[str, List[List[float]]]],
+ unit: Optional[str],
+ requests: int,
+) -> Dict[str, object]:
+ """Turn fetched aggregates and samples into the report; no I/O."""
+ base_aggregates, cand_aggregates = aggregates
+ base_samples, cand_samples = samples
+ names = choose_transactions(base_aggregates, cand_aggregates, options)
+ sampled = [
+ name
+ for name in names
+ if sample_shortfall(
+ base_samples.get(name, []), cand_samples.get(name, []), options
+ )
+ is None
+ ]
+ quantile_family = len(QUANTILES) * len(sampled)
+ failure_family = len(
+ eligible_transactions(names, base_aggregates, cand_aggregates, options)
+ )
+ level = adjusted_confidence(options.confidence, quantile_family)
+ run_options = replace(
+ options, resamples=effective_resamples(options.resamples, level)
+ )
+ results = [
+ compare_transaction(
+ name,
+ base_aggregates.get(name),
+ cand_aggregates.get(name),
+ base_samples.get(name, []),
+ cand_samples.get(name, []),
+ run_options,
+ quantile_family,
+ failure_family,
+ )
+ for name in names
+ ]
+
+ def counts(name: str) -> Tuple[int, int]:
+ return (
+ int(base_aggregates.get(name, {}).get("count") or 0),
+ int(cand_aggregates.get(name, {}).get("count") or 0),
+ )
+
+ overview_names = sorted(
+ set(base_aggregates) | set(cand_aggregates),
+ key=lambda name: (-min(counts(name)), -sum(counts(name)), name),
+ )
+ return {
+ "schema_version": SCHEMA_VERSION,
+ "field": options.field,
+ "unit": unit,
+ "baseline": dict(
+ baseline.selector(),
+ project_id=project_ids.get(baseline.label),
+ aggregate_query=build_query(baseline, options.field, options.transactions),
+ ),
+ "candidate": dict(
+ candidate.selector(),
+ project_id=project_ids.get(candidate.label),
+ aggregate_query=build_query(candidate, options.field, options.transactions),
+ ),
+ "settings": {
+ "transactions": options.transactions or None,
+ "top": options.top,
+ "sample_cap": options.sample_cap,
+ "sample_sort": SAMPLE_SORT,
+ "min_count": options.min_count,
+ "min_traces": options.min_traces,
+ "min_effect": options.min_effect,
+ "min_failure_delta": options.min_failure_delta,
+ "confidence": options.confidence,
+ "comparisons": quantile_family,
+ "failure_comparisons": failure_family,
+ "adjusted_confidence": level,
+ "resamples": run_options.resamples,
+ "seed": options.seed,
+ },
+ "overview": [
+ {
+ "transaction": name,
+ "baseline": base_aggregates.get(name),
+ "candidate": cand_aggregates.get(name),
+ }
+ for name in overview_names
+ ],
+ "transactions": results,
+ "overall": overall_outcome(results),
+ "requests": requests,
+ }
+
+
+# Rendering ------------------------------------------------------------------
+
+
+def format_number(value: Optional[float]) -> str:
+ if value is None:
+ return "n/a"
+ size = abs(value)
+ if size >= 1000:
+ return f"{value:,.0f}"
+ if size >= 100:
+ return f"{value:.0f}"
+ if size >= 10:
+ return f"{value:.1f}"
+ if size >= 1:
+ return f"{value:.2f}"
+ return f"{value:.3g}"
+
+
+def format_signed(value: Optional[float]) -> str:
+ if value is None:
+ return "n/a"
+ text = format_number(value)
+ return text if text.startswith("-") else "+" + text
+
+
+def format_rel(value: Optional[float]) -> str:
+ return "n/a" if value is None else f"{value * 100:+.1f}%"
+
+
+def format_points(value: Optional[float]) -> str:
+ return "n/a" if value is None else f"{value * 100:+.2f} pp"
+
+
+def format_percent(value: Optional[float]) -> str:
+ return "n/a" if value is None else f"{value * 100:.2f}%"
+
+
+def format_change(stat: Dict[str, object], suffix: str) -> str:
+ if stat["diff"] is None:
+ return "n/a"
+ return (
+ f"{format_signed(stat['diff'])}{suffix} ({format_rel(stat['rel'])}); "
+ f"Sentry {format_rel(stat['sentry_rel'])}"
+ )
+
+
+def format_interval(stat: Dict[str, object], suffix: str) -> str:
+ if not stat["ci_adjusted"]:
+ return "n/a"
+ low, high = stat["ci_adjusted"]
+ return f"[{format_signed(low)}, {format_signed(high)}]{suffix}"
+
+
+def format_level(level: float) -> str:
+ return f"{level * 100:.3g}%"
+
+
+def cell(text: object) -> str:
+ return str(text).replace("|", "\\|").replace("\n", " ")
+
+
+def describe_side(selector: Dict[str, object]) -> str:
+ windows = ", ".join(f"{pair[0]} to {pair[1]}" for pair in selector["windows"])
+ parts = [
+ f"project `{selector['project']}`",
+ f"env `{selector['environment'] or 'any'}`",
+ f"release `{selector['release'] or 'any'}`",
+ ]
+ if selector.get("query"):
+ parts.append(f"query `{selector['query']}`")
+ return ", ".join(parts) + f"; windows {windows}"
+
+
+def render_markdown(report: Dict[str, object]) -> str:
+ unit = UNIT_SUFFIX.get(str(report.get("unit")), "")
+ suffix = f" {unit}" if unit else ""
+ settings = report["settings"]
+ lines = [
+ f"## perf-compare: `{report['field']}`" + (f" ({unit})" if unit else ""),
+ "",
+ f"- **Baseline:** {describe_side(report['baseline'])}",
+ f"- **Candidate:** {describe_side(report['candidate'])}",
+ "",
+ "### Sentry aggregates (every matching event)",
+ "",
+ "| transaction | n base | n cand | p50 base | p50 cand | p95 base "
+ "| p95 cand | p99 base | p99 cand | fail base | fail cand |",
+ "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|",
+ ]
+ empty: Dict[str, object] = {}
+ for row in report["overview"][:OVERVIEW_ROWS]:
+ base = row["baseline"] or empty
+ cand = row["candidate"] or empty
+ lines.append(
+ "| "
+ + " | ".join(
+ cell(value)
+ for value in (
+ row["transaction"],
+ base.get("count", 0),
+ cand.get("count", 0),
+ format_number(base.get("p50")),
+ format_number(cand.get("p50")),
+ format_number(base.get("p95")),
+ format_number(cand.get("p95")),
+ format_number(base.get("p99")),
+ format_number(cand.get("p99")),
+ format_percent(base.get("failure_rate")),
+ format_percent(cand.get("failure_rate")),
+ )
+ )
+ + " |"
+ )
+ hidden = len(report["overview"]) - OVERVIEW_ROWS
+ if hidden > 0:
+ lines.append(f"\n{hidden} more transactions in `--json` output.")
+ confidence = format_level(float(settings["confidence"]))
+ adjusted = format_level(float(settings["adjusted_confidence"]))
+ lines += [
+ "",
+ "### Verdicts (candidate minus baseline)",
+ "",
+ f"Intervals are {adjusted} bootstrap CIs: {confidence} held jointly over "
+ f"the {settings['comparisons']} percentile comparisons below "
+ "(Bonferroni). Changes are from the sample, then from Sentry's "
+ "percentiles over every event.",
+ "",
+ "| transaction | samples base / cand | p50 change | p50 CI | p95 change "
+ "| p95 CI | failure rate change | verdict |",
+ "|---|---:|---:|---:|---:|---:|---:|---|",
+ ]
+ for result in report["transactions"]:
+ base, cand = result["baseline"], result["candidate"]
+ failure = result["failure_rate"]
+ failure_text = format_points(failure["diff"])
+ if failure["verdict"] in (HIGHER, LOWER):
+ failure_text += f" ({failure['verdict']})"
+ verdict = f"**{result['verdict']}**"
+ if failure["verdict"] == HIGHER:
+ verdict = f"**{result['verdict']}, more failures**"
+ if result.get("note"):
+ verdict += f" ({result['note']})"
+ lines.append(
+ "| "
+ + " | ".join(
+ cell(value)
+ for value in (
+ result["transaction"],
+ f"{base['sample_n']} / {cand['sample_n']} "
+ f"({base['sample_traces']} / {cand['sample_traces']} traces)",
+ format_change(result["p50"], suffix),
+ format_interval(result["p50"], suffix),
+ format_change(result["p95"], suffix),
+ format_interval(result["p95"], suffix),
+ failure_text,
+ verdict,
+ )
+ )
+ + " |"
+ )
+ overall = report["overall"]
+ lines += [
+ "",
+ f"**Overall: {overall['verdict']}** (exit {overall['exit_code']})",
+ ]
+ lines += [f"- {reason}" for reason in overall["reasons"]]
+ lines += [
+ "",
+ f"Rules: a side needs {settings['min_count']} samples from "
+ f"{settings['min_traces']} traces. Faster or slower needs the CI to "
+ "exclude zero, and the sample change and Sentry's change to both be at "
+ f"least {float(settings['min_effect']) * 100:g}% in that direction. "
+ "Failure rates use Fisher's exact test over "
+ f"{settings['failure_comparisons']} transactions (Bonferroni). Up to "
+ f"{settings['sample_cap']} samples per side in span-id order, resampled "
+ f"by trace {settings['resamples']} times with seed {settings['seed']}. "
+ f"{report['requests']} Sentry requests.",
+ "",
+ ]
+ return "\n".join(lines)
+
+
+def render_queries(
+ org: str,
+ region_url: str,
+ baseline: Side,
+ candidate: Side,
+ options: Options,
+ now: datetime,
+) -> str:
+ events_url = "{}/api/0/organizations/{}/events/".format(region_url.rstrip("/"), org)
+ names = options.transactions or ["<TRANSACTION>"]
+ lines = [
+ "# Queries perf_compare.py sends (Events API, dataset=spans)",
+ "",
+ "REST needs the numeric project id (GET /api/0/projects/{org}/{slug}/ "
+ "returns it). The MCP takes the slug. The query string is identical for "
+ "both, and its timestamp filter pins the exact windows, so the MCP "
+ "`period` only has to cover them.",
+ "",
+ "Raw samples are sorted by span `id`, which is random, so the first N "
+ "rows are a uniform random sample of the windows. REST pages through "
+ "them with the `Link` cursor. The MCP stops at 100 rows per call: for a "
+ "bigger sample, cut the window into sub-windows (narrow the timestamp "
+ "filter), get count() for each, and take the first rows by `id` from "
+ "each sub-window in proportion to its count.",
+ ]
+ for side in (baseline, candidate):
+ plans = [
+ (
+ "aggregate per transaction",
+ build_query(side, options.field, options.transactions),
+ aggregate_fields(options.field),
+ "-count()",
+ PAGE_SIZE,
+ )
+ ]
+ plans.extend(
+ (
+ f"raw sample for {name} (first {options.sample_cap} rows by id)",
+ build_query(side, options.field, [name]),
+ sample_fields(options.field),
+ SAMPLE_SORT,
+ PAGE_SIZE,
+ )
+ for name in names
+ )
+ for title, query, fields, sort, limit in plans:
+ params = events_params(
+ "PROJECT_ID", query, fields, sort, side.windows, limit
+ )
+ lines += [
+ "",
+ f"## {side.label}: {title}",
+ "",
+ "REST:",
+ "",
+ f" GET {events_url}?{urllib.parse.urlencode(params)}",
+ "",
+ f"(PROJECT_ID = id of `{side.project}`)",
+ "",
+ "MCP `search_events` arguments (or `search_traces` without "
+ "`dataset`):",
+ "",
+ json.dumps(
+ mcp_arguments(
+ org, region_url, side, query, fields, sort, limit, now
+ ),
+ indent=2,
+ ),
+ ]
+ return "\n".join(lines) + "\n"
+
+
+# Command line ---------------------------------------------------------------
+
+
+def build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ prog="perf_compare.py",
+ description=(
+ "Compare a numeric span field (duration by default) between a "
+ "baseline and a candidate selector in Sentry, per transaction, and "
+ "say whether the candidate is faster, slower, unchanged, or "
+ "under-sampled."
+ ),
+ epilog=EXIT_CODES_HELP
+ + "\nselector keys for --baseline/--candidate: "
+ + ", ".join(SIDE_KEYS)
+ + "\nexample:\n perf_compare.py --project builder --env app:prod-behave:cloud "
+ "\\\n --baseline release=OLD_SHA --candidate release=NEW_SHA --period 7d",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ selectors = parser.add_argument_group("selectors")
+ selectors.add_argument(
+ "--baseline",
+ nargs="+",
+ required=True,
+ metavar="KEY=VALUE",
+ help="baseline selector, e.g. release=SHA env=pr-123 window=START/END",
+ )
+ selectors.add_argument(
+ "--candidate",
+ nargs="+",
+ required=True,
+ metavar="KEY=VALUE",
+ help="candidate selector, same keys as --baseline",
+ )
+ selectors.add_argument("--project", help="project slug used by both sides")
+ selectors.add_argument(
+ "--env", help="environment used by both sides, e.g. app:prod-behave:cloud"
+ )
+ selectors.add_argument(
+ "--query",
+ help="extra Sentry search filter for both sides, e.g. span.op:pageload",
+ )
+ timing = parser.add_argument_group(
+ "time (both sides; a side's own window=, start=/end= or period= wins)"
+ )
+ timing.add_argument(
+ "--period", help=f"window ending now, e.g. 24h (default {DEFAULT_PERIOD})"
+ )
+ timing.add_argument("--start", help="window start, ISO 8601 UTC")
+ timing.add_argument("--end", help="window end, ISO 8601 UTC (default now)")
+ timing.add_argument(
+ "--window",
+ action="append",
+ default=[],
+ help="START/END in UTC; repeat for interleaved runs",
+ )
+ what = parser.add_argument_group("what to compare")
+ what.add_argument(
+ "--transaction",
+ action="append",
+ default=[],
+ help="transaction name to compare; repeat for several "
+ "(default: the --top busiest transactions on both sides)",
+ )
+ what.add_argument(
+ "--top",
+ type=int,
+ default=5,
+ help="without --transaction, compare this many of the busiest "
+ "transactions present on both sides (default 5)",
+ )
+ what.add_argument(
+ "--field",
+ default=DEFAULT_FIELD,
+ help="numeric field, e.g. span.duration, measurements.lcp, "
+ "gen_ai.usage.input_tokens, tags[my.attribute,number]",
+ )
+ stats = parser.add_argument_group("statistics")
+ stats.add_argument(
+ "--sample-cap",
+ type=int,
+ default=1000,
+ help="raw values fetched per side per transaction, a random sample in "
+ f"span-id order (default 1000, at most {MAX_SAMPLE_CAP})",
+ )
+ stats.add_argument(
+ "--min-count",
+ type=int,
+ default=200,
+ help="samples each side needs before any verdict (default 200)",
+ )
+ stats.add_argument(
+ "--min-traces",
+ type=int,
+ default=30,
+ help="distinct traces each side's samples must come from before any "
+ "verdict (default 30)",
+ )
+ stats.add_argument(
+ "--min-effect",
+ type=float,
+ default=0.05,
+ help="smallest relative change that can be called faster or slower "
+ "(default 0.05 = 5%%)",
+ )
+ stats.add_argument(
+ "--min-failure-delta",
+ type=float,
+ default=0.001,
+ help="smallest failure-rate change, as a fraction, that counts "
+ "(default 0.001 = 0.1 pp)",
+ )
+ stats.add_argument(
+ "--confidence",
+ type=float,
+ default=0.95,
+ help="confidence held jointly over every percentile compared in the "
+ "run; each interval is Bonferroni-adjusted (default 0.95)",
+ )
+ stats.add_argument(
+ "--resamples", type=int, default=4000, help="bootstrap resamples (default 4000)"
+ )
+ stats.add_argument(
+ "--seed", type=int, default=1234, help="bootstrap seed (default 1234)"
+ )
+ output = parser.add_argument_group("output and connection")
+ output.add_argument("--json", action="store_true", help="machine-readable output")
+ output.add_argument(
+ "--print-queries",
+ action="store_true",
+ help="print the REST and MCP queries without calling Sentry",
+ )
+ output.add_argument("--org", default=DEFAULT_ORG, help=f"default {DEFAULT_ORG}")
+ output.add_argument(
+ "--region-url", default=DEFAULT_REGION_URL, help=f"default {DEFAULT_REGION_URL}"
+ )
+ output.add_argument(
+ "--concurrency",
+ type=int,
+ default=4,
+ help="parallel Sentry requests (default 4)",
+ )
+ output.add_argument(
+ "--timeout", type=float, default=30.0, help="seconds per request (default 30)"
+ )
+ return parser
+
+
+def check_region_url(url: str) -> None:
+ """https with a host, or http only for a server on this machine."""
+ parts = urllib.parse.urlsplit(url)
+ secure = parts.scheme == "https" and bool(parts.hostname)
+ local = parts.scheme == "http" and parts.hostname in LOCAL_HOSTS
+ bare = parts.path in ("", "/") and not parts.query and not parts.fragment
+ if not ((secure or local) and bare):
+ raise UsageError(
+ f"bad --region-url {url!r}: use the scheme and host only, such as "
+ f"{DEFAULT_REGION_URL} (http is allowed only for localhost)"
+ )
+
+
+def build_options(args: argparse.Namespace) -> Options:
+ checks = (
+ (args.top >= 1, "--top must be at least 1"),
+ (args.sample_cap >= 1, "--sample-cap must be at least 1"),
+ (args.min_count >= 1, "--min-count must be at least 1"),
+ (args.min_traces >= 1, "--min-traces must be at least 1"),
+ (args.min_effect >= 0, "--min-effect cannot be negative"),
+ (args.min_failure_delta >= 0, "--min-failure-delta cannot be negative"),
+ (0 < args.confidence < 1, "--confidence must be between 0 and 1"),
+ (args.resamples >= 100, "--resamples must be at least 100"),
+ (args.concurrency >= 1, "--concurrency must be at least 1"),
+ (args.timeout > 0, "--timeout must be more than 0 seconds"),
+ (bool(FIELD_PATTERN.match(args.field)), f"bad --field {args.field!r}"),
+ )
+ for passed, message in checks:
+ if not passed:
+ raise UsageError(message)
+ check_region_url(args.region_url)
+ sample_cap = min(args.sample_cap, MAX_SAMPLE_CAP)
+ if sample_cap < args.min_count:
+ raise UsageError(
+ f"--sample-cap {sample_cap} is below --min-count {args.min_count}, "
+ "so no transaction could get a verdict"
+ )
+ most = len(QUANTILES) * (len(set(args.transaction)) or args.top)
+ level = adjusted_confidence(args.confidence, most)
+ if RESAMPLE_CAP * tail_share(level) < MIN_TAIL_DRAWS:
+ raise UsageError(
+ f"{most} percentile comparisons need more than {RESAMPLE_CAP} "
+ "bootstrap resamples; compare fewer transactions"
+ )
+ return Options(
+ field=args.field,
+ transactions=list(args.transaction),
+ top=args.top,
+ sample_cap=sample_cap,
+ min_count=args.min_count,
+ min_traces=args.min_traces,
+ min_effect=args.min_effect,
+ min_failure_delta=args.min_failure_delta,
+ confidence=args.confidence,
+ resamples=args.resamples,
+ seed=args.seed,
+ concurrency=args.concurrency,
+ )
+
+
+def build_sides(args: argparse.Namespace, now: datetime) -> Tuple[Side, Side]:
+ shared_windows = resolve_windows(
+ "shared time", args.window, args.start, args.end, args.period, now
+ ) or [Window(now - parse_period(DEFAULT_PERIOD), now)]
+ shared = Shared(args.project, args.env, args.query, shared_windows)
+ baseline = parse_side("baseline", args.baseline, shared, now)
+ candidate = parse_side("candidate", args.candidate, shared, now)
+ if baseline.selector() == candidate.selector():
+ raise UsageError(
+ "baseline and candidate select the same events; change a release, "
+ "environment, project, query or window on one side"
+ )
+ return baseline, candidate
+
+
+def main(
+ argv: Optional[Sequence[str]] = None,
+ environ: Optional[Dict[str, str]] = None,
+ transport: Optional[Transport] = None,
+ now: Optional[datetime] = None,
+ stdout=None,
+ stderr=None,
+ sleep: Callable[[float], None] = time.sleep,
+) -> int:
+ environ = os.environ if environ is None else environ
+ stdout = stdout or sys.stdout
+ stderr = stderr or sys.stderr
+ args = build_parser().parse_args(argv)
+
+ def fail(kind: str, message: str, code: int) -> int:
+ stderr.write(message if message.endswith("\n") else message + "\n")
+ if args.json:
+ error = {"type": kind, "message": message.strip()}
+ stdout.write(
+ json.dumps(
+ {
+ "schema_version": SCHEMA_VERSION,
+ "error": error,
+ "exit_code": code,
+ }
+ )
+ + "\n"
+ )
+ return code
+
+ try:
+ moment = (now or datetime.now(timezone.utc)).replace(microsecond=0)
+ options = build_options(args)
+ if options.sample_cap < args.sample_cap:
+ stderr.write(
+ f"perf-compare: --sample-cap is limited to {MAX_SAMPLE_CAP}; "
+ f"using {options.sample_cap}\n"
+ )
+ baseline, candidate = build_sides(args, moment)
+ if args.print_queries:
+ stdout.write(
+ render_queries(
+ args.org, args.region_url, baseline, candidate, options, moment
+ )
+ )
+ return 0
+ token = (environ.get("SENTRY_AUTH_TOKEN") or "").strip()
+ if not token:
+ return fail("auth", MISSING_TOKEN_MESSAGE, EXIT_AUTH)
+ client = SentryClient(
+ token,
+ args.org,
+ args.region_url,
+ transport=transport,
+ timeout=args.timeout,
+ sleep=sleep,
+ )
+ report = compare(client, baseline, candidate, options)
+ except PerfCompareError as error:
+ return fail(error.kind, f"perf-compare: {error}", error.exit_code)
+ except Exception as error:
+ return fail(
+ "internal",
+ f"perf-compare: unexpected {type(error).__name__}: {error}",
+ EXIT_API,
+ )
+ report["generated_at"] = format_filter_time(moment)
+ report["org"] = args.org
+ if args.json:
+ stdout.write(json.dumps(report, indent=2) + "\n")
+ else:
+ stdout.write(render_markdown(report))
+ return int(report["overall"]["exit_code"])
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.claude/skills/perf-compare/scripts/test_perf_compare.py b/.claude/skills/perf-compare/scripts/test_perf_compare.py
new file mode 100644
index 0000000000..c97a21da94
--- /dev/null
+++ b/.claude/skills/perf-compare/scripts/test_perf_compare.py
@@ -0,0 +1,1175 @@
+"""Offline tests for perf_compare.py.
+
+Run from this directory with ``python3 -m unittest``. Sentry is replaced by a
+fake transport, or by a local ``http.server``, that serves the synthetic
+responses in ``fixtures/``.
+"""
+
+import contextlib
+import http.server
+import io
+import json
+import math
+import os
+import random
+import socket
+import sys
+import threading
+import unittest
+import urllib.parse
+from datetime import datetime, timedelta, timezone
+from fractions import Fraction
+from pathlib import Path
+from unittest import mock
+
+HERE = Path(__file__).resolve().parent
+if str(HERE) not in sys.path:
+ sys.path.insert(0, str(HERE))
+
+import perf_compare as pc # noqa: E402
+
+FIXTURES = HERE / "fixtures"
+NOW = datetime(2026, 10, 12, 15, 0, tzinfo=timezone.utc)
+BASE_SHA = "a" * 40
+CAND_SHA = "b" * 40
+WINDOW_B = "2026-10-10T14:00:00Z/2026-10-11T14:00:00Z"
+WINDOW_C = "2026-10-11T14:00:00Z/2026-10-12T14:00:00Z"
+EVENTS_PATH = "/api/0/organizations/significant-gravitas/events/"
+PROJECT_PATH = "/api/0/projects/significant-gravitas/builder/"
+LIBRARY = "GET /api/library/agents"
+STORE = "GET /api/store/agents"
+
+
+def fixture(name):
+ return json.loads((FIXTURES / name).read_text())
+
+
+def respond(name, status=200):
+ data = fixture(name)
+ return status, data["headers"], json.dumps(data["body"]).encode()
+
+
+class FakeTransport:
+ """Answers through ``route`` and records every request."""
+
+ def __init__(self, route):
+ self.route = route
+ self.calls = []
+
+ def __call__(self, url, headers, timeout):
+ parts = urllib.parse.urlsplit(url)
+ params = urllib.parse.parse_qs(parts.query, keep_blank_values=True)
+ self.calls.append({"path": parts.path, "params": params, "headers": headers})
+ return self.route(parts.path, params)
+
+
+def sentry_route(candidate_data="candidate"):
+ """Route like Sentry: project lookup, aggregates, then paged samples."""
+
+ def route(path, params):
+ if path == PROJECT_PATH:
+ return respond("project_builder.json")
+ if path != EVENTS_PATH:
+ return 404, {}, b'{"detail": "not found"}'
+ query = params["query"][0]
+ side = "baseline" if f'release:"{BASE_SHA}"' in query else candidate_data
+ if "count()" in params.get("field", []):
+ return respond(f"aggregate_{side}.json")
+ page = 2 if params.get("cursor") == ["0:100:0"] else 1
+ return respond(f"samples_{side}_page{page}.json")
+
+ return route
+
+
+def side(**overrides):
+ values = dict(
+ label="baseline",
+ project="builder",
+ environment="app:prod-behave:cloud",
+ release=BASE_SHA,
+ query=None,
+ windows=[pc.parse_window(WINDOW_B)],
+ )
+ values.update(overrides)
+ return pc.Side(**values)
+
+
+def run_main(argv, environ=None, transport=None, sleep=None):
+ stdout, stderr = io.StringIO(), io.StringIO()
+ code = pc.main(
+ argv,
+ environ={"SENTRY_AUTH_TOKEN": "test-token"} if environ is None else environ,
+ transport=transport,
+ now=NOW,
+ stdout=stdout,
+ stderr=stderr,
+ sleep=sleep or (lambda seconds: None),
+ )
+ return code, stdout.getvalue(), stderr.getvalue()
+
+
+def exact_fisher(base_failures, base_total, cand_failures, cand_total):
+ """Two-sided Fisher p-value in exact integer arithmetic."""
+ total = base_total + cand_total
+ failures = base_failures + cand_failures
+
+ def weight(count):
+ return math.comb(failures, count) * math.comb(
+ total - failures, cand_total - count
+ )
+
+ observed = weight(cand_failures)
+ low = max(0, cand_total - (total - failures))
+ high = min(failures, cand_total)
+ tail = sum(weight(c) for c in range(low, high + 1) if weight(c) <= observed)
+ return float(Fraction(tail, math.comb(total, cand_total)))
+
+
+COMPARE_ARGS = [
+ "--project",
+ "builder",
+ "--env",
+ "app:prod-behave:cloud",
+ "--baseline",
+ f"release={BASE_SHA}",
+ f"window={WINDOW_B}",
+ "--candidate",
+ f"release={CAND_SHA}",
+ f"window={WINDOW_C}",
+ "--min-count",
+ "150",
+ "--resamples",
+ "500",
+]
+
+
+class QueryBuildingTest(unittest.TestCase):
+ def test_release_side_query_has_every_filter_and_the_window(self):
+ self.assertEqual(
+ pc.build_query(side(), "span.duration"),
+ 'is_transaction:true environment:"app:prod-behave:cloud" '
+ f'release:"{BASE_SHA}" '
+ "timestamp:>=2026-10-10T14:00:00Z timestamp:<2026-10-11T14:00:00Z",
+ )
+
+ def test_non_duration_field_requires_the_field_to_exist(self):
+ query = pc.build_query(side(environment=None, release=None), "measurements.lcp")
+ self.assertTrue(query.startswith("is_transaction:true has:measurements.lcp "))
+
+ def test_extra_query_is_grouped_so_an_or_cannot_escape(self):
+ query = pc.build_query(
+ side(query="span.op:pageload OR span.op:navigation"), "span.duration"
+ )
+ self.assertIn("(span.op:pageload OR span.op:navigation)", query)
+
+ def test_several_transactions_become_one_or_group_with_escaping(self):
+ query = pc.build_query(side(), "span.duration", [LIBRARY, 'say "hi" \\ bye'])
+ self.assertIn(
+ f'(transaction:"{LIBRARY}" OR transaction:"say \\"hi\\" \\\\ bye")', query
+ )
+
+ def test_interleaved_windows_become_one_or_group(self):
+ windows = [
+ pc.parse_window("2026-10-11T10:00:00Z/2026-10-11T10:10:00Z"),
+ pc.parse_window("2026-10-11T10:20:00Z/2026-10-11T10:30:00Z"),
+ ]
+ self.assertTrue(
+ pc.build_query(side(windows=windows), "span.duration").endswith(
+ "(timestamp:>=2026-10-11T10:00:00Z timestamp:<2026-10-11T10:10:00Z "
+ "OR timestamp:>=2026-10-11T10:20:00Z timestamp:<2026-10-11T10:30:00Z)"
+ )
+ )
+
+ def test_events_params_cover_the_union_of_windows(self):
+ windows = [
+ pc.parse_window("2026-10-11T10:00:00Z/2026-10-11T10:10:00Z"),
+ pc.parse_window("2026-10-11T10:20:00Z/2026-10-11T10:30:00Z"),
+ ]
+ params = pc.events_params(
+ "42", "q", pc.aggregate_fields("span.duration"), "-count()", windows, 100
+ )
+ self.assertEqual(
+ params,
+ [
+ ("dataset", "spans"),
+ ("project", "42"),
+ ("field", "transaction"),
+ ("field", "count()"),
+ ("field", "p50(span.duration)"),
+ ("field", "p95(span.duration)"),
+ ("field", "p99(span.duration)"),
+ ("field", "failure_rate()"),
+ ("query", "q"),
+ ("sort", "-count()"),
+ ("per_page", "100"),
+ ("start", "2026-10-11T10:00:00"),
+ ("end", "2026-10-11T10:30:00"),
+ ],
+ )
+
+ def test_samples_are_sorted_by_span_id(self):
+ self.assertEqual(pc.SAMPLE_SORT, "id")
+ self.assertEqual(
+ pc.sample_fields("measurements.lcp"), ["measurements.lcp", "trace", "id"]
+ )
+
+ def test_mcp_arguments_use_a_period_that_covers_the_window(self):
+ arguments = pc.mcp_arguments(
+ "org",
+ "https://us.sentry.io",
+ side(),
+ "q",
+ ["count()"],
+ "-count()",
+ 100,
+ NOW,
+ )
+ self.assertEqual(arguments["period"], "50h")
+ self.assertEqual(arguments["projectSlug"], "builder")
+ self.assertEqual(arguments["dataset"], "spans")
+
+
+class TimeTest(unittest.TestCase):
+ def test_parse_time_accepts_utc_offsets_naive_and_date_only(self):
+ expected = datetime(2026, 10, 11, 14, 0, tzinfo=timezone.utc)
+ for text in (
+ "2026-10-11T14:00:00Z",
+ "2026-10-11T14:00:00z",
+ "2026-10-11T16:00:00+02:00",
+ "2026-10-11T09:00:00-05:00",
+ "2026-10-11T14:00:00",
+ "2026-10-11T14:00:00.987654Z",
+ ):
+ with self.subTest(text=text):
+ self.assertEqual(pc.parse_time(text), expected)
+ self.assertEqual(
+ pc.parse_time("2026-10-11"), datetime(2026, 10, 11, tzinfo=timezone.utc)
+ )
+
+ def test_parse_time_rejects_junk(self):
+ for text in ("yesterday", "", "2026-13-01T00:00:00Z", "14:00"):
+ with self.subTest(text=text), self.assertRaises(pc.UsageError):
+ pc.parse_time(text)
+
+ def parse(self, *extra):
+ return pc.build_parser().parse_args(
+ ["--project", "builder", "--baseline", "release=a"]
+ + ["--candidate", "release=b", *extra]
+ )
+
+ def test_shared_start_and_end_apply_to_both_sides(self):
+ baseline, candidate = pc.build_sides(
+ self.parse(
+ "--start", "2026-10-11T10:00:00Z", "--end", "2026-10-11T12:00:00Z"
+ ),
+ NOW,
+ )
+ window = pc.parse_window("2026-10-11T10:00:00Z/2026-10-11T12:00:00Z")
+ self.assertEqual(baseline.windows, [window])
+ self.assertEqual(candidate.windows, [window])
+
+ def test_shared_start_alone_runs_until_now(self):
+ baseline, _ = pc.build_sides(self.parse("--start", "2026-10-12T09:00:00Z"), NOW)
+ self.assertEqual(
+ baseline.windows,
+ [pc.Window(datetime(2026, 10, 12, 9, tzinfo=timezone.utc), NOW)],
+ )
+
+ def test_a_side_window_replaces_all_shared_time_flags(self):
+ args = pc.build_parser().parse_args(
+ ["--project", "builder", "--period", "24h"]
+ + ["--baseline", "release=a", "period=1h", "--candidate", "release=b"]
+ )
+ baseline, candidate = pc.build_sides(args, NOW)
+ self.assertEqual(baseline.windows, [pc.Window(NOW - timedelta(hours=1), NOW)])
+ self.assertEqual(candidate.windows, [pc.Window(NOW - timedelta(hours=24), NOW)])
+
+ def test_mixing_shared_time_flags_is_a_usage_error(self):
+ with self.assertRaises(pc.UsageError):
+ pc.build_sides(self.parse("--period", "24h", "--start", "2026-10-11"), NOW)
+
+ def test_default_window_is_the_last_seven_days(self):
+ baseline, _ = pc.build_sides(self.parse(), NOW)
+ self.assertEqual(baseline.windows, [pc.Window(NOW - timedelta(days=7), NOW)])
+
+
+class SelectorTest(unittest.TestCase):
+ def shared(self, **overrides):
+ values = dict(
+ project="builder",
+ env="app:prod-behave:cloud",
+ query=None,
+ windows=[pc.Window(NOW - timedelta(days=7), NOW)],
+ )
+ values.update(overrides)
+ return pc.Shared(**values)
+
+ def test_side_keys_override_shared_defaults(self):
+ parsed = pc.parse_side(
+ "candidate",
+ ["environment=pr-123", "release=abc", "period=24h"],
+ self.shared(),
+ NOW,
+ )
+ self.assertEqual(parsed.environment, "pr-123")
+ self.assertEqual(parsed.release, "abc")
+ self.assertEqual(parsed.windows, [pc.Window(NOW - timedelta(hours=24), NOW)])
+
+ def test_empty_env_clears_the_shared_environment(self):
+ parsed = pc.parse_side("baseline", ["env="], self.shared(), NOW)
+ self.assertIsNone(parsed.environment)
+
+ def test_unknown_or_repeated_keys_are_usage_errors(self):
+ for tokens in (["colour=red"], ["release"], ["release=a", "release=b"]):
+ with self.subTest(tokens=tokens), self.assertRaises(pc.UsageError):
+ pc.parse_side("baseline", tokens, self.shared(), NOW)
+
+ def test_missing_project_is_a_usage_error(self):
+ with self.assertRaises(pc.UsageError):
+ pc.parse_side("baseline", ["release=a"], self.shared(project=None), NOW)
+
+ def test_bad_windows_are_usage_errors(self):
+ cases = (
+ ["window=2026-10-11T10:00:00Z/2026-10-11T09:00:00Z"],
+ [
+ "window=2026-10-11T10:00:00Z/2026-10-11T11:00:00Z",
+ "window=2026-10-11T10:30:00Z/2026-10-11T12:00:00Z",
+ ],
+ ["window=2026-10-11T10:00:00Z"],
+ ["period=24h", "start=2026-10-11T10:00:00Z"],
+ ["period=0h"],
+ )
+ for tokens in cases:
+ with self.subTest(tokens=tokens), self.assertRaises(pc.UsageError):
+ pc.parse_side("baseline", tokens, self.shared(), NOW)
+
+ def test_identical_selectors_are_refused(self):
+ code, _, err = run_main(
+ [
+ "--project",
+ "builder",
+ "--baseline",
+ "release=a",
+ "--candidate",
+ "release=a",
+ ]
+ )
+ self.assertEqual(code, pc.EXIT_USAGE)
+ self.assertIn("select the same events", err)
+
+ def test_malformed_selector_exits_with_usage_code(self):
+ with contextlib.redirect_stderr(io.StringIO()), self.assertRaises(
+ SystemExit
+ ) as raised:
+ pc.build_parser().parse_args(["--baseline"])
+ self.assertEqual(raised.exception.code, pc.EXIT_USAGE)
+
+
+class OptionsTest(unittest.TestCase):
+ def refuse(self, path, params):
+ raise AssertionError("invalid options must not reach Sentry")
+
+ def assert_usage_error(self, extra, message):
+ transport = FakeTransport(self.refuse)
+ code, out, err = run_main(
+ COMPARE_ARGS + extra + ["--json"], transport=transport
+ )
+ self.assertEqual(code, pc.EXIT_USAGE)
+ self.assertIn(message, err)
+ self.assertEqual(json.loads(out)["error"]["type"], "usage")
+ self.assertEqual(transport.calls, [])
+
+ def test_region_url_must_be_https_with_a_host(self):
+ for url in (
+ "us.sentry.io",
+ "http://us.sentry.io",
+ "ftp://us.sentry.io",
+ "https://",
+ "https://us.sentry.io/api/0",
+ "https://us.sentry.io?x=1",
+ ):
+ with self.subTest(url=url):
+ self.assert_usage_error(["--region-url", url], "bad --region-url")
+
+ def test_plain_http_is_allowed_only_for_this_machine(self):
+ for url in (
+ "http://localhost:9000",
+ "http://127.0.0.1:9000",
+ "https://us.sentry.io/",
+ ):
+ with self.subTest(url=url):
+ code, _, _ = run_main(
+ COMPARE_ARGS + ["--region-url", url, "--print-queries"], environ={}
+ )
+ self.assertEqual(code, 0)
+
+ def test_timeout_must_be_positive(self):
+ for value in ("0", "-1"):
+ with self.subTest(value=value):
+ self.assert_usage_error(["--timeout", value], "--timeout")
+
+ def test_sample_cap_below_min_count_is_refused(self):
+ self.assert_usage_error(
+ ["--sample-cap", "100"], "--sample-cap 100 is below --min-count 150"
+ )
+
+ def test_min_traces_must_be_positive(self):
+ self.assert_usage_error(["--min-traces", "0"], "--min-traces")
+
+ def test_sample_cap_is_clamped_to_what_paging_can_fetch(self):
+ code, out, err = run_main(
+ COMPARE_ARGS + ["--sample-cap", "50000", "--print-queries"], environ={}
+ )
+ self.assertEqual(code, 0)
+ self.assertIn(f"limited to {pc.MAX_SAMPLE_CAP}", err)
+ self.assertIn(f"first {pc.MAX_SAMPLE_CAP} rows by id", out)
+
+ def test_too_many_comparisons_for_the_resample_cap_are_refused(self):
+ self.assert_usage_error(["--top", "200"], "compare fewer transactions")
+
+
+class PaginationTest(unittest.TestCase):
+ def client(self, route, sleep=None):
+ transport = FakeTransport(route)
+ client = pc.SentryClient(
+ "test-token", transport=transport, sleep=sleep or (lambda seconds: None)
+ )
+ return client, transport
+
+ def test_link_header_cursor_parsing(self):
+ first = fixture("samples_baseline_page1.json")["headers"]["link"]
+ last = fixture("samples_baseline_page2.json")["headers"]["link"]
+ self.assertEqual(pc.next_cursor(first), "0:100:0")
+ self.assertIsNone(pc.next_cursor(last))
+ self.assertIsNone(pc.next_cursor(""))
+
+ def test_paginate_follows_the_cursor_until_sentry_has_no_more(self):
+ client, transport = self.client(sentry_route())
+ rows = client.paginate([("query", f'release:"{BASE_SHA}"')], 1000)
+ self.assertEqual(len(rows), 160)
+ self.assertEqual(len(transport.calls), 2)
+ self.assertNotIn("cursor", transport.calls[0]["params"])
+ self.assertEqual(transport.calls[1]["params"]["cursor"], ["0:100:0"])
+
+ def test_paginate_stops_at_the_cap(self):
+ client, transport = self.client(sentry_route())
+ self.assertEqual(len(client.paginate([("query", "x")], 50)), 50)
+ self.assertEqual(len(transport.calls), 1)
+ self.assertEqual(len(client.paginate([("query", "x")], 130)), 130)
+ self.assertEqual(len(transport.calls), 3)
+
+ def test_paginate_stops_at_max_pages(self):
+ def endless(path, params):
+ body = {"data": [{"span.duration": 1.0, "trace": "t", "id": "x"}]}
+ link = (
+ '<https://x/?cursor=0:1:0>; rel="next"; results="true"; cursor="0:1:0"'
+ )
+ return 200, {"link": link}, json.dumps(body).encode()
+
+ client, transport = self.client(endless)
+ with mock.patch.object(pc, "MAX_PAGES", 3):
+ rows = client.paginate([("query", "x")], 1000)
+ self.assertEqual(len(rows), 3)
+ self.assertEqual(len(transport.calls), 3)
+
+ def test_paginate_stops_on_an_empty_page_that_claims_more(self):
+ def empty(path, params):
+ link = (
+ '<https://x/?cursor=0:100:0>; rel="next"; results="true"; '
+ 'cursor="0:100:0"'
+ )
+ return 200, {"link": link}, b'{"data": []}'
+
+ client, transport = self.client(empty)
+ self.assertEqual(client.paginate([("query", "x")], 1000), [])
+ self.assertEqual(len(transport.calls), 1)
+
+ def test_rate_limits_are_retried_after_the_advertised_delay(self):
+ replies = [(429, {"retry-after": "7"}, b"{}"), respond("project_builder.json")]
+ slept = []
+ client, _ = self.client(lambda path, params: replies.pop(0), slept.append)
+ self.assertEqual(client.project_id("builder"), "1000001")
+ self.assertEqual(slept, [7.0])
+
+ def test_network_errors_are_retried(self):
+ replies = [OSError("reset"), respond("project_builder.json")]
+
+ def route(path, params):
+ reply = replies.pop(0)
+ if isinstance(reply, Exception):
+ raise reply
+ return reply
+
+ client, _ = self.client(route)
+ self.assertEqual(client.project_id("builder"), "1000001")
+
+ def test_server_errors_give_up_after_the_retries(self):
+ slept = []
+ client, transport = self.client(
+ lambda path, params: (503, {}, b"unavailable"), slept.append
+ )
+ with self.assertRaises(pc.SentryApiError):
+ client.project_id("builder")
+ self.assertEqual(len(transport.calls), 4)
+ self.assertEqual(slept, [1.0, 2.0, 4.0])
+
+ def test_rejected_token_is_an_auth_error(self):
+ client, _ = self.client(lambda path, params: (403, {}, b'{"detail": "no"}'))
+ with self.assertRaises(pc.AuthError) as raised:
+ client.project_id("builder")
+ self.assertIn("event:read", str(raised.exception))
+
+ def test_rejected_query_is_a_usage_error_with_sentry_detail(self):
+ client, _ = self.client(lambda path, params: respond("error_400.json", 400))
+ with self.assertRaises(pc.UsageError) as raised:
+ client.events([("query", "x")])
+ self.assertIn("p50(nope)", str(raised.exception))
+
+ def test_unknown_project_is_a_usage_error(self):
+ client, _ = self.client(lambda path, params: (404, {}, b'{"detail": "x"}'))
+ with self.assertRaises(pc.UsageError):
+ client.project_id("nope")
+
+ def test_project_ids_are_looked_up_once(self):
+ client, transport = self.client(sentry_route())
+ client.project_id("builder")
+ client.project_id("builder")
+ self.assertEqual(client.project_id("12345"), "12345")
+ self.assertEqual(len(transport.calls), 1)
+
+
+class SentryStub(http.server.BaseHTTPRequestHandler):
+ """Serves ``server.route`` over real HTTP for urllib_transport."""
+
+ def do_GET(self):
+ parts = urllib.parse.urlsplit(self.path)
+ params = urllib.parse.parse_qs(parts.query, keep_blank_values=True)
+ self.server.seen.append((parts.path, self.headers.get("Authorization")))
+ status, headers, body = self.server.route(parts.path, params)
+ self.send_response(status)
+ for name, value in headers.items():
+ self.send_header(name, value)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *args):
+ pass
+
+
+class LiveHttpTest(unittest.TestCase):
+ def setUp(self):
+ patcher = mock.patch.dict(os.environ, {"no_proxy": "127.0.0.1,localhost"})
+ patcher.start()
+ self.addCleanup(patcher.stop)
+
+ def serve(self, route):
+ server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), SentryStub)
+ server.route = route
+ server.seen = []
+ thread = threading.Thread(
+ target=server.serve_forever, kwargs={"poll_interval": 0.02}, daemon=True
+ )
+ thread.start()
+ self.addCleanup(server.server_close)
+ self.addCleanup(server.shutdown)
+ return server, f"http://127.0.0.1:{server.server_address[1]}"
+
+ def test_transport_returns_status_lowercased_headers_and_body(self):
+ _, base = self.serve(sentry_route())
+ status, headers, body = pc.urllib_transport(
+ base + EVENTS_PATH + "?" + urllib.parse.urlencode({"query": "x"}), {}, 5
+ )
+ self.assertEqual(status, 200)
+ self.assertEqual(pc.next_cursor(headers["link"]), "0:100:0")
+ self.assertEqual(len(json.loads(body)["data"]), 100)
+
+ def test_transport_returns_error_statuses_with_headers(self):
+ _, base = self.serve(lambda path, params: (429, {"Retry-After": "3"}, b"{}"))
+ status, headers, _ = pc.urllib_transport(base + PROJECT_PATH, {}, 5)
+ self.assertEqual((status, headers["retry-after"]), (429, "3"))
+
+ def test_full_run_over_http(self):
+ server, base = self.serve(sentry_route())
+ code, out, _ = run_main(COMPARE_ARGS + ["--region-url", base, "--json"])
+ self.assertEqual(code, pc.EXIT_SLOWER)
+ self.assertEqual(json.loads(out)["requests"], 7)
+ self.assertEqual(len(server.seen), 7)
+ self.assertTrue(all(auth == "Bearer test-token" for _, auth in server.seen))
+
+ def test_rate_limit_over_http_is_retried(self):
+ replies = [(429, {"Retry-After": "0"}, b"{}")]
+ route = sentry_route()
+ slept = []
+ _, base = self.serve(
+ lambda path, params: replies.pop(0) if replies else route(path, params)
+ )
+ code, _, _ = run_main(
+ COMPARE_ARGS + ["--region-url", base, "--json"], sleep=slept.append
+ )
+ self.assertEqual(code, pc.EXIT_SLOWER)
+ self.assertEqual(slept, [0.0])
+
+ def test_rejected_token_over_http_exits_with_the_auth_code(self):
+ _, base = self.serve(
+ lambda path, params: (401, {}, b'{"detail": "Invalid token"}')
+ )
+ code, _, err = run_main(COMPARE_ARGS + ["--region-url", base])
+ self.assertEqual(code, pc.EXIT_AUTH)
+ self.assertIn("Invalid token", err)
+
+ def test_connection_refused_exits_with_the_api_code(self):
+ with socket.socket() as probe:
+ probe.bind(("127.0.0.1", 0))
+ port = probe.getsockname()[1]
+ slept = []
+ code, out, err = run_main(
+ COMPARE_ARGS + ["--region-url", f"http://127.0.0.1:{port}", "--json"],
+ sleep=slept.append,
+ )
+ self.assertEqual(code, pc.EXIT_API)
+ self.assertIn("network error", err)
+ self.assertEqual(json.loads(out)["error"]["type"], "api")
+ self.assertEqual(slept, [1.0, 2.0, 4.0])
+
+
+class StatisticsTest(unittest.TestCase):
+ def test_percentile_interpolates_between_ranks(self):
+ values = [1.0, 2.0, 3.0, 4.0, 5.0]
+ self.assertEqual(pc.percentile(values, 0.0), 1.0)
+ self.assertEqual(pc.percentile(values, 0.5), 3.0)
+ self.assertAlmostEqual(pc.percentile(values, 0.95), 4.8)
+ self.assertEqual(pc.percentile(values, 1.0), 5.0)
+ self.assertEqual(pc.percentile([7.0], 0.95), 7.0)
+ with self.assertRaises(ValueError):
+ pc.percentile([], 0.5)
+
+ def test_confidence_interval_takes_the_matching_tails(self):
+ low, high = pc.confidence_interval([float(i) for i in range(1, 101)], 0.9)
+ self.assertAlmostEqual(low, 5.95)
+ self.assertAlmostEqual(high, 95.05)
+
+ def test_bonferroni_adjustment(self):
+ self.assertAlmostEqual(pc.adjusted_confidence(0.95, 10), 0.995)
+ self.assertEqual(pc.adjusted_confidence(0.95, 1), 0.95)
+ self.assertEqual(pc.adjusted_confidence(0.95, 0), 0.95)
+
+ def test_resamples_are_raised_until_each_tail_has_twenty_draws(self):
+ self.assertEqual(pc.required_resamples(0.95), 800)
+ self.assertEqual(pc.required_resamples(pc.adjusted_confidence(0.95, 2)), 1600)
+ self.assertEqual(pc.required_resamples(pc.adjusted_confidence(0.95, 10)), 8000)
+ self.assertEqual(pc.required_resamples(1 - 1e-9), pc.RESAMPLE_CAP)
+ self.assertEqual(pc.effective_resamples(5000, 0.95), 5000)
+ self.assertEqual(pc.effective_resamples(500, 0.975), 1600)
+
+ def test_bootstrap_of_a_constant_shift_is_exact(self):
+ diffs = pc.bootstrap_quantile_diffs(
+ [[100.0]] * 50, [[110.0]] * 50, [0.5, 0.95], 200, random.Random(1)
+ )
+ self.assertEqual(set(diffs[0.5]), {10.0})
+ self.assertEqual(set(diffs[0.95]), {10.0})
+
+ def test_bootstrap_resamples_whole_traces(self):
+ draw = pc._drawer([[5.0, 5.0, 5.0], [1.0]], random.Random(3))
+ for _ in range(50):
+ values = draw()
+ self.assertEqual(values.count(5.0) % 3, 0)
+ self.assertEqual(len(values), values.count(5.0) + values.count(1.0))
+
+ def test_clusters_are_keyed_on_trace_not_span_id(self):
+ rows = [
+ {"span.duration": 1.0, "trace": "t1", "id": "a"},
+ {"span.duration": 2.0, "trace": "t1", "id": "b"},
+ {"span.duration": 3.0, "trace": "t2", "id": "c"},
+ {"span.duration": 4.0, "id": "d"},
+ {"span.duration": None, "trace": "t3", "id": "e"},
+ {"span.duration": "slow", "trace": "t4", "id": "f"},
+ {"span.duration": True, "trace": "t5", "id": "g"},
+ {"span.duration": float("nan"), "trace": "t6", "id": "h"},
+ ]
+ clusters = pc.to_clusters(rows, "span.duration")
+ self.assertEqual(clusters, [[1.0, 2.0], [3.0], [4.0]])
+ summary = pc.side_summary({"count": 8}, clusters)
+ self.assertEqual((summary["sample_n"], summary["sample_traces"]), (4, 3))
+
+ def test_bootstrap_is_deterministic_for_a_seed(self):
+ rng = random.Random(11)
+ baseline = [[rng.lognormvariate(4.8, 0.4)] for _ in range(120)]
+ candidate = [[rng.lognormvariate(4.9, 0.4)] for _ in range(120)]
+
+ def run(seed):
+ return pc.compare_quantiles(
+ baseline,
+ candidate,
+ random.Random(seed),
+ pc.Options(min_count=100, resamples=300),
+ )
+
+ self.assertEqual(run(5), run(5))
+ self.assertNotEqual(run(5)["p95"]["ci"], run(6)["p95"]["ci"])
+
+ def test_bootstrap_matches_pinned_values(self):
+ rng = random.Random(2024)
+ baseline = [[round(rng.uniform(90, 110), 3)] for _ in range(40)]
+ candidate = [[round(rng.uniform(100, 125), 3)] for _ in range(40)]
+ result = pc.compare_quantiles(
+ baseline,
+ candidate,
+ random.Random(99),
+ pc.Options(min_count=30, resamples=200),
+ family_size=1,
+ )
+ self.assertAlmostEqual(result["p50"]["diff"], PINNED_P50_DIFF, places=9)
+ self.assertEqual(
+ [round(bound, 9) for bound in result["p50"]["ci"]], PINNED_P50_CI
+ )
+ self.assertEqual(
+ [round(bound, 9) for bound in result["p95"]["ci"]], PINNED_P95_CI
+ )
+
+ def test_a_known_shift_is_found_and_no_shift_is_not(self):
+ rng = random.Random(42)
+ baseline = [[rng.lognormvariate(4.8, 0.3)] for _ in range(400)]
+ slower = [[value * 1.3 for value in cluster] for cluster in baseline]
+ same = [[rng.lognormvariate(4.8, 0.3)] for _ in range(400)]
+ options = pc.Options(resamples=400)
+ shifted = pc.compare_quantiles(baseline, slower, random.Random(1), options)
+ self.assertEqual(shifted["p50"]["verdict"], pc.SLOWER)
+ self.assertEqual(shifted["p95"]["verdict"], pc.SLOWER)
+ steady = pc.compare_quantiles(baseline, same, random.Random(1), options)
+ self.assertEqual(steady["p50"]["verdict"], pc.NO_CHANGE)
+ self.assertEqual(steady["p95"]["verdict"], pc.NO_CHANGE)
+
+ def test_many_spans_from_few_traces_is_not_enough_data(self):
+ few = [[100.0 + i for i in range(100)] for _ in range(3)]
+ many = [[200.0] for _ in range(300)]
+ options = pc.Options(resamples=100)
+ result = pc.compare_quantiles(few, many, random.Random(1), options)
+ self.assertEqual(result["p50"]["verdict"], pc.NOT_ENOUGH_DATA)
+ self.assertIn(
+ "needs 30 traces per side, has 3 / 300",
+ pc.sample_shortfall(few, many, options),
+ )
+ enough = pc.compare_quantiles(
+ few, many, random.Random(1), pc.Options(resamples=100, min_traces=3)
+ )
+ self.assertEqual(enough["p50"]["verdict"], pc.SLOWER)
+
+
+class CalibrationTest(unittest.TestCase):
+ """A/A runs: identical distributions must rarely be called a change."""
+
+ REPLICATES = 60
+ RESAMPLES = 800
+
+ @staticmethod
+ def iid_side(rng, count):
+ return [[rng.lognormvariate(4.8, 0.4)] for _ in range(count)]
+
+ @staticmethod
+ def clustered_side(rng, traces):
+ side = []
+ for _ in range(traces):
+ level = rng.gauss(0, 0.5)
+ side.append(
+ [
+ rng.lognormvariate(4.8 + level, 0.15)
+ for _ in range(rng.randint(1, 8))
+ ]
+ )
+ return side
+
+ def false_change_rate(self, make_pair, seed):
+ rng = random.Random(seed)
+ options = pc.Options(
+ min_count=50, min_traces=10, resamples=self.RESAMPLES, min_effect=0.0
+ )
+ changed = 0
+ for index in range(self.REPLICATES):
+ baseline, candidate = make_pair(rng)
+ result = pc.compare_quantiles(
+ baseline, candidate, random.Random(index), options, family_size=2
+ )
+ verdict = pc.combine_verdicts([stat["verdict"] for stat in result.values()])
+ changed += verdict in (pc.FASTER, pc.SLOWER)
+ return changed / self.REPLICATES
+
+ def test_independent_spans_are_called_a_change_about_five_percent_or_less(self):
+ rate = self.false_change_rate(
+ lambda rng: (self.iid_side(rng, 120), self.iid_side(rng, 120)), 1
+ )
+ self.assertLessEqual(rate, 0.07)
+
+ def test_clustered_spans_stay_calibrated_only_when_resampled_by_trace(self):
+ def clustered(rng):
+ return self.clustered_side(rng, 40), self.clustered_side(rng, 40)
+
+ def flattened(rng):
+ baseline, candidate = clustered(rng)
+ return (
+ [[value] for cluster in baseline for value in cluster],
+ [[value] for cluster in candidate for value in cluster],
+ )
+
+ self.assertLessEqual(self.false_change_rate(clustered, 2), 0.07)
+ self.assertGreaterEqual(self.false_change_rate(flattened, 2), 0.25)
+
+
+class VerdictTest(unittest.TestCase):
+ def test_quantile_verdict_rules(self):
+ cases = [
+ ((-10, -2, (-0.10, -0.08)), pc.FASTER),
+ ((2, 10, (0.10, 0.12)), pc.SLOWER),
+ ((2, 10, (0.10, None)), pc.SLOWER),
+ ((2, 10, (None, None)), pc.SLOWER),
+ ((-10, 2, (-0.10, -0.10)), pc.NO_CHANGE),
+ ((0, 10, (0.50, 0.50)), pc.NO_CHANGE),
+ ((-10, -2, (-0.10, -0.03)), pc.NO_CHANGE),
+ ((-10, -2, (-0.03, -0.10)), pc.NO_CHANGE),
+ ((-10, -2, (-0.10, 0.06)), pc.NO_CHANGE),
+ ]
+ for (low, high, effects), expected in cases:
+ with self.subTest(low=low, high=high, effects=effects):
+ self.assertEqual(
+ pc.quantile_verdict(low, high, effects, 0.05), expected
+ )
+
+ def test_too_few_samples_is_not_enough_data(self):
+ baseline = [[100.0]] * 199
+ candidate = [[200.0]] * 400
+ result = pc.compare_quantiles(
+ baseline, candidate, random.Random(1), pc.Options(resamples=100)
+ )
+ self.assertEqual(result["p50"]["verdict"], pc.NOT_ENOUGH_DATA)
+ self.assertIsNone(result["p50"]["ci"])
+ self.assertEqual(result["p50"]["diff"], 100.0)
+
+ def test_transaction_verdict_takes_the_worst_percentile(self):
+ self.assertEqual(pc.combine_verdicts([pc.FASTER, pc.SLOWER]), pc.SLOWER)
+ self.assertEqual(pc.combine_verdicts([pc.FASTER, pc.NO_CHANGE]), pc.FASTER)
+ self.assertEqual(
+ pc.combine_verdicts([pc.NO_CHANGE, pc.NO_CHANGE]), pc.NO_CHANGE
+ )
+ self.assertEqual(
+ pc.combine_verdicts([pc.NOT_ENOUGH_DATA, pc.NOT_ENOUGH_DATA]),
+ pc.NOT_ENOUGH_DATA,
+ )
+ self.assertEqual(pc.combine_verdicts([]), pc.NOT_ENOUGH_DATA)
+
+ def test_fisher_matches_exact_arithmetic(self):
+ for case in (
+ (0, 1000, 4, 1000),
+ (3, 1500, 43, 1450),
+ (10, 1000, 24, 1000),
+ (2, 400, 9, 400),
+ (5, 200, 5, 200),
+ (0, 300, 0, 300),
+ ):
+ with self.subTest(case=case):
+ self.assertAlmostEqual(
+ pc.fisher_exact_p(*case), exact_fisher(*case), places=10
+ )
+
+ def test_fisher_stays_fast_and_exact_on_huge_failure_counts(self):
+ def full_support(base_failures, base_total, cand_failures, cand_total):
+ total = base_total + cand_total
+ failures = base_failures + cand_failures
+ low = max(0, cand_total - (total - failures))
+ high = min(failures, cand_total)
+
+ def log_p(count):
+ return (
+ pc.log_comb(failures, count)
+ + pc.log_comb(total - failures, cand_total - count)
+ - pc.log_comb(total, cand_total)
+ )
+
+ observed = log_p(cand_failures) + math.log1p(1e-7)
+ values = (log_p(count) for count in range(low, high + 1))
+ return sum(math.exp(value) for value in values if value <= observed)
+
+ for case in ((40000, 100000, 40600, 100000), (30000, 100000, 30000, 100000)):
+ with self.subTest(case=case):
+ self.assertAlmostEqual(
+ pc.fisher_exact_p(*case) / full_support(*case), 1.0, places=9
+ )
+ with mock.patch.object(pc, "log_comb", wraps=pc.log_comb) as counted:
+ pc.fisher_exact_p(5_000_000, 10_000_000, 5_002_000, 10_000_000)
+ self.assertLess(counted.call_count, 200_000)
+
+ def test_failure_rate_verdicts(self):
+ def check(base, cand, family=1, **overrides):
+ options = pc.Options(**overrides)
+ return pc.compare_failure_rates(base, cand, options, family)["verdict"]
+
+ def agg(failures, count):
+ return {"count": count, "failure_rate": failures / count}
+
+ self.assertEqual(check(agg(0, 1000), agg(4, 1000)), pc.NO_CHANGE)
+ self.assertEqual(check(agg(3, 1500), agg(43, 1450)), pc.HIGHER)
+ self.assertEqual(check(agg(43, 1450), agg(3, 1500)), pc.LOWER)
+ self.assertEqual(check(agg(3, 1500), agg(4, 1500)), pc.NO_CHANGE)
+ self.assertEqual(check(agg(0, 1500), agg(0, 1500)), pc.NO_CHANGE)
+ self.assertEqual(check(agg(10, 1000), agg(24, 1000)), pc.HIGHER)
+ self.assertEqual(check(agg(10, 1000), agg(24, 1000), family=5), pc.NO_CHANGE)
+ big = (agg(1000, 100000), agg(1150, 100000))
+ self.assertEqual(check(*big), pc.HIGHER)
+ self.assertEqual(check(*big, min_failure_delta=0.002), pc.NO_CHANGE)
+ self.assertEqual(check(agg(0, 50), agg(25, 50)), pc.NOT_ENOUGH_DATA)
+ missing = {"count": 1500, "failure_rate": None}
+ self.assertEqual(check(missing, agg(9, 1500)), pc.NOT_ENOUGH_DATA)
+
+ def test_failure_rate_reports_the_p_value(self):
+ result = pc.compare_failure_rates(
+ {"count": 1000, "failure_rate": 0.0},
+ {"count": 1000, "failure_rate": 0.004},
+ pc.Options(),
+ )
+ self.assertAlmostEqual(result["p_value"], 0.1246249063, places=9)
+ self.assertAlmostEqual(result["diff"], 0.004)
+
+ def test_overall_outcome_and_exit_codes(self):
+ def result(verdict, failure=pc.NO_CHANGE):
+ stat = {"verdict": verdict, "rel": 0.1, "sentry_rel": 0.1}
+ return {
+ "transaction": "t",
+ "p50": stat,
+ "p95": stat,
+ "failure_rate": {"verdict": failure, "diff": 0.01},
+ "verdict": verdict,
+ }
+
+ cases = [
+ ([result(pc.FASTER), result(pc.SLOWER)], pc.SLOWER, pc.EXIT_SLOWER),
+ (
+ [result(pc.FASTER, failure=pc.HIGHER)],
+ pc.MORE_FAILURES,
+ pc.EXIT_SLOWER,
+ ),
+ ([result(pc.FASTER), result(pc.NO_CHANGE)], pc.FASTER, pc.EXIT_FASTER),
+ (
+ [result(pc.NO_CHANGE), result(pc.NOT_ENOUGH_DATA)],
+ pc.NO_CHANGE,
+ pc.EXIT_NO_CHANGE,
+ ),
+ ([result(pc.NOT_ENOUGH_DATA)], pc.NOT_ENOUGH_DATA, pc.EXIT_NOT_ENOUGH_DATA),
+ ([], pc.NOT_ENOUGH_DATA, pc.EXIT_NOT_ENOUGH_DATA),
+ ]
+ for results, verdict, code in cases:
+ with self.subTest(verdict=verdict):
+ outcome = pc.overall_outcome(results)
+ self.assertEqual(outcome["verdict"], verdict)
+ self.assertEqual(outcome["exit_code"], code)
+
+
+class ReportTest(unittest.TestCase):
+ def test_two_sampled_transactions_make_four_comparisons(self):
+ rng = random.Random(7)
+
+ def sample():
+ return [[rng.lognormvariate(4.8, 0.3)] for _ in range(60)]
+
+ def agg(count):
+ return {
+ "count": count,
+ "p50": 120.0,
+ "p95": 200.0,
+ "p99": 260.0,
+ "failure_rate": 0.0,
+ }
+
+ aggregates = {LIBRARY: agg(500), STORE: agg(400)}
+ report = pc.build_report(
+ side(),
+ side(label="candidate", release=CAND_SHA),
+ pc.Options(min_count=50, min_traces=10, resamples=200),
+ {},
+ (aggregates, aggregates),
+ (
+ {LIBRARY: sample(), STORE: sample()},
+ {LIBRARY: sample(), STORE: sample()},
+ ),
+ "millisecond",
+ 0,
+ )
+ settings = report["settings"]
+ self.assertEqual(settings["comparisons"], 4)
+ self.assertEqual(settings["failure_comparisons"], 2)
+ self.assertAlmostEqual(settings["adjusted_confidence"], 0.9875)
+ self.assertEqual(settings["resamples"], 3200)
+ for result in report["transactions"]:
+ for name in ("p50", "p95"):
+ low, high = result[name]["ci"]
+ adjusted_low, adjusted_high = result[name]["ci_adjusted"]
+ self.assertLessEqual(adjusted_low, low)
+ self.assertGreaterEqual(adjusted_high, high)
+
+
+class MissingTokenTest(unittest.TestCase):
+ def refuse(self, path, params):
+ raise AssertionError("no request may be sent without a token")
+
+ def test_missing_token_explains_the_scopes_and_sends_nothing(self):
+ transport = FakeTransport(self.refuse)
+ code, out, err = run_main(COMPARE_ARGS, environ={}, transport=transport)
+ self.assertEqual(code, pc.EXIT_AUTH)
+ self.assertIn("SENTRY_AUTH_TOKEN is not set", err)
+ self.assertIn("org:read project:read event:read", err)
+ self.assertIn("never looks for tokens on disk", err)
+ self.assertEqual(out, "")
+ self.assertEqual(transport.calls, [])
+
+ def test_blank_token_counts_as_missing_and_json_mode_reports_it(self):
+ code, out, _ = run_main(
+ COMPARE_ARGS + ["--json"],
+ environ={"SENTRY_AUTH_TOKEN": " "},
+ transport=FakeTransport(self.refuse),
+ )
+ self.assertEqual(code, pc.EXIT_AUTH)
+ payload = json.loads(out)
+ self.assertEqual(payload["exit_code"], pc.EXIT_AUTH)
+ self.assertEqual(payload["error"]["type"], "auth")
+
+ def test_printing_queries_needs_no_token(self):
+ code, out, _ = run_main(
+ COMPARE_ARGS + ["--transaction", LIBRARY, "--print-queries"],
+ environ={},
+ transport=FakeTransport(self.refuse),
+ )
+ self.assertEqual(code, 0)
+ self.assertIn('"sort": "id"', out)
+ self.assertIn('"period": "50h"', out)
+ self.assertIn(f'transaction:\\"{LIBRARY}\\"', out)
+
+
+class ExitCodeTest(unittest.TestCase):
+ def events_reply(self, reply):
+ route = sentry_route()
+
+ def answer(path, params):
+ return route(path, params) if path == PROJECT_PATH else reply
+
+ return FakeTransport(answer)
+
+ def test_unexpected_errors_exit_with_the_api_code_not_as_a_regression(self):
+ def explode(path, params):
+ raise RuntimeError("boom")
+
+ code, out, err = run_main(
+ COMPARE_ARGS + ["--json"], transport=FakeTransport(explode)
+ )
+ self.assertEqual(code, pc.EXIT_API)
+ self.assertIn("RuntimeError: boom", err)
+ payload = json.loads(out)
+ self.assertEqual(payload["error"]["type"], "internal")
+ self.assertEqual(payload["exit_code"], pc.EXIT_API)
+
+ def test_sentry_status_codes_map_to_exit_codes(self):
+ cases = [
+ (respond("error_400.json", 400), pc.EXIT_USAGE),
+ ((403, {}, b'{"detail": "forbidden"}'), pc.EXIT_AUTH),
+ ((503, {}, b"unavailable"), pc.EXIT_API),
+ ((418, {}, b"teapot"), pc.EXIT_API),
+ ((200, {}, b"not json"), pc.EXIT_API),
+ ((200, {}, b'{"rows": []}'), pc.EXIT_API),
+ ]
+ for reply, expected in cases:
+ with self.subTest(status=reply[0], body=reply[2][:12]):
+ code, _, _ = run_main(COMPARE_ARGS, transport=self.events_reply(reply))
+ self.assertEqual(code, expected)
+
+
+class EndToEndTest(unittest.TestCase):
+ def test_slower_candidate_with_more_failures_exits_one(self):
+ transport = FakeTransport(sentry_route())
+ code, out, _ = run_main(COMPARE_ARGS + ["--json"], transport=transport)
+ self.assertEqual(code, pc.EXIT_SLOWER)
+ report = json.loads(out)
+ library, store = report["transactions"]
+ self.assertEqual(library["transaction"], LIBRARY)
+ self.assertEqual(library["verdict"], pc.SLOWER)
+ self.assertIsNone(library["note"])
+ self.assertEqual(library["failure_rate"]["verdict"], pc.HIGHER)
+ self.assertLess(library["failure_rate"]["p_value"], 1e-6)
+ self.assertTrue(library["regression"])
+ self.assertEqual(library["baseline"]["sample_n"], 160)
+ self.assertEqual(library["candidate"]["sample_n"], 160)
+ self.assertGreater(library["p50"]["rel"], 0.15)
+ self.assertAlmostEqual(library["p50"]["sentry_rel"], 150.6 / 120.4 - 1)
+ self.assertEqual(store["verdict"], pc.NOT_ENOUGH_DATA)
+ self.assertEqual(store["note"], "needs 150 samples per side, has 0 / 0")
+ self.assertEqual(store["baseline"]["sample_n"], 0)
+ self.assertEqual(report["settings"]["comparisons"], 2)
+ self.assertEqual(report["settings"]["resamples"], 1600)
+ self.assertEqual(report["overall"]["verdict"], pc.SLOWER)
+ self.assertEqual(report["unit"], "millisecond")
+ self.assertEqual(report["requests"], 7)
+
+ def test_requests_match_the_documented_queries(self):
+ transport = FakeTransport(sentry_route())
+ run_main(COMPARE_ARGS + ["--json"], transport=transport)
+ paths = [call["path"] for call in transport.calls]
+ self.assertEqual(paths.count(PROJECT_PATH), 1)
+ self.assertEqual(paths.count(EVENTS_PATH), 6)
+ for call in transport.calls:
+ self.assertEqual(call["headers"]["Authorization"], "Bearer test-token")
+ aggregate = transport.calls[1]["params"]
+ self.assertEqual(aggregate["dataset"], ["spans"])
+ self.assertEqual(aggregate["project"], ["1000001"])
+ self.assertEqual(aggregate["sort"], ["-count()"])
+ self.assertEqual(aggregate["field"], pc.aggregate_fields("span.duration"))
+ self.assertEqual(aggregate["start"], ["2026-10-10T14:00:00"])
+ self.assertEqual(aggregate["end"], ["2026-10-11T14:00:00"])
+ samples = [
+ call["params"]
+ for call in transport.calls
+ if "trace" in call["params"].get("field", [])
+ ]
+ self.assertEqual(len(samples), 4)
+ for params in samples:
+ self.assertEqual(params["sort"], ["id"])
+ self.assertEqual(params["per_page"], ["100"])
+ self.assertIn(f'transaction:"{LIBRARY}"', params["query"][0])
+
+ def test_markdown_report_names_the_verdict(self):
+ code, out, _ = run_main(COMPARE_ARGS, transport=FakeTransport(sentry_route()))
+ self.assertEqual(code, pc.EXIT_SLOWER)
+ self.assertIn("**Overall: slower** (exit 1)", out)
+ self.assertIn(f"| {LIBRARY} | 160 / 160", out)
+ self.assertIn("**slower, more failures**", out)
+ self.assertIn("**not enough data** (needs 150 samples per side", out)
+
+ def test_identical_data_is_no_detectable_change(self):
+ transport = FakeTransport(sentry_route(candidate_data="baseline"))
+ code, out, _ = run_main(COMPARE_ARGS + ["--json"], transport=transport)
+ self.assertEqual(code, pc.EXIT_NO_CHANGE)
+ library = json.loads(out)["transactions"][0]
+ self.assertEqual(library["p50"]["diff"], 0.0)
+ self.assertEqual(library["verdict"], pc.NO_CHANGE)
+
+ def test_too_little_traffic_is_not_enough_data(self):
+ transport = FakeTransport(sentry_route())
+ code, out, _ = run_main(
+ COMPARE_ARGS + ["--min-count", "2000", "--sample-cap", "2000", "--json"],
+ transport=transport,
+ )
+ self.assertEqual(code, pc.EXIT_NOT_ENOUGH_DATA)
+ self.assertEqual(json.loads(out)["overall"]["verdict"], pc.NOT_ENOUGH_DATA)
+ self.assertEqual(len(transport.calls), 3)
+
+ def test_rejected_token_exits_with_the_auth_code(self):
+ code, _, err = run_main(
+ COMPARE_ARGS, transport=FakeTransport(lambda p, q: (401, {}, b"{}"))
+ )
+ self.assertEqual(code, pc.EXIT_AUTH)
+ self.assertIn("HTTP 401", err)
+
+
+PINNED_P50_DIFF = 11.6705
+PINNED_P50_CI = [6.881525, 18.8584125]
+PINNED_P95_CI = [12.03550875, 18.80014375]
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/autogpt_platform/backend/load-tests/README.md b/autogpt_platform/backend/load-tests/README.md
index f4a48bf0aa..5f989a86ed 100644
--- a/autogpt_platform/backend/load-tests/README.md
+++ b/autogpt_platform/backend/load-tests/README.md
@@ -94,8 +94,9 @@ npm run cloud
### Environment Configuration
- **LOCAL**: `http://localhost:8006` (local development)
-- **DEV**: `https://dev-api.agpt.co` (development environment)
+- **DEV**: `https://dev-server.agpt.co` (development environment)
- **PROD**: `https://api.agpt.co` (production environment - coordinate with team!)
+- **PREVIEW**: `https://pr-<N>-server.agpt.co`, the `!deploy` preview of PR `<N>`. Set `PR_NUMBER=<N>` next to `K6_ENVIRONMENT=PREVIEW`. A preview starts with an empty database and has its own sign-in, so `generate-tokens.js` tokens are not valid there; `tests/marketplace/public-access-test.js` needs no token. `tests/basic/connectivity-test.js` also calls the preview's own Supabase project, so it needs `SUPABASE_URL` and `SUPABASE_ANON_KEY` for that project. Measuring a preview against a baseline is covered in `.claude/skills/perf-compare/SKILL.md`.
## 📊 Performance Testing Features
diff --git a/autogpt_platform/backend/load-tests/configs/environment.js b/autogpt_platform/backend/load-tests/configs/environment.js
index 46f8040820..aff0bdba50 100644
--- a/autogpt_platform/backend/load-tests/configs/environment.js
+++ b/autogpt_platform/backend/load-tests/configs/environment.js
@@ -29,9 +29,30 @@ export const ENV_CONFIG = {
// Get environment config based on K6_ENVIRONMENT variable (default: DEV)
export function getEnvironmentConfig() {
const env = __ENV.K6_ENVIRONMENT || "DEV";
+ if (env === "PREVIEW") {
+ return getPreviewConfig(__ENV.PR_NUMBER);
+ }
return ENV_CONFIG[env];
}
+// K6_ENVIRONMENT=PREVIEW PR_NUMBER=<n> targets the `!deploy` preview of PR <n>.
+// Each preview has its own Supabase project that nothing derives from <n>, so
+// tests that call Supabase directly need SUPABASE_URL and SUPABASE_ANON_KEY.
+export function getPreviewConfig(prNumber) {
+ if (!/^[1-9]\d{0,6}$/.test(prNumber || "")) {
+ throw new Error(
+ "K6_ENVIRONMENT=PREVIEW needs PR_NUMBER=<pull request number>",
+ );
+ }
+ return {
+ API_BASE_URL: `https://pr-${prNumber}-server.agpt.co`,
+ BUILDER_BASE_URL: `https://autogpt-pr-${prNumber}.vercel.app`,
+ WS_BASE_URL: `wss://pr-${prNumber}-ws.agpt.co`,
+ SUPABASE_URL: __ENV.SUPABASE_URL,
+ SUPABASE_ANON_KEY: __ENV.SUPABASE_ANON_KEY,
+ };
+}
+
// Authentication configuration
export const AUTH_CONFIG = {
// Test user credentials - REPLACE WITH ACTUAL TEST ACCOUNTS