The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
feat(skills): add performance skill evals (Closes #253)
Liyu3c3ed3e92aa8
diff --git a/skills/traceway/evals/README.md b/skills/traceway/evals/README.mdnew file mode 100644index 00000000..18676638--- /dev/null+++ b/skills/traceway/evals/README.md@@ -0,0 +1,48 @@+# Performance Skill Evals++Scenario-based evals for the `traceway` skill's **Performance** flow+(`/traceway perf <endpoint or symptom>`). Each eval hands an agent a+synthetic-but-realistic Traceway telemetry snapshot and asks it to diagnose+the bottleneck using the skill; the expected diagnosis is pinned to one row+of the bottleneck checklist in `../performance.md`.++## How to run an eval++1. Give the agent the `traceway` skill (or the `traceway` CLI against a+ staging instance seeded with the snapshot data).+2. Paste the **Task** prompt from the scenario file verbatim.+3. Grade the answer against **Expected diagnosis** and **Pass criteria**.+ A pass requires naming the right checklist suspect *and* citing the+ telemetry that confirms it — a correct guess with no evidence is a fail.++The snapshots are static text, so evals also work offline: the agent should+reason from the snapshot exactly as it would from live `traceway` output.++## Coverage++| ID | Area | Framework | Checklist suspect |+|---|---|---|---|+| `backend-go-select-n-plus-1` | backend | Go (Gin) | Select N+1 |+| `backend-python-pool-exhaustion` | backend | Python (FastAPI) | Connection-pool exhaustion |+| `backend-node-sequential-calls` | backend | Node.js (NestJS) | Sequential calls that could run in parallel |+| `backend-missing-index` | backend | any (PostgreSQL) | Missing or unused index |+| `backend-sync-background-task` | backend | Go (Gin) | Synchronous work that belongs in a background task |+| `frontend-js-large-payloads` | frontend | browser JS | Large request/response payloads |+| `frontend-js-no-cache` | frontend | browser JS | No cache / low hit ratio |+| `mobile-ios-cold-start` | mobile | iOS | Cold start (synchronous init on launch) |+| `mobile-android-gc-pauses` | mobile | Android | Memory pressure / GC pauses |+| `mobile-flutter-shader-jank` | mobile | Flutter | Cold start / JIT / shader warmup |+| `infra-cpu-saturation` | infrastructure | n/a | CPU saturation |+| `infra-cache-stampede` | infrastructure | n/a | Cache stampede / thundering herd |++## Adding a new eval++1. Copy the file format below: header table, `## Telemetry snapshot`,+ `## Task`, `## Expected diagnosis`, `## Pass criteria`.+2. Build the snapshot from a real incident when possible; keep numbers+ internally consistent (percentiles, span durations, and metric values+ must agree with each other).+3. Pin the expected diagnosis to exactly one `performance.md` checklist+ row, and make the snapshot contain the confirming signal that row+ asks for — plus one plausible distractor the agent must rule out.+4. Add a row to the coverage table above.diff --git a/skills/traceway/evals/backend-go-select-n-plus-1.md b/skills/traceway/evals/backend-go-select-n-plus-1.mdnew file mode 100644index 00000000..afbb1892--- /dev/null+++ b/skills/traceway/evals/backend-go-select-n-plus-1.md@@ -0,0 +1,55 @@+# Eval: backend-go-select-n-plus-1++- **ID**: `backend-go-select-n-plus-1`+- **Area**: backend+- **Framework**: Go (Gin + PostgreSQL)+- **Checklist item**: Select N+1 (`performance.md` → Database and data access)++## Telemetry snapshot++`traceway endpoints list --search "GET /api/orders" --since 24h --output json`:++```json+{ "endpoint": "GET /api/orders", "count": 48210, "p50": 412, "p95": 2380, "p99": 3110, "errorCount": 0 }+```++`traceway endpoints show <endpointId> --recorded-at <t>` — span waterfall of one+p95 request (total 2.31s):++```+GET /api/orders ............ 2310ms+└── db.query ............... 3ms SELECT * FROM orders WHERE user_id = $1 LIMIT 50+└── db.query ............... 2ms SELECT * FROM order_items WHERE order_id = $1+└── db.query ............... 2ms SELECT * FROM order_items WHERE order_id = $1+... (347 more near-identical order_items spans, 1–3ms each)+└── json.encode ............ 41ms+```++`traceway metrics query --name system.cpu.utilization --aggregation max --since 24h`:+flat at 22–31% all day. p50/p95 have crept up ~15% per week for a month with+no deploy in that window (`git log` clean).++## Task++> `GET /api/orders` p95 is 2.4s and climbing ~15% a week with no recent+> deploy. Diagnose the bottleneck to a specific checklist cause and propose+> the fix.++## Expected diagnosis++- **Root cause**: Select N+1 — the handler loads 50 orders, then issues one+ `order_items` query per order (≈348 near-identical short spans); total DB+ time dominates while each call is tiny.+- **Key evidence**: repeated near-identical query spans in the waterfall;+ flat CPU (rules out saturation); latency creeping with data growth and no+ deploy (rules out a regression).+- **Fix**: single JOIN / batched `WHERE order_id IN (...)` load, or a+ dataloader.++## Pass criteria++- Names **Select N+1** (not "slow query" or "missing index" — the individual+ queries are 1–3ms, so neither fits).+- Cites the repeated spans *and* rules out saturation/regression from the+ flat CPU and clean deploy history.+- Proposes batching the item load.diff --git a/skills/traceway/evals/backend-missing-index.md b/skills/traceway/evals/backend-missing-index.mdnew file mode 100644index 00000000..731f217f--- /dev/null+++ b/skills/traceway/evals/backend-missing-index.md@@ -0,0 +1,53 @@+# Eval: backend-missing-index++- **ID**: `backend-missing-index`+- **Area**: backend+- **Framework**: any (PostgreSQL)+- **Checklist item**: Missing or unused index (`performance.md` → Database and data access)++## Telemetry snapshot++`traceway endpoints list --search "GET /api/users/lookup" --since 24h --output json`:++```json+{ "endpoint": "GET /api/users/lookup", "count": 88100, "p50": 940, "p95": 1210, "p99": 1480, "errorCount": 0 }+```++`traceway endpoints show <endpointId> --recorded-at <t>` — one request+(total 1.02s):++```+GET /api/users/lookup ..... 1020ms+└── db.query .............. 980ms SELECT id, name, email FROM users WHERE email = $1+└── json.encode ........... 12ms+```++The same query span dominates in *every* sampled trace, at all hours, at all+traffic levels. `endpoints chart --metric-type p50 --since 30d` shows p50+rising smoothly from 210ms to 940ms over two months; `git log` shows no+deploy touching this path; the `users` table grew from 2M to 9M rows in the+same window. `EXPLAIN` (from a DBA paste in the linked issue) shows+`Seq Scan on users`.++## Task++> `GET /api/users/lookup` p50 crept from 210ms to 940ms over two months with+> no deploy on that path. Diagnose the bottleneck to a specific checklist+> cause.++## Expected diagnosis++- **Root cause**: Missing index on `users(email)` — one query span is the+ long pole in every trace, its duration grows with table size, and the+ latency curve tracks data growth, not deploys.+- **Key evidence**: single dominating query span across all traces; smooth+ multi-week climb with no deploy; table growth 2M→9M rows; `Seq Scan`.+- **Fix**: `CREATE INDEX CONCURRENTLY ON users (email)`; verify p50 drops+ back.++## Pass criteria++- Names **missing index** (not N+1 — there is exactly one query; not+ saturation — CPU flat, all-hours slowness).+- Connects the multi-week climb + table growth + no deploy to data-driven+ latency, and cites the `Seq Scan`.diff --git a/skills/traceway/evals/backend-node-sequential-calls.md b/skills/traceway/evals/backend-node-sequential-calls.mdnew file mode 100644index 00000000..9da87559--- /dev/null+++ b/skills/traceway/evals/backend-node-sequential-calls.md@@ -0,0 +1,54 @@+# Eval: backend-node-sequential-calls++- **ID**: `backend-node-sequential-calls`+- **Area**: backend+- **Framework**: Node.js (NestJS, microservice fan-out)+- **Checklist item**: Sequential calls that could run in parallel+ (`performance.md` → Network, IO, and external dependencies)++## Telemetry snapshot++`traceway endpoints list --search "GET /api/dashboard" --since 24h --output json`:++```json+{ "endpoint": "GET /api/dashboard", "count": 31200, "p50": 3890, "p95": 4520, "p99": 5100, "errorCount": 0 }+```++`traceway traces show <traceId> --recorded-at <t>` — cross-service timeline+(total 4.41s):++```+GET /api/dashboard (edge) .............. 4410ms+└── svc-users GET /profile ............. 890ms ─┐+└── svc-billing GET /balance ........... 920ms ├ staircase: each starts+└── svc-notify GET /unread ............. 870ms │ only when the previous+└── svc-reco GET /suggestions .......... 940ms ─┘ ends+└── json.encode ........................ 120ms+```++Each downstream service is healthy on its own (their p95s are 120–200ms in+their own dashboards); the 900ms here is mostly network round-trip per hop.+`system.cpu.utilization` flat at ~30%.++## Task++> `GET /api/dashboard` p50 is 3.9s. Every downstream service looks healthy+> on its own. Diagnose the bottleneck to a specific checklist cause.++## Expected diagnosis++- **Root cause**: Sequential downstream calls that could run in parallel —+ the staircase waterfall shows four independent service hops serialized;+ each hop's own latency is fine, the sum is the problem.+- **Key evidence**: staircase shape in `traces show`; downstream services+ healthy in isolation; flat CPU rules out saturation.+- **Fix**: `Promise.all` the four independent calls (they share no data+ dependencies); expected p50 ≈ 1.1s.++## Pass criteria++- Names **sequential calls that could run in parallel** (not "slow+ upstream" — upstreams are healthy; not "chatty fan-out" — four calls is+ not chatty, the serialization is the issue).+- Reads the staircase waterfall correctly and notes the calls are+ independent.diff --git a/skills/traceway/evals/backend-python-pool-exhaustion.md b/skills/traceway/evals/backend-python-pool-exhaustion.mdnew file mode 100644index 00000000..47c5aa92--- /dev/null+++ b/skills/traceway/evals/backend-python-pool-exhaustion.md@@ -0,0 +1,55 @@+# Eval: backend-python-pool-exhaustion++- **ID**: `backend-python-pool-exhaustion`+- **Area**: backend+- **Framework**: Python (FastAPI + PostgreSQL, SQLAlchemy pool size 20)+- **Checklist item**: Connection-pool exhaustion (`performance.md` → Database and data access)++## Telemetry snapshot++`traceway endpoints list --search "POST /api/checkout" --since 24h --output json`:++```json+{ "endpoint": "POST /api/checkout", "count": 96400, "p50": 118, "p95": 940, "p99": 8210, "errorCount": 12 }+```++p50 flat all day; p99 climbs steeply during the 12:00–14:00 and 19:00–21:00+traffic peaks and collapses between them.++`traceway endpoints show <endpointId> --recorded-at <t>` — one p99 request+(total 8.04s):++```+POST /api/checkout ....... 8040ms+├── <gap: no spans> ...... 3120ms+└── db.query ............. 44ms SELECT ... FROM inventory WHERE sku = $1+└── db.query ............. 61ms UPDATE orders SET ...+└── http.post ............ 280ms POST https://payments.example.com/charge+```++`traceway metrics query --name system.cpu.utilization --aggregation max --since 24h`:+35–48%, no spike aligned with the p99 humps. `traceway logs query --search "pool"`:+`sqlalchemy.pool.impl.QueuePool: "QueuePool limit of size 20 overflow 10 reached, connection timed out"` WARN lines during both peaks.++## Task++> `POST /api/checkout` p50 is fine at 118ms but p99 hits 8.2s during traffic+> peaks. Diagnose the bottleneck to a specific checklist cause.++## Expected diagnosis++- **Root cause**: Connection-pool exhaustion — the 3.1s gap *before* the+ first DB span is time waiting to acquire a connection, not work; p99 rises+ with concurrency while p50 stays flat; pool-timeout warnings in the logs+ confirm it.+- **Key evidence**: the pre-DB gap in the waterfall; p99-vs-p50 divergence+ tracking traffic; `QueuePool limit reached` log lines.+- **Fix**: raise pool size / add overflow, shorten checkout transaction scope,+ or shed load (queue) at the peak.++## Pass criteria++- Names **connection-pool exhaustion** (not "slow DB" — the queries are+ 44–61ms; not "slow upstream" — the payment call is 280ms).+- Points at the gap *before* the first DB span as the finding, and cites the+ p99/p50 divergence plus the pool log lines.diff --git a/skills/traceway/evals/backend-sync-background-task.md b/skills/traceway/evals/backend-sync-background-task.mdnew file mode 100644index 00000000..18c38873--- /dev/null+++ b/skills/traceway/evals/backend-sync-background-task.md@@ -0,0 +1,56 @@+# Eval: backend-sync-background-task++- **ID**: `backend-sync-background-task`+- **Area**: backend+- **Framework**: Go (Gin)+- **Checklist item**: Synchronous work that belongs in a background task+ (`performance.md` → Application and compute)++## Telemetry snapshot++`traceway endpoints list --search "POST /api/reports" --since 24h --output json`:++```json+{ "endpoint": "POST /api/reports", "count": 1400, "p50": 9400, "p95": 12100, "p99": 14800, "errorCount": 0 }+```++`traceway endpoints show <endpointId> --recorded-at <t>` — one request+(total 11.9s):++```+POST /api/reports ........ 11900ms+└── db.query ............. 320ms INSERT INTO reports ...+└── pdf.render ............ 9100ms (in-process)+└── s3.put ................ 410ms PUT reports/2026-10-16/...+└── json.encode ........... 18ms {"report_id": "...", "status": "ready"}+```++The response returns only after the PDF is fully rendered and uploaded; the+user does not need the bytes synchronously (the UI polls `GET+/api/reports/:id`). `system.cpu.utilization` spikes to 91% on the single+instance during each render. No queue depth anywhere — there is no task+system on this path.++## Task++> `POST /api/reports` p50 is 9.4s. Diagnose the bottleneck to a specific+> checklist cause and propose the fix.++## Expected diagnosis++- **Root cause**: Synchronous work that belongs in a background task — the+ 9.1s `pdf.render` span runs before the response returns, for work the user+ does not wait on.+- **Key evidence**: the in-process render span dominating the waterfall;+ response only returned after upload; UI already polls for completion, so+ async is safe.+- **Fix**: return `202 {"status": "processing"}` immediately and move render+ + upload into a `CaptureTask` background task.++## Pass criteria++- Names **synchronous work that belongs in a background task** (not "CPU+ saturation" — CPU spikes because of this render, it is the effect, not the+ cause; not "slow upstream" — S3 put is 410ms).+- Notes the user doesn't need the result synchronously (polling UI) and+ proposes the background-task fix.diff --git a/skills/traceway/evals/frontend-js-large-payloads.md b/skills/traceway/evals/frontend-js-large-payloads.mdnew file mode 100644index 00000000..45717ccf--- /dev/null+++ b/skills/traceway/evals/frontend-js-large-payloads.md@@ -0,0 +1,51 @@+# Eval: frontend-js-large-payloads++- **ID**: `frontend-js-large-payloads`+- **Area**: frontend+- **Framework**: browser JS (React SPA)+- **Checklist item**: Large request/response payloads (`performance.md` →+ Network, IO, and external dependencies)++## Telemetry snapshot++`traceway endpoints list --search "GET /api/feed" --since 24h --output json`:++```json+{ "endpoint": "GET /api/feed", "count": 120400, "p50": 1810, "p95": 3400, "p99": 5200, "errorCount": 0 }+```++`traceway endpoints show <endpointId> --recorded-at <t>` — one request+(total 3.28s):++```+GET /api/feed ............ 3280ms+└── db.query ............. 210ms SELECT ... LIMIT 50+└── json.encode .......... 180ms+└── transfer ............. 2840ms response 2.3 MB+```++The response carries 50 feed cards with full-resolution image URLs inlined+as base64 (`data:` URIs, ~45KB each). p50/p95 correlate with the `?page`+depth, not with server load; `system.cpu.utilization` flat at ~25%. Mobile+web p95 is 2× desktop — transfer-dominated.++## Task++> `GET /api/feed` p50 is 1.8s, worse on mobile web. Server-side spans are+> fast. Diagnose the bottleneck to a specific checklist cause.++## Expected diagnosis++- **Root cause**: Large response payloads — 2.3MB per response, dominated by+ inlined base64 images; transfer time is the long pole, not server work.+- **Key evidence**: `transfer` span 2.84s vs 210ms DB; latency tracks page+ depth and is worse on mobile (bandwidth-bound signature); flat CPU.+- **Fix**: thumbnail URLs + lazy-load full images, paginate or compress,+ strip base64 inlining.++## Pass criteria++- Names **large payloads** (not "slow DB" — DB is 210ms; not "no cache" —+ there is no evidence about caching here).+- Identifies transfer as the long pole and ties the mobile-web skew to+ bandwidth.diff --git a/skills/traceway/evals/frontend-js-no-cache.md b/skills/traceway/evals/frontend-js-no-cache.mdnew file mode 100644index 00000000..65c5353f--- /dev/null+++ b/skills/traceway/evals/frontend-js-no-cache.md@@ -0,0 +1,44 @@+# Eval: frontend-js-no-cache++- **ID**: `frontend-js-no-cache`+- **Area**: frontend+- **Framework**: browser JS (React SPA)+- **Checklist item**: No cache / low hit ratio (`performance.md` → Caching and delivery)++## Telemetry snapshot++`traceway endpoints list --search "GET /api/products" --since 24h --output json`:++```json+{ "endpoint": "GET /api/products", "count": 388000, "p50": 640, "p95": 980, "p99": 1400, "errorCount": 0 }+```++The product catalog changes at most twice a day, yet the endpoint is hit on+*every* page navigation — 388k hits/day for ~40k daily users. The+`cache.hit_ratio` metric (reported by the CDN edge) sits at 4%. Waterfall of+one request is unremarkable: 380ms DB + 120ms serialize, no single slow span.++`traceway endpoints chart --metric-type total_time --since 7d` shows total+server time for this endpoint is 3× the next endpoint — it is the fleet's+biggest cost center by volume, not by per-request latency.++## Task++> `GET /api/products` per-request latency looks fine, but it is the fleet's+> biggest total-time cost center. Diagnose to a specific checklist cause.++## Expected diagnosis++- **Root cause**: No effective cache — a twice-daily-changing catalog is+ fetched on every navigation with a 4% edge hit ratio; the DB is hit on+ every request.+- **Key evidence**: 388k hits vs 40k users; 4% hit ratio; total_time+ dominance without per-request slownness.+- **Fix**: edge-cache the catalog response with a 6h TTL + surrogate-key+ purge on catalog publish; or `stale-while-revalidate`.++## Pass criteria++- Names **no cache / low hit ratio** (not "slow query" — per-request spans+ are fine; the problem is volume × repetition).+- Uses the hit-ratio metric and the hits-per-user ratio, not just latency.diff --git a/skills/traceway/evals/infra-cache-stampede.md b/skills/traceway/evals/infra-cache-stampede.mdnew file mode 100644index 00000000..1cbfb4e1--- /dev/null+++ b/skills/traceway/evals/infra-cache-stampede.md@@ -0,0 +1,46 @@+# Eval: infra-cache-stampede++- **ID**: `infra-cache-stampede`+- **Area**: infrastructure+- **Framework**: n/a (Redis-backed API)+- **Checklist item**: Cache stampede / thundering herd (`performance.md` →+ Caching and delivery)++## Telemetry snapshot++`traceway endpoints list --search "GET /api/homepage" --since 24h --output json`:++```json+{ "endpoint": "GET /api/homepage", "count": 210000, "p50": 95, "p95": 210, "p99": 4800, "errorCount": 0 }+```++`traceway endpoints chart --metric-type p99 --since 24h --interval-minutes 5`+for this endpoint: p99 is ~200ms all day except sharp 4–5s spikes at+**:00, :10, :20, :30, :40, :50** — exactly the 10-minute cache TTL boundary.++During a spike window, `traces show` reveals hundreds of near-identical+traces all recomputing the homepage payload (12 DB queries each) within the+same second; between spikes, the cache-hit metric reads 99%. The stampede+traces show no lock contention — every worker independently missed and+recomputed.++## Task++> `GET /api/homepage` p99 spikes to 5s exactly every 10 minutes, aligned to+> the cache TTL. Diagnose to a specific checklist cause.++## Expected diagnosis++- **Root cause**: Cache stampede / thundering herd — at each TTL expiry a+ burst of workers simultaneously miss and recompute the same payload.+- **Key evidence**: spikes phase-locked to the TTL boundary; burst of+ identical recompute traces; 99% hit ratio between spikes (rules out "no+ cache").+- **Fix**: probabilistic early refresh / request coalescing (singleflight)+ / staggered TTLs.++## Pass criteria++- Names **cache stampede** (not "no cache" — hit ratio is 99% between+ spikes; not "slow DB" — queries are fine, the burst is the problem).+- Cites the TTL-aligned periodicity as the decisive signal.diff --git a/skills/traceway/evals/infra-cpu-saturation.md b/skills/traceway/evals/infra-cpu-saturation.mdnew file mode 100644index 00000000..9727c05b--- /dev/null+++ b/skills/traceway/evals/infra-cpu-saturation.md@@ -0,0 +1,49 @@+# Eval: infra-cpu-saturation++- **ID**: `infra-cpu-saturation`+- **Area**: infrastructure+- **Framework**: n/a (fleet-wide)+- **Checklist item**: CPU saturation (`performance.md` → Infrastructure and saturation)++## Telemetry snapshot++`traceway endpoints list --since 24h --order-by p95 --page-size 10`: the top+8 endpoints *all* show p95 2–4× their 7-day baseline, starting at 15:40+yesterday. No single span dominates — waterfalls show every span type+(DB, serialize, upstream) uniformly slower, with no long pole.++`traceway metrics query --name system.cpu.utilization --aggregation max --since 24h --interval-minutes 60`:++```+14:00 38% 15:00 44% 16:00 97% 17:00 98% 18:00 96% ...+```++The CPU step at ~15:40 lines up exactly with the latency rise across all+endpoints. `git log` shows the only deploy in the window was a config-only+change at 15:35 that doubled the log verbosity (see also+`backend-excessive-logging`, not this eval). `mem.used` flat.++## Task++> Eight unrelated endpoints all went 2–4× slower at 15:40 yesterday, with no+> single slow span. Diagnose to a specific checklist cause — and say what it+> is *not*.++## Expected diagnosis++- **Root cause**: CPU saturation — fleet-wide latency rise with no long+ pole, aligned exactly with the `system.cpu.utilization` step to 97%+.+- **Key evidence**: broad rise across endpoints (rules out a code bug in one+ path); no dominating span (rules out a slow query/upstream); metric+ inflection at the same timestamp.+- **What it is not**: not a code regression (config-only deploy, no single+ culprit span); the doubled log verbosity is a *contributor* to CPU, worth+ fixing, but the finding is saturation.+- **Fix**: scale out / move the verbose logging off the hot path, then+ re-check headroom.++## Pass criteria++- Names **CPU saturation** and explicitly rules out a single code/query+ cause using the breadth of the rise and the metric alignment.+- Does not blame the config deploy as a code regression without evidence.diff --git a/skills/traceway/evals/mobile-android-gc-pauses.md b/skills/traceway/evals/mobile-android-gc-pauses.mdnew file mode 100644index 00000000..40718057--- /dev/null+++ b/skills/traceway/evals/mobile-android-gc-pauses.md@@ -0,0 +1,52 @@+# Eval: mobile-android-gc-pauses++- **ID**: `mobile-android-gc-pauses`+- **Area**: mobile+- **Framework**: Android (native, ART)+- **Checklist item**: Memory pressure / GC pauses (`performance.md` →+ Infrastructure and saturation)++## Telemetry snapshot++`scroll.jank` trace group (feed scroll): p50 frame time 22ms, p99 190ms.+The p99 frames cluster in sessions with >3 minutes of continuous scrolling.++`traceway traces show <traceId> --recorded-at <t>` — one janky scroll+window:++```+scroll.window ............ 2400ms+├── frame ................ 24ms+├── frame ................ 28ms+├── <gap: no spans> ...... 165ms ← ART GC pause (concurrent → blocking)+├── frame ................ 188ms+└── image.decode ......... 96ms (full-res bitmap, 4096×4096)+```++`mem.used` climbs monotonically across the session; the image cache has no+size cap (in-memory LRU unbounded). GC pause frequency tracks scroll depth,+not network or server state. Server spans for the same feed are fast+(p95 300ms).++## Task++> Android feed scroll p99 frame time is 190ms, worsening the longer the+> session runs. Server side is fine. Diagnose to a specific checklist cause.++## Expected diagnosis++- **Root cause**: Memory pressure / GC pauses — unbounded in-memory image+ cache grows with scroll depth; ART GC pauses surface as gaps in the frame+ timeline.+- **Key evidence**: the span-less gap (GC, not work); `mem.used` climbing+ with session length; full-res decodes feeding an uncapped cache; server+ side clean.+- **Fix**: cap the image cache (e.g. 128MB LRU), decode at display size,+ recycle bitmaps.++## Pass criteria++- Names **memory pressure / GC pauses** and reads the gap as GC rather than+ missing work.+- Connects the uncapped cache + `mem.used` climb to the pauses (not "slow+ images" alone — decode is 96ms, the 165ms gap is the killer).diff --git a/skills/traceway/evals/mobile-flutter-shader-jank.md b/skills/traceway/evals/mobile-flutter-shader-jank.mdnew file mode 100644index 00000000..bddfac4f--- /dev/null+++ b/skills/traceway/evals/mobile-flutter-shader-jank.md@@ -0,0 +1,47 @@+# Eval: mobile-flutter-shader-jank++- **ID**: `mobile-flutter-shader-jank`+- **Area**: mobile+- **Framework**: Flutter+- **Checklist item**: Cold start / JIT / connection warmup (`performance.md`+ → Caching and delivery; "latency normalizes after warmup")++## Telemetry snapshot++`route.transition` trace group (product detail page): p50 320ms, p99 2.1s.+The p99s are *first visits per install* to any page with the animated hero+carousel — `firstSeen`-correlated per route; second visits render at ~300ms.++`traceway traces show <traceId> --recorded-at <t>` — one first visit (2.0s):++```+route.transition ......... 2000ms+├── build/layout ......... 180ms+├── <gap: raster thread> . 1450ms ← shader compilation (SkSL warmup)+└── first-frame .......... 370ms+```++`system.cpu.utilization` on device spikes during the gap (raster thread+compiling shaders); no network spans in the window — the page data was+already cached. After the first visit, subsequent transitions are smooth.++## Task++> Flutter product-page transitions hit 2.1s p99, but only the first visit+> per install; data is cached and the network is idle. Diagnose to a+> specific checklist cause.++## Expected diagnosis++- **Root cause**: Shader/JIT warmup jank — first-frame SkSL compilation on+ the raster thread; classic warmup pattern (bad once, then fine).+- **Key evidence**: first-visit-only outliers; raster-thread gap with no+ spans and no network; CPU spike on device during the gap.+- **Fix**: ship SkSL warmup (`flutter build --bundle-sksl-path`), or+ pre-warm the carousel offscreen.++## Pass criteria++- Names **shader/JIT warmup** (not "slow build" — build/layout is 180ms;+ not network — idle).+- Recognizes the first-visit-only + normalizes-after-warmup signature.diff --git a/skills/traceway/evals/mobile-ios-cold-start.md b/skills/traceway/evals/mobile-ios-cold-start.mdnew file mode 100644index 00000000..1f09f00b--- /dev/null+++ b/skills/traceway/evals/mobile-ios-cold-start.md@@ -0,0 +1,49 @@+# Eval: mobile-ios-cold-start++- **ID**: `mobile-ios-cold-start`+- **Area**: mobile+- **Framework**: iOS (native)+- **Checklist item**: Cold start (`performance.md` → Caching and delivery;+ "first-request-after-deploy outliers … latency normalizes after warmup")++## Telemetry snapshot++Mobile `app.start` trace group, last 24h: p50 1.1s, p95 3.8s. The p95 is+entirely first launches after install/update — `firstSeen`-correlated; warm+starts sit at 0.4s.++`traceway traces show <traceId> --recorded-at <t>` — one cold start (3.6s):++```+app.start ................ 3600ms+└── didFinishLaunching ... 2200ms+│ ├── sdk.init ......... 900ms analytics + crash + attribution SDKs, sequential+│ ├── db.migrate ....... 700ms CoreData lightweight migration on main thread+│ └── config.fetch ..... 600ms synchronous remote-config fetch, blocks return+└── first-frame .......... 1400ms+```++No network saturation (config endpoint p95 120ms); the 600ms is one+blocking call plus main-thread contention. Subsequent launches skip the+migration and use cached config.++## Task++> iOS cold start p95 is 3.8s, but only on first launch after install/update.+> Diagnose to a specific checklist cause.++## Expected diagnosis++- **Root cause**: Cold start with synchronous init on the main thread —+ sequential SDK init, main-thread CoreData migration, and a blocking+ remote-config fetch inside `didFinishLaunching`.+- **Key evidence**: first-launch-only outliers (`firstSeen`-correlated);+ warm starts fine; the three init spans serialized on the main thread.+- **Fix**: defer non-critical SDK init off the critical path, move migration+ off main thread, make remote config async with cached fallback.++## Pass criteria++- Names **cold start / synchronous launch init** and ties it to+ first-launch-only data.+- Identifies all three serialized init spans (not just "slow SDK").