The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
Started structured evals as part of cli tests (#406)
narrow-adventureb2582cc426c1
diff --git a/.plugin-scanner.toml b/.plugin-scanner.tomlindex 159fbfa8..25466ea9 100644--- a/.plugin-scanner.toml+++ b/.plugin-scanner.toml@@ -6,6 +6,7 @@ ignore_paths = ["backend/static/frontend/*","benchmarks/*","cli/internal/state/state.go",+ "cli/test/evals/testdata/*","docs/*","examples/*","testing/*",diff --git a/CLAUDE.md b/CLAUDE.mdindex 84a2a7d9..dc955920 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -7,7 +7,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with coTraceway is an error tracking and monitoring platform consisting of:- **Frontend**: SvelteKit 2 dashboard application with Svelte 5- **Backend**: Go/Gin API server with ClickHouse database-- **CLI**: Go/Cobra command-line client for the backend HTTP API (`/cli`), built twice from one source: read-only `traceway`, and `traceway-write` (`-tags write`, own config dir and OAuth client, write commands in `cmd/traceway/edition_write.go`). Skills: `skills/traceway` (read), `skills/traceway-learn` (the product guide, one file per area from `knowledge/learn/`, also served as `traceway://knowledge/learn/*`) and `plugins/tw-write/skills/traceway-write` (write, separate plugin), all generated from `cli/pkg/mcpserver/knowledge` by `just gen-skills`+- **CLI**: Go/Cobra command-line client for the backend HTTP API (`/cli`), built twice from one source: read-only `traceway`, and `traceway-write` (`-tags write`, own config dir and OAuth client, write commands in `cmd/traceway/edition_write.go`). Skills: `skills/traceway` (read), `skills/traceway-learn` (the product guide, one file per area from `knowledge/learn/`, also served as `traceway://knowledge/learn/*`) and `plugins/tw-write/skills/traceway-write` (write, separate plugin), all generated from `cli/pkg/mcpserver/knowledge` by `just gen-skills`. The skill and MCP evals live in `cli/test/evals/` (own Go module, see its README), never under `skills/`: anything in a skill directory ships to users and to the agent under test---diff --git a/cli/test/evals/README.md b/cli/test/evals/README.mdnew file mode 100644index 00000000..7c4241f3--- /dev/null+++ b/cli/test/evals/README.md@@ -0,0 +1,57 @@+# Traceway skill and MCP evals++**Manual only.** Nothing here runs automatically, and nothing here calls a+model yet: this directory holds eval cases and their fixtures. The harness+that seeds a backend, drives an agent and grades the result is added next,+in its own Go module under this directory.++The cases started as the performance scenarios of PR #405+(`skills/traceway/evals/*.md`). They moved out of `skills/` because anything+in a skill directory ships to users and to the agent under test, which would+hand it the expected answers.++## What changed in the port++Each Markdown scenario became a structured case with one correct answer, one+named distractor, and telemetry that the real SDKs and the real ingest path+can produce:++| PR #405 file | Case | What was fixed |+|---|---|---|+| `backend-go-select-n-plus-1` | `perf-db-n-plus-1` | Consistent counts: N item queries for N orders |+| `backend-missing-index` | `perf-db-missing-index` | No `EXPLAIN` handed to the agent; a 30-day window that matches the claim |+| `backend-node-sequential-calls` | `perf-net-sequential` | No unexplained 700ms per hop |+| `backend-python-pool-exhaustion` | `perf-db-pool-exhaustion` | Pool-wait gaps that add up to the request time |+| `backend-sync-background-task` | `perf-app-sync-background` | Fix advice that does not depend on the retired `CaptureTask` |+| `frontend-js-large-payloads` | `perf-net-large-payload` | A real `http.response.body.size` attribute instead of a fictional `transfer` span |+| `frontend-js-no-cache` | `perf-cache-no-cache` | A browser project is present, and only one checklist row is correct |+| `infra-cpu-saturation` | `perf-infra-cpu-saturation` | Saturation driven by traffic, not by a deploy |+| `infra-cache-stampede` | `perf-cache-stampede` | Realistic request rates |+| `mobile-ios-cold-start` | `perf-xs-ios-watchdog` | Built only from telemetry the iOS SDK sends: a launch watchdog crash whose cause is a slow backend endpoint |+| `mobile-android-gc-pauses` | `perf-xs-android-timeouts`, `debug-android-npe` | The Android SDK sends crashes, actions and logs, not frame or GC metrics |+| `mobile-flutter-shader-jank` | `debug-flutter-null-check` | The Flutter SDK sends errors, not frame timings |++## Layout++```+cases/perf/<id>.yaml 11 performance cases+cases/debug/<id>.yaml 2 mobile debugging cases+testdata/repos/<repo>/ the fixture repository each case's agent works in+testdata/oracle/<case>/ files the reference trajectory writes (debug fixes)+```++A case names its seed (the Go function that will generate its telemetry),+its arms (`cli+skill`, `cli-noskill`), the prompt, budgets, the expected+answer, the tool calls that must, may and must not happen, a judge rubric,+and an `oracle`: the reference trajectory, one tool call per step, ending in+the structured answer.++A fixture repo can carry git history: `history.yaml` lists commits, each an+overlay under `.history/` (only the files that differ from the fixture, plus+an `absent:` list of files the commit lacks), with ages relative to the run so+`git log` lines up with the seeded deploys.++## Credits++The first performance scenarios were contributed by Liyu in PR #405+("feat(skills): add performance skill evals").diff --git a/cli/test/evals/cases/debug/debug-android-npe.yaml b/cli/test/evals/cases/debug/debug-android-npe.yamlnew file mode 100644index 00000000..90d43759--- /dev/null+++ b/cli/test/evals/cases/debug/debug-android-npe.yaml@@ -0,0 +1,80 @@+id: debug-android-npe+suite: debug+title: Android feed crashes while scrolling since the 5.8.0 rollout+tags: [mobile, android, kotlin]+seed: debug/android-npe+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: An OutOfMemoryError group in the full-size image viewer (FullImageActivity.decodeOriginal) on low-memory phones has been around for weeks on every version; the feed crash is new with 5.8.0 and is an NPE in the adapter.+workspace: {repo: feed-android}+prompt: |+ /traceway debug {{seed.hash}}+ Android users report the app crashing while they scroll the home feed. Find the cause and fix it.+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 14000, timeout: 10m}+user:+ - default: "I don't know more than that. Please fix it in the app."+expect:+ answer:+ issue_hash: "{{seed.hash}}"+ category: null_reference+ culprit: {file: app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt, symbol: bindImage, line: 55}+ needs_instrumentation: false+ mentions_all: ["text[- ]?(only )?posts?|without (an )?image|no image|null image|image (is|was) null|image (field|object|key) is (missing|absent|omitted)", "5\\.8\\.0"]+ grounding:+ - "FeedAdapter.kt:55"+ calls:+ required:+ - {op: exceptions.show, args: {hash: "{{seed.hash}}"}}+ forbidden:+ - {op: "*", write: true}+ order:+ - [exceptions.show, write_file]+ - [exceptions.show, edit_file]+ max_failed_calls: 5+ max_identical_repeats: 2+ code:+ static: true+ touches_culprit: true+ must_change: [app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt]+ must_not_change: [app/src/main/java/com/example/pinboard/media/FullImageActivity.kt]+judge:+ expected: >-+ The group is a NullPointerException ("Attempt to invoke virtual method 'int ...ImageRef.getWidth()' on a null object+ reference") in FeedAdapter.bindImage (FeedAdapter.kt:55), reached from onBindViewHolder during RecyclerView prefetch+ while the feed scrolls, only on 5.8.0, first seen with its rollout. 5.8.0 made FeedItem.image non-null and sizes the+ thumbnail from item.image.width; text posts have no image and Gson ignores Kotlin nullability, so image is null at+ runtime. The fix declares image nullable and hides the thumbnail for text posts (keeping the aspect-ratio sizing for+ image posts). The OutOfMemoryError in the full-size viewer is an older, separate issue.+ rubric:+ - Reads the deobfuscated trace to the exact line and explains why a non-null Kotlin property is null (Gson, text posts without an image).+ - Ties the crash to 5.8.0 (only version, first seen at the rollout, the git change that removed the null check).+ - The diff makes image nullable or otherwise handles a missing image in bindImage, without touching the unrelated image viewer.+oracle:+ - shell: >-+ traceway exceptions show {{seed.hash}} --page-size 30 --output json+ | jq '{group: (.group | {exceptionHash, firstSeen, lastSeen, count, stackTrace}), versions: ([.occurrences[].appVersion] | unique),+ devices: ([.occurrences[].serverName] | unique), occurrences: [.occurrences[0] | {id, recordedAt, appVersion, serverName}]}'+ - shell: >-+ traceway exceptions occurrence {{calls.c01.occurrences.0.id}} --recorded-at {{calls.c01.occurrences.0.recordedAt}} --output json+ | jq '{exception: (.exception | {recordedAt, appVersion}), recording: .sessionRecording | {actions, logs}}'+ - shell: git log --format='%h %ad %s' --date=iso -5+ - read_file: app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt+ - read_file: app/src/main/java/com/example/pinboard/feed/FeedItem.kt+ - write_file: {path: app/src/main/java/com/example/pinboard/feed/FeedItem.kt, from: oracle/debug-android-npe/FeedItem.kt}+ - write_file: {path: app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt, from: oracle/debug-android-npe/FeedAdapter.kt}+ - submit:+ issue_hash: "{{seed.hash}}"+ category: null_reference+ culprit: {file: app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt, symbol: bindImage, line: 55}+ evidence:+ - {call_id: c01, claim: "NullPointerException on ImageRef.getWidth() in FeedAdapter.bindImage (FeedAdapter.kt:55) from onBindViewHolder during RecyclerView prefetch, only on 5.8.0"}+ - {call_id: c02, claim: "the crash comes right after a feed page loads while scrolling"}+ - {call_id: c03, claim: "5.8.0 changed feed images to their own aspect ratio"}+ - {call_id: c04, claim: "bindImage reads item.image.width with no check for a missing image"}+ - {call_id: c05, claim: "FeedItem.image is declared non-null, but text posts carry no image and Gson does not enforce Kotlin nullability"}+ fix_summary: FeedItem.image is nullable; bindImage hides the thumbnail for posts without an image and keeps the aspect-ratio sizing for the others.+ files_changed: [app/src/main/java/com/example/pinboard/feed/FeedItem.kt, app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt]+ needs_instrumentation: false+ confidence: high+ report: |+ The feed crash is a NullPointerException in FeedAdapter.bindImage (FeedAdapter.kt:55): "Attempt to invoke virtual method 'int com.example.pinboard.feed.ImageRef.getWidth()' on a null object reference", hit from onBindViewHolder while RecyclerView prefetches the next posts during scrolling. It only happens on 5.8.0 and started with its rollout. 5.8.0 sizes thumbnails from item.image.width and made FeedItem.image non-null, but text posts have no image and Gson fills fields without Kotlin's null checks, so image is null for them. I made image nullable again and bindImage hides the thumbnail when there is no image. The OutOfMemoryError in the full-size image viewer is a separate, older issue on low-memory phones.diff --git a/cli/test/evals/cases/debug/debug-flutter-null-check.yaml b/cli/test/evals/cases/debug/debug-flutter-null-check.yamlnew file mode 100644index 00000000..58852c66--- /dev/null+++ b/cli/test/evals/cases/debug/debug-flutter-null-check.yaml@@ -0,0 +1,81 @@+id: debug-flutter-null-check+suite: debug+title: Flutter product pages fail with a null check on listings without photos+tags: [mobile, flutter, dart]+seed: debug/flutter-null-check+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: >-+ No app release lines up with the crash (it hits 4.2.1 and 4.3.0 alike, and 4.3.0 shipped nine days ago); a steady+ trickle of "Failed host lookup" ClientExceptions from offline phones is unrelated.+workspace: {repo: shop-flutter}+prompt: |+ /traceway debug {{seed.hash}}+ For about a day now, some product pages in the app show a grey screen instead of the product. Find the cause and fix it.+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 14000, timeout: 10m}+user:+ - default: "I don't know more than that. Please fix it in the app."+expect:+ answer:+ issue_hash: "{{seed.hash}}"+ category: null_reference+ culprit: {file: lib/features/product/widgets/product_hero_carousel.dart, symbol: build, line: 24}+ needs_instrumentation: false+ mentions_all: ["media", "4\\.2\\.1|both (app )?versions|every (app )?version|no (app )?release|not (an? )?(app )?release|catalog|listing"]+ grounding:+ - "product_hero_carousel.dart:24"+ calls:+ required:+ - {op: exceptions.show, args: {hash: "{{seed.hash}}"}}+ forbidden:+ - {op: "*", write: true}+ order:+ - [exceptions.show, write_file]+ - [exceptions.show, edit_file]+ max_failed_calls: 5+ max_identical_repeats: 2+ code:+ static: true+ touches_culprit: true+ must_change: [lib/features/product/widgets/product_hero_carousel.dart]+ must_not_change: [lib/core/api_client.dart]+judge:+ expected: >-+ The group is "_TypeError: Null check operator used on a null value" in ProductHeroCarousel.build+ (product_hero_carousel.dart:24), building the product page's carousel. It hits app versions 4.2.1 and 4.3.0 alike and+ began about a day ago, with no app release then: the data changed, not the app. Product.media is documented as+ omitted when a listing has no media yet, and the carousel null-checks it (product.media!.first). The fix renders a+ placeholder (or nothing) for products without media instead of the null check. The host-lookup failures are an+ unrelated, steady background of offline phones.+ rubric:+ - Points at the null check on product.media at line 24 and explains that the catalog omits media for some listings.+ - Notices that both app versions crash and no release lines up, so the trigger is new data rather than an app change.+ - The diff handles a missing or empty media list in the carousel without changing the API client.+oracle:+ - shell: >-+ traceway exceptions show {{seed.hash}} --page-size 30 --output json+ | jq '{group: (.group | {exceptionHash, firstSeen, lastSeen, count, stackTrace}), versions: ([.occurrences[].appVersion] | group_by(.) | map({version: .[0], n: length})),+ occurrences: [.occurrences[0] | {id, recordedAt, appVersion, serverName}]}'+ - shell: >-+ traceway exceptions occurrence {{calls.c01.occurrences.0.id}} --recorded-at {{calls.c01.occurrences.0.recordedAt}} --output json+ | jq '{exception: (.exception | {recordedAt, appVersion}), actions: .sessionRecording.actions}'+ - shell: git log --format='%h %ad %s' --date=iso -5+ - read_file: lib/features/product/widgets/product_hero_carousel.dart+ - read_file: lib/features/product/models/product.dart+ - write_file: {path: lib/features/product/widgets/product_hero_carousel.dart, from: oracle/debug-flutter-null-check/product_hero_carousel.dart}+ - submit:+ issue_hash: "{{seed.hash}}"+ category: null_reference+ culprit: {file: lib/features/product/widgets/product_hero_carousel.dart, symbol: build, line: 24}+ evidence:+ - {call_id: c01, claim: "_TypeError: Null check operator used on a null value in ProductHeroCarousel.build (product_hero_carousel.dart:24), on both 4.2.1 and 4.3.0, first seen about a day ago"}+ - {call_id: c02, claim: "it happens right after the product page loads GET /v1/products/{id}"}+ - {call_id: c03, claim: "the last app release, 4.3.0, shipped nine days ago, so no release lines up with the onset"}+ - {call_id: c04, claim: "the carousel does product.media!.first"}+ - {call_id: c05, claim: "Product.media is omitted by the catalog when a listing has no media yet"}+ fix_summary: The carousel treats a missing media list as empty and shows a placeholder instead of null-checking product.media.+ files_changed: [lib/features/product/widgets/product_hero_carousel.dart]+ needs_instrumentation: false+ confidence: high+ report: |+ The grey screen is Flutter's error widget for "_TypeError: Null check operator used on a null value" in ProductHeroCarousel.build (lib/features/product/widgets/product_hero_carousel.dart:24), which does product.media!.first. Product.media is omitted by the catalog for listings that have no photos yet. The crash hits app versions 4.2.1 and 4.3.0 alike and started about a day ago, while the last release shipped nine days ago, so the trigger is new catalog data (listings without media), not an app change. I made the carousel treat a missing media list as empty and show a placeholder. The "Failed host lookup" errors are a steady, unrelated background from phones without connectivity.diff --git a/cli/test/evals/cases/perf/perf-app-sync-background.yaml b/cli/test/evals/cases/perf/perf-app-sync-background.yamlnew file mode 100644index 00000000..a5151ca3--- /dev/null+++ b/cli/test/evals/cases/perf/perf-app-sync-background.yaml@@ -0,0 +1,79 @@+id: perf-app-sync-background+suite: perf+title: Generating a statement holds the request open while the PDF renders+tags: [application, go]+seed: perf/app-sync-background+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: Host CPU jumps to ~80% whenever a statement renders, which reads like saturation, but average CPU is ~20%, the endpoint is just as slow when the host is otherwise idle, and every other endpoint on the host stays fast. The spike is the render itself, not the cause of the latency.+workspace: {repo: statements-api-go}+prompt: |+ /traceway perf POST /api/statements+ Customers say clicking "Generate statement" in the billing dashboard hangs for ten seconds or more before anything happens, and on flaky connections the request sometimes times out and they click again. Find where the time goes and tell me what we should change.+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: synchronous_work_that_belongs_in_a_background_task+ not: [cpu_saturation, cpu_bound_serialization, slow_third_party_upstream_api, inefficient_algorithm_o_n2_hot_path]+ locus: {endpoint: "POST /api/statements", span_contains: "render"}+ grounding:+ - "render statement pdf"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~system\\.cpu"}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Synchronous work that belongs in a background task. Create renders the statement PDF in the request (a 6-11s "render statement pdf" span, ~9.4s p50), uploads it to S3 (~0.3s) and only then answers 201, although the dashboard already polls GET /api/statements/{id} for the status. Every request pays it (p50 close to p95), it has been the same all week, and the DB and S3 spans are small. CPU reaches ~80% only while a render runs and averages ~20%; other endpoints are fast, so it is not saturation. Fix - insert the statement as queued, enqueue a render job for a worker (a durable queue or job table, with the worker's processing span instrumented as an OTel CONSUMER span so it shows up as a task), answer 202 with the id, and let the existing poll pick up ready/failed.+ rubric:+ - Attributes the latency to the in-request PDF render span (6-11s) that runs before the response, and notes the client polls for the result anyway, so the user does not need to wait on it.+ - Rules out CPU saturation (the CPU spikes are the renders themselves; average CPU is low and other endpoints are fast) and a slow upstream (the S3 PUT is a few hundred ms).+ - Proposes answering 202 right after inserting the statement and rendering in a worker or queue consumer, with the status endpoint reporting progress; does not suggest the retired Traceway Go SDK's CaptureTask API.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: traceway logs query --since 6h --search /api/statements --page-size 10 --output json | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, status: .endpoint.statusCode, total_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, kind: .spanKind, ms: (.duration/1e6|round), detail: (.attributes["db.query.text"] // .attributes["url.full"] // .attributes["statement.pages"])}]}'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ | jq '.results[0].series | map_values(map(.value * 100 | round))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by state --since 24h --interval-minutes 360 --output json+ | jq '.results[0].series | map_values(map(.value * 100 | round))'+ - shell: >-+ traceway endpoints chart --metric-type p50 --since 7d --interval-minutes 1440 --output json+ | jq '[.series[] | select(.endpoint == "POST /api/statements") | {t: .timestamp, p50_ms: (.value|round)}]'+ - read_file: internal/statements/handler.go+ - read_file: web/statements.js+ - submit:+ checklist_item: synchronous_work_that_belongs_in_a_background_task+ category: application+ locus: {endpoint: "POST /api/statements", span: "render statement pdf", service: statements-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "POST /api/statements has a ~9s p50 and a p95 close to it, so every request pays the cost; the other endpoints answer in milliseconds"}+ - {call_id: c03, claim: "the slowest request spends almost all of its time in the in-process 'render statement pdf' span before the S3 PUT and the 201; the DB queries and the upload are small"}+ - {call_id: c04, claim: "CPU peaks around 80% only at the moments a render runs"}+ - {call_id: c05, claim: "average CPU is around 20%, so the host is not saturated"}+ - {call_id: c06, claim: "the p50 has been ~9s every day of the week, so nothing regressed"}+ ruled_out:+ - {checklist_item: cpu_saturation, call_id: c05, reason: "average CPU ~20%; the spikes are the renders themselves and the other endpoints stay fast"}+ - {checklist_item: slow_third_party_upstream_api, call_id: c03, reason: "the S3 PUT takes a few hundred milliseconds"}+ fix: Insert the statement as queued, hand the render and upload to a bounded worker pool (or a durable queue consumer whose job span is an OTel CONSUMER span), answer 202 with the id right away, and let the dashboard's existing poll of GET /api/statements/{id} pick up ready or failed.+ confidence: high+ report: |+ POST /api/statements is slow because it renders the statement PDF inside the request. The slowest request of the last day spent ~12s of its ~12.1s in the in-process "render statement pdf" span, then ~0.3s uploading to S3, before answering 201. The p50 is ~9.4s and has been the same all week. The dashboard already polls GET /api/statements/{id} for the status (web/statements.js), so nobody needs to wait on the render. CPU reaches ~80% only while a render runs and averages ~20%, and every other endpoint is fast, so this is not saturation. I changed Create to insert the statement, queue the render for two worker goroutines and answer 202 at once; the workers mark the statement ready or failed, which the existing poll picks up. A durable queue would survive restarts; instrument the worker with an OTel CONSUMER span so renders show up as tasks in Traceway.diff --git a/cli/test/evals/cases/perf/perf-cache-no-cache.yaml b/cli/test/evals/cases/perf/perf-cache-no-cache.yamlnew file mode 100644index 00000000..812a49e3--- /dev/null+++ b/cli/test/evals/cases/perf/perf-cache-no-cache.yaml@@ -0,0 +1,79 @@+id: perf-cache-no-cache+suite: perf+title: The catalog endpoint keeps the database busy all day+tags: [caching, node, react, multi-project, browser]+seed: perf/cache-no-cache+arms: [cli+skill, cli-noskill]+invoke: natural+distractor: The catalog query is the long pole of every request (~45ms of ~55ms over ~1200 rows), which reads like a slow query, but it is steady and unremarkable per request; the load is volume times repetition, ~9000 identical builds a day of a catalog that changes twice a day, because the cache never hits.+workspace: {repo: shop-catalog}+prompt: |+ Our catalog database is busy all day and the DB team traced most of the load to GET /api/catalog. No single request is slow, it is just constant. The catalog only changes when merchandising publishes, about twice a day. Can you find out what is going on and what we should change? The storefront web app (the shop-web project) is in the same repo.+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: no_cache_low_hit_ratio+ not: [expensive_join_or_aggregation, missing_or_unused_index, over_fetching, cpu_saturation]+ locus: {endpoint: "GET /api/catalog", service: catalog-api}+ grounding:+ - "SET catalog:/api/catalog?locale=en-US"+ - "cache.hit_ratio"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: No cache / low hit ratio. GET /api/catalog is by far the busiest endpoint (~9000 requests a day, each ~55ms, ~45ms of it the catalog query over ~1200 rows plus serialization of ~410KB). catalog-api does have a Redis cache, but keys it by the full URL (catalog-cache.js cacheKey(req.originalUrl)), and the web app's useCatalog hook fetches the catalog on every route change with a cache-busting _=Date.now() parameter, so every request misses (cache.hit_ratio for the catalog cache ~3%, against ~97% for the session cache), rebuilds and stores a key nobody reads again. The shop-web occurrences show the same catalog URL fetched again after every navigation. Fix by keying the cache by locale only, dropping the buster, fetching once per page load in the SPA, and sending Cache-Control so browsers and the CDN reuse it.+ rubric:+ - Shows that the cache exists but never hits (unique _= keys in the GET/SET spans, the low catalog cache.hit_ratio) and that the endpoint's cost is volume, not per-request slowness.+ - Uses the shop-web project's captured navigations and network actions to show the catalog is re-fetched on every route change with a cache buster.+ - Proposes keying the server cache by locale, removing the buster and fetching once per page load (plus HTTP caching), rather than tuning the query.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), total_s: (.avgDuration * .count / 1e9 | round)}] | sort_by(-.total_s)'+ - shell: >-+ traceway logs query --since 6h --search /api/catalog --page-size 5 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6), bodySize: .endpoint.bodySize, query: .endpoint.attributes["url.query"],+ spans: [.spans | sort_by(.startTime)[] | {name, ms: (.duration / 1e6), statement: (.attributes["db.query.text"] // null | if . then .[0:120] else . end), rows: .attributes["db.response.returned_rows"]}]}'+ - shell: >-+ traceway metrics query --name cache.hit_ratio --aggregation avg --group-by cache --since 24h --interval-minutes 360 --output json+ - shell: traceway projects list --output json+ - shell: >-+ traceway exceptions list --project {{env.projects.shop-web.id}} --since 24h --output json+ | jq '[.data[] | {exceptionHash, count, lastSeen, top: (.stackTrace | split("\n")[0])}]'+ - shell: >-+ traceway exceptions show {{seed.webHash}} --project {{env.projects.shop-web.id}} --page-size 3 --output json+ | jq '{occurrences: [.occurrences[] | {id, recordedAt, sessionId}]}'+ - shell: >-+ traceway exceptions occurrence {{calls.c07.occurrences.0.id}} --recorded-at {{calls.c07.occurrences.0.recordedAt}} --project {{env.projects.shop-web.id}} --output json+ | jq '{stack: (.exception.stackTrace | split("\n")[0]), actions: [.sessionRecording.actions[] | {timestamp, type, to, url, statusCode, durationMs, responseBytes, error}]}'+ - read_file: web/src/hooks/useCatalog.js+ - read_file: api/src/routes/catalog.js+ - submit:+ checklist_item: no_cache_low_hit_ratio+ category: caching+ locus: {endpoint: "GET /api/catalog", span: "GET catalog:/api/catalog?locale=en-US&_=<timestamp> (miss) then SELECT products and SET", service: catalog-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/catalog serves ~9000 requests a day at ~55ms and is by far the largest total time; nothing is slow per request"}+ - {call_id: c03, claim: "each request looks up a Redis key built from the full URL including a _=<timestamp> parameter, misses, runs the catalog query over ~1200 rows and stores a key no one will ask for again"}+ - {call_id: c04, claim: "the catalog cache hit ratio is ~3% (the session cache ~97%)"}+ - {call_id: c08, claim: "in the web app the same catalog URL, with a new _= timestamp, is fetched after every navigation, ~410KB each time"}+ ruled_out:+ - {checklist_item: expensive_join_or_aggregation, call_id: c03, reason: "the query is ~45ms and steady; the problem is running it on every request for data that changes twice a day"}+ fix: Key the catalog cache by locale only (ignore _ and other query noise), drop the cache buster and fetch the catalog once per page load in useCatalog, and send Cache-Control (public, max-age with stale-while-revalidate) from the endpoint.+ confidence: high+ report: |+ GET /api/catalog keeps the database busy because its cache never hits. catalog-api caches the response in Redis, but under a key built from the full URL (api/src/catalog-cache.js), and the web app's useCatalog hook (web/src/hooks/useCatalog.js) re-fetches the catalog on every route change with _=Date.now() appended. Every request therefore misses, runs the ~45ms catalog query over ~1200 products, serializes ~410KB and writes a key nobody reads again: ~9000 times a day for a catalog that changes twice. The catalog cache's hit ratio is ~3% (the session cache's is ~97%), and the shop-web error occurrences show the same catalog URL fetched again after every navigation. I keyed the cache by locale only and changed useCatalog to fetch once per page load without the buster; the endpoint should also send Cache-Control so browsers and the CDN reuse the response.diff --git a/cli/test/evals/cases/perf/perf-cache-stampede.yaml b/cli/test/evals/cases/perf/perf-cache-stampede.yamlnew file mode 100644index 00000000..9cbd6dc4--- /dev/null+++ b/cli/test/evals/cases/perf/perf-cache-stampede.yaml@@ -0,0 +1,78 @@+id: perf-cache-stampede+suite: perf+title: Home page requests stall for seconds in bursts+tags: [smoke, caching, go, redis]+seed: perf/cache-stampede+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: The slow requests run all twelve home queries against the database on the request path, which reads like a missing cache, but the cache is there and serves ~98% of requests in ~3ms; the slow requests all land in the seconds after the cached feed expires, when every request in flight rebuilds it at once.+workspace: {repo: home-api-go}+prompt: |+ /traceway perf GET /api/home+ The home endpoint is usually instant, but every few minutes a batch of customers waits 7 to 10 seconds for the home page, then it is fine again. What is causing it, and how do we fix it?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: cache_stampede_thundering_herd+ not: [no_cache_low_hit_ratio, missing_query_result_caching, expensive_join_or_aggregation, connection_pool_exhaustion, cpu_saturation]+ locus: {endpoint: "GET /api/home", service: home-api}+ grounding:+ - "home:feed:v3"+ - "ex 600"+ calls:+ required:+ - {op: endpoints.list}+ - {op: logs.query}+ any_of:+ - [{op: endpoints.chart}]+ - [{op: endpoints.list, args: {from: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Cache stampede / thundering herd. GET /api/home serves the feed from Redis (home:feed:v3, set with a 10-minute TTL, "ex 600"); p50 is ~3ms but p99 is ~7s. Every ~10 minutes, at the moment the key expires, every request that arrives before the first rebuild has written the key back misses and rebuilds the feed itself (12 section queries), so 15-25 identical ~7-10s traces start within the same few seconds; the concurrent rebuilds also slow each other down in the database. Between expiries the cache-hit ratio is ~98% and requests take a few ms, so the cache works. Fix with request coalescing (one rebuild per key, others wait for it, e.g. singleflight), plus early/background refresh and TTL jitter.+ rubric:+ - Identifies the periodicity (p99 spikes every ~10 minutes, matching the 600s TTL) and the burst of identical rebuild traces within seconds of each other.+ - Rules out "no cache" with the high hit ratio between bursts and the fast cache hits, and explains why the rebuilds are slow only during the burst.+ - Proposes coalescing concurrent misses (singleflight/lock), with early refresh or TTL jitter, in home.Service.Feed.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}] | sort_by(-.p99_ms)'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 2h --interval-minutes 5 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/home") | {t: .timestamp, p99_ms: (.value | round)}]'+ - shell: >-+ traceway logs query --from {{seed.spikeFrom}} --to {{seed.spikeTo}} --search /api/home --page-size 40 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.missRequest.traceId}} --recorded-at {{seed.missRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6),+ spans: [.spans | sort_by([.startTime, -.duration])[] | {name, ms: (.duration / 1e6 | round), statement: (.attributes["db.query.text"] // null | if . then .[0:90] else . end)}]}'+ - shell: >-+ traceway metrics query --name cache.hit_ratio --aggregation min --group-by cache --since 2h --interval-minutes 5 --output json+ | jq '.results[0].series | map_values(map({t: .timestamp, ratio: (.value * 1000 | round / 1000)}))'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 2h --interval-minutes 30 --output json+ - read_file: internal/home/service.go+ - submit:+ checklist_item: cache_stampede_thundering_herd+ category: caching+ locus: {endpoint: "GET /api/home", span: "get home:feed:v3 miss followed by buildHomeFeed (12 queries) and set home:feed:v3 ex 600, in many concurrent requests", service: home-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/home p50 ~3ms, p95 ~5ms, p99 ~7s: a rare but severe tail"}+ - {call_id: c02, claim: "p99 spikes to 7-10s in every other 5-minute bucket, i.e. every ~10 minutes"}+ - {call_id: c03, claim: "inside one spike, about twenty 7-10s requests start within the same ~10 seconds"}+ - {call_id: c04, claim: "each slow request misses home:feed:v3, runs the twelve section queries and sets the key back with ex 600 (a 10-minute TTL)"}+ - {call_id: c05, claim: "the home-feed cache hits ~98% of the time and dips only in the spike buckets"}+ - {call_id: c06, claim: "CPU is moderate and flat"}+ ruled_out:+ - {checklist_item: no_cache_low_hit_ratio, call_id: c05, reason: "hit ratio ~98% between spikes; hits take a few ms"}+ - {checklist_item: cpu_saturation, call_id: c06, reason: "CPU flat"}+ fix: Coalesce misses in home.Service.Feed so one rebuild per key runs and concurrent misses wait for it (singleflight), and add TTL jitter or an early background refresh so the key never expires under load.+ confidence: high+ report: |+ The bursts are a cache stampede on the home feed. GET /api/home serves home:feed:v3 from Redis and is ~3ms; the key is set with a 10-minute TTL (ex 600). When it expires, every request that arrives before the first rebuild finishes misses and rebuilds the feed itself, running the twelve section queries; at ~2.4 requests per second that is about twenty identical rebuilds in a few seconds, and they slow each other down in the database, so each takes 7-10s. That is why p99 spikes every ~10 minutes while the cache-hit ratio stays ~98% in between. I changed home.Service.Feed so concurrent misses share a single rebuild (the others wait for its result) and the TTL gets a little jitter; a background refresh before expiry would remove the wait entirely.diff --git a/cli/test/evals/cases/perf/perf-db-missing-index.yaml b/cli/test/evals/cases/perf/perf-db-missing-index.yamlnew file mode 100644index 00000000..6c47b1b0--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-missing-index.yaml@@ -0,0 +1,80 @@+id: perf-db-missing-index+suite: perf+title: The user lookup gets slower every week as the users table grows+tags: [database, python, fastapi]+seed: perf/db-missing-index+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: Release 1.13.0 two weeks ago changed POST /api/invites (two primary-key reads and an email enqueue, a few milliseconds); the lookup's curve climbs steadily through the whole month with no step at the release, and the release did not touch the lookup.+workspace: {repo: accounts-api-py}+prompt: |+ /traceway perf GET /api/users/lookup+ The member lookup behind the invite dialog and the admin user search has been getting slower for about a month. It's around half a second now for everyone, not only the big orgs. We shipped the invite email change two weeks ago, but nobody touched the lookup itself. What is slowing it down, and what should we change?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: missing_or_unused_index+ not: [select_n_plus_1, over_fetching, no_pagination_unbounded_result_set, expensive_join_or_aggregation, cpu_saturation, slow_third_party_upstream_api]+ locus: {endpoint: "GET /api/users/lookup", span_contains: "users"}+ grounding:+ - "WHERE users.email = $1"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}, {op: endpoints.chart}]+ - [{op: traces.show, args: {recorded_at: present}}, {op: endpoints.chart}]+ - [{op: endpoints.show, args: {recorded_at: present}}, {op: endpoints.list, args: {from: present}}]+ - [{op: traces.show, args: {recorded_at: present}}, {op: endpoints.list, args: {from: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, traces.show]+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Missing index on users.email. Every lookup runs one query, SELECT ... FROM users WHERE users.email = $1, that returns one row (or none) yet takes about 500ms in the last day and about 250ms a month ago; it grows smoothly with the users table, so Postgres is reading the whole table. Primary-key reads of users (GET /api/me, the invite flow) stay at 1-2ms, CPU is flat around 20-25%, and the 1.13.0 release two weeks ago only changed POST /api/invites. The model and migrations index org_id but not email. Fix with CREATE INDEX CONCURRENTLY on users(email) (an Alembic migration plus index=True on the model).+ rubric:+ - Attributes the latency to the single users-by-email query, which returns at most one row but takes hundreds of milliseconds and grows with the table.+ - Rules out the 1.13.0 release (no step in the lookup's curve, it only changed POST /api/invites) and saturation (flat CPU).+ - Proposes an index on users.email, created concurrently (migration and model), and says how to confirm it (the span drops to a few ms).+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6)}]'+ - shell: >-+ traceway logs query --since 6h --search /api/users/lookup --page-size 5 --output json+ | jq '{data: [.data[] | {traceId, timestamp, body}]}'+ - shell: >-+ traceway traces show {{calls.c02.data.0.traceId}} --recorded-at {{calls.c02.data.0.timestamp}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, id: .endpoint.id, recordedAt: .endpoint.recordedAt, version: .endpoint.appVersion, ms: (.endpoint.duration / 1e6),+ spans: [.spans[] | {name, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: >-+ traceway endpoints chart --metric-type p50 --since 29d --interval-minutes 10080 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/users/lookup" or .endpoint == "POST /api/invites") | {endpoint, t: .timestamp, p50_ms: .value}]'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - shell: git log --format='%h %ad %s' --date=iso --stat -3+ - read_file: app/users.py+ - read_file: app/models.py+ - submit:+ checklist_item: missing_or_unused_index+ category: database+ locus: {endpoint: "GET /api/users/lookup", span: "SELECT users ... WHERE users.email = $1::VARCHAR", service: accounts-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/users/lookup has p50 and p95 both around half a second, so every request pays it"}+ - {call_id: c03, claim: "the request spends almost all its time in one users query filtered by email that returns a single row; the orgs primary-key read next to it takes about a millisecond"}+ - {call_id: c04, claim: "the lookup's p50 roughly doubled over four weeks, climbing every week with no step at the release; POST /api/invites only moved by a few ms"}+ - {call_id: c05, claim: "CPU stayed flat all day, so this is not saturation"}+ - {call_id: c08, claim: "users.email has no index (only org_id is indexed in the model and the migrations)"}+ ruled_out:+ - {checklist_item: select_n_plus_1, call_id: c03, reason: "one users query per request"}+ - {checklist_item: no_pagination_unbounded_result_set, call_id: c03, reason: "the query returns at most one row"}+ - {checklist_item: cpu_saturation, call_id: c05, reason: "CPU flat"}+ fix: Add an index on users(email) with CREATE INDEX CONCURRENTLY (Alembic migration 0003, index=True on the model), then confirm the users span drops to a few milliseconds.+ confidence: high+ report: |+ GET /api/users/lookup is slow because users.email has no index. Each lookup runs one query, SELECT ... FROM users WHERE users.email = $1, which returns one row or none but takes about half a second: Postgres reads the whole users table every time. The query got slower every week as the table grew (p50 roughly doubled over the month) with no step at the 1.13.0 release, which only touched POST /api/invites. Primary-key reads of users and orgs take a millisecond or two and CPU is flat. I added migration 0003 that creates ix_users_email concurrently and marked the column indexed in the model.diff --git a/cli/test/evals/cases/perf/perf-db-n-plus-1.yaml b/cli/test/evals/cases/perf/perf-db-n-plus-1.yamlnew file mode 100644index 00000000..d37d48ef--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-n-plus-1.yaml@@ -0,0 +1,68 @@+id: perf-db-n-plus-1+suite: perf+title: Orders endpoint slows down as customers accumulate orders+tags: [smoke, database, go]+seed: perf/db-n-plus-1+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: The orders query selects every column (over-fetching), but it takes 3-6ms; CPU stays flat and nothing was deployed in the window.+workspace: {repo: orders-api-go}+prompt: |+ /traceway perf GET /api/orders+ Customers with a long order history say the orders page takes over a second to load, and it has been getting worse for weeks. Find the cause and tell me how to fix it.+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: select_n_plus_1+ not: [missing_or_unused_index, over_fetching, no_pagination_unbounded_result_set, cpu_saturation]+ locus: {endpoint: "GET /api/orders", span_contains: "order_items"}+ grounding:+ - "FROM order_items WHERE order_id"+ calls:+ required:+ - {op: endpoints.list}+ - {op: endpoints.show, args: {recorded_at: present}}+ any_of:+ - [{op: metrics.query, args: {name: "~system\\.cpu"}}]+ - [{op: endpoints.chart}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Select N+1. ListOrders loads up to 100 orders and then runs one order_items query per order (60-90 per request in the last day, 6-14ms each). Orders per customer grew over the month, so latency grew without a deploy; CPU is flat around 25%.+ rubric:+ - Attributes the latency to one order_items query per order and quantifies it (dozens of queries per request).+ - Rules out saturation (flat CPU) and a deploy regression (gradual climb, no release in the window).+ - Proposes a batched load (JOIN, IN (...) or ANY) of the items in ListOrders.+oracle:+ - shell: traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ - shell: traceway logs query --since 6h --search "/api/orders " --page-size 5 --output json | jq -r '.data[] | "\(.traceId) \(.body)"'+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{duration: .endpoint.duration, spans: ([.spans[] | {q: (.attributes["db.query.text"] // .name), ms: (.duration / 1e6)}]+ | group_by(.q) | map({query: .[0].q, count: length, total_ms: (map(.ms) | add)}))}'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - shell: traceway endpoints chart --metric-type p95 --since 29d --interval-minutes 10080 --output json | jq '[.series[] | select(.endpoint == "GET /api/orders")]'+ - read_file: internal/handlers/orders.go+ - submit:+ checklist_item: select_n_plus_1+ category: database+ locus: {endpoint: "GET /api/orders", span: "SELECT id, sku, qty, unit_price FROM order_items WHERE order_id = $1", service: orders-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/orders is the slowest endpoint; p50 and p95 are both high, so every request pays the cost"}+ - {call_id: c03, claim: "the slowest request of the last day runs one order_items query per order, dozens per request, which make up most of its duration"}+ - {call_id: c04, claim: "CPU stayed flat all day, so this is not saturation"}+ - {call_id: c05, claim: "p95 climbed week over week with no step, which is data growth rather than a deploy"}+ ruled_out:+ - {checklist_item: over_fetching, call_id: c03, reason: "the SELECT * on orders takes a few milliseconds"}+ - {checklist_item: cpu_saturation, call_id: c04, reason: "CPU flat"}+ fix: Load the items of all listed orders in one query (WHERE order_id IN (...) or a JOIN) instead of one query per order in ListOrders.+ confidence: high+ report: |+ GET /api/orders is slow because of a Select N+1 in ListOrders (internal/handlers/orders.go): after loading up to 100 orders it runs one order_items query per order. The slowest request of the last day ran dozens of those queries, which make up most of its time. CPU is flat and nothing shipped in the window; the latency grew as customers accumulated orders. I batched the items into one IN (...) query (store.ItemsByOrders).diff --git a/cli/test/evals/cases/perf/perf-db-pool-exhaustion.yaml b/cli/test/evals/cases/perf/perf-db-pool-exhaustion.yamlnew file mode 100644index 00000000..feb66a5d--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-pool-exhaustion.yaml@@ -0,0 +1,84 @@+id: perf-db-pool-exhaustion+suite: perf+title: Checkout stalls for seconds at the lunch and evening rushes+tags: [smoke, database, python, fastapi]+seed: perf/db-pool-exhaustion+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: A slow database or the payment provider. The inventory query takes 40-60ms and the payment call 150-260ms at every hour, rushes included; the seconds go to a gap before the first query, and GET /api/cart, which only runs a 3ms query on the same pool, stalls at the same hours.+workspace: {repo: checkout-api-py}+prompt: |+ /traceway perf POST /api/checkout+ Checkout is fine most of the day, but during the lunch and evening rushes some customers wait 10+ seconds and a few get a 500. The median looks normal, it's the tail. Our payment provider's status page is green the whole time. Here's one of the slow ones from today: {{env.url}}/endpoints/POST%20%2Fapi%2Fcheckout/{{seed.slowRequest.id}}?t={{seed.slowRequest.recordedAt}}+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: connection_pool_exhaustion+ not: [slow_third_party_upstream_api, missing_or_unused_index, expensive_join_or_aggregation, lock_transaction_contention, cpu_saturation, no_timeouts_aggressive_retries]+ locus: {endpoint: "POST /api/checkout", service: checkout-api}+ grounding:+ - "QueuePool limit of size 10 overflow 5 reached"+ calls:+ required:+ - {op: endpoints.show, args: {id: "{{seed.slowRequest.id}}", recorded_at: present}}+ - {op: endpoints.list}+ any_of:+ - [{op: logs.query}]+ - [{op: exceptions.list}]+ - [{op: exceptions.show}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Connection-pool exhaustion. The slow request spends most of its time in a gap before its first query (seconds with no span), then runs its normal work (inventory query 40-60ms, payment call ~200ms). Checkout holds its SQLAlchemy session, and so a pooled connection, for the whole request including the payment provider call, so at the lunch and evening rushes the 15 connections (pool_size 10 + max_overflow 5) are all in use (db.client.connection.count used=15, idle=0) and requests queue for one; the few that wait the whole 30s pool_timeout fail with sqlalchemy.exc.TimeoutError ("QueuePool limit of size 10 overflow 5 reached, connection timed out, timeout 30.00"). p50 stays ~300ms while p99 is many seconds, only in the rush hours; GET /api/cart, whose only query takes 3ms, stalls the same way. CPU is moderate. Fix by releasing the connection before the payment call (short transactions around it), then size the pool (and pool_timeout) for the peak concurrency.+ rubric:+ - Points at the gap before the first DB span as time waiting for a connection, not query time, and quantifies it.+ - Uses the QueuePool timeout logs/TimeoutError issues and the rush-hour-only p99 (or the pool metric at 15 used, 0 idle) to confirm the pool, and rules out the database and the payment provider with their per-span timings.+ - Proposes not holding the session across the payment call (two short transactions), and only then pool sizing.+oracle:+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq 'def secs: .[11:] | rtrimstr("Z") | split(":") | ((.[0] | tonumber) * 3600 + (.[1] | tonumber) * 60 + (.[2] | tonumber));+ .endpoint.recordedAt as $t0 | {endpoint: .endpoint.endpoint, status: .endpoint.statusCode, ms: (.endpoint.duration / 1e6 | round),+ spans: [.spans[] | {name, start_ms: (((.startTime | secs) - ($t0 | secs)) * 1000 | round), ms: (.duration / 1e6 | round), query: .attributes["db.query.text"], url: .attributes["url.full"]}]}'+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 2d --interval-minutes 60 --output json+ | jq '[.series[] | select(.endpoint == "POST /api/checkout" or .endpoint == "GET /api/cart") | {endpoint, t: .timestamp, p99_ms: (.value | round)}]'+ - shell: >-+ traceway logs query --since 24h --min-severity 13 --page-size 8 --output json+ | jq '{data: [.data[] | {timestamp, severityText, traceId, body, exception: .logAttributes["exception.message"]}]}'+ - shell: >-+ traceway exceptions list --since 3d --output json+ | jq '[.data[] | {hash: .exceptionHash, count, firstSeen, lastSeen, top: (.stackTrace | split("\n") | last)}]'+ - shell: >-+ traceway metrics query --name db.client.connection.count --group-by db.client.connection.state --aggregation max --since 24h --interval-minutes 60 --output json+ | jq '.results[0].series | map_values([.[] | {t: .timestamp, v: (.value | round)}])'+ - read_file: app/checkout.py+ - read_file: app/db.py+ - submit:+ checklist_item: connection_pool_exhaustion+ category: database+ locus: {endpoint: "POST /api/checkout", span: "gap before the first SELECT cart_items (QueuePool checkout)", service: checkout-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "the slow request waits seconds before its first query starts; the queries and the payment call after that take their usual few hundred ms"}+ - {call_id: c02, claim: "checkout p50 is ~300ms while p95 and p99 are many seconds; GET /api/cart, a single 3ms query, has a multi-second p99 too"}+ - {call_id: c03, claim: "the p99 spikes only in the lunch and evening rush hours, for both endpoints"}+ - {call_id: c04, claim: "QueuePool limit of size 10 overflow 5 reached, connection timed out, timeout 30.00 in the WARN/ERROR logs at the rushes"}+ - {call_id: c05, claim: "sqlalchemy.exc.TimeoutError issues on checkout and cart"}+ - {call_id: c06, claim: "the pool sits at 15 connections used and 0 idle in exactly those hours"}+ - {call_id: c07, claim: "checkout keeps one session (and its connection) open across payments.charge"}+ ruled_out:+ - {checklist_item: slow_third_party_upstream_api, call_id: c01, reason: "the payment call takes ~200ms even in the slow request; the time is before the first query"}+ - {checklist_item: missing_or_unused_index, call_id: c01, reason: "the inventory query is 40-60ms, and GET /api/cart stalls although its query is 3ms"}+ - {checklist_item: lock_transaction_contention, call_id: c01, reason: "the DB spans themselves are normal; the wait happens before any query runs"}+ fix: Release the connection before the payment call (read and check stock in one short session, charge, then write the order in a second short transaction), then size pool_size/max_overflow for the rush concurrency and lower pool_timeout so overload fails fast.+ confidence: high+ report: |+ Checkout's tail is connection-pool exhaustion. The slow request waits several seconds before its first query even starts; after that, the inventory query (~50ms) and the payment call (~200ms) take their usual time. Checkout holds its SQLAlchemy session, and with it a pooled connection, across the payment provider call, so at the lunch and evening rushes all 15 connections (pool_size 10 + max_overflow 5) are checked out and requests queue for one. The ones that wait the full 30s fail with sqlalchemy.exc.TimeoutError ("QueuePool limit of size 10 overflow 5 reached ..."). p50 stays ~300ms while p99 is many seconds only in those hours, GET /api/cart (one 3ms query on the same pool) stalls the same way, and the pool metric shows 15 used, 0 idle. I split checkout into a short read session, the charge with no connection held, and a short write transaction.diff --git a/cli/test/evals/cases/perf/perf-infra-cpu-saturation.yaml b/cli/test/evals/cases/perf/perf-infra-cpu-saturation.yamlnew file mode 100644index 00000000..7c252f06--- /dev/null+++ b/cli/test/evals/cases/perf/perf-infra-cpu-saturation.yaml@@ -0,0 +1,92 @@+id: perf-infra-cpu-saturation+suite: perf+title: Traffic tripled and every endpoint slowed down with the CPU pinned+tags: [smoke, infrastructure, node]+seed: perf/infra-cpu-saturation+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: A code regression; but service.version never changed, the last commit is nine days old, and every endpoint got 2-4x slower at the same moment the request rate tripled, with CPU on both instances going from ~25% to ~95% at that time. The waterfalls show the same spans, just stretched, plus wait time between them; no new or longer query.+workspace: {repo: commerce-api-node}+prompt: |+ /traceway perf+ For about a day now the whole commerce API has been sluggish, every page and not one in particular, and it hasn't recovered on its own. Find out what's going on and what we should do about it.+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: cpu_saturation+ not: [connection_pool_exhaustion, slow_third_party_upstream_api, memory_pressure_gc_pauses, excessive_logging_telemetry_in_the_hot_path, cold_start_jit_connection_warmup]+ locus: {service: commerce-api}+ onset: {at: "{{seed.onsetAt}}", tolerance_minutes: 60}+ grounding:+ - "system.cpu.utilization"+ - "GET /api/products/:id"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~system\\.cpu"}}+ any_of:+ - [{op: endpoints.chart}]+ - [{op: endpoints.list, args: {from: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, metrics.query]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: CPU saturation. At ~14:20 UTC the request rate on every commerce-api endpoint tripled within about 40 minutes and stayed there. In the same window CPU on both instances went from ~25% busy to 90-97% at the daily peak (idle near 0-8%), and every endpoint's p50 and p95 rose 2-4x together (GET /api/products/:id p95 ~14ms to ~40ms), more at peak hours than at night. Waterfalls contain the same spans as before, each a bit longer, with growing wait between them; no span is a new long pole. There was no deploy (same service.version, last commit nine days ago). Fix - add capacity now (more replicas; the deployment is fixed at 2 with 1-CPU limits) and add a CPU-based HorizontalPodAutoscaler so the next traffic jump scales out; then look at per-request CPU cost.+ rubric:+ - Shows that the latency rise is broad (every endpoint, p50 and p95 alike) and coincides with both the CPU climbing to ~95% and the request count tripling.+ - Rules out a code regression (no new version or commit, no new long-pole span) and a single slow dependency.+ - Recommends scaling out (more replicas / autoscaling on CPU) rather than a code fix to one endpoint.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 3d --interval-minutes 180 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, p95_ms: map("\(.timestamp[5:16]) \(.value|round)")})'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by state --since 3d --interval-minutes 180 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[5:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation min --tag state=idle --group-by host.name --since 24h --interval-minutes 180 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[5:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway endpoints list --from {{seed.beforeFrom}} --to {{seed.beforeTo}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --from {{seed.afterFrom}} --to {{seed.afterTo}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints show {{seed.earlierRequest.id}} --recorded-at {{seed.earlierRequest.recordedAt}} --output json+ | jq '{version: .endpoint.appVersion, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, start: .startTime[11:23], ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{version: .endpoint.appVersion, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, start: .startTime[11:23], ms: ((.duration/1e5|round)/10)}]}'+ - shell: git log --date=iso-strict --format='%h %ad %an %s' -n 5+ - read_file: deploy/k8s/deployment.yaml+ - submit:+ checklist_item: cpu_saturation+ category: infrastructure+ locus: {service: commerce-api, host: "both instances"}+ onset: "{{seed.onsetAt}}"+ evidence:+ - {call_id: c01, claim: "every endpoint is slower, not one: the rise is broad"}+ - {call_id: c02, claim: "all top endpoints stepped up in the same bucket and stayed up, peaking each afternoon"}+ - {call_id: c03, claim: "CPU busy went from ~25% to over 90% in the same bucket"}+ - {call_id: c04, claim: "both instances are near zero idle at peak, so it is fleet-wide, not one host"}+ - {call_id: c05, claim: "the day before, the same hours had a third of the requests at the old latency"}+ - {call_id: c06, claim: "since the step, request counts are about 3x and p50/p95 2-4x on every endpoint"}+ - {call_id: c07, claim: "before: the same two queries, the request takes ~10-15ms"}+ - {call_id: c08, claim: "after: the same two queries, a little longer each, plus wait time before and between them; no new long pole and the same version"}+ - {call_id: c09, claim: "no commit since nine days ago"}+ ruled_out:+ - {checklist_item: connection_pool_exhaustion, call_id: c08, reason: "the wait is spread across the whole request and every endpoint, including the ones without database calls, and tracks CPU"}+ - {checklist_item: slow_third_party_upstream_api, call_id: c06, reason: "endpoints without any upstream call slowed down as much"}+ fix: Scale out now (raise replicas from 2), and add a CPU-based HorizontalPodAutoscaler (target ~60%) so traffic jumps add pods instead of queueing; afterwards profile per-request CPU.+ confidence: high+ report: |+ commerce-api is CPU-saturated by a traffic increase, not broken by a change. At about 14:20 UTC the request rate on every endpoint tripled within ~40 minutes and has stayed there. In the same window CPU on both instances went from ~25% busy to 90-97% at the daily peak, and every endpoint's p50 and p95 rose 2-4x together (GET /api/products/:id p95 ~14ms to ~40ms), worst in the afternoon and better at night. A request after the step runs the same two queries as before, each a bit longer, with wait time before and between them, which is time spent queued for the CPU. service.version did not change and the last commit is nine days old. The deployment is pinned at 2 replicas with 1-CPU limits; scale it out now and add the HorizontalPodAutoscaler I put in deploy/k8s/hpa.yaml (CPU target 60%, 2-10 replicas).diff --git a/cli/test/evals/cases/perf/perf-net-large-payload.yaml b/cli/test/evals/cases/perf/perf-net-large-payload.yamlnew file mode 100644index 00000000..7d3215e7--- /dev/null+++ b/cli/test/evals/cases/perf/perf-net-large-payload.yaml@@ -0,0 +1,75 @@+id: perf-net-large-payload+suite: perf+title: The feed is slow, worst on phones+tags: [network, go]+seed: perf/net-large-payload+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: The feed query runs on every request with nothing cached in front of it, which reads like a missing cache, but it takes ~15ms; the request time is the ~2.3MB body going out, and it tracks the body size and the client's link, not the database.+workspace: {repo: feed-api-go}+prompt: |+ /traceway perf GET /api/feed+ The feed is slow to load, and it is worst for people on their phones. The DB team says the feed query only takes a few milliseconds. What is going on, and what should we change?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: large_request_response_payloads+ not: [no_cache_low_hit_ratio, missing_query_result_caching, cpu_bound_serialization, cpu_saturation]+ locus: {endpoint: "GET /api/feed", service: feed-api}+ grounding:+ - "http.response.body.size"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Large request/response payloads. GET /api/feed returns 50 cards with their images inlined as base64 data URIs (feed_cards.image_data and the author avatars), so every response is ~2.3MB (http.response.body.size). The feed query takes ~15ms and ranking a few ms; the rest of the server span, after the last child span, is the body being written, which blocks on the client's link, so phones (okhttp, iPhone) take 1.2-2.5s while desktops take 150-300ms for the same size, and the small card endpoint is fast. Not a cache problem and not CPU (flat). Fix by sending CDN image URLs instead of inline data (and not selecting image_data), which cuts the response to tens of KB.+ rubric:+ - Ties the latency to the response size (http.response.body.size ~2.3MB) and shows that it follows the client's bandwidth (phone vs desktop, the small card endpoint), with the DB and handler work small.+ - Rules out a missing cache or a slow query from the data (the query is ~15ms) and CPU saturation.+ - Proposes shrinking the payload (CDN image URLs instead of inline base64, thumbnails, paging or compression) in the feed handler/store.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}] | sort_by(-.p95_ms)'+ - shell: >-+ traceway logs query --since 6h --search /api/feed --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.mobileRequest.traceId}} --recorded-at {{seed.mobileRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6), bodySize: .endpoint.bodySize,+ body_size_attr: .endpoint.attributes["http.response.body.size"], user_agent: .endpoint.attributes["user_agent.original"],+ spans: [.spans[] | {name, start: .startTime, ms: (.duration / 1e6)}]}'+ - shell: >-+ traceway traces show {{seed.desktopRequest.traceId}} --recorded-at {{seed.desktopRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6), bodySize: .endpoint.bodySize,+ body_size_attr: .endpoint.attributes["http.response.body.size"], user_agent: .endpoint.attributes["user_agent.original"],+ spans: [.spans[] | {name, start: .startTime, ms: (.duration / 1e6)}]}'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - read_file: internal/feed/handler.go+ - read_file: internal/feed/store.go+ - submit:+ checklist_item: large_request_response_payloads+ category: network+ locus: {endpoint: "GET /api/feed", span: "GET /api/feed (time after the last child span: writing the ~2.3MB body)", service: feed-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/feed p50 ~490ms, p95 ~2.6s; the single-card endpoint is fast"}+ - {call_id: c03, claim: "a phone request took well over a second with a ~2.3MB body (http.response.body.size); its query and ranking take a few tens of ms, the rest is writing the body"}+ - {call_id: c04, claim: "a desktop request with a body of the same size took a few hundred ms: latency follows size over the client's bandwidth"}+ - {call_id: c05, claim: "CPU is flat"}+ ruled_out:+ - {checklist_item: no_cache_low_hit_ratio, call_id: c03, reason: "the feed query takes ~15ms; caching it would not move a multi-second request"}+ - {checklist_item: cpu_saturation, call_id: c05, reason: "CPU flat"}+ fix: Stop inlining images; send CDN URLs for card images and avatars (and stop selecting image_data/avatar_data), so the feed is tens of KB instead of ~2.3MB.+ confidence: high+ report: |+ GET /api/feed is slow because each response is about 2.3MB: the handler inlines every card image and avatar as a base64 data URI (feed_cards.image_data, authors.avatar_data; internal/feed/handler.go and store.go). The feed query takes ~15ms and ranking a few ms; the remaining time of the request, after the last child span, is the body being written to the client, so it depends on the client's link. A phone request took well over a second and a desktop request a few hundred ms for the same ~2.3MB, which is why mobile users see it most. CPU is flat and the query is fast, so caching would not help. I changed the feed to return CDN URLs for card images and avatars and stopped selecting the inline image columns; the response drops to tens of KB.diff --git a/cli/test/evals/cases/perf/perf-net-sequential.yaml b/cli/test/evals/cases/perf/perf-net-sequential.yamlnew file mode 100644index 00000000..7e24c23d--- /dev/null+++ b/cli/test/evals/cases/perf/perf-net-sequential.yaml@@ -0,0 +1,75 @@+id: perf-net-sequential+suite: perf+title: Portal dashboard is slow although every service behind it is fast+tags: [smoke, network, node, nestjs, multi-service]+seed: perf/net-sequential+arms: [cli+skill, cli-noskill]+invoke: slash+distractor: recommendations-service is the longest single hop, which reads like a slow upstream, but it is an internal service running at its own steady p95 (no spikes in the window) and it is only about a third of the request; the four hops add up.+workspace: {repo: portal-web-api-nest}+prompt: |+ /traceway perf GET /api/dashboard+ The portal home page is the slowest page we have. The dashboard API takes around 300ms, sometimes 400, and we budgeted 150ms for it. The services behind it all look healthy on their own dashboards. What is making it slow, and how do we fix it?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: sequential_calls_that_could_run_in_parallel+ not: [slow_third_party_upstream_api, chatty_ipc_microservice_fan_out, cpu_saturation, expensive_join_or_aggregation]+ locus: {endpoint: "GET /api/dashboard", service: web-api}+ grounding:+ - "GET /internal/notifications/unread"+ - "GET /internal/recommendations"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, traces.show]+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Sequential calls that could run in parallel. DashboardService.load (src/dashboard/dashboard.service.ts) awaits four internal reads one after another (accounts profile ~45ms, billing summary ~75ms, unread notifications ~40ms, recommendations ~100ms at p50); each needs only the account id. The waterfall is a staircase where each CLIENT span starts when the previous one ends, so the dashboard pays the sum (~280ms p50) instead of the slowest hop (~100ms). Every downstream endpoint is steady at its own p95 over three days, CPU is flat. Fix with Promise.all over the four calls.+ rubric:+ - Shows the staircase (each downstream call starts only when the previous one finished) and that the four calls are independent (all keyed by the account id).+ - Quantifies it with the downstream services' own latencies (their endpoint percentiles add up to the dashboard's) and rules out a slow upstream (no single hop dominates or spikes) and saturation.+ - Proposes running the four calls concurrently (Promise.all) in DashboardService.load, with the expected latency of roughly the slowest call.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}] | sort_by(-.p95_ms)'+ - shell: >-+ traceway logs query --since 12h --search /api/dashboard --page-size 5 --output json+ | jq '{data: [.data[] | {traceId, timestamp, body}]}'+ - shell: >-+ traceway traces show {{calls.c02.data.0.traceId}} --recorded-at {{calls.c02.data.0.timestamp}} --output json+ | jq '{nodes: [.nodes[] | {endpoint: .endpoint.endpoint, service: .endpoint.serverName, start: .endpoint.recordedAt, ms: (.endpoint.duration / 1e6)}],+ calls: [.nodes[0].spans[] | select(.spanKind == 3 and .serviceName == "web-api") | {url: .attributes["url.full"], start: .startTime, ms: (.duration / 1e6)}]}'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 3d --interval-minutes 720 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, p95_ms: map(.value | round)})'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - read_file: src/dashboard/dashboard.service.ts+ - submit:+ checklist_item: sequential_calls_that_could_run_in_parallel+ category: network+ locus: {endpoint: "GET /api/dashboard", span: "GET http://recommendations-service.internal:8080/internal/recommendations (fourth of four sequential calls)", service: web-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "the dashboard p50 is about the sum of the four internal endpoints' own p50s (profile, billing summary, unread notifications, recommendations), each of which is fast"}+ - {call_id: c03, claim: "each of the four CLIENT calls starts only when the previous one has returned: a staircase, and all four take the same account id"}+ - {call_id: c04, claim: "the dashboard and every downstream endpoint are flat over three days: no upstream spike or regression"}+ - {call_id: c05, claim: "web-api CPU is flat and low"}+ ruled_out:+ - {checklist_item: slow_third_party_upstream_api, call_id: c04, reason: "recommendations is the longest hop but steady at its own p95 and only about a third of the request; no hop dominates"}+ - {checklist_item: cpu_saturation, call_id: c05, reason: "CPU flat"}+ fix: Run the four independent reads concurrently with Promise.all in DashboardService.load, so the dashboard waits for the slowest service (~100ms) instead of the sum (~280ms).+ confidence: high+ report: |+ GET /api/dashboard is slow because DashboardService.load (src/dashboard/dashboard.service.ts) awaits its four internal reads one after another. The trace is a staircase: accounts profile, then billing summary, then unread notifications, then recommendations, each starting when the previous one returned, although all four only need the account id. Each service is fast and steady on its own (their endpoint p50s add up to the dashboard's ~280ms), CPU is flat, and nothing changed over three days. I changed load() to issue the four calls with Promise.all; the endpoint should drop to roughly the slowest call, about 100ms.diff --git a/cli/test/evals/cases/perf/perf-xs-android-timeouts.yaml b/cli/test/evals/cases/perf/perf-xs-android-timeouts.yamlnew file mode 100644index 00000000..c64484bf--- /dev/null+++ b/cli/test/evals/cases/perf/perf-xs-android-timeouts.yaml@@ -0,0 +1,94 @@+id: perf-xs-android-timeouts+suite: perf+title: Android checkout crashes with socket timeouts while the checkout API waits on row locks+tags: [cross-surface, android, kotlin, java, database]+seed: perf/xs-android-timeouts+arms: [cli+skill, cli-noskill]+invoke: natural+distractor: An app bug. The crash is an uncaught java.net.SocketTimeoutException from CheckoutRepository (OkHttp's 10s read timeout, no catch in the view model), so catching it or raising the timeout looks like the fix. But each crash's network action is POST /api/checkout giving up at 10s, and on commerce-api that endpoint's p95 went from under a second to about 7s when the sale started; the time is inside update inventory (queued on the sale item's row lock, which every checkout holds through its payment call), not before the first query (no pool wait) and not in the payment call itself, and deadlock errors appear in the same hours.+workspace: {repo: shop-android-and-api}+prompt: |+ The Android app started crashing at checkout this afternoon, right when the autumn sale went live. Here is the crash in Traceway: {{env.url}}/issues/{{seed.hash}}+ Can you work out why it happens and what we should fix?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I only have the crash reports. Please find out from the data."+expect:+ answer:+ checklist_item: lock_transaction_contention+ not: [connection_pool_exhaustion, none_found, slow_third_party_upstream_api, memory_pressure_gc_pauses, cpu_saturation, no_timeouts_aggressive_retries]+ locus: {endpoint: "POST /api/checkout", span_contains: "inventory"}+ grounding:+ - "SocketTimeoutException"+ - "update inventory set reserved"+ calls:+ required:+ - {op: exceptions.show, args: {hash: "{{seed.hash}}"}}+ any_of:+ - [{op: "endpoints.*", args: {project_id: "{{env.projects.commerce-api.id}}"}}]+ - [{op: "logs.query", args: {project_id: "{{env.projects.commerce-api.id}}"}}]+ - [{op: projects.use, args: {project_id: "{{env.projects.commerce-api.id}}"}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: >-+ The crash is an uncaught java.net.SocketTimeoutException: OkHttp's default 10s read timeout fires on POST+ /api/checkout (the occurrence's network action gives up at ~10s with no status) and the view model does not catch+ it. The slowness is on commerce-api: since the sale started about three hours ago POST /api/checkout has p50 about+ 1.6s and p95 about 7s (it was under a second before), with traffic up many times. Slow checkouts spend their time+ inside "update inventory set reserved=reserved+? ...": checkouts of the sale item queue on that inventory row's lock,+ which CheckoutService.placeOrder holds for the whole transaction, including the payment call. There is no gap before+ the first query (the pool is fine), the payment call takes its usual ~0.5s, CPU is moderate, and Postgres reports+ "deadlock detected" for carts that lock two rows in opposite order. Lock / transaction contention. Fix: keep the+ row lock short (reserve and commit, charge outside the transaction, then confirm or release; or a conditional+ decrement without holding the lock across the payment), lock SKUs in a fixed order, and on the app catch the+ timeout and retry the checkout idempotently.+ rubric:+ - Follows the Android crash to POST /api/checkout on the backend project by the request URL and the times, and quantifies its latency since the sale.+ - Shows the time is spent inside the inventory UPDATE under the write peak (not a pool wait before the first query, not the payment call), backed by the deadlock logs.+ - Proposes shortening the locking transaction (payment outside it, consistent lock order) as the main fix, with crash-proof handling in the app as secondary.+oracle:+ - shell: traceway exceptions show {{seed.hash}} --page-size 5 --output json+ - shell: traceway exceptions occurrence {{calls.c01.occurrences.0.id}} --recorded-at {{calls.c01.occurrences.0.recordedAt}} --output json+ - shell: traceway projects list --output json+ - shell: >-+ traceway endpoints list --project {{env.projects.commerce-api.id}} --search /api/checkout --since 24h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --project {{env.projects.commerce-api.id}} --metric-type p95 --since 12h --interval-minutes 30 --output json+ | jq '[.series[] | select(.endpoint == "POST /api/checkout") | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: >-+ traceway logs query --project {{env.projects.commerce-api.id}} --search /api/checkout --since 4h --page-size 50 --output json+ | jq -r '.data[] | select((.logAttributes.duration_ms // "0" | tonumber) > 8000) | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {project: .projectName, endpoint: .endpoint.endpoint, start: .endpoint.recordedAt, duration_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, start: .startTime, ms: (.duration/1e6|round), query: .attributes["db.query.text"], url: .attributes["url.full"]}]}]'+ - shell: >-+ traceway logs query --project {{env.projects.commerce-api.id}} --min-severity 13 --since 4h --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.severityText) \(.traceId) \(.body | split("\n")[0])"'+ - shell: traceway metrics query --project {{env.projects.commerce-api.id}} --name system.cpu.utilization --aggregation max --group-by state --since 12h --interval-minutes 120 --output json+ - read_file: backend/src/main/java/com/example/shop/checkout/CheckoutService.java+ - read_file: android/app/src/main/java/com/example/shop/checkout/CheckoutRepository.kt+ - submit:+ checklist_item: lock_transaction_contention+ category: database+ locus: {endpoint: "POST /api/checkout", span: "update inventory set reserved=reserved+? where sku=? and available-reserved>=?", service: checkout-api}+ onset: "{{seed.saleAt}}"+ evidence:+ - {call_id: c02, claim: "the crash is an uncaught SocketTimeoutException; its network action is POST /api/checkout giving up at ~10s with no response"}+ - {call_id: c04, claim: "on commerce-api, POST /api/checkout has p95 of several seconds over the last day"}+ - {call_id: c05, claim: "its p95 jumped from under a second to about 7s when the sale started"}+ - {call_id: c06, claim: "checkouts finishing in 10-20s at the times the app crashed"}+ - {call_id: c07, claim: "a slow checkout starts its first query at once and then waits seconds inside update inventory; the payment call takes about half a second"}+ - {call_id: c08, claim: "deadlock detected errors on the same statement during the sale"}+ - {call_id: c09, claim: "CPU is moderate, not saturated"}+ ruled_out:+ - {checklist_item: connection_pool_exhaustion, call_id: c07, reason: "no gap before the first query; the wait is inside the UPDATE"}+ - {checklist_item: slow_third_party_upstream_api, call_id: c07, reason: "the payment call is about half a second; the UPDATE takes seconds"}+ fix: Shorten the locking transaction in CheckoutService.placeOrder (reserve and commit, charge the card outside the transaction, then confirm or release the reservation), take row locks in a fixed SKU order to stop the deadlocks, and in the app catch the timeout and retry the checkout with an idempotency key instead of crashing.+ confidence: high+ report: |+ The Android crashes are the client side of lock waits in the checkout API. The app crashes with an uncaught `java.net.SocketTimeoutException`: OkHttp's 10s read timeout fires on POST /api/checkout (the occurrence's network action gives up after ~10s) and the view model does not catch it. On commerce-api, POST /api/checkout went from p95 under a second to about 7s when the sale started, with traffic up many times. Slow checkouts run their first query immediately (no pool wait) and then sit for seconds inside `update inventory set reserved=reserved+? ...`: every checkout of the sale item needs that row's lock, and CheckoutService.placeOrder holds it for the whole transaction, payment call included, so they queue. Postgres also reports `deadlock detected` for carts that lock two rows in opposite order. Fix the transaction: reserve and commit first, charge outside the transaction, then confirm or release; lock SKUs in a fixed order. In the app, catch the timeout and retry with an idempotency key so a slow checkout does not crash the app or double-charge.diff --git a/cli/test/evals/cases/perf/perf-xs-ios-watchdog.yaml b/cli/test/evals/cases/perf/perf-xs-ios-watchdog.yamlnew file mode 100644index 00000000..f1d2e521--- /dev/null+++ b/cli/test/evals/cases/perf/perf-xs-ios-watchdog.yaml@@ -0,0 +1,95 @@+id: perf-xs-ios-watchdog+suite: perf+title: iOS launch watchdog kills while the config endpoint waits for database connections+tags: [cross-surface, ios, swift, go, database]+seed: perf/xs-ios-watchdog+arms: [cli+skill, cli-noskill]+invoke: natural+distractor: A pure client bug. The crash is a launch watchdog termination (0x8badf00d) with the main thread blocked in ConfigClient.fetchBlocking, a synchronous GET /api/config with a 30s timeout inside didFinishLaunching, and that is a real client defect. But it has shipped for three weeks without crashing; the kills started with two waves of launches today, when GET /api/config on mobile-api began taking 20-30s, all of it waiting before the first query for one of the API's 8 database connections (the feature-flag queries themselves take a few milliseconds), with POST /api/devices and the feed queued in the same waves.+workspace: {repo: shop-ios-and-api}+prompt: |+ Since earlier today a bunch of iOS users say the app freezes on the splash screen and then closes. We haven't shipped an iOS release in three weeks. This is the crash in Traceway: {{env.url}}/issues/{{seed.hash}}+ What is causing it, and what should we fix?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "That's all I know from support. Please look at the data."+expect:+ answer:+ checklist_item: connection_pool_exhaustion+ not: [synchronous_work_that_belongs_in_a_background_task, cold_start_jit_connection_warmup, none_found, cache_stampede_thundering_herd, slow_third_party_upstream_api, cpu_saturation]+ locus: {endpoint: "GET /api/config"}+ grounding:+ - "0x8badf00d"+ - "FROM feature_flags"+ calls:+ required:+ - {op: exceptions.show, args: {hash: "{{seed.hash}}"}}+ any_of:+ - [{op: "endpoints.*", args: {project_id: "{{env.projects.mobile-api.id}}"}}]+ - [{op: "logs.query", args: {project_id: "{{env.projects.mobile-api.id}}"}}]+ - [{op: projects.use, args: {project_id: "{{env.projects.mobile-api.id}}"}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: >-+ The crash is the launch watchdog (0x8badf00d, 20s wall-clock allowance) killing the app while the main thread waits+ in ConfigClient.fetchBlocking: RemoteConfig.loadAtLaunch calls GET /api/config synchronously from+ didFinishLaunching with a 30s timeout. That code is three weeks old; the kills started today in two waves (about+ five hours and about 1.5 hours ago). In those waves mobile-api's GET /api/config, POST /api/devices and GET /api/feed+ all see far more requests than usual (a burst of app launches, most likely a push campaign) and GET /api/config goes+ from a few milliseconds to 20-30s: the time is a gap before the first query (waiting for one of the 8 connections+ database/sql is capped at, which the feed holds for hundreds of milliseconds), while the feature_flags and+ app_versions queries take milliseconds. Connection-pool exhaustion on the API, made fatal by the blocking launch+ fetch. Fix both: on the API raise the pool size / give config its own pool or cache the config in memory (it is+ the same for every user of a platform), and on iOS load the config asynchronously with the cached or bundled+ flags and a short timeout, never blocking didFinishLaunching.+ rubric:+ - Reads the crash as the launch watchdog killing a main thread blocked on the synchronous config fetch, and connects it to GET /api/config on the backend project by the code path and the crash times.+ - Shows that GET /api/config's time in the crash windows is waiting before its first query (a pool wait), with traffic bursts on the launch endpoints and fast queries, rather than a slow query or a slow dependency.+ - Recommends fixing the API's pool/config caching and making the iOS launch fetch asynchronous, and explains why it started without an app release.+oracle:+ - shell: >-+ traceway exceptions show {{seed.hash}} --page-size 5 --output json+ | jq '{group: (.group | {exceptionHash, firstSeen, lastSeen, count, stackTrace}), occurrences: [.occurrences[] | {id, recordedAt, appVersion, serverName}]}'+ - shell: traceway projects list --output json+ - shell: >-+ traceway endpoints list --project {{env.projects.mobile-api.id}} --search /api/config --since 24h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --project {{env.projects.mobile-api.id}} --metric-type p99 --since 12h --interval-minutes 15 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/config" and .value > 1000) | {t: .timestamp, p99_ms: (.value|round)}]'+ - shell: >-+ traceway endpoints list --project {{env.projects.mobile-api.id}} --from {{seed.wave2From}} --to {{seed.wave2To}} --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway logs query --project {{env.projects.mobile-api.id}} --search /api/config --from {{seed.wave2From}} --to {{seed.wave2To}} --page-size 50 --output json+ | jq -r '.data[] | select((.logAttributes.duration_ms // "0" | tonumber) > 15000) | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {project: .projectName, endpoint: .endpoint.endpoint, start: .endpoint.recordedAt, duration_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, start: .startTime, ms: (.duration/1e6), query: .attributes["db.query.text"]}]}]'+ - shell: git log --format='%h %ad %s' --date=iso -3+ - read_file: ios/Shop/Config/ConfigClient.swift+ - read_file: api/cmd/server/main.go+ - submit:+ checklist_item: connection_pool_exhaustion+ category: database+ locus: {endpoint: "GET /api/config", span: "wait before SELECT key, value, rollout_pct FROM feature_flags (database/sql pool, 8 connections)", service: mobile-api}+ onset: "{{seed.push1}}"+ evidence:+ - {call_id: c01, claim: "launch watchdog terminations (0x8badf00d) with the main thread blocked in ConfigClient.fetchBlocking under didFinishLaunching, on app 4.12.0, first seen today"}+ - {call_id: c03, claim: "GET /api/config has p50 of a few milliseconds but p99 above 15s over the last day"}+ - {call_id: c04, claim: "its p99 spikes to tens of seconds in two short windows, about five hours and about 1.5 hours ago"}+ - {call_id: c05, claim: "in the second window /api/config, /api/devices and /api/feed all carry far more requests than usual and all are slow"}+ - {call_id: c06, claim: "config requests taking 20-30s in that window"}+ - {call_id: c07, claim: "a 20s+ config request spends almost all of it before its first query; the feature_flags and app_versions queries take milliseconds"}+ - {call_id: c10, claim: "the API caps database/sql at 8 open connections, shared with the feed"}+ ruled_out:+ - {checklist_item: synchronous_work_that_belongs_in_a_background_task, call_id: c08, reason: "the blocking launch fetch has shipped for three weeks; it only kills launches since the API started taking 20s+"}+ - {checklist_item: slow_third_party_upstream_api, call_id: c07, reason: "no outgoing call; the time is a wait before the first query"}+ fix: On mobile-api, stop the launch endpoints from queuing behind each other for 8 connections (raise MaxOpenConns to what Postgres can take, cache the per-platform config in memory so GET /api/config needs no connection, keep the feed off the same pool); on iOS, load the config asynchronously from the cached or bundled flags with a short timeout instead of blocking didFinishLaunching.+ confidence: high+ report: |+ The iOS crash is the launch watchdog (0x8badf00d, 20s wall-clock allowance) killing the app while its main thread waits in ConfigClient.fetchBlocking: RemoteConfig.loadAtLaunch calls GET /api/config synchronously from didFinishLaunching with a 30s timeout. That code is three weeks old, so what changed is the API. Today, in two waves (about five and about 1.5 hours ago), launches spiked (POST /api/devices, GET /api/config and the feed all jump, most likely a push campaign) and GET /api/config went from a few milliseconds to 20-30s. The slow requests spend nearly all of that before their first query, waiting for one of the 8 database connections the API allows, which the feed holds for hundreds of milliseconds; the feature_flags and app_versions queries themselves take milliseconds. That is connection-pool exhaustion, and the blocking launch fetch turns it into a crash. Fix both sides: on the API, size the pool for launch bursts and cache the per-platform config in memory so GET /api/config does not need a connection at all; on iOS, start from the cached or bundled flags and refresh the config asynchronously with a short timeout.diff --git a/cli/test/evals/testdata/oracle/debug-android-npe/FeedAdapter.kt b/cli/test/evals/testdata/oracle/debug-android-npe/FeedAdapter.ktnew file mode 100644index 00000000..a0a2159b--- /dev/null+++ b/cli/test/evals/testdata/oracle/debug-android-npe/FeedAdapter.kt@@ -0,0 +1,75 @@+package com.example.pinboard.feed++import android.view.LayoutInflater+import android.view.View+import android.view.ViewGroup+import android.widget.ImageView+import android.widget.TextView+import androidx.constraintlayout.widget.ConstraintLayout+import androidx.core.view.updateLayoutParams+import androidx.recyclerview.widget.DiffUtil+import androidx.recyclerview.widget.ListAdapter+import androidx.recyclerview.widget.RecyclerView+import coil.load+import coil.transform.CircleCropTransformation+import com.example.pinboard.R++/**+ * Binds feed posts. Each card shows the author, the text and, since 5.8.0,+ * the image at its own aspect ratio instead of a fixed square crop.+ */+class FeedAdapter(+ private val onOpen: (FeedItem) -> Unit,+) : ListAdapter<FeedItem, FeedAdapter.PostViewHolder>(DIFF) {++ class PostViewHolder(view: View) : RecyclerView.ViewHolder(view) {+ val avatar: ImageView = view.findViewById(R.id.avatar)+ val author: TextView = view.findViewById(R.id.author)+ val text: TextView = view.findViewById(R.id.text)+ val thumbnail: ImageView = view.findViewById(R.id.thumbnail)+ val likes: TextView = view.findViewById(R.id.likes)+ }++ override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): PostViewHolder {+ val view = LayoutInflater.from(parent.context).inflate(R.layout.item_post, parent, false)+ return PostViewHolder(view)+ }++ override fun onBindViewHolder(holder: PostViewHolder, position: Int) {+ val item = getItem(position)+ holder.author.text = item.author.name+ holder.avatar.load(item.author.avatarUrl) {+ crossfade(true)+ transformations(CircleCropTransformation())+ placeholder(R.drawable.avatar_placeholder)+ }+ holder.text.text = item.text+ holder.likes.text = holder.itemView.resources.getQuantityString(R.plurals.likes, item.likes, item.likes)+ bindImage(holder, item)+ holder.itemView.setOnClickListener { onOpen(item) }+ }++ private fun bindImage(holder: PostViewHolder, item: FeedItem) {+ val image = item.image+ if (image == null) {+ // Text posts have no image.+ holder.thumbnail.visibility = View.GONE+ return+ }+ holder.thumbnail.visibility = View.VISIBLE+ holder.thumbnail.updateLayoutParams<ConstraintLayout.LayoutParams> {+ dimensionRatio = "${image.width}:${image.height}"+ }+ holder.thumbnail.load(image.url) {+ crossfade(true)+ size(holder.thumbnail.width.coerceAtLeast(1), coil.size.Dimension.Undefined)+ }+ }++ private companion object {+ val DIFF = object : DiffUtil.ItemCallback<FeedItem>() {+ override fun areItemsTheSame(a: FeedItem, b: FeedItem) = a.id == b.id+ override fun areContentsTheSame(a: FeedItem, b: FeedItem) = a == b+ }+ }+}diff --git a/cli/test/evals/testdata/oracle/debug-android-npe/FeedItem.kt b/cli/test/evals/testdata/oracle/debug-android-npe/FeedItem.ktnew file mode 100644index 00000000..b8d6c38d--- /dev/null+++ b/cli/test/evals/testdata/oracle/debug-android-npe/FeedItem.kt@@ -0,0 +1,19 @@+package com.example.pinboard.feed++/**+ * One post of the home feed, as GET /v2/feed returns it (parsed by Gson).+ * Text posts carry no "image" object. Gson does not enforce Kotlin+ * nullability, so anything the API may omit must be declared nullable.+ */+data class FeedItem(+ val id: String,+ val author: Author,+ val text: String,+ val image: ImageRef?,+ val likes: Int,+ val createdAt: String,+)++data class Author(val id: String, val name: String, val avatarUrl: String?)++data class ImageRef(val url: String, val width: Int, val height: Int)diff --git a/cli/test/evals/testdata/oracle/debug-flutter-null-check/product_hero_carousel.dart b/cli/test/evals/testdata/oracle/debug-flutter-null-check/product_hero_carousel.dartnew file mode 100644index 00000000..c8ad7692--- /dev/null+++ b/cli/test/evals/testdata/oracle/debug-flutter-null-check/product_hero_carousel.dart@@ -0,0 +1,45 @@+import 'package:cached_network_image/cached_network_image.dart';+import 'package:flutter/material.dart';++import '../models/product.dart';++/// The swipeable hero images at the top of the product page.+class ProductHeroCarousel extends StatelessWidget {+ const ProductHeroCarousel({super.key, required this.product, this.height = 360});++ final Product product;+ final double height;++ static Widget _image(ProductMedia m) => CachedNetworkImage(+ imageUrl: m.url,+ fit: BoxFit.cover,+ placeholder: (_, __) => const ColoredBox(color: Color(0xFFF2F2F2)),+ );++ @override+ Widget build(BuildContext context) {+ // Size the carousel to the first image so portrait and landscape+ // photos both fill the width without letterboxing.+ final media = product.media ?? const <ProductMedia>[];+ if (media.isEmpty) {+ // Listings without photos yet (the catalog omits media) get a+ // placeholder instead of a carousel.+ return SizedBox(+ height: 240,+ child: ColoredBox(+ color: const Color(0xFFF2F2F2),+ child: Center(child: Icon(Icons.image_not_supported_outlined, size: 48, color: Colors.grey.shade500)),+ ),+ );+ }+ final ratio = MediaQuery.sizeOf(context).width;+ final first = media.first;+ final h = (ratio * first.height / first.width).clamp(240.0, height);+ return SizedBox(+ height: h,+ child: PageView(+ children: [for (final m in media) _image(m)],+ ),+ );+ }+}diff --git a/cli/test/evals/testdata/repos/accounts-api-py/.history/01/app/invites.py b/cli/test/evals/testdata/repos/accounts-api-py/.history/01/app/invites.pynew file mode 100644index 00000000..0f85ef75--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/.history/01/app/invites.py@@ -0,0 +1,59 @@+import secrets+from datetime import datetime, timezone++from fastapi import APIRouter, Depends, HTTPException+from pydantic import BaseModel, EmailStr+from sqlalchemy import select+from sqlalchemy.ext.asyncio import AsyncSession++from app.db import current_user, get_session+from app.models import Invite, OrgMember++router = APIRouter(prefix="/api")+++class InviteIn(BaseModel):+ org_id: int+ email: EmailStr+ role: str = "member"++[email protected]("/invites")+async def list_invites(org_id: int, _: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ rows = await session.execute(+ select(Invite.id, Invite.email, Invite.role, Invite.created_at)+ .where(Invite.org_id == org_id, Invite.accepted_at.is_(None))+ .order_by(Invite.created_at.desc())+ )+ return [dict(r._mapping) for r in rows]++[email protected]("/invites", status_code=201)+async def create_invite(body: InviteIn, user_id: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ role = (+ await session.execute(select(OrgMember.role).where(OrgMember.org_id == body.org_id, OrgMember.user_id == user_id))+ ).scalar_one_or_none()+ if role not in ("owner", "admin"):+ raise HTTPException(status_code=403, detail="only admins can invite")+ invite = Invite(+ org_id=body.org_id,+ email=body.email,+ role=body.role,+ token=secrets.token_urlsafe(32),+ invited_by=user_id,+ created_at=datetime.now(timezone.utc).replace(tzinfo=None),+ )+ session.add(invite)+ await session.commit()+ return {"id": invite.id, "token": invite.token}++[email protected]("/invites/{token}/accept")+async def accept_invite(token: str, user_id: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ invite = (await session.execute(select(Invite).where(Invite.token == token))).scalar_one_or_none()+ if invite is None or invite.accepted_at is not None:+ raise HTTPException(status_code=404, detail="invite not found")+ session.add(OrgMember(org_id=invite.org_id, user_id=user_id, role=invite.role))+ invite.accepted_at = datetime.now(timezone.utc).replace(tzinfo=None)+ await session.commit()+ return {"orgId": invite.org_id}diff --git a/cli/test/evals/testdata/repos/accounts-api-py/.history/01/requirements.txt b/cli/test/evals/testdata/repos/accounts-api-py/.history/01/requirements.txtnew file mode 100644index 00000000..0bd05ca2--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/.history/01/requirements.txt@@ -0,0 +1,10 @@+fastapi==0.118.0+uvicorn[standard]==0.37.0+sqlalchemy[asyncio]==2.0.43+asyncpg==0.30.0+alembic==1.16.5+pydantic[email]==2.11.9+opentelemetry-distro==0.58b0+opentelemetry-exporter-otlp==1.37.0+opentelemetry-instrumentation-fastapi==0.58b0+opentelemetry-instrumentation-sqlalchemy==0.58b0diff --git a/cli/test/evals/testdata/repos/accounts-api-py/README.md b/cli/test/evals/testdata/repos/accounts-api-py/README.mdnew file mode 100644index 00000000..db8b8640--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/README.md@@ -0,0 +1,19 @@+# accounts-api++Users, organizations and invites for Teamspace. Python 3.12, FastAPI,+SQLAlchemy 2.0 on PostgreSQL (asyncpg), Alembic migrations. Traces, logs and+metrics go to Traceway over OTLP through the OpenTelemetry auto-instrumentation+(`opentelemetry-instrument uvicorn app.main:app`).++```+pip install -r requirements.txt+alembic upgrade head+opentelemetry-instrument uvicorn app.main:app --port 8080+```++Routes:++- `GET /api/me`, `PATCH /api/me` – the signed-in user+- `GET /api/users/lookup?email=` – find a user by email (invite dialog, admin search)+- `GET /api/orgs/{org_id}/members` – members of an organization+- `GET /api/invites`, `POST /api/invites`, `POST /api/invites/{token}/accept`diff --git a/cli/test/evals/testdata/repos/accounts-api-py/app/db.py b/cli/test/evals/testdata/repos/accounts-api-py/app/db.pynew file mode 100644index 00000000..02465c0d--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/app/db.py@@ -0,0 +1,30 @@+import hmac+import os++from fastapi import Cookie, HTTPException+from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine++engine = create_async_engine(+ os.environ.get("DATABASE_URL", "postgresql+asyncpg://accounts@localhost/accounts"),+ pool_size=10,+ max_overflow=5,+ pool_pre_ping=True,+)+SessionLocal = async_sessionmaker(engine, expire_on_commit=False)+_SESSION_SECRET = os.environ.get("SESSION_SECRET", "dev").encode()+++async def get_session() -> AsyncSession:+ async with SessionLocal() as session:+ yield session+++async def current_user(session_id: str | None = Cookie(default=None)) -> int:+ """Session cookies are '<user id>.<hmac>'."""+ if not session_id or "." not in session_id:+ raise HTTPException(status_code=401, detail="not signed in")+ user_id, sig = session_id.split(".", 1)+ expected = hmac.new(_SESSION_SECRET, user_id.encode(), "sha256").hexdigest()+ if not hmac.compare_digest(sig, expected):+ raise HTTPException(status_code=401, detail="not signed in")+ return int(user_id)diff --git a/cli/test/evals/testdata/repos/accounts-api-py/app/invites.py b/cli/test/evals/testdata/repos/accounts-api-py/app/invites.pynew file mode 100644index 00000000..e15e9fa3--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/app/invites.py@@ -0,0 +1,71 @@+import json+import os+import secrets+from datetime import datetime, timezone++import redis.asyncio as redis+from fastapi import APIRouter, Depends, HTTPException+from pydantic import BaseModel, EmailStr+from sqlalchemy import select+from sqlalchemy.ext.asyncio import AsyncSession++from app.db import current_user, get_session+from app.models import Invite, Org, OrgMember, User++router = APIRouter(prefix="/api")+queue = redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379/0"))+++class InviteIn(BaseModel):+ org_id: int+ email: EmailStr+ role: str = "member"++[email protected]("/invites")+async def list_invites(org_id: int, _: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ rows = await session.execute(+ select(Invite.id, Invite.email, Invite.role, Invite.created_at)+ .where(Invite.org_id == org_id, Invite.accepted_at.is_(None))+ .order_by(Invite.created_at.desc())+ )+ return [dict(r._mapping) for r in rows]++[email protected]("/invites", status_code=201)+async def create_invite(body: InviteIn, user_id: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ role = (+ await session.execute(select(OrgMember.role).where(OrgMember.org_id == body.org_id, OrgMember.user_id == user_id))+ ).scalar_one_or_none()+ if role not in ("owner", "admin"):+ raise HTTPException(status_code=403, detail="only admins can invite")+ invite = Invite(+ org_id=body.org_id,+ email=body.email,+ role=body.role,+ token=secrets.token_urlsafe(32),+ invited_by=user_id,+ created_at=datetime.now(timezone.utc).replace(tzinfo=None),+ )+ session.add(invite)+ await session.commit()++ # The invite email names who sent it and which organization it is for.+ org = await session.get(Org, body.org_id)+ inviter = await session.get(User, user_id)+ await queue.rpush(+ "queue:emails",+ json.dumps({"template": "invite", "to": invite.email, "org": org.name, "inviter": inviter.display_name, "token": invite.token}),+ )+ return {"id": invite.id}++[email protected]("/invites/{token}/accept")+async def accept_invite(token: str, user_id: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ invite = (await session.execute(select(Invite).where(Invite.token == token))).scalar_one_or_none()+ if invite is None or invite.accepted_at is not None:+ raise HTTPException(status_code=404, detail="invite not found")+ session.add(OrgMember(org_id=invite.org_id, user_id=user_id, role=invite.role))+ invite.accepted_at = datetime.now(timezone.utc).replace(tzinfo=None)+ await session.commit()+ return {"orgId": invite.org_id}diff --git a/cli/test/evals/testdata/repos/accounts-api-py/app/main.py b/cli/test/evals/testdata/repos/accounts-api-py/app/main.pynew file mode 100644index 00000000..4554bf19--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/app/main.py@@ -0,0 +1,7 @@+from fastapi import FastAPI++from app import invites, users++app = FastAPI(title="accounts-api")+app.include_router(users.router)+app.include_router(invites.router)diff --git a/cli/test/evals/testdata/repos/accounts-api-py/app/models.py b/cli/test/evals/testdata/repos/accounts-api-py/app/models.pynew file mode 100644index 00000000..6928f090--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/app/models.py@@ -0,0 +1,47 @@+from datetime import datetime++from sqlalchemy import DateTime, ForeignKey, Integer, String, func+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column+++class Base(DeclarativeBase):+ pass+++class Org(Base):+ __tablename__ = "orgs"++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ name: Mapped[str] = mapped_column(String(200))+ plan: Mapped[str] = mapped_column(String(32), default="free")+++class User(Base):+ __tablename__ = "users"++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ email: Mapped[str] = mapped_column(String(320), nullable=False)+ display_name: Mapped[str] = mapped_column(String(200))+ org_id: Mapped[int] = mapped_column(ForeignKey("orgs.id"), index=True)+ created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now())+++class OrgMember(Base):+ __tablename__ = "org_members"++ org_id: Mapped[int] = mapped_column(ForeignKey("orgs.id"), primary_key=True)+ user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), primary_key=True)+ role: Mapped[str] = mapped_column(String(32), default="member")+++class Invite(Base):+ __tablename__ = "invites"++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ org_id: Mapped[int] = mapped_column(ForeignKey("orgs.id"), index=True)+ email: Mapped[str] = mapped_column(String(320))+ role: Mapped[str] = mapped_column(String(32))+ token: Mapped[str] = mapped_column(String(64), unique=True)+ invited_by: Mapped[int] = mapped_column(ForeignKey("users.id"))+ created_at: Mapped[datetime] = mapped_column(DateTime)+ accepted_at: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)diff --git a/cli/test/evals/testdata/repos/accounts-api-py/app/users.py b/cli/test/evals/testdata/repos/accounts-api-py/app/users.pynew file mode 100644index 00000000..b304f000--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/app/users.py@@ -0,0 +1,45 @@+from fastapi import APIRouter, Depends, HTTPException+from pydantic import BaseModel, EmailStr+from sqlalchemy import select, update+from sqlalchemy.ext.asyncio import AsyncSession++from app.db import current_user, get_session+from app.models import Org, User++router = APIRouter(prefix="/api")+++class UserOut(BaseModel):+ id: int+ email: str+ display_name: str+ org_id: int+ org_name: str | None = None++[email protected]("/me", response_model=UserOut)+async def me(user_id: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ user = await session.get(User, user_id)+ org = await session.get(Org, user.org_id)+ return UserOut(id=user.id, email=user.email, display_name=user.display_name, org_id=user.org_id, org_name=org.name)+++class ProfileIn(BaseModel):+ display_name: str++[email protected]("/me", response_model=UserOut)+async def update_me(body: ProfileIn, user_id: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ await session.execute(update(User).where(User.id == user_id).values(display_name=body.display_name))+ await session.commit()+ return await me(user_id, session)++[email protected]("/users/lookup", response_model=UserOut)+async def lookup(email: EmailStr, _: int = Depends(current_user), session: AsyncSession = Depends(get_session)):+ """Find a user by email address: the invite dialog and the admin user search."""+ user = (await session.execute(select(User).where(User.email == email))).scalar_one_or_none()+ if user is None:+ raise HTTPException(status_code=404, detail="no user with that email")+ org = await session.get(Org, user.org_id)+ return UserOut(id=user.id, email=user.email, display_name=user.display_name, org_id=user.org_id, org_name=org.name)diff --git a/cli/test/evals/testdata/repos/accounts-api-py/history.yaml b/cli/test/evals/testdata/repos/accounts-api-py/history.yamlnew file mode 100644index 00000000..0bb323a7--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/history.yaml@@ -0,0 +1,7 @@+commits:+- dir: .history/01+ message: 'accounts: users, orgs, members and invites'+ age: 3400h+head:+ message: 'invites: email the invitee, naming the inviter and the organization'+ age: 340hdiff --git a/cli/test/evals/testdata/repos/accounts-api-py/migrations/versions/0001_initial.py b/cli/test/evals/testdata/repos/accounts-api-py/migrations/versions/0001_initial.pynew file mode 100644index 00000000..94e56dea--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/migrations/versions/0001_initial.py@@ -0,0 +1,35 @@+"""users, orgs and memberships++Revision ID: 0001+Revises:+Create Date: 2025-11-03 09:12:44+"""+import sqlalchemy as sa+from alembic import op++revision = "0001"+down_revision = None+++def upgrade() -> None:+ op.create_table(+ "orgs",+ sa.Column("id", sa.Integer, primary_key=True),+ sa.Column("name", sa.String(200), nullable=False),+ sa.Column("plan", sa.String(32), nullable=False, server_default="free"),+ )+ op.create_table(+ "users",+ sa.Column("id", sa.Integer, primary_key=True),+ sa.Column("email", sa.String(320), nullable=False),+ sa.Column("display_name", sa.String(200), nullable=False),+ sa.Column("org_id", sa.Integer, sa.ForeignKey("orgs.id"), nullable=False),+ sa.Column("created_at", sa.DateTime, server_default=sa.func.now(), nullable=False),+ )+ op.create_index("ix_users_org_id", "users", ["org_id"])+ op.create_table(+ "org_members",+ sa.Column("org_id", sa.Integer, sa.ForeignKey("orgs.id"), primary_key=True),+ sa.Column("user_id", sa.Integer, sa.ForeignKey("users.id"), primary_key=True),+ sa.Column("role", sa.String(32), nullable=False, server_default="member"),+ )diff --git a/cli/test/evals/testdata/repos/accounts-api-py/migrations/versions/0002_invites.py b/cli/test/evals/testdata/repos/accounts-api-py/migrations/versions/0002_invites.pynew file mode 100644index 00000000..3c77c56d--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/migrations/versions/0002_invites.py@@ -0,0 +1,26 @@+"""invites++Revision ID: 0002+Revises: 0001+Create Date: 2026-02-17 14:40:02+"""+import sqlalchemy as sa+from alembic import op++revision = "0002"+down_revision = "0001"+++def upgrade() -> None:+ op.create_table(+ "invites",+ sa.Column("id", sa.Integer, primary_key=True),+ sa.Column("org_id", sa.Integer, sa.ForeignKey("orgs.id"), nullable=False),+ sa.Column("email", sa.String(320), nullable=False),+ sa.Column("role", sa.String(32), nullable=False),+ sa.Column("token", sa.String(64), nullable=False, unique=True),+ sa.Column("invited_by", sa.Integer, sa.ForeignKey("users.id"), nullable=False),+ sa.Column("created_at", sa.DateTime, nullable=False),+ sa.Column("accepted_at", sa.DateTime, nullable=True),+ )+ op.create_index("ix_invites_org_id", "invites", ["org_id"])diff --git a/cli/test/evals/testdata/repos/accounts-api-py/requirements.txt b/cli/test/evals/testdata/repos/accounts-api-py/requirements.txtnew file mode 100644index 00000000..03dfa799--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-py/requirements.txt@@ -0,0 +1,12 @@+fastapi==0.118.0+uvicorn[standard]==0.37.0+sqlalchemy[asyncio]==2.0.43+asyncpg==0.30.0+alembic==1.16.5+redis==6.4.0+pydantic[email]==2.11.9+opentelemetry-distro==0.58b0+opentelemetry-exporter-otlp==1.37.0+opentelemetry-instrumentation-fastapi==0.58b0+opentelemetry-instrumentation-sqlalchemy==0.58b0+opentelemetry-instrumentation-redis==0.58b0diff --git a/cli/test/evals/testdata/repos/checkout-api-py/README.md b/cli/test/evals/testdata/repos/checkout-api-py/README.mdnew file mode 100644index 00000000..c6c111a5--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/README.md@@ -0,0 +1,11 @@+# checkout-api++Cart and checkout for the marketplace. Python 3.12, FastAPI, SQLAlchemy 2.0+(sync sessions, psycopg 3) on PostgreSQL, run with uvicorn (one process per+container). Payments go to the provider's REST API with httpx. Telemetry goes+to Traceway over OTLP through the OpenTelemetry auto-instrumentation.++```+pip install -r requirements.txt+opentelemetry-instrument uvicorn app.main:app --port 8080+```diff --git a/cli/test/evals/testdata/repos/checkout-api-py/app/cart.py b/cli/test/evals/testdata/repos/checkout-api-py/app/cart.pynew file mode 100644index 00000000..99b91c93--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/app/cart.py@@ -0,0 +1,22 @@+import uuid++from fastapi import APIRouter, Header+from sqlalchemy import select++from app.db import SessionLocal+from app.models import CartItem++router = APIRouter(prefix="/api")++[email protected]("/cart")+def get_cart(x_cart_id: str = Header()):+ cart_id = uuid.UUID(x_cart_id)+ # A short session: the connection goes back to the pool as soon as the+ # rows are read.+ with SessionLocal() as session:+ rows = session.execute(select(CartItem).where(CartItem.cart_id == cart_id)).scalars().all()+ return {+ "items": [{"sku": r.sku, "qty": r.qty, "unitPriceCents": r.unit_price_cents} for r in rows],+ "totalCents": sum(r.qty * r.unit_price_cents for r in rows),+ }diff --git a/cli/test/evals/testdata/repos/checkout-api-py/app/checkout.py b/cli/test/evals/testdata/repos/checkout-api-py/app/checkout.pynew file mode 100644index 00000000..8630c50d--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/app/checkout.py@@ -0,0 +1,60 @@+import logging+import uuid++from fastapi import APIRouter, Header, HTTPException+from pydantic import BaseModel+from sqlalchemy import delete, select, update+from sqlalchemy.exc import TimeoutError as PoolTimeoutError++from app import payments+from app.db import SessionLocal+from app.models import CartItem, Inventory, Order++log = logging.getLogger(__name__)+router = APIRouter(prefix="/api")+++class CheckoutIn(BaseModel):+ cart_id: uuid.UUID+++class CheckoutOut(BaseModel):+ order_id: uuid.UUID+ total_cents: int++[email protected]("/checkout", status_code=201, response_model=CheckoutOut)+def create_checkout(body: CheckoutIn, x_user_id: int = Header()):+ """Places the order for a cart: checks stock, charges the card, writes the+ order and decrements inventory, all in one transaction so a failed charge+ leaves nothing behind."""+ try:+ return _checkout(body, x_user_id)+ except PoolTimeoutError as exc:+ log.warning("checkout failed for cart %s: %s", body.cart_id, exc)+ raise+++def _checkout(body: CheckoutIn, user_id: int) -> CheckoutOut:+ with SessionLocal.begin() as session:+ items = session.execute(select(CartItem).where(CartItem.cart_id == body.cart_id)).scalars().all()+ if not items:+ raise HTTPException(status_code=404, detail="cart is empty")+ skus = [i.sku for i in items]+ stock = dict(session.execute(select(Inventory.sku, Inventory.available).where(Inventory.sku.in_(skus))).all())+ for item in items:+ if stock.get(item.sku, 0) < item.qty:+ raise HTTPException(status_code=409, detail=f"{item.sku} is out of stock")++ total = sum(i.qty * i.unit_price_cents for i in items)+ payments.charge(user_id, total, idempotency_key=str(body.cart_id))++ order = Order(cart_id=body.cart_id, user_id=user_id, total_cents=total, status="paid")+ session.add(order)+ session.flush()+ for item in items:+ session.execute(+ update(Inventory).where(Inventory.sku == item.sku).values(available=Inventory.available - item.qty)+ )+ session.execute(delete(CartItem).where(CartItem.cart_id == body.cart_id))+ return CheckoutOut(order_id=order.id, total_cents=total)diff --git a/cli/test/evals/testdata/repos/checkout-api-py/app/db.py b/cli/test/evals/testdata/repos/checkout-api-py/app/db.pynew file mode 100644index 00000000..b773eadd--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/app/db.py@@ -0,0 +1,13 @@+import os++from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker++engine = create_engine(+ os.environ.get("DATABASE_URL", "postgresql+psycopg://[email protected]:5432/checkout"),+ pool_size=10,+ max_overflow=5,+ pool_timeout=30,+ pool_pre_ping=True,+)+SessionLocal = sessionmaker(engine, expire_on_commit=False)diff --git a/cli/test/evals/testdata/repos/checkout-api-py/app/main.py b/cli/test/evals/testdata/repos/checkout-api-py/app/main.pynew file mode 100644index 00000000..2ff4cf97--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/app/main.py@@ -0,0 +1,18 @@+from fastapi import FastAPI++from app import cart, checkout++app = FastAPI(title="checkout-api")+app.include_router(cart.router)+app.include_router(checkout.router)+++SHIPPING_OPTIONS = [+ {"id": "standard", "label": "Standard (3-5 days)", "priceCents": 0},+ {"id": "express", "label": "Express (1-2 days)", "priceCents": 990},+]++[email protected]("/api/shipping/options")+def shipping_options():+ return SHIPPING_OPTIONSdiff --git a/cli/test/evals/testdata/repos/checkout-api-py/app/models.py b/cli/test/evals/testdata/repos/checkout-api-py/app/models.pynew file mode 100644index 00000000..aa0cf8de--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/app/models.py@@ -0,0 +1,37 @@+import uuid+from datetime import datetime++from sqlalchemy import DateTime, Integer, String, Uuid, func+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column+++class Base(DeclarativeBase):+ pass+++class CartItem(Base):+ __tablename__ = "cart_items"++ id: Mapped[int] = mapped_column(Integer, primary_key=True)+ cart_id: Mapped[uuid.UUID] = mapped_column(Uuid, index=True)+ sku: Mapped[str] = mapped_column(String(64))+ qty: Mapped[int] = mapped_column(Integer)+ unit_price_cents: Mapped[int] = mapped_column(Integer)+++class Inventory(Base):+ __tablename__ = "inventory"++ sku: Mapped[str] = mapped_column(String(64), primary_key=True)+ available: Mapped[int] = mapped_column(Integer)+++class Order(Base):+ __tablename__ = "orders"++ id: Mapped[uuid.UUID] = mapped_column(Uuid, primary_key=True, default=uuid.uuid4)+ cart_id: Mapped[uuid.UUID] = mapped_column(Uuid)+ user_id: Mapped[int] = mapped_column(Integer, index=True)+ total_cents: Mapped[int] = mapped_column(Integer)+ status: Mapped[str] = mapped_column(String(16))+ created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now())diff --git a/cli/test/evals/testdata/repos/checkout-api-py/app/payments.py b/cli/test/evals/testdata/repos/checkout-api-py/app/payments.pynew file mode 100644index 00000000..d95cbfcc--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/app/payments.py@@ -0,0 +1,26 @@+import os++import httpx++_client = httpx.Client(+ base_url="https://api.payments.example",+ headers={"Authorization": f"Bearer {os.environ.get('PAYMENTS_API_KEY', '')}"},+ timeout=httpx.Timeout(10.0, connect=3.0),+)+++class PaymentDeclined(Exception):+ pass+++def charge(user_id: int, amount_cents: int, idempotency_key: str) -> str:+ """Charges the user's saved card and returns the charge id."""+ resp = _client.post(+ "/v1/charges",+ json={"customer": f"user_{user_id}", "amount": amount_cents, "currency": "eur"},+ headers={"Idempotency-Key": idempotency_key},+ )+ if resp.status_code == 402:+ raise PaymentDeclined(resp.json().get("message", "card declined"))+ resp.raise_for_status()+ return resp.json()["id"]diff --git a/cli/test/evals/testdata/repos/checkout-api-py/history.yaml b/cli/test/evals/testdata/repos/checkout-api-py/history.yamlnew file mode 100644index 00000000..8403ef84--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/history.yaml@@ -0,0 +1,4 @@+commits: []+head:+ message: 'checkout: cart, checkout and payments'+ age: 3100hdiff --git a/cli/test/evals/testdata/repos/checkout-api-py/requirements.txt b/cli/test/evals/testdata/repos/checkout-api-py/requirements.txtnew file mode 100644index 00000000..5d3306e0--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-py/requirements.txt@@ -0,0 +1,10 @@+fastapi==0.118.0+uvicorn[standard]==0.37.0+sqlalchemy==2.0.43+psycopg[binary]==3.2.10+httpx==0.28.1+opentelemetry-distro==0.58b0+opentelemetry-exporter-otlp==1.37.0+opentelemetry-instrumentation-fastapi==0.58b0+opentelemetry-instrumentation-sqlalchemy==0.58b0+opentelemetry-instrumentation-httpx==0.58b0diff --git a/cli/test/evals/testdata/repos/commerce-api-node/.history/01/src/pricing.js b/cli/test/evals/testdata/repos/commerce-api-node/.history/01/src/pricing.jsnew file mode 100644index 00000000..c6532352--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/.history/01/src/pricing.js@@ -0,0 +1,7 @@+// Cart and order totals. Prices are integer cents.+export function priceCart(lines) {+ const items = lines.map((l) => ({ id: l.id, name: l.name, qty: l.qty, lineCents: l.price_cents * l.qty }));+ const subtotalCents = items.reduce((s, i) => s + i.lineCents, 0);+ const shippingCents = items.length === 0 ? 0 : 495;+ return { items, subtotalCents, shippingCents, totalCents: subtotalCents + shippingCents };+}diff --git a/cli/test/evals/testdata/repos/commerce-api-node/.history/01/src/pricing.test.js b/cli/test/evals/testdata/repos/commerce-api-node/.history/01/src/pricing.test.jsnew file mode 100644index 00000000..a61fae02--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/.history/01/src/pricing.test.js@@ -0,0 +1,12 @@+import { test } from 'node:test';+import assert from 'node:assert/strict';+import { priceCart } from './pricing.js';++test('flat shipping', () => {+ assert.equal(priceCart([{ id: 1, name: 'Mug', price_cents: 2500, qty: 2 }]).shippingCents, 495);+});++test('totals add up', () => {+ const c = priceCart([{ id: 1, name: 'Mug', price_cents: 1200, qty: 3 }, { id: 2, name: 'Tea', price_cents: 800, qty: 1 }]);+ assert.deepEqual([c.subtotalCents, c.totalCents], [4400, 4895]);+});diff --git a/cli/test/evals/testdata/repos/commerce-api-node/README.md b/cli/test/evals/testdata/repos/commerce-api-node/README.mdnew file mode 100644index 00000000..7cc032f6--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/README.md@@ -0,0 +1,13 @@+# commerce-api++The shop's API: catalog, search, cart, checkout and recommendations. Node 24,+Express 5, PostgreSQL (`pg`), Redis (`ioredis`) and the search cluster.+Traces and metrics go to Traceway over OTLP through the OpenTelemetry Node SDK+(`src/tracing.js`); host metrics come from the OTel Collector DaemonSet.++Runs on Kubernetes, see `deploy/k8s/`.++```+npm test+npm start+```diff --git a/cli/test/evals/testdata/repos/commerce-api-node/deploy/k8s/deployment.yaml b/cli/test/evals/testdata/repos/commerce-api-node/deploy/k8s/deployment.yamlnew file mode 100644index 00000000..8f27ee73--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/deploy/k8s/deployment.yaml@@ -0,0 +1,27 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: commerce-api+ labels: {app: commerce-api}+spec:+ replicas: 2+ selector:+ matchLabels: {app: commerce-api}+ template:+ metadata:+ labels: {app: commerce-api}+ spec:+ containers:+ - name: commerce-api+ image: registry.example.com/commerce-api:4.18.2+ ports: [{containerPort: 8080}]+ env:+ - {name: OTEL_SERVICE_NAME, value: commerce-api}+ - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector.observability:4318"}+ envFrom: [{secretRef: {name: commerce-api}}]+ resources:+ requests: {cpu: "1", memory: 512Mi}+ limits: {cpu: "1", memory: 1Gi}+ readinessProbe:+ httpGet: {path: /api/categories, port: 8080}+ periodSeconds: 10diff --git a/cli/test/evals/testdata/repos/commerce-api-node/history.yaml b/cli/test/evals/testdata/repos/commerce-api-node/history.yamlnew file mode 100644index 00000000..dc2ac221--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: 'checkout: create payment intents before inserting the order'+ age: 530h+ author: Lena Fischer <[email protected]>+head:+ message: 'cart: free shipping from 50 EUR'+ age: 216h+ author: Marco Bianchi <[email protected]>diff --git a/cli/test/evals/testdata/repos/commerce-api-node/package.json b/cli/test/evals/testdata/repos/commerce-api-node/package.jsonnew file mode 100644index 00000000..f0c21ba9--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/package.json@@ -0,0 +1,18 @@+{+ "name": "commerce-api",+ "version": "4.18.2",+ "private": true,+ "type": "module",+ "main": "src/server.js",+ "scripts": {+ "start": "node --import ./src/tracing.js src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "^0.64.1",+ "@opentelemetry/sdk-node": "^0.205.0",+ "express": "^5.1.0",+ "ioredis": "^5.8.0",+ "pg": "^8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/commerce-api-node/src/db.js b/cli/test/evals/testdata/repos/commerce-api-node/src/db.jsnew file mode 100644index 00000000..07173ee1--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/src/db.js@@ -0,0 +1,5 @@+import pg from 'pg';+import Redis from 'ioredis';++export const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL, max: 20 });+export const redis = new Redis(process.env.REDIS_URL);diff --git a/cli/test/evals/testdata/repos/commerce-api-node/src/pricing.js b/cli/test/evals/testdata/repos/commerce-api-node/src/pricing.jsnew file mode 100644index 00000000..9b85859a--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/src/pricing.js@@ -0,0 +1,7 @@+// Cart and order totals. Prices are integer cents.+export function priceCart(lines) {+ const items = lines.map((l) => ({ id: l.id, name: l.name, qty: l.qty, lineCents: l.price_cents * l.qty }));+ const subtotalCents = items.reduce((s, i) => s + i.lineCents, 0);+ const shippingCents = subtotalCents >= 5000 || items.length === 0 ? 0 : 495;+ return { items, subtotalCents, shippingCents, totalCents: subtotalCents + shippingCents };+}diff --git a/cli/test/evals/testdata/repos/commerce-api-node/src/pricing.test.js b/cli/test/evals/testdata/repos/commerce-api-node/src/pricing.test.jsnew file mode 100644index 00000000..3c0e222f--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/src/pricing.test.js@@ -0,0 +1,13 @@+import { test } from 'node:test';+import assert from 'node:assert/strict';+import { priceCart } from './pricing.js';++test('free shipping from 50 EUR', () => {+ assert.equal(priceCart([{ id: 1, name: 'Mug', price_cents: 2500, qty: 2 }]).shippingCents, 0);+ assert.equal(priceCart([{ id: 1, name: 'Mug', price_cents: 2500, qty: 1 }]).shippingCents, 495);+});++test('totals add up', () => {+ const c = priceCart([{ id: 1, name: 'Mug', price_cents: 1200, qty: 3 }, { id: 2, name: 'Tea', price_cents: 800, qty: 1 }]);+ assert.deepEqual([c.subtotalCents, c.totalCents], [4400, 4895]);+});diff --git a/cli/test/evals/testdata/repos/commerce-api-node/src/server.js b/cli/test/evals/testdata/repos/commerce-api-node/src/server.jsnew file mode 100644index 00000000..aebe9088--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/src/server.js@@ -0,0 +1,86 @@+import express from 'express';+import { pool, redis } from './db.js';+import { priceCart } from './pricing.js';++const app = express();+app.use(express.json({ limit: '100kb' }));++app.get('/api/products', async (req, res) => {+ const { rows } = await pool.query(+ 'SELECT id, sku, name, price_cents, thumbnail_url FROM products WHERE category_id = $1 AND visible ORDER BY rank LIMIT 48',+ [req.query.category]);+ res.json(rows);+});++app.get('/api/products/:id', async (req, res) => {+ const { rows: [product] } = await pool.query('SELECT id, sku, name, description, price_cents, stock FROM products WHERE id = $1', [req.params.id]);+ if (!product) return res.sendStatus(404);+ const { rows: images } = await pool.query('SELECT url, alt FROM product_images WHERE product_id = $1 ORDER BY position', [product.id]);+ res.json({ ...product, images });+});++app.get('/api/search', async (req, res) => {+ const r = await fetch('http://search.internal:9200/products/_search', {+ method: 'POST',+ headers: { 'content-type': 'application/json' },+ body: JSON.stringify({ query: { multi_match: { query: String(req.query.q ?? ''), fields: ['name^3', 'description'] } }, size: 24 }),+ });+ const body = await r.json();+ res.json(body.hits.hits.map((h) => ({ id: h._id, ...h._source })));+});++app.get('/api/cart', async (req, res) => {+ const cached = await redis.get(`cart:${req.get('x-session-id')}`);+ if (cached) return res.type('json').send(cached);+ const { rows } = await pool.query(+ 'SELECT p.id, p.name, p.price_cents, ci.qty FROM cart_items ci JOIN products p ON p.id = ci.product_id WHERE ci.cart_id = $1',+ [req.get('x-cart-id')]);+ res.json(priceCart(rows));+});++app.post('/api/cart/items', async (req, res) => {+ const { productId, qty } = req.body ?? {};+ if (!productId || !(qty > 0)) return res.status(422).json({ error: 'productId and qty are required' });+ await pool.query(+ 'INSERT INTO cart_items (cart_id, product_id, qty) VALUES ($1, $2, $3) ON CONFLICT (cart_id, product_id) DO UPDATE SET qty = cart_items.qty + EXCLUDED.qty',+ [req.get('x-cart-id'), productId, qty]);+ await redis.del(`cart:${req.get('x-session-id')}`);+ res.sendStatus(201);+});++app.get('/api/categories', async (_req, res) => {+ res.type('json').send(await redis.get('categories:tree'));+});++app.get('/api/recommendations', async (req, res) => {+ const { rows } = await pool.query('SELECT product_id, score FROM recommendations WHERE user_id = $1 ORDER BY score DESC LIMIT 12', [req.get('x-user-id')]);+ res.json(rows);+});++app.post('/api/checkout', async (req, res) => {+ const client = await pool.connect();+ try {+ await client.query('BEGIN');+ const { rows } = await client.query(+ 'SELECT p.id, p.price_cents, p.stock, ci.qty FROM cart_items ci JOIN products p ON p.id = ci.product_id WHERE ci.cart_id = $1 FOR UPDATE',+ [req.get('x-cart-id')]);+ const order = priceCart(rows);+ const pay = await fetch('https://api.payments.example/v1/payment_intents', {+ method: 'POST',+ headers: { authorization: `Bearer ${process.env.PAYMENTS_KEY}`, 'content-type': 'application/json' },+ body: JSON.stringify({ amount: order.totalCents, currency: 'eur' }),+ });+ if (!pay.ok) throw new Error(`payment failed: ${pay.status}`);+ await client.query("INSERT INTO orders (id, user_id, total_cents, status, created_at) VALUES ($1, $2, $3, 'paid', now())",+ [crypto.randomUUID(), req.get('x-user-id'), order.totalCents]);+ await client.query('COMMIT');+ res.status(201).json(order);+ } catch (err) {+ await client.query('ROLLBACK');+ throw err;+ } finally {+ client.release();+ }+});++app.listen(Number(process.env.PORT ?? 8080));diff --git a/cli/test/evals/testdata/repos/commerce-api-node/src/tracing.js b/cli/test/evals/testdata/repos/commerce-api-node/src/tracing.jsnew file mode 100644index 00000000..980f4de2--- /dev/null+++ b/cli/test/evals/testdata/repos/commerce-api-node/src/tracing.js@@ -0,0 +1,8 @@+import { NodeSDK } from '@opentelemetry/sdk-node';+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';++// OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME come from the deployment.+const sdk = new NodeSDK({+ instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false } })],+});+sdk.start();diff --git a/cli/test/evals/testdata/repos/feed-android/.history/01/app/build.gradle.kts b/cli/test/evals/testdata/repos/feed-android/.history/01/app/build.gradle.ktsnew file mode 100644index 00000000..59fea894--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/.history/01/app/build.gradle.kts@@ -0,0 +1,32 @@+plugins {+ id("com.android.application")+ id("org.jetbrains.kotlin.android")+ id("com.tracewayapp.symbols") version "1.0.1"+}++android {+ namespace = "com.example.pinboard"+ compileSdk = 35+ defaultConfig {+ applicationId = "com.example.pinboard"+ minSdk = 26+ targetSdk = 35+ versionCode = 5702+ versionName = "5.7.2"+ }+ buildTypes {+ release {+ isMinifyEnabled = true+ proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro")+ }+ }+}++dependencies {+ implementation("com.tracewayapp:traceway:1.0.1")+ implementation("androidx.recyclerview:recyclerview:1.4.0")+ implementation("androidx.constraintlayout:constraintlayout:2.2.1")+ implementation("com.squareup.retrofit2:retrofit:2.11.0")+ implementation("com.squareup.retrofit2:converter-gson:2.11.0")+ implementation("io.coil-kt:coil:2.7.0")+}diff --git a/cli/test/evals/testdata/repos/feed-android/.history/01/app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt b/cli/test/evals/testdata/repos/feed-android/.history/01/app/src/main/java/com/example/pinboard/feed/FeedAdapter.ktnew file mode 100644index 00000000..de4940f4--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/.history/01/app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt@@ -0,0 +1,68 @@+package com.example.pinboard.feed++import android.view.LayoutInflater+import android.view.View+import android.view.ViewGroup+import android.widget.ImageView+import android.widget.TextView+import androidx.recyclerview.widget.DiffUtil+import androidx.recyclerview.widget.ListAdapter+import androidx.recyclerview.widget.RecyclerView+import coil.load+import coil.transform.CircleCropTransformation+import com.example.pinboard.R++/**+ * Binds feed posts. Each card shows the author, the text and the image as a+ * square crop.+ */+class FeedAdapter(+ private val onOpen: (FeedItem) -> Unit,+) : ListAdapter<FeedItem, FeedAdapter.PostViewHolder>(DIFF) {++ class PostViewHolder(view: View) : RecyclerView.ViewHolder(view) {+ val avatar: ImageView = view.findViewById(R.id.avatar)+ val author: TextView = view.findViewById(R.id.author)+ val text: TextView = view.findViewById(R.id.text)+ val thumbnail: ImageView = view.findViewById(R.id.thumbnail)+ val likes: TextView = view.findViewById(R.id.likes)+ }++ override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): PostViewHolder {+ val view = LayoutInflater.from(parent.context).inflate(R.layout.item_post, parent, false)+ return PostViewHolder(view)+ }++ override fun onBindViewHolder(holder: PostViewHolder, position: Int) {+ val item = getItem(position)+ holder.author.text = item.author.name+ holder.avatar.load(item.author.avatarUrl) {+ crossfade(true)+ transformations(CircleCropTransformation())+ placeholder(R.drawable.avatar_placeholder)+ }+ holder.text.text = item.text+ holder.likes.text = holder.itemView.resources.getQuantityString(R.plurals.likes, item.likes, item.likes)+ bindImage(holder, item)+ holder.itemView.setOnClickListener { onOpen(item) }+ }++ private fun bindImage(holder: PostViewHolder, item: FeedItem) {+ val image = item.image+ if (image == null) {+ holder.thumbnail.visibility = View.GONE+ return+ }+ holder.thumbnail.visibility = View.VISIBLE+ holder.thumbnail.load(image.url) {+ crossfade(true)+ }+ }++ private companion object {+ val DIFF = object : DiffUtil.ItemCallback<FeedItem>() {+ override fun areItemsTheSame(a: FeedItem, b: FeedItem) = a.id == b.id+ override fun areContentsTheSame(a: FeedItem, b: FeedItem) = a == b+ }+ }+}diff --git a/cli/test/evals/testdata/repos/feed-android/.history/01/app/src/main/java/com/example/pinboard/feed/FeedItem.kt b/cli/test/evals/testdata/repos/feed-android/.history/01/app/src/main/java/com/example/pinboard/feed/FeedItem.ktnew file mode 100644index 00000000..2781e097--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/.history/01/app/src/main/java/com/example/pinboard/feed/FeedItem.kt@@ -0,0 +1,18 @@+package com.example.pinboard.feed++/**+ * One post of the home feed, as GET /v2/feed returns it (parsed by Gson).+ * Text posts carry no "image" object.+ */+data class FeedItem(+ val id: String,+ val author: Author,+ val text: String,+ val image: ImageRef?,+ val likes: Int,+ val createdAt: String,+)++data class Author(val id: String, val name: String, val avatarUrl: String?)++data class ImageRef(val url: String, val width: Int, val height: Int)diff --git a/cli/test/evals/testdata/repos/feed-android/README.md b/cli/test/evals/testdata/repos/feed-android/README.mdnew file mode 100644index 00000000..db00f5ec--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/README.md@@ -0,0 +1,12 @@+# Pinboard for Android++Kotlin, AndroidX, RecyclerView, Retrofit + Gson, Coil. Crashes go to+Traceway through `com.tracewayapp:traceway`; release builds are minified with+R8 and the `com.tracewayapp.symbols` plugin uploads `mapping.txt`, so traces+arrive deobfuscated.++The home feed (`feed/`) pages through `GET /v2/feed` and binds each post in+`FeedAdapter`.++Releases are tagged in the commit message (`5.8.0: ...`) and rolled out in+stages from the Play Console.diff --git a/cli/test/evals/testdata/repos/feed-android/app/build.gradle.kts b/cli/test/evals/testdata/repos/feed-android/app/build.gradle.ktsnew file mode 100644index 00000000..768e69f1--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/build.gradle.kts@@ -0,0 +1,32 @@+plugins {+ id("com.android.application")+ id("org.jetbrains.kotlin.android")+ id("com.tracewayapp.symbols") version "1.0.1"+}++android {+ namespace = "com.example.pinboard"+ compileSdk = 35+ defaultConfig {+ applicationId = "com.example.pinboard"+ minSdk = 26+ targetSdk = 35+ versionCode = 5800+ versionName = "5.8.0"+ }+ buildTypes {+ release {+ isMinifyEnabled = true+ proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro")+ }+ }+}++dependencies {+ implementation("com.tracewayapp:traceway:1.0.1")+ implementation("androidx.recyclerview:recyclerview:1.4.0")+ implementation("androidx.constraintlayout:constraintlayout:2.2.1")+ implementation("com.squareup.retrofit2:retrofit:2.11.0")+ implementation("com.squareup.retrofit2:converter-gson:2.11.0")+ implementation("io.coil-kt:coil:2.7.0")+}diff --git a/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/PinboardApp.kt b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/PinboardApp.ktnew file mode 100644index 00000000..be702664--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/PinboardApp.kt@@ -0,0 +1,33 @@+package com.example.pinboard++import android.app.Application+import com.example.pinboard.data.FeedApi+import com.tracewayapp.traceway.Traceway+import com.tracewayapp.traceway.TracewayOptions+import com.tracewayapp.traceway.network.TracewayOkHttpInterceptor+import okhttp3.OkHttpClient+import retrofit2.Retrofit+import retrofit2.converter.gson.GsonConverterFactory++class PinboardApp : Application() {+ companion object {+ lateinit var http: OkHttpClient+ lateinit var api: FeedApi+ }++ override fun onCreate() {+ super.onCreate()+ Traceway.init(+ application = this,+ connectionString = BuildConfig.TRACEWAY_DSN,+ options = TracewayOptions(version = BuildConfig.VERSION_NAME, proguardUuid = BuildConfig.TRACEWAY_PROGUARD_UUID),+ )+ http = OkHttpClient.Builder().addInterceptor(TracewayOkHttpInterceptor()).build()+ api = Retrofit.Builder()+ .baseUrl("https://api.pinboard.example/")+ .client(http)+ .addConverterFactory(GsonConverterFactory.create())+ .build()+ .create(FeedApi::class.java)+ }+}diff --git a/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/data/FeedApi.kt b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/data/FeedApi.ktnew file mode 100644index 00000000..fd691276--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/data/FeedApi.kt@@ -0,0 +1,16 @@+package com.example.pinboard.data++import com.example.pinboard.feed.FeedItem+import retrofit2.http.GET+import retrofit2.http.Path+import retrofit2.http.Query++data class FeedPage(val items: List<FeedItem>, val nextCursor: String?)++interface FeedApi {+ @GET("v2/feed")+ suspend fun feed(@Query("cursor") cursor: String?, @Query("limit") limit: Int = 20): FeedPage++ @GET("v2/posts/{id}")+ suspend fun post(@Path("id") id: String): FeedItem+}diff --git a/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedActivity.kt b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedActivity.ktnew file mode 100644index 00000000..ab6e6870--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedActivity.kt@@ -0,0 +1,52 @@+package com.example.pinboard.feed++import android.content.Intent+import android.os.Bundle+import androidx.appcompat.app.AppCompatActivity+import androidx.lifecycle.lifecycleScope+import androidx.recyclerview.widget.LinearLayoutManager+import androidx.recyclerview.widget.RecyclerView+import com.example.pinboard.PinboardApp+import com.example.pinboard.R+import com.example.pinboard.media.FullImageActivity+import kotlinx.coroutines.launch++class FeedActivity : AppCompatActivity() {+ private val adapter = FeedAdapter(onOpen = ::open)+ private var cursor: String? = null+ private var loading = false++ override fun onCreate(savedInstanceState: Bundle?) {+ super.onCreate(savedInstanceState)+ setContentView(R.layout.activity_feed)+ val list = findViewById<RecyclerView>(R.id.feed)+ list.layoutManager = LinearLayoutManager(this)+ list.adapter = adapter+ list.addOnScrollListener(object : RecyclerView.OnScrollListener() {+ override fun onScrolled(rv: RecyclerView, dx: Int, dy: Int) {+ val lm = rv.layoutManager as LinearLayoutManager+ if (lm.findLastVisibleItemPosition() > adapter.itemCount - 5) loadMore()+ }+ })+ loadMore()+ }++ private fun loadMore() {+ if (loading) return+ loading = true+ lifecycleScope.launch {+ try {+ val page = PinboardApp.api.feed(cursor)+ cursor = page.nextCursor+ adapter.submitList(adapter.currentList + page.items)+ println("feed page loaded: ${page.items.size} items")+ } finally {+ loading = false+ }+ }+ }++ private fun open(item: FeedItem) {+ startActivity(Intent(this, FullImageActivity::class.java).putExtra(FullImageActivity.EXTRA_POST, item.id))+ }+}diff --git a/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedAdapter.ktnew file mode 100644index 00000000..7068de2d--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedAdapter.kt@@ -0,0 +1,69 @@+package com.example.pinboard.feed++import android.view.LayoutInflater+import android.view.View+import android.view.ViewGroup+import android.widget.ImageView+import android.widget.TextView+import androidx.constraintlayout.widget.ConstraintLayout+import androidx.core.view.updateLayoutParams+import androidx.recyclerview.widget.DiffUtil+import androidx.recyclerview.widget.ListAdapter+import androidx.recyclerview.widget.RecyclerView+import coil.load+import coil.transform.CircleCropTransformation+import com.example.pinboard.R++/**+ * Binds feed posts. Each card shows the author, the text and, since 5.8.0,+ * the image at its own aspect ratio instead of a fixed square crop.+ */+class FeedAdapter(+ private val onOpen: (FeedItem) -> Unit,+) : ListAdapter<FeedItem, FeedAdapter.PostViewHolder>(DIFF) {++ class PostViewHolder(view: View) : RecyclerView.ViewHolder(view) {+ val avatar: ImageView = view.findViewById(R.id.avatar)+ val author: TextView = view.findViewById(R.id.author)+ val text: TextView = view.findViewById(R.id.text)+ val thumbnail: ImageView = view.findViewById(R.id.thumbnail)+ val likes: TextView = view.findViewById(R.id.likes)+ }++ override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): PostViewHolder {+ val view = LayoutInflater.from(parent.context).inflate(R.layout.item_post, parent, false)+ return PostViewHolder(view)+ }++ override fun onBindViewHolder(holder: PostViewHolder, position: Int) {+ val item = getItem(position)+ holder.author.text = item.author.name+ holder.avatar.load(item.author.avatarUrl) {+ crossfade(true)+ transformations(CircleCropTransformation())+ placeholder(R.drawable.avatar_placeholder)+ }+ holder.text.text = item.text+ holder.likes.text = holder.itemView.resources.getQuantityString(R.plurals.likes, item.likes, item.likes)+ bindImage(holder, item)+ holder.itemView.setOnClickListener { onOpen(item) }+ }++ private fun bindImage(holder: PostViewHolder, item: FeedItem) {+ holder.thumbnail.visibility = View.VISIBLE+ holder.thumbnail.updateLayoutParams<ConstraintLayout.LayoutParams> {+ dimensionRatio = "${item.image.width}:${item.image.height}"+ }+ holder.thumbnail.load(item.image.url) {+ crossfade(true)+ size(holder.thumbnail.width.coerceAtLeast(1), coil.size.Dimension.Undefined)+ }+ }++ private companion object {+ val DIFF = object : DiffUtil.ItemCallback<FeedItem>() {+ override fun areItemsTheSame(a: FeedItem, b: FeedItem) = a.id == b.id+ override fun areContentsTheSame(a: FeedItem, b: FeedItem) = a == b+ }+ }+}diff --git a/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedItem.kt b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedItem.ktnew file mode 100644index 00000000..81684ea5--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/feed/FeedItem.kt@@ -0,0 +1,18 @@+package com.example.pinboard.feed++/**+ * One post of the home feed, as GET /v2/feed returns it (parsed by Gson).+ * Text posts carry no "image" object.+ */+data class FeedItem(+ val id: String,+ val author: Author,+ val text: String,+ val image: ImageRef,+ val likes: Int,+ val createdAt: String,+)++data class Author(val id: String, val name: String, val avatarUrl: String?)++data class ImageRef(val url: String, val width: Int, val height: Int)diff --git a/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/media/FullImageActivity.kt b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/media/FullImageActivity.ktnew file mode 100644index 00000000..5df6d7c8--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/app/src/main/java/com/example/pinboard/media/FullImageActivity.kt@@ -0,0 +1,55 @@+package com.example.pinboard.media++import android.graphics.Bitmap+import android.graphics.BitmapFactory+import android.os.Bundle+import android.widget.ImageView+import androidx.appcompat.app.AppCompatActivity+import androidx.lifecycle.lifecycleScope+import com.example.pinboard.PinboardApp+import com.example.pinboard.R+import kotlinx.coroutines.Dispatchers+import kotlinx.coroutines.launch+import kotlinx.coroutines.withContext+import okhttp3.Request++/**+ * Shows one post's image at full resolution, pinch-to-zoom.+ */+class FullImageActivity : AppCompatActivity() {++ companion object {+ const val EXTRA_POST = "post_id"+ }++ private lateinit var image: ImageView++ override fun onCreate(savedInstanceState: Bundle?) {+ super.onCreate(savedInstanceState)+ setContentView(R.layout.activity_full_image)+ image = findViewById(R.id.full_image)+ val postId = intent.getStringExtra(EXTRA_POST) ?: return finish()+ lifecycleScope.launch {+ val url = PinboardApp.api.post(postId).image?.url ?: return@launch finish()+ val bitmap = withContext(Dispatchers.IO) { download(url) }+ image.setImageBitmap(bitmap)+ }+ }++ private fun download(url: String): Bitmap {+ val request = Request.Builder().url(url).build()+ PinboardApp.http.newCall(request).execute().use { response ->+ val body = response.body ?: error("empty body for $url")+ return decodeOriginal(body.byteStream())+ }+ }++ // Originals are up to 6000x4000; decoded at full size so zoom stays sharp.+ private fun decodeOriginal(stream: java.io.InputStream): Bitmap {+ val options = BitmapFactory.Options().apply {+ inPreferredConfig = Bitmap.Config.ARGB_8888+ inMutable = false+ }+ return BitmapFactory.decodeStream(stream, null, options) ?: error("undecodable image")+ }+}diff --git a/cli/test/evals/testdata/repos/feed-android/history.yaml b/cli/test/evals/testdata/repos/feed-android/history.yamlnew file mode 100644index 00000000..b1ff07b0--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: '5.7.2: fix like counts on tablets'+ age: 600h+ author: Aisha Bello <[email protected]>+head:+ message: '5.8.0: feed images at their own aspect ratio'+ age: 26h40m+ author: Lukas Novak <[email protected]>diff --git a/cli/test/evals/testdata/repos/feed-android/settings.gradle.kts b/cli/test/evals/testdata/repos/feed-android/settings.gradle.ktsnew file mode 100644index 00000000..9773f5ae--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-android/settings.gradle.kts@@ -0,0 +1,2 @@+rootProject.name = "pinboard"+include(":app")diff --git a/cli/test/evals/testdata/repos/feed-api-go/README.md b/cli/test/evals/testdata/repos/feed-api-go/README.mdnew file mode 100644index 00000000..e4b8f9bf--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-api-go/README.md@@ -0,0 +1,19 @@+# feed-api++The home feed of the social app (Go, `net/http`, PostgreSQL). Serves the web+app, the iOS app and the Android app.++- `GET /api/feed` - the latest 50 cards for the signed-in member+- `GET /api/feed/cards/{id}` - one card+- `GET /api/authors/{id}` - an author's profile++Card images are stored with the card (`feed_cards.image_data`, a data URI+written by the importer) and also uploaded to the media CDN under+`cards/<id>.jpg` (`CDN_BASE_URL`).++Traces go to Traceway over OTLP (otelhttp, installed by `withTracing`).++```+go test ./...+go run ./cmd/server+```diff --git a/cli/test/evals/testdata/repos/feed-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/feed-api-go/cmd/server/main.gonew file mode 100644index 00000000..f85f9b53--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-api-go/cmd/server/main.go@@ -0,0 +1,38 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"++ "example.com/feed-api/internal/feed"+)++func main() {+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ h := &feed.Handler{Store: &feed.SQL{DB: db}, CDNBase: envOr("CDN_BASE_URL", "https://media.social.example")}+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/feed", h.Feed)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, withTracing(mux)); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}++// withTracing installs the OpenTelemetry HTTP middleware (otelhttp.NewHandler+// in production builds).+func withTracing(h http.Handler) http.Handler { return h }diff --git a/cli/test/evals/testdata/repos/feed-api-go/go.mod b/cli/test/evals/testdata/repos/feed-api-go/go.modnew file mode 100644index 00000000..d9a81eb9--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/feed-api++go 1.26diff --git a/cli/test/evals/testdata/repos/feed-api-go/internal/feed/handler.go b/cli/test/evals/testdata/repos/feed-api-go/internal/feed/handler.gonew file mode 100644index 00000000..5b57290c--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-api-go/internal/feed/handler.go@@ -0,0 +1,68 @@+// Package feed serves the home feed.+package feed++import (+ "encoding/json"+ "net/http"+ "sort"+ "time"+)++// Handler serves the feed routes.+type Handler struct {+ Store Store+ CDNBase string+}++type cardJSON struct {+ ID int64 `json:"id"`+ Kind string `json:"kind"`+ Title string `json:"title"`+ Body string `json:"body"`+ Image string `json:"image"`+ PublishedAt time.Time `json:"publishedAt"`+ Author authorJSON `json:"author"`+}++type authorJSON struct {+ ID int64 `json:"id"`+ Name string `json:"name"`+ Avatar string `json:"avatar"`+}++// Feed serves GET /api/feed: the latest 50 cards, ranked.+func (h *Handler) Feed(w http.ResponseWriter, r *http.Request) {+ cards, err := h.Store.Latest(r.Context(), audiences(r), 50)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ rank(cards)+ out := make([]cardJSON, 0, len(cards))+ for _, c := range cards {+ out = append(out, cardJSON{+ ID: c.ID, Kind: c.Kind, Title: c.Title, Body: c.Body, Image: c.ImageData, PublishedAt: c.PublishedAt,+ Author: authorJSON{ID: c.Author.ID, Name: c.Author.DisplayName, Avatar: c.Author.AvatarData},+ })+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(map[string]any{"cards": out})+}++func audiences(r *http.Request) []string {+ if a := r.Header.Get("X-Audience"); a != "" {+ return []string{"all", a}+ }+ return []string{"all"}+}++// rank orders cards by recency, pinned announcements first.+func rank(cards []Card) {+ sort.SliceStable(cards, func(i, j int) bool {+ pi, pj := cards[i].Kind == "announcement", cards[j].Kind == "announcement"+ if pi != pj {+ return pi+ }+ return cards[i].PublishedAt.After(cards[j].PublishedAt)+ })+}diff --git a/cli/test/evals/testdata/repos/feed-api-go/internal/feed/handler_test.go b/cli/test/evals/testdata/repos/feed-api-go/internal/feed/handler_test.gonew file mode 100644index 00000000..fbe53c2a--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-api-go/internal/feed/handler_test.go@@ -0,0 +1,35 @@+package feed++import (+ "context"+ "encoding/json"+ "net/http/httptest"+ "testing"+ "time"+)++type fakeStore struct{ cards []Card }++func (f fakeStore) Latest(context.Context, []string, int) ([]Card, error) { return f.cards, nil }++func TestFeedRanksAnnouncementsFirst(t *testing.T) {+ now := time.Now()+ h := &Handler{Store: fakeStore{cards: []Card{+ {ID: 1, Kind: "post", Title: "a", PublishedAt: now},+ {ID: 2, Kind: "announcement", Title: "b", PublishedAt: now.Add(-time.Hour)},+ {ID: 3, Kind: "post", Title: "c", PublishedAt: now.Add(-time.Minute)},+ }}}+ rec := httptest.NewRecorder()+ h.Feed(rec, httptest.NewRequest("GET", "/api/feed", nil))+ var out struct {+ Cards []struct {+ ID int64 `json:"id"`+ } `json:"cards"`+ }+ if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {+ t.Fatal(err)+ }+ if len(out.Cards) != 3 || out.Cards[0].ID != 2 || out.Cards[1].ID != 1 || out.Cards[2].ID != 3 {+ t.Fatalf("order %+v", out.Cards)+ }+}diff --git a/cli/test/evals/testdata/repos/feed-api-go/internal/feed/store.go b/cli/test/evals/testdata/repos/feed-api-go/internal/feed/store.gonew file mode 100644index 00000000..30577ecf--- /dev/null+++ b/cli/test/evals/testdata/repos/feed-api-go/internal/feed/store.go@@ -0,0 +1,53 @@+package feed++import (+ "context"+ "database/sql"+ "time"+)++// Card is one feed card as stored.+type Card struct {+ ID int64+ Kind string+ Title string+ Body string+ ImageData string // data:image/jpeg;base64,... (about 45KB per card)+ PublishedAt time.Time+ Author Author+}++// Author is a card's author.+type Author struct {+ ID int64+ DisplayName string+ AvatarData string // data:image/png;base64,...+}++// Store reads cards.+type Store interface {+ Latest(ctx context.Context, audiences []string, limit int) ([]Card, error)+}++// SQL implements Store on PostgreSQL.+type SQL struct{ DB *sql.DB }++// Latest returns the newest cards for the given audiences.+func (s *SQL) Latest(ctx context.Context, audiences []string, limit int) ([]Card, error) {+ rows, err := s.DB.QueryContext(ctx, `SELECT c.id, c.kind, c.title, c.body, c.image_data, c.published_at, a.id, a.display_name, a.avatar_data FROM feed_cards c JOIN authors a ON a.id = c.author_id WHERE c.audience = ANY($1) ORDER BY c.published_at DESC LIMIT $2`, audiences, limit)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Card+ for rows.Next() {+ var c Card+ var image, avatar sql.NullString+ if err := rows.Scan(&c.ID, &c.Kind, &c.Title, &c.Body, &image, &c.PublishedAt, &c.Author.ID, &c.Author.DisplayName, &avatar); err != nil {+ return nil, err+ }+ c.ImageData, c.Author.AvatarData = image.String, avatar.String+ out = append(out, c)+ }+ return out, rows.Err()+}diff --git a/cli/test/evals/testdata/repos/home-api-go/README.md b/cli/test/evals/testdata/repos/home-api-go/README.mdnew file mode 100644index 00000000..1cddd3ff--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/README.md@@ -0,0 +1,17 @@+# home-api++Serves the storefront home feed (`GET /api/home`): banners, collections,+top sellers, new arrivals and the other home sections, assembled from+PostgreSQL and cached in Redis as one JSON document (`home:feed:v3`).++- `internal/home/service.go` - cache lookup and rebuild+- `internal/home/build.go` - the section queries+- `internal/home/handler.go` - HTTP handler++Redis is go-redis with redisotel, PostgreSQL is pgx with otelpgx; traces go+to Traceway over OTLP.++```+go test ./...+go run ./cmd/server+```diff --git a/cli/test/evals/testdata/repos/home-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/home-api-go/cmd/server/main.gonew file mode 100644index 00000000..10b59293--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/cmd/server/main.go@@ -0,0 +1,34 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"++ "example.com/home-api/internal/home"+)++func main() {+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ svc := &home.Service{Cache: newRedisCache(os.Getenv("REDIS_URL")), Builder: &home.SQLBuilder{DB: db, Season: envOr("SEASON", "autumn")}}+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/home", svc.Handler)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, mux); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/home-api-go/cmd/server/redis.go b/cli/test/evals/testdata/repos/home-api-go/cmd/server/redis.gonew file mode 100644index 00000000..254407d6--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/cmd/server/redis.go@@ -0,0 +1,18 @@+package main++import (+ "context"+ "time"+)++// redisCache adapts go-redis (with redisotel tracing) to home.Cache in+// production builds; this stand-in keeps the module dependency-free.+type redisCache struct{ url string }++func newRedisCache(url string) *redisCache { return &redisCache{url: url} }++func (c *redisCache) Get(ctx context.Context, key string) ([]byte, bool, error) { return nil, false, nil }++func (c *redisCache) Set(ctx context.Context, key string, val []byte, ttl time.Duration) error {+ return nil+}diff --git a/cli/test/evals/testdata/repos/home-api-go/go.mod b/cli/test/evals/testdata/repos/home-api-go/go.modnew file mode 100644index 00000000..f38d9568--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/home-api++go 1.26diff --git a/cli/test/evals/testdata/repos/home-api-go/internal/home/build.go b/cli/test/evals/testdata/repos/home-api-go/internal/home/build.gonew file mode 100644index 00000000..36c8ac5e--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/internal/home/build.go@@ -0,0 +1,96 @@+package home++import (+ "context"+ "database/sql"+)++// Feed is the home page document.+type Feed struct {+ Banners []Row `json:"banners"`+ Collections []Row `json:"collections"`+ TopSellers []Row `json:"topSellers"`+ NewArrivals []Row `json:"newArrivals"`+ Editorial []Row `json:"editorial"`+ Trending []Row `json:"trending"`+ Categories []Row `json:"categories"`+ Deals []Row `json:"deals"`+ Brands []Row `json:"brands"`+ TopReviewed []Row `json:"topReviewed"`+ Stories []Row `json:"stories"`+ ForYou []Row `json:"forYou"`+}++// Row is one result row, column name to value.+type Row = map[string]any++// sections are the home queries, in page order.+var sections = []struct {+ field func(*Feed) *[]Row+ query string+}{+ {func(f *Feed) *[]Row { return &f.Banners }, `SELECT id, image_url, target_url, starts_at, ends_at FROM banners WHERE placement = 'home' AND now() BETWEEN starts_at AND ends_at ORDER BY priority`},+ {func(f *Feed) *[]Row { return &f.Collections }, `SELECT c.id, c.title, c.slug, c.hero_image FROM collections c WHERE c.featured ORDER BY c.position LIMIT 6`},+ {func(f *Feed) *[]Row { return &f.TopSellers }, `SELECT ol.sku, SUM(ol.qty) AS sold FROM order_lines ol JOIN orders o ON o.id = ol.order_id WHERE o.placed_at > now() - interval '7 days' GROUP BY ol.sku ORDER BY sold DESC LIMIT 24`},+ {func(f *Feed) *[]Row { return &f.NewArrivals }, `SELECT id, sku, title, price_minor, image_url FROM products WHERE published_at > now() - interval '14 days' ORDER BY published_at DESC LIMIT 24`},+ {func(f *Feed) *[]Row { return &f.Editorial }, `SELECT p.id, p.sku, p.title, p.price_minor, p.image_url FROM editorial_picks e JOIN products p ON p.id = e.product_id WHERE e.week = date_trunc('week', now()) ORDER BY e.position`},+ {func(f *Feed) *[]Row { return &f.Trending }, `SELECT term, count(*) AS n FROM search_events WHERE created_at > now() - interval '1 day' GROUP BY term ORDER BY n DESC LIMIT 10`},+ {func(f *Feed) *[]Row { return &f.Categories }, `SELECT id, slug, title, tile_image FROM categories WHERE parent_id IS NULL ORDER BY position`},+ {func(f *Feed) *[]Row { return &f.Deals }, `SELECT d.id, d.product_id, d.discount_pct, d.ends_at FROM deals d WHERE d.active AND d.ends_at > now() ORDER BY d.discount_pct DESC LIMIT 12`},+ {func(f *Feed) *[]Row { return &f.Brands }, `SELECT b.id, b.name, b.logo_url FROM brands b WHERE b.spotlight_until > now() LIMIT 4`},+ {func(f *Feed) *[]Row { return &f.TopReviewed }, `SELECT r.product_id, avg(r.rating) AS rating, count(*) AS n FROM reviews r WHERE r.created_at > now() - interval '3 days' GROUP BY r.product_id ORDER BY n DESC LIMIT 12`},+ {func(f *Feed) *[]Row { return &f.Stories }, `SELECT id, title, image_url FROM seasonal_stories WHERE season = $1 ORDER BY position LIMIT 3`},+ {func(f *Feed) *[]Row { return &f.ForYou }, `SELECT sku FROM product_scores WHERE segment = 'default' ORDER BY score DESC LIMIT 24`},+}++// SQLBuilder builds the feed with one query per section.+type SQLBuilder struct {+ DB *sql.DB+ Season string+}++// Build runs the section queries in page order.+func (b *SQLBuilder) Build(ctx context.Context) (*Feed, error) {+ f := &Feed{}+ for _, s := range sections {+ var args []any+ if s.field(f) == &f.Stories {+ args = append(args, b.Season)+ }+ rows, err := b.DB.QueryContext(ctx, s.query, args...)+ if err != nil {+ return nil, err+ }+ items, err := scanMaps(rows)+ if err != nil {+ return nil, err+ }+ *s.field(f) = items+ }+ return f, nil+}++func scanMaps(rows *sql.Rows) ([]Row, error) {+ defer rows.Close()+ cols, err := rows.Columns()+ if err != nil {+ return nil, err+ }+ var out []Row+ for rows.Next() {+ vals := make([]any, len(cols))+ ptrs := make([]any, len(cols))+ for i := range vals {+ ptrs[i] = &vals[i]+ }+ if err := rows.Scan(ptrs...); err != nil {+ return nil, err+ }+ m := Row{}+ for i, c := range cols {+ m[c] = vals[i]+ }+ out = append(out, m)+ }+ return out, rows.Err()+}diff --git a/cli/test/evals/testdata/repos/home-api-go/internal/home/handler.go b/cli/test/evals/testdata/repos/home-api-go/internal/home/handler.gonew file mode 100644index 00000000..56eb02df--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/internal/home/handler.go@@ -0,0 +1,17 @@+package home++import (+ "encoding/json"+ "net/http"+)++// Handler serves GET /api/home.+func (s *Service) Handler(w http.ResponseWriter, r *http.Request) {+ f, err := s.Feed(r.Context())+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(f)+}diff --git a/cli/test/evals/testdata/repos/home-api-go/internal/home/service.go b/cli/test/evals/testdata/repos/home-api-go/internal/home/service.gonew file mode 100644index 00000000..8d20c967--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/internal/home/service.go@@ -0,0 +1,50 @@+// Package home assembles and caches the storefront home feed.+package home++import (+ "context"+ "encoding/json"+ "time"+)++// FeedKey is the Redis key of the cached home feed.+const FeedKey = "home:feed:v3"++// FeedTTL is how long a built feed is served from the cache.+const FeedTTL = 10 * time.Minute++// Cache is the subset of Redis the service uses. Get reports ok=false on a+// miss.+type Cache interface {+ Get(ctx context.Context, key string) (val []byte, ok bool, err error)+ Set(ctx context.Context, key string, val []byte, ttl time.Duration) error+}++// Builder assembles the feed from the database.+type Builder interface {+ Build(ctx context.Context) (*Feed, error)+}++// Service returns the home feed.+type Service struct {+ Cache Cache+ Builder Builder+}++// Feed returns the cached feed, rebuilding and caching it on a miss.+func (s *Service) Feed(ctx context.Context) (*Feed, error) {+ if raw, ok, err := s.Cache.Get(ctx, FeedKey); err == nil && ok {+ var f Feed+ if err := json.Unmarshal(raw, &f); err == nil {+ return &f, nil+ }+ }+ f, err := s.Builder.Build(ctx)+ if err != nil {+ return nil, err+ }+ if raw, err := json.Marshal(f); err == nil {+ _ = s.Cache.Set(ctx, FeedKey, raw, FeedTTL)+ }+ return f, nil+}diff --git a/cli/test/evals/testdata/repos/home-api-go/internal/home/service_test.go b/cli/test/evals/testdata/repos/home-api-go/internal/home/service_test.gonew file mode 100644index 00000000..63d73007--- /dev/null+++ b/cli/test/evals/testdata/repos/home-api-go/internal/home/service_test.go@@ -0,0 +1,53 @@+package home++import (+ "context"+ "sync"+ "testing"+ "time"+)++type memCache struct {+ mu sync.Mutex+ m map[string][]byte+}++func (c *memCache) Get(_ context.Context, k string) ([]byte, bool, error) {+ c.mu.Lock()+ defer c.mu.Unlock()+ v, ok := c.m[k]+ return v, ok, nil+}++func (c *memCache) Set(_ context.Context, k string, v []byte, _ time.Duration) error {+ c.mu.Lock()+ defer c.mu.Unlock()+ c.m[k] = v+ return nil+}++type countingBuilder struct {+ mu sync.Mutex+ calls int+}++func (b *countingBuilder) Build(context.Context) (*Feed, error) {+ b.mu.Lock()+ b.calls+++ b.mu.Unlock()+ return &Feed{Banners: []Row{{"id": 1}}}, nil+}++func TestFeedIsCachedAfterFirstBuild(t *testing.T) {+ b := &countingBuilder{}+ s := &Service{Cache: &memCache{m: map[string][]byte{}}, Builder: b}+ for i := 0; i < 3; i++ {+ f, err := s.Feed(context.Background())+ if err != nil || len(f.Banners) != 1 {+ t.Fatalf("feed %v %v", f, err)+ }+ }+ if b.calls != 1 {+ t.Fatalf("built %d times, want 1", b.calls)+ }+}diff --git a/cli/test/evals/testdata/repos/orders-api-go/.history/01/internal/handlers/orders.go b/cli/test/evals/testdata/repos/orders-api-go/.history/01/internal/handlers/orders.gonew file mode 100644index 00000000..081e8d22--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/.history/01/internal/handlers/orders.go@@ -0,0 +1,59 @@+// Package handlers serves the orders API.+package handlers++import (+ "encoding/json"+ "net/http"+ "strconv"++ "example.com/orders-api/internal/store"+)++// Handlers holds the HTTP handlers.+type Handlers struct{ store store.Store }++// New returns the handlers.+func New(s store.Store) *Handlers { return &Handlers{store: s} }++// ListOrders serves GET /api/orders: the signed-in customer's orders.+func (h *Handlers) ListOrders(w http.ResponseWriter, r *http.Request) {+ userID, err := strconv.ParseInt(r.Header.Get("X-User-Id"), 10, 64)+ if err != nil {+ http.Error(w, "unauthorized", http.StatusUnauthorized)+ return+ }+ orders, err := h.store.OrdersByUser(r.Context(), userID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ writeJSON(w, orders)+}++// GetOrder serves GET /api/orders/{id}.+func (h *Handlers) GetOrder(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad id", http.StatusBadRequest)+ return+ }+ o, err := h.store.OrderByID(r.Context(), id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if o == nil {+ http.NotFound(w, r)+ return+ }+ if o.Items, err = h.store.ItemsByOrder(r.Context(), o.ID); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ writeJSON(w, o)+}++func writeJSON(w http.ResponseWriter, v any) {+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(v)+}diff --git a/cli/test/evals/testdata/repos/orders-api-go/.history/01/internal/handlers/orders_test.go b/cli/test/evals/testdata/repos/orders-api-go/.history/01/internal/handlers/orders_test.gonew file mode 100644index 00000000..4e6bb866--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/.history/01/internal/handlers/orders_test.go@@ -0,0 +1,61 @@+package handlers++import (+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "testing"++ "example.com/orders-api/internal/store"+)++type fakeStore struct {+ orders map[int64][]store.Order+ items map[int64][]store.Item+}++func (f *fakeStore) OrdersByUser(_ context.Context, userID int64) ([]store.Order, error) {+ return append([]store.Order{}, f.orders[userID]...), nil+}++func (f *fakeStore) OrderByID(_ context.Context, id int64) (*store.Order, error) {+ for _, os := range f.orders {+ for _, o := range os {+ if o.ID == id {+ return &o, nil+ }+ }+ }+ return nil, nil+}++func (f *fakeStore) ItemsByOrder(_ context.Context, orderID int64) ([]store.Item, error) {+ return f.items[orderID], nil+}++func TestListOrdersIncludesItems(t *testing.T) {+ fs := &fakeStore{+ orders: map[int64][]store.Order{7: {{ID: 1, UserID: 7}, {ID: 2, UserID: 7}}},+ items: map[int64][]store.Item{1: {{ID: 10, SKU: "A"}}, 2: {{ID: 11, SKU: "B"}, {ID: 12, SKU: "C"}}},+ }+ req := httptest.NewRequest(http.MethodGet, "/api/orders", nil)+ req.Header.Set("X-User-Id", "7")+ rec := httptest.NewRecorder()+ New(fs).ListOrders(rec, req)+ var got []store.Order+ if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {+ t.Fatal(err)+ }+ if len(got) != 2 {+ t.Fatalf("unexpected orders: %+v", got)+ }+}++func TestListOrdersNeedsUser(t *testing.T) {+ rec := httptest.NewRecorder()+ New(&fakeStore{}).ListOrders(rec, httptest.NewRequest(http.MethodGet, "/api/orders", nil))+ if rec.Code != http.StatusUnauthorized {+ t.Fatalf("status %d", rec.Code)+ }+}diff --git a/cli/test/evals/testdata/repos/orders-api-go/README.md b/cli/test/evals/testdata/repos/orders-api-go/README.mdnew file mode 100644index 00000000..a7262ae3--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/README.md@@ -0,0 +1,15 @@+# orders-api++The storefront's order service: order history and order detail for signed-in+customers. Go, `net/http`, PostgreSQL through `database/sql`. Traces, logs and+metrics go to Traceway over OTLP (`OTEL_EXPORTER_OTLP_ENDPOINT`).++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/orders` – the signed-in customer's orders, newest first, with their line items+- `GET /api/orders/{id}` – one order with its line itemsdiff --git a/cli/test/evals/testdata/repos/orders-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/orders-api-go/cmd/server/main.gonew file mode 100644index 00000000..c747d525--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/cmd/server/main.go@@ -0,0 +1,40 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"++ "example.com/orders-api/internal/handlers"+ "example.com/orders-api/internal/store"+)++func main() {+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ h := handlers.New(store.New(db))+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/orders", h.ListOrders)+ mux.HandleFunc("GET /api/orders/{id}", h.GetOrder)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, withTracing(mux)); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}++// withTracing is where the OpenTelemetry HTTP middleware is installed+// (otelhttp.NewHandler in production builds).+func withTracing(h http.Handler) http.Handler { return h }diff --git a/cli/test/evals/testdata/repos/orders-api-go/go.mod b/cli/test/evals/testdata/repos/orders-api-go/go.modnew file mode 100644index 00000000..e05415bf--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/orders-api++go 1.26diff --git a/cli/test/evals/testdata/repos/orders-api-go/history.yaml b/cli/test/evals/testdata/repos/orders-api-go/history.yamlnew file mode 100644index 00000000..1c257580--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/history.yaml@@ -0,0 +1,7 @@+commits:+- dir: .history/01+ message: 'orders: list and detail endpoints'+ age: 2400h+head:+ message: 'orders: include line items in the order list'+ age: 1500hdiff --git a/cli/test/evals/testdata/repos/orders-api-go/internal/handlers/orders.go b/cli/test/evals/testdata/repos/orders-api-go/internal/handlers/orders.gonew file mode 100644index 00000000..8508e6c2--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/internal/handlers/orders.go@@ -0,0 +1,68 @@+// Package handlers serves the orders API.+package handlers++import (+ "encoding/json"+ "net/http"+ "strconv"++ "example.com/orders-api/internal/store"+)++// Handlers holds the HTTP handlers.+type Handlers struct{ store store.Store }++// New returns the handlers.+func New(s store.Store) *Handlers { return &Handlers{store: s} }++// ListOrders serves GET /api/orders: the signed-in customer's orders with+// their items.+func (h *Handlers) ListOrders(w http.ResponseWriter, r *http.Request) {+ userID, err := strconv.ParseInt(r.Header.Get("X-User-Id"), 10, 64)+ if err != nil {+ http.Error(w, "unauthorized", http.StatusUnauthorized)+ return+ }+ orders, err := h.store.OrdersByUser(r.Context(), userID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ for i := range orders {+ items, err := h.store.ItemsByOrder(r.Context(), orders[i].ID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ orders[i].Items = items+ }+ writeJSON(w, orders)+}++// GetOrder serves GET /api/orders/{id}.+func (h *Handlers) GetOrder(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad id", http.StatusBadRequest)+ return+ }+ o, err := h.store.OrderByID(r.Context(), id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if o == nil {+ http.NotFound(w, r)+ return+ }+ if o.Items, err = h.store.ItemsByOrder(r.Context(), o.ID); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ writeJSON(w, o)+}++func writeJSON(w http.ResponseWriter, v any) {+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(v)+}diff --git a/cli/test/evals/testdata/repos/orders-api-go/internal/handlers/orders_test.go b/cli/test/evals/testdata/repos/orders-api-go/internal/handlers/orders_test.gonew file mode 100644index 00000000..fc08084a--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/internal/handlers/orders_test.go@@ -0,0 +1,61 @@+package handlers++import (+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "testing"++ "example.com/orders-api/internal/store"+)++type fakeStore struct {+ orders map[int64][]store.Order+ items map[int64][]store.Item+}++func (f *fakeStore) OrdersByUser(_ context.Context, userID int64) ([]store.Order, error) {+ return append([]store.Order{}, f.orders[userID]...), nil+}++func (f *fakeStore) OrderByID(_ context.Context, id int64) (*store.Order, error) {+ for _, os := range f.orders {+ for _, o := range os {+ if o.ID == id {+ return &o, nil+ }+ }+ }+ return nil, nil+}++func (f *fakeStore) ItemsByOrder(_ context.Context, orderID int64) ([]store.Item, error) {+ return f.items[orderID], nil+}++func TestListOrdersIncludesItems(t *testing.T) {+ fs := &fakeStore{+ orders: map[int64][]store.Order{7: {{ID: 1, UserID: 7}, {ID: 2, UserID: 7}}},+ items: map[int64][]store.Item{1: {{ID: 10, SKU: "A"}}, 2: {{ID: 11, SKU: "B"}, {ID: 12, SKU: "C"}}},+ }+ req := httptest.NewRequest(http.MethodGet, "/api/orders", nil)+ req.Header.Set("X-User-Id", "7")+ rec := httptest.NewRecorder()+ New(fs).ListOrders(rec, req)+ var got []store.Order+ if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {+ t.Fatal(err)+ }+ if len(got) != 2 || len(got[0].Items) != 1 || len(got[1].Items) != 2 {+ t.Fatalf("unexpected orders: %+v", got)+ }+}++func TestListOrdersNeedsUser(t *testing.T) {+ rec := httptest.NewRecorder()+ New(&fakeStore{}).ListOrders(rec, httptest.NewRequest(http.MethodGet, "/api/orders", nil))+ if rec.Code != http.StatusUnauthorized {+ t.Fatalf("status %d", rec.Code)+ }+}diff --git a/cli/test/evals/testdata/repos/orders-api-go/internal/store/store.go b/cli/test/evals/testdata/repos/orders-api-go/internal/store/store.gonew file mode 100644index 00000000..15100274--- /dev/null+++ b/cli/test/evals/testdata/repos/orders-api-go/internal/store/store.go@@ -0,0 +1,87 @@+// Package store reads orders from PostgreSQL.+package store++import (+ "context"+ "database/sql"+ "time"+)++// Order is one customer order.+type Order struct {+ ID int64 `json:"id"`+ UserID int64 `json:"userId"`+ Status string `json:"status"`+ TotalCents int64 `json:"totalCents"`+ CreatedAt time.Time `json:"createdAt"`+ Items []Item `json:"items"`+}++// Item is one order line.+type Item struct {+ ID int64 `json:"id"`+ SKU string `json:"sku"`+ Qty int `json:"qty"`+ UnitPrice int64 `json:"unitPrice"`+}++// Store is the data access layer.+type Store interface {+ OrdersByUser(ctx context.Context, userID int64) ([]Order, error)+ OrderByID(ctx context.Context, id int64) (*Order, error)+ ItemsByOrder(ctx context.Context, orderID int64) ([]Item, error)+}++// SQL implements Store on database/sql.+type SQL struct{ db *sql.DB }++// New returns a Store backed by db.+func New(db *sql.DB) *SQL { return &SQL{db: db} }++// OrdersByUser returns a user's latest 100 orders.+func (s *SQL) OrdersByUser(ctx context.Context, userID int64) ([]Order, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT * FROM orders WHERE user_id = $1 ORDER BY created_at DESC LIMIT 100`, userID)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Order+ for rows.Next() {+ var o Order+ var shippingAddress, notes sql.NullString+ if err := rows.Scan(&o.ID, &o.UserID, &o.Status, &o.TotalCents, &shippingAddress, ¬es, &o.CreatedAt); err != nil {+ return nil, err+ }+ out = append(out, o)+ }+ return out, rows.Err()+}++// OrderByID returns one order.+func (s *SQL) OrderByID(ctx context.Context, id int64) (*Order, error) {+ var o Order+ err := s.db.QueryRowContext(ctx, `SELECT id, user_id, status, total_cents, created_at FROM orders WHERE id = $1`, id).+ Scan(&o.ID, &o.UserID, &o.Status, &o.TotalCents, &o.CreatedAt)+ if err == sql.ErrNoRows {+ return nil, nil+ }+ return &o, err+}++// ItemsByOrder returns one order's line items.+func (s *SQL) ItemsByOrder(ctx context.Context, orderID int64) ([]Item, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT id, sku, qty, unit_price FROM order_items WHERE order_id = $1`, orderID)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Item+ for rows.Next() {+ var it Item+ if err := rows.Scan(&it.ID, &it.SKU, &it.Qty, &it.UnitPrice); err != nil {+ return nil, err+ }+ out = append(out, it)+ }+ return out, rows.Err()+}diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/README.md b/cli/test/evals/testdata/repos/portal-web-api-nest/README.mdnew file mode 100644index 00000000..4833a54e--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/README.md@@ -0,0 +1,17 @@+# portal web-api++The backend-for-frontend of the customer portal (NestJS). It serves the+portal's pages by calling the internal services:++| Service | What the portal reads |+|---|---|+| accounts-service | account profile, members, feature flags |+| billing-service | open invoices and the current balance |+| notifications-service | unread notifications |+| recommendations-service | suggested catalog items |++Traces go to Traceway over OTLP (`src/tracing.ts`, loaded with `node -r`).++```+npm run build && npm start+```diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/package.json b/cli/test/evals/testdata/repos/portal-web-api-nest/package.jsonnew file mode 100644index 00000000..2338f036--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/package.json@@ -0,0 +1,29 @@+{+ "name": "portal-web-api",+ "version": "3.7.0",+ "private": true,+ "scripts": {+ "build": "nest build",+ "start": "node -r ./dist/tracing.js dist/main.js",+ "test": "jest"+ },+ "dependencies": {+ "@nestjs/common": "^11.1.6",+ "@nestjs/core": "^11.1.6",+ "@nestjs/platform-express": "^11.1.6",+ "@opentelemetry/api": "^1.9.0",+ "@opentelemetry/auto-instrumentations-node": "^0.62.0",+ "@opentelemetry/exporter-trace-otlp-proto": "^0.203.0",+ "@opentelemetry/sdk-node": "^0.203.0",+ "reflect-metadata": "^0.2.2",+ "rxjs": "^7.8.2"+ },+ "devDependencies": {+ "@nestjs/cli": "^11.0.10",+ "@nestjs/testing": "^11.1.6",+ "@types/jest": "^30.0.0",+ "jest": "^30.0.5",+ "ts-jest": "^29.4.1",+ "typescript": "^5.9.2"+ }+}diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/src/clients/internal-client.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/src/clients/internal-client.tsnew file mode 100644index 00000000..b67e5423--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/src/clients/internal-client.ts@@ -0,0 +1,21 @@+import { Injectable } from '@nestjs/common';++const BASE = {+ accounts: process.env.ACCOUNTS_URL ?? 'http://accounts-service.internal:8080',+ billing: process.env.BILLING_URL ?? 'http://billing-service.internal:8080',+ notifications: process.env.NOTIFICATIONS_URL ?? 'http://notifications-service.internal:8080',+ recommendations: process.env.RECOMMENDATIONS_URL ?? 'http://recommendations-service.internal:8080',+};++export type Service = keyof typeof BASE;++@Injectable()+export class InternalClient {+ async get<T>(service: Service, path: string): Promise<T> {+ const res = await fetch(BASE[service] + path, { headers: { accept: 'application/json' } });+ if (!res.ok) {+ throw new Error(`${service} ${path} answered ${res.status}`);+ }+ return (await res.json()) as T;+ }+}diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.controller.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.controller.tsnew file mode 100644index 00000000..f0dbe4b8--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.controller.ts@@ -0,0 +1,12 @@+import { Controller, Get, Headers } from '@nestjs/common';+import { DashboardService } from './dashboard.service';++@Controller('api/dashboard')+export class DashboardController {+ constructor(private readonly dashboard: DashboardService) {}++ @Get()+ getDashboard(@Headers('x-account-id') accountId: string) {+ return this.dashboard.load(accountId);+ }+}diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.presenter.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.presenter.tsnew file mode 100644index 00000000..ac42d161--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.presenter.ts@@ -0,0 +1,29 @@+import { Injectable } from '@nestjs/common';+import { trace } from '@opentelemetry/api';++export interface Profile { id: string; name: string; plan: string; seats: number; members: { userId: string; role: string }[] }+export interface BillingSummary { balanceCents: number; invoices: { number: string; status: string; dueAt: string }[] }+export interface Unread { count: number; latest: { id: string; kind: string; title: string }[] }+export interface Suggestion { itemId: string; category: string; score: number }++const tracer = trace.getTracer('web-api');++@Injectable()+export class DashboardPresenter {+ compose(profile: Profile, billing: BillingSummary, unread: Unread, suggestions: Suggestion[]) {+ return tracer.startActiveSpan('DashboardPresenter.compose', (span) => {+ try {+ return {+ account: { id: profile.id, name: profile.name, plan: profile.plan, seats: profile.seats },+ team: profile.members.length,+ balance: billing.balanceCents,+ invoices: billing.invoices.filter((i) => i.status !== 'paid'),+ notifications: unread,+ suggestions: suggestions.slice(0, 8),+ };+ } finally {+ span.end();+ }+ });+ }+}diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.service.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.service.tsnew file mode 100644index 00000000..0b31527a--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/src/dashboard/dashboard.service.ts@@ -0,0 +1,24 @@+import { Injectable } from '@nestjs/common';+import { InternalClient } from '../clients/internal-client';+import { DashboardPresenter, Profile, BillingSummary, Unread, Suggestion } from './dashboard.presenter';++@Injectable()+export class DashboardService {+ constructor(+ private readonly client: InternalClient,+ private readonly presenter: DashboardPresenter,+ ) {}++ // Everything the portal home page shows for one account.+ async load(accountId: string) {+ const id = encodeURIComponent(accountId);+ const profile = await this.client.get<Profile>('accounts', `/internal/accounts/${id}/profile`);+ const billing = await this.client.get<BillingSummary>('billing', `/internal/billing/${id}/summary`);+ const unread = await this.client.get<Unread>('notifications', `/internal/notifications/unread?accountId=${id}`);+ const suggestions = await this.client.get<Suggestion[]>(+ 'recommendations',+ `/internal/recommendations?accountId=${id}&limit=8`,+ );+ return this.presenter.compose(profile, billing, unread, suggestions);+ }+}diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/src/main.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/src/main.tsnew file mode 100644index 00000000..db4cbc33--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/src/main.ts@@ -0,0 +1,19 @@+import { NestFactory } from '@nestjs/core';+import { Module } from '@nestjs/common';+import { DashboardController } from './dashboard/dashboard.controller';+import { DashboardService } from './dashboard/dashboard.service';+import { DashboardPresenter } from './dashboard/dashboard.presenter';+import { InternalClient } from './clients/internal-client';++@Module({+ controllers: [DashboardController],+ providers: [DashboardService, DashboardPresenter, InternalClient],+})+class AppModule {}++async function bootstrap() {+ const app = await NestFactory.create(AppModule);+ await app.listen(process.env.PORT ?? 3000);+}++bootstrap();diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/src/tracing.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/src/tracing.tsnew file mode 100644index 00000000..c8f982d4--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/src/tracing.ts@@ -0,0 +1,11 @@+import { NodeSDK } from '@opentelemetry/sdk-node';+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';+import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';++const sdk = new NodeSDK({+ serviceName: 'web-api',+ traceExporter: new OTLPTraceExporter(),+ instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false } })],+});++sdk.start();diff --git a/cli/test/evals/testdata/repos/portal-web-api-nest/test/dashboard.service.spec.ts b/cli/test/evals/testdata/repos/portal-web-api-nest/test/dashboard.service.spec.tsnew file mode 100644index 00000000..0b88f1cb--- /dev/null+++ b/cli/test/evals/testdata/repos/portal-web-api-nest/test/dashboard.service.spec.ts@@ -0,0 +1,19 @@+import { DashboardService } from '../src/dashboard/dashboard.service';+import { DashboardPresenter } from '../src/dashboard/dashboard.presenter';++describe('DashboardService', () => {+ it('composes the dashboard from the four services', async () => {+ const responses: Record<string, unknown> = {+ accounts: { id: '42', name: 'Acme', plan: 'team', seats: 10, members: [{ userId: '1', role: 'owner' }] },+ billing: { balanceCents: 1200, invoices: [{ number: 'INV-1', status: 'open', dueAt: '2026-11-01' }] },+ notifications: { count: 2, latest: [] },+ recommendations: [{ itemId: 'a', category: 'x', score: 0.9 }],+ };+ const client = { get: jest.fn(async (service: string) => responses[service]) };+ const svc = new DashboardService(client as never, new DashboardPresenter());+ const out = await svc.load('42');+ expect(out.account.name).toBe('Acme');+ expect(out.invoices).toHaveLength(1);+ expect(client.get).toHaveBeenCalledTimes(4);+ });+});diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/README.md b/cli/test/evals/testdata/repos/shop-android-and-api/README.mdnew file mode 100644index 00000000..901af476--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/README.md@@ -0,0 +1,9 @@+# shop (Android app + checkout API)++- `android/`: the Android shopping app (Kotlin, Retrofit + OkHttp,+ coroutines). Reports crashes to Traceway with the Android SDK (project+ **shop-android**), with the network calls `TracewayOkHttpInterceptor`+ records.+- `backend/`: the checkout API (Java 21, Spring Boot 3, JPA on PostgreSQL,+ HikariCP). Traced with the OpenTelemetry Java agent (project+ **commerce-api**).diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/checkout/CheckoutRepository.kt b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/checkout/CheckoutRepository.ktnew file mode 100644index 00000000..48e0cb57--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/checkout/CheckoutRepository.kt@@ -0,0 +1,39 @@+package com.example.shop.checkout++import com.example.shop.net.ApiClient+import com.example.shop.net.CheckoutRequest+import com.example.shop.net.OrderDto+import com.example.shop.net.ShopApi+import kotlinx.coroutines.Dispatchers+import kotlinx.coroutines.withContext+import retrofit2.HttpException++/**+ * Places orders. The call is blocking (Call.execute) on the IO dispatcher so+ * the response can be mapped in one place.+ */+class CheckoutRepository(+ private val api: ShopApi = ApiClient.shop,+) {+ sealed interface Result {+ data class Placed(val order: OrderDto) : Result+ data class Rejected(val code: Int, val message: String) : Result+ }++ /**+ * POST /api/checkout. Returns Placed or Rejected; HttpException for+ * other HTTP failures.+ */+ suspend fun placeOrder(cartId: String, paymentMethodId: String): Result =+ withContext(Dispatchers.IO) {+ val response = api+ .checkout(CheckoutRequest(cartId, paymentMethodId))+ .execute()+ when {+ response.isSuccessful -> Result.Placed(response.body()!!)+ response.code() == 409 || response.code() == 422 ->+ Result.Rejected(response.code(), response.errorBody()?.string().orEmpty())+ else -> throw HttpException(response)+ }+ }+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/checkout/CheckoutViewModel.kt b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/checkout/CheckoutViewModel.ktnew file mode 100644index 00000000..f5312277--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/checkout/CheckoutViewModel.kt@@ -0,0 +1,32 @@+package com.example.shop.checkout++import androidx.lifecycle.ViewModel+import androidx.lifecycle.viewModelScope+import kotlinx.coroutines.flow.MutableStateFlow+import kotlinx.coroutines.flow.StateFlow+import kotlinx.coroutines.launch++class CheckoutViewModel(+ private val repository: CheckoutRepository = CheckoutRepository(),+) : ViewModel() {+ sealed interface UiState {+ data object Idle : UiState+ data object Placing : UiState+ data class Done(val orderId: String) : UiState+ data class Failed(val message: String) : UiState+ }++ private val _state = MutableStateFlow<UiState>(UiState.Idle)+ val state: StateFlow<UiState> = _state++ fun placeOrder(cartId: String, itemCount: Int, paymentMethodId: String) {+ println("CheckoutViewModel: placing order ($itemCount items)")+ _state.value = UiState.Placing+ viewModelScope.launch {+ when (val result = repository.placeOrder(cartId, paymentMethodId)) {+ is CheckoutRepository.Result.Placed -> _state.value = UiState.Done(result.order.id)+ is CheckoutRepository.Result.Rejected -> _state.value = UiState.Failed(result.message)+ }+ }+ }+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/ApiClient.kt b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/ApiClient.ktnew file mode 100644index 00000000..03fb6d94--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/ApiClient.kt@@ -0,0 +1,23 @@+package com.example.shop.net++import com.example.shop.BuildConfig+import com.tracewayapp.traceway.network.TracewayOkHttpInterceptor+import okhttp3.OkHttpClient+import retrofit2.Retrofit+import retrofit2.converter.moshi.MoshiConverterFactory++object ApiClient {+ // OkHttp's defaults: 10s connect, 10s read, 10s write.+ private val http = OkHttpClient.Builder()+ .addInterceptor(AuthInterceptor())+ .addInterceptor(TracewayOkHttpInterceptor())+ .build()++ val retrofit: Retrofit = Retrofit.Builder()+ .baseUrl(BuildConfig.API_BASE_URL)+ .client(http)+ .addConverterFactory(MoshiConverterFactory.create())+ .build()++ val shop: ShopApi = retrofit.create(ShopApi::class.java)+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/AuthInterceptor.kt b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/AuthInterceptor.ktnew file mode 100644index 00000000..7a99483e--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/AuthInterceptor.kt@@ -0,0 +1,20 @@+package com.example.shop.net++import com.example.shop.session.SessionStore+import okhttp3.Interceptor+import okhttp3.Response++/** Adds the signed-in user's bearer token to every API call. */+class AuthInterceptor : Interceptor {+ override fun intercept(chain: Interceptor.Chain): Response {+ val token = SessionStore.accessToken()+ val request = if (token == null) {+ chain.request()+ } else {+ chain.request().newBuilder()+ .header("Authorization", "Bearer $token")+ .build()+ }+ return chain.proceed(request)+ }+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/ShopApi.kt b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/ShopApi.ktnew file mode 100644index 00000000..5cc58a7b--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/android/app/src/main/java/com/example/shop/net/ShopApi.kt@@ -0,0 +1,19 @@+package com.example.shop.net++import retrofit2.Call+import retrofit2.http.Body+import retrofit2.http.GET+import retrofit2.http.POST++data class CartDto(val id: String, val items: List<CartItemDto>, val totalCents: Long)+data class CartItemDto(val sku: String, val qty: Int, val unitPriceCents: Long)+data class CheckoutRequest(val cartId: String, val paymentMethodId: String)+data class OrderDto(val id: String, val status: String, val totalCents: Long)++interface ShopApi {+ @GET("api/cart")+ fun cart(): Call<CartDto>++ @POST("api/checkout")+ fun checkout(@Body body: CheckoutRequest): Call<OrderDto>+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/CheckoutController.java b/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/CheckoutController.javanew file mode 100644index 00000000..acf90698--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/CheckoutController.java@@ -0,0 +1,30 @@+package com.example.shop.checkout;++import org.springframework.http.ResponseEntity;+import org.springframework.web.bind.annotation.PostMapping;+import org.springframework.web.bind.annotation.RequestBody;+import org.springframework.web.bind.annotation.RequestMapping;+import org.springframework.web.bind.annotation.RestController;++@RestController+@RequestMapping("/api")+public class CheckoutController {++ public record CheckoutRequest(String cartId, String paymentMethodId) {}++ public record OrderResponse(String id, String status, long totalCents) {}++ private final CheckoutService checkoutService;++ public CheckoutController(CheckoutService checkoutService) {+ this.checkoutService = checkoutService;+ }++ /** POST /api/checkout: reserve stock, charge, create the order. */+ @PostMapping("/checkout")+ public ResponseEntity<OrderResponse> checkout(@RequestBody CheckoutRequest req) {+ // The authenticated customer comes from the security context.+ var order = checkoutService.placeOrder(req.cartId(), req.paymentMethodId());+ return ResponseEntity.ok(new OrderResponse(order.id(), order.status(), order.totalCents()));+ }+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/CheckoutService.java b/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/CheckoutService.javanew file mode 100644index 00000000..291a9a25--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/CheckoutService.java@@ -0,0 +1,52 @@+package com.example.shop.checkout;++import java.util.UUID;+import org.springframework.stereotype.Service;+import org.springframework.transaction.annotation.Transactional;++@Service+public class CheckoutService {++ public record Order(String id, String status, long totalCents) {}++ private final CartRepository carts;+ private final InventoryRepository inventory;+ private final PaymentClient payments;+ private final OrderRepository orders;++ public CheckoutService(CartRepository carts, InventoryRepository inventory, PaymentClient payments, OrderRepository orders) {+ this.carts = carts;+ this.inventory = inventory;+ this.payments = payments;+ this.orders = orders;+ }++ /**+ * One transaction: reserve every line's stock, charge the card, write the+ * order. If the charge fails the reservation rolls back with it.+ */+ @Transactional+ public Order placeOrder(String cartId, String paymentMethodId) {+ var cart = carts.findWithItems(cartId)+ .orElseThrow(() -> new CartNotFoundException(cartId));+ if (cart.items().isEmpty()) {+ throw new EmptyCartException(cartId);+ }++ // Reserve in cart order. UPDATE ... WHERE available - reserved >= qty+ // takes the row lock on the inventory row until commit.+ for (var item : cart.items()) {+ int updated = inventory.reserve(item.sku(), item.qty());+ if (updated == 0) {+ throw new OutOfStockException(item.sku());+ }+ inventory.flush();+ }++ var charge = payments.charge(paymentMethodId, cart.totalCents(), cart.currency());++ var orderId = UUID.randomUUID().toString();+ orders.insert(orderId, cart, charge.reference());+ return new Order(orderId, "placed", cart.totalCents());+ }+}diff --git a/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/InventoryRepository.java b/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/InventoryRepository.javanew file mode 100644index 00000000..c408a1c2--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-android-and-api/backend/src/main/java/com/example/shop/checkout/InventoryRepository.java@@ -0,0 +1,16 @@+package com.example.shop.checkout;++import org.springframework.data.jpa.repository.Modifying;+import org.springframework.data.jpa.repository.Query;+import org.springframework.data.repository.Repository;+import org.springframework.data.repository.query.Param;++public interface InventoryRepository extends Repository<InventoryItem, String> {++ /** Reserves qty units of sku; 0 rows updated means not enough stock. */+ @Modifying+ @Query(value = "update inventory set reserved=reserved+:qty where sku=:sku and available-reserved>=:qty", nativeQuery = true)+ int reserve(@Param("sku") String sku, @Param("qty") int qty);++ void flush();+}diff --git a/cli/test/evals/testdata/repos/shop-catalog/README.md b/cli/test/evals/testdata/repos/shop-catalog/README.mdnew file mode 100644index 00000000..b30571fa--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/README.md@@ -0,0 +1,14 @@+# shop++The shop's catalog API and web storefront.++- `api/` - catalog-api (Express, PostgreSQL, Redis). `GET /api/catalog`+ returns every active product with its category and price for a locale+ (~1200 products). Merchandising publishes catalog changes twice a day.+- `web/` - the storefront SPA (React, Vite) at shop.example.com, reporting+ errors and sessions to Traceway with `@tracewayapp/react`.++```+cd api && npm test+cd web && npm run dev+```diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/package.json b/cli/test/evals/testdata/repos/shop-catalog/api/package.jsonnew file mode 100644index 00000000..be3316d8--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/package.json@@ -0,0 +1,20 @@+{+ "name": "catalog-api",+ "version": "6.2.1",+ "private": true,+ "main": "src/server.js",+ "scripts": {+ "start": "node --require ./src/tracing.js src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/api": "^1.9.0",+ "@opentelemetry/auto-instrumentations-node": "^0.62.0",+ "@opentelemetry/exporter-metrics-otlp-proto": "^0.203.0",+ "@opentelemetry/exporter-trace-otlp-proto": "^0.203.0",+ "@opentelemetry/sdk-node": "^0.203.0",+ "express": "^5.1.0",+ "ioredis": "^5.7.0",+ "pg": "^8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/src/cache-metrics.js b/cli/test/evals/testdata/repos/shop-catalog/api/src/cache-metrics.jsnew file mode 100644index 00000000..549bfcb9--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/src/cache-metrics.js@@ -0,0 +1,27 @@+'use strict';+const { metrics } = require('@opentelemetry/api');++// Hit ratio per cache over each export interval, as the cache.hit_ratio+// gauge (attribute: cache).+const counts = new Map();++function record(cache, hit) {+ const c = counts.get(cache) ?? { hit: 0, miss: 0 };+ c[hit ? 'hit' : 'miss'] += 1;+ counts.set(cache, c);+}++metrics+ .getMeter('catalog-api')+ .createObservableGauge('cache.hit_ratio', { unit: '1', description: 'Share of cache lookups that hit' })+ .addCallback((result) => {+ for (const [cache, c] of counts) {+ const total = c.hit + c.miss;+ if (total > 0) {+ result.observe(c.hit / total, { cache });+ }+ counts.set(cache, { hit: 0, miss: 0 });+ }+ });++module.exports = { record };diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/src/catalog-cache.js b/cli/test/evals/testdata/repos/shop-catalog/api/src/catalog-cache.jsnew file mode 100644index 00000000..33e66590--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/src/catalog-cache.js@@ -0,0 +1,11 @@+'use strict';++// How long a built catalog response stays in Redis.+const CACHE_TTL_SECONDS = 6 * 60 * 60;++// cacheKey is the Redis key a catalog request is cached under.+function cacheKey(originalUrl) {+ return `catalog:${originalUrl}`;+}++module.exports = { CACHE_TTL_SECONDS, cacheKey };diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/src/routes/catalog.js b/cli/test/evals/testdata/repos/shop-catalog/api/src/routes/catalog.jsnew file mode 100644index 00000000..8d8098f6--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/src/routes/catalog.js@@ -0,0 +1,35 @@+'use strict';+const express = require('express');+const { CACHE_TTL_SECONDS, cacheKey } = require('../catalog-cache');+const cacheMetrics = require('../cache-metrics');++const CATALOG_SQL = `SELECT p.id, p.sku, p.title, p.brand, p.image_url, c.slug AS category, pr.amount_minor, pr.currency+ FROM products p JOIN categories c ON c.id = p.category_id JOIN prices pr ON pr.product_id = p.id AND pr.price_list = $1+ WHERE p.active ORDER BY c.position, p.rank`;++function priceList(locale) {+ return locale === 'en-GB' ? 'gbp' : locale === 'en-US' ? 'usd' : 'eur';+}++function catalogRouter({ db, redis }) {+ const router = express.Router();+ router.get('/api/catalog', async (req, res, next) => {+ try {+ const key = cacheKey(req.originalUrl);+ const cached = await redis.get(key);+ cacheMetrics.record('catalog', Boolean(cached));+ if (cached) {+ return res.type('json').send(cached);+ }+ const { rows } = await db.query(CATALOG_SQL, [priceList(req.query.locale)]);+ const body = JSON.stringify({ products: rows });+ await redis.set(key, body, 'EX', CACHE_TTL_SECONDS);+ res.type('json').send(body);+ } catch (err) {+ next(err);+ }+ });+ return router;+}++module.exports = { catalogRouter, priceList };diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/src/routes/reviews.js b/cli/test/evals/testdata/repos/shop-catalog/api/src/routes/reviews.jsnew file mode 100644index 00000000..9bdc6522--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/src/routes/reviews.js@@ -0,0 +1,25 @@+'use strict';+const express = require('express');+const cacheMetrics = require('../cache-metrics');++function reviewsRouter({ db, redis }) {+ const router = express.Router();+ router.get('/api/products/:sku/reviews', async (req, res, next) => {+ try {+ const sid = req.get('cookie')?.match(/sid=([a-f0-9]+)/)?.[1];+ const session = sid ? await redis.get(`sess:${sid}`) : null;+ cacheMetrics.record('session', Boolean(session));+ const page = Math.max(1, Number(req.query.page) || 1);+ const { rows } = await db.query(+ 'SELECT id, rating, title, body, created_at FROM reviews WHERE product_sku = $1 ORDER BY created_at DESC LIMIT 10 OFFSET $2',+ [req.params.sku, (page - 1) * 10],+ );+ res.json({ reviews: rows, page });+ } catch (err) {+ next(err);+ }+ });+ return router;+}++module.exports = { reviewsRouter };diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/src/server.js b/cli/test/evals/testdata/repos/shop-catalog/api/src/server.jsnew file mode 100644index 00000000..44576823--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/src/server.js@@ -0,0 +1,14 @@+'use strict';+const express = require('express');+const Redis = require('ioredis');+const { Pool } = require('pg');+const { catalogRouter } = require('./routes/catalog');+const { reviewsRouter } = require('./routes/reviews');++const db = new Pool({ connectionString: process.env.DATABASE_URL });+const redis = new Redis(process.env.REDIS_URL || 'redis://cache.internal:6379');++const app = express();+app.use(catalogRouter({ db, redis }));+app.use(reviewsRouter({ db, redis }));+app.listen(Number(process.env.PORT || 3000));diff --git a/cli/test/evals/testdata/repos/shop-catalog/api/test/catalog-cache.test.js b/cli/test/evals/testdata/repos/shop-catalog/api/test/catalog-cache.test.jsnew file mode 100644index 00000000..f603d97d--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/api/test/catalog-cache.test.js@@ -0,0 +1,9 @@+'use strict';+const test = require('node:test');+const assert = require('node:assert');+const { CACHE_TTL_SECONDS, cacheKey } = require('../src/catalog-cache');++test('catalog cache keys are namespaced', () => {+ assert.match(cacheKey('/api/catalog?locale=en-US'), /^catalog:/);+ assert.ok(CACHE_TTL_SECONDS >= 60);+});diff --git a/cli/test/evals/testdata/repos/shop-catalog/web/package.json b/cli/test/evals/testdata/repos/shop-catalog/web/package.jsonnew file mode 100644index 00000000..113f5f33--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/web/package.json@@ -0,0 +1,20 @@+{+ "name": "shop-web",+ "version": "2026.10.3",+ "private": true,+ "type": "module",+ "scripts": {+ "dev": "vite",+ "build": "vite build"+ },+ "dependencies": {+ "@tracewayapp/react": "^1.3.0",+ "react": "^19.1.1",+ "react-dom": "^19.1.1",+ "react-router": "^7.9.1"+ },+ "devDependencies": {+ "@vitejs/plugin-react": "^5.0.2",+ "vite": "^7.1.5"+ }+}diff --git a/cli/test/evals/testdata/repos/shop-catalog/web/src/App.jsx b/cli/test/evals/testdata/repos/shop-catalog/web/src/App.jsxnew file mode 100644index 00000000..d5a5028c--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/web/src/App.jsx@@ -0,0 +1,30 @@+import { BrowserRouter, Outlet, Route, Routes } from 'react-router';+import { TracewayProvider } from '@tracewayapp/react';+import { useCatalog } from './hooks/useCatalog';+import { CategoryPage } from './pages/CategoryPage';+import { ProductPage } from './pages/ProductPage';++function Layout() {+ const catalog = useCatalog();+ return (+ <main>+ <Outlet context={{ catalog }} />+ </main>+ );+}++export function App() {+ return (+ <TracewayProvider connectionString={import.meta.env.VITE_TRACEWAY_CONNECTION} recordAllSessions>+ <BrowserRouter>+ <Routes>+ <Route element={<Layout />}>+ <Route index element={<CategoryPage />} />+ <Route path="c/:slug" element={<CategoryPage />} />+ <Route path="p/:sku" element={<ProductPage />} />+ </Route>+ </Routes>+ </BrowserRouter>+ </TracewayProvider>+ );+}diff --git a/cli/test/evals/testdata/repos/shop-catalog/web/src/hooks/useCatalog.js b/cli/test/evals/testdata/repos/shop-catalog/web/src/hooks/useCatalog.jsnew file mode 100644index 00000000..974a924a--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/web/src/hooks/useCatalog.js@@ -0,0 +1,19 @@+import { useEffect, useState } from 'react';+import { useLocation } from 'react-router';++// useCatalog returns the product catalog for the shopper's locale.+export function useCatalog() {+ const location = useLocation();+ const [catalog, setCatalog] = useState(null);++ useEffect(() => {+ const controller = new AbortController();+ // Always ask for a fresh copy so shoppers never see stale prices.+ fetch(`/api/catalog?locale=${encodeURIComponent(navigator.language)}&_=${Date.now()}`, { signal: controller.signal })+ .then((res) => res.json())+ .then((body) => setCatalog(body.products));+ return () => controller.abort();+ }, [location.key]);++ return catalog;+}diff --git a/cli/test/evals/testdata/repos/shop-catalog/web/src/pages/CategoryPage.jsx b/cli/test/evals/testdata/repos/shop-catalog/web/src/pages/CategoryPage.jsxnew file mode 100644index 00000000..141a36a1--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/web/src/pages/CategoryPage.jsx@@ -0,0 +1,20 @@+import { useOutletContext, useParams } from 'react-router';++export function CategoryPage() {+ const { slug } = useParams();+ const { catalog } = useOutletContext();+ if (!catalog) return <p>Loading…</p>;+ const products = slug ? catalog.filter((p) => p.category === slug) : catalog.slice(0, 48);+ return (+ <ul className="grid">+ {products.map((p) => (+ <li key={p.id}>+ <a href={`/p/${p.sku}`}>+ <img src={p.image_url} alt="" loading="lazy" />+ {p.title} · {(p.amount_minor / 100).toFixed(2)} {p.currency}+ </a>+ </li>+ ))}+ </ul>+ );+}diff --git a/cli/test/evals/testdata/repos/shop-catalog/web/src/pages/ProductPage.jsx b/cli/test/evals/testdata/repos/shop-catalog/web/src/pages/ProductPage.jsxnew file mode 100644index 00000000..7fd7253a--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-catalog/web/src/pages/ProductPage.jsx@@ -0,0 +1,22 @@+import { useEffect, useState } from 'react';+import { useOutletContext, useParams } from 'react-router';++export function ProductPage() {+ const { sku } = useParams();+ const { catalog } = useOutletContext();+ const [reviews, setReviews] = useState([]);+ useEffect(() => {+ fetch(`/api/products/${encodeURIComponent(sku)}/reviews?page=1`)+ .then((res) => res.json())+ .then((body) => setReviews(body.reviews));+ }, [sku]);+ const product = catalog?.find((p) => p.sku === sku);+ if (!product) return <p>Loading…</p>;+ return (+ <article>+ <h1>{product.title}</h1>+ <p>{(product.amount_minor / 100).toFixed(2)} {product.currency}</p>+ <section>{reviews.map((r) => <blockquote key={r.id}>{r.title}</blockquote>)}</section>+ </article>+ );+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/.history/01/lib/features/product/product_page.dart b/cli/test/evals/testdata/repos/shop-flutter/.history/01/lib/features/product/product_page.dartnew file mode 100644index 00000000..c2570fd9--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/.history/01/lib/features/product/product_page.dart@@ -0,0 +1,42 @@+import 'package:flutter/material.dart';++import '../../core/api_client.dart';+import 'models/product.dart';+import 'widgets/product_hero_carousel.dart';++class ProductPage extends StatefulWidget {+ const ProductPage({super.key, required this.productId});++ final String productId;++ @override+ State<ProductPage> createState() => _ProductPageState();+}++class _ProductPageState extends State<ProductPage> {+ late final Future<Product> _product = ApiClient().product(widget.productId);++ @override+ Widget build(BuildContext context) {+ return Scaffold(+ appBar: AppBar(),+ body: FutureBuilder<Product>(+ future: _product,+ builder: (context, snap) {+ if (snap.hasError) return const Center(child: Text('Could not load this product.'));+ if (!snap.hasData) return const Center(child: CircularProgressIndicator());+ final p = snap.data!;+ return ListView(+ children: [+ ProductHeroCarousel(product: p),+ Padding(+ padding: const EdgeInsets.all(16),+ child: Text(p.title, style: Theme.of(context).textTheme.headlineSmall),+ ),+ ],+ );+ },+ ),+ );+ }+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/.history/01/lib/main.dart b/cli/test/evals/testdata/repos/shop-flutter/.history/01/lib/main.dartnew file mode 100644index 00000000..8087a41e--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/.history/01/lib/main.dart@@ -0,0 +1,32 @@+import 'package:flutter/material.dart';+import 'package:traceway/traceway.dart';++import 'features/product/product_page.dart';++const _tracewayConnectionString = String.fromEnvironment('TRACEWAY_CONNECTION_STRING');++void main() {+ Traceway.run(+ connectionString: _tracewayConnectionString,+ options: TracewayOptions(screenCapture: true, version: '4.2.1'),+ child: const ShopApp(),+ );+}++class ShopApp extends StatelessWidget {+ const ShopApp({super.key});++ @override+ Widget build(BuildContext context) {+ return MaterialApp(+ title: 'Shop',+ onGenerateRoute: (settings) {+ final uri = Uri.parse(settings.name ?? '/');+ if (uri.pathSegments.length == 2 && uri.pathSegments.first == 'products') {+ return MaterialPageRoute(builder: (_) => ProductPage(productId: uri.pathSegments[1]));+ }+ return MaterialPageRoute(builder: (_) => const Placeholder());+ },+ );+ }+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/.history/01/pubspec.yaml b/cli/test/evals/testdata/repos/shop-flutter/.history/01/pubspec.yamlnew file mode 100644index 00000000..c83a0ffe--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/.history/01/pubspec.yaml@@ -0,0 +1,20 @@+name: shop_app+description: The storefront app.+publish_to: none+version: 4.2.1+421++environment:+ sdk: ^3.9.0+ flutter: ">=3.35.0"++dependencies:+ flutter:+ sdk: flutter+ http: ^1.5.0+ traceway: ^1.2.0+ cached_network_image: ^3.4.1++dev_dependencies:+ flutter_test:+ sdk: flutter+ flutter_lints: ^6.0.0diff --git a/cli/test/evals/testdata/repos/shop-flutter/README.md b/cli/test/evals/testdata/repos/shop-flutter/README.mdnew file mode 100644index 00000000..6228f5bc--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/README.md@@ -0,0 +1,10 @@+# Shop (Flutter)++The storefront app for iOS and Android. Flutter 3.35, Dart 3.9. Errors go to+Traceway through the `traceway` package (`Traceway.run` in `lib/main.dart`);+release builds are not obfuscated, so traces are readable as reported.++Product data comes from the catalog API (`lib/core/api_client.dart`): every+product page loads `GET /v1/products/{id}`.++Releases are tagged in the commit message (`4.3.0: ...`).diff --git a/cli/test/evals/testdata/repos/shop-flutter/history.yaml b/cli/test/evals/testdata/repos/shop-flutter/history.yamlnew file mode 100644index 00000000..e4ebafa5--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: '4.2.1: hero carousel sized to the first photo'+ age: 900h+ author: Sofia Marin <[email protected]>+head:+ message: '4.3.0: show the seller on marketplace listings'+ age: 230h+ author: Noah Kim <[email protected]>diff --git a/cli/test/evals/testdata/repos/shop-flutter/lib/core/api_client.dart b/cli/test/evals/testdata/repos/shop-flutter/lib/core/api_client.dartnew file mode 100644index 00000000..ec2ac604--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/lib/core/api_client.dart@@ -0,0 +1,20 @@+import 'dart:convert';++import 'package:http/http.dart' as http;++import '../features/product/models/product.dart';++class ApiClient {+ ApiClient({http.Client? client, this.baseUrl = 'https://api.shop.example'}) : _client = client ?? http.Client();++ final http.Client _client;+ final String baseUrl;++ Future<Product> product(String id) async {+ final res = await _client.get(Uri.parse('$baseUrl/v1/products/$id'));+ if (res.statusCode != 200) {+ throw http.ClientException('GET /v1/products/$id: ${res.statusCode}');+ }+ return Product.fromJson(jsonDecode(res.body) as Map<String, dynamic>);+ }+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/models/product.dart b/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/models/product.dartnew file mode 100644index 00000000..23b46b2c--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/models/product.dart@@ -0,0 +1,43 @@+/// A product as GET /v1/products/{id} returns it.+class Product {+ const Product({+ required this.id,+ required this.title,+ required this.priceCents,+ required this.currency,+ this.media,+ this.seller,+ });++ final String id;+ final String title;+ final int priceCents;+ final String currency;++ /// Photos and videos, in display order. The catalog omits the field when a+ /// listing has no media yet.+ final List<ProductMedia>? media;++ /// Set for marketplace listings sold by a third-party seller.+ final String? seller;++ factory Product.fromJson(Map<String, dynamic> json) => Product(+ id: json['id'] as String,+ title: json['title'] as String,+ priceCents: json['price_cents'] as int,+ currency: json['currency'] as String,+ media: (json['media'] as List<dynamic>?)?.map((m) => ProductMedia.fromJson(m as Map<String, dynamic>)).toList(),+ seller: json['seller'] as String?,+ );+}++class ProductMedia {+ const ProductMedia({required this.url, required this.width, required this.height});++ final String url;+ final int width;+ final int height;++ factory ProductMedia.fromJson(Map<String, dynamic> json) =>+ ProductMedia(url: json['url'] as String, width: json['width'] as int, height: json['height'] as int);+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/product_page.dart b/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/product_page.dartnew file mode 100644index 00000000..48ac12a1--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/product_page.dart@@ -0,0 +1,47 @@+import 'package:flutter/material.dart';++import '../../core/api_client.dart';+import 'models/product.dart';+import 'widgets/product_hero_carousel.dart';++class ProductPage extends StatefulWidget {+ const ProductPage({super.key, required this.productId});++ final String productId;++ @override+ State<ProductPage> createState() => _ProductPageState();+}++class _ProductPageState extends State<ProductPage> {+ late final Future<Product> _product = ApiClient().product(widget.productId);++ @override+ Widget build(BuildContext context) {+ return Scaffold(+ appBar: AppBar(),+ body: FutureBuilder<Product>(+ future: _product,+ builder: (context, snap) {+ if (snap.hasError) return const Center(child: Text('Could not load this product.'));+ if (!snap.hasData) return const Center(child: CircularProgressIndicator());+ final p = snap.data!;+ return ListView(+ children: [+ ProductHeroCarousel(product: p),+ Padding(+ padding: const EdgeInsets.all(16),+ child: Text(p.title, style: Theme.of(context).textTheme.headlineSmall),+ ),+ if (p.seller != null)+ Padding(+ padding: const EdgeInsets.symmetric(horizontal: 16),+ child: Text('Sold by ${p.seller}'),+ ),+ ],+ );+ },+ ),+ );+ }+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/widgets/product_hero_carousel.dart b/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/widgets/product_hero_carousel.dartnew file mode 100644index 00000000..67cc25c1--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/lib/features/product/widgets/product_hero_carousel.dart@@ -0,0 +1,33 @@+import 'package:cached_network_image/cached_network_image.dart';+import 'package:flutter/material.dart';++import '../models/product.dart';++/// The swipeable hero images at the top of the product page.+class ProductHeroCarousel extends StatelessWidget {+ const ProductHeroCarousel({super.key, required this.product, this.height = 360});++ final Product product;+ final double height;++ static Widget _image(ProductMedia m) => CachedNetworkImage(+ imageUrl: m.url,+ fit: BoxFit.cover,+ placeholder: (_, __) => const ColoredBox(color: Color(0xFFF2F2F2)),+ );++ @override+ Widget build(BuildContext context) {+ // Size the carousel to the first image so portrait and landscape+ // photos both fill the width without letterboxing.+ final ratio = MediaQuery.sizeOf(context).width;+ final first = product.media!.first;+ final h = (ratio * first.height / first.width).clamp(240.0, height);+ return SizedBox(+ height: h,+ child: PageView(+ children: [for (final m in product.media!) _image(m)],+ ),+ );+ }+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/lib/main.dart b/cli/test/evals/testdata/repos/shop-flutter/lib/main.dartnew file mode 100644index 00000000..cfebc1b2--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/lib/main.dart@@ -0,0 +1,32 @@+import 'package:flutter/material.dart';+import 'package:traceway/traceway.dart';++import 'features/product/product_page.dart';++const _tracewayConnectionString = String.fromEnvironment('TRACEWAY_CONNECTION_STRING');++void main() {+ Traceway.run(+ connectionString: _tracewayConnectionString,+ options: TracewayOptions(screenCapture: true, version: '4.3.0'),+ child: const ShopApp(),+ );+}++class ShopApp extends StatelessWidget {+ const ShopApp({super.key});++ @override+ Widget build(BuildContext context) {+ return MaterialApp(+ title: 'Shop',+ onGenerateRoute: (settings) {+ final uri = Uri.parse(settings.name ?? '/');+ if (uri.pathSegments.length == 2 && uri.pathSegments.first == 'products') {+ return MaterialPageRoute(builder: (_) => ProductPage(productId: uri.pathSegments[1]));+ }+ return MaterialPageRoute(builder: (_) => const Placeholder());+ },+ );+ }+}diff --git a/cli/test/evals/testdata/repos/shop-flutter/pubspec.yaml b/cli/test/evals/testdata/repos/shop-flutter/pubspec.yamlnew file mode 100644index 00000000..f31e6f37--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-flutter/pubspec.yaml@@ -0,0 +1,20 @@+name: shop_app+description: The storefront app.+publish_to: none+version: 4.3.0+430++environment:+ sdk: ^3.9.0+ flutter: ">=3.35.0"++dependencies:+ flutter:+ sdk: flutter+ http: ^1.5.0+ traceway: ^1.2.0+ cached_network_image: ^3.4.1++dev_dependencies:+ flutter_test:+ sdk: flutter+ flutter_lints: ^6.0.0diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/.history/01/api/internal/feed/handler.go b/cli/test/evals/testdata/repos/shop-ios-and-api/.history/01/api/internal/feed/handler.gonew file mode 100644index 00000000..aaedc938--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/.history/01/api/internal/feed/handler.go@@ -0,0 +1,85 @@+// Package feed serves the home feed: promotions for the user's segments and+// their recently viewed products.+package feed++import (+ "database/sql"+ "encoding/json"+ "net/http"+ "strconv"+ "strings"+)++type promotion struct {+ ID int64 `json:"id"`+ Title string `json:"title"`+ ImageURL string `json:"imageUrl"`+ PriceCents int64 `json:"priceCents"`+}++// Handler serves GET /api/feed.+func Handler(db *sql.DB) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ ctx := r.Context()+ userID, _ := strconv.ParseInt(r.Header.Get("X-User-Id"), 10, 64)+ segments := strings.Split(r.Header.Get("X-Segments"), ",")+ rows, err := db.QueryContext(ctx, `SELECT p.id, p.title, p.image_url, p.price_cents FROM promotions p WHERE p.active AND p.segment = ANY($1) ORDER BY p.priority DESC LIMIT 30`, segments)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ promos := []promotion{}+ for rows.Next() {+ var p promotion+ if err := rows.Scan(&p.ID, &p.Title, &p.ImageURL, &p.PriceCents); err != nil {+ rows.Close()+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ promos = append(promos, p)+ }+ rows.Close()+ viewed := map[int64]int{}+ vrows, err := db.QueryContext(ctx, `SELECT product_id, count(*) FROM recent_views WHERE user_id = $1 GROUP BY product_id ORDER BY count(*) DESC LIMIT 20`, userID)+ if err == nil {+ for vrows.Next() {+ var id int64+ var n int+ if vrows.Scan(&id, &n) == nil {+ viewed[id] = n+ }+ }+ vrows.Close()+ }+ _ = json.NewEncoder(w).Encode(map[string]any{"promotions": promos, "recentlyViewed": viewed})+ })+}++// Product serves GET /api/products/{id}.+func Product(db *sql.DB) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad product id", http.StatusBadRequest)+ return+ }+ var p struct {+ ID int64 `json:"id"`+ Name string `json:"name"`+ Description string `json:"description"`+ PriceCents int64 `json:"priceCents"`+ Images string `json:"images"`+ }+ err = db.QueryRowContext(r.Context(), `SELECT id, name, description, price_cents, images FROM products WHERE id = $1`, id).+ Scan(&p.ID, &p.Name, &p.Description, &p.PriceCents, &p.Images)+ if err == sql.ErrNoRows {+ http.NotFound(w, r)+ return+ }+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ _ = json.NewEncoder(w).Encode(p)+ })+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/README.md b/cli/test/evals/testdata/repos/shop-ios-and-api/README.mdnew file mode 100644index 00000000..086211e5--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/README.md@@ -0,0 +1,8 @@+# shop (iOS app + mobile API)++- `ios/`: the iOS shopping app (Swift, UIKit). Crashes go to Traceway with+ the `traceway-ios` SDK (project **shop-ios**); launch watchdog terminations+ are forwarded from MetricKit by `Diagnostics/CrashDiagnostics.swift`. dSYMs+ are uploaded on every release build.+- `api/`: the mobile API the app talks to (Go, `net/http`, PostgreSQL through+ `database/sql`, OpenTelemetry; project **mobile-api**).diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/api/cmd/server/main.go b/cli/test/evals/testdata/repos/shop-ios-and-api/api/cmd/server/main.gonew file mode 100644index 00000000..3019c2a6--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/api/cmd/server/main.go@@ -0,0 +1,46 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"+ "time"++ "example.com/mobile-api/internal/config"+ "example.com/mobile-api/internal/feed"+)++func main() {+ // otelsql.Open("pgx", ...) in production builds.+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ // Sized for the steady state: about 40 requests a second at a few+ // milliseconds each, plus the feed.+ db.SetMaxOpenConns(8)+ db.SetMaxIdleConns(8)+ db.SetConnMaxLifetime(30 * time.Minute)++ mux := http.NewServeMux()+ mux.Handle("GET /api/config", config.Handler(config.NewStore(db)))+ mux.Handle("POST /api/devices", config.RegisterDevice(db))+ mux.Handle("GET /api/feed", feed.Handler(db))+ mux.Handle("GET /api/products/{id}", feed.Product(db))+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ srv := &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}+ if err := srv.ListenAndServe(); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/api/go.mod b/cli/test/evals/testdata/repos/shop-ios-and-api/api/go.modnew file mode 100644index 00000000..39bce767--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/api/go.mod@@ -0,0 +1,3 @@+module example.com/mobile-api++go 1.26diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/config/handler.go b/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/config/handler.gonew file mode 100644index 00000000..99caecd0--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/config/handler.go@@ -0,0 +1,99 @@+// Package config serves the app's launch config and device registration.+package config++import (+ "context"+ "database/sql"+ "encoding/json"+ "net/http"+)++// Payload is what the apps read at launch.+type Payload struct {+ Flags map[string]bool `json:"flags"`+ MinimumVersion string `json:"minimumVersion"`+ LatestVersion string `json:"latestVersion"`+}++// Store loads the config.+type Store interface {+ Flags(ctx context.Context, platform string) (map[string]bool, error)+ Versions(ctx context.Context, platform string) (min, latest string, err error)+}++// SQLStore implements Store on database/sql.+type SQLStore struct{ db *sql.DB }++// NewStore returns the store.+func NewStore(db *sql.DB) *SQLStore { return &SQLStore{db: db} }++// Flags returns the enabled flags of a platform.+func (s *SQLStore) Flags(ctx context.Context, platform string) (map[string]bool, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT key, value, rollout_pct FROM feature_flags WHERE platform = $1 AND enabled`, platform)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ out := map[string]bool{}+ for rows.Next() {+ var key string+ var value bool+ var rollout int+ if err := rows.Scan(&key, &value, &rollout); err != nil {+ return nil, err+ }+ out[key] = value && rollout > 0+ }+ return out, rows.Err()+}++// Versions returns the minimum supported and latest app version.+func (s *SQLStore) Versions(ctx context.Context, platform string) (string, string, error) {+ var min, latest string+ err := s.db.QueryRowContext(ctx, `SELECT min_supported, latest FROM app_versions WHERE platform = $1`, platform).Scan(&min, &latest)+ return min, latest, err+}++// Handler serves GET /api/config?platform=ios.+func Handler(s Store) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ platform := r.URL.Query().Get("platform")+ if platform == "" {+ platform = "ios"+ }+ flags, err := s.Flags(r.Context(), platform)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ min, latest, err := s.Versions(r.Context(), platform)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(Payload{Flags: flags, MinimumVersion: min, LatestVersion: latest})+ })+}++// RegisterDevice serves POST /api/devices: stores the push token.+func RegisterDevice(db *sql.DB) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ var in struct {+ UserID int64 `json:"userId"`+ Platform string `json:"platform"`+ Token string `json:"token"`+ }+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ http.Error(w, "invalid body", http.StatusBadRequest)+ return+ }+ _, err := db.ExecContext(r.Context(), `INSERT INTO push_tokens (user_id, platform, token, updated_at) VALUES ($1, $2, $3, now()) ON CONFLICT (token) DO UPDATE SET user_id = EXCLUDED.user_id, updated_at = now()`,+ in.UserID, in.Platform, in.Token)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.WriteHeader(http.StatusNoContent)+ })+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/config/handler_test.go b/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/config/handler_test.gonew file mode 100644index 00000000..d1dccfc6--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/config/handler_test.go@@ -0,0 +1,31 @@+package config++import (+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "testing"+)++type fakeStore struct{}++func (fakeStore) Flags(context.Context, string) (map[string]bool, error) {+ return map[string]bool{"new_home": true}, nil+}++func (fakeStore) Versions(context.Context, string) (string, string, error) {+ return "4.8.0", "4.12.0", nil+}++func TestHandler(t *testing.T) {+ rec := httptest.NewRecorder()+ Handler(fakeStore{}).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/config?platform=ios", nil))+ var p Payload+ if err := json.Unmarshal(rec.Body.Bytes(), &p); err != nil {+ t.Fatal(err)+ }+ if !p.Flags["new_home"] || p.MinimumVersion != "4.8.0" || p.LatestVersion != "4.12.0" {+ t.Fatalf("payload %+v", p)+ }+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/feed/handler.go b/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/feed/handler.gonew file mode 100644index 00000000..dfee7a7f--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/api/internal/feed/handler.go@@ -0,0 +1,86 @@+// Package feed serves the home feed: promotions for the user's segments and+// their recently viewed products.+package feed++import (+ "database/sql"+ "encoding/json"+ "net/http"+ "strconv"+ "strings"+)++type promotion struct {+ ID int64 `json:"id"`+ Title string `json:"title"`+ ImageURL string `json:"imageUrl"`+ PriceCents int64 `json:"priceCents"`+ Badge string `json:"badge"`+}++// Handler serves GET /api/feed.+func Handler(db *sql.DB) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ ctx := r.Context()+ userID, _ := strconv.ParseInt(r.Header.Get("X-User-Id"), 10, 64)+ segments := strings.Split(r.Header.Get("X-Segments"), ",")+ rows, err := db.QueryContext(ctx, `SELECT p.id, p.title, p.image_url, p.price_cents, p.badge FROM promotions p WHERE p.active AND p.segment = ANY($1) ORDER BY p.priority DESC LIMIT 30`, segments)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ promos := []promotion{}+ for rows.Next() {+ var p promotion+ if err := rows.Scan(&p.ID, &p.Title, &p.ImageURL, &p.PriceCents, &p.Badge); err != nil {+ rows.Close()+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ promos = append(promos, p)+ }+ rows.Close()+ viewed := map[int64]int{}+ vrows, err := db.QueryContext(ctx, `SELECT product_id, count(*) FROM recent_views WHERE user_id = $1 GROUP BY product_id ORDER BY count(*) DESC LIMIT 20`, userID)+ if err == nil {+ for vrows.Next() {+ var id int64+ var n int+ if vrows.Scan(&id, &n) == nil {+ viewed[id] = n+ }+ }+ vrows.Close()+ }+ _ = json.NewEncoder(w).Encode(map[string]any{"promotions": promos, "recentlyViewed": viewed})+ })+}++// Product serves GET /api/products/{id}.+func Product(db *sql.DB) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad product id", http.StatusBadRequest)+ return+ }+ var p struct {+ ID int64 `json:"id"`+ Name string `json:"name"`+ Description string `json:"description"`+ PriceCents int64 `json:"priceCents"`+ Images string `json:"images"`+ }+ err = db.QueryRowContext(r.Context(), `SELECT id, name, description, price_cents, images FROM products WHERE id = $1`, id).+ Scan(&p.ID, &p.Name, &p.Description, &p.PriceCents, &p.Images)+ if err == sql.ErrNoRows {+ http.NotFound(w, r)+ return+ }+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ _ = json.NewEncoder(w).Encode(p)+ })+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/history.yaml b/cli/test/evals/testdata/repos/shop-ios-and-api/history.yamlnew file mode 100644index 00000000..7d175a55--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: 'ios 4.12.0: new home screen behind the new_home flag'+ age: 504h+ author: Aiko Tanaka <[email protected]>+head:+ message: 'api 2.9.0: badge field on feed promotions'+ age: 456h+ author: Daniel Okoro <[email protected]>diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/AppDelegate.swift b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/AppDelegate.swiftnew file mode 100644index 00000000..16de6eef--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/AppDelegate.swift@@ -0,0 +1,21 @@+import UIKit+import Traceway++@main+class AppDelegate: UIResponder, UIApplicationDelegate {+ func application(+ _ application: UIApplication,+ didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?+ ) -> Bool {+ Traceway.start(+ connectionString: Secrets.tracewayConnectionString,+ options: TracewayOptions(version: AppInfo.version)+ )+ CrashDiagnostics.shared.start()++ // Flags decide which home screen and payment methods to show.+ RemoteConfig.shared.loadAtLaunch()+ PushRegistration.shared.register(application)+ return true+ }+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Config/ConfigClient.swift b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Config/ConfigClient.swiftnew file mode 100644index 00000000..ff4b0d65--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Config/ConfigClient.swift@@ -0,0 +1,36 @@+import Foundation++struct ConfigPayload: Decodable {+ let flags: [String: Bool]+ let minimumVersion: String?+}++enum ConfigError: Error { case timedOut, badResponse }++final class ConfigClient {+ private let baseURL: URL+ private let session: URLSession++ init(baseURL: URL, session: URLSession = .shared) {+ self.baseURL = baseURL+ self.session = session+ }++ /// GET /api/config, waiting for the answer on the calling thread.+ func fetchBlocking(timeout: TimeInterval) throws -> ConfigPayload {+ var request = URLRequest(url: baseURL.appendingPathComponent("api/config"))+ request.timeoutInterval = timeout+ let semaphore = DispatchSemaphore(value: 0)+ var result: Result<ConfigPayload, Error> = .failure(ConfigError.timedOut)+ session.dataTask(with: request) { data, response, error in+ defer { semaphore.signal() }+ guard let data, (response as? HTTPURLResponse)?.statusCode == 200 else {+ result = .failure(error ?? ConfigError.badResponse)+ return+ }+ result = Result { try JSONDecoder().decode(ConfigPayload.self, from: data) }+ }.resume()+ _ = semaphore.wait(timeout: .now() + timeout)+ return try result.get()+ }+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Config/RemoteConfig.swift b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Config/RemoteConfig.swiftnew file mode 100644index 00000000..14d0e0bc--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Config/RemoteConfig.swift@@ -0,0 +1,27 @@+import Foundation++/// Feature flags and the minimum supported version, from GET /api/config.+final class RemoteConfig {+ static let shared = RemoteConfig()++ private(set) var flags: [String: Bool] = [:]+ private(set) var minimumVersion: String?++ private let client = ConfigClient(baseURL: AppInfo.apiBaseURL)++ /// Loads the config before the first screen is built, so the home screen+ /// never renders with stale flags. Falls back to the bundled defaults if+ /// the request fails.+ func loadAtLaunch() {+ flags = Self.bundledDefaults+ do {+ let config = try client.fetchBlocking(timeout: 30)+ flags = config.flags+ minimumVersion = config.minimumVersion+ } catch {+ Traceway.capture(error)+ }+ }++ static let bundledDefaults: [String: Bool] = ["new_home": false, "apple_pay": true]+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/CallStackFrames.swift b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/CallStackFrames.swiftnew file mode 100644index 00000000..337849a5--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/CallStackFrames.swift@@ -0,0 +1,31 @@+import Foundation+import MetricKit++/// The crashed thread's frames from an MXCallStackTree, innermost first.+enum CallStackFrames {+ struct Frame {+ let binaryUUID: String+ let offsetIntoBinaryTextSegment: UInt64+ let binaryName: String+ }++ static func crashedThread(_ tree: MXCallStackTree) -> [Frame] {+ guard+ let root = try? JSONSerialization.jsonObject(with: tree.jsonRepresentation()) as? [String: Any],+ let stacks = root["callStacks"] as? [[String: Any]],+ let crashed = stacks.first(where: { $0["threadAttributed"] as? Bool == true }),+ var node = (crashed["callStackRootFrames"] as? [[String: Any]])?.first+ else { return [] }+ var frames: [Frame] = []+ while true {+ frames.append(Frame(+ binaryUUID: (node["binaryUUID"] as? String ?? "").replacingOccurrences(of: "-", with: ""),+ offsetIntoBinaryTextSegment: (node["offsetIntoBinaryTextSegment"] as? NSNumber)?.uint64Value ?? 0,+ binaryName: node["binaryName"] as? String ?? "?"+ ))+ guard let next = (node["subFrames"] as? [[String: Any]])?.first else { break }+ node = next+ }+ return frames+ }+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/CrashDiagnostics.swift b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/CrashDiagnostics.swiftnew file mode 100644index 00000000..b166a2dd--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/CrashDiagnostics.swift@@ -0,0 +1,40 @@+import Foundation+import MetricKit+import Traceway++/// Forwards MetricKit crash diagnostics (watchdog terminations included,+/// which no in-process handler can catch) to Traceway on the next launch, in+/// the Traceway iOS frame format so the uploaded dSYM symbolicates them.+final class CrashDiagnostics: NSObject, MXMetricManagerSubscriber {+ static let shared = CrashDiagnostics()++ func start() { MXMetricManager.shared.add(self) }++ func didReceive(_ payloads: [MXDiagnosticPayload]) {+ for payload in payloads {+ for crash in payload.crashDiagnostics ?? [] {+ TracewayReport.send(stackTrace: Self.render(crash), recordedAt: payload.timeStampEnd,+ attributes: ["source": "metrickit", "phase": "launch"])+ }+ }+ }++ /// "Watchdog termination: EXC_CRASH (SIGKILL), FRONTBOARD 0x8badf00d ..."+ /// then the *** separator, the os/arch line and one "#NN <uuid> 0x<offset>+ /// <image>" line per frame of the crashed thread.+ static func render(_ crash: MXCrashDiagnostic) -> String {+ var lines = [Self.preamble(crash), String(repeating: "*** ", count: 16).trimmingCharacters(in: .whitespaces), "os: ios arch: arm64"]+ for (i, frame) in CallStackFrames.crashedThread(crash.callStackTree).enumerated() {+ lines.append(String(format: "#%02d %@ 0x%016llx %@", i, frame.binaryUUID, frame.offsetIntoBinaryTextSegment, frame.binaryName))+ }+ return lines.joined(separator: "\n")+ }++ private static func preamble(_ crash: MXCrashDiagnostic) -> String {+ let reason = crash.terminationReason ?? "unknown"+ if reason.contains("0x8badf00d") {+ return "Watchdog termination: EXC_CRASH (SIGKILL), \(reason)"+ }+ return "Fatal Signal \(crash.signal ?? 0): \(reason)"+ }+}diff --git a/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/TracewayReport.swift b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/TracewayReport.swiftnew file mode 100644index 00000000..d5210292--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-ios-and-api/ios/Shop/Diagnostics/TracewayReport.swift@@ -0,0 +1,27 @@+import Foundation++/// Posts one exception to Traceway's /api/report with the app's connection+/// string, for reports the SDK cannot capture itself (MetricKit diagnostics).+enum TracewayReport {+ static func send(stackTrace: String, recordedAt: Date, attributes: [String: String]) {+ let (token, url) = Secrets.tracewayConnection+ let body: [String: Any] = [+ "appVersion": AppInfo.version,+ "serverName": AppInfo.deviceDescription,+ "collectionFrames": [[+ "stackTraces": [[+ "stackTrace": stackTrace,+ "recordedAt": ISO8601DateFormatter().string(from: recordedAt),+ "attributes": attributes,+ "isMessage": false,+ ]],+ ]],+ ]+ var request = URLRequest(url: url)+ request.httpMethod = "POST"+ request.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization")+ request.setValue("application/json", forHTTPHeaderField: "Content-Type")+ request.httpBody = try? JSONSerialization.data(withJSONObject: body)+ URLSession.shared.dataTask(with: request).resume()+ }+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/README.md b/cli/test/evals/testdata/repos/statements-api-go/README.mdnew file mode 100644index 00000000..01f545b6--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/README.md@@ -0,0 +1,20 @@+# statements-api++Account statements for the billing dashboard: a customer picks a period and+gets a PDF of every invoice issued in it. Go, `net/http`, PostgreSQL through+`pgx`, PDFs stored in S3. Traces, logs and metrics go to Traceway over OTLP+(`OTEL_EXPORTER_OTLP_ENDPOINT`).++```+go test ./...+go run ./cmd/server+```++Routes:++- `POST /api/statements` – create a statement for `{accountId, periodStart, periodEnd}`+- `GET /api/statements/{id}` – a statement's status and object key+- `GET /api/statements/{id}/download` – redirect to a presigned S3 URL+- `GET /api/invoices` – the account's latest invoices++The dashboard client is in `web/statements.js`.diff --git a/cli/test/evals/testdata/repos/statements-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/statements-api-go/cmd/server/main.gonew file mode 100644index 00000000..6510bf5e--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/cmd/server/main.go@@ -0,0 +1,41 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"+ "time"++ "example.com/statements-api/internal/objects"+ "example.com/statements-api/internal/pdf"+ "example.com/statements-api/internal/statements"+)++func main() {+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ font, err := os.ReadFile(os.Getenv("STATEMENT_FONT"))+ if err != nil {+ slog.Error("read font", "err", err)+ os.Exit(1)+ }+ bucket := &objects.Bucket{+ Endpoint: os.Getenv("STATEMENTS_BUCKET_URL"),+ Signer: objects.SidecarSigner(os.Getenv("SIGNER_URL")),+ Client: &http.Client{Timeout: 30 * time.Second},+ }+ h := statements.New(statements.NewSQLStore(db), pdf.NewRenderer(font), bucket)++ mux := http.NewServeMux()+ h.Routes(mux)+ srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 10 * time.Second}+ slog.Info("listening", "addr", srv.Addr)+ if err := srv.ListenAndServe(); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/go.mod b/cli/test/evals/testdata/repos/statements-api-go/go.modnew file mode 100644index 00000000..fd0053e4--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/statements-api++go 1.26diff --git a/cli/test/evals/testdata/repos/statements-api-go/internal/objects/bucket.go b/cli/test/evals/testdata/repos/statements-api-go/internal/objects/bucket.gonew file mode 100644index 00000000..c9ab25ab--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/internal/objects/bucket.go@@ -0,0 +1,62 @@+// Package objects stores statement PDFs in S3.+package objects++import (+ "bytes"+ "context"+ "fmt"+ "net/http"+ "time"+)++// Bucket is an S3 bucket reached through presigned requests from the+// signing sidecar (see deploy/).+type Bucket struct {+ Endpoint string // https://<bucket>.s3.<region>.amazonaws.com+ Signer func(method, key string, ttl time.Duration) (string, error)+ Client *http.Client+}++// Put uploads body under key.+func (b *Bucket) Put(ctx context.Context, key string, body []byte) error {+ url, err := b.Signer(http.MethodPut, key, 5*time.Minute)+ if err != nil {+ return err+ }+ req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body))+ if err != nil {+ return err+ }+ req.Header.Set("Content-Type", "application/pdf")+ res, err := b.Client.Do(req)+ if err != nil {+ return err+ }+ defer res.Body.Close()+ if res.StatusCode != http.StatusOK {+ return fmt.Errorf("put %s: %s", key, res.Status)+ }+ return nil+}++// PresignGet returns a download URL for key.+func (b *Bucket) PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error) {+ return b.Signer(http.MethodGet, key, ttl)+}++// SidecarSigner asks the signing sidecar for a presigned URL.+func SidecarSigner(base string) func(method, key string, ttl time.Duration) (string, error) {+ client := &http.Client{Timeout: 2 * time.Second}+ return func(method, key string, ttl time.Duration) (string, error) {+ res, err := client.Get(fmt.Sprintf("%s/sign?method=%s&key=%s&ttl=%d", base, method, key, int(ttl.Seconds())))+ if err != nil {+ return "", err+ }+ defer res.Body.Close()+ var buf bytes.Buffer+ if _, err := buf.ReadFrom(res.Body); err != nil {+ return "", err+ }+ return buf.String(), nil+ }+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/internal/pdf/render.go b/cli/test/evals/testdata/repos/statements-api-go/internal/pdf/render.gonew file mode 100644index 00000000..fa8a8859--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/internal/pdf/render.go@@ -0,0 +1,56 @@+// Package pdf renders account statements.+package pdf++import (+ "bytes"+ "compress/flate"+ "context"+ "fmt"++ "example.com/statements-api/internal/statements"+)++// Renderer lays out a statement and compresses its page streams. The font is+// embedded so the PDF prints identically everywhere.+type Renderer struct {+ Font []byte+}++// NewRenderer returns a renderer with the embedded statement font.+func NewRenderer(font []byte) *Renderer { return &Renderer{Font: font} }++// Render returns the PDF bytes for the account's invoices.+func (r *Renderer) Render(ctx context.Context, acct *statements.Account, invoices []statements.Invoice) ([]byte, error) {+ var doc bytes.Buffer+ doc.WriteString("%PDF-1.7\n")+ doc.Write(r.Font)+ pages := 1 + len(invoices)/25+ for p := 0; p < pages; p++ {+ if err := ctx.Err(); err != nil {+ return nil, err+ }+ var page bytes.Buffer+ fmt.Fprintf(&page, "BT /F1 11 Tf 56 790 Td (%s - statement page %d) Tj ET\n", acct.LegalName, p+1)+ for i := p * 25; i < len(invoices) && i < (p+1)*25; i++ {+ inv := invoices[i]+ fmt.Fprintf(&page, "BT 56 %d Td (%s %s %d.%02d %s) Tj ET\n", 760-(i%25)*28, inv.Number,+ inv.IssuedAt.Format("2006-01-02"), inv.TotalCents/100, inv.TotalCents%100, acct.Currency)+ }+ var z bytes.Buffer+ fw, err := flate.NewWriter(&z, flate.BestCompression)+ if err != nil {+ return nil, err+ }+ if _, err := fw.Write(page.Bytes()); err != nil {+ return nil, err+ }+ if err := fw.Close(); err != nil {+ return nil, err+ }+ fmt.Fprintf(&doc, "%d 0 obj << /Length %d /Filter /FlateDecode >> stream\n", p+4, z.Len())+ doc.Write(z.Bytes())+ doc.WriteString("\nendstream endobj\n")+ }+ doc.WriteString("%%EOF\n")+ return doc.Bytes(), nil+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/internal/statements/handler.go b/cli/test/evals/testdata/repos/statements-api-go/internal/statements/handler.gonew file mode 100644index 00000000..1638bf2b--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/internal/statements/handler.go@@ -0,0 +1,185 @@+// Package statements serves account statements: a PDF of every invoice an+// account was issued in a period.+package statements++import (+ "context"+ "crypto/rand"+ "encoding/hex"+ "encoding/json"+ "fmt"+ "net/http"+ "time"+)++// Account is the billed customer.+type Account struct {+ ID int64+ LegalName string+ Currency string+ BillingEmail string+}++// Invoice is one issued invoice.+type Invoice struct {+ ID int64+ Number string+ IssuedAt time.Time+ DueAt time.Time+ TotalCents int64+ Status string+}++// Statement is a rendered (or rendering) statement.+type Statement struct {+ ID string `json:"id"`+ AccountID int64 `json:"accountId"`+ PeriodStart time.Time `json:"periodStart"`+ PeriodEnd time.Time `json:"periodEnd"`+ Status string `json:"status"`+ ObjectKey string `json:"objectKey,omitempty"`+ RenderedAt *time.Time `json:"renderedAt,omitempty"`+}++// Store is the statements database.+type Store interface {+ Account(ctx context.Context, id int64) (*Account, error)+ InvoicesForPeriod(ctx context.Context, accountID int64, from, to time.Time) ([]Invoice, error)+ CreateStatement(ctx context.Context, st *Statement) error+ MarkReady(ctx context.Context, id, objectKey string) error+ MarkFailed(ctx context.Context, id string) error+ Statement(ctx context.Context, id string) (*Statement, error)+}++// Renderer turns an account's invoices into a PDF.+type Renderer interface {+ Render(ctx context.Context, acct *Account, invoices []Invoice) ([]byte, error)+}++// Uploader stores rendered PDFs.+type Uploader interface {+ Put(ctx context.Context, key string, body []byte) error+ PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error)+}++// Handler holds the statement routes.+type Handler struct {+ store Store+ renderer Renderer+ uploader Uploader+}++// New returns the handler.+func New(store Store, renderer Renderer, uploader Uploader) *Handler {+ return &Handler{store: store, renderer: renderer, uploader: uploader}+}++// Routes registers the statement routes on mux.+func (h *Handler) Routes(mux *http.ServeMux) {+ mux.HandleFunc("POST /api/statements", h.Create)+ mux.HandleFunc("GET /api/statements/{id}", h.Get)+ mux.HandleFunc("GET /api/statements/{id}/download", h.Download)+}++type createRequest struct {+ AccountID int64 `json:"accountId"`+ PeriodStart time.Time `json:"periodStart"`+ PeriodEnd time.Time `json:"periodEnd"`+}++// Create serves POST /api/statements.+func (h *Handler) Create(w http.ResponseWriter, r *http.Request) {+ var req createRequest+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {+ http.Error(w, "invalid body", http.StatusBadRequest)+ return+ }+ if req.AccountID == 0 || !req.PeriodEnd.After(req.PeriodStart) {+ http.Error(w, "accountId and a valid period are required", http.StatusUnprocessableEntity)+ return+ }+ ctx := r.Context()+ acct, err := h.store.Account(ctx, req.AccountID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if acct == nil {+ http.NotFound(w, r)+ return+ }+ invoices, err := h.store.InvoicesForPeriod(ctx, acct.ID, req.PeriodStart, req.PeriodEnd)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ st := &Statement{ID: newID(), AccountID: acct.ID, PeriodStart: req.PeriodStart, PeriodEnd: req.PeriodEnd, Status: "rendering"}+ if err := h.store.CreateStatement(ctx, st); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }++ pdf, err := h.renderer.Render(ctx, acct, invoices)+ if err != nil {+ _ = h.store.MarkFailed(ctx, st.ID)+ http.Error(w, "render failed", http.StatusInternalServerError)+ return+ }+ key := fmt.Sprintf("statements/%d/%s.pdf", st.PeriodStart.Year(), st.ID)+ if err := h.uploader.Put(ctx, key, pdf); err != nil {+ _ = h.store.MarkFailed(ctx, st.ID)+ http.Error(w, "upload failed", http.StatusInternalServerError)+ return+ }+ if err := h.store.MarkReady(ctx, st.ID, key); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ st.Status, st.ObjectKey = "ready", key+ writeJSON(w, http.StatusCreated, st)+}++// Get serves GET /api/statements/{id}; the dashboard polls it.+func (h *Handler) Get(w http.ResponseWriter, r *http.Request) {+ st, err := h.store.Statement(r.Context(), r.PathValue("id"))+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if st == nil {+ http.NotFound(w, r)+ return+ }+ writeJSON(w, http.StatusOK, st)+}++// Download serves GET /api/statements/{id}/download.+func (h *Handler) Download(w http.ResponseWriter, r *http.Request) {+ st, err := h.store.Statement(r.Context(), r.PathValue("id"))+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if st == nil || st.Status != "ready" {+ http.NotFound(w, r)+ return+ }+ url, err := h.uploader.PresignGet(r.Context(), st.ObjectKey, 15*time.Minute)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ http.Redirect(w, r, url, http.StatusFound)+}++func newID() string {+ var b [16]byte+ _, _ = rand.Read(b[:])+ return hex.EncodeToString(b[:])+}++func writeJSON(w http.ResponseWriter, status int, v any) {+ w.Header().Set("Content-Type", "application/json")+ w.WriteHeader(status)+ _ = json.NewEncoder(w).Encode(v)+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/internal/statements/handler_test.go b/cli/test/evals/testdata/repos/statements-api-go/internal/statements/handler_test.gonew file mode 100644index 00000000..20a0b0e8--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/internal/statements/handler_test.go@@ -0,0 +1,124 @@+package statements++import (+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync"+ "testing"+ "time"+)++type memStore struct {+ mu sync.Mutex+ statements map[string]*Statement+}++func newMemStore() *memStore { return &memStore{statements: map[string]*Statement{}} }++func (m *memStore) Account(_ context.Context, id int64) (*Account, error) {+ if id != 7 {+ return nil, nil+ }+ return &Account{ID: 7, LegalName: "Acme GmbH", Currency: "EUR", BillingEmail: "[email protected]"}, nil+}++func (m *memStore) InvoicesForPeriod(context.Context, int64, time.Time, time.Time) ([]Invoice, error) {+ return []Invoice{{ID: 1, Number: "INV-1001", TotalCents: 12900, Status: "paid"}}, nil+}++func (m *memStore) CreateStatement(_ context.Context, st *Statement) error {+ m.mu.Lock()+ defer m.mu.Unlock()+ cp := *st+ m.statements[st.ID] = &cp+ return nil+}++func (m *memStore) MarkReady(_ context.Context, id, key string) error {+ m.mu.Lock()+ defer m.mu.Unlock()+ m.statements[id].Status, m.statements[id].ObjectKey = "ready", key+ return nil+}++func (m *memStore) MarkFailed(_ context.Context, id string) error {+ m.mu.Lock()+ defer m.mu.Unlock()+ m.statements[id].Status = "failed"+ return nil+}++func (m *memStore) Statement(_ context.Context, id string) (*Statement, error) {+ m.mu.Lock()+ defer m.mu.Unlock()+ st, ok := m.statements[id]+ if !ok {+ return nil, nil+ }+ cp := *st+ return &cp, nil+}++type fakeRenderer struct{}++func (fakeRenderer) Render(context.Context, *Account, []Invoice) ([]byte, error) {+ return []byte("%PDF-1.7"), nil+}++type fakeBucket struct {+ mu sync.Mutex+ puts []string+}++func (b *fakeBucket) Put(_ context.Context, key string, _ []byte) error {+ b.mu.Lock()+ defer b.mu.Unlock()+ b.puts = append(b.puts, key)+ return nil+}++func (b *fakeBucket) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) {+ return "https://bucket.example/" + key, nil+}++func TestCreateRejectsInvalidPeriod(t *testing.T) {+ mux := http.NewServeMux()+ New(newMemStore(), fakeRenderer{}, &fakeBucket{}).Routes(mux)+ req := httptest.NewRequest(http.MethodPost, "/api/statements", strings.NewReader(`{"accountId":7,"periodStart":"2026-10-01T00:00:00Z","periodEnd":"2026-09-01T00:00:00Z"}`))+ rec := httptest.NewRecorder()+ mux.ServeHTTP(rec, req)+ if rec.Code != http.StatusUnprocessableEntity {+ t.Fatalf("status %d, want 422", rec.Code)+ }+}++func TestStatementBecomesDownloadable(t *testing.T) {+ store, bucket := newMemStore(), &fakeBucket{}+ mux := http.NewServeMux()+ New(store, fakeRenderer{}, bucket).Routes(mux)+ req := httptest.NewRequest(http.MethodPost, "/api/statements", strings.NewReader(`{"accountId":7,"periodStart":"2026-09-01T00:00:00Z","periodEnd":"2026-10-01T00:00:00Z"}`))+ rec := httptest.NewRecorder()+ mux.ServeHTTP(rec, req)+ if rec.Code/100 != 2 {+ t.Fatalf("status %d: %s", rec.Code, rec.Body.String())+ }+ var st Statement+ if err := json.NewDecoder(rec.Body).Decode(&st); err != nil || st.ID == "" {+ t.Fatalf("no statement id in %q (%v)", rec.Body.String(), err)+ }+ deadline := time.Now().Add(2 * time.Second)+ for {+ rec = httptest.NewRecorder()+ mux.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/statements/"+st.ID+"/download", nil))+ if rec.Code == http.StatusFound {+ break+ }+ if time.Now().After(deadline) {+ t.Fatalf("statement never became downloadable (last status %d)", rec.Code)+ }+ time.Sleep(10 * time.Millisecond)+ }+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/internal/statements/store_sql.go b/cli/test/evals/testdata/repos/statements-api-go/internal/statements/store_sql.gonew file mode 100644index 00000000..2eba52b6--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/internal/statements/store_sql.go@@ -0,0 +1,76 @@+package statements++import (+ "context"+ "database/sql"+ "errors"+ "time"+)++// SQLStore is the PostgreSQL Store.+type SQLStore struct{ db *sql.DB }++// NewSQLStore returns a Store backed by db.+func NewSQLStore(db *sql.DB) *SQLStore { return &SQLStore{db: db} }++func (s *SQLStore) Account(ctx context.Context, id int64) (*Account, error) {+ var a Account+ err := s.db.QueryRowContext(ctx, `SELECT id, legal_name, currency, billing_email FROM accounts WHERE id = $1`, id).+ Scan(&a.ID, &a.LegalName, &a.Currency, &a.BillingEmail)+ if errors.Is(err, sql.ErrNoRows) {+ return nil, nil+ }+ return &a, err+}++func (s *SQLStore) InvoicesForPeriod(ctx context.Context, accountID int64, from, to time.Time) ([]Invoice, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT id, number, issued_at, due_at, total_cents, status FROM invoices WHERE account_id = $1 AND issued_at >= $2 AND issued_at < $3 ORDER BY issued_at`, accountID, from, to)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Invoice+ for rows.Next() {+ var inv Invoice+ if err := rows.Scan(&inv.ID, &inv.Number, &inv.IssuedAt, &inv.DueAt, &inv.TotalCents, &inv.Status); err != nil {+ return nil, err+ }+ out = append(out, inv)+ }+ return out, rows.Err()+}++func (s *SQLStore) CreateStatement(ctx context.Context, st *Statement) error {+ _, err := s.db.ExecContext(ctx, `INSERT INTO statements (id, account_id, period_start, period_end, status, created_at) VALUES ($1, $2, $3, $4, 'rendering', now())`,+ st.ID, st.AccountID, st.PeriodStart, st.PeriodEnd)+ return err+}++func (s *SQLStore) MarkReady(ctx context.Context, id, objectKey string) error {+ _, err := s.db.ExecContext(ctx, `UPDATE statements SET status = 'ready', object_key = $2, rendered_at = now() WHERE id = $1`, id, objectKey)+ return err+}++func (s *SQLStore) MarkFailed(ctx context.Context, id string) error {+ _, err := s.db.ExecContext(ctx, `UPDATE statements SET status = 'failed' WHERE id = $1`, id)+ return err+}++func (s *SQLStore) Statement(ctx context.Context, id string) (*Statement, error) {+ var st Statement+ var key sql.NullString+ var rendered sql.NullTime+ err := s.db.QueryRowContext(ctx, `SELECT id, account_id, status, object_key, rendered_at FROM statements WHERE id = $1`, id).+ Scan(&st.ID, &st.AccountID, &st.Status, &key, &rendered)+ if errors.Is(err, sql.ErrNoRows) {+ return nil, nil+ }+ if err != nil {+ return nil, err+ }+ st.ObjectKey = key.String+ if rendered.Valid {+ st.RenderedAt = &rendered.Time+ }+ return &st, nil+}diff --git a/cli/test/evals/testdata/repos/statements-api-go/web/statements.js b/cli/test/evals/testdata/repos/statements-api-go/web/statements.jsnew file mode 100644index 00000000..47f054fb--- /dev/null+++ b/cli/test/evals/testdata/repos/statements-api-go/web/statements.js@@ -0,0 +1,29 @@+// Statement generation in the billing dashboard.++const POLL_INTERVAL_MS = 2000;++export async function generateStatement(accountId, periodStart, periodEnd, onProgress) {+ const res = await fetch('/api/statements', {+ method: 'POST',+ headers: { 'Content-Type': 'application/json' },+ body: JSON.stringify({ accountId, periodStart, periodEnd }),+ });+ if (!res.ok) throw new Error(`statement request failed: ${res.status}`);+ const { id } = await res.json();+ onProgress?.('rendering');+ return waitUntilReady(id, onProgress);+}++async function waitUntilReady(id, onProgress) {+ for (;;) {+ const res = await fetch(`/api/statements/${id}`);+ if (!res.ok) throw new Error(`statement status failed: ${res.status}`);+ const st = await res.json();+ if (st.status === 'ready') {+ onProgress?.('ready');+ return `/api/statements/${id}/download`;+ }+ if (st.status === 'failed') throw new Error('statement rendering failed');+ await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));+ }+}diff --git a/skills/traceway/evals/README.md b/skills/traceway/evals/README.mddeleted file mode 100644index 18676638..00000000--- a/skills/traceway/evals/README.md+++ /dev/null@@ -1,48 +0,0 @@-# Performance Skill Evals--Scenario-based evals for the `traceway` skill's **Performance** flow-(`/traceway perf <endpoint or symptom>`). Each eval hands an agent a-synthetic-but-realistic Traceway telemetry snapshot and asks it to diagnose-the bottleneck using the skill; the expected diagnosis is pinned to one row-of the bottleneck checklist in `../performance.md`.--## How to run an eval--1. Give the agent the `traceway` skill (or the `traceway` CLI against a- staging instance seeded with the snapshot data).-2. Paste the **Task** prompt from the scenario file verbatim.-3. Grade the answer against **Expected diagnosis** and **Pass criteria**.- A pass requires naming the right checklist suspect *and* citing the- telemetry that confirms it — a correct guess with no evidence is a fail.--The snapshots are static text, so evals also work offline: the agent should-reason from the snapshot exactly as it would from live `traceway` output.--## Coverage--| ID | Area | Framework | Checklist suspect |-|---|---|---|---|-| `backend-go-select-n-plus-1` | backend | Go (Gin) | Select N+1 |-| `backend-python-pool-exhaustion` | backend | Python (FastAPI) | Connection-pool exhaustion |-| `backend-node-sequential-calls` | backend | Node.js (NestJS) | Sequential calls that could run in parallel |-| `backend-missing-index` | backend | any (PostgreSQL) | Missing or unused index |-| `backend-sync-background-task` | backend | Go (Gin) | Synchronous work that belongs in a background task |-| `frontend-js-large-payloads` | frontend | browser JS | Large request/response payloads |-| `frontend-js-no-cache` | frontend | browser JS | No cache / low hit ratio |-| `mobile-ios-cold-start` | mobile | iOS | Cold start (synchronous init on launch) |-| `mobile-android-gc-pauses` | mobile | Android | Memory pressure / GC pauses |-| `mobile-flutter-shader-jank` | mobile | Flutter | Cold start / JIT / shader warmup |-| `infra-cpu-saturation` | infrastructure | n/a | CPU saturation |-| `infra-cache-stampede` | infrastructure | n/a | Cache stampede / thundering herd |--## Adding a new eval--1. Copy the file format below: header table, `## Telemetry snapshot`,- `## Task`, `## Expected diagnosis`, `## Pass criteria`.-2. Build the snapshot from a real incident when possible; keep numbers- internally consistent (percentiles, span durations, and metric values- must agree with each other).-3. Pin the expected diagnosis to exactly one `performance.md` checklist- row, and make the snapshot contain the confirming signal that row- asks for — plus one plausible distractor the agent must rule out.-4. Add a row to the coverage table above.diff --git a/skills/traceway/evals/backend-go-select-n-plus-1.md b/skills/traceway/evals/backend-go-select-n-plus-1.mddeleted file mode 100644index afbb1892..00000000--- a/skills/traceway/evals/backend-go-select-n-plus-1.md+++ /dev/null@@ -1,55 +0,0 @@-# Eval: backend-go-select-n-plus-1--- **ID**: `backend-go-select-n-plus-1`-- **Area**: backend-- **Framework**: Go (Gin + PostgreSQL)-- **Checklist item**: Select N+1 (`performance.md` → Database and data access)--## Telemetry snapshot--`traceway endpoints list --search "GET /api/orders" --since 24h --output json`:--```json-{ "endpoint": "GET /api/orders", "count": 48210, "p50": 412, "p95": 2380, "p99": 3110, "errorCount": 0 }-```--`traceway endpoints show <endpointId> --recorded-at <t>` — span waterfall of one-p95 request (total 2.31s):--```-GET /api/orders ............ 2310ms-└── db.query ............... 3ms SELECT * FROM orders WHERE user_id = $1 LIMIT 50-└── db.query ............... 2ms SELECT * FROM order_items WHERE order_id = $1-└── db.query ............... 2ms SELECT * FROM order_items WHERE order_id = $1-... (347 more near-identical order_items spans, 1–3ms each)-└── json.encode ............ 41ms-```--`traceway metrics query --name system.cpu.utilization --aggregation max --since 24h`:-flat at 22–31% all day. p50/p95 have crept up ~15% per week for a month with-no deploy in that window (`git log` clean).--## Task--> `GET /api/orders` p95 is 2.4s and climbing ~15% a week with no recent-> deploy. Diagnose the bottleneck to a specific checklist cause and propose-> the fix.--## Expected diagnosis--- **Root cause**: Select N+1 — the handler loads 50 orders, then issues one- `order_items` query per order (≈348 near-identical short spans); total DB- time dominates while each call is tiny.-- **Key evidence**: repeated near-identical query spans in the waterfall;- flat CPU (rules out saturation); latency creeping with data growth and no- deploy (rules out a regression).-- **Fix**: single JOIN / batched `WHERE order_id IN (...)` load, or a- dataloader.--## Pass criteria--- Names **Select N+1** (not "slow query" or "missing index" — the individual- queries are 1–3ms, so neither fits).-- Cites the repeated spans *and* rules out saturation/regression from the- flat CPU and clean deploy history.-- Proposes batching the item load.diff --git a/skills/traceway/evals/backend-missing-index.md b/skills/traceway/evals/backend-missing-index.mddeleted file mode 100644index 731f217f..00000000--- a/skills/traceway/evals/backend-missing-index.md+++ /dev/null@@ -1,53 +0,0 @@-# Eval: backend-missing-index--- **ID**: `backend-missing-index`-- **Area**: backend-- **Framework**: any (PostgreSQL)-- **Checklist item**: Missing or unused index (`performance.md` → Database and data access)--## Telemetry snapshot--`traceway endpoints list --search "GET /api/users/lookup" --since 24h --output json`:--```json-{ "endpoint": "GET /api/users/lookup", "count": 88100, "p50": 940, "p95": 1210, "p99": 1480, "errorCount": 0 }-```--`traceway endpoints show <endpointId> --recorded-at <t>` — one request-(total 1.02s):--```-GET /api/users/lookup ..... 1020ms-└── db.query .............. 980ms SELECT id, name, email FROM users WHERE email = $1-└── json.encode ........... 12ms-```--The same query span dominates in *every* sampled trace, at all hours, at all-traffic levels. `endpoints chart --metric-type p50 --since 30d` shows p50-rising smoothly from 210ms to 940ms over two months; `git log` shows no-deploy touching this path; the `users` table grew from 2M to 9M rows in the-same window. `EXPLAIN` (from a DBA paste in the linked issue) shows-`Seq Scan on users`.--## Task--> `GET /api/users/lookup` p50 crept from 210ms to 940ms over two months with-> no deploy on that path. Diagnose the bottleneck to a specific checklist-> cause.--## Expected diagnosis--- **Root cause**: Missing index on `users(email)` — one query span is the- long pole in every trace, its duration grows with table size, and the- latency curve tracks data growth, not deploys.-- **Key evidence**: single dominating query span across all traces; smooth- multi-week climb with no deploy; table growth 2M→9M rows; `Seq Scan`.-- **Fix**: `CREATE INDEX CONCURRENTLY ON users (email)`; verify p50 drops- back.--## Pass criteria--- Names **missing index** (not N+1 — there is exactly one query; not- saturation — CPU flat, all-hours slowness).-- Connects the multi-week climb + table growth + no deploy to data-driven- latency, and cites the `Seq Scan`.diff --git a/skills/traceway/evals/backend-node-sequential-calls.md b/skills/traceway/evals/backend-node-sequential-calls.mddeleted file mode 100644index 9da87559..00000000--- a/skills/traceway/evals/backend-node-sequential-calls.md+++ /dev/null@@ -1,54 +0,0 @@-# Eval: backend-node-sequential-calls--- **ID**: `backend-node-sequential-calls`-- **Area**: backend-- **Framework**: Node.js (NestJS, microservice fan-out)-- **Checklist item**: Sequential calls that could run in parallel- (`performance.md` → Network, IO, and external dependencies)--## Telemetry snapshot--`traceway endpoints list --search "GET /api/dashboard" --since 24h --output json`:--```json-{ "endpoint": "GET /api/dashboard", "count": 31200, "p50": 3890, "p95": 4520, "p99": 5100, "errorCount": 0 }-```--`traceway traces show <traceId> --recorded-at <t>` — cross-service timeline-(total 4.41s):--```-GET /api/dashboard (edge) .............. 4410ms-└── svc-users GET /profile ............. 890ms ─┐-└── svc-billing GET /balance ........... 920ms ├ staircase: each starts-└── svc-notify GET /unread ............. 870ms │ only when the previous-└── svc-reco GET /suggestions .......... 940ms ─┘ ends-└── json.encode ........................ 120ms-```--Each downstream service is healthy on its own (their p95s are 120–200ms in-their own dashboards); the 900ms here is mostly network round-trip per hop.-`system.cpu.utilization` flat at ~30%.--## Task--> `GET /api/dashboard` p50 is 3.9s. Every downstream service looks healthy-> on its own. Diagnose the bottleneck to a specific checklist cause.--## Expected diagnosis--- **Root cause**: Sequential downstream calls that could run in parallel —- the staircase waterfall shows four independent service hops serialized;- each hop's own latency is fine, the sum is the problem.-- **Key evidence**: staircase shape in `traces show`; downstream services- healthy in isolation; flat CPU rules out saturation.-- **Fix**: `Promise.all` the four independent calls (they share no data- dependencies); expected p50 ≈ 1.1s.--## Pass criteria--- Names **sequential calls that could run in parallel** (not "slow- upstream" — upstreams are healthy; not "chatty fan-out" — four calls is- not chatty, the serialization is the issue).-- Reads the staircase waterfall correctly and notes the calls are- independent.diff --git a/skills/traceway/evals/backend-python-pool-exhaustion.md b/skills/traceway/evals/backend-python-pool-exhaustion.mddeleted file mode 100644index 47c5aa92..00000000--- a/skills/traceway/evals/backend-python-pool-exhaustion.md+++ /dev/null@@ -1,55 +0,0 @@-# Eval: backend-python-pool-exhaustion--- **ID**: `backend-python-pool-exhaustion`-- **Area**: backend-- **Framework**: Python (FastAPI + PostgreSQL, SQLAlchemy pool size 20)-- **Checklist item**: Connection-pool exhaustion (`performance.md` → Database and data access)--## Telemetry snapshot--`traceway endpoints list --search "POST /api/checkout" --since 24h --output json`:--```json-{ "endpoint": "POST /api/checkout", "count": 96400, "p50": 118, "p95": 940, "p99": 8210, "errorCount": 12 }-```--p50 flat all day; p99 climbs steeply during the 12:00–14:00 and 19:00–21:00-traffic peaks and collapses between them.--`traceway endpoints show <endpointId> --recorded-at <t>` — one p99 request-(total 8.04s):--```-POST /api/checkout ....... 8040ms-├── <gap: no spans> ...... 3120ms-└── db.query ............. 44ms SELECT ... FROM inventory WHERE sku = $1-└── db.query ............. 61ms UPDATE orders SET ...-└── http.post ............ 280ms POST https://payments.example.com/charge-```--`traceway metrics query --name system.cpu.utilization --aggregation max --since 24h`:-35–48%, no spike aligned with the p99 humps. `traceway logs query --search "pool"`:-`sqlalchemy.pool.impl.QueuePool: "QueuePool limit of size 20 overflow 10 reached, connection timed out"` WARN lines during both peaks.--## Task--> `POST /api/checkout` p50 is fine at 118ms but p99 hits 8.2s during traffic-> peaks. Diagnose the bottleneck to a specific checklist cause.--## Expected diagnosis--- **Root cause**: Connection-pool exhaustion — the 3.1s gap *before* the- first DB span is time waiting to acquire a connection, not work; p99 rises- with concurrency while p50 stays flat; pool-timeout warnings in the logs- confirm it.-- **Key evidence**: the pre-DB gap in the waterfall; p99-vs-p50 divergence- tracking traffic; `QueuePool limit reached` log lines.-- **Fix**: raise pool size / add overflow, shorten checkout transaction scope,- or shed load (queue) at the peak.--## Pass criteria--- Names **connection-pool exhaustion** (not "slow DB" — the queries are- 44–61ms; not "slow upstream" — the payment call is 280ms).-- Points at the gap *before* the first DB span as the finding, and cites the- p99/p50 divergence plus the pool log lines.diff --git a/skills/traceway/evals/backend-sync-background-task.md b/skills/traceway/evals/backend-sync-background-task.mddeleted file mode 100644index 18c38873..00000000--- a/skills/traceway/evals/backend-sync-background-task.md+++ /dev/null@@ -1,56 +0,0 @@-# Eval: backend-sync-background-task--- **ID**: `backend-sync-background-task`-- **Area**: backend-- **Framework**: Go (Gin)-- **Checklist item**: Synchronous work that belongs in a background task- (`performance.md` → Application and compute)--## Telemetry snapshot--`traceway endpoints list --search "POST /api/reports" --since 24h --output json`:--```json-{ "endpoint": "POST /api/reports", "count": 1400, "p50": 9400, "p95": 12100, "p99": 14800, "errorCount": 0 }-```--`traceway endpoints show <endpointId> --recorded-at <t>` — one request-(total 11.9s):--```-POST /api/reports ........ 11900ms-└── db.query ............. 320ms INSERT INTO reports ...-└── pdf.render ............ 9100ms (in-process)-└── s3.put ................ 410ms PUT reports/2026-10-16/...-└── json.encode ........... 18ms {"report_id": "...", "status": "ready"}-```--The response returns only after the PDF is fully rendered and uploaded; the-user does not need the bytes synchronously (the UI polls `GET-/api/reports/:id`). `system.cpu.utilization` spikes to 91% on the single-instance during each render. No queue depth anywhere — there is no task-system on this path.--## Task--> `POST /api/reports` p50 is 9.4s. Diagnose the bottleneck to a specific-> checklist cause and propose the fix.--## Expected diagnosis--- **Root cause**: Synchronous work that belongs in a background task — the- 9.1s `pdf.render` span runs before the response returns, for work the user- does not wait on.-- **Key evidence**: the in-process render span dominating the waterfall;- response only returned after upload; UI already polls for completion, so- async is safe.-- **Fix**: return `202 {"status": "processing"}` immediately and move render- + upload into a `CaptureTask` background task.--## Pass criteria--- Names **synchronous work that belongs in a background task** (not "CPU- saturation" — CPU spikes because of this render, it is the effect, not the- cause; not "slow upstream" — S3 put is 410ms).-- Notes the user doesn't need the result synchronously (polling UI) and- proposes the background-task fix.diff --git a/skills/traceway/evals/frontend-js-large-payloads.md b/skills/traceway/evals/frontend-js-large-payloads.mddeleted file mode 100644index 45717ccf..00000000--- a/skills/traceway/evals/frontend-js-large-payloads.md+++ /dev/null@@ -1,51 +0,0 @@-# Eval: frontend-js-large-payloads--- **ID**: `frontend-js-large-payloads`-- **Area**: frontend-- **Framework**: browser JS (React SPA)-- **Checklist item**: Large request/response payloads (`performance.md` →- Network, IO, and external dependencies)--## Telemetry snapshot--`traceway endpoints list --search "GET /api/feed" --since 24h --output json`:--```json-{ "endpoint": "GET /api/feed", "count": 120400, "p50": 1810, "p95": 3400, "p99": 5200, "errorCount": 0 }-```--`traceway endpoints show <endpointId> --recorded-at <t>` — one request-(total 3.28s):--```-GET /api/feed ............ 3280ms-└── db.query ............. 210ms SELECT ... LIMIT 50-└── json.encode .......... 180ms-└── transfer ............. 2840ms response 2.3 MB-```--The response carries 50 feed cards with full-resolution image URLs inlined-as base64 (`data:` URIs, ~45KB each). p50/p95 correlate with the `?page`-depth, not with server load; `system.cpu.utilization` flat at ~25%. Mobile-web p95 is 2× desktop — transfer-dominated.--## Task--> `GET /api/feed` p50 is 1.8s, worse on mobile web. Server-side spans are-> fast. Diagnose the bottleneck to a specific checklist cause.--## Expected diagnosis--- **Root cause**: Large response payloads — 2.3MB per response, dominated by- inlined base64 images; transfer time is the long pole, not server work.-- **Key evidence**: `transfer` span 2.84s vs 210ms DB; latency tracks page- depth and is worse on mobile (bandwidth-bound signature); flat CPU.-- **Fix**: thumbnail URLs + lazy-load full images, paginate or compress,- strip base64 inlining.--## Pass criteria--- Names **large payloads** (not "slow DB" — DB is 210ms; not "no cache" —- there is no evidence about caching here).-- Identifies transfer as the long pole and ties the mobile-web skew to- bandwidth.diff --git a/skills/traceway/evals/frontend-js-no-cache.md b/skills/traceway/evals/frontend-js-no-cache.mddeleted file mode 100644index 65c5353f..00000000--- a/skills/traceway/evals/frontend-js-no-cache.md+++ /dev/null@@ -1,44 +0,0 @@-# Eval: frontend-js-no-cache--- **ID**: `frontend-js-no-cache`-- **Area**: frontend-- **Framework**: browser JS (React SPA)-- **Checklist item**: No cache / low hit ratio (`performance.md` → Caching and delivery)--## Telemetry snapshot--`traceway endpoints list --search "GET /api/products" --since 24h --output json`:--```json-{ "endpoint": "GET /api/products", "count": 388000, "p50": 640, "p95": 980, "p99": 1400, "errorCount": 0 }-```--The product catalog changes at most twice a day, yet the endpoint is hit on-*every* page navigation — 388k hits/day for ~40k daily users. The-`cache.hit_ratio` metric (reported by the CDN edge) sits at 4%. Waterfall of-one request is unremarkable: 380ms DB + 120ms serialize, no single slow span.--`traceway endpoints chart --metric-type total_time --since 7d` shows total-server time for this endpoint is 3× the next endpoint — it is the fleet's-biggest cost center by volume, not by per-request latency.--## Task--> `GET /api/products` per-request latency looks fine, but it is the fleet's-> biggest total-time cost center. Diagnose to a specific checklist cause.--## Expected diagnosis--- **Root cause**: No effective cache — a twice-daily-changing catalog is- fetched on every navigation with a 4% edge hit ratio; the DB is hit on- every request.-- **Key evidence**: 388k hits vs 40k users; 4% hit ratio; total_time- dominance without per-request slownness.-- **Fix**: edge-cache the catalog response with a 6h TTL + surrogate-key- purge on catalog publish; or `stale-while-revalidate`.--## Pass criteria--- Names **no cache / low hit ratio** (not "slow query" — per-request spans- are fine; the problem is volume × repetition).-- Uses the hit-ratio metric and the hits-per-user ratio, not just latency.diff --git a/skills/traceway/evals/infra-cache-stampede.md b/skills/traceway/evals/infra-cache-stampede.mddeleted file mode 100644index 1cbfb4e1..00000000--- a/skills/traceway/evals/infra-cache-stampede.md+++ /dev/null@@ -1,46 +0,0 @@-# Eval: infra-cache-stampede--- **ID**: `infra-cache-stampede`-- **Area**: infrastructure-- **Framework**: n/a (Redis-backed API)-- **Checklist item**: Cache stampede / thundering herd (`performance.md` →- Caching and delivery)--## Telemetry snapshot--`traceway endpoints list --search "GET /api/homepage" --since 24h --output json`:--```json-{ "endpoint": "GET /api/homepage", "count": 210000, "p50": 95, "p95": 210, "p99": 4800, "errorCount": 0 }-```--`traceway endpoints chart --metric-type p99 --since 24h --interval-minutes 5`-for this endpoint: p99 is ~200ms all day except sharp 4–5s spikes at-**:00, :10, :20, :30, :40, :50** — exactly the 10-minute cache TTL boundary.--During a spike window, `traces show` reveals hundreds of near-identical-traces all recomputing the homepage payload (12 DB queries each) within the-same second; between spikes, the cache-hit metric reads 99%. The stampede-traces show no lock contention — every worker independently missed and-recomputed.--## Task--> `GET /api/homepage` p99 spikes to 5s exactly every 10 minutes, aligned to-> the cache TTL. Diagnose to a specific checklist cause.--## Expected diagnosis--- **Root cause**: Cache stampede / thundering herd — at each TTL expiry a- burst of workers simultaneously miss and recompute the same payload.-- **Key evidence**: spikes phase-locked to the TTL boundary; burst of- identical recompute traces; 99% hit ratio between spikes (rules out "no- cache").-- **Fix**: probabilistic early refresh / request coalescing (singleflight)- / staggered TTLs.--## Pass criteria--- Names **cache stampede** (not "no cache" — hit ratio is 99% between- spikes; not "slow DB" — queries are fine, the burst is the problem).-- Cites the TTL-aligned periodicity as the decisive signal.diff --git a/skills/traceway/evals/infra-cpu-saturation.md b/skills/traceway/evals/infra-cpu-saturation.mddeleted file mode 100644index 9727c05b..00000000--- a/skills/traceway/evals/infra-cpu-saturation.md+++ /dev/null@@ -1,49 +0,0 @@-# Eval: infra-cpu-saturation--- **ID**: `infra-cpu-saturation`-- **Area**: infrastructure-- **Framework**: n/a (fleet-wide)-- **Checklist item**: CPU saturation (`performance.md` → Infrastructure and saturation)--## Telemetry snapshot--`traceway endpoints list --since 24h --order-by p95 --page-size 10`: the top-8 endpoints *all* show p95 2–4× their 7-day baseline, starting at 15:40-yesterday. No single span dominates — waterfalls show every span type-(DB, serialize, upstream) uniformly slower, with no long pole.--`traceway metrics query --name system.cpu.utilization --aggregation max --since 24h --interval-minutes 60`:--```-14:00 38% 15:00 44% 16:00 97% 17:00 98% 18:00 96% ...-```--The CPU step at ~15:40 lines up exactly with the latency rise across all-endpoints. `git log` shows the only deploy in the window was a config-only-change at 15:35 that doubled the log verbosity (see also-`backend-excessive-logging`, not this eval). `mem.used` flat.--## Task--> Eight unrelated endpoints all went 2–4× slower at 15:40 yesterday, with no-> single slow span. Diagnose to a specific checklist cause — and say what it-> is *not*.--## Expected diagnosis--- **Root cause**: CPU saturation — fleet-wide latency rise with no long- pole, aligned exactly with the `system.cpu.utilization` step to 97%+.-- **Key evidence**: broad rise across endpoints (rules out a code bug in one- path); no dominating span (rules out a slow query/upstream); metric- inflection at the same timestamp.-- **What it is not**: not a code regression (config-only deploy, no single- culprit span); the doubled log verbosity is a *contributor* to CPU, worth- fixing, but the finding is saturation.-- **Fix**: scale out / move the verbose logging off the hot path, then- re-check headroom.--## Pass criteria--- Names **CPU saturation** and explicitly rules out a single code/query- cause using the breadth of the rise and the metric alignment.-- Does not blame the config deploy as a code regression without evidence.diff --git a/skills/traceway/evals/mobile-android-gc-pauses.md b/skills/traceway/evals/mobile-android-gc-pauses.mddeleted file mode 100644index 40718057..00000000--- a/skills/traceway/evals/mobile-android-gc-pauses.md+++ /dev/null@@ -1,52 +0,0 @@-# Eval: mobile-android-gc-pauses--- **ID**: `mobile-android-gc-pauses`-- **Area**: mobile-- **Framework**: Android (native, ART)-- **Checklist item**: Memory pressure / GC pauses (`performance.md` →- Infrastructure and saturation)--## Telemetry snapshot--`scroll.jank` trace group (feed scroll): p50 frame time 22ms, p99 190ms.-The p99 frames cluster in sessions with >3 minutes of continuous scrolling.--`traceway traces show <traceId> --recorded-at <t>` — one janky scroll-window:--```-scroll.window ............ 2400ms-├── frame ................ 24ms-├── frame ................ 28ms-├── <gap: no spans> ...... 165ms ← ART GC pause (concurrent → blocking)-├── frame ................ 188ms-└── image.decode ......... 96ms (full-res bitmap, 4096×4096)-```--`mem.used` climbs monotonically across the session; the image cache has no-size cap (in-memory LRU unbounded). GC pause frequency tracks scroll depth,-not network or server state. Server spans for the same feed are fast-(p95 300ms).--## Task--> Android feed scroll p99 frame time is 190ms, worsening the longer the-> session runs. Server side is fine. Diagnose to a specific checklist cause.--## Expected diagnosis--- **Root cause**: Memory pressure / GC pauses — unbounded in-memory image- cache grows with scroll depth; ART GC pauses surface as gaps in the frame- timeline.-- **Key evidence**: the span-less gap (GC, not work); `mem.used` climbing- with session length; full-res decodes feeding an uncapped cache; server- side clean.-- **Fix**: cap the image cache (e.g. 128MB LRU), decode at display size,- recycle bitmaps.--## Pass criteria--- Names **memory pressure / GC pauses** and reads the gap as GC rather than- missing work.-- Connects the uncapped cache + `mem.used` climb to the pauses (not "slow- images" alone — decode is 96ms, the 165ms gap is the killer).diff --git a/skills/traceway/evals/mobile-flutter-shader-jank.md b/skills/traceway/evals/mobile-flutter-shader-jank.mddeleted file mode 100644index bddfac4f..00000000--- a/skills/traceway/evals/mobile-flutter-shader-jank.md+++ /dev/null@@ -1,47 +0,0 @@-# Eval: mobile-flutter-shader-jank--- **ID**: `mobile-flutter-shader-jank`-- **Area**: mobile-- **Framework**: Flutter-- **Checklist item**: Cold start / JIT / connection warmup (`performance.md`- → Caching and delivery; "latency normalizes after warmup")--## Telemetry snapshot--`route.transition` trace group (product detail page): p50 320ms, p99 2.1s.-The p99s are *first visits per install* to any page with the animated hero-carousel — `firstSeen`-correlated per route; second visits render at ~300ms.--`traceway traces show <traceId> --recorded-at <t>` — one first visit (2.0s):--```-route.transition ......... 2000ms-├── build/layout ......... 180ms-├── <gap: raster thread> . 1450ms ← shader compilation (SkSL warmup)-└── first-frame .......... 370ms-```--`system.cpu.utilization` on device spikes during the gap (raster thread-compiling shaders); no network spans in the window — the page data was-already cached. After the first visit, subsequent transitions are smooth.--## Task--> Flutter product-page transitions hit 2.1s p99, but only the first visit-> per install; data is cached and the network is idle. Diagnose to a-> specific checklist cause.--## Expected diagnosis--- **Root cause**: Shader/JIT warmup jank — first-frame SkSL compilation on- the raster thread; classic warmup pattern (bad once, then fine).-- **Key evidence**: first-visit-only outliers; raster-thread gap with no- spans and no network; CPU spike on device during the gap.-- **Fix**: ship SkSL warmup (`flutter build --bundle-sksl-path`), or- pre-warm the carousel offscreen.--## Pass criteria--- Names **shader/JIT warmup** (not "slow build" — build/layout is 180ms;- not network — idle).-- Recognizes the first-visit-only + normalizes-after-warmup signature.diff --git a/skills/traceway/evals/mobile-ios-cold-start.md b/skills/traceway/evals/mobile-ios-cold-start.mddeleted file mode 100644index 1f09f00b..00000000--- a/skills/traceway/evals/mobile-ios-cold-start.md+++ /dev/null@@ -1,49 +0,0 @@-# Eval: mobile-ios-cold-start--- **ID**: `mobile-ios-cold-start`-- **Area**: mobile-- **Framework**: iOS (native)-- **Checklist item**: Cold start (`performance.md` → Caching and delivery;- "first-request-after-deploy outliers … latency normalizes after warmup")--## Telemetry snapshot--Mobile `app.start` trace group, last 24h: p50 1.1s, p95 3.8s. The p95 is-entirely first launches after install/update — `firstSeen`-correlated; warm-starts sit at 0.4s.--`traceway traces show <traceId> --recorded-at <t>` — one cold start (3.6s):--```-app.start ................ 3600ms-└── didFinishLaunching ... 2200ms-│ ├── sdk.init ......... 900ms analytics + crash + attribution SDKs, sequential-│ ├── db.migrate ....... 700ms CoreData lightweight migration on main thread-│ └── config.fetch ..... 600ms synchronous remote-config fetch, blocks return-└── first-frame .......... 1400ms-```--No network saturation (config endpoint p95 120ms); the 600ms is one-blocking call plus main-thread contention. Subsequent launches skip the-migration and use cached config.--## Task--> iOS cold start p95 is 3.8s, but only on first launch after install/update.-> Diagnose to a specific checklist cause.--## Expected diagnosis--- **Root cause**: Cold start with synchronous init on the main thread —- sequential SDK init, main-thread CoreData migration, and a blocking- remote-config fetch inside `didFinishLaunching`.-- **Key evidence**: first-launch-only outliers (`firstSeen`-correlated);- warm starts fine; the three init spans serialized on the main thread.-- **Fix**: defer non-critical SDK init off the critical path, move migration- off main thread, make remote config async with cached fallback.--## Pass criteria--- Names **cold start / synchronous launch init** and ties it to- first-launch-only data.-- Identifies all three serialized init spans (not just "slow SDK").