The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
Perf cases (#411)
narrow-adventure4d96466a265b
diff --git a/cli/test/evals/README.md b/cli/test/evals/README.mdindex 3fdc8947..76c64bcb 100644--- a/cli/test/evals/README.md+++ b/cli/test/evals/README.md@@ -312,7 +312,11 @@ Backend facts that bite (each one cost a case author an afternoon):- Rare events confined to a short window can come out empty at some hours ofthe day (the arrival process follows the wall-clock hour). Use `Burst`, orseed the case at every hour of the day into `seed.NewDrySink()` and check- the event is there before relying on it.+ the event is there before relying on it. The same goes for a `DetailPolicy`+ sample: overnight it can leave the last few hours with no detailed request,+ so a trace (or access log) an oracle step reaches through a log or list+ query must be forced by exposing it, `s.Expose(key, e.Pick(...))` or+ `e.Slowest(...)`, which keeps that request's detail.- Numeric OTel attributes must be integers, which the DSL does for you.- Projects with a browser framework never get endpoints from OTLP.diff --git a/cli/test/evals/cases/perf/perf-app-cpu-serialization.yaml b/cli/test/evals/cases/perf/perf-app-cpu-serialization.yamlnew file mode 100644index 00000000..f5d61c85--- /dev/null+++ b/cli/test/evals/cases/perf/perf-app-cpu-serialization.yaml@@ -0,0 +1,77 @@+id: perf-app-cpu-serialization+suite: perf+title: The transaction export burns seconds of CPU building its JSON body+tags: [application, dotnet]+seed: perf/app-cpu-serialization+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: The export bodies are large (0.1-4.7MB, logged with every export), which reads like a payload problem, but the time is a gap between the transactions SELECT and the export_audit INSERT, before a single byte of the body is sent, and the body is written in a few milliseconds afterwards. Each row costs ~0.35ms of CPU, while the paged transaction list serializes 50 of the same rows in about 2ms.+workspace: {repo: accounts-api-dotnet}+prompt: |+ Finance keeps complaining that the transaction export in the accounts dashboard takes several seconds for big accounts, sometimes most of ten, and the accounts-api CPU graph jumps whenever someone runs one. The query itself is supposed to be fast. Can you look at it in Traceway and tell me what's going on and how to fix it?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: cpu_bound_serialization+ not: [large_request_response_payloads, no_pagination_unbounded_result_set, over_fetching, cpu_saturation]+ locus: {endpoint: "GET /api/accounts/{accountId}/transactions/export"}+ grounding:+ - "INSERT export_audit"+ - "Exported"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~system\\.cpu"}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: CPU-bound serialization. In the export's waterfall the transactions SELECT returns in tens to a couple of hundred milliseconds, then nothing happens on the trace for 0.3-7s until the export_audit INSERT, and the body goes out a few ms later. That gap is the endpoint building the JSON body in memory - ~0.35ms of CPU per row, so it scales with the account's row count (p50 ~1.3s, p95 ~4s) - and CPU jumps while it runs. The code creates a new JsonSerializerOptions for every row (no metadata cache), parses each row's metadata with JsonDocument, resolves the currency culture by enumerating every culture for every amount, and indents the output, all into one string. The bodies are 0.1-4.7MB but are written in milliseconds after the gap, so it is not a transfer problem; the paged list endpoint serializes 50 rows in ~2ms. Fix - one static JsonSerializerOptions (or a source-generated context), cache the culture, write the metadata raw, and stream rows with Utf8JsonWriter to the response body instead of building a string.+ rubric:+ - Locates the time in the untraced gap between the transactions SELECT and the export_audit INSERT and identifies it as in-process JSON building, with CPU up while it runs.+ - Rules out the payload size as the cause (the gap happens before anything is sent, sending takes milliseconds) and a slow query (the SELECT is short).+ - Proposes reusing one JsonSerializerOptions or a source-generated context, caching the currency culture, and streaming rows with Utf8JsonWriter instead of building the body per row.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: traceway logs query --since 24h --search Exported --page-size 15 --output json | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: traceway logs query --since 24h --search transactions/export --page-size 15 --output json | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, started: .endpoint.recordedAt[11:23], total_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {start: .startTime[11:23], ms: (.duration/1e6|round), name, query: .attributes["db.query.text"]}]}'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --tag state=user --from {{seed.slowFrom}} --to {{seed.slowTo}} --interval-minutes 1 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by state --since 24h --interval-minutes 360 --output json+ | jq '.results[0].series | map_values(map(.value * 100 | round))'+ - read_file: src/AccountsApi/Exports/TransactionExportEndpoints.cs+ - read_file: src/AccountsApi/Serialization/MoneyConverter.cs+ - submit:+ checklist_item: cpu_bound_serialization+ category: application+ locus: {endpoint: "GET /api/accounts/{accountId}/transactions/export", span: "gap between SELECT transactions and INSERT export_audit", service: accounts-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "the export is the only slow endpoint, with a p50 over a second and a p95 of several seconds"}+ - {call_id: c02, claim: "exports log from hundreds to over ten thousand rows each, with bodies of up to a few MB"}+ - {call_id: c03, claim: "matched to c02 by trace id, export durations grow with the logged row counts, about a third of a millisecond per row"}+ - {call_id: c04, claim: "in the slowest export of the day the transactions SELECT returns in under 200ms, then the trace is idle for over six seconds until INSERT export_audit; the body goes out right after"}+ - {call_id: c05, claim: "user CPU jumps from ~13% to ~50% in the minute the export was serializing, and only in minutes with an export"}+ - {call_id: c06, claim: "average CPU is low over the day (~20% busy), so the host is not saturated"}+ ruled_out:+ - {checklist_item: large_request_response_payloads, call_id: c04, reason: "the time is spent before anything is sent; the response is written within milliseconds after the audit insert"}+ - {checklist_item: no_pagination_unbounded_result_set, call_id: c04, reason: "the full-period read is what an export is for, and the SELECT itself is short"}+ - {checklist_item: cpu_saturation, call_id: c06, reason: "average CPU is low; the spikes are the exports themselves"}+ fix: Reuse one JsonSerializerOptions (or a source-generated JsonSerializerContext), resolve the currency culture once, write each row's metadata raw instead of parsing it, drop the indentation, and stream the rows with Utf8JsonWriter straight to the response body instead of building a string per row.+ confidence: high+ report: |+ The export is slow because it builds its JSON body row by row on the CPU. In the slowest export of the day (~6.5s) the transactions SELECT took ~160ms, then the trace shows nothing for ~6.3s until INSERT export_audit, and the body went out within milliseconds after that. The gap grows with the logged row count (about 0.35ms per row) and user CPU jumps from ~13% to ~50% in the minute it runs; average CPU is ~20%. TransactionExportEndpoints.SerializeRow creates a new JsonSerializerOptions for every row, so System.Text.Json rebuilds its type metadata each time, parses every row's metadata with JsonDocument, and MoneyConverter enumerates every specific culture for every amount. The multi-MB bodies are not the problem: nothing is sent until the gap is over. Fix: stream the rows with one Utf8JsonWriter (and one static JsonSerializerOptions or a source-generated context) straight to the response, write the metadata raw, drop the indentation and resolve the currency culture once.diff --git a/cli/test/evals/cases/perf/perf-app-excessive-logging.yaml b/cli/test/evals/cases/perf/perf-app-excessive-logging.yamlnew file mode 100644index 00000000..69ef30fd--- /dev/null+++ b/cli/test/evals/cases/perf/perf-app-excessive-logging.yaml@@ -0,0 +1,89 @@+id: perf-app-excessive-logging+suite: perf+title: A release that only turned on debug logs slows every endpoint and doubles CPU+tags: [application, go, deploy]+seed: perf/app-excessive-logging+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: Host CPU on both pods stepped from ~35% to ~78%, which reads like saturation and invites scaling out, but traffic at the same time of day is unchanged, the CPU step comes five minutes after release 5.3.1 rolled out, and from that release on every request writes ~15-20x as many log lines (one per decoded row or image), which is where the extra time per request goes.+workspace: {repo: listings-api-go}+prompt: |+ /traceway perf listings-api+ For the last day or so every endpoint on listings-api has been slower, p95 more than doubled, and CPU on its pods went from around a third to about 80%. Should we scale it out, or is something else going on?+budget: {max_turns: 40, max_tool_calls: 34, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: excessive_logging_telemetry_in_the_hot_path+ not: [cpu_saturation, undersized_noisy_neighbor_instance, cold_start_jit_connection_warmup, select_n_plus_1]+ locus: {service: listings-api}+ onset: {at: "{{seed.deployAt}}", tolerance_minutes: 30}+ grounding:+ - "decoded listing"+ - "LOG_LEVEL=debug"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~system\\.cpu"}}+ - {op: logs.query}+ any_of:+ - [{op: endpoints.chart}]+ - [{op: endpoints.list, args: {from: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: Excessive logging in the hot path. Release 5.3.1 (git - "debug logs on to chase the ranking complaints") changed only deploy/production.env, LOG_LEVEL=info to debug. From the deploy on, every endpoint's latency stepped up (GET /api/listings p95 ~18ms to ~42ms, GET /api/listings/{id} ~10ms to ~22ms) and five minutes later, once both pods had restarted, CPU went from ~35% to ~78%. Request counts at the same time of day did not change. A search request now writes 40-50 log lines (one "decoded listing" per row plus cache/query lines) against 2 before, and overall log volume per hour went up ~15-20x; with AddSource on at debug, each line costs about half a millisecond on the request path. It is not saturation from load - fix the config (LOG_LEVEL=info), and if debug detail is needed, sample it or enable it per request instead of logging per row.+ rubric:+ - Ties the latency step and the later CPU step to the 5.3.1 release, and shows from the logs that requests write many times more lines since then (per trace and in total).+ - Rules out load-driven saturation with request counts at the same time of day before and after the release, and does not recommend scaling out as the fix.+ - Proposes reverting LOG_LEVEL to info (or sampling / per-request debug) rather than adding capacity, and points at the per-row debug logging in the listings repo.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 48h --interval-minutes 120 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, p95_ms: map("\(.timestamp[5:16]) \(.value|round)")})'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by state --since 48h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[5:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway endpoints list --from {{seed.beforeFrom}} --to {{seed.deployAt}} --output json+ | jq '[.data[] | {endpoint, count, p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --from {{seed.afterFrom}} --to {{seed.afterTo}} --output json+ | jq '[.data[] | {endpoint, count, p95_ms: (.p95Duration/1e6|round)}]'+ - shell: traceway logs query --from {{seed.beforeHourFrom}} --to {{seed.beforeHourTo}} --page-size 1 --output json | jq '.pagination'+ - shell: traceway logs query --from {{seed.afterHourFrom}} --to {{seed.afterHourTo}} --page-size 1 --output json | jq '.pagination'+ - shell: >-+ traceway logs query --trace-id {{seed.afterRequest.traceId}} --since 6h --page-size 100 --output json+ | jq '{lines: .pagination.total, version: .data[0].resourceAttributes["service.version"], bodies: ([.data[] | "\(.severityText) \(.body)"] | group_by(.) | map({line: .[0], n: length}))}'+ - shell: >-+ traceway logs query --trace-id {{seed.beforeRequest.traceId}} --from {{seed.beforeFrom}} --to {{seed.deployAt}} --page-size 100 --output json+ | jq '{lines: .pagination.total, version: .data[0].resourceAttributes["service.version"], bodies: [.data[] | "\(.severityText) \(.body)"]}'+ - shell: git log -p -n 1 --date=iso-strict --format='%h %ad %an %s'+ - submit:+ checklist_item: excessive_logging_telemetry_in_the_hot_path+ category: application+ locus: {service: listings-api, span: "log writes between the spans of every request"}+ onset: "{{seed.deployAt}}"+ evidence:+ - {call_id: c02, claim: "every endpoint's p95 stepped up in the same bucket and stayed there"}+ - {call_id: c03, claim: "CPU stepped from ~35% to ~78% right after that"}+ - {call_id: c04, claim: "in the six hours before the release every endpoint ran at its old p95"}+ - {call_id: c05, claim: "the same six hours of the day after it served about the same request counts as before (c04), with p95 roughly doubled: no extra load"}+ - {call_id: c06, claim: "an hour before the release, at the same time of day as c07, gives the baseline log volume"}+ - {call_id: c07, claim: "the same hour of the day after the release wrote roughly 15-20 times as many lines"}+ - {call_id: c08, claim: "one search request on 5.3.1 writes 40-50 lines, one 'decoded listing' DEBUG line per row"}+ - {call_id: c09, claim: "a search request on 5.3.0 wrote two lines"}+ - {call_id: c10, claim: "the release 5.3.1 commit changed only deploy/production.env: LOG_LEVEL=info to LOG_LEVEL=debug"}+ ruled_out:+ - {checklist_item: cpu_saturation, call_id: c05, reason: "request counts are flat; CPU rose because each request writes far more logs, right after the config release"}+ - {checklist_item: cold_start_jit_connection_warmup, call_id: c02, reason: "latency stays elevated for a day instead of recovering after the rollout"}+ fix: Set LOG_LEVEL back to info in deploy/production.env and roll it out; if the ranking investigation needs detail, sample the debug lines or enable debug per request instead of logging every decoded row.+ confidence: high+ report: |+ listings-api is not short of capacity; release 5.3.1 turned on debug logging. The release (git - "debug logs on to chase the ranking complaints") only changed deploy/production.env from LOG_LEVEL=info to LOG_LEVEL=debug. From its deploy every endpoint got slower (GET /api/listings p95 from ~18ms to ~42ms) and CPU on both pods stepped from ~35% to ~78% five minutes later, once the rollout finished. Traffic at the same time of day is unchanged. A search request now writes ~45 log lines, one "decoded listing" DEBUG line per row plus cache and query lines, where it wrote 2 before; total log volume per hour went up about 15-20 times. Fix: set LOG_LEVEL back to info in deploy/production.env and roll it out; latency and CPU should return to their previous levels. Scaling out would only spread the logging cost. If the ranking investigation needs the detail, sample the debug lines or enable debug per request instead of logging every decoded row.diff --git a/cli/test/evals/cases/perf/perf-app-large-alloc.yaml b/cli/test/evals/cases/perf/perf-app-large-alloc.yamlnew file mode 100644index 00000000..339454f7--- /dev/null+++ b/cli/test/evals/cases/perf/perf-app-large-alloc.yaml@@ -0,0 +1,87 @@+id: perf-app-large-alloc+suite: perf+title: The retention report loads every event of the range into the JVM heap+tags: [application, java]+seed: perf/app-large-alloc+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The JVM looks like the culprit (old generation jumping from ~0.6GB to 3GB+ and young collections jumping from ~3 to 20-30 a minute with ~40ms average pauses), but that only happens while a retention request runs, the collections add up to about a second of a request that takes several, the heap falls back to ~0.6GB after each one (no leak), and every other endpoint on the same JVM keeps its normal p99. The allocation is the endpoint's own work.+workspace: {repo: insights-api-java}+prompt: |+ /traceway perf GET /api/insights/retention+ The retention report in the insights dashboard takes anywhere from one to ten seconds. Ops thinks it's the JVM: the insights-api graphs show the old generation filling up and GC time spiking whenever it's slow, and they want to double the heap. Is that the right fix, or is something else going on?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: unbounded_in_memory_work_large_allocations+ not: [memory_pressure_gc_pauses, no_pagination_unbounded_result_set, expensive_join_or_aggregation, inefficient_algorithm_o_n2_hot_path, cpu_bound_serialization]+ locus: {endpoint: "GET /api/insights/retention", span_contains: "buildCohorts"}+ grounding:+ - "RetentionService.buildCohorts"+ - "G1 Old Gen"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~^jvm\\."}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, endpoints.show]+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: Unbounded in-memory work / large allocations. GET /api/insights/retention (p50 ~1.5s, p99 ~9s) spends its time in the RetentionService.buildCohorts span; only ~20-25% of it is the events SELECT, the rest is the JVM grouping, sorting and bucketing every event of the requested range (0.1-2.5M rows; the 180-day ranges are the slow ones). While a request runs the G1 old generation jumps from ~0.6GB to 3GB+ and young collections spike to 20-30/min at ~40ms, then everything returns to baseline; used-after-GC falls back to ~0.6GB, so there is no leak, and the other endpoints' p99 is unaffected. GC is a symptom, and doubling the heap would only delay it. Fix - compute the cohort x period counts in PostgreSQL (DISTINCT user/period + GROUP BY) so the JVM receives a few hundred rows, or at least stream with a fetch size and aggregate incrementally.+ rubric:+ - Locates the time in the retention endpoint's own buildCohorts work over the full event range, with heap and GC activity rising only while it runs.+ - Rules out GC or memory pressure as the root cause (no leak, heap returns to baseline, other endpoints unaffected, the pauses are a small share) and explains why more heap would not fix it.+ - Proposes moving the cohort aggregation into SQL (or streaming with incremental aggregation) instead of materializing every event as Java objects.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: traceway logs query --since 24h --search /api/insights/retention --page-size 10 --output json | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{query: .endpoint.attributes["url.query"], total_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, ms: (.duration/1e6|round), start: .startTime[11:23], span: .spanId, parent: .parentSpanId}]}'+ - shell: >-+ traceway endpoints show {{seed.fastRequest.id}} --recorded-at {{seed.fastRequest.recordedAt}} --output json+ | jq '{query: .endpoint.attributes["url.query"], total_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: (.duration/1e6|round)}]}'+ - shell: >-+ traceway metrics query --name jvm.memory.used --aggregation max --group-by jvm.memory.pool.name --from {{seed.slowFrom}} --to {{seed.slowTo}} --interval-minutes 1 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value/1e6|round)MB"))'+ - shell: >-+ traceway metrics query --name jvm.gc.duration.count --aggregation sum --group-by jvm.gc.name --from {{seed.slowFrom}} --to {{seed.slowTo}} --interval-minutes 1 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value|round)"))'+ - shell: >-+ traceway metrics query --name jvm.gc.duration.avg --aggregation max --group-by jvm.gc.name --from {{seed.slowFrom}} --to {{seed.slowTo}} --interval-minutes 1 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value*1000|round)ms"))'+ - shell: >-+ traceway metrics query --name jvm.memory.used_after_last_gc --aggregation min --since 24h --interval-minutes 180 --output json+ | jq '.results[0].series | map_values(map(.value/1e6|round))'+ - read_file: src/main/java/com/acme/insights/retention/RetentionService.java+ - submit:+ checklist_item: unbounded_in_memory_work_large_allocations+ category: application+ locus: {endpoint: "GET /api/insights/retention", span: "RetentionService.buildCohorts", service: insights-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "only the retention endpoint is slow (p99 several seconds); funnels and dashboards on the same JVM keep their usual p99"}+ - {call_id: c03, claim: "the slowest sampled request over a 180-day range spends most of its time inside RetentionService.buildCohorts after the events SELECT returned, i.e. in Java over the loaded rows"}+ - {call_id: c04, claim: "a 30-day request finishes in well under a second: the cost follows the size of the requested range, i.e. the number of events loaded"}+ - {call_id: c05, claim: "the G1 old generation jumps from ~0.6GB to over 3GB in the minute that request ran, then drops straight back"}+ - {call_id: c06, claim: "young collections spike only in the minutes a retention request runs"}+ - {call_id: c07, claim: "their average pause stays around 40ms, a small share of a multi-second request"}+ - {call_id: c08, claim: "heap after GC returns to ~0.6GB, so nothing leaks"}+ ruled_out:+ - {checklist_item: memory_pressure_gc_pauses, call_id: c08, reason: "no leak and no fleet-wide pauses; GC activity is a consequence of this endpoint loading every event, and the other endpoints are unaffected"}+ - {checklist_item: expensive_join_or_aggregation, call_id: c03, reason: "the SQL is a plain range scan that returns in a fraction of the request; the aggregation happens in Java"}+ fix: Compute the cohort x period distinct-user counts in PostgreSQL (DISTINCT user/period, first period per user, GROUP BY) so buildCohorts receives a few hundred rows instead of every event; a bigger heap would only postpone the collections.+ confidence: high+ report: |+ Doubling the heap would not fix this. The retention endpoint materializes every event of the requested range as Java objects (up to millions of rows for 180-day ranges) and groups them in RetentionService.buildCohorts. In the slowest sampled request the events SELECT took about 1.9s and the remaining ~6.6s of the 8.5s were spent in Java inside buildCohorts. In that minute the G1 old generation jumped from ~0.6GB to ~3.4GB and young collections went from 3 to ~20 per minute at ~40ms, then everything dropped back; heap after GC sits at ~0.6GB, so there is no leak, and funnels and dashboards on the same JVM keep their normal p99. The GC activity is caused by this endpoint's allocation. Fix: move the aggregation into PostgreSQL, with one query that returns the distinct users per cohort and period, so the JVM handles a few hundred rows instead of every event.diff --git a/cli/test/evals/cases/perf/perf-app-quadratic.yaml b/cli/test/evals/cases/perf/perf-app-quadratic.yamlnew file mode 100644index 00000000..fac35993--- /dev/null+++ b/cli/test/evals/cases/perf/perf-app-quadratic.yaml@@ -0,0 +1,77 @@+id: perf-app-quadratic+suite: perf+title: Quotes for large carts take seconds while small carts are instant+tags: [application, python]+seed: perf/app-quadratic+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: Large carts look like a memory problem (big in-memory work per request), but process RSS and host memory are flat, and the time is in-process compute in apply_promotions that grows with the square of cart.items (~20ms at 40 items, ~0.3s at 170, over a second at 330+) while both queries stay under ~10ms.+workspace: {repo: pricing-service-py}+prompt: |+ /traceway perf POST /api/quotes+ Some quote requests at checkout take more than a second, mostly from wholesale accounts, while the rest come back instantly. As far as I know nothing changed on our side. What's causing it and how do we fix it?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: inefficient_algorithm_o_n2_hot_path+ not: [unbounded_in_memory_work_large_allocations, cpu_bound_serialization, missing_or_unused_index, cpu_saturation]+ locus: {endpoint: "POST /api/quotes", span_contains: "apply_promotions"}+ grounding:+ - "apply_promotions"+ - "cart.items"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Inefficient algorithm (O(n^2) hot path). POST /api/quotes has a low p50 (~12ms) and a p99 around a second. In every trace the two queries take a few ms and nearly all the time is the apply_promotions compute span, whose duration grows with the square of the root span's cart.items attribute (~20ms at 40 items, ~0.3s at 170, 1.2s+ at 330). apply_promotions sums the quantity of every other line in the cart for every line (and rebuilds the SKU list per bundle check), so pricing is quadratic in cart size. Memory is flat and nothing was deployed. Fix - compute the per-family totals and the SKU set once (Counter/set) and index promotions by family and SKU, making it linear.+ rubric:+ - Compares several traces and shows that apply_promotions time grows superlinearly (roughly quadratically) with cart.items while the DB spans stay small.+ - Rules out a database cause (fast, constant queries) and memory pressure or large allocations (flat RSS).+ - Points at the nested scan in apply_promotions and proposes precomputing family totals and the SKU set once so the pass is linear.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: traceway logs query --since 12h --search /api/quotes --page-size 20 --output json | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway endpoints show {{seed.bigQuote.id}} --recorded-at {{seed.bigQuote.recordedAt}} --output json+ | jq '{cart_items: .endpoint.attributes["cart.items"], total_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway endpoints show {{seed.midQuote.id}} --recorded-at {{seed.midQuote.recordedAt}} --output json+ | jq '{cart_items: .endpoint.attributes["cart.items"], total_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway endpoints show {{seed.smallQuote.id}} --recorded-at {{seed.smallQuote.recordedAt}} --output json+ | jq '{cart_items: .endpoint.attributes["cart.items"], total_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway metrics query --name process.runtime.cpython.memory --aggregation max --group-by type --since 24h --interval-minutes 360 --output json+ | jq '.results[0].series | map_values(map(.value / 1e6 | round))'+ - read_file: app/pricing/promotions.py+ - submit:+ checklist_item: inefficient_algorithm_o_n2_hot_path+ category: application+ locus: {endpoint: "POST /api/quotes", span: "apply_promotions", service: pricing-service}+ onset: null+ evidence:+ - {call_id: c01, claim: "POST /api/quotes has a ~12ms p50 but a p99 around a second: a minority of requests are slow"}+ - {call_id: c03, claim: "the largest cart spends over a second in apply_promotions; both queries take under 10ms"}+ - {call_id: c04, claim: "a cart with about half as many items spends roughly a quarter of that time in apply_promotions"}+ - {call_id: c05, claim: "a cart of a few dozen items spends under 30ms in apply_promotions: the cost grows with the square of cart.items"}+ - {call_id: c06, claim: "process RSS is flat, so this is not memory growth"}+ ruled_out:+ - {checklist_item: unbounded_in_memory_work_large_allocations, call_id: c06, reason: "flat RSS; the time is compute that scales with the square of the item count"}+ - {checklist_item: missing_or_unused_index, call_id: c03, reason: "the price and promotion queries take under 10ms even for the largest cart"}+ fix: In apply_promotions, compute per-family quantities with a Counter and the cart's SKUs as a set once, and index the promotions by family and SKU, so each line is priced in constant time and the pass is linear in the cart size.+ confidence: high+ report: |+ Quote latency grows with the square of the cart size. In the traces I compared, the price and promotion queries take a few milliseconds regardless of size, and nearly all the time is the apply_promotions span: ~27ms for a 47-item cart, ~0.3s at 169 items and ~1.26s at 332. apply_promotions (app/pricing/promotions.py) sums the quantity of every line of the cart for every line, and rebuilds the cart's SKU list for each bundle check, so the work is O(n^2) in the number of lines; wholesale carts are the large ones. Memory is flat and nothing was deployed. Fix: in apply_promotions, build the family totals (a Counter), the cart's SKU set and a promotion index by family and SKU once before the loop, which keeps the same discounts and makes the pass linear.diff --git a/cli/test/evals/cases/perf/perf-app-sync-background.yaml b/cli/test/evals/cases/perf/perf-app-sync-background.yamlindex 663abf98..67ebe97f 100644--- a/cli/test/evals/cases/perf/perf-app-sync-background.yaml+++ b/cli/test/evals/cases/perf/perf-app-sync-background.yaml@@ -76,4 +76,4 @@ oracle:fix: Insert the statement as queued, hand the render and upload to a bounded worker pool (or a durable queue consumer whose job span is an OTel CONSUMER span), answer 202 with the id right away, and let the dashboard's existing poll of GET /api/statements/{id} pick up ready or failed.confidence: highreport: |- POST /api/statements is slow because it renders the statement PDF inside the request. The slowest request of the last day spent ~12s of its ~12.1s in the in-process "render statement pdf" span, then ~0.3s uploading to S3, before answering 201. The p50 is ~9.4s and has been the same all week. The dashboard already polls GET /api/statements/{id} for the status (web/statements.js), so nobody needs to wait on the render. CPU reaches ~80% only while a render runs and averages ~20%, and every other endpoint is fast, so this is not saturation. I changed Create to insert the statement, queue the render for two worker goroutines and answer 202 at once; the workers mark the statement ready or failed, which the existing poll picks up. A durable queue would survive restarts; instrument the worker with an OTel CONSUMER span so renders show up as tasks in Traceway.+ POST /api/statements is slow because it renders the statement PDF inside the request. The slowest request of the last day spent ~12s of its ~12.1s in the in-process "render statement pdf" span, then ~0.3s uploading to S3, before answering 201. The p50 is ~9.4s and has been the same all week. The dashboard already polls GET /api/statements/{id} for the status (web/statements.js), so nobody needs to wait on the render. CPU reaches ~80% only while a render runs and averages ~20%, and every other endpoint is fast, so this is not saturation. Fix: have Create insert the statement, queue the render for a small pool of worker goroutines and answer 202 at once; the workers mark the statement ready or failed, which the existing poll picks up. A durable queue would survive restarts; instrument the worker with an OTel CONSUMER span so renders show up as tasks in Traceway.diff --git a/cli/test/evals/cases/perf/perf-cache-cold-start.yaml b/cli/test/evals/cases/perf/perf-cache-cold-start.yamlnew file mode 100644index 00000000..90798b2e--- /dev/null+++ b/cli/test/evals/cases/perf/perf-cache-cold-start.yaml@@ -0,0 +1,83 @@+id: perf-cache-cold-start+suite: perf+title: Pricing times out for a few minutes after every release+tags: [caching, java, spring-boot, deploy]+seed: perf/cache-cold-start+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: 2.16.0, shipped ~3h ago, rewrote the promotion rules engine, which invites a regression reading, but the same 5-7s first requests followed 2.14.1, 2.15.0 and 2.15.1 on every pod, and steady-state p50/p95 after 2.16.0 equal the day before it.+workspace: {repo: pricing-service-java}+prompt: |+ Every time we release pricing-service, checkout logs a handful of pricing timeouts (its client gives up after 3 seconds) for the first few minutes, and then it settles down. We shipped 2.16.0 a few hours ago with the rewritten promotion rules engine and it happened again. Is 2.16.0 slower, and what should we do about it?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: cold_start_jit_connection_warmup+ not: [connection_pool_exhaustion, cpu_saturation, expensive_join_or_aggregation, inefficient_algorithm_o_n2_hot_path]+ locus: {endpoint: "POST /api/prices/quote", service: pricing-service}+ grounding:+ - "HikariPool-1 - Starting..."+ - "2.15.1"+ calls:+ required:+ - {op: logs.query}+ any_of:+ - [{op: endpoints.chart}, {op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.chart}, {op: endpoints.show, args: {recorded_at: present}}]+ - [{op: endpoints.list, args: {from: present}}, {op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.list, args: {from: present}}, {op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Cold start / connection warm-up, not a 2.16.0 regression. pricing-service runs with spring.main.lazy-initialization=true (src/main/resources/application.yml), so a new pod reports started in ~7s but its first request initializes the DispatcherServlet, creates the web and data beans and starts the Hikari pool on its first query (the request carries "Initializing Spring DispatcherServlet" and "HikariPool-1 - Starting..." logs) and takes 5-7s; the next couple of requests run cold, interpreted code at 0.4-0.8s, and after a handful of requests per pod latency is back to ~19ms p50 / ~28ms p95. The same pattern follows every release in the window (2.14.1, 2.15.0, 2.15.1, 2.16.0) on each of the three pods, and steady-state latency after 2.16.0 equals the day before. Fix by creating beans eagerly (drop lazy initialization), pre-filling the pool, and warming the quote path before the pod reports ready (an ApplicationRunner, which Spring finishes before readiness flips), so the rollout only sends traffic to warm pods.+ rubric:+ - Shows the outliers are the first requests on each new pod after every release (5-7s, with the DispatcherServlet and Hikari initialization logs inside them), and that latency normalizes after a few requests.+ - Rules out a 2.16.0 regression with the data (the same outliers after older releases; steady-state p50/p95 before and after 2.16.0 match) and does not blame the promotion rules rewrite.+ - Proposes removing lazy initialization and warming the pool and the hot path before readiness, so traffic reaches only warmed pods.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --search /api/prices --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 10d --interval-minutes 360 --output json+ | jq '[.series[] | select(.endpoint == "POST /api/prices/quote") | {t: .timestamp, p99_ms: (.value | round)}]'+ - shell: >-+ traceway logs query --since 10d --service pricing-service --search PricingServiceApplication --page-size 30 --output json+ | jq -r '.data[] | "\(.timestamp) \(.resourceAttributes["host.name"]) \(.resourceAttributes["service.version"]) \(.body)"'+ - shell: >-+ traceway logs query --since 3d --service pricing-service --search HikariPool-1 --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.resourceAttributes["host.name"]) \(.resourceAttributes["service.version"]) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.coldRequest.traceId}} --recorded-at {{seed.coldRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, start: .endpoint.recordedAt, ms: (.endpoint.duration / 1e6), version: .endpoint.appVersion,+ spans: [.spans | sort_by(.startTime)[] | {name, start: .startTime, ms: (.duration / 1e6 | round)}]}'+ - shell: >-+ traceway endpoints list --search /api/prices/quote --from {{seed.beforeFrom}} --to {{seed.deploy4}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round)}]'+ - shell: >-+ traceway endpoints list --search /api/prices/quote --from {{seed.afterFrom}} --to {{env.now}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round)}]'+ - shell: git log --format='%h %ad %s' --date=iso -6+ - read_file: src/main/resources/application.yml+ - submit:+ checklist_item: cold_start_jit_connection_warmup+ category: caching+ locus: {endpoint: "POST /api/prices/quote", span: "first request on each new pod: lazy bean creation and HikariPool-1 start before the first JDBC span", service: pricing-service}+ onset: null+ evidence:+ - {call_id: c02, claim: "p99 jumps only in the 6-hour buckets that contain a release, for every release in the last 10 days"}+ - {call_id: c03, claim: "each release restarts all three pods; they report started in ~7s"}+ - {call_id: c04, claim: "the first request on each pod after 2.16.0 and after 2.15.1 starts HikariPool-1 itself (Starting..., Added connection, Start completed)"}+ - {call_id: c05, claim: "that first request takes 5-7s, mostly before its first JDBC span, and its pricing work is slow too"}+ - {call_id: c06, claim: "p50/p95 over the day before 2.16.0"}+ - {call_id: c07, claim: "p50/p95 since 2.16.0 warmed up are the same: no regression in steady state"}+ ruled_out:+ - {checklist_item: connection_pool_exhaustion, call_id: c04, reason: "the wait before the first query is the pool being created on the first request of a new pod, not queueing for a connection; it does not recur"}+ fix: Turn off spring.main.lazy-initialization, pre-fill the Hikari pool, and run a warm-up of the quote path in an ApplicationRunner so pods report ready only when warm.+ confidence: high+ report: |+ 2.16.0 is not slower. The timeouts come from cold pods: pricing-service sets spring.main.lazy-initialization=true (src/main/resources/application.yml), so after every release each new pod starts in ~7s but its first request creates the beans, initializes the DispatcherServlet and starts the Hikari pool (you can see "HikariPool-1 - Starting..." inside those requests) and takes 5-7s, over checkout's 3s timeout; the next few requests run cold code. The same outliers followed 2.14.1, 2.15.0 and 2.15.1 on all three pods, and the p50/p95 since 2.16.0 warmed up match the day before it, so the promotion rules rewrite did not change steady-state latency. Fix: turn lazy initialization off, pre-fill the pool and add a warm-up ApplicationRunner that exercises the quote path before the pod reports ready, so the rolling update only sends traffic to warm pods.diff --git a/cli/test/evals/cases/perf/perf-cache-stampede.yaml b/cli/test/evals/cases/perf/perf-cache-stampede.yamlindex 42eb1735..e028cbeb 100644--- a/cli/test/evals/cases/perf/perf-cache-stampede.yaml+++ b/cli/test/evals/cases/perf/perf-cache-stampede.yaml@@ -75,4 +75,4 @@ oracle:fix: Coalesce misses in home.Service.Feed so one rebuild per key runs and concurrent misses wait for it (singleflight), and add TTL jitter or an early background refresh so the key never expires under load.confidence: highreport: |- The bursts are a cache stampede on the home feed. GET /api/home serves home:feed:v3 from Redis and is ~3ms; the key is set with a 10-minute TTL (ex 600). When it expires, every request that arrives before the first rebuild finishes misses and rebuilds the feed itself, running the twelve section queries; at ~2.4 requests per second that is about twenty identical rebuilds in a few seconds, and they slow each other down in the database, so each takes 7-10s. That is why p99 spikes every ~10 minutes while the cache-hit ratio stays ~98% in between. I changed home.Service.Feed so concurrent misses share a single rebuild (the others wait for its result) and the TTL gets a little jitter; a background refresh before expiry would remove the wait entirely.+ The bursts are a cache stampede on the home feed. GET /api/home serves home:feed:v3 from Redis and is ~3ms; the key is set with a 10-minute TTL (ex 600). When it expires, every request that arrives before the first rebuild finishes misses and rebuilds the feed itself, running the twelve section queries; at ~2.4 requests per second that is about twenty identical rebuilds in a few seconds, and they slow each other down in the database, so each takes 7-10s. That is why p99 spikes every ~10 minutes while the cache-hit ratio stays ~98% in between. Fix: make home.Service.Feed share a single rebuild between concurrent misses (the others wait for its result) and add a little jitter to the TTL; a background refresh before expiry would remove the wait entirely.diff --git a/cli/test/evals/cases/perf/perf-ctl-alert-handoff.yaml b/cli/test/evals/cases/perf/perf-ctl-alert-handoff.yamlnew file mode 100644index 00000000..e6392630--- /dev/null+++ b/cli/test/evals/cases/perf/perf-ctl-alert-handoff.yaml@@ -0,0 +1,105 @@+id: perf-ctl-alert-handoff+suite: perf+title: A pasted p95 alert that fired right after a release restarted the catalog+tags: [control, go, alert, caching]+seed: perf/ctl-alert-handoff+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: A code regression in v5.1.0, which shipped minutes before the alert. But v5.1.0 serves the listing as fast as v5.0.4 once warm (p95 about 20ms before and after), the slow requests all ran in the first minutes after the restart, each on an instance's first request for a category (the listing query plus new pool connections), and the same transient followed the v5.0.4 restart 30 hours earlier.+workspace: {repo: catalog-api-go}+prompt: |+ Got this alert by email about 40 minutes ago while I was in a meeting. Can you check what happened and whether we need to do something?++ [WARNING] [catalog] P95 latency {{seed.alertP95}}ms on GET /api/categories/:slug/products++ The P95 latency for GET /api/categories/:slug/products has reached {{seed.alertP95}}ms over the last 10 minutes (threshold: 1000ms).++ {{env.url}}/endpoints?from={{seed.linkFrom}}&to={{seed.linkTo}}++ You are receiving this because the notification rule "Category listing p95 above 1s" fired.+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I only have the email. Please look into it."+expect:+ answer:+ checklist_item: cold_start_jit_connection_warmup+ not: [none_found, insufficient_evidence, cache_stampede_thundering_herd, no_cache_low_hit_ratio, expensive_join_or_aggregation, missing_or_unused_index]+ locus: {endpoint: "GET /api/categories/:slug/products"}+ onset: {at: "{{seed.deployAt}}", tolerance_minutes: 30}+ grounding:+ - "SELECT id FROM categories WHERE slug"+ calls:+ required:+ - {op: endpoints.slow, args: {endpoint: "GET /api/categories/:slug/products"}}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: >-+ The endpoint is not marked slow (offsetMs 0), so the 1000ms rule applies. v5.1.0 restarted all three catalog instances+ five minutes before the alert. Each instance starts with an empty in-process category cache (a background preload fills+ one category every ~25s) and an empty pgx pool, so for several minutes an instance's first request for a category runs+ the listing query (one to two seconds, products JOIN prices JOIN inventory) and the first requests open connections (connect+ spans). p95 jumped to over a second right after the restart and was back to ~20ms about ten minutes later, well before the user looked; warm v5.1.0+ is as fast as v5.0.4, and the v5.0.4 restart 30 hours earlier shows the same transient. Cold start / connection warmup,+ not a regression. Fix: do not report the instance ready (health check, load balancer) until the preload has filled the+ cache, or load the cache synchronously before serving, and warm the pool (pool_min_conns); a shared cache would also+ survive restarts.+ rubric:+ - Checks whether the endpoint is marked slow and anchors the investigation on the alert's fire time rather than only on the current, healthy window.+ - Shows the latency normalized after the restart (warm v5.1.0 matches v5.0.4) and ties the slow requests to first-request-per-instance cache misses and connection setup.+ - Recommends gating readiness on the cache preload or warming the cache and pool before taking traffic, not reverting v5.1.0.+oracle:+ - shell: >-+ traceway endpoints list --search /api/categories/ --since 1h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round), impactReason}]'+ - op: endpoints.slow+ args: {endpoint: "GET /api/categories/:slug/products"}+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 3h --interval-minutes 5 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/categories/:slug/products") | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: >-+ traceway endpoints list --search /api/categories/ --from {{seed.deployMinus60m}} --to {{seed.deployAt}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --search /api/categories/ --since 30m --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway logs query --search /api/categories/ --from {{seed.deployAt}} --to {{seed.deployPlus15m}} --page-size 30 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.resourceAttributes["host.name"]) \(.resourceAttributes["service.version"]) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, path: .endpoint.attributes["url.path"], version: .endpoint.appVersion, duration_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, ms: (.duration/1e6|round), scope: .scopeName, query: .attributes["db.query.text"]}]}]'+ - shell: >-+ traceway traces show {{seed.warmRequest.traceId}} --recorded-at {{seed.warmRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, version: .endpoint.appVersion, duration_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: (.duration/1e6|round)}]}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 48h --interval-minutes 30 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/categories/:slug/products" and .value > 200) | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: git log --format='%h %ad %s' --date=iso -3+ - read_file: cmd/server/main.go+ - submit:+ checklist_item: cold_start_jit_connection_warmup+ category: caching+ locus: {endpoint: "GET /api/categories/:slug/products", span: "category listing query on a cold instance (products JOIN product_prices JOIN inventory) plus connect", service: catalog-api}+ onset: "{{seed.deployAt}}"+ evidence:+ - {call_id: c02, claim: "the endpoint is not marked slow (offsetMs 0), so the 1000ms alert applies as is"}+ - {call_id: c03, claim: "p95 jumped above a second in the minutes right after the v5.1.0 restart and was back at ~20ms about ten minutes later"}+ - {call_id: c04, claim: "in the hour before the release p95 was about 20ms"}+ - {call_id: c05, claim: "on v5.1.0, warm, p95 is again about 20ms: no regression"}+ - {call_id: c07, claim: "a slow request right after the restart ran the full category listing query (products JOIN product_prices JOIN inventory), over a second and nearly all of its time"}+ - {call_id: c08, claim: "a warm request is served from the in-process cache in about 15ms, with no database span"}+ - {call_id: c09, claim: "the v5.0.4 restart 30 hours earlier produced the same short spike"}+ ruled_out:+ - {checklist_item: none_found, call_id: c03, reason: "users did get second-long listings for several minutes after the release"}+ - {checklist_item: cache_stampede_thundering_herd, call_id: c09, reason: "the spikes follow restarts only, not cache expiry; refreshes run off the request path"}+ fix: Gate readiness on the cache preload (have /healthz answer 503 until Preload has filled every category, or preload before ListenAndServe), and warm the pgx pool (pool_min_conns) so a restarted instance takes traffic only when it is warm.+ confidence: high+ report: |+ The alert was a warmup spike from the v5.1.0 release, not a regression. GET /api/categories/:slug/products is not marked slow (offsetMs 0), so the 1000ms rule applied. v5.1.0 restarted all three catalog instances about five minutes before the alert. A restarted instance starts with an empty in-process category cache, which a background preload fills one category at a time, and an empty pgx pool; meanwhile an instance's first request for a category runs the full listing query (one to two seconds) and the first requests open connections. p95 went over a second right after the restart, was back at ~20ms about ten minutes later and has stayed there; warm v5.1.0 is as fast as v5.0.4 was, and the v5.0.4 restart 30 hours earlier shows the same spike. Nothing needs reverting. To stop it recurring on every release, keep /healthz failing until the preload has filled the cache (or preload before serving) and set pool_min_conns so connections are open before traffic arrives.diff --git a/cli/test/evals/cases/perf/perf-ctl-healthy.yaml b/cli/test/evals/cases/perf/perf-ctl-healthy.yamlnew file mode 100644index 00000000..cf8fb075--- /dev/null+++ b/cli/test/evals/cases/perf/perf-ctl-healthy.yaml@@ -0,0 +1,79 @@+id: perf-ctl-healthy+suite: perf+title: Search "feels slow" but the endpoint is healthy and stable+tags: [control, node, healthy]+seed: perf/ctl-healthy+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The search query matches with ILIKE '%term%' on name and brand, which reads like it needs a trigram index, and POST /api/checkout is far slower than search. But the ILIKE query takes about 15ms on this catalog, GET /api/search has p50 about 25ms and p95 about 45ms with no errors, the same as a week ago, and CPU is flat; checkout's time is its payment call and unrelated.+workspace: {repo: storefront-api-node}+prompt: |+ /traceway perf GET /api/search+ A couple of people on the team say search feels slow lately. Is something wrong with it?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 10000, timeout: 10m}+user:+ - default: "That's all I've heard. Please check the data."+expect:+ answer:+ checklist_item: none_found+ not: [missing_or_unused_index, select_n_plus_1, over_fetching, cpu_saturation, slow_third_party_upstream_api]+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.chart}]+ - [{op: endpoints.list, args: {from: present}}]+ - [{op: endpoints.list, args: {since: "~^([2-9]|[1-9][0-9])d$"}}]+ forbidden:+ - {op: "*", write: true}+ - {op: edit_file}+ - {op: write_file}+ max_failed_calls: 4+ max_identical_repeats: 2+ report:+ contains:+ - "p95|95th"+ - "[1-9][0-9]\\s*ms"+judge:+ expected: Nothing is wrong with GET /api/search. Over the last day p50 is about 25ms, p95 about 45ms and p99 about 60ms, with no errors; the week before looks the same, there is no step at the 1.43.0 release five days ago, and CPU is flat. The slowest searches spend their time in the ILIKE product query (tens of milliseconds) and the facet count. The answer is none_found; if the team still perceives slowness, it is outside this endpoint (the browser, the network, or another page), and the next step is to measure there rather than to change the query.+ rubric:+ - Reports the actual percentiles (p50/p95) and that they are stable over the week, with no errors and no saturation.+ - Concludes there is no backend problem with search instead of inventing a bottleneck.+ - Does not propose or make a code or index change for a healthy endpoint; at most suggests where to measure the perceived slowness.+oracle:+ - shell: >-+ traceway endpoints list --search /api/search --since 24h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round), impactReason}]'+ - shell: >-+ traceway endpoints list --search /api/search --since 7d --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 7d --interval-minutes 1440 --output json+ | jq '{endpoints, search: [.series[] | select(.endpoint == "GET /api/search") | {t: .timestamp, p95_ms: (.value|round)}]}'+ - shell: >-+ traceway logs query --since 24h --search /api/search --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.severityText) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, duration_ms: (.endpoint.duration/1e6), spans: [.spans[] | {name, ms: (.duration/1e6), rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - read_file: src/search.js+ - submit:+ checklist_item: none_found+ category: none+ locus: {endpoint: "GET /api/search", service: storefront-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "over the last day GET /api/search has p50 about 25ms, p95 about 45ms, p99 about 60ms"}+ - {call_id: c02, claim: "the week's percentiles are the same: nothing got slower lately"}+ - {call_id: c03, claim: "daily p95 stays in the mid-40s of milliseconds across the week, with no step up"}+ - {call_id: c04, claim: "search requests all answer 200 in tens of milliseconds"}+ - {call_id: c05, claim: "the slowest search of the day is the ILIKE product query plus the facet count, well under a tenth of a second"}+ - {call_id: c06, claim: "CPU is flat and low all day"}+ ruled_out:+ - {checklist_item: missing_or_unused_index, call_id: c05, reason: "even in the slowest search of the day the ILIKE query takes tens of milliseconds on this catalog"}+ - {checklist_item: cpu_saturation, call_id: c06, reason: "CPU stays low and flat"}+ fix: No change needed for GET /api/search. If the team still sees slow search, measure on the client side (browser timing, network) or check the page around it.+ confidence: high+ report: |+ Nothing is wrong with GET /api/search on the backend. Over the last day it has p50 about 25ms, p95 about 45ms and p99 about 60ms, with no errors, and the week looks the same: daily p95 is flat with no step, and CPU is low and flat. The slowest search of the day spent its time in the ILIKE product query and the facet count, tens of milliseconds in total. So there is no regression to fix here. If people still find search slow, the time is going somewhere else (the browser, the network, or the page around it), and that is where to measure next; I would not change the query or add an index for an endpoint answering in under 50ms.diff --git a/cli/test/evals/cases/perf/perf-ctl-marked-slow-ok.yaml b/cli/test/evals/cases/perf/perf-ctl-marked-slow-ok.yamlnew file mode 100644index 00000000..5571f28d--- /dev/null+++ b/cli/test/evals/cases/perf/perf-ctl-marked-slow-ok.yaml@@ -0,0 +1,79 @@+id: perf-ctl-marked-slow-ok+suite: perf+title: Ledger export is the slowest endpoint but sits inside its marked-slow allowance+tags: [smoke, control, go, marked-slow]+seed: perf/ctl-marked-slow-ok+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The export is the slowest endpoint by far (p95 about 1.7-1.8s) and its waterfall is one large range query plus a CSV encode, which reads like an expensive query or an unbounded result set; but an operator marked it slow with a 2000ms allowance, p95 and even p99 stay under it, and it has held that level all month.+workspace: {repo: ledger-api-go}+prompt: |+ /traceway perf GET /api/reports/export+ Someone in finance says the ledger export takes almost two seconds and asked me whether something is wrong with it. Is it slower than it should be?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that. Please check and tell me."+expect:+ answer:+ checklist_item: none_found+ not: [expensive_join_or_aggregation, no_pagination_unbounded_result_set, over_fetching, large_request_response_payloads, cpu_bound_serialization]+ grounding:+ - "Up to 2s is fine"+ calls:+ required:+ - {op: endpoints.list}+ - {op: endpoints.slow, args: {endpoint: "GET /api/reports/export"}}+ any_of:+ - [{op: endpoints.chart}]+ - [{op: endpoints.list, args: {from: present}}]+ - [{op: endpoints.list, args: {since: "~^([2-9]|[1-9][0-9])d$"}}]+ forbidden:+ - {op: "*", write: true}+ - {op: edit_file}+ - {op: write_file}+ max_failed_calls: 4+ max_identical_repeats: 2+ report:+ contains:+ - "1[.,][5-9][0-9]*\\s*s|1,?[5-9][0-9]{2}\\s*ms"+ - "2(\\.0+)?\\s*s|2,?000\\s*ms"+ - "allowance|offset|marked (as )?slow|accepted"+judge:+ expected: Nothing is wrong. GET /api/reports/export has p95 about 1.7-1.8s (p50 about 1.1-1.2s, p99 under 1.9s, no request over 2s), but an operator marked it slow with a 2000ms allowance ("Full ledger CSV export for the finance team ... Up to 2s is fine"), so it is inside its accepted baseline. The p95 has been flat for the whole month (no deploy, no step), CPU is flat, and the time goes to the one range query and the CSV encode, both proportional to the rows exported. Answer none_found; no fix is needed.+ rubric:+ - States the p95 together with the 2000ms allowance and concludes it is within the accepted baseline (endpoints slow was checked).+ - Shows the latency has not regressed over time (flat weekly p95, no deploy) rather than judging one window.+ - Does not invent a fix or change code; at most mentions what would help if finance wants it faster.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round), impact, impactReason}]'+ - op: endpoints.slow+ args: {endpoint: "GET /api/reports/export"}+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 30d --interval-minutes 10080 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/reports/export") | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: >-+ traceway logs query --since 24h --search /api/reports/export --page-size 50 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, duration_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: (.duration/1e6|round), rows: .attributes["db.response.returned_rows"]}]}]'+ - read_file: internal/reports/export.go+ - submit:+ checklist_item: none_found+ category: none+ locus: {endpoint: "GET /api/reports/export", service: ledger-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/reports/export is the slowest endpoint: p50 about 1.1-1.2s, p95 about 1.7-1.8s, p99 under 1.9s over the last day"}+ - {call_id: c02, claim: "an operator marked it slow with offsetMs 2000 (Full ledger CSV export for the finance team; up to 2s is fine)"}+ - {call_id: c03, claim: "weekly p95 has stayed around 1.8s for the whole month: no step, no regression"}+ - {call_id: c05, claim: "the slowest export of the day spends its time in the one range query and the CSV encode, both proportional to the 50k+ rows it returned"}+ ruled_out:+ - {checklist_item: expensive_join_or_aggregation, call_id: c05, reason: "a single range query streamed in posting order; its cost tracks the rows exported and stays inside the allowance"}+ - {checklist_item: no_pagination_unbounded_result_set, call_id: c03, reason: "the range is one quarter by design and the latency has not grown over the month"}+ fix: No change needed; the export is inside its 2000ms accepted baseline.+ confidence: high+ report: |+ Nothing is wrong with GET /api/reports/export. Its p95 over the last day is about 1.7-1.8s (p50 about 1.1-1.2s, p99 under 1.9s), and the endpoint is marked slow with a 2000ms allowance ("Full ledger CSV export for the finance team ... Up to 2s is fine"), so it sits inside its accepted baseline. The weekly p95 has been flat around 1.8s for the whole month, with no step. The time goes to the one range query over the quarter's ledger entries and the CSV encode, both proportional to the rows exported (up to ~54k for the largest accounts). No fix is needed; "almost two seconds" is the expected cost of a full quarter export.diff --git a/cli/test/evals/cases/perf/perf-ctl-marked-slow-regressed.yaml b/cli/test/evals/cases/perf/perf-ctl-marked-slow-regressed.yamlnew file mode 100644index 00000000..ec05b6e3--- /dev/null+++ b/cli/test/evals/cases/perf/perf-ctl-marked-slow-regressed.yaml@@ -0,0 +1,88 @@+id: perf-ctl-marked-slow-regressed+suite: perf+title: Marked-slow invoice export regressed past its allowance after a release+tags: [control, go, database, marked-slow, regression]+seed: perf/ctl-marked-slow-regressed+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The export is marked slow with a 2000ms allowance, so its high latency can be waved through as accepted; and its big invoices/lines JOIN (about a second) looks like an expensive query. But the JOIN did not change; p95 sat near 1.5s until v4.8.0, which added one invoice_tax_lines query per invoice and took p95 to about 4s, twice the allowance.+workspace: {repo: invoicing-api-go}+prompt: |+ /traceway perf GET /api/exports/invoices+ Accounting says the monthly invoice export got a lot slower recently; they are seeing four seconds or more. Can you find out why and how to fix it?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 14000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: select_n_plus_1+ not: [none_found, expensive_join_or_aggregation, missing_or_unused_index, no_pagination_unbounded_result_set]+ locus: {endpoint: "GET /api/exports/invoices", span_contains: "invoice_tax_lines"}+ onset: {at: "{{seed.deployAt}}", tolerance_minutes: 60}+ grounding:+ - "FROM invoice_tax_lines WHERE invoice_id"+ - "Up to 2s is expected"+ calls:+ required:+ - {op: endpoints.slow, args: {endpoint: "GET /api/exports/invoices"}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+ report:+ contains:+ - "allowance|offset|marked (as )?slow"+ - "2(\\.0+)?\\s*s|2,?000\\s*ms"+judge:+ expected: Select N+1 introduced by v4.8.0. The export is marked slow with a 2000ms allowance and its p95 sat around 1.5s, inside it, until the v4.8.0 deploy about 26 hours ago; since then p95 is about 4s (p50 about 2.9s), double the allowance. Slow requests after the deploy run one "SELECT rate_code, rate_bps, amount_cents FROM invoice_tax_lines WHERE invoice_id = $1" per invoice (60-125 per export, about 20ms each) after the unchanged invoices/lines JOIN; requests before the deploy have no tax-line queries. The fix loads every invoice's tax lines in one query (WHERE invoice_id = ANY($1) or a JOIN) in the export handler.+ rubric:+ - Checks the marked-slow allowance and states that the latency (p95 about 4s) is past the 2000ms allowance, so it is a real regression, not the accepted baseline.+ - Dates the onset to the v4.8.0 deploy (endpoints chart or window bisection plus git history) and contrasts a request from before the deploy with one after.+ - Attributes the extra time to one tax-line query per invoice and proposes a batched load, leaving the existing JOIN alone.+oracle:+ - shell: >-+ traceway endpoints list --search /api/exports/invoices --since 24h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round), impact, impactReason}]'+ - op: endpoints.slow+ args: {endpoint: "GET /api/exports/invoices"}+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 7d --interval-minutes 360 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/exports/invoices") | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 48h --interval-minutes 60 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/exports/invoices") | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: >-+ traceway logs query --since 24h --search /api/exports/invoices --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body) \(.resourceAttributes["service.version"])"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, version: .endpoint.appVersion, duration_ms: (.endpoint.duration/1e6|round),+ spans: ([.spans[] | {q: (.attributes["db.query.text"] // .name), ms: (.duration/1e6)}] | group_by(.q) | map({query: .[0].q, count: length, total_ms: (map(.ms) | add | round)}))}]'+ - shell: >-+ traceway traces show {{seed.beforeRequest.traceId}} --recorded-at {{seed.beforeRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, version: .endpoint.appVersion, duration_ms: (.endpoint.duration/1e6|round),+ spans: ([.spans[] | {q: (.attributes["db.query.text"] // .name), ms: (.duration/1e6)}] | group_by(.q) | map({query: .[0].q, count: length, total_ms: (map(.ms) | add | round)}))}]'+ - shell: git log --format='%h %ad %an %s' --date=iso -5+ - read_file: internal/exports/invoices.go+ - submit:+ checklist_item: select_n_plus_1+ category: database+ locus: {endpoint: "GET /api/exports/invoices", span: "SELECT rate_code, rate_bps, amount_cents FROM invoice_tax_lines WHERE invoice_id = $1", service: invoicing-api}+ onset: "{{seed.deployAt}}"+ evidence:+ - {call_id: c01, claim: "over the last day the export's p95 is around 4s and p50 near 3s"}+ - {call_id: c02, claim: "the endpoint is marked slow with offsetMs 2000 (Monthly invoice CSV for accounting ... Up to 2s is expected), so p95 is double the allowance"}+ - {call_id: c04, claim: "hourly p95 sits around 1.5s, below the 2s allowance, until about 26 hours ago, then steps well past it (mostly 3-4s) within one hour"}+ - {call_id: c06, claim: "a slow export on 4.8.0 runs one invoice_tax_lines query per invoice, over a hundred per request, which add seconds after the unchanged JOIN"}+ - {call_id: c07, claim: "an export on 4.7.3 runs only the invoices/lines JOIN and the CSV encode"}+ - {call_id: c08, claim: "v4.8.0 (tax breakdown per invoice in the monthly export) was committed right before the step"}+ ruled_out:+ - {checklist_item: none_found, call_id: c02, reason: "p95 is far past the 2000ms allowance, so this is not the accepted baseline"}+ - {checklist_item: expensive_join_or_aggregation, call_id: c07, reason: "the JOIN is the same before and after the deploy and fits inside the allowance"}+ fix: Load the tax lines of all exported invoices in one query (WHERE invoice_id = ANY($1)) instead of one query per invoice in the export handler.+ confidence: high+ report: |+ GET /api/exports/invoices regressed past its allowance. It is marked slow with a 2000ms offset ("Monthly invoice CSV for accounting ... Up to 2s is expected") and sat around 1.5s at p95 until the v4.8.0 deploy about 26 hours ago; since then p95 is about 4s, twice the allowance. v4.8.0 added the tax breakdown, and the export now runs one `SELECT ... FROM invoice_tax_lines WHERE invoice_id = $1` per invoice (60-125 per export, about 20ms each) after the unchanged invoices/lines JOIN: a Select N+1. Requests from before the deploy have no tax-line queries. The fix is to load the tax lines of all exported invoices with one `WHERE invoice_id = ANY($1)` query (a batched store method instead of the per-invoice TaxLines loop in the Invoices handler, internal/exports/invoices.go), which brings it back to one extra statement per export.diff --git a/cli/test/evals/cases/perf/perf-ctl-metric-p95-trap.yaml b/cli/test/evals/cases/perf/perf-ctl-metric-p95-trap.yamlnew file mode 100644index 00000000..cc564f74--- /dev/null+++ b/cli/test/evals/cases/perf/perf-ctl-metric-p95-trap.yaml@@ -0,0 +1,100 @@+id: perf-ctl-metric-p95-trap+suite: perf+title: Afternoon slowdown where the relayed "p95 CPU" is really an average+tags: [control, go, infrastructure, signal-hygiene]+seed: perf/ctl-metric-p95-trap+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: Ops' "p95 CPU around 45%", which is what metrics query returns for --aggregation p95 over a day. The server has no quantile aggregation for metric points and silently averages, so that number is the daily mean. The max per bucket shows both query hosts pegged at 97-100% busy every afternoon while traffic more than triples, and every endpoint (not one span) slows down in the same hours.+workspace: {repo: query-api-go}+prompt: |+ /traceway perf+ Search gets sluggish every afternoon. Ops pulled the p95 CPU for the query hosts and says it is only around 45%, so they don't think it's the hosts. Can you check the CPU p95 yourself and figure out what is actually going on?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know how ops got the number; they just told me the p95 CPU. Please look at the data."+expect:+ answer:+ checklist_item: cpu_saturation+ not: [none_found, inefficient_algorithm_o_n2_hot_path, missing_or_unused_index, connection_pool_exhaustion, slow_third_party_upstream_api]+ grounding:+ - "system.cpu.utilization"+ calls:+ required:+ - {op: "endpoints.*"}+ any_of:+ - [{op: metrics.query, args: {name: "~system\\.cpu", aggregation: "max"}}]+ - [{op: metrics.query, args: {name: "~system\\.cpu", aggregation: "min"}}]+ - [{op: metrics.query, args: {name: "~system\\.cpu", aggregation: "avg", interval_minutes: {max: 60}}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+ report:+ contains:+ - "average|\\bavg\\b|\\bmean\\b"+ - "percentile|quantile|p95"+ - "(9[5-9]|100)(\\.[0-9]+)?\\s*%|0\\.9[5-9]"+judge:+ expected: >-+ CPU saturation. metrics query has no percentile aggregation for metric points: --aggregation p95 (p50/p99) is+ silently computed as avg on the CLI (and rejected outright over MCP), so ops' "p95 CPU around 45%" is the daily+ average busy CPU. With --aggregation max (grouped by state, user+system, or 1 - min idle) per 30-60 minute+ bucket, both query-api hosts sit at 97-100% busy every afternoon (about 13:00-16:30 UTC) while traffic on every+ endpoint more than triples (about four times the morning's rate). In the same hours the p95 of GET /api/query goes+ from about 100ms to about 340ms and the other endpoints slow down too; a slow afternoon request has no single long+ pole, its in-process spans (rank results, render highlights) are stretched the most and the database query less.+ Fix: add capacity (more instances or bigger ones, CPU-based autoscaling; the deployment is two 2-vCPU instances+ with autoscaling off), then look at trimming per-request CPU in ranking.+ rubric:+ - Explains that metrics query --aggregation p95 is not a percentile (the server averages), so the relayed 45% is the mean, and reports the CPU honestly with max (or the bucketed series) instead.+ - Ties the afternoon slowdown to the hosts pegged at ~98% CPU while traffic more than triples, with every endpoint slower and no single span to blame.+ - Recommends more capacity (scale out or up, autoscaling) rather than a code fix for one endpoint.+oracle:+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation p95 --group-by state --since 24h --interval-minutes 60 --output json+ | jq '.results[0].series as $s | {busy_pct_24h: ((([$s.user[].value] | add) + ([$s.system[].value] | add)) / ($s.user | length) * 100 | round),+ hourly_busy_pct: [range(0; ($s.user | length)) | (($s.user[.].value + $s.system[.].value) * 100 | round)]}'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by state --since 24h --interval-minutes 60 --output json+ | jq '.results[0].series as $s | {busy_pct_24h: ((([$s.user[].value] | add) + ([$s.system[].value] | add)) / ($s.user | length) * 100 | round),+ hourly_busy_pct: [range(0; ($s.user | length)) | (($s.user[.].value + $s.system[.].value) * 100 | round)]}'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 30 --output json+ | jq '.results[0].series as $s | [range(0; ($s.user | length)) | {t: $s.user[.].timestamp, busy_pct: ((($s.user[.].value + $s.system[.].value) * 100) | round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 48h --interval-minutes 60 --output json+ | jq '{endpoints, series: [.series[] | select(.endpoint != "Other") | {t: .timestamp, endpoint, p95_ms: (.value|round)}]}'+ - shell: >-+ traceway endpoints list --from {{seed.peakFrom}} --to {{seed.peakTo}} --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --from {{seed.quietFrom}} --to {{seed.quietTo}} --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway traces show {{seed.peakRequest.traceId}} --recorded-at {{seed.peakRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, duration_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: (.duration/1e6|round)}]}]'+ - shell: >-+ traceway traces show {{seed.quietRequest.traceId}} --recorded-at {{seed.quietRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, duration_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: (.duration/1e6|round)}]}]'+ - read_file: deploy/query-api.yaml+ - submit:+ checklist_item: cpu_saturation+ category: infrastructure+ locus: {service: query-api, host: "query-api-1, query-api-2"}+ onset: null+ evidence:+ - {call_id: c02, claim: "averaged over the last 24 hours, busy CPU (user plus system) is about 45%: the number ops called p95; the p95 aggregation returns the same averages on the CLI"}+ - {call_id: c03, claim: "with max per 30-minute bucket, busy CPU is pinned at 97-100% through the afternoon (13:00-16:30 UTC)"}+ - {call_id: c04, claim: "every top endpoint's p95 rises in the same afternoon hours"}+ - {call_id: c05, claim: "during the peak the request rate is about four times the morning's and GET /api/query p95 is about 340ms"}+ - {call_id: c06, claim: "in the quiet morning GET /api/query p95 is about 100ms"}+ - {call_id: c07, claim: "a peak request has no single long pole: most of its ~320ms is in-process ranking and highlighting, spread over several spans"}+ - {call_id: c09, claim: "two 2-vCPU instances with autoscaling disabled"}+ ruled_out:+ - {checklist_item: none_found, call_id: c03, reason: "the 45% is a daily average; the hosts are pegged in the afternoon"}+ - {checklist_item: inefficient_algorithm_o_n2_hot_path, call_id: c08, reason: "the same endpoint runs the same spans in about 90ms in the quiet morning; only the load changes"}+ fix: Add capacity for the afternoon peak (more instances or larger ones, CPU-based autoscaling), then reduce per-request CPU in ranking if needed.+ confidence: high+ report: |+ The afternoon slowdown is CPU saturation on the query hosts. The "p95 CPU around 45%" is not a percentile: metrics query has no quantile aggregation for metric points, so --aggregation p95 silently returns the average (over MCP it is refused). Over a day the average busy CPU is about 45%, but with max per 30-minute bucket the query-api hosts are pinned at 97-100% busy every afternoon (about 13:00-16:30 UTC), when the request rate is about four times the morning's. In those hours GET /api/query's p95 goes from about 100ms to about 340ms and every other endpoint slows down too; a slow afternoon request has no single long pole, most of its time is in-process ranking and highlighting, which take a fraction of that in the quiet morning. The service runs on two 2-vCPU instances with autoscaling off. Fix: add capacity for the peak (more or larger instances, CPU-based autoscaling); trimming per-request CPU in ranking would help after that.diff --git a/cli/test/evals/cases/perf/perf-ctl-onset-bisect.yaml b/cli/test/evals/cases/perf/perf-ctl-onset-bisect.yamlnew file mode 100644index 00000000..1c7853db--- /dev/null+++ b/cli/test/evals/cases/perf/perf-ctl-onset-bisect.yaml@@ -0,0 +1,94 @@+id: perf-ctl-onset-bisect+suite: perf+title: Account page slowed two days ago; the route is not in the chart's top five+tags: [control, python, onset, database]+seed: perf/ctl-onset-bisect+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: Data growth. The orders count query on the same request has crept up all through the window, so a coarse first-days-versus-now comparison reads like gradual growth; but the p95 stepped from about 25ms to about 450ms inside one hour, right at the 2.32.0 deploy, when a new monthly-spend aggregate (orders JOIN order_items GROUP BY month) appeared in every request.+workspace: {repo: members-api-py}+prompt: |+ /traceway perf GET /api/account/summary+ The account page in the members app has felt slower for a few days. When did it start, and what changed?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know exactly; a few days ago. Please find out from the data."+expect:+ answer:+ checklist_item: expensive_join_or_aggregation+ not: [select_n_plus_1, missing_or_unused_index, no_pagination_unbounded_result_set, cpu_saturation, none_found]+ locus: {endpoint: "GET /api/account/summary"}+ onset: {at: "{{seed.deployAt}}", tolerance_minutes: 60}+ mentions_all: ["date_trunc|group(ed)? by month|monthly spend|spend_cents|spend per month"]+ grounding:+ - "date_trunc"+ calls:+ required:+ - {op: endpoints.list, args: {from: present}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: >-+ The account summary regressed at the 2.32.0 deploy about 52 hours ago. Before it the route's p95 was about 20-25ms (creeping up slowly with the orders table); within one hour of the deploy it stepped to about 450ms. The route is not in the top five of endpoints chart (imports, reports, avatar uploads, recommendations, search and login are slower), so the onset comes from comparing endpoints list --search windows. Slow requests carry a new aggregate, SELECT date_trunc('month', ...) ... FROM orders JOIN order_items ... GROUP BY month, taking 200-600ms; it is the monthly spend chart added in 2.32.0. The indexes on orders(account_id, created_at) and order_items(order_id) exist; the cost is aggregating the account's whole history on every page view. Fix: precompute the monthly totals (a rollup maintained on order writes, or a materialized view), cache them per account, or bound the range.+ rubric:+ - Dates the onset to within an hour of the 2.32.0 deploy by comparing the route's percentiles across narrowing --from/--to windows (the chart does not show it).+ - Rejects gradual data growth as the explanation (the step is abrupt; the slow creep before it is tens of milliseconds).+ - Names the new monthly-spend aggregate as the long pole and proposes precomputing or caching it rather than an index.+oracle:+ - shell: >-+ traceway endpoints list --search /api/account/summary --since 24h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: traceway endpoints chart --metric-type p95 --since 7d --interval-minutes 360 --output json | jq '.endpoints'+ - shell: >-+ traceway endpoints list --search /api/account/summary --from {{seed.ago240h}} --to {{seed.ago120h}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --search /api/account/summary --from {{seed.ago120h}} --to {{seed.ago60h}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --search /api/account/summary --from {{seed.ago60h}} --to {{seed.ago48h}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --search /api/account/summary --from {{seed.ago54h}} --to {{seed.ago51h}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --search /api/account/summary --from {{seed.ago54h}} --to {{seed.ago52h30m}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints list --search /api/account/summary --from {{seed.ago52h30m}} --to {{seed.ago51h}} --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway logs query --since 24h --search /api/account/summary --page-size 10 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body) \(.resourceAttributes["service.version"])"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, version: .endpoint.appVersion, duration_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, ms: (.duration/1e6|round), query: .attributes["db.query.text"]}]}]'+ - shell: git log --format='%h %ad %an %s' --date=iso -5+ - submit:+ checklist_item: expensive_join_or_aggregation+ category: database+ locus: {endpoint: "GET /api/account/summary", span: "SELECT date_trunc(...) AS month, count(DISTINCT orders.id), sum(order_items.qty * order_items.unit_price_cents) FROM orders JOIN order_items ... GROUP BY month", service: members-api}+ onset: "{{seed.deployAt}}"+ evidence:+ - {call_id: c01, claim: "over the last day the account summary has p95 about 450ms"}+ - {call_id: c02, claim: "the route is not among the top five endpoints of the p95 chart, so its onset has to be bisected"}+ - {call_id: c03, claim: "five to ten days ago its p95 was about 20ms"}+ - {call_id: c04, claim: "from 120h to 60h ago the p95 is still about 20-25ms"}+ - {call_id: c05, claim: "the 60h-48h window already has the slow p95"}+ - {call_id: c07, claim: "54h-52h30m ago the p95 is still fast"}+ - {call_id: c08, claim: "52h30m-51h ago the p95 is slow: the step is inside that bracket"}+ - {call_id: c10, claim: "a slow request on 2.32.0 spends most of its time in a new monthly-spend aggregate over orders JOIN order_items GROUP BY month"}+ - {call_id: c11, claim: "2.32.0 (monthly spend chart on the account page) was committed just before the step"}+ ruled_out:+ - {checklist_item: no_pagination_unbounded_result_set, call_id: c04, reason: "the slow creep before the deploy is a few milliseconds; the step is abrupt"}+ - {checklist_item: missing_or_unused_index, call_id: c10, reason: "the other orders queries on the same request (the count and the latest five) take milliseconds; the time is the new aggregate grouping the account's whole order history"}+ fix: Stop aggregating the account's whole order history on every page view. Maintain monthly totals in a rollup table (updated on order writes) or a materialized view, or cache the result per account; at minimum bound the range to the last 12 months.+ confidence: high+ report: |+ The account page slowed at the 2.32.0 deploy, about 52 hours ago (between 52h30m and 51h ago by window bisection). Before it GET /api/account/summary had p95 around 20-25ms; right after it the p95 is about 450ms. The route is not in endpoints chart's top five, so I bisected endpoints list --search over narrowing --from/--to windows. Slow requests on 2.32.0 run a new aggregate, `SELECT date_trunc('month', orders.created_at) ... FROM orders JOIN order_items ... GROUP BY month`, for the monthly spend chart added in that release; it takes a few hundred milliseconds on every page view because it aggregates the account's whole history. The orders count query has crept up a little with data growth, but that is a few milliseconds and not the regression. Fix: precompute the monthly totals (rollup table or materialized view, refreshed on order writes) or cache them per account, and bound the range.diff --git a/cli/test/evals/cases/perf/perf-db-expensive-join.yaml b/cli/test/evals/cases/perf/perf-db-expensive-join.yamlnew file mode 100644index 00000000..a2bc06e0--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-expensive-join.yaml@@ -0,0 +1,76 @@+id: perf-db-expensive-join+suite: perf+title: The revenue report takes two seconds every time, busy or quiet+tags: [database, php, laravel]+seed: perf/db-expensive-join+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: Saturation (ops wants more PHP-FPM workers). CPU follows the daily traffic gently and stays under 40%, every other endpoint answers in about 15ms or less at the same moments, and the report takes ~1.8s at the quietest hour exactly as at the peak.+workspace: {repo: merchant-api-php}+prompt: |+ /traceway perf GET /api/stores/{store}/reports/revenue+ The revenue chart on the merchant dashboard takes about two seconds to load, every single time, morning or night. Ops thinks the API servers are overloaded and wants to add more PHP-FPM workers. Here is one of the slow requests: {{env.url}}/endpoints/GET%20%2Fapi%2Fstores%2F%7Bstore%7D%2Freports%2Frevenue/{{seed.slowRequest.id}}?t={{seed.slowRequest.recordedAt}}+ Would more workers fix it?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: expensive_join_or_aggregation+ not: [cpu_saturation, undersized_noisy_neighbor_instance, missing_or_unused_index, select_n_plus_1, no_pagination_unbounded_result_set, missing_query_result_caching, connection_pool_exhaustion]+ locus: {endpoint: "GET /api/stores/{store}/reports/revenue", span_contains: "orders"}+ grounding:+ - "inner join order_items on order_items.order_id = orders.id"+ calls:+ required:+ - {op: endpoints.show, args: {id: "{{seed.slowRequest.id}}", recorded_at: present}}+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.chart}]+ - [{op: metrics.query, args: {name: "~system\\.cpu"}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Expensive JOIN/aggregation. The report runs one query that joins 90 days of the store's orders to order_items and products and groups by day and category (count distinct buyers, sum of revenue); it takes ~1.8s in every request, p50 ~1.84s and p95 ~1.98s, at night and at the daily peak alike, and has for the whole two weeks. The other queries in the request take about a millisecond. CPU follows the daily traffic gently and stays under 40%, other endpoints stay at about 15ms or less, so more PHP-FPM workers would not help. Fix by pre-aggregating (a store/day/category rollup table refreshed by a scheduled job, or a summary table updated as orders are paid) and reading that, or by caching the computed report per store per day.+ rubric:+ - Identifies the single aggregate query (orders joined to order_items and products, grouped by day and category) as ~all of the request's time, and shows it is the same at low and high traffic.+ - Rules out saturation with data (CPU well below capacity, other endpoints fast at the same time, latency flat across the day) and answers the workers question.+ - Proposes precomputing the aggregate (rollup or summary table, materialized result) or caching it per store and day, rather than scaling the servers.+oracle:+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, path: .endpoint.attributes["url.path"], ms: (.endpoint.duration / 1e6),+ spans: [.spans[] | {name, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}'+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6)}]'+ - shell: >-+ traceway endpoints chart --metric-type p50 --since 2d --interval-minutes 120 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/stores/{store}/reports/revenue" or .endpoint == "GET /api/stores/{store}/orders") | {endpoint, t: .timestamp, p50_ms: .value}]'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 2d --interval-minutes 120 --output json+ | jq '.results[0].series | {user: [.user[] | {t: .timestamp, v: (.value * 1000 | round / 1000)}], idle: [.idle[] | {t: .timestamp, v: (.value * 1000 | round / 1000)}]}'+ - shell: git log --format='%h %ad %s' --date=iso -5+ - read_file: app/Http/Controllers/RevenueReportController.php+ - read_file: database/migrations/2025_01_14_101500_create_orders_tables.php+ - submit:+ checklist_item: expensive_join_or_aggregation+ category: database+ locus: {endpoint: "GET /api/stores/{store}/reports/revenue", span: "SELECT orders (orders join order_items join products, group by day, category_id)", service: merchant-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "the request is ~2.0s, almost all of it one query joining orders, order_items and products and grouping by day and category; the token and store lookups take about a millisecond each"}+ - {call_id: c02, claim: "p50 and p95 are both ~1.8-2.0s, so every request pays it; every other endpoint answers in about 15ms or less"}+ - {call_id: c03, claim: "the report's p50 is flat at ~1.8s in every two-hour bucket, night and peak, while the order list stays at ~13ms"}+ - {call_id: c04, claim: "CPU moves with the daily traffic but stays under 40% busy (user at most ~27%)"}+ - {call_id: c06, claim: "the controller aggregates 90 days of raw orders on every request"}+ ruled_out:+ - {checklist_item: cpu_saturation, call_id: c04, reason: "CPU well below capacity and latency independent of the time of day"}+ - {checklist_item: missing_or_unused_index, call_id: c07, reason: "orders is already indexed on (store_id, status, created_at), and the latency is flat (c03) rather than growing with the tables"}+ - {checklist_item: select_n_plus_1, call_id: c01, reason: "three queries per request"}+ fix: Pre-aggregate revenue into a store/day/category rollup table refreshed by a scheduled command (reports:roll-up-revenue every 10 minutes) and read that in the controller, or cache the computed report per store and day. More PHP-FPM workers would not change the per-request time.+ confidence: high+ report: |+ More workers would not help: the revenue report is slow because of one expensive aggregate query, not load. Every request runs a query that joins 90 days of the store's orders to order_items and products and groups by day and category; it takes ~1.8s every time (p50 1.8s, p95 2.0s), at night and at the daily peak alike. CPU stays under 40% and other endpoints answer in about 15ms or less at the same moments. Fix: add a store_daily_revenue rollup table, refresh it with a scheduled reports:roll-up-revenue command every 10 minutes, and have the controller read the rollup, so the request reads at most a few hundred pre-computed rows (or cache the computed report per store and day).diff --git a/cli/test/evals/cases/perf/perf-db-lock-contention.yaml b/cli/test/evals/cases/perf/perf-db-lock-contention.yamlnew file mode 100644index 00000000..d64cb461--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-lock-contention.yaml@@ -0,0 +1,75 @@+id: perf-db-lock-contention+suite: perf+title: Stock reservations stall and fail during the evening product drop+tags: [database, go, pgx]+seed: perf/db-lock-contention+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: Connection-pool exhaustion. Release 1.32.0 a week ago raised the pgx pool from 20 to 40 connections and the evening spikes look the same before and after; in the slow requests BEGIN starts within a millisecond (no wait for a connection) and the time is inside the UPDATE inventory spans.+workspace: {repo: inventory-api-go}+prompt: |+ Stock reservations (POST /api/reservations on inventory-api) get really slow every evening during the 18:00 product drop, and some carts fail with 500s or 503s. Last week we raised the pgx pool from 20 to 40 connections and it didn't change anything. Can you use Traceway to figure out what's actually going on and what we should fix in the code?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: lock_transaction_contention+ not: [connection_pool_exhaustion, cpu_saturation, missing_or_unused_index, slow_third_party_upstream_api, no_timeouts_aggressive_retries, select_n_plus_1]+ locus: {endpoint: "POST /api/reservations", span_contains: "inventory"}+ grounding:+ - "deadlock detected (SQLSTATE 40P01)"+ calls:+ required:+ - {op: endpoints.list}+ - {op: logs.query}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Lock / transaction contention. During the 18:00 drop many reservations update the same few hot inventory rows. Reserve updates each cart line in cart order inside one transaction, so concurrent carts queue on the row locks (the UPDATE inventory spans run for hundreds of ms to seconds, while BEGIN starts immediately), and carts that lock two hot SKUs in opposite orders deadlock (ERROR deadlock detected, SQLSTATE 40P01, after ~1s) or hit lock_timeout (55P03 after 5s). p50 stays ~8ms; p99 is seconds only in the drop window, every evening before and after the pool change, which therefore did not help. Fix by locking rows in a consistent order (sort the lines by warehouse and SKU, or one UPDATE ... FROM unnest for all lines), keeping the transaction short, and retrying 40P01 once.+ rubric:+ - Locates the time inside the UPDATE inventory spans (lock waits in the database) and cites the deadlock / lock timeout errors from the logs.+ - Rules out pool exhaustion with evidence (no gap before BEGIN, spikes unchanged after the pool was doubled).+ - Proposes a consistent lock order (sorted SKUs or a single statement) and short transactions, not a bigger pool.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}]'+ - shell: >-+ traceway logs query --since 24h --min-severity 13 --page-size 6 --output json+ | jq '{data: [.data[] | {timestamp, severityText, traceId, body, detail: .logAttributes.pg_detail, where: .logAttributes.pg_where}]}'+ - shell: >-+ traceway traces show {{calls.c02.data.0.traceId}} --recorded-at {{calls.c02.data.0.timestamp}} --output json+ | jq 'def secs: .[11:] | rtrimstr("Z") | split(":") | ((.[0] | tonumber) * 3600 + (.[1] | tonumber) * 60 + (.[2] | tonumber));+ [.nodes[] | .endpoint.recordedAt as $t0 | {endpoint: .endpoint.endpoint, status: .endpoint.statusCode, ms: (.endpoint.duration / 1e6 | round),+ spans: [.spans[] | {name, start_ms: (((.startTime | secs) - ($t0 | secs)) * 1000 | round), ms: (.duration / 1e6 | round), status: .statusCode, query: .attributes["db.query.text"]}]}]'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 9d --interval-minutes 360 --output json+ | jq '[.series[] | select(.endpoint == "POST /api/reservations") | {t: .timestamp, p99_ms: (.value | round)}]'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - shell: git log --format='%h %ad %s' --date=iso --stat -3+ - read_file: internal/reserve/reserve.go+ - submit:+ checklist_item: lock_transaction_contention+ category: database+ locus: {endpoint: "POST /api/reservations", span: "UPDATE inventory SET reserved = reserved + $1 ... WHERE sku = $2 AND warehouse_id = $3", service: inventory-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "POST /api/reservations has p50 ~8ms but p99 in the seconds"}+ - {call_id: c02, claim: "the drop produces ERROR deadlock detected (40P01) while updating inventory rows, with Postgres naming two processes that each wait for the other"}+ - {call_id: c03, claim: "in a failing request BEGIN starts within a millisecond, and the time is inside the UPDATE inventory statements: the one that fails waits on a row lock for about a second before Postgres reports the deadlock"}+ - {call_id: c04, claim: "p99 spikes in the evening bucket every day, before and after the pool was raised to 40"}+ - {call_id: c05, claim: "CPU is flat"}+ - {call_id: c07, claim: "Reserve updates the cart lines in cart order inside one transaction, so two carts can lock the same SKUs in opposite orders"}+ ruled_out:+ - {checklist_item: connection_pool_exhaustion, call_id: c03, reason: "no wait before BEGIN; doubling the pool (c06) left the evening spikes unchanged (c04)"}+ - {checklist_item: cpu_saturation, call_id: c05, reason: "CPU flat during the drop"}+ fix: Lock inventory rows in one global order (sort the lines by warehouse and SKU before updating, or update them all in one UPDATE ... FROM unnest statement), keep the reservation transaction short, and retry a 40P01 once. A bigger pool does not help.+ confidence: high+ report: |+ Reservations stall on row locks, not on the pool. During the 18:00 drop many carts reserve the same few SKUs. Reserve updates each cart line in cart order inside one transaction, so concurrent carts queue on the hot inventory rows: the UPDATE inventory spans run for hundreds of ms to seconds while BEGIN starts immediately, and carts that hold two hot SKUs in opposite orders deadlock (40P01, after deadlock_timeout); a statement that waits past the 5s lock_timeout (55P03) is what Reserve answers with a 503. p50 is ~8ms; p99 jumps only in the evening window, and it did the same before and after the pool was doubled. Fix: have Reserve lock the lines in one global order (sort by warehouse and SKU, or one UPDATE ... FROM unnest), so carts queue behind each other instead of deadlocking, keep the transaction short, and retry a 40P01 once; a bigger pool does not help.diff --git a/cli/test/evals/cases/perf/perf-db-missing-index.yaml b/cli/test/evals/cases/perf/perf-db-missing-index.yamlindex ee0d9892..8dc1f268 100644--- a/cli/test/evals/cases/perf/perf-db-missing-index.yaml+++ b/cli/test/evals/cases/perf/perf-db-missing-index.yaml@@ -77,4 +77,4 @@ oracle:fix: Add an index on users(email) with CREATE INDEX CONCURRENTLY (Alembic migration 0003, index=True on the model), then confirm the users span drops to a few milliseconds.confidence: highreport: |- GET /api/users/lookup is slow because users.email has no index. Each lookup runs one query, SELECT ... FROM users WHERE users.email = $1, which returns one row or none but takes about half a second: Postgres reads the whole users table every time. The query got slower every week as the table grew (p50 roughly doubled over the month) with no step at the 1.13.0 release, which only touched POST /api/invites. Primary-key reads of users and orgs take a millisecond or two and CPU is flat. I added migration 0003 that creates ix_users_email concurrently and marked the column indexed in the model.+ GET /api/users/lookup is slow because users.email has no index. Each lookup runs one query, SELECT ... FROM users WHERE users.email = $1, which returns one row or none but takes about half a second: Postgres reads the whole users table every time. The query got slower every week as the table grew (p50 roughly doubled over the month) with no step at the 1.13.0 release, which only touched POST /api/invites. Primary-key reads of users and orgs take a millisecond or two and CPU is flat. Fix: add a migration that creates an index on users.email concurrently, and mark the column indexed in the model.diff --git a/cli/test/evals/cases/perf/perf-db-missing-query-cache.yaml b/cli/test/evals/cases/perf/perf-db-missing-query-cache.yamlnew file mode 100644index 00000000..5dd20589--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-missing-query-cache.yaml@@ -0,0 +1,80 @@+id: perf-db-missing-query-cache+suite: perf+title: Every storefront request reloads the feature flags from Postgres+tags: [database, node, nestjs]+seed: perf/db-missing-query-cache+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: No cache / a low hit ratio. Product data is already behind Redis and the app's cache.hit_ratio gauge stays between about 96% and 99%, and the product spans show Redis hits; what is not cached is one specific query, the parameterless feature_flags read that a global guard runs before every handler.+workspace: {repo: storefront-api-nest}+prompt: |+ /traceway perf GET /api/me+ Every endpoint of the storefront API is slower than it should be. Even GET /api/me, which reads a single customer row, takes 20-30ms. Product data already sits behind Redis with a ~98% hit rate, so I don't think it's a caching problem. Where is the time going?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: missing_query_result_caching+ not: [no_cache_low_hit_ratio, select_n_plus_1, missing_or_unused_index, expensive_join_or_aggregation, cpu_saturation, chatty_ipc_microservice_fan_out]+ locus: {endpoint: "GET /api/me", span_contains: "feature_flags"}+ grounding:+ - "FROM feature_flags WHERE archived_at IS NULL"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, traces.show]+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Missing query/result caching. A global NestJS guard (FeatureFlagsGuard) calls FeatureFlagsService.load on every request, which runs the same parameterless query, SELECT key, enabled, rollout_percent, rules FROM feature_flags WHERE archived_at IS NULL ORDER BY key, returning the same 214 rows each time and taking ~15-30ms. It is most of GET /api/me (the customer read takes ~1.5ms) and it is in front of every endpoint, so every endpoint pays it. Redis works for product data (hit ratio 96-99%, ~98% on average; product spans are Redis hits); the flags are simply not cached. Flags only need to propagate within a minute (README), so cache them in memory with a short TTL (or refresh in the background / on change) instead of querying per request.+ rubric:+ - Identifies the feature_flags query as the dominant span of GET /api/me and shows it runs identically (same text, no parameters, same row count) on other endpoints too.+ - Rules out a low cache hit ratio (the product cache stays at 96-99% and its spans are Redis hits) and N+1 (one flags query per request).+ - Proposes caching the flag set in process with a short TTL or change-driven refresh, consistent with the one-minute propagation requirement.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6)}]'+ - shell: >-+ traceway logs query --since 8h --search /api/me --page-size 5 --output json+ | jq '{data: [.data[] | {traceId, timestamp, body}]}'+ - shell: >-+ traceway traces show {{calls.c02.data.0.traceId}} --recorded-at {{calls.c02.data.0.timestamp}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6),+ spans: [.spans[] | {name, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: >-+ traceway logs query --since 8h --search /api/categories/ --page-size 3 --output json+ | jq '{data: [.data[] | {traceId, timestamp, body}]}'+ - shell: >-+ traceway traces show {{calls.c04.data.0.traceId}} --recorded-at {{calls.c04.data.0.timestamp}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6),+ spans: [.spans[] | {name, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: traceway metrics query --name cache.hit_ratio --aggregation min --since 24h --interval-minutes 360 --output json+ - read_file: src/flags/feature-flags.guard.ts+ - read_file: src/flags/feature-flags.service.ts+ - submit:+ checklist_item: missing_query_result_caching+ category: database+ locus: {endpoint: "GET /api/me", span: "SELECT feature_flags: SELECT key, enabled, rollout_percent, rules FROM feature_flags WHERE archived_at IS NULL ORDER BY key", service: storefront-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "every endpoint has a p50 of 20ms or more, even trivial ones like GET /api/me"}+ - {call_id: c03, claim: "GET /api/me spends most of its time in the feature_flags query (214 rows); the customer read is ~1.5ms"}+ - {call_id: c05, claim: "the category listing runs the identical feature_flags query (same text, 214 rows) before its handler looks the page up in Redis"}+ - {call_id: c06, claim: "the product cache hit ratio never drops below ~96% (the lowest value in each 6-hour bucket), so the existing cache is fine"}+ - {call_id: c07, claim: "a global guard loads the flags on every request"}+ ruled_out:+ - {checklist_item: no_cache_low_hit_ratio, call_id: c06, reason: "the Redis product cache hit ratio stays above 95%; the uncached work is one specific query"}+ - {checklist_item: select_n_plus_1, call_id: c03, reason: "one flags query per request, not a loop"}+ fix: Cache the flag set in memory in FeatureFlagsService with a short TTL (30s, single-flight refresh, serve the previous copy while refreshing), which still meets the one-minute propagation the back office expects; or refresh on change via LISTEN/NOTIFY.+ confidence: high+ report: |+ The time is a query that runs on every request: FeatureFlagsGuard is a global guard, and it reloads every feature flag (SELECT key, enabled, rollout_percent, rules FROM feature_flags WHERE archived_at IS NULL ORDER BY key, 214 rows) before any handler runs. That query takes ~15-30ms and is most of GET /api/me, whose own customer read takes ~1.5ms; the category listing and the other endpoints run the identical query before their own work. Redis is fine (the product cache hit ratio stays above 95%); the flags just are not cached. Fix: have FeatureFlagsService keep the flags in memory for 30 seconds with a single-flight refresh (serving the previous copy while it refreshes), so a request reads them from memory and a flag change still reaches the API within a minute.diff --git a/cli/test/evals/cases/perf/perf-db-n-plus-1.yaml b/cli/test/evals/cases/perf/perf-db-n-plus-1.yamlindex c62ffeda..eabdcbc7 100644--- a/cli/test/evals/cases/perf/perf-db-n-plus-1.yaml+++ b/cli/test/evals/cases/perf/perf-db-n-plus-1.yaml@@ -65,4 +65,4 @@ oracle:fix: Load the items of all listed orders in one query (WHERE order_id IN (...) or a JOIN) instead of one query per order in ListOrders.confidence: highreport: |- GET /api/orders is slow because of a Select N+1 in ListOrders (internal/handlers/orders.go): after loading up to 100 orders it runs one order_items query per order. The slowest request of the last day ran dozens of those queries, which make up most of its time. CPU is flat and nothing shipped in the window; the latency grew as customers accumulated orders. I batched the items into one IN (...) query (store.ItemsByOrders).+ GET /api/orders is slow because of a Select N+1 in ListOrders (internal/handlers/orders.go): after loading up to 100 orders it runs one order_items query per order. The slowest request of the last day ran dozens of those queries, which make up most of its time. CPU is flat and nothing shipped in the window; the latency grew as customers accumulated orders. Fix: load the items of all listed orders in one query (WHERE order_id IN (...) or a JOIN) instead of one query per order.diff --git a/cli/test/evals/cases/perf/perf-db-over-fetching.yaml b/cli/test/evals/cases/perf/perf-db-over-fetching.yamlnew file mode 100644index 00000000..00092dea--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-over-fetching.yaml@@ -0,0 +1,82 @@+id: perf-db-over-fetching+suite: perf+title: The category grid got slow when products gained a large column+tags: [database, node, express]+seed: perf/db-over-fetching+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: JSON encoding (a teammate's guess). The response is 48 small product cards encoded in about 2ms after the query, and CPU rose only a few points; the time is in the SELECT * span, which returns the same 48 rows as before release 4.8.0 but now includes the large supplier_payload column. Search reads the same table and 48 rows with an explicit column list and stays fast.+workspace: {repo: catalog-api-node}+prompt: |+ /traceway perf GET /api/categories/:categoryId/products+ Category pages got slow about ten days ago and never recovered: the product grid API went from around 15ms to around 400ms, and the API's memory went up as well. The listing code hasn't changed in months. Someone on the team wants to swap res.json for a faster JSON serializer. Is that the right fix, or is something else going on?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: over_fetching+ not: [cpu_bound_serialization, large_request_response_payloads, missing_or_unused_index, no_pagination_unbounded_result_set, select_n_plus_1, expensive_join_or_aggregation, cpu_saturation]+ locus: {endpoint: "GET /api/categories/:categoryId/products", span_contains: "products"}+ grounding:+ - "SELECT * FROM products WHERE category_id = $1"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}, {op: endpoints.chart}]+ - [{op: traces.show, args: {recorded_at: present}}, {op: endpoints.chart}]+ - [{op: endpoints.show, args: {recorded_at: present}}, {op: endpoints.list, args: {from: present}}]+ - [{op: traces.show, args: {recorded_at: present}}, {op: endpoints.list, args: {from: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, traces.show]+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Over-fetching. The grid runs SELECT * FROM products ... LIMIT 48. Release 4.8.0 (about ten days ago) added products.supplier_payload, the supplier's raw record (~120KB a row), and the importer filled it over the next day and a half; the query still returns 48 rows, but its span went from ~12ms to ~400ms and Node heap and host memory rose in step. GET /api/search reads the same table and 48 rows with an explicit column list and stays at ~13ms. The response is 48 small cards; encoding takes ~2ms and CPU rose only a few points, so a faster serializer would not help. Fix by selecting only the card columns (id, name, slug, price_cents, currency, thumbnail_url) in categoryProducts (and an explicit list on the product page).+ rubric:+ - Puts the time in the SELECT * products span (same 48 rows, much wider rows) and ties the onset to the 4.8.0 release that added supplier_payload, not to a change in the listing code.+ - Rules out JSON encoding (a few ms after the query, small response, CPU only slightly up) and contrasts with the fast explicit-column search query.+ - Proposes an explicit column list for the grid (and the product page) instead of SELECT *.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6)}]'+ - shell: >-+ traceway logs query --since 6h --search /api/categories/ --page-size 5 --output json+ | jq '{data: [.data[] | {traceId, timestamp, body}]}'+ - shell: >-+ traceway traces show {{calls.c02.data.0.traceId}} --recorded-at {{calls.c02.data.0.timestamp}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, id: .endpoint.id, recordedAt: .endpoint.recordedAt, version: .endpoint.appVersion, ms: (.endpoint.duration / 1e6),+ spans: [.spans[] | {name, start: .startTime, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: >-+ traceway endpoints chart --metric-type p50 --since 14d --interval-minutes 1440 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/categories/:categoryId/products" or .endpoint == "GET /api/search") | {endpoint, t: .timestamp, p50_ms: .value}]'+ - shell: traceway metrics query --name system.memory.utilization --tag state=used --aggregation max --since 14d --interval-minutes 1440 --output json+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - shell: git log --format='%h %ad %s' --date=iso --stat -3+ - read_file: src/catalog/queries.js+ - read_file: migrations/0012_products_supplier_payload.sql+ - submit:+ checklist_item: over_fetching+ category: database+ locus: {endpoint: "GET /api/categories/:categoryId/products", span: "SELECT * FROM products WHERE category_id = $1 AND is_active = true ORDER BY sort_rank DESC LIMIT 48", service: catalog-api}+ onset: "{{seed.deployAt}}"+ evidence:+ - {call_id: c01, claim: "the grid endpoint's p50 is around 400ms while GET /api/search, which reads the same products table, is around 13ms"}+ - {call_id: c03, claim: "almost all of the request is the SELECT * FROM products span, which returns 48 rows; encoding the response after it takes about 2ms"}+ - {call_id: c04, claim: "the grid went from ~15ms to ~400ms over a day and a half starting at the 4.8.0 release, while search stayed flat"}+ - {call_id: c05, claim: "host memory rose by the same step"}+ - {call_id: c06, claim: "CPU is around 30%, not saturated, so serialization CPU is not the limit"}+ - {call_id: c09, claim: "4.8.0 added products.supplier_payload, the supplier's raw record, which SELECT * now reads for every grid row"}+ ruled_out:+ - {checklist_item: cpu_bound_serialization, call_id: c03, reason: "encoding the 48 cards takes ~2ms; the query span is the cost"}+ - {checklist_item: no_pagination_unbounded_result_set, call_id: c03, reason: "the query is LIMIT 48 and returns 48 rows, as before"}+ - {checklist_item: missing_or_unused_index, call_id: c04, reason: "search on the same table is fast and the step lines up with the new column, not with table growth"}+ fix: Select only the card columns in categoryProducts (id, name, slug, price_cents, currency, thumbnail_url) and use an explicit list on the product page, so supplier_payload is never read for listings. A faster JSON serializer would not change anything.+ confidence: high+ report: |+ The grid is slow because its query over-fetches. categoryProducts runs SELECT * FROM products ... LIMIT 48. Release 4.8.0 added products.supplier_payload (the supplier's raw record) and the importer filled it over the next day and a half; the query still returns 48 rows, but each row now carries that payload, so the span went from ~12ms to ~400ms and memory went up with it. GET /api/search reads the same table and 48 rows with an explicit column list and stays at ~13ms. Encoding the response takes about 2ms and CPU is near 30%, so swapping res.json would not help. Fix: have categoryProducts select only the card columns (id, name, slug, price_cents, currency, thumbnail_url), and give the product page an explicit column list too.diff --git a/cli/test/evals/cases/perf/perf-db-unbounded-result.yaml b/cli/test/evals/cases/perf/perf-db-unbounded-result.yamlnew file mode 100644index 00000000..a1732ca6--- /dev/null+++ b/cli/test/evals/cases/perf/perf-db-unbounded-result.yaml@@ -0,0 +1,86 @@+id: perf-db-unbounded-result+suite: perf+title: The admin audit log gets slower every week as organizations accumulate events+tags: [database, java, spring]+seed: perf/db-unbounded-result+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: A missing index on audit_events (the user's guess). The dashboard widget reads the same table through the same (org_id, created_at) index with LIMIT 20 in about 2ms, and the audit-log query's cost per returned row stays flat (~20µs) while the rows it returns grow every week.+workspace: {repo: admin-api-java}+prompt: |+ Org admins keep telling us the Audit log page in the admin console gets slower every week. It was snappy when we launched it in the spring; now it's a few hundred ms for most orgs and several seconds for the biggest ones. My guess is that audit_events needs another index, it has grown a lot. Can you check what Traceway shows for GET /api/admin/audit-log and tell me what to change?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: no_pagination_unbounded_result_set+ not: [missing_or_unused_index, over_fetching, select_n_plus_1, expensive_join_or_aggregation, large_request_response_payloads, cpu_bound_serialization, cpu_saturation]+ locus: {endpoint: "GET /api/admin/audit-log", span_contains: "audit_events"}+ grounding:+ - "e.user_agent, e.created_at FROM audit_events e WHERE e.org_id = ?"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}, {op: endpoints.chart}]+ - [{op: traces.show, args: {recorded_at: present}}, {op: endpoints.chart}]+ - [{op: endpoints.show, args: {recorded_at: present}}, {op: endpoints.list, args: {from: present}}]+ - [{op: traces.show, args: {recorded_at: present}}, {op: endpoints.list, args: {from: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, traces.show]+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: No pagination / unbounded result set. GET /api/admin/audit-log runs SELECT ... FROM audit_events e WHERE e.org_id = ? ORDER BY e.created_at DESC with no LIMIT, so every request returns the organization's whole history (thousands to over a hundred thousand rows, db.response.returned_rows) and then serializes all of it. Events accrue daily, so the rows per request and p50 grow week over week (roughly 170-190ms in the first week to ~370ms now, p95 in the seconds for the biggest orgs) with no deploy and flat CPU. It is not an index problem; the time per returned row is constant, and the dashboard widget's LIMIT 20 query on the same index takes ~2ms. Fix with keyset pagination on (created_at, id) and a page size (for example 100, max 500), with the UI loading further pages on demand.+ rubric:+ - Shows the query has no LIMIT and returns a growing number of rows (returned_rows in the thousands and up), and ties the weekly p50 creep to the event history growing.+ - Rules out the user's index theory with evidence (cost per row constant, the LIMIT 20 query on the same index is fast, the migration already indexes (org_id, created_at)).+ - Proposes pagination (keyset/cursor on created_at, id, with a page size) instead of an index.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6), p99_ms: (.p99Duration / 1e6)}]'+ - shell: >-+ traceway logs query --since 12h --search /api/admin/audit-log --page-size 20 --output json+ | jq '{data: ([.data[] | {traceId, timestamp, body, ms: (.logAttributes.duration_ms | tonumber)}] | sort_by(-.ms))}'+ - shell: >-+ traceway traces show {{calls.c02.data.0.traceId}} --recorded-at {{calls.c02.data.0.timestamp}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, id: .endpoint.id, recordedAt: .endpoint.recordedAt, ms: (.endpoint.duration / 1e6),+ spans: [.spans[] | {name, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: >-+ traceway endpoints chart --metric-type p50 --since 29d --interval-minutes 10080 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/admin/audit-log" or .endpoint == "GET /api/admin/dashboard/activity") | {endpoint, t: .timestamp, p50_ms: .value}]'+ - shell: >-+ traceway logs query --since 6h --search /api/admin/dashboard/activity --page-size 3 --output json+ | jq '{data: [.data[] | {traceId, timestamp, body}]}'+ - shell: >-+ traceway traces show {{calls.c05.data.0.traceId}} --recorded-at {{calls.c05.data.0.timestamp}} --output json+ | jq '[.nodes[] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6), spans: [.spans[] | {name, ms: (.duration / 1e6), query: .attributes["db.query.text"], rows: .attributes["db.response.returned_rows"]}]}]'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - shell: git log --format='%h %ad %s' --date=iso -5+ - read_file: src/main/java/com/ledgerly/admin/audit/AuditEventRepository.java+ - read_file: src/main/resources/db/migration/V14__audit_events.sql+ - submit:+ checklist_item: no_pagination_unbounded_result_set+ category: database+ locus: {endpoint: "GET /api/admin/audit-log", span: "SELECT audit_events (WHERE e.org_id = ? ORDER BY e.created_at DESC, no LIMIT)", service: admin-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/admin/audit-log has a p50 of a few hundred ms and a p95 of several seconds"}+ - {call_id: c03, claim: "the request spends its time in one audit_events query that has no LIMIT and returns thousands of rows (db.response.returned_rows), then serializing them inside the controller span"}+ - {call_id: c04, claim: "p50 climbed every week over the month with no step, while the dashboard widget stayed flat"}+ - {call_id: c06, claim: "the widget's LIMIT 20 query on the same table and index returns in about 2ms"}+ - {call_id: c07, claim: "CPU is flat around 25%"}+ - {call_id: c10, claim: "audit_events is already indexed on (org_id, created_at DESC)"}+ ruled_out:+ - {checklist_item: missing_or_unused_index, call_id: c06, reason: "the same index serves the LIMIT 20 query in ~2ms; the big query's time is proportional to the rows it returns"}+ - {checklist_item: over_fetching, call_id: c03, reason: "explicit column list; the cost is the number of rows, not their width"}+ - {checklist_item: cpu_saturation, call_id: c07, reason: "CPU flat"}+ fix: Paginate the audit log with a keyset cursor on (created_at, id) and a page size (default 100, max 500) instead of returning the whole history; the UI loads more pages on demand. No new index is needed.+ confidence: high+ report: |+ The Audit log page is slow because GET /api/admin/audit-log has no pagination. AuditEventRepository.findByOrg runs SELECT ... FROM audit_events e WHERE e.org_id = ? ORDER BY e.created_at DESC with no LIMIT, so each request returns the organization's entire history (thousands to over a hundred thousand rows) and serializes it. Events pile up every day, so p50 has climbed week over week with no deploy and flat CPU. An index would not help: audit_events is already indexed on (org_id, created_at), the dashboard widget's LIMIT 20 query on it takes ~2ms, and the big query's time is proportional to the rows it returns. Fix: paginate the endpoint with a keyset cursor on (created_at, id) and a page size (default 100, max 500), with the UI loading more pages on demand.diff --git a/cli/test/evals/cases/perf/perf-infra-disk-io.yaml b/cli/test/evals/cases/perf/perf-infra-disk-io.yamlnew file mode 100644index 00000000..22e9d5bb--- /dev/null+++ b/cli/test/evals/cases/perf/perf-infra-disk-io.yaml@@ -0,0 +1,94 @@+id: perf-infra-disk-io+suite: perf+title: Every query got several times slower while the database host's disk queue filled up+tags: [infrastructure, database, node]+seed: perf/infra-disk-io+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The availability search is the slowest query and the user suspects a missing index, but every query slowed by the same factor at the same minute, primary-key lookups included, nothing was deployed to the API and its tables did not grow; at that minute the database host's disk queue went from ~1 to ~36 operations and iowait from ~2% to ~35%, with its CPU still mostly idle.+workspace: {repo: bookings-api-node}+prompt: |+ /traceway perf bookings-api+ Since earlier today every query from bookings-api is slow. Availability search is the worst, but even looking a booking up by id takes several times longer than usual. We haven't deployed the API. I'm guessing the availability table needs a new index, can you confirm and tell me which one?+budget: {max_turns: 40, max_tool_calls: 34, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: disk_io_bottleneck+ not: [missing_or_unused_index, lock_transaction_contention, connection_pool_exhaustion, cpu_saturation, expensive_join_or_aggregation]+ locus: {host: pg-bookings-1}+ onset: {at: "{{seed.onsetAt}}", tolerance_minutes: 60}+ grounding:+ - "system.disk"+ - "pg-bookings-1"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~system\\.(disk|cpu)"}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: Disk / IO bottleneck on the database host pg-bookings-1. At the onset every bookings-api query became roughly 4.5-7.5x slower at once (availability search p50 ~15ms to ~70ms, primary-key booking lookups ~1-3ms to ~5-20ms per query, writes 6.5-9.5x), while request counts only followed the normal daily curve and the API's CPU stayed flat. At the same minute the database host's disk read throughput went from ~15MB/s to ~110MB/s and writes from ~8 to ~55MB/s, the disk queue (system.disk.pending_operations) from ~1 to ~36 and iowait (system.cpu.utilization state=wait) from ~2% to ~35%, with user CPU still ~13%. Nothing was deployed to the API; twelve minutes before the onset a commit added the rebuild-search-documents Job, which bulk-reads and writes 50,000 listings per statement with no pause. Not a missing index - indexed point lookups slowed just as much. Fix - stop or throttle the rebuild (small batches with pauses, off-peak, or on a replica) and give the volume IOPS headroom.+ rubric:+ - Shows that all queries, including primary-key lookups, slowed together at one moment, which an index cannot explain.+ - Finds the database host's disk saturation (queue depth, iowait, throughput) with its CPU still low, and the onset time.+ - Connects it to the rebuild job that started then and proposes throttling, rescheduling or moving it, or more disk IOPS, instead of adding an index.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p50 --since 24h --interval-minutes 60 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, p50_ms: map("\(.timestamp[11:16]) \(.value|round)")})'+ - shell: >-+ traceway endpoints show {{seed.earlierBooking.id}} --recorded-at {{seed.earlierBooking.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10), query: .attributes["db.query.text"]}]}'+ - shell: >-+ traceway endpoints show {{seed.slowBooking.id}} --recorded-at {{seed.slowBooking.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10), query: .attributes["db.query.text"]}]}'+ - shell: >-+ traceway endpoints show {{seed.slowAvailability.id}} --recorded-at {{seed.slowAvailability.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10), rows: .attributes["db.response.returned_rows"]}]}'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by host.name --tag state=user --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --group-by state --tag host.name=pg-bookings-1 --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway metrics query --name system.disk.pending_operations --aggregation avg --group-by host.name --since 24h --interval-minutes 60 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 10 | round / 10)"))'+ - shell: >-+ traceway metrics query --name system.disk.io --aggregation max --group-by direction --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value / 1e9 | round)GB"))'+ - shell: git log --date=iso-strict --format='%h %ad %an %s' --stat -n 3+ - read_file: jobs/rebuild-search-documents.js+ - submit:+ checklist_item: disk_io_bottleneck+ category: infrastructure+ locus: {service: bookings-db, host: pg-bookings-1, span: "every database query of bookings-api"}+ onset: "{{seed.onsetAt}}"+ evidence:+ - {call_id: c02, claim: "every endpoint's p50 stepped up about 4.5-7x in the same hour and stayed up"}+ - {call_id: c03, claim: "before that, a booking lookup's two primary-key queries took 1-3ms each"}+ - {call_id: c04, claim: "after it, the same two primary-key queries each take several times as long: indexed point lookups slowed as much as anything"}+ - {call_id: c05, claim: "the availability search query alone now takes ~60ms for a few hundred rows, against an endpoint p50 of ~15ms before: the same few-fold slowdown as the primary-key lookups"}+ - {call_id: c06, claim: "user CPU on the API host is flat and on the database host only rose from ~10% to ~13%"}+ - {call_id: c07, claim: "on pg-bookings-1 iowait went from ~2% to ~35% at the step"}+ - {call_id: c08, claim: "its disk queue went from about 1 to over 30 operations at the same time"}+ - {call_id: c09, claim: "disk reads and writes went up about 7x from then on"}+ - {call_id: c10, claim: "nothing changed in the API; minutes before the step a commit added the rebuild-search-documents Job"}+ ruled_out:+ - {checklist_item: missing_or_unused_index, call_id: c04, reason: "primary-key lookups slowed as much as the availability search, all at the same moment, without any table growth"}+ - {checklist_item: cpu_saturation, call_id: c07, reason: "the database host's CPU is mostly idle; it is waiting on the disk"}+ - {checklist_item: lock_transaction_contention, call_id: c04, reason: "plain reads slowed too, and the rebuild writes to a separate table"}+ fix: Stop the rebuild-search-documents Job, then rerun it throttled (small batches with pauses, off-peak, or against a replica) so it cannot take the primary's whole disk; check the volume has IOPS headroom for both.+ confidence: high+ report: |+ This is not an index problem. Every query bookings-api runs got roughly 5-9x slower in the same minute about ten hours ago, including primary-key lookups (a booking by id went from 1-3ms per query to several times that), and the API's CPU and traffic did not change. At that minute the database host pg-bookings-1 went from ~2% to ~35% iowait with its CPU otherwise idle, its disk queue from ~1 to over 30 operations, and its disk reads and writes up about 7x: the database is waiting on its disk. Twelve minutes earlier a commit added the rebuild-search-documents Job, which reads and rewrites 50,000 listings per statement with no pause on the primary. Fix: stop it, then rerun it throttled (batches of a few thousand listings with a pause between them), off-peak or against a replica, and make sure the volume has IOPS headroom.diff --git a/cli/test/evals/cases/perf/perf-infra-memory-gc.yaml b/cli/test/evals/cases/perf/perf-infra-memory-gc.yamlnew file mode 100644index 00000000..e84bd844--- /dev/null+++ b/cli/test/evals/cases/perf/perf-infra-memory-gc.yaml@@ -0,0 +1,86 @@+id: perf-infra-memory-gc+suite: perf+title: The heap climbs to its limit every seven hours and every endpoint stalls until the pod restarts+tags: [infrastructure, go]+seed: perf/infra-memory-gc+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The batch geocode endpoint is the slowest route and allocates the most per request, so it looks like the culprit, but it runs ~60 times a day, its latency is mostly its own bounded work (it only picks up the same stalls as everything else near the top of the cycle), the heap climbs at the same steady rate day and night with no step at its calls and only drops at restarts, and the stalls hit every endpoint, including the one-query place lookup.+workspace: {repo: geo-api-go}+prompt: |+ /traceway perf geo-api+ Every few hours geo-api gets slow on every endpoint until Kubernetes restarts the pod (OOMKilled), and right after the restart everything is fast again. The batch geocoding endpoint is by far the heaviest thing it runs, so I suspect that. What's going on and how do we fix it?+budget: {max_turns: 40, max_tool_calls: 34, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: memory_pressure_gc_pauses+ not: [unbounded_in_memory_work_large_allocations, connection_pool_exhaustion, cache_stampede_thundering_herd, cpu_saturation]+ locus: {service: geo-api}+ grounding:+ - "go.memory.used"+ - "starting geo-api"+ calls:+ required:+ - {op: endpoints.list}+ - {op: metrics.query, args: {name: "~memory"}}+ any_of:+ - [{op: endpoints.show, args: {recorded_at: present}}]+ - [{op: traces.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: Memory pressure / GC pauses. go.memory.used climbs steadily from ~0.3GB to ~0.9GB over seven hours, reaching the 900MiB GOMEMLIMIT with the GC goal pinned at the limit; each time it drops back the startup log "starting geo-api" shows the restart (every seven hours). As the heap fills, requests on every endpoint stall in gaps with no span (each up to ~0.4s, including after their query has returned), so p50 stays at a few ms on the interactive endpoints while p95/p99 build up through the cycle and reset at each restart. The batch endpoint is slow because of its own bounded work and is rare; it is not what fills the heap. The cause is the autocomplete memo (internal/memo) - expired entries are only ignored on read and never deleted, and its locale|prefix key space is unbounded, so it retains everything. Fix - bound the memo (delete expired entries, cap its size or use an LRU), then confirm the heap stays flat.+ rubric:+ - Connects the latency cycle to the heap climbing to the memory limit and the restarts (memory metric sawtooth plus startup logs), with stalls as gaps in otherwise fast requests on every endpoint.+ - Rules out the batch endpoint's allocations as the cause (rare, bounded, heap growth independent of it) and a connection pool problem (stalls also after the query, tracking heap rather than load).+ - Finds the never-evicting memo and proposes bounding it (expiry sweep, size cap or LRU) rather than only raising the memory limit.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 10 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 24h --interval-minutes 60 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, p99_ms: map("\(.timestamp[11:16]) \(.value|round)")})'+ - shell: >-+ traceway metrics query --name go.memory.used --aggregation max --group-by go.memory.type --since 24h --interval-minutes 60 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value/1e6|round)MB"))'+ - shell: >-+ traceway metrics query --name go.memory.gc.goal --aggregation max --since 24h --interval-minutes 60 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value/1e6|round)MB"))'+ - shell: traceway logs query --since 24h --search starting --output json | jq -r '.data[] | "\(.timestamp) \(.body) \(.logAttributes)"'+ - shell: >-+ traceway endpoints show {{seed.pausedRequest.id}} --recorded-at {{seed.pausedRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, started: .endpoint.recordedAt[11:23], total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, start: .startTime[11:23], ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway endpoints show {{seed.calmRequest.id}} --recorded-at {{seed.calmRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, started: .endpoint.recordedAt[11:23], total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, start: .startTime[11:23], ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway endpoints show {{seed.batchRequest.id}} --recorded-at {{seed.batchRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, total_ms: (.endpoint.duration/1e6|round), spans: [.spans[] | {name, ms: (.duration/1e6|round), size: .attributes["batch.size"]}]}'+ - read_file: internal/places/places.go+ - read_file: internal/memo/memo.go+ - submit:+ checklist_item: memory_pressure_gc_pauses+ category: infrastructure+ locus: {service: geo-api, host: geo-api-5d8c7b9f4-hq2mz, span: "untraced pauses in requests on every endpoint"}+ onset: null+ evidence:+ - {call_id: c01, claim: "autocomplete, place lookup and reverse all have a p50 of a few ms but a p95/p99 of hundreds: a minority of requests stall on every interactive endpoint"}+ - {call_id: c02, claim: "the p99 of those three builds up over several hours and collapses at the same moments"}+ - {call_id: c03, claim: "the heap climbs steadily from ~0.3GB to ~0.9GB and drops back at those moments"}+ - {call_id: c04, claim: "the GC goal sits at the 900MiB limit for the last hours of each cycle"}+ - {call_id: c05, claim: "each drop is a process restart (startup log)"}+ - {call_id: c06, claim: "a one-query place lookup near the top of the cycle stalls for ~0.2s or more after its ~2ms query returned, with no span"}+ - {call_id: c07, claim: "the same lookup right after a restart takes a few ms"}+ - {call_id: c08, claim: "the heaviest batch while the heap was low spends its time in its own decode, query and matching spans, sized by the batch"}+ ruled_out:+ - {checklist_item: unbounded_in_memory_work_large_allocations, call_id: c03, reason: "the heap grows at a steady rate day and night and only drops at restarts; the batch endpoint is rare and its work is bounded"}+ - {checklist_item: connection_pool_exhaustion, call_id: c06, reason: "the stall comes after the query returned and follows the heap, not the load"}+ fix: Bound the autocomplete memo - delete expired entries as it writes and cap its size (or use an LRU), so the heap stays flat; keep GOMEMLIMIT below the container limit.+ confidence: high+ report: |+ geo-api is running out of memory, and the slowdown is the Go GC fighting the 900MiB GOMEMLIMIT. go.memory.used climbs steadily from ~0.3GB to ~0.9GB over seven hours, with the GC goal pinned at the limit near the end; then the pod is killed and "starting geo-api" shows the restart, after which the cycle repeats. As the heap fills, requests on every endpoint stall in pauses with no span (a one-query place lookup took hundreds of ms, much of it after its ~2ms query had returned) and p95/p99 build up until the restart, while p50 stays at a few ms. The batch geocode endpoint is not the cause: it runs about 60 times a day, its latency is its own bounded work, and the heap climbs smoothly with no step at its calls. The heap is retained by the autocomplete memo (internal/memo): expired entries are ignored on read but never deleted, and its locale|prefix keys are unbounded. Fix: make the memo drop expired entries as it writes and cap its size (or use an LRU), then check that go.memory.used stays flat.diff --git a/cli/test/evals/cases/perf/perf-infra-noisy-neighbor.yaml b/cli/test/evals/cases/perf/perf-infra-noisy-neighbor.yamlnew file mode 100644index 00000000..52d98722--- /dev/null+++ b/cli/test/evals/cases/perf/perf-infra-noisy-neighbor.yaml@@ -0,0 +1,91 @@+id: perf-infra-noisy-neighbor+suite: perf+title: One of three VMs loses a third of its CPU to steal and serves every request slowly+tags: [infrastructure, go]+seed: perf/infra-noisy-neighbor+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: Fleet-wide CPU saturation (every endpoint's p95 doubled, so add servers); but traffic is flat, tickets-node-a and tickets-node-b are ~70% idle with unchanged latency, and only tickets-node-c is slow, where ~30% of the CPU time goes to steal while its own user and system time did not grow.+workspace: {repo: tickets-api-go}+prompt: |+ /traceway perf tickets-api+ For the last half day or so the p95 of every tickets-api endpoint has been up. Some people say the seat map takes ages, others say it's fine for them, and the CPU graph doesn't look alarming. Do we need more servers?+budget: {max_turns: 40, max_tool_calls: 34, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: undersized_noisy_neighbor_instance+ not: [cpu_saturation, memory_pressure_gc_pauses, connection_pool_exhaustion, slow_third_party_upstream_api]+ locus: {host: node-c}+ onset: {at: "{{seed.onsetAt}}", tolerance_minutes: 60}+ grounding:+ - "tickets-node-c"+ - "system.cpu.utilization"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: metrics.query, args: {name: "~system\\.cpu", group_by: "~host"}}, {op: metrics.query, args: {name: "~http\\.server\\.request\\.duration", group_by: "~host"}}]+ - [{op: metrics.query, args: {name: "~system\\.cpu", group_by: "~host"}}, {op: logs.query}]+ - [{op: metrics.query, args: {name: "~system\\.cpu", tag_filters: "~host"}}, {op: metrics.query, args: {name: "~http\\.server\\.request\\.duration", group_by: "~host"}}]+ - [{op: metrics.query, args: {name: "~system\\.cpu", tag_filters: "~host"}}, {op: logs.query}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: Undersized / noisy-neighbour instance - tickets-node-c. Since the onset about 14 hours ago, only requests served by tickets-node-c are slow (http.server.request.duration around 43ms there against ~17ms on node-a and node-b, and in the access logs every slow request comes from node-c); because the load balancer spreads requests evenly, a third of every endpoint's requests are slow, which lifted p95 everywhere while p50 barely moved. At the same moment node-c started losing 25-35% of its CPU time to steal (system.cpu.utilization state=steal), its idle fell from ~69% to ~39%, and its user and system time stayed where they were, like the other two hosts. Traffic is flat and nothing was deployed, so this is not saturation - more servers would not help node-c. Fix - replace or migrate tickets-node-c (stop/start onto another host, or let the ASG replace it), take it out of the target group meanwhile, and consider dedicated tenancy or alerting on steal.+ rubric:+ - Separates the latency by host (request-duration metric by host.name or the access logs) and finds that only tickets-node-c is slow.+ - Identifies CPU steal on tickets-node-c, with its user/system time and the other hosts normal, and rules out fleet-wide saturation and a code change.+ - Recommends draining and replacing or migrating that one instance instead of adding servers.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 8 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --metric-type p95 --since 24h --interval-minutes 120 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, p95_ms: map("\(.timestamp[11:16]) \(.value|round)")})'+ - shell: >-+ traceway metrics query --name http.server.request.duration.avg --aggregation avg --group-by host.name --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 1000 | round)ms"))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --tag state=idle --group-by host.name --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --tag state=steal --group-by host.name --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 100 | round)%"))'+ - shell: >-+ traceway metrics query --name system.cpu.utilization --aggregation avg --tag state=user --group-by host.name --since 24h --interval-minutes 120 --output json+ | jq '.results[0].series | map_values(map("\(.timestamp[11:16]) \(.value * 100 | round)%"))'+ - shell: traceway logs query --since 3h --search /api/events/ --page-size 20 --output json | jq -r '.data[] | "\(.timestamp) \(.resourceAttributes["host.name"]) \(.traceId) \(.body)"'+ - shell: >-+ traceway endpoints show {{seed.slowRequest.id}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10)}]}'+ - shell: >-+ traceway endpoints show {{seed.fastRequest.id}} --recorded-at {{seed.fastRequest.recordedAt}} --output json+ | jq '{endpoint: .endpoint.endpoint, total_ms: ((.endpoint.duration/1e5|round)/10), spans: [.spans[] | {name, ms: ((.duration/1e5|round)/10)}]}'+ - read_file: deploy/terraform/instances.tf+ - submit:+ checklist_item: undersized_noisy_neighbor_instance+ category: infrastructure+ locus: {service: tickets-api, host: tickets-node-c}+ onset: "{{seed.onsetAt}}"+ evidence:+ - {call_id: c01, claim: "every endpoint's p95 is 2-3x its p50 over the last day: a share of requests is slow on every route"}+ - {call_id: c02, claim: "the p95 of all endpoints stepped up in the same bucket"}+ - {call_id: c03, claim: "per host, only tickets-node-c's request duration rose (~43ms against ~17ms on node-a and node-b), from that same bucket"}+ - {call_id: c04, claim: "node-a and node-b stay ~69% idle; node-c fell to ~39% at the step"}+ - {call_id: c05, claim: "the lost idle on node-c is steal: ~30% from the step on, zero on the other hosts"}+ - {call_id: c06, claim: "user CPU is the same on all three hosts, so node-c is not doing more work"}+ - {call_id: c07, claim: "in the access logs the slow requests all come from tickets-node-c"}+ - {call_id: c08, claim: "a slow request on node-c runs the same two queries as a fast one, only a little longer; most of its time is the untraced CPU work around them, several times what it takes on node-a; no single long pole"}+ - {call_id: c09, claim: "the same endpoint on node-a takes about 20ms"}+ ruled_out:+ - {checklist_item: cpu_saturation, call_id: c04, reason: "two of three hosts are ~70% idle and traffic is flat; only node-c, and only through steal"}+ - {checklist_item: connection_pool_exhaustion, call_id: c08, reason: "the slow requests are slow throughout on one host, not waiting before their queries"}+ fix: Take tickets-node-c out of the target group and replace it (stop/start the instance so it lands on another host, or terminate it and let it be recreated); more servers are not needed. Alert on CPU steal per host so the next one is caught quickly; consider dedicated tenancy if it recurs.+ confidence: high+ report: |+ No, more servers would not help: one of the three VMs is being starved by its hypervisor. Since about 14 hours ago every request served by tickets-node-c takes ~2.5-3x longer (http.server.request.duration ~43ms there against ~17ms on node-a and node-b; in the access logs the 40-65ms requests all come from node-c). The load balancer sends it a third of the traffic, which is why every endpoint's p95 rose while p50 barely moved and some users see no problem. From the same moment node-c loses ~30% of its CPU time to steal, so its idle fell from ~69% to ~39% although its own user and system time did not change; node-a and node-b have no steal and stay ~69% idle, and traffic is flat. Drain tickets-node-c from the target group and replace it (a stop/start moves it to another host), and add an alert on per-host CPU steal.diff --git a/cli/test/evals/cases/perf/perf-net-chatty-fanout.yaml b/cli/test/evals/cases/perf/perf-net-chatty-fanout.yamlnew file mode 100644index 00000000..939cd12f--- /dev/null+++ b/cli/test/evals/cases/perf/perf-net-chatty-fanout.yaml@@ -0,0 +1,81 @@+id: perf-net-chatty-fanout+suite: perf+title: Cart page slows down with the size of the cart+tags: [network, node, express, multi-project]+seed: perf/net-chatty-fanout+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The hops run one after another, which reads like "sequential calls that could run in parallel", but they are dozens of tiny per-line calls to the same three endpoints, each spending ~9ms on the wire for ~2ms of work; the cost is the number of round trips, and the batch endpoints (POST /products/batch is already used by the wishlist) remove them.+workspace: {repo: storefront-gateway-node}+prompt: |+ /traceway perf GET /api/cart+ Customers with bigger carts say the cart page is sluggish, and it is our slowest storefront endpoint. The services it depends on belong to the catalog and fulfillment teams, who say their endpoints answer in a couple of milliseconds. Find out where the time goes and what we should change.+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: chatty_ipc_microservice_fan_out+ not: [sequential_calls_that_could_run_in_parallel, select_n_plus_1, slow_third_party_upstream_api, cpu_saturation]+ locus: {endpoint: "GET /api/cart", service: storefront-gateway}+ grounding:+ - "GET /stock/:sku"+ - "GET /prices/:sku"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ order:+ - [endpoints.list, traces.show]+ - [endpoints.list, endpoints.show]+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Chatty IPC / microservice fan-out. buildCart (src/routes/cart.js) asks catalog-service, pricing-service (catalog project) and inventory-service (fulfillment project) about every cart line, one request per line per service, so a ten-line cart is 31 hops, 30 of them to the other two projects. Each downstream call does ~1-3ms of work but its CLIENT span lasts ~9-12ms, so roughly 280ms of a ~330ms request is round trips; latency grows with cart size (p50 ~350ms, p95 ~530ms). Fix by fetching each service once per cart with the batch endpoints (catalog POST /products/batch, already used by the wishlist; pricing and inventory getMany), which collapses ~31 hops to 4.+ rubric:+ - Uses a cart request's trace to count the hops (about three per cart line, into the catalog and fulfillment projects) and shows that the round trips, not the downstream work, make up most of the request.+ - Explains why this is chattiness rather than serialization of a few independent calls; the fix is fewer calls (batch per service), not merely running the same thirty calls concurrently.+ - Proposes batching in buildCart with the existing getMany/batch endpoints, keeping the cart output (titles, prices, stock flags, total) unchanged.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}] | sort_by(-.p95_ms)'+ - shell: >-+ traceway logs query --since 6h --search /api/cart --page-size 10 --output json+ | jq -r '.data[] | "\(.traceId) \(.timestamp) \(.body)"'+ - shell: >-+ traceway traces show {{seed.cartRequest.traceId}} --recorded-at {{seed.cartRequest.recordedAt}} --output json+ | jq '{gateway_ms: (.nodes[0].endpoint.duration / 1e6), downstream_nodes: ([.nodes[1:][]] | length),+ by_endpoint: ([.nodes[1:][] | {project: .projectName, endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6)}] | group_by(.endpoint)+ | map({endpoint: .[0].endpoint, project: .[0].project, calls: length, server_ms_total: (map(.ms) | add | round)})),+ client_calls: ([.nodes[0].spans[] | select(.spanKind == 3 and .serviceName == "storefront-gateway" and .attributes["url.full"] != null)]+ | {count: length, total_ms: (map(.duration / 1e6) | add | round), first: (.[0:4] | map({url: .attributes["url.full"], start: .startTime, ms: (.duration / 1e6)}))})}'+ - shell: traceway projects list --output json+ - shell: >-+ traceway endpoints list --project {{env.projects.catalog.id}} --since 24h --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6), p95_ms: (.p95Duration / 1e6)}]'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 24h --interval-minutes 360 --output json+ - read_file: src/routes/cart.js+ - read_file: src/clients/index.js+ - submit:+ checklist_item: chatty_ipc_microservice_fan_out+ category: network+ locus: {endpoint: "GET /api/cart", span: "per-line GET /products/:sku, GET /prices/:sku, GET /stock/:sku calls from buildCart", service: storefront-gateway}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/cart is the slowest storefront endpoint; GET /carts/:id behind it is fast"}+ - {call_id: c03, claim: "a cart of about ten lines makes ~30 hops into the catalog and fulfillment projects (one each to products, prices and stock per line) plus one to cart-service; the downstream servers spend 1-2ms per call while each client call takes ~10ms, so the client calls add up to nearly the whole request"}+ - {call_id: c05, claim: "catalog endpoints answer in a millisecond or two; POST /products/batch exists and is fast"}+ - {call_id: c06, claim: "gateway CPU is flat and low"}+ ruled_out:+ - {checklist_item: sequential_calls_that_could_run_in_parallel, call_id: c03, reason: "the calls are three per cart line to the same endpoints and each is mostly round trip; the fix is fewer calls, not concurrency"}+ - {checklist_item: slow_third_party_upstream_api, call_id: c05, reason: "the downstream services are fast; no hop is slow on its own"}+ - {checklist_item: cpu_saturation, call_id: c06, reason: "CPU flat"}+ fix: In buildCart, fetch products, prices and stock once per cart with catalog.getMany (POST /products/batch), pricing.getMany and inventory.getMany instead of three calls per line; 31 hops become 4.+ confidence: high+ report: |+ GET /api/cart is slow because buildCart (src/routes/cart.js) calls three services in other projects for every cart line: GET /products/:sku and GET /prices/:sku (catalog project) and GET /stock/:sku (fulfillment project), one line at a time. A cart of about ten lines makes ~30 cross-project hops (three per line), plus one to cart-service; the downstream endpoints spend a millisecond or two per call doing work, but each call costs ~9-12ms end to end, so most of the request is round trips, and the latency grows with the cart. The downstream teams are right that their endpoints are fast; CPU is flat. Fix: change buildCart to fetch each service once for all SKUs with the batch methods the clients already have (POST /products/batch is what the wishlist uses), which takes a ten-line cart from 31 hops to 4.diff --git a/cli/test/evals/cases/perf/perf-net-large-payload.yaml b/cli/test/evals/cases/perf/perf-net-large-payload.yamlindex 7a81e958..8b83181d 100644--- a/cli/test/evals/cases/perf/perf-net-large-payload.yaml+++ b/cli/test/evals/cases/perf/perf-net-large-payload.yaml@@ -72,4 +72,4 @@ oracle:fix: Stop inlining images; send CDN URLs for card images and avatars (and stop selecting image_data/avatar_data), so the feed is tens of KB instead of ~2.3MB.confidence: highreport: |- GET /api/feed is slow because each response is about 2.3MB: the handler inlines every card image and avatar as a base64 data URI (feed_cards.image_data, authors.avatar_data; internal/feed/handler.go and store.go). The feed query takes ~15ms and ranking a few ms; the remaining time of the request, after the last child span, is the body being written to the client, so it depends on the client's link. A phone request took well over a second and a desktop request a few hundred ms for the same ~2.3MB, which is why mobile users see it most. CPU is flat and the query is fast, so caching would not help. I changed the feed to return CDN URLs for card images and avatars and stopped selecting the inline image columns; the response drops to tens of KB.+ GET /api/feed is slow because each response is about 2.3MB: the handler inlines every card image and avatar as a base64 data URI (feed_cards.image_data, authors.avatar_data; internal/feed/handler.go and store.go). The feed query takes ~15ms and ranking a few ms; the remaining time of the request, after the last child span, is the body being written to the client, so it depends on the client's link. A phone request took well over a second and a desktop request a few hundred ms for the same ~2.3MB, which is why mobile users see it most. CPU is flat and the query is fast, so caching would not help. Fix: return CDN URLs for card images and avatars and stop selecting the inline image columns; the response drops to tens of KB.diff --git a/cli/test/evals/cases/perf/perf-net-no-conn-reuse.yaml b/cli/test/evals/cases/perf/perf-net-no-conn-reuse.yamlnew file mode 100644index 00000000..7bf2ec82--- /dev/null+++ b/cli/test/evals/cases/perf/perf-net-no-conn-reuse.yaml@@ -0,0 +1,67 @@+id: perf-net-no-conn-reuse+suite: perf+title: Store finder spends most of its time in calls to the maps provider+tags: [network, go]+seed: perf/net-no-conn-reuse+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: The maps provider calls are the long pole, which reads like a slow upstream, but inside every call ~80% is DNS, TCP connect and the TLS handshake on a brand-new connection (http.conn.reused=false); once the request is on the wire the provider answers in ~20ms.+workspace: {repo: store-locator-go}+prompt: |+ The store finder (GET /api/stores/nearby) takes about a quarter of a second, and almost all of it is spent in the calls to our maps provider. Before we pay for their premium tier, can you check whether the provider is really the slow part, and what we could change on our side?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: no_connection_reuse+ not: [slow_third_party_upstream_api, sequential_calls_that_could_run_in_parallel, cpu_saturation]+ locus: {endpoint: "GET /api/stores/nearby", service: store-locator-api}+ grounding:+ - "http.tls"+ - "http.conn.reused"+ calls:+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: No connection reuse. maps.Client.get (internal/maps/client.go) builds a new http.Client and http.Transport for every call, so neither keep-alive connections nor TLS sessions survive between calls. Every call to maps.geo.example (two per request, geocode then distance matrix) shows http.getconn with http.conn.reused=false and http.dns (~15ms), http.connect (~25ms) and http.tls (~50ms) under it; the provider itself answers ~20-25ms after http.send. Of a ~250ms request ~180-200ms is connection setup. The inventory client, which shares one transport, shows reused connections and ~0.03ms getconn. Fix by creating one transport in maps.New and reusing it (and draining bodies), which cuts the endpoint to roughly 50-60ms.+ rubric:+ - Breaks a provider call down with the httptrace child spans and shows the DNS/connect/TLS cost repeats on every call with http.conn.reused=false.+ - Rules out a slow provider from the data (time from send to first byte is ~20ms) and contrasts with a reused connection elsewhere.+ - Proposes a single long-lived http.Client/Transport in the maps client, with the expected saving.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round)}] | sort_by(-.p95_ms)'+ - shell: >-+ traceway logs query --since 6h --search /api/stores/ --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6),+ spans: [.spans | sort_by(.startTime)[] | {name, ms: (.duration / 1e6 * 10 | round / 10), url: .attributes["url.full"], reused: .attributes["http.conn.reused"]}]}'+ - shell: >-+ traceway traces show {{seed.availRequest.traceId}} --recorded-at {{seed.availRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6),+ spans: [.spans | sort_by(.startTime)[] | {name, ms: (.duration / 1e6 * 100 | round / 100), url: .attributes["url.full"], reused: .attributes["http.conn.reused"]}]}'+ - read_file: internal/maps/client.go+ - submit:+ checklist_item: no_connection_reuse+ category: network+ locus: {endpoint: "GET /api/stores/nearby", span: "http.getconn > http.dns / http.connect / http.tls under GET https://maps.geo.example/...", service: store-locator-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "GET /api/stores/nearby p50 ~240ms; the other store routes are fast"}+ - {call_id: c03, claim: "both maps.geo.example calls open a new connection (http.conn.reused=false): DNS, TCP connect and TLS take roughly 100ms or more per call, while the provider's own part (from http.send to http.receive) is only a few tens of ms"}+ - {call_id: c04, claim: "the inventory call reuses its connection (reused=true, getconn in microseconds)"}+ ruled_out:+ - {checklist_item: slow_third_party_upstream_api, call_id: c03, reason: "send-to-first-byte is a few tens of ms against roughly 100ms of setup per call; the provider is not slow, the connection setup is"}+ fix: Build the http.Transport once in maps.New and reuse it for every call (keep-alive and TLS session reuse), and drain response bodies so connections return to the pool; the endpoint should drop from ~250ms to ~50-60ms.+ confidence: high+ report: |+ The maps provider is not slow; we pay a fresh connection for every call. maps.Client.get (internal/maps/client.go) creates a new http.Client with a new http.Transport per request, so both calls of GET /api/stores/nearby (geocode, then distance matrix) show http.getconn with http.conn.reused=false and DNS (~15ms), TCP connect (~25ms) and the TLS handshake (~50ms) underneath. That setup is about 180-200ms of the ~250ms request; the provider answers 20-30ms after the request is sent. The inventory client, which shares a transport, reuses its connection. Fix: build the transport once in maps.New so connections and TLS sessions are reused (and drain the body); the endpoint should drop to roughly 50-60ms. The premium tier would not help.diff --git a/cli/test/evals/cases/perf/perf-net-retries.yaml b/cli/test/evals/cases/perf/perf-net-retries.yamlnew file mode 100644index 00000000..f7185668--- /dev/null+++ b/cli/test/evals/cases/perf/perf-net-retries.yaml@@ -0,0 +1,71 @@+id: perf-net-retries+suite: perf+title: Shipping quotes occasionally take 6 to 13 seconds+tags: [network, python, fastapi]+seed: perf/net-retries+arms: [cli+skill, cli-noskill, mcp]+invoke: slash+distractor: The carrier's rate API is the long pole of the slow requests, which reads like a slow upstream, but its successful answers take ~150ms and the slow requests are the client waiting out httpx's default 5s timeout and backing off, one or two times, before the same call succeeds.+workspace: {repo: shipping-quotes-py}+prompt: |+ /traceway perf POST /api/shipping/quotes+ Most shipping quotes come back quickly, but every now and then checkout sits on the shipping step for 6 to 13 seconds. The carrier's status page shows no incidents. What is going on, and what should we change?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: no_timeouts_aggressive_retries+ not: [slow_third_party_upstream_api, connection_pool_exhaustion, cpu_saturation]+ locus: {endpoint: "POST /api/shipping/quotes", span_contains: "rates.carrier.example"}+ grounding:+ - "Retrying app.carriers.client.CarrierClient.fetch_rates"+ - "ReadTimeout"+ calls:+ required:+ - {op: endpoints.list}+ - {op: logs.query}+ any_of:+ - [{op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: No timeouts / aggressive retries. CarrierClient.fetch_rates (app/carriers/client.py) uses an httpx client with no explicit timeout, so httpx's default 5s applies, inside a tenacity retry of 3 attempts with exponential waits of 1s and 2s. A few percent of attempts hang until the 5s read timeout (or get a quick 503); the request then sleeps and calls the same endpoint again, so a slow request shows the same POST to rates.carrier.example two or three times with 1s/2s gaps, and takes ~6s (one retry) or ~13s (two). p50 is ~160ms and a successful carrier call takes ~150ms, so the carrier is not slow in general; the tail is the client's timeout and retry policy, and the WARN logs show the retries all day. Fix with a tight explicit timeout (sub-second read), at most one quick retry with jitter and an overall deadline.+ rubric:+ - Shows the same carrier span repeated within one trace, with the 5s timeouts and the backoff gaps, and ties it to the tenacity retry WARN lines.+ - Rules out a slow upstream from the data (successful attempts take ~150ms, p50 is low, the slow requests are timeouts plus sleeps, spread through the day).+ - Proposes an explicit short httpx timeout plus a bounded retry (fewer attempts, jittered short backoff, total deadline) in CarrierClient.+oracle:+ - shell: >-+ traceway endpoints list --since 24h --order-by p95 --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}] | sort_by(-.p99_ms)'+ - shell: >-+ traceway logs query --since 24h --search Retrying --page-size 30 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.severityText) \(.body | split("\n")[0])"'+ - shell: >-+ traceway traces show {{seed.retriedRequest.traceId}} --recorded-at {{seed.retriedRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6), status: .endpoint.statusCode,+ spans: [.spans[] | {name, url: .attributes["url.full"], start: .startTime, ms: (.duration / 1e6), status: .attributes["http.response.status_code"], error: .attributes["error.type"]}]}'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 24h --interval-minutes 180 --output json+ | jq '.series | map(select(.endpoint == "POST /api/shipping/quotes")) | map({t: .timestamp, p99_ms: (.value | round)})'+ - read_file: app/carriers/client.py+ - submit:+ checklist_item: no_timeouts_aggressive_retries+ category: network+ locus: {endpoint: "POST /api/shipping/quotes", span: "POST https://rates.carrier.example/v2/rates (retried after ReadTimeout)", service: quotes-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "p50 is ~160ms but p99 is over 6s: a long tail on an otherwise fast endpoint"}+ - {call_id: c02, claim: "tenacity WARN lines every few minutes: 'Retrying app.carriers.client.CarrierClient.fetch_rates in 1.0 seconds as it raised ReadTimeout', and some 'in 2.0 seconds' second retries"}+ - {call_id: c03, claim: "a 13s request: two 5s POSTs to rates.carrier.example ending in ReadTimeout, 1s and 2s gaps, then the same call succeeds in a fraction of a second"}+ - {call_id: c04, claim: "p99 is over 6s in most 3-hour buckets of the day: the tail is spread across the day, not a provider incident window"}+ ruled_out:+ - {checklist_item: slow_third_party_upstream_api, call_id: c03, reason: "successful carrier calls take ~150ms; the time is spent waiting out 5s timeouts and backoff sleeps"}+ fix: Give the carrier client an explicit short timeout (connect 0.5s, read 1s) instead of httpx's 5s default, and bound the retry to two attempts with a short jittered backoff and a 2.5s total deadline.+ confidence: high+ report: |+ The slow shipping quotes are the client's timeout and retry policy, not a slow carrier. CarrierClient.fetch_rates (app/carriers/client.py) posts to rates.carrier.example with an httpx client that sets no timeout, so each attempt can hang for httpx's default 5s, and tenacity retries up to three times with 1s and 2s waits. A few percent of attempts hang; those requests show the same carrier POST two or three times in one trace (5s ReadTimeout, 1s sleep, 5s ReadTimeout, 2s sleep, then a quick success), which is the 6s and 13s you see. The WARN logs show these retries every few minutes, the p99 tail is there all day, and successful carrier calls take about 150ms. Fix: set an explicit timeout (0.5s connect, 1s read) and limit the retry to two attempts with a short jittered backoff and a 2.5s overall deadline.diff --git a/cli/test/evals/cases/perf/perf-net-sequential.yaml b/cli/test/evals/cases/perf/perf-net-sequential.yamlindex aa568a64..6579fda8 100644--- a/cli/test/evals/cases/perf/perf-net-sequential.yaml+++ b/cli/test/evals/cases/perf/perf-net-sequential.yaml@@ -72,4 +72,4 @@ oracle:fix: Run the four independent reads concurrently with Promise.all in DashboardService.load, so the dashboard waits for the slowest service (~100ms) instead of the sum (~280ms).confidence: highreport: |- GET /api/dashboard is slow because DashboardService.load (src/dashboard/dashboard.service.ts) awaits its four internal reads one after another. The trace is a staircase: accounts profile, then billing summary, then unread notifications, then recommendations, each starting when the previous one returned, although all four only need the account id. Each service is fast and steady on its own (their endpoint p50s add up to the dashboard's ~280ms), CPU is flat, and nothing changed over three days. I changed load() to issue the four calls with Promise.all; the endpoint should drop to roughly the slowest call, about 100ms.+ GET /api/dashboard is slow because DashboardService.load (src/dashboard/dashboard.service.ts) awaits its four internal reads one after another. The trace is a staircase: accounts profile, then billing summary, then unread notifications, then recommendations, each starting when the previous one returned, although all four only need the account id. Each service is fast and steady on its own (their endpoint p50s add up to the dashboard's ~280ms), CPU is flat, and nothing changed over three days. Fix: issue the four calls in load() together with Promise.all; the endpoint should drop to roughly the slowest call, about 100ms.diff --git a/cli/test/evals/cases/perf/perf-net-slow-third-party.yaml b/cli/test/evals/cases/perf/perf-net-slow-third-party.yamlnew file mode 100644index 00000000..611cfa5c--- /dev/null+++ b/cli/test/evals/cases/perf/perf-net-slow-third-party.yaml@@ -0,0 +1,79 @@+id: perf-net-slow-third-party+suite: perf+title: Checkout hangs every day or so, then recovers+tags: [network, go]+seed: perf/net-slow-third-party+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: During the spikes goroutines pile up (go.goroutine.count from under 100 to ~400), which reads like requests queueing for database connections, but the DB spans start right away and stay at a few ms; the wait is inside the POST to api.payments.example, and only the routes that call the provider spike.+workspace: {repo: checkout-api-go}+prompt: |+ Every day or so checkout hangs for several seconds and some customers get an error instead of their order confirmation; a while later everything is fine again. Can you figure out what is going on with POST /api/checkout and what we should do about it?+budget: {max_turns: 40, max_tool_calls: 30, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I don't know more than that, please investigate."+expect:+ answer:+ checklist_item: slow_third_party_upstream_api+ not: [connection_pool_exhaustion, no_timeouts_aggressive_retries, cpu_saturation, memory_pressure_gc_pauses]+ locus: {endpoint: "POST /api/checkout", span_contains: "api.payments.example"}+ grounding:+ - "api.payments.example/v1/charges"+ - "POST /api/refunds"+ calls:+ required:+ - {op: endpoints.list}+ any_of:+ - [{op: endpoints.chart}, {op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.chart}, {op: endpoints.show, args: {recorded_at: present}}]+ - [{op: endpoints.list, args: {from: present}}, {op: traces.show, args: {recorded_at: present}}]+ - [{op: endpoints.list, args: {from: present}}, {op: endpoints.show, args: {recorded_at: present}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 4+ max_identical_repeats: 2+judge:+ expected: Slow third-party / upstream API. Three times in three days (25-40 minutes each) the payment provider slowed down; the charge CLIENT span (POST https://api.payments.example/v1/charges) went from ~230ms to several seconds, and some calls hit the client's 10s timeout (the *url.Error "context deadline exceeded (Client.Timeout exceeded while awaiting headers)" group, answered 502). POST /api/checkout and POST /api/refunds, the two routes that call the provider, spike together in the same hours while GET /api/orders/:id and the rest stay flat; inside a slow checkout the DB spans start immediately and take a few ms, CPU is flat. Goroutines rise during the windows because requests are waiting on the provider, not on the pool. Mitigations on our side are a tighter timeout for the charge call, a circuit breaker, an asynchronous "payment pending" flow with the idempotency key, and raising it with the provider.+ rubric:+ - Localizes the time to the external charge span and correlates the spikes with the provider (checkout and refunds spike together, routes that do not call it stay flat; the timeouts are on that call).+ - Rules out connection-pool exhaustion and saturation from the data (no gap before the DB spans, fast DB spans, flat CPU; goroutines are a symptom).+ - Proposes upstream-dependency mitigations (timeout budget, circuit breaker, async/pending payment with idempotency, provider follow-up) rather than DB or code tuning.+oracle:+ - shell: >-+ traceway endpoints list --since 3d --order-by p95 --page-size 20 --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration / 1e6 | round), p95_ms: (.p95Duration / 1e6 | round), p99_ms: (.p99Duration / 1e6 | round)}] | sort_by(-.p99_ms)'+ - shell: >-+ traceway endpoints chart --metric-type p99 --since 3d --interval-minutes 60 --output json+ | jq '.series | group_by(.endpoint) | map({endpoint: .[0].endpoint, over_1s: [.[] | select(.value > 1000) | {t: .timestamp, p99_ms: (.value | round)}]})'+ - shell: >-+ traceway exceptions list --since 3d --output json+ | jq '[.data[] | {exceptionHash, count, firstSeen, lastSeen, top: (.stackTrace | split("\n")[0])}]'+ - shell: >-+ traceway logs query --from {{seed.incidentFrom}} --to {{seed.incidentTo}} --search /api/checkout --page-size 20 --output json+ | jq -r '.data[] | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.incidentRequest.traceId}} --recorded-at {{seed.incidentRequest.recordedAt}} --output json+ | jq '.nodes[0] | {endpoint: .endpoint.endpoint, ms: (.endpoint.duration / 1e6), status: .endpoint.statusCode,+ spans: [.spans | sort_by(.startTime)[] | {name, url: .attributes["url.full"], start: .startTime, ms: (.duration / 1e6)}]}'+ - shell: traceway metrics query --name go.goroutine.count --aggregation max --since 3d --interval-minutes 60 --output json | jq '[.results[0].series[][] | select(.value > 150) | .timestamp]'+ - shell: traceway metrics query --name system.cpu.utilization --aggregation max --group-by state --since 3d --interval-minutes 720 --output json+ - read_file: internal/payments/client.go+ - submit:+ checklist_item: slow_third_party_upstream_api+ category: network+ locus: {endpoint: "POST /api/checkout", span: "POST https://api.payments.example/v1/charges", service: checkout-api}+ onset: null+ evidence:+ - {call_id: c01, claim: "POST /api/checkout and POST /api/refunds have p50 ~240ms but p99 of several seconds; the other routes are fast"}+ - {call_id: c02, claim: "checkout and refunds spike in the same three windows in three days; routes that do not call the provider stay flat"}+ - {call_id: c03, claim: "the timeout error group is the charge call to api.payments.example hitting the 10s client timeout, first and last seen inside the slow windows; any other group is unrelated"}+ - {call_id: c05, claim: "a slow checkout spends nearly all its time in POST https://api.payments.example/v1/charges; its DB spans start immediately and take a few ms"}+ - {call_id: c06, claim: "goroutines rise only during the same windows, as requests wait on the provider"}+ - {call_id: c07, claim: "CPU is flat"}+ ruled_out:+ - {checklist_item: connection_pool_exhaustion, call_id: c05, reason: "no gap before the first DB span and the DB spans are fast; the wait is in the provider call"}+ - {checklist_item: cpu_saturation, call_id: c07, reason: "CPU flat"}+ fix: Not our code, so contain it. Give the charge call a tighter timeout budget with a circuit breaker, move to an asynchronous 'payment pending' confirmation using the existing idempotency key when the provider is slow, and raise the incidents with the provider.+ confidence: high+ report: |+ The checkout hangs are the payment provider. Three times in the last three days, for 25-40 minutes, the charge call (POST https://api.payments.example/v1/charges) went from ~230ms to several seconds, and some calls hit our 10s client timeout, which is the "context deadline exceeded (Client.Timeout exceeded while awaiting headers)" error customers got as a 502. POST /api/refunds, the other route that calls the provider, spiked in exactly the same hours, while every route that does not call it stayed flat. Inside a slow checkout the database work starts immediately and takes a few milliseconds; the goroutine count rises only because requests are waiting on the provider, and CPU is flat. There is nothing to fix in our queries. We should contain it: a shorter timeout for the charge with a circuit breaker, a "payment pending" path that confirms asynchronously with the idempotency key we already send, and a ticket with the provider for those three windows.diff --git a/cli/test/evals/cases/perf/perf-xs-frontend-timeouts.yaml b/cli/test/evals/cases/perf/perf-xs-frontend-timeouts.yamlnew file mode 100644index 00000000..b6015c48--- /dev/null+++ b/cli/test/evals/cases/perf/perf-xs-frontend-timeouts.yaml@@ -0,0 +1,88 @@+id: perf-xs-frontend-timeouts+suite: perf+title: Web shop search times out in the browser; the API's search provider is slow+tags: [cross-surface, browser, react, node, network]+seed: perf/xs-frontend-timeouts+arms: [cli+skill, cli-noskill, mcp]+invoke: natural+distractor: A frontend bug. The web app gives up after 8 seconds (AbortSignal.timeout) and the issue's stack points into its search client, so raising the timeout or retrying in the browser looks like the fix. But every timed-out search lines up with a GET /api/search on shop-api that ran past 8s, and those requests spend nearly all their time in the POST to the hosted search index (shop-prod.search-cloud.example), which got slow about ten hours ago; the API's own database work stays at a few milliseconds and CPU is flat.+workspace: {repo: shop-web-and-api}+prompt: |+ Customers keep telling support that product search on the web shop spins for a while and then says it is taking too long. This is the issue Traceway shows for it: {{env.url}}/issues/{{seed.hash}}+ What is going on, and what should we change?+budget: {max_turns: 40, max_tool_calls: 32, max_output_tokens: 12000, timeout: 10m}+user:+ - default: "I only know what support told me. Please find out from the data."+expect:+ answer:+ checklist_item: slow_third_party_upstream_api+ not: [none_found, no_timeouts_aggressive_retries, large_request_response_payloads, cpu_saturation, missing_or_unused_index]+ locus: {endpoint: "GET /api/search", span_contains: "search-cloud.example"}+ grounding:+ - "signal timed out"+ - "search-cloud.example"+ calls:+ required:+ - {op: exceptions.show, args: {hash: "{{seed.hash}}"}}+ any_of:+ - [{op: "endpoints.*", args: {project_id: "{{env.projects.shop-api.id}}"}}]+ - [{op: "logs.query", args: {project_id: "{{env.projects.shop-api.id}}"}}]+ - [{op: projects.use, args: {project_id: "{{env.projects.shop-api.id}}"}}]+ forbidden:+ - {op: "*", write: true}+ max_failed_calls: 5+ max_identical_repeats: 2+judge:+ expected: >-+ The browser TimeoutError ("signal timed out") is the web app's own 8-second AbortSignal.timeout on the search+ fetch; the network action on each occurrence is GET /api/search taking 8000ms with no response. On the shop-api+ project, GET /api/search has been slow for about ten hours: p95 around 7-8s (it was a few hundred ms before), with+ a tail well past 8 seconds, and every browser timeout lines up with an API access log for a search that started 8s+ earlier and ran longer than that (often 10-15s, some 20-30s).+ Those requests spend nearly all their time in the outgoing POST to the hosted search index+ (shop-prod.search-cloud.example); the product_stock query and the merge take a few milliseconds and CPU is flat.+ The provider is slow, not the frontend and not our code. Fix: put a timeout under 8s on the index call in the API+ with a fallback (cached or degraded results), raise it with the provider, and do not raise the browser timeout.+ rubric:+ - Follows the browser issue to the backend project by the URL path and timestamps (the native SDK trace id does not join the backend trace) and quantifies GET /api/search on shop-api.+ - Shows the backend requests' time is in the external search-index call, not in the database or CPU, and that it started degrading hours ago.+ - Recommends a server-side timeout and fallback for the index call and escalating to the provider, rather than a longer browser timeout.+oracle:+ - shell: traceway exceptions show {{seed.hash}} --page-size 5 --output json+ - shell: traceway exceptions occurrence {{calls.c01.occurrences.0.id}} --recorded-at {{calls.c01.occurrences.0.recordedAt}} --output json+ - shell: traceway projects list --output json+ - shell: >-+ traceway endpoints list --project {{env.projects.shop-api.id}} --search /api/search --since 24h --output json+ | jq '[.data[] | {endpoint, count, p50_ms: (.p50Duration/1e6|round), p95_ms: (.p95Duration/1e6|round), p99_ms: (.p99Duration/1e6|round)}]'+ - shell: >-+ traceway endpoints chart --project {{env.projects.shop-api.id}} --metric-type p95 --since 24h --interval-minutes 60 --output json+ | jq '[.series[] | select(.endpoint == "GET /api/search") | {t: .timestamp, p95_ms: (.value|round)}]'+ - shell: >-+ traceway logs query --project {{env.projects.shop-api.id}} --search /api/search --since 12h --page-size 50 --output json+ | jq -r '.data[] | select((.logAttributes.duration_ms // "0" | tonumber) > 5000) | "\(.timestamp) \(.traceId) \(.body)"'+ - shell: >-+ traceway traces show {{seed.slowRequest.traceId}} --recorded-at {{seed.slowRequest.recordedAt}} --output json+ | jq '[.nodes[] | {project: .projectName, endpoint: .endpoint.endpoint, duration_ms: (.endpoint.duration/1e6|round),+ spans: [.spans[] | {name, ms: (.duration/1e6|round), url: .attributes["url.full"], server: .attributes["server.address"]}]}]'+ - shell: traceway metrics query --project {{env.projects.shop-api.id}} --name system.cpu.utilization --aggregation max --group-by state --since 12h --interval-minutes 120 --output json+ - read_file: web/src/api/search.ts+ - read_file: api/src/search.js+ - submit:+ checklist_item: slow_third_party_upstream_api+ category: network+ locus: {endpoint: "GET /api/search", span: "POST https://shop-prod.search-cloud.example/1/indexes/products/query", service: shop-api}+ onset: "{{seed.degradedAt}}"+ evidence:+ - {call_id: c02, claim: "the browser occurrence is a TimeoutError (signal timed out): the GET /api/search network action ran 8000ms with no response before the app gave up"}+ - {call_id: c04, claim: "on shop-api, GET /api/search has p95 in the seconds over the last day, with a tail past 8s"}+ - {call_id: c05, claim: "the endpoint's hourly p95 went from a few hundred ms to seconds about ten hours ago"}+ - {call_id: c06, claim: "the browser timeouts line up with API searches that started 8s earlier and took longer than that, often 10s or more"}+ - {call_id: c07, claim: "a search that ran past 8s spent nearly all of it in the POST to shop-prod.search-cloud.example; the stock query and merge took milliseconds"}+ - {call_id: c08, claim: "CPU on the API host is flat"}+ ruled_out:+ - {checklist_item: no_timeouts_aggressive_retries, call_id: c07, reason: "one provider call per request, no retries; the provider itself answers slowly"}+ - {checklist_item: cpu_saturation, call_id: c08, reason: "CPU flat"}+ fix: Put a client timeout below the browser's 8s on the search-index call in api/src/search.js with a fallback (cached or degraded results), escalate the latency to the search provider, and keep the browser timeout as it is.+ confidence: high+ report: |+ The browser timeouts are the symptom; the slowness is the hosted search provider behind the API. The web app aborts a search after 8 seconds (AbortSignal.timeout in web/src/api/search.ts) and reports the TimeoutError ("signal timed out"); each occurrence's network action is GET /api/search taking 8000ms with no response. On shop-api, GET /api/search got slow about ten hours ago (hourly p95 from a few hundred ms to seconds, tail past 8s), and each browser timeout lines up with an access log for a search that started 8 seconds earlier and took longer than that, often 10-15s. Those requests spend nearly all their time in the POST to shop-prod.search-cloud.example; the product_stock query and the merge take milliseconds and CPU is flat. Fix it on the server: give the index call a timeout below 8s with a fallback (cached or degraded results) in api/src/search.js, and take the latency up with the provider. Raising the browser timeout would only make shoppers wait longer.diff --git a/cli/test/evals/internal/scenarios/perf/app_cpu_serialization.go b/cli/test/evals/internal/scenarios/perf/app_cpu_serialization.gonew file mode 100644index 00000000..6e24d1af--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/app_cpu_serialization.go@@ -0,0 +1,114 @@+package perf++import (+ "fmt"+ "math"+ "sort"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/app-cpu-serialization", papSeedCPUSerialization) }++// papSeedCPUSerialization: the transaction export reads the rows quickly,+// then spends 0.3-7s building the JSON body in memory (a gap between the+// SELECT and the audit INSERT, with no span) before writing it out in a few+// milliseconds. CPU jumps while an export serializes. Distractor: the+// bodies are large (up to ~4.7MB, logged per export), but the time is spent+// before a single byte is sent, and the cost per row (~0.35ms) is CPU, not+// transfer.+func papSeedCPUSerialization(s *seed.Scenario) error {+ p := s.Project("ledger", seed.OTel)+ api := p.Service("accounts-api", seed.DotNet()).Versions("4.2.0")+ window := 7 * 24 * time.Hour++ var busy []papInterval+ var sorted bool+ export := api.Endpoint("GET", "/api/accounts/{accountId}/transactions/export")+ export.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(window))+ export.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(1.2, 2.8)) // JWT validation, model binding+ var account int+ path, _ := r.Request().Root.Attrs["url.path"].(string)+ fmt.Sscanf(path, "/api/accounts/%d/", &account)+ r.DB("SELECT id, number, currency, owner_id, balance, updated_at FROM accounts WHERE id = @AccountId", r.Ms(1.2, 3))+ rows := int(math.Min(20000, math.Max(300, float64(r.LogNormal(3400*time.Millisecond, 16000*time.Millisecond).Milliseconds()))))+ r.DB("SELECT t.id, t.posted_at, t.amount, t.currency, t.counterparty, t.reference, t.category, t.balance_after, t.metadata FROM transactions t WHERE t.account_id = @AccountId AND t.posted_at >= @From AND t.posted_at < @To ORDER BY t.posted_at",+ time.Duration(float64(rows)*0.009*(0.85+0.3*r.Float())*float64(time.Millisecond))+r.Ms(6, 14))+ ser := time.Duration(float64(rows) * 0.35 * (0.9 + 0.2*r.Float()) * float64(time.Millisecond))+ at := r.At()+ r.Gap(ser)+ // A host sample covers the minute before it, so it shows every+ // export that serialized in that minute.+ busy = append(busy, papInterval{at, at.Add(ser + time.Minute)})+ sorted = false+ bytes := rows*(205+r.IntN(30)) + 2048+ r.DB("INSERT INTO export_audit (account_id, requested_by, row_count, byte_count, created_at) VALUES (@AccountId, @UserId, @RowCount, @ByteCount, now())", r.Ms(1.5, 4))+ r.Log(seed.SevInfo, fmt.Sprintf("Exported %d transactions (%d bytes) for account %d", rows, bytes, account),+ seed.A("RowCount", rows), seed.A("ByteCount", bytes), seed.A("AccountId", account),+ seed.A("category.name", "Accounts.Api.Exports.TransactionExportEndpointsLog"))+ r.Gap(time.Duration(float64(bytes)/1e9*float64(time.Second)) + r.Ms(2, 6)) // write the body to Kestrel+ })++ page := api.Endpoint("GET", "/api/accounts/{accountId}/transactions")+ page.Traffic(seed.Daily(600), seed.Diurnal(), seed.Over(window))+ page.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(1.2, 2.8))+ r.DB("SELECT t.id, t.posted_at, t.amount, t.currency, t.counterparty, t.reference, t.category, t.balance_after, t.metadata FROM transactions t WHERE t.account_id = @AccountId ORDER BY t.posted_at DESC LIMIT @PageSize OFFSET @Offset",+ r.Ms(3, 9))+ r.Gap(r.Ms(1.5, 3.5)) // 50 rows through a cached JsonSerializerOptions+ })++ acct := api.Endpoint("GET", "/api/accounts/{accountId}")+ acct.Traffic(seed.Daily(450), seed.Diurnal(), seed.Over(window))+ acct.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(1.2, 2.8))+ r.DB("SELECT id, number, currency, owner_id, balance, updated_at FROM accounts WHERE id = @AccountId", r.Ms(1.2, 3))+ r.Gap(r.Ms(0.3, 0.8))+ })++ s.Background(api, seed.TypicalTraffic(6))++ // Two vCPUs; a serializing export keeps one busy. Exports rarely+ // overlap, so a sample shows one export's worth.+ serLoad := func(t time.Time) float64 {+ if !sorted {+ sort.Slice(busy, func(i, j int) bool { return busy[i].from.Before(busy[j].from) })+ sorted = true+ }+ return 0.48 * math.Min(1, float64(papActive(busy, t, 90*time.Second)))+ }+ api.Host().CPU(seed.Plus(seed.Flat(0.12, 0.2), serLoad))+ api.Host().Memory(seed.Flat(0.46, 0.55))++ // The slowest export of the day, leaving out the last minutes, which no+ // host sample covers yet.+ var slow *seed.Request+ slowest := func() *seed.Request {+ if slow != nil {+ return slow+ }+ for _, rq := range export.Requests() {+ if rq.Start.Before(s.Ago(24*time.Hour)) || rq.Start.After(s.Ago(3*time.Minute)) {+ continue+ }+ if slow == nil || rq.Duration() > slow.Duration() {+ slow = rq+ }+ }+ slow.KeepDetail()+ return slow+ }+ s.Expose("slowRequest", func() any { return slowest().Ref() })+ s.Expose("slowFrom", func() any {+ return slowest().Start.Add(-10 * time.Minute).Truncate(time.Minute).Format(time.RFC3339)+ })+ s.Expose("slowTo", func() any {+ return slowest().Start.Add(11 * time.Minute).Truncate(time.Minute).Format(time.RFC3339)+ })+ s.Expose("p50Ms24h", func() any { return export.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return export.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/app_excessive_logging.go b/cli/test/evals/internal/scenarios/perf/app_excessive_logging.gonew file mode 100644index 00000000..2d7c1070--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/app_excessive_logging.go@@ -0,0 +1,134 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/app-excessive-logging", papSeedExcessiveLogging) }++// papSeedExcessiveLogging: release 5.3.1 of listings-api, deployed 26h ago,+// only switches LOG_LEVEL to debug. From then on every request writes ~15-20x+// as many log lines (one per decoded row and resolved image), each costing+// ~0.5ms on the request path, so every endpoint gets slower at the deploy and+// host CPU steps from ~35% to ~78% five minutes later, once the rollout has+// finished. Traffic is flat. Distractor: CPU saturation; but the CPU rise+// follows the deploy, request counts did not change, and the extra time per+// request is spent writing logs.+func papSeedExcessiveLogging(s *seed.Scenario) error {+ p := s.Project("marketplace", seed.OTel)+ deployAt := s.Ago(26*time.Hour - 9*time.Minute)+ api := p.Service("listings-api", seed.Go("nethttp", "pgx")).+ Versions("5.3.0", seed.Deploy{At: deployAt, Version: "5.3.1"}).+ Instances("listings-api-6c9f7d8b5-k2x4q", "listings-api-6c9f7d8b5-p8n2m")+ window := 48 * time.Hour+ debug := func(r *seed.Req) bool { return !r.Start().Before(deployAt) }+ // logLine writes a debug line when the release logs at debug level, and+ // pays its cost on the request path.+ logLine := func(r *seed.Req, body string, attrs ...seed.KV) {+ if debug(r) {+ r.Log(seed.SevDebug, body, attrs...)+ r.Gap(r.Ms(0.35, 0.75))+ }+ }++ search := api.Endpoint("GET", "/api/listings")+ search.Traffic(papJitter("search", 0.07), seed.Daily(650), seed.Diurnal(), seed.Over(window))+ search.Handler(func(r *seed.Req) {+ r.KeepDetail() // every request has its access log line+ r.Gap(r.Ms(0.4, 1.0))+ category := 100 + r.IntN(400)+ key := fmt.Sprintf("listings:search:%d:%x", category, r.IntN(1<<20))+ logLine(r, "cache lookup", seed.A("cache.key", key))+ r.Cache("GET", key, r.Ms(0.3, 0.9))+ rows := 36 + r.IntN(9)+ r.DB("SELECT id, seller_id, title, price_cents, currency, thumbnail_url, ranked_at FROM listings WHERE category_id = $1 AND status = 'active' ORDER BY ranked_at DESC LIMIT 44",+ r.Ms(6, 15), seed.Rows(rows))+ for i := 0; i < rows; i++ {+ logLine(r, "decoded listing", seed.A("listing.id", 300000+r.IntN(600000)), seed.A("seller.id", 1000+r.IntN(90000)))+ }+ logLine(r, "query executed", seed.A("db.table", "listings"), seed.A("rows", rows))+ r.Gap(r.Ms(0.3, 0.9)) // marshal the page for the cache+ r.Cache("SET", key, r.Ms(0.3, 0.9))+ r.Log(seed.SevInfo, "search served", seed.A("category.id", category), seed.A("results", rows))+ r.Gap(r.Ms(0.5, 1.5)) // JSON encode+ })++ show := api.Endpoint("GET", "/api/listings/{id}")+ show.Traffic(papJitter("show", 0.07), seed.Daily(600), seed.Diurnal(), seed.Over(window))+ show.Handler(func(r *seed.Req) {+ r.KeepDetail()+ r.Gap(r.Ms(0.4, 1.0))+ var id int+ path, _ := r.Request().Root.Attrs["url.path"].(string)+ fmt.Sscanf(path, "/api/listings/%d", &id)+ logLine(r, "cache lookup", seed.A("cache.key", fmt.Sprintf("listing:%d", id)))+ r.Cache("GET", fmt.Sprintf("listing:%d", id), r.Ms(0.3, 0.9))+ r.DB("SELECT id, seller_id, title, description, price_cents, currency, status, ranked_at FROM listings WHERE id = $1", r.Ms(1.5, 4), seed.Rows(1))+ logLine(r, "query executed", seed.A("db.table", "listings"), seed.A("rows", 1))+ images := 12 + r.IntN(10)+ r.DB("SELECT url, width, height, position FROM listing_images WHERE listing_id = $1 ORDER BY position", r.Ms(1.5, 4), seed.Rows(images))+ logLine(r, "query executed", seed.A("db.table", "listing_images"), seed.A("rows", images))+ logLine(r, "decoded listing", seed.A("listing.id", id))+ for i := 0; i < images; i++ {+ logLine(r, "image url resolved", seed.A("listing.id", id), seed.A("position", i))+ }+ r.Gap(r.Ms(0.5, 1.5))+ })++ views := api.Endpoint("POST", "/api/listings/{id}/views")+ views.Traffic(papJitter("views", 0.07), seed.Daily(350), seed.Diurnal(), seed.Over(window))+ views.Handler(func(r *seed.Req) {+ r.KeepDetail()+ r.Gap(r.Ms(0.3, 0.8))+ logLine(r, "recording view")+ r.DB("INSERT INTO listing_views (listing_id, viewer_id, viewed_at) VALUES ($1, $2, now())", r.Ms(1.5, 4))+ logLine(r, "query executed", seed.A("db.table", "listing_views"), seed.A("rows", 1))+ r.Status(204)+ })++ cats := api.Endpoint("GET", "/api/categories")+ cats.Traffic(papJitter("cats", 0.07), seed.Daily(200), seed.Diurnal(), seed.Over(window))+ cats.Handler(func(r *seed.Req) {+ r.KeepDetail()+ r.Gap(r.Ms(0.3, 0.8))+ logLine(r, "cache lookup", seed.A("cache.key", "categories:v3"))+ r.Cache("GET", "categories:v3", r.Ms(0.3, 0.8))+ logLine(r, "cache hit", seed.A("cache.key", "categories:v3"))+ r.Gap(r.Ms(0.6, 1.6))+ })++ rolled := deployAt.Add(5 * time.Minute)+ for i, h := range api.Hosts {+ api.HostNamed(h).CPU(seed.Step(rolled, seed.Flat(0.3, 0.38), seed.Flat(0.74, 0.82)))+ api.HostNamed(h).Memory(seed.Flat(0.38, 0.43))+ api.LogOn(h, deployAt.Add(time.Duration(i)*2*time.Minute+40*time.Second), seed.SevInfo, "starting listings-api",+ seed.A("version", "5.3.1"), seed.A("log.level", "debug"), seed.A("addr", ":8080"))+ }++ s.Expose("deployAt", deployAt.Format(time.RFC3339))+ // Windows at the same time of day before and after the release, so the+ // diurnal traffic curve does not confound the comparison.+ rfc := func(t time.Time) string { return t.Format(time.RFC3339) }+ s.Expose("beforeFrom", rfc(deployAt.Add(-6*time.Hour)))+ s.Expose("afterFrom", rfc(deployAt.Add(18*time.Hour)))+ s.Expose("afterTo", rfc(deployAt.Add(24*time.Hour)))+ s.Expose("beforeHourFrom", rfc(deployAt.Add(-3*time.Hour)))+ s.Expose("beforeHourTo", rfc(deployAt.Add(-2*time.Hour)))+ s.Expose("afterHourFrom", rfc(deployAt.Add(21*time.Hour)))+ s.Expose("afterHourTo", rfc(deployAt.Add(22*time.Hour)))+ s.Expose("beforeRequest", search.Pick(func(rq *seed.Request) bool {+ return rq.Start.Before(deployAt.Add(-2*time.Hour)) && rq.Start.After(deployAt.Add(-4*time.Hour))+ }))+ s.Expose("afterRequest", search.Pick(func(rq *seed.Request) bool { return rq.Start.After(s.Ago(6 * time.Hour)) }))+ s.Expose("searchP95Before", func() any {+ return search.PercentileBetween(0.95, deployAt.Add(-6*time.Hour), deployAt).Seconds() * 1000+ })+ s.Expose("searchP95After", func() any {+ return search.PercentileBetween(0.95, deployAt.Add(18*time.Hour), deployAt.Add(24*time.Hour)).Seconds() * 1000+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/app_large_alloc.go b/cli/test/evals/internal/scenarios/perf/app_large_alloc.gonew file mode 100644index 00000000..108ac70c--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/app_large_alloc.go@@ -0,0 +1,171 @@+package perf++import (+ "fmt"+ "math"+ "sort"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/app-large-alloc", papSeedLargeAlloc) }++// papWeighted is an interval of in-memory work with the rows it holds.+type papWeighted struct {+ from, to time.Time+ rows float64+}++// papSeedLargeAlloc: GET /api/insights/retention loads every event of the+// requested range into memory (0.1-2.5M rows) and builds the cohort table in+// Java. While one runs, the G1 old generation jumps by ~1.4KB per row and+// young collections run back to back, then the heap falls back to its+// baseline. Distractor: GC as the root cause; but the collections only+// happen while this endpoint runs, they add up to a small share of its time,+// the heap after GC returns to ~0.6GB (no leak), and every other endpoint on+// the JVM keeps its normal p99.+func papSeedLargeAlloc(s *seed.Scenario) error {+ p := s.Project("insights", seed.OTel)+ api := p.Service("insights-api", seed.Java("spring")).Versions("2.31.0")+ window := 7 * 24 * time.Hour++ var work []papWeighted+ var sorted bool+ workOf := map[*seed.Request]papWeighted{}+ daysOf := map[*seed.Request]int{}+ ranges := []int{30, 90, 180}+ retention := api.Endpoint("GET", "/api/insights/retention")+ retention.Traffic(seed.Daily(300), seed.Diurnal(), seed.Over(window))+ retention.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.1, Max: 30})+ retention.Handler(func(r *seed.Req) {+ days := ranges[r.IntN(len(ranges))]+ daysOf[r.Request()] = days+ workspace := 1000 + r.IntN(8000)+ end := r.Start().UTC().Truncate(24 * time.Hour)+ r.RootAttr("url.query", fmt.Sprintf("workspace=%d&from=%s&to=%s&interval=week", workspace,+ end.AddDate(0, 0, -days).Format("2006-01-02"), end.Format("2006-01-02")))+ size := float64(r.LogNormal(350*time.Millisecond, 1700*time.Millisecond).Milliseconds()) * 1000+ rows := math.Min(2.6e6, math.Max(40000, size*float64(days)/90))+ r.Gap(r.Ms(0.8, 2))+ var w papWeighted+ spanAt := r.At() // Req.At inside Span counts from the span's start+ r.Span("RetentionService.buildCohorts", func(r *seed.Req) {+ r.DB("SELECT id, name, timezone FROM workspaces WHERE id = ?", r.Ms(0.8, 2))+ fetch := time.Duration((0.15 + rows*0.9e-6) * (0.9 + 0.2*r.Float()) * float64(time.Second))+ start := spanAt.Add(r.At().Sub(r.Start()))+ r.DB("SELECT user_id, event_name, occurred_at, properties FROM events WHERE workspace_id = ? AND occurred_at >= ? AND occurred_at < ? ORDER BY occurred_at", fetch)+ build := time.Duration(rows * 3.2e-6 * (0.9 + 0.2*r.Float()) * float64(time.Second))+ r.Gap(build) // group by user, sort, bucket into cohorts+ w = papWeighted{start, start.Add(fetch + build), rows}+ }, seed.A("code.function", "buildCohorts"), seed.A("code.namespace", "com.acme.insights.retention.RetentionService"))+ work = append(work, w)+ workOf[r.Request()] = w+ sorted = false+ r.Gap(r.Ms(2, 6)) // Jackson writes the 26x26 cohort table+ })++ funnels := api.Endpoint("GET", "/api/insights/funnels")+ funnels.Traffic(seed.Daily(300), seed.Diurnal(), seed.Over(window))+ funnels.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 2))+ r.DB("SELECT step, count(DISTINCT user_id) FROM funnel_steps WHERE workspace_id = ? AND funnel_id = ? AND day >= ? GROUP BY step ORDER BY step", r.LogNormal(60*time.Millisecond, 180*time.Millisecond))+ r.Gap(r.Ms(1, 3))+ })++ dash := api.Endpoint("GET", "/api/insights/dashboards/{id}")+ dash.Traffic(seed.Daily(500), seed.Diurnal(), seed.Over(window))+ dash.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 2))+ r.DB("SELECT id, name, layout, updated_at FROM dashboards WHERE id = ? AND workspace_id = ?", r.Ms(1, 3))+ r.DB("SELECT id, kind, query, position FROM dashboard_cards WHERE dashboard_id = ? ORDER BY position", r.Ms(1.5, 4))+ r.Gap(r.Ms(1, 2.5))+ })++ s.Background(api, seed.TypicalTraffic(6))++ ensure := func() {+ if !sorted {+ sort.Slice(work, func(i, j int) bool { return work[i].from.Before(work[j].from) })+ sorted = true+ }+ }+ held := func(t time.Time) float64 { // rows in memory at t+ ensure()+ i := sort.Search(len(work), func(i int) bool { return !work[i].from.Before(t.Add(-time.Minute)) })+ v := 0.0+ for ; i < len(work) && !work[i].from.After(t); i++ {+ if t.Before(work[i].to) {+ v += work[i].rows+ }+ }+ return v+ }+ churn := func(t time.Time) float64 { // million-row-seconds of work in the minute before t+ ensure()+ lo := t.Add(-time.Minute)+ i := sort.Search(len(work), func(i int) bool { return !work[i].from.Before(lo.Add(-time.Minute)) })+ v := 0.0+ for ; i < len(work) && work[i].from.Before(t); i++ {+ a, b := work[i].from, work[i].to+ if a.Before(lo) {+ a = lo+ }+ if b.After(t) {+ b = t+ }+ if b.After(a) {+ v += b.Sub(a).Seconds() * work[i].rows / 1e6+ }+ }+ return v+ }++ pool := func(name string) map[string]any {+ return map[string]any{"jvm.memory.type": "heap", "jvm.memory.pool.name": name}+ }+ old := func(t time.Time) float64 { return math.Min(3.7e9, 6.2e8+held(t)*1400) }+ api.Gauge("jvm.memory.used", "By", pool("G1 Old Gen"), seed.Plus(seed.Flat(-2e7, 2e7), old))+ api.Gauge("jvm.memory.used", "By", pool("G1 Eden Space"), seed.Flat(1.2e8, 7.8e8))+ api.Gauge("jvm.memory.used", "By", pool("G1 Survivor Space"), seed.Plus(seed.Flat(1e7, 3e7), func(t time.Time) float64 { return math.Min(1.2e8, held(t)*40) }))+ api.Gauge("jvm.memory.used_after_last_gc", "By", pool("G1 Old Gen"), seed.Plus(seed.Flat(5.9e8, 6.4e8), func(t time.Time) float64 { return held(t) * 1100 }))+ api.Gauge("jvm.memory.limit", "By", pool("G1 Old Gen"), seed.Const(4.0e9))+ young := map[string]any{"jvm.gc.name": "G1 Young Generation", "jvm.gc.action": "end of minor GC"}+ api.Histogram("jvm.gc.duration", "s", young,+ func(t time.Time) float64 { return 0.008 + 0.0022*math.Min(14, churn(t)) },+ func(t time.Time) float64 { return 3 + 1.2*churn(t) })+ api.Host().CPU(seed.Plus(seed.Flat(0.18, 0.26), func(t time.Time) float64 { return math.Min(0.4, 0.35*math.Min(1, held(t)/1e6)) }))+ api.Host().Memory(seed.Flat(0.71, 0.74)) // the JVM reserves its 4GB heap up front++ var sampled *seed.Request+ slowSampled := func() *seed.Request {+ if sampled != nil {+ return sampled+ }+ for _, rq := range retention.Requests() {+ w := workOf[rq]+ if rq.Start.Before(s.Ago(24*time.Hour)) || w.from.Truncate(time.Minute).Equal(w.to.Truncate(time.Minute)) {+ continue+ }+ if sampled == nil || rq.Duration() > sampled.Duration() {+ sampled = rq+ }+ }+ sampled.KeepDetail()+ return sampled+ }+ s.Expose("slowRequest", func() any { return slowSampled().Ref() })+ s.Expose("slowFrom", func() any {+ return slowSampled().Start.Add(-10 * time.Minute).Truncate(time.Minute).Format(time.RFC3339)+ })+ s.Expose("slowTo", func() any {+ return slowSampled().Start.Add(11 * time.Minute).Truncate(time.Minute).Format(time.RFC3339)+ })+ s.Expose("fastRequest", retention.Pick(func(rq *seed.Request) bool {+ return rq.Start.After(s.Ago(24*time.Hour)) && daysOf[rq] == 30 && rq.Duration() < 900*time.Millisecond+ }))+ s.Expose("p50Ms24h", func() any { return retention.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p99Ms24h", func() any { return retention.Percentile(0.99, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/app_quadratic.go b/cli/test/evals/internal/scenarios/perf/app_quadratic.gonew file mode 100644index 00000000..c505850c--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/app_quadratic.go@@ -0,0 +1,69 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/app-quadratic", papSeedQuadratic) }++// papSeedQuadratic: POST /api/quotes prices a cart; the apply_promotions span+// costs ~0.011ms x items^2, so a 40-item cart takes ~20ms and a 350-item+// wholesale cart well over a second. The two queries are fast and constant.+// Distractor: a large in-memory allocation; but process RSS and host memory+// stay flat, and the time is pure compute that grows with the square of+// cart.items, not with any data volume the service holds.+func papSeedQuadratic(s *seed.Scenario) error {+ p := s.Project("shop", seed.OTel)+ api := p.Service("pricing-service", seed.Python("fastapi", "psycopg")).Versions("3.8.1")+ window := 3 * 24 * time.Hour++ quotes := api.Endpoint("POST", "/api/quotes")+ quotes.Traffic(seed.Daily(1700), seed.Diurnal(), seed.Over(window))+ quotes.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.04, Max: 50})+ quotes.Handler(func(r *seed.Req) {+ items := int(math.Min(450, math.Max(1, float64(r.LogNormal(12*time.Millisecond, 330*time.Millisecond).Milliseconds()))))+ n := float64(items)+ r.RootAttr("cart.items", items)+ r.Gap(r.Ms(0.6, 1.4) + time.Duration(n*0.012*float64(time.Millisecond))) // request validation+ r.DB("SELECT sku, family, unit_price_cents, tax_class FROM prices WHERE sku = ANY(%(skus)s)",+ r.Ms(2.5, 6)+time.Duration(n*0.01*float64(time.Millisecond)), seed.Rows(items))+ r.DB("SELECT id, kind, family, sku, partner_sku, min_quantity, percent, amount_cents FROM promotions WHERE active AND starts_at <= now() AND ends_at > now()",+ r.Ms(1.5, 3.5), seed.Rows(18))+ promo := time.Duration((0.011*n*n + 0.04*n) * (0.92 + 0.16*r.Float()) * float64(time.Millisecond))+ r.Compute("apply_promotions", promo, seed.A("promotions.active", 18))+ r.Compute("calculate_tax", r.Ms(0.2, 0.6)+time.Duration(n*0.004*float64(time.Millisecond)))+ r.Gap(r.Ms(0.4, 1.2) + time.Duration(n*0.006*float64(time.Millisecond))) // response model+ })++ carts := api.Endpoint("GET", "/api/price-lists/{price_list_id}")+ carts.Traffic(seed.Daily(700), seed.Diurnal(), seed.Over(window))+ carts.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("SELECT sku, family, unit_price_cents, tax_class FROM prices WHERE price_list_id = %(price_list_id)s ORDER BY sku LIMIT 200", r.Ms(4, 11), seed.Rows(200))+ r.Gap(r.Ms(1.5, 3.5))+ })++ s.Background(api, seed.TypicalTraffic(8))+ api.Host().CPU(seed.Flat(0.21, 0.31))+ api.Host().Memory(seed.Flat(0.44, 0.49))+ api.Gauge("process.runtime.cpython.memory", "By", map[string]any{"type": "rss"}, seed.Flat(212e6, 228e6))+ api.Gauge("process.runtime.cpython.memory", "By", map[string]any{"type": "vms"}, seed.Flat(612e6, 640e6))++ pick := func(lo, hi int) func() any {+ return quotes.Pick(func(rq *seed.Request) bool {+ v, _ := rq.Root.Attrs["cart.items"].(int)+ return rq.Start.After(s.Ago(12*time.Hour)) && v >= lo && v <= hi+ })+ }+ s.Expose("bigQuote", pick(330, 450))+ s.Expose("midQuote", pick(150, 190))+ s.Expose("smallQuote", pick(35, 50))+ s.Expose("p50Ms24h", func() any { return quotes.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return quotes.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p99Ms24h", func() any { return quotes.Percentile(0.99, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/cache_cold_start.go b/cli/test/evals/internal/scenarios/perf/cache_cold_start.gonew file mode 100644index 00000000..7b2eaaf0--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/cache_cold_start.go@@ -0,0 +1,151 @@+package perf++import (+ "fmt"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/cache-cold-start", pncSeedColdStart) }++const pncAnnotationsScope = "io.opentelemetry.opentelemetry-instrumentation-annotations-1.16"++// pncSeedColdStart: pricing-service (Spring Boot) runs with+// spring.main.lazy-initialization=true, so a new pod starts in seconds but+// its first request creates the web and data beans and the Hikari pool+// (5-8s), and the next few run cold, interpreted code. Every release, on+// every pod, the first requests are outliers and latency is back to normal+// after a dozen requests. Distractor: the latest release (2.16.0, ~3h ago)+// rewrote the promotion rules, which invites a regression reading, but the+// same outliers follow the older releases and steady-state latency did not+// move.+func pncSeedColdStart(s *seed.Scenario) error {+ window := 10 * 24 * time.Hour+ deploys := []seed.Deploy{+ {At: s.Ago(216*time.Hour + 14*time.Minute), Version: "2.14.1"},+ {At: s.Ago(144*time.Hour + 41*time.Minute), Version: "2.15.0"},+ {At: s.Ago(48*time.Hour + 6*time.Minute), Version: "2.15.1"},+ {At: s.Ago(3*time.Hour + 2*time.Minute), Version: "2.16.0"},+ }+ p := s.Project("pricing", seed.OTel)+ svc := p.Service("pricing-service", seed.Java("spring-boot")).Versions("2.14.0", deploys...).+ Instances("pricing-service-0", "pricing-service-1", "pricing-service-2")++ deployIndex := func(t time.Time) int {+ i := -1+ for k, d := range deploys {+ if !t.Before(d.At) {+ i = k+ }+ }+ return i+ }+ served := map[string]int{} // deploy|host -> requests served since the pod became ready+ readyAt := map[string]time.Time{} // deploy|host -> when the new pod reported started+ ordinal := map[*seed.Request]int{}++ quote := svc.Endpoint("POST", "/api/prices/quote")+ quote.Traffic(seed.Daily(1500), seed.Diurnal(), seed.Over(window))+ quote.Detail(seed.DetailPolicy{Recent: 2 * time.Hour, Rate: 0.03, Max: 8})+ quote.Handler(func(r *seed.Req) {+ // Pods that were already running when the window opened, and old pods+ // still serving while their replacement starts, are warm.+ n := 1000+ if di := deployIndex(r.Start()); di >= 0 {+ key := fmt.Sprintf("%d|%s", di, r.Host())+ if !r.Start().Before(readyAt[key]) {+ served[key]+++ n = served[key]+ }+ }+ ordinal[r.Request()] = n+ lines := 1 + r.IntN(8)+ engine := r.LogNormal(9*time.Millisecond, 22*time.Millisecond)+ dbScale := 1.0+ thread := seed.A("thread.name", "http-nio-8080-exec-1")+ if n == 1 {+ r.KeepDetail()+ r.Log(seed.SevInfo, "Initializing Spring DispatcherServlet 'dispatcherServlet'", thread)+ r.Log(seed.SevInfo, "Initializing Servlet 'dispatcherServlet'", thread)+ d := r.Ms(900, 1400)+ r.Gap(d)+ r.Log(seed.SevInfo, fmt.Sprintf("Completed initialization in %d ms", d.Milliseconds()), thread)+ r.Gap(r.Ms(2000, 3200)) // lazy bean creation for the quote path+ engine = r.Ms(700, 1300)+ dbScale = 6+ } else if n <= 3 {+ engine += r.Ms(350, 800)+ dbScale = 2.5+ } else if n <= 8 {+ engine += time.Duration(float64(r.Ms(60, 160)) * float64(9-n) / 5)+ }+ r.Gap(r.Ms(0.5, 1.5))+ eng := r.Span("PricingEngine.quote", func(r *seed.Req) {+ if n == 1 {+ // The pool starts on the first getConnection, inside quote().+ r.Log(seed.SevInfo, "HikariPool-1 - Starting...", thread)+ r.Gap(r.Ms(600, 1100))+ r.Log(seed.SevInfo, fmt.Sprintf("HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@%x", 0x10000000+r.IntN(0x6fffffff)), thread)+ r.Log(seed.SevInfo, "HikariPool-1 - Start completed.", thread)+ }+ r.DB("SELECT sku, list_price_minor FROM price_list_items WHERE price_list_id = ? AND sku IN ("+strings.TrimSuffix(strings.Repeat("?, ", lines), ", ")+")",+ time.Duration(float64(r.LogNormal(4*time.Millisecond, 11*time.Millisecond))*dbScale), seed.Rows(lines))+ r.DB("SELECT id, kind, priority, conditions, effect FROM promotion_rules WHERE active AND ? BETWEEN starts_at AND ends_at ORDER BY priority",+ time.Duration(float64(r.LogNormal(3*time.Millisecond, 8*time.Millisecond))*dbScale), seed.Rows(30+r.IntN(20)))+ r.Gap(engine)+ }, seed.A("code.function", "quote"), seed.A("code.namespace", "com.example.pricing.engine.PricingEngine"))+ eng.Scope = pncAnnotationsScope+ r.Gap(r.Ms(0.5, 1.5))+ })++ item := svc.Endpoint("GET", "/api/prices/:sku")+ item.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(window))+ item.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.5))+ r.DB("SELECT sku, list_price_minor, currency FROM price_list_items WHERE price_list_id = ? AND sku = ?", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))+ r.Gap(r.Ms(0.5, 1.5))+ })++ // Startup lines of every pod at every release (no request context);+ // the new pods take traffic as soon as they report started.+ for i, d := range append([]seed.Deploy{{At: s.Ago(window + time.Hour), Version: "2.14.0"}}, deploys...) {+ for k, h := range svc.Hosts {+ at := d.At.Add(time.Duration(k*3) * time.Second)+ secs := 6.4 + 0.9*s.RNG.Float64()+ readyAt[fmt.Sprintf("%d|%s", i-1, h)] = at.Add(time.Duration(secs * float64(time.Second)))+ svc.LogOn(h, at, seed.SevInfo, fmt.Sprintf("Starting PricingServiceApplication v%s using Java 21.0.4 with PID 1 (/app/app.jar started by app in /app)", d.Version), seed.A("thread.name", "main"))+ svc.LogOn(h, at.Add(1200*time.Millisecond), seed.SevInfo, "Tomcat initialized with port 8080 (http)", seed.A("thread.name", "main"))+ svc.LogOn(h, at.Add(time.Duration(secs*float64(time.Second))), seed.SevInfo, fmt.Sprintf("Started PricingServiceApplication in %.3f seconds (process running for %.3f)", secs, secs+0.7), seed.A("thread.name", "main"))+ }+ }++ s.Background(svc, seed.TrafficOpts{Endpoints: 2, PerDay: 60})+ for _, h := range svc.Hosts {+ svc.HostNamed(h).CPU(seed.Flat(0.16, 0.26))+ svc.HostNamed(h).Memory(seed.Flat(0.55, 0.62))+ }++ s.Expose("coldRequest", quote.Pick(func(r *seed.Request) bool {+ return ordinal[r] == 1 && !r.Start.Before(deploys[3].At)+ }))+ s.Expose("olderColdRequest", quote.Pick(func(r *seed.Request) bool {+ return ordinal[r] == 1 && !r.Start.Before(deploys[2].At) && r.Start.Before(deploys[3].At)+ }))+ for i, d := range deploys {+ s.Expose(fmt.Sprintf("deploy%d", i+1), d.At.UTC().Format(time.RFC3339))+ }+ s.Expose("beforeFrom", deploys[3].At.Add(-24*time.Hour).UTC().Format(time.RFC3339))+ s.Expose("afterFrom", deploys[3].At.Add(time.Hour).UTC().Format(time.RFC3339))+ s.Expose("p50MsBefore", func() any {+ return quote.PercentileBetween(0.5, deploys[3].At.Add(-24*time.Hour), deploys[3].At).Seconds() * 1000+ })+ s.Expose("p50MsAfter", func() any { return quote.PercentileBetween(0.5, deploys[3].At.Add(time.Hour), s.Now).Seconds() * 1000 })+ s.Expose("p95MsBefore", func() any {+ return quote.PercentileBetween(0.95, deploys[3].At.Add(-24*time.Hour), deploys[3].At).Seconds() * 1000+ })+ s.Expose("p95MsAfter", func() any { return quote.PercentileBetween(0.95, deploys[3].At.Add(time.Hour), s.Now).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/infra_disk_io.go b/cli/test/evals/internal/scenarios/perf/infra_disk_io.gonew file mode 100644index 00000000..9458a5e7--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/infra_disk_io.go@@ -0,0 +1,114 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/infra-disk-io", papSeedDiskIO) }++// papSeedDiskIO: ~9h40m ago a Kubernetes Job started rebuilding the+// search_documents table on the bookings database. Ever since, the+// database host's disk runs at its limit (reads 15 -> ~110MB/s, writes 8 ->+// ~55MB/s, ~35 operations queued, iowait ~2% -> ~35%) while its CPU stays+// low, and every query of bookings-api, indexed primary-key lookups+// included, takes 5-8x longer. The API's own CPU is flat. Distractor: a+// missing index on the slowest query (the availability search); but every+// query slowed by the same factor at the same minute, with no deploy of the+// API and no growth in its tables.+func papSeedDiskIO(s *seed.Scenario) error {+ p := s.Project("bookings", seed.OTel)+ api := p.Service("bookings-api", seed.Node("express", "pg")).Versions("6.4.1")+ db := p.Service("bookings-db", seed.Go("nethttp", "pgx")).Instances("pg-bookings-1")+ window := 24 * time.Hour+ onset := s.Ago(9*time.Hour + 40*time.Minute)++ slow := func(r *seed.Req, d time.Duration, write bool) time.Duration {+ if r.Start().Before(onset) {+ return d+ }+ f := 4.5 + 3*r.Float()+ if write {+ f += 2 // commits wait on the WAL fsync too+ }+ return time.Duration(float64(d) * f)+ }+ detail := seed.DetailPolicy{Recent: 20 * time.Hour, Rate: 0.03, Max: 30}++ avail := api.Endpoint("GET", "/api/availability")+ avail.Traffic(papJitter("avail", 0.06), seed.Daily(2200), seed.Diurnal(), seed.Over(window))+ avail.Detail(detail)+ avail.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.2))+ r.DB("SELECT listing_id, day, units_free, price_cents FROM availability WHERE listing_id = ANY($1) AND day BETWEEN $2 AND $3 ORDER BY listing_id, day",+ slow(r, r.Ms(7, 16), false), seed.Rows(120+r.IntN(400)))+ r.Gap(r.Ms(1.5, 4))+ })++ booking := api.Endpoint("GET", "/api/bookings/:id")+ booking.Traffic(papJitter("booking", 0.06), seed.Daily(1300), seed.Diurnal(), seed.Over(window))+ booking.Detail(detail)+ booking.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.2))+ r.DB("SELECT id, listing_id, guest_id, check_in, check_out, status, total_cents FROM bookings WHERE id = $1", slow(r, r.Ms(1, 2.8), false), seed.Rows(1))+ r.DB("SELECT id, title, city, host_id FROM listings WHERE id = $1", slow(r, r.Ms(0.9, 2.5), false), seed.Rows(1))+ r.Gap(r.Ms(0.3, 0.9))+ })++ create := api.Endpoint("POST", "/api/bookings")+ create.Traffic(papJitter("create", 0.06), seed.Daily(500), seed.Diurnal(), seed.Over(window))+ create.Detail(detail)+ create.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.5))+ r.DB("BEGIN", r.Ms(0.2, 0.5))+ r.DB("SELECT units_free FROM availability WHERE listing_id = $1 AND day >= $2 AND day < $3 FOR UPDATE", slow(r, r.Ms(1.5, 4), false))+ r.DB("UPDATE availability SET units_free = units_free - 1 WHERE listing_id = $1 AND day >= $2 AND day < $3", slow(r, r.Ms(1.5, 4), true))+ r.DB("INSERT INTO bookings (id, listing_id, guest_id, check_in, check_out, status, total_cents, created_at) VALUES ($1, $2, $3, $4, $5, 'confirmed', $6, now())", slow(r, r.Ms(1.2, 3), true))+ r.DB("COMMIT", slow(r, r.Ms(1.5, 4), true))+ r.Status(201)+ })++ reviews := api.Endpoint("GET", "/api/listings/:id/reviews")+ reviews.Traffic(papJitter("reviews", 0.06), seed.Daily(900), seed.Diurnal(), seed.Over(window))+ reviews.Detail(detail)+ reviews.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.2))+ r.DB("SELECT id, author_name, rating, body, created_at FROM reviews WHERE listing_id = $1 ORDER BY created_at DESC LIMIT 20", slow(r, r.Ms(2.5, 7), false), seed.Rows(20))+ r.Gap(r.Ms(0.8, 2))+ })++ api.Host().CPU(seed.Flat(0.17, 0.25))+ api.Host().Memory(seed.Flat(0.39, 0.44))++ pg := db.Host()+ pg.CPU(seed.Step(onset, seed.Flat(0.1, 0.15), seed.Flat(0.14, 0.2)),+ seed.CPUExtra{Wait: seed.Step(onset, seed.Flat(0.012, 0.028), seed.Flat(0.3, 0.4))})+ pg.Disk("nvme1n1",+ seed.Step(onset, seed.Flat(1.1e7, 1.9e7), seed.Flat(1.04e8, 1.16e8)),+ seed.Step(onset, seed.Flat(5e6, 1.1e7), seed.Flat(4.8e7, 6.2e7)),+ seed.Step(onset, seed.Flat(0.4, 1.6), seed.Flat(28, 44)))+ pg.Memory(seed.Flat(0.61, 0.64))+ pg.Filesystem("/var/lib/postgresql", seed.Plus(seed.Flat(0.57, 0.58), func(t time.Time) float64 {+ if t.Before(onset) {+ return 0+ }+ return 0.006 * t.Sub(onset).Hours() // the rebuilt table grows on the same volume+ }))++ rfc := func(t time.Time) string { return t.UTC().Format(time.RFC3339) }+ s.Expose("onsetAt", rfc(onset))+ s.Expose("beforeFrom", rfc(onset.Add(-6*time.Hour)))+ s.Expose("afterTo", rfc(onset.Add(6*time.Hour)))+ s.Expose("slowAvailability", avail.Pick(func(rq *seed.Request) bool {+ return rq.Start.After(onset.Add(time.Hour)) && rq.Duration() > 60*time.Millisecond+ }))+ s.Expose("slowBooking", booking.Pick(func(rq *seed.Request) bool {+ return rq.Start.After(onset.Add(time.Hour)) && rq.Duration() > 15*time.Millisecond+ }))+ s.Expose("earlierBooking", booking.Pick(func(rq *seed.Request) bool {+ return rq.Start.Before(onset.Add(-time.Hour)) && rq.Start.After(onset.Add(-8*time.Hour))+ }))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/infra_memory_gc.go b/cli/test/evals/internal/scenarios/perf/infra_memory_gc.gonew file mode 100644index 00000000..457cf7ab--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/infra_memory_gc.go@@ -0,0 +1,145 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/infra-memory-gc", papSeedMemoryGC) }++// papSeedMemoryGC: geo-api's autocomplete memo never evicts, so the live heap+// climbs from ~0.2GB to its 900MiB GOMEMLIMIT over seven hours; near the+// limit the GC runs back to back and requests on every endpoint stall in+// pauses (gaps with no span, up to ~0.4s), until the container is OOM killed+// and restarts with an empty heap. Distractor: the batch geocode endpoint,+// which allocates a lot per request; but its traffic is flat and small, its+// time is its own bounded work, the heap climbs smoothly with no step at its+// calls, and the pauses hit every endpoint.+func papSeedMemoryGC(s *seed.Scenario) error {+ p := s.Project("maps", seed.OTel)+ host := "geo-api-5d8c7b9f4-hq2mz"+ api := p.Service("geo-api", seed.Go("nethttp", "pgx")).Versions("2.9.4").Instances(host)+ window := 3 * 24 * time.Hour+ period := 7 * time.Hour++ // fill is how far the heap has climbed since the last restart (0..1).+ fill := func(t time.Time) float64 { return float64(t.Sub(t.Truncate(period))) / float64(period) }+ pause := func(r *seed.Req) {+ h := fill(r.At())+ if r.Chance(0.02 + 0.55*math.Pow(h, 4)) {+ r.Gap(time.Duration((15 + 380*math.Pow(h, 3)*(0.4+0.6*r.Float())) * float64(time.Millisecond)))+ }+ }+ detail := seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.06, Max: 45}++ auto := api.Endpoint("GET", "/api/autocomplete")+ auto.Traffic(papJitter("auto", 0.06), seed.Daily(1500), seed.Diurnal(), seed.Over(window))+ auto.Detail(detail)+ auto.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ pause(r)+ r.Gap(r.Ms(0.05, 0.2)) // memo lookup+ r.DB("SELECT id, name, kind, country, lat, lon FROM places WHERE name_prefix @@ to_tsquery($1) ORDER BY popularity DESC LIMIT 10", r.Ms(3, 9), seed.Rows(10))+ pause(r)+ r.Gap(r.Ms(0.4, 1.2))+ })++ place := api.Endpoint("GET", "/api/places/:id")+ place.Traffic(papJitter("place", 0.06), seed.Daily(900), seed.Diurnal(), seed.Over(window))+ place.Detail(detail)+ place.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ pause(r)+ r.DB("SELECT id, name, kind, country, lat, lon, address, timezone FROM places WHERE id = $1", r.Ms(1.2, 3.5))+ pause(r)+ r.Gap(r.Ms(0.2, 0.6))+ })++ reverse := api.Endpoint("GET", "/api/reverse")+ reverse.Traffic(papJitter("reverse", 0.06), seed.Daily(600), seed.Diurnal(), seed.Over(window))+ reverse.Detail(detail)+ reverse.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ pause(r)+ r.DB("SELECT id, name, kind, country FROM places WHERE ST_DWithin(geom, ST_SetSRID(ST_MakePoint($1, $2), 4326), 200) ORDER BY geom <-> ST_SetSRID(ST_MakePoint($1, $2), 4326) LIMIT 5", r.Ms(4, 12))+ pause(r)+ r.Gap(r.Ms(0.3, 0.9))+ })++ batch := api.Endpoint("POST", "/api/geocode/batch")+ batch.Traffic(papJitter("batch", 0.1), seed.Daily(60), seed.Over(window))+ batch.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.3, Max: 15})+ batch.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.5))+ n := 500 + r.IntN(4500)+ r.Compute("decode batch", time.Duration(float64(n)*0.004*float64(time.Millisecond)), seed.A("batch.size", n))+ pause(r)+ r.DB("SELECT q.idx, p.id, p.lat, p.lon FROM places p JOIN unnest($1::text[]) WITH ORDINALITY AS q(addr, idx) ON p.normalized_address = q.addr", time.Duration(float64(n)*0.05*float64(time.Millisecond))+r.Ms(10, 25), seed.Rows(n))+ r.Compute("match addresses", time.Duration(float64(n)*0.06*float64(time.Millisecond)))+ pause(r)+ r.Gap(time.Duration(float64(n)*0.01*float64(time.Millisecond)) + r.Ms(1, 3))+ })++ limit := 900.0 * (1 << 20)+ heap := func(t time.Time) float64 { return 1.9e8 + (8.9e8-1.9e8)*fill(t) }+ api.Gauge("go.memory.used", "By", map[string]any{"go.memory.type": "other"}, seed.Plus(seed.Flat(-1e7, 1e7), heap))+ api.Gauge("go.memory.used", "By", map[string]any{"go.memory.type": "stack"}, seed.Flat(5.5e6, 7.5e6))+ // The DSL jitters every gauge by up to 1.5%, so the goal is capped a+ // little under GOMEMLIMIT to stay below it.+ api.Gauge("go.memory.gc.goal", "By", nil, func(t time.Time) float64 { return math.Min(0.984*limit, 2*(heap(t)-6e7)) })+ api.Gauge("go.goroutine.count", "{goroutine}", nil, seed.Flat(120, 175))+ busy := func(t time.Time) float64 { return 0.21 + 0.4*math.Pow(fill(t), 5) } // GC CPU near the limit+ api.Host().CPU(seed.Plus(seed.Flat(-0.02, 0.03), busy))+ api.Host().Memory(seed.Plus(seed.Flat(0.17, 0.19), func(t time.Time) float64 { return heap(t) / 1.6e9 }))++ var lastRestart time.Time+ for t := s.Ago(window).Truncate(period).Add(period); t.Before(s.Now); t = t.Add(period) {+ at := t.Add(time.Duration(4+int(t.Unix()/3600)%5) * time.Second)+ api.Log(at, seed.SevInfo, "starting geo-api", seed.A("version", "2.9.4"), seed.A("addr", ":8080"), seed.A("gomemlimit", "900MiB"))+ lastRestart = t+ }++ // A request that stalled after its query returned: the pause is plainly+ // not a wait for a connection.+ gappy := func(e *seed.Endpoint, lo, hi time.Duration) func() any {+ return e.Pick(func(rq *seed.Request) bool {+ cs := rq.Root.Children+ if len(cs) == 0 || !rq.Start.After(s.Ago(24*time.Hour)) {+ return false+ }+ last := cs[len(cs)-1]+ tail := rq.Duration() - (last.Start + last.Dur)+ return tail >= lo && tail <= hi+ })+ }+ s.Expose("pausedRequest", gappy(place, 180*time.Millisecond, time.Second))+ s.Expose("pausedAutocomplete", gappy(auto, 150*time.Millisecond, time.Second))+ s.Expose("calmRequest", place.Pick(func(rq *seed.Request) bool {+ return rq.Start.After(s.Ago(24*time.Hour)) && fill(rq.Start) < 0.2 && rq.Duration() < 5*time.Millisecond+ }))+ s.Expose("lastRestart", lastRestart.Format(time.RFC3339))+ s.Expose("restartFrom", lastRestart.Add(-2*time.Hour).Format(time.RFC3339))+ s.Expose("restartTo", lastRestart.Add(30*time.Minute).Format(time.RFC3339))+ // The heaviest batch of the day while the heap was still low, so its+ // spans account for its time: the pauses near the limit hit it too.+ s.Expose("batchRequest", func() any {+ var best *seed.Request+ for _, rq := range batch.Requests() {+ if rq.Start.Before(s.Ago(24*time.Hour)) || fill(rq.Start) >= 0.4 {+ continue+ }+ if best == nil || rq.Duration() > best.Duration() {+ best = rq+ }+ }+ if best == nil {+ return nil+ }+ best.KeepDetail()+ return best.Ref()+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/infra_noisy_neighbor.go b/cli/test/evals/internal/scenarios/perf/infra_noisy_neighbor.gonew file mode 100644index 00000000..5342f758--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/infra_noisy_neighbor.go@@ -0,0 +1,176 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/infra-noisy-neighbor", papSeedNoisyNeighbor) }++// papSeedNoisyNeighbor: tickets-api runs on three VMs behind a round-robin+// load balancer. ~14h ago the hypervisor under tickets-node-c got busy:+// that VM now loses 25-35% of its CPU time to steal and every request it+// serves takes ~3x longer, while node-a and node-b are unchanged. Endpoint+// rows mix all three hosts, so the p95 of every endpoint rose, but only+// because a third of the requests are slow. Per-host latency is visible in+// otelhttp's http.server.request.duration grouped by host.name, and in the+// access logs. Distractor: fleet-wide CPU saturation; but node-a and node-b+// are ~70% idle, node-c's own user+system time did not grow, and traffic is+// flat.+func papSeedNoisyNeighbor(s *seed.Scenario) error {+ p := s.Project("ticketing", seed.OTel)+ hosts := []string{"tickets-node-a", "tickets-node-b", "tickets-node-c"}+ api := p.Service("tickets-api", seed.Go("nethttp", "pgx")).Versions("3.2.7").Instances(hosts...)+ window := 2 * 24 * time.Hour+ onset := s.Ago(14*time.Hour + 20*time.Minute)+ bad := hosts[2]++ stretched := func(r *seed.Req) bool { return r.Host() == bad && !r.Start().Before(onset) }+ cpu := func(r *seed.Req, d time.Duration) time.Duration {+ if stretched(r) {+ return time.Duration(float64(d) * (2.6 + 0.9*r.Float()))+ }+ return d+ }+ db := func(r *seed.Req, d time.Duration) time.Duration {+ if stretched(r) {+ return time.Duration(float64(d) * (1.5 + 0.5*r.Float()))+ }+ return d+ }+ detail := seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.06, Max: 30}++ events := api.Endpoint("GET", "/api/events")+ events.Traffic(papJitter("events", 0.06), seed.Daily(1300), seed.Diurnal(), seed.Over(window))+ events.Detail(detail)+ events.Handler(func(r *seed.Req) {+ r.Gap(cpu(r, r.Ms(0.5, 1.2)))+ r.DB("SELECT id, title, venue_id, starts_at, min_price_cents FROM events WHERE city_id = $1 AND starts_at > now() ORDER BY starts_at LIMIT 30", db(r, r.Ms(4, 10)), seed.Rows(30))+ r.Gap(cpu(r, r.Ms(6, 14))) // render the listing JSON+ })+ event := api.Endpoint("GET", "/api/events/:id")+ event.Traffic(papJitter("event", 0.06), seed.Daily(1100), seed.Diurnal(), seed.Over(window))+ event.Detail(detail)+ event.Handler(func(r *seed.Req) {+ r.Gap(cpu(r, r.Ms(0.5, 1.2)))+ r.DB("SELECT id, title, description, venue_id, starts_at FROM events WHERE id = $1", db(r, r.Ms(1.2, 3)))+ r.DB("SELECT section, row_label, seat_count, price_cents, available FROM seat_blocks WHERE event_id = $1", db(r, r.Ms(3, 8)), seed.Rows(60))+ r.Gap(cpu(r, r.Ms(8, 18))) // seat map+ })+ reserve := api.Endpoint("POST", "/api/orders")+ reserve.Traffic(papJitter("reserve", 0.06), seed.Daily(320), seed.Diurnal(), seed.Over(window))+ reserve.Detail(detail)+ reserve.Handler(func(r *seed.Req) {+ r.Gap(cpu(r, r.Ms(0.8, 2)))+ r.DB("UPDATE seat_blocks SET available = available - $2 WHERE id = $1 AND available >= $2", db(r, r.Ms(2, 5)))+ r.DB("INSERT INTO orders (id, event_id, buyer_id, seats, total_cents, status, created_at) VALUES ($1, $2, $3, $4, $5, 'reserved', now())", db(r, r.Ms(2, 5)))+ r.Gap(cpu(r, r.Ms(3, 8))) // sign the reservation token+ r.Status(201)+ })+ order := api.Endpoint("GET", "/api/orders/:id")+ order.Traffic(papJitter("order", 0.06), seed.Daily(500), seed.Diurnal(), seed.Over(window))+ order.Detail(detail)+ order.Handler(func(r *seed.Req) {+ r.Gap(cpu(r, r.Ms(0.5, 1.2)))+ r.DB("SELECT id, event_id, buyer_id, seats, total_cents, status FROM orders WHERE id = $1", db(r, r.Ms(1.2, 3)))+ r.Gap(cpu(r, r.Ms(1, 3)))+ })++ // otelhttp's http.server.request.duration, per host and method: the+ // mean and count of the requests since the previous point, from the+ // generated requests. Points come every minute in the last day and every+ // ten before that (seed.MetricStep), so each point covers one step; a+ // per-minute share of a longer window would round to no count at night+ // and drop the point.+ type agg struct {+ sum time.Duration+ n int+ }+ var perMin map[string]map[time.Time]*agg+ build := func() {+ perMin = map[string]map[time.Time]*agg{}+ for _, e := range []*seed.Endpoint{events, event, reserve, order} {+ for _, rq := range e.Requests() {+ k := rq.Host + "|" + e.Method+ if perMin[k] == nil {+ perMin[k] = map[time.Time]*agg{}+ }+ m := rq.Start.Truncate(time.Minute)+ a := perMin[k][m]+ if a == nil {+ a = &agg{}+ perMin[k][m] = a+ }+ a.sum += rq.Duration()+ a.n+++ }+ }+ }+ recentCut := s.Now.Add(-time.Minute - 24*time.Hour)+ step := func(t time.Time) time.Duration {+ if t.Before(recentCut) {+ return seed.MetricStep.Old+ }+ return seed.MetricStep.Recent+ }+ for _, h := range hosts {+ for _, method := range []string{"GET", "POST"} {+ k := h + "|" + method+ attrs := map[string]any{"http.request.method": method, "http.response.status_code": 200, "url.scheme": "https", "network.protocol.version": "1.1"}+ if method == "POST" {+ attrs["http.response.status_code"] = 201+ }+ windowAt := func(t time.Time) agg {+ if perMin == nil {+ build()+ }+ var out agg+ for m := t.Add(-step(t)); m.Before(t); m = m.Add(time.Minute) {+ if a := perMin[k][m]; a != nil {+ out.sum += a.sum+ out.n += a.n+ }+ }+ return out+ }+ mean := func(t time.Time) float64 {+ if a := windowAt(t); a.n > 0 {+ return (a.sum / time.Duration(a.n)).Seconds()+ }+ return 0+ }+ perMinute := func(t time.Time) float64 {+ return float64(windowAt(t).n) / step(t).Minutes()+ }+ api.HistogramOn(h, "http.server.request.duration", "s", attrs, mean, perMinute)+ }+ }++ for _, h := range hosts {+ host := api.HostNamed(h)+ if h == bad {+ host.CPU(seed.Flat(0.27, 0.35), seed.CPUExtra{Steal: seed.Step(onset, seed.Flat(0, 0.004), seed.Flat(0.24, 0.34))})+ } else {+ host.CPU(seed.Flat(0.26, 0.34))+ }+ host.Memory(seed.Flat(0.47, 0.53))+ }++ rfc := func(t time.Time) string { return t.UTC().Format(time.RFC3339) }+ s.Expose("onsetAt", rfc(onset))+ s.Expose("slowRequest", event.Pick(func(rq *seed.Request) bool {+ return rq.Host == bad && rq.Start.After(s.Ago(6*time.Hour)) && rq.Duration() > 45*time.Millisecond+ }))+ s.Expose("fastRequest", event.Pick(func(rq *seed.Request) bool {+ return rq.Host == hosts[0] && rq.Start.After(s.Ago(6*time.Hour)) && rq.Duration() < 25*time.Millisecond+ }))+ s.Expose("eventP95Before", func() any {+ return event.PercentileBetween(0.95, onset.Add(-8*time.Hour), onset).Seconds() * 1000+ })+ s.Expose("eventP95After", func() any {+ return event.PercentileBetween(0.95, onset, s.Now).Seconds() * 1000+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/net_chatty_fanout.go b/cli/test/evals/internal/scenarios/perf/net_chatty_fanout.gonew file mode 100644index 00000000..03e84e27--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/net_chatty_fanout.go@@ -0,0 +1,119 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/net-chatty-fanout", pncSeedChattyFanout) }++var pncSkuPrefixes = []string{"SH", "TS", "JK", "BG", "SK", "HT", "SC", "BT"}++func pncSku(r *seed.Req) string {+ return fmt.Sprintf("%s-%05d", pncSkuPrefixes[r.IntN(len(pncSkuPrefixes))], 1000+r.IntN(80000))+}++// pncSeedChattyFanout: the storefront gateway builds the cart view by asking+// three services in two other projects about every cart line, one line at a+// time: catalog-service and pricing-service (catalog project) and+// inventory-service (fulfillment project). A ten-line cart is ~31 hops, each+// spending ~10ms on the wire for ~2ms of server work. Distractor: the hops+// are sequential, which reads like "run them in parallel", but they are+// dozens of tiny per-line calls to the same three endpoints; the batch+// endpoints that exist (POST /products/batch, used by the wishlist) remove+// the round trips.+func pncSeedChattyFanout(s *seed.Scenario) error {+ window := 24 * time.Hour+ store := s.Project("storefront", seed.OTel)+ catalogP := s.Project("catalog", seed.OTel)+ fulfil := s.Project("fulfillment", seed.OTel)+ s.Expose("primaryProject", "storefront")++ gw := store.Service("storefront-gateway", seed.Node("express", "pg")).Versions("4.18.2")+ carts := store.Service("cart-service", seed.Go("chi", "pgx")).Versions("2.6.0")+ catalog := catalogP.Service("catalog-service", seed.Go("nethttp", "pgx")).Versions("7.3.1")+ pricing := catalogP.Service("pricing-service", seed.Java("spring-boot")).Versions("3.11.0")+ inventory := fulfil.Service("inventory-service", seed.Python("fastapi", "psycopg")).Versions("1.9.4")++ sku := ""+ cartGet := carts.Endpoint("GET", "/carts/:id")+ cartGet.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.5))+ r.DB("SELECT sku, qty, added_at FROM cart_lines WHERE cart_id = $1 ORDER BY added_at", r.LogNormal(3*time.Millisecond, 8*time.Millisecond))+ r.Compute("encode response", r.Ms(0.2, 0.6))+ })+ product := catalog.Endpoint("GET", "/products/:sku")+ product.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.1, 0.3))+ r.Cache("get", "product:"+sku, r.Ms(0.3, 0.9))+ if r.Chance(0.08) {+ r.DB("SELECT sku, title, brand, image_url, price_list FROM products WHERE sku = $1", r.Ms(2, 5))+ }+ r.Compute("encode response", r.Ms(0.1, 0.4))+ })+ productsBatch := catalog.Endpoint("POST", "/products/batch")+ productsBatch.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.1, 0.3))+ r.Cache("mget", "product:"+pncSku(r), r.Ms(0.5, 1.4))+ r.Compute("encode response", r.Ms(0.4, 1.2))+ })+ price := pricing.Endpoint("GET", "/prices/:sku")+ price.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.6))+ r.DB("SELECT amount_minor, currency, valid_until FROM price_book WHERE sku = ? AND price_list = ? AND currency = ?", r.LogNormal(1500*time.Microsecond, 4*time.Millisecond))+ })+ stock := inventory.Endpoint("GET", "/stock/:sku")+ stock.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.5))+ r.Cache("GET", "stock:eu-central:"+sku, r.Ms(0.4, 1.1))+ r.Compute("encode response", r.Ms(0.1, 0.3))+ })++ rtt := func(r *seed.Req) time.Duration { return r.LogNormal(9*time.Millisecond, 15*time.Millisecond) }+ lines := map[*seed.Request]int{}+ cart := gw.Endpoint("GET", "/api/cart")+ cart.Traffic(seed.Daily(420), seed.Diurnal(), seed.Over(window))+ cart.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.03, Max: 4})+ cart.Handler(func(r *seed.Req) {+ n := 5 + r.IntN(12)+ lines[r.Request()] = n+ cartID := fmt.Sprintf("c_%07d", r.IntN(9000000))+ r.Gap(r.Ms(0.4, 1.0))+ r.Cache("GET", "sess:"+fmt.Sprintf("%x", r.IntN(1<<30)), r.Ms(0.4, 1.2))+ pncCall(r, cartGet, rtt(r), "/carts/"+cartID, "")+ for range n {+ sku = pncSku(r)+ pncCall(r, product, rtt(r), "/products/"+sku, "")+ pncCall(r, price, rtt(r), "/prices/"+sku, "currency=EUR")+ pncCall(r, stock, rtt(r), "/stock/"+sku, "")+ }+ r.Gap(r.Ms(1.7, 4.6)) // build the view+ })++ wish := gw.Endpoint("GET", "/api/wishlist")+ wish.Traffic(seed.Daily(240), seed.Diurnal(), seed.Over(window))+ wish.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.0))+ r.Cache("GET", "sess:"+fmt.Sprintf("%x", r.IntN(1<<30)), r.Ms(0.4, 1.2))+ r.DB("SELECT sku FROM wishlist_items WHERE customer_id = $1 ORDER BY added_at DESC LIMIT 50", r.LogNormal(3*time.Millisecond, 8*time.Millisecond), seed.Rows(5+r.IntN(30)))+ pncCall(r, productsBatch, rtt(r), "/products/batch", "")+ r.Gap(r.Ms(0.8, 2))+ })++ s.Background(gw, seed.TrafficOpts{Endpoints: 4, PerDay: 150})+ gw.Host().CPU(seed.Flat(0.14, 0.24))+ gw.Host().Memory(seed.Flat(0.41, 0.49))+ catalog.Host().CPU(seed.Flat(0.11, 0.19))+ inventory.Host().CPU(seed.Flat(0.09, 0.16))++ s.Expose("cartRequest", cart.Pick(func(r *seed.Request) bool {+ n := lines[r]+ return n >= 8 && n <= 11 && r.Start.After(s.Ago(6*time.Hour))+ }))+ s.Expose("p50Ms24h", func() any { return cart.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return cart.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/net_no_conn_reuse.go b/cli/test/evals/internal/scenarios/perf/net_no_conn_reuse.gonew file mode 100644index 00000000..b95d38e0--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/net_no_conn_reuse.go@@ -0,0 +1,118 @@+package perf++import (+ "fmt"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/net-no-conn-reuse", pncSeedNoConnReuse) }++const pncHTTPTraceScope = "go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace"++// pncTracedGet records an outgoing GET the way otelhttp plus otelhttptrace+// report it: the CLIENT span, with http.getconn (and, on a new connection,+// http.dns, http.connect and http.tls inside it), http.send and+// http.receive as children. server is the time from the request being sent+// to the first response byte.+func pncTracedGet(r *seed.Req, url, host, remote string, fresh bool, server time.Duration) *seed.Span {+ port := 443+ if strings.HasPrefix(url, "http://") {+ port = 8080+ }+ addr := fmt.Sprintf("%s:%d", remote, port)+ var client *seed.Span+ client = r.Span("GET", func(r *seed.Req) {+ gc := r.Span("http.getconn", func(r *seed.Req) {+ if fresh {+ dns := r.Compute("http.dns", r.LogNormal(16*time.Millisecond, 42*time.Millisecond), seed.A("net.host.name", host), seed.A("http.dns.addrs", remote))+ dns.Scope = pncHTTPTraceScope+ conn := r.Compute("http.connect", r.LogNormal(26*time.Millisecond, 58*time.Millisecond), seed.A("http.remote", addr), seed.A("http.conn.start.network", "tcp"))+ conn.Scope = pncHTTPTraceScope+ tls := r.Compute("http.tls", r.LogNormal(48*time.Millisecond, 105*time.Millisecond))+ tls.Scope = pncHTTPTraceScope+ } else {+ r.Gap(r.Ms(0.01, 0.06))+ }+ }, seed.A("net.host.name", host), seed.A("http.remote", addr), seed.A("http.local", fmt.Sprintf("10.4.%d.%d:%d", r.IntN(8), 10+r.IntN(200), 40000+r.IntN(20000))),+ seed.A("http.conn.reused", !fresh), seed.A("http.conn.wasidle", !fresh))+ gc.Scope = pncHTTPTraceScope+ send := r.Compute("http.send", r.Ms(0.02, 0.12))+ send.Scope = pncHTTPTraceScope+ r.Gap(server)+ recv := r.Compute("http.receive", r.Ms(0.3, 1.8))+ recv.Scope = pncHTTPTraceScope+ }, seed.A("http.request.method", "GET"), seed.A("url.full", url), seed.A("server.address", host), seed.A("server.port", port),+ seed.A("http.response.status_code", 200), seed.A("network.protocol.version", "1.1"))+ client.Kind = seed.KindClient+ client.Scope = "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"+ return client+}++var pncPlaces = []struct{ q, ll string }{+ {"10115+Berlin", "52.5323,13.3846"}, {"80331+M%C3%BCnchen", "48.1374,11.5755"}, {"20095+Hamburg", "53.5503,10.0007"},+ {"50667+K%C3%B6ln", "50.9384,6.9599"}, {"60311+Frankfurt", "50.1109,8.6821"}, {"70173+Stuttgart", "48.7784,9.1800"},+ {"04109+Leipzig", "51.3397,12.3731"}, {"01067+Dresden", "51.0504,13.7373"},+}++// pncSeedNoConnReuse: the store finder geocodes the query and asks for+// driving distances from a maps provider. The provider client builds a new+// http.Transport for every call, so each call pays DNS, TCP and TLS+// (http.conn.reused=false) before ~20ms of provider work. Distractor: the+// provider calls are the long pole, which reads like a slow upstream, but+// the provider answers quickly once a request is on the wire.+func pncSeedNoConnReuse(s *seed.Scenario) error {+ window := 2 * 24 * time.Hour+ p := s.Project("stores", seed.OTel)+ api := p.Service("store-locator-api", seed.Go("nethttp", "pgx")).Versions("1.14.0")+ const mapsHost = "maps.geo.example"++ nearby := api.Endpoint("GET", "/api/stores/nearby")+ nearby.Traffic(seed.Daily(1800), seed.Diurnal(), seed.Over(window))+ nearby.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.02, Max: 10})+ nearby.Handler(func(r *seed.Req) {+ remote := fmt.Sprintf("198.51.100.%d", 20+r.IntN(4))+ r.Gap(r.Ms(0.2, 0.6))+ place := pncPlaces[r.IntN(len(pncPlaces))]+ pncTracedGet(r, "https://"+mapsHost+"/v1/geocode?q="+place.q+"®ion=de", mapsHost, remote, true, r.LogNormal(19*time.Millisecond, 45*time.Millisecond))+ r.DB("SELECT id, name, address, lat, lng FROM stores WHERE ST_DWithin(location, ST_MakePoint($1, $2)::geography, $3) ORDER BY location <-> ST_MakePoint($1, $2)::geography LIMIT 10",+ r.LogNormal(4*time.Millisecond, 10*time.Millisecond), seed.Rows(10))+ dest := make([]string, 10)+ for i := range dest {+ dest[i] = fmt.Sprintf("store%%3A%d", 1000+r.IntN(400))+ }+ pncTracedGet(r, "https://"+mapsHost+"/v1/distancematrix?origins="+place.ll+"&destinations="+strings.Join(dest, "%7C")+"&mode=driving", mapsHost, remote, true, r.LogNormal(24*time.Millisecond, 55*time.Millisecond))+ r.Compute("encode response", r.Ms(0.4, 1.2))+ })++ avail := api.Endpoint("GET", "/api/stores/:id/availability")+ avail.Traffic(seed.Daily(900), seed.Diurnal(), seed.Over(window))+ avail.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.6))+ pncTracedGet(r, fmt.Sprintf("http://inventory.internal:8080/v2/stock?store=%d", 100+r.IntN(300)), "inventory.internal", fmt.Sprintf("10.20.3.%d", 10+r.IntN(5)), false, r.LogNormal(6*time.Millisecond, 14*time.Millisecond))+ r.Compute("encode response", r.Ms(0.2, 0.8))+ })++ hours := api.Endpoint("GET", "/api/stores/:id/hours")+ hours.Traffic(seed.Daily(700), seed.Diurnal(), seed.Over(window))+ hours.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.6))+ r.DB("SELECT weekday, opens_at, closes_at FROM store_hours WHERE store_id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond), seed.Rows(7))+ r.Compute("encode response", r.Ms(0.1, 0.5))+ })++ s.Background(api, seed.TrafficOpts{Endpoints: 3, PerDay: 150})+ api.Host().CPU(seed.Flat(0.12, 0.2))+ api.Host().Memory(seed.Flat(0.38, 0.45))++ s.Expose("slowRequest", nearby.Pick(func(r *seed.Request) bool {+ return r.Start.After(s.Ago(3*time.Hour)) && r.Duration() > 260*time.Millisecond && r.Duration() < 420*time.Millisecond+ }))+ s.Expose("availRequest", avail.Pick(func(r *seed.Request) bool { return r.Start.After(s.Ago(3 * time.Hour)) }))+ s.Expose("p50Ms24h", func() any { return nearby.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return nearby.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/net_retries.go b/cli/test/evals/internal/scenarios/perf/net_retries.gonew file mode 100644index 00000000..ca308934--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/net_retries.go@@ -0,0 +1,128 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/net-retries", pncSeedRetries) }++const (+ pncRatesURL = "https://rates.carrier.example/v2/rates"+ pncRetryFn = "app.carriers.client.CarrierClient.fetch_rates"+ pncReadTimeout = "The read operation timed out"+ pnc503Message = "Server error '503 Service Unavailable' for url 'https://rates.carrier.example/v2/rates'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503"+ pncRetryErrType = "tenacity.RetryError"+)++const pncRetryErrorStack = `Traceback (most recent call last):+ File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__+ result = await fn(*args, **kwargs)+ File "/app/app/carriers/client.py", line 36, in fetch_rates+ response = await self._http.post(RATES_URL, json=payload)+ File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1859, in post+ return await self.request(+ File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions+ raise mapped_exc(message) from exc+httpx.ReadTimeout: The read operation timed out++The above exception was the direct cause of the following exception:++Traceback (most recent call last):+ File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app+ response = await f(request)+ File "/app/app/routes/quotes.py", line 27, in create_quote+ rates = await carrier.fetch_rates(parcel)+ File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped+ return await copy(fn, *args, **kwargs)+ File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 153, in iter+ raise retry_exc from fut.exception()+tenacity.RetryError: RetryError[<Future at 0x7f3a2c1b9d50 state=finished raised ReadTimeout>]`++// pncSeedRetries: the shipping quote endpoint asks a carrier's rate API+// through httpx with no explicit timeout (httpx's default 5s) inside a+// tenacity retry (3 attempts, exponential wait 1s then 2s). A few percent of+// attempts hang until the read timeout or get a quick 503; each failure is+// waited out and retried, so a minority of requests take 6-13s with the same+// carrier span two or three times. Distractor: the carrier looks slow, but+// its successful answers take ~150ms; the tail is the client's own timeout+// and backoff.+func pncSeedRetries(s *seed.Scenario) error {+ window := 2 * 24 * time.Hour+ p := s.Project("logistics", seed.OTel)+ api := p.Service("quotes-api", seed.Python("fastapi", "psycopg")).Versions("2.3.0")++ var forced *seed.Request+ forceFrom, forceTo := s.Ago(110*time.Minute), s.Ago(95*time.Minute)+ quotes := api.Endpoint("POST", "/api/shipping/quotes")+ quotes.Traffic(seed.Daily(2000), seed.Diurnal(), seed.Over(window))+ quotes.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.02, Max: 12})+ quotes.Handler(func(r *seed.Req) {+ route := "POST /api/shipping/quotes"+ r.Compute(route+" http receive", r.Ms(0.05, 0.3))+ r.DB("SELECT id, weight_grams, length_mm, width_mm, height_mm, origin_postcode, destination_postcode, destination_country FROM parcels WHERE id = %s", r.LogNormal(2*time.Millisecond, 6*time.Millisecond))+ ok := false+ force := false+ if forced == nil && !r.Start().Before(forceFrom) && r.Start().Before(forceTo) {+ forced, force = r.Request(), true+ }+ for attempt := 1; attempt <= 3; attempt++ {+ x := r.Float()+ if force {+ x = map[bool]float64{true: 0.01, false: 0.5}[attempt < 3]+ }+ var cause string+ switch {+ case x < 0.035:+ sp := r.HTTP("POST", pncRatesURL, 5*time.Second+r.Ms(1, 8), 0)+ delete(sp.Attrs, "http.response.status_code")+ sp.Attrs["error.type"] = "ReadTimeout"+ sp.Status, sp.StatusMsg = seed.StatusError, pncReadTimeout+ cause = "ReadTimeout: " + pncReadTimeout+ case x < 0.05:+ r.HTTP("POST", pncRatesURL, r.LogNormal(35*time.Millisecond, 90*time.Millisecond), 503, seed.A("error.type", "503"))+ cause = "HTTPStatusError: " + pnc503Message+ default:+ r.HTTP("POST", pncRatesURL, r.LogNormal(150*time.Millisecond, 400*time.Millisecond), 200)+ ok = true+ }+ if ok {+ break+ }+ if attempt < 3 {+ wait := float64(int(1) << (attempt - 1))+ r.Log(seed.SevWarn, fmt.Sprintf("Retrying %s in %.1f seconds as it raised %s.", pncRetryFn, wait, cause),+ seed.A("code.filepath", "/usr/local/lib/python3.12/site-packages/tenacity/before_sleep.py"), seed.A("code.function", "log_it"), seed.A("code.lineno", 65))+ r.Gap(time.Duration(wait * float64(time.Second)))+ }+ }+ if !ok {+ r.Error(seed.Exc{Type: pncRetryErrType, Message: "RetryError[<Future at 0x7f3a2c1b9d50 state=finished raised ReadTimeout>]", Stack: pncRetryErrorStack, Escaped: true})+ r.Status(500)+ r.Compute(route+" http send", r.Ms(0.05, 0.2))+ return+ }+ r.DB("INSERT INTO quotes (parcel_id, carrier, service_level, amount_minor, currency, expires_at) VALUES (%s, %s, %s, %s, %s, %s)", r.LogNormal(2500*time.Microsecond, 6*time.Millisecond))+ r.Compute(route+" http send", r.Ms(0.05, 0.3))+ })++ track := api.Endpoint("GET", "/api/shipments/:id/tracking")+ track.Traffic(seed.Daily(1100), seed.Diurnal(), seed.Over(window))+ track.Handler(func(r *seed.Req) {+ r.DB("SELECT status, location, occurred_at FROM tracking_events WHERE shipment_id = %s ORDER BY occurred_at DESC LIMIT 20", r.LogNormal(3*time.Millisecond, 8*time.Millisecond), seed.Rows(1+r.IntN(12)))+ r.Compute("GET /api/shipments/:id/tracking http send", r.Ms(0.05, 0.3))+ })++ s.Background(api, seed.TrafficOpts{Endpoints: 4, PerDay: 150})+ api.Host().CPU(seed.Flat(0.15, 0.24))+ api.Host().Memory(seed.Flat(0.5, 0.57))++ s.Expose("retriedRequest", quotes.Pick(func(r *seed.Request) bool { return r == forced }))+ s.Expose("p50Ms24h", func() any { return quotes.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return quotes.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p99Ms24h", func() any { return quotes.Percentile(0.99, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/net_slow_third_party.go b/cli/test/evals/internal/scenarios/perf/net_slow_third_party.gonew file mode 100644index 00000000..0826df8c--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/net_slow_third_party.go@@ -0,0 +1,130 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/net-slow-third-party", pncSeedSlowThirdParty) }++type pncWindow struct{ from, to time.Time }++func (w pncWindow) has(t time.Time) bool { return !t.Before(w.from) && t.Before(w.to) }++func pncInAny(ws []pncWindow, t time.Time) bool {+ for _, w := range ws {+ if w.has(t) {+ return true+ }+ }+ return false+}++// pncSeedSlowThirdParty: checkout charges the card through an external+// payment provider. Three times in three days the provider slowed down for+// 25-40 minutes: the charge CLIENT span grew from ~230ms to seconds (some+// calls hit the 10s client timeout), and checkout and refunds spiked+// together while routes that do not call the provider stayed flat.+// Distractor: goroutines pile up during the spikes, which reads like pool+// exhaustion, but the DB spans start immediately and stay fast.+func pncSeedSlowThirdParty(s *seed.Scenario) error {+ window := 3 * 24 * time.Hour+ incidents := []pncWindow{+ {s.Ago(61*time.Hour + 12*time.Minute), s.Ago(61*time.Hour + 12*time.Minute - 35*time.Minute)},+ {s.Ago(30*time.Hour + 47*time.Minute), s.Ago(30*time.Hour + 47*time.Minute - 25*time.Minute)},+ {s.Ago(5*time.Hour + 21*time.Minute), s.Ago(5*time.Hour + 21*time.Minute - 40*time.Minute)},+ }+ p := s.Project("shop", seed.OTel)+ api := p.Service("checkout-api", seed.Go("nethttp", "pgx")).Versions("5.2.0")++ providerCall := func(r *seed.Req, path string, timeout bool) (time.Duration, bool) {+ d := r.LogNormal(230*time.Millisecond, 560*time.Millisecond)+ if pncInAny(incidents, r.At()) {+ d = r.LogNormal(4*time.Second, 16*time.Second)+ }+ if timeout {+ d = 12 * time.Second+ }+ url := "https://api.payments.example" + path+ if d >= 10*time.Second {+ sp := r.HTTP("POST", url, 10*time.Second, 0)+ delete(sp.Attrs, "http.response.status_code")+ sp.Attrs["error.type"] = "*url.Error"+ sp.Status = seed.StatusError+ sp.StatusMsg = fmt.Sprintf("Post %q: context deadline exceeded (Client.Timeout exceeded while awaiting headers)", url)+ return d, false+ }+ r.HTTP("POST", url, d, 200, seed.A("http.request.body.size", 380+r.IntN(200)))+ return d, true+ }++ checkout := api.Endpoint("POST", "/api/checkout")+ checkout.Traffic(seed.Daily(1500), seed.Diurnal(), seed.Over(window))+ checkout.Detail(seed.DetailPolicy{Recent: 8 * time.Hour, Rate: 0.03, Max: 20})+ forcedTimeout := false+ checkout.Handler(func(r *seed.Req) {+ // At least one charge in the latest window runs into the client timeout.+ timeout := !forcedTimeout && incidents[2].has(r.Start()) && !r.Start().Before(incidents[2].from.Add(10*time.Minute))+ forcedTimeout = forcedTimeout || timeout+ r.Gap(r.Ms(0.4, 1.0))+ r.DB("SELECT id, status, total_cents, currency FROM orders WHERE id = $1 AND customer_id = $2", r.LogNormal(2500*time.Microsecond, 7*time.Millisecond))+ r.DB("SELECT sku, qty, unit_price_cents FROM order_lines WHERE order_id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond), seed.Rows(1+r.IntN(6)))+ if _, ok := providerCall(r, "/v1/charges", timeout); !ok {+ r.Error(seed.Exc{Type: "*url.Error", Message: `Post "https://api.payments.example/v1/charges": context deadline exceeded (Client.Timeout exceeded while awaiting headers)`})+ r.Log(seed.SevError, "charge failed", seed.A("error", `Post "https://api.payments.example/v1/charges": context deadline exceeded (Client.Timeout exceeded while awaiting headers)`))+ r.Status(502)+ r.Gap(r.Ms(0.2, 0.6))+ return+ }+ r.DB("UPDATE orders SET status = 'paid', paid_at = now(), charge_id = $2 WHERE id = $1", r.LogNormal(2500*time.Microsecond, 6*time.Millisecond))+ r.DB("INSERT INTO payment_events (order_id, kind, payload) VALUES ($1, $2, $3)", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))+ r.Compute("encode response", r.Ms(0.3, 1))+ })++ refunds := api.Endpoint("POST", "/api/refunds")+ refunds.Traffic(seed.Daily(320), seed.Diurnal(), seed.Over(window))+ refunds.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.0))+ r.DB("SELECT id, status, charge_id, total_cents FROM orders WHERE id = $1", r.LogNormal(2500*time.Microsecond, 7*time.Millisecond))+ if _, ok := providerCall(r, "/v1/refunds", false); !ok {+ r.Status(502)+ return+ }+ r.DB("INSERT INTO payment_events (order_id, kind, payload) VALUES ($1, $2, $3)", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))+ })++ order := api.Endpoint("GET", "/api/orders/:id")+ order.Traffic(seed.Daily(900), seed.Diurnal(), seed.Over(window))+ order.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.DB("SELECT id, status, total_cents, currency, paid_at FROM orders WHERE id = $1", r.LogNormal(2500*time.Microsecond, 7*time.Millisecond))+ r.DB("SELECT sku, qty, unit_price_cents FROM order_lines WHERE order_id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))+ r.Compute("encode response", r.Ms(0.3, 1))+ })++ s.Background(api, seed.TrafficOpts{Endpoints: 5, PerDay: 150})+ api.Host().CPU(seed.Flat(0.18, 0.27))+ api.Host().Memory(seed.Flat(0.46, 0.53))+ goroutines := func(t time.Time) float64 {+ if pncInAny(incidents, t) {+ return 380+ }+ return 54+ }+ api.Gauge("go.goroutine.count", "{goroutine}", nil, seed.Plus(goroutines, seed.Flat(0, 30)))++ s.Expose("incidentRequest", checkout.Pick(func(r *seed.Request) bool {+ return incidents[2].has(r.Start) && r.Status == 200 && r.Duration() > 3*time.Second && r.Duration() < 8*time.Second+ }))+ s.Expose("incidentFrom", incidents[2].from.Truncate(time.Hour).Format(time.RFC3339))+ s.Expose("incidentTo", incidents[2].to.Truncate(time.Hour).Add(time.Hour).Format(time.RFC3339))+ for i, w := range incidents {+ s.Expose(fmt.Sprintf("incident%d", i+1), w.from.Format(time.RFC3339))+ }+ s.Expose("p50Ms3d", func() any { return checkout.Percentile(0.5, window).Seconds() * 1000 })+ s.Expose("p99Ms3d", func() any { return checkout.Percentile(0.99, window).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_alert_handoff.go b/cli/test/evals/internal/scenarios/perf/pct_alert_handoff.gonew file mode 100644index 00000000..6e411713--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_alert_handoff.go@@ -0,0 +1,176 @@+package perf++import (+ "fmt"+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/ctl-alert-handoff", pctSeedAlertHandoff) }++// pctCategories are the storefront's category slugs, most popular first.+var pctCategories = []string{+ "shoes", "jackets", "t-shirts", "jeans", "sneakers", "dresses", "hoodies", "bags", "accessories", "socks",+ "shirts", "shorts", "skirts", "sweaters", "boots", "hats", "scarves", "belts", "swimwear", "sportswear",+ "underwear", "sunglasses", "watches", "gloves",+}++// pctSeedAlertHandoff: the p95 alert on the category listing fired five+// minutes after v5.1.0 restarted all three catalog instances. Each instance+// starts with an empty in-process category cache, which a background preload+// fills one category every 25s, and an empty pgx pool; until a category is+// preloaded, the first request for it on that instance runs the listing+// query (one to two seconds), and the first few requests open connections.+// Ten minutes after the restart the listing is back to ~20ms on v5.1.0;+// v5.0.4, 30h earlier, shows the same transient. Expected:+// cold_start_jit_connection_warmup. Distractor: a code regression in v5.1.0+// (steady-state latency is unchanged).+func pctSeedAlertHandoff(s *seed.Scenario) error {+ p := s.Project("catalog", seed.OTel)+ window := 48 * time.Hour+ prevDeploy := s.Ago(30 * time.Hour)+ deployAt := s.Ago(48 * time.Minute)+ fireAt := deployAt.Add(5 * time.Minute)+ api := p.Service("catalog-api", seed.Go("nethttp", "pgx")).+ Instances("catalog-api-a", "catalog-api-b", "catalog-api-c").+ Versions("5.0.3", seed.Deploy{At: prevDeploy, Version: "5.0.4"}, seed.Deploy{At: deployAt, Version: "5.1.0"})++ // Per-instance process state: the version it runs (a change means the+ // process restarted), the categories it has cached, the connections+ // its pool has opened. Each endpoint generates its requests minute by+ // minute on its own, so each keeps its own view of the processes.+ type proc struct {+ version string+ started time.Time+ cached map[string]bool+ conns int+ }+ restartOf := func(t time.Time) time.Time {+ switch {+ case !t.Before(deployAt):+ return deployAt+ case !t.Before(prevDeploy):+ return prevDeploy+ }+ return s.Ago(window + time.Hour)+ }+ newState := func() func(r *seed.Req) *proc {+ procs := map[string]*proc{}+ return func(r *seed.Req) *proc {+ st := procs[r.Host()]+ if st == nil || st.version != r.Version() {+ fresh := &proc{version: r.Version(), started: restartOf(r.Start()), cached: map[string]bool{}}+ if st == nil {+ // The window opens on warm processes.+ for _, c := range pctCategories {+ fresh.cached[c] = true+ }+ fresh.conns = 10+ }+ st = fresh+ procs[r.Host()] = st+ }+ return st+ }+ }+ weights := make([]float64, len(pctCategories))+ total := 0.0+ for i := range weights {+ weights[i] = 1 / math.Sqrt(float64(i+1))+ total += weights[i]+ }+ pickCategory := func(r *seed.Req) string {+ x := r.Float() * total+ for i, w := range weights {+ x -= w+ if x <= 0 {+ return pctCategories[i]+ }+ }+ return pctCategories[len(pctCategories)-1]+ }+ connect := func(r *seed.Req, st *proc) {+ if st.conns >= 10 || !r.Chance(0.8) {+ return+ }+ st.conns+++ sp := r.Compute("connect", r.Ms(90, 220), seed.A("db.system.name", "postgresql"), seed.A("server.address", "catalog-db.internal"), seed.A("server.port", 5432))+ sp.Kind, sp.Scope = seed.KindClient, "github.com/exaring/otelpgx"+ }++ listing := api.Endpoint("GET", "/api/categories/:slug/products")+ listing.Traffic(seed.Daily(3600), seed.Diurnal(), seed.Over(window))+ listing.Detail(seed.DetailPolicy{Recent: 70 * time.Minute, Rate: 0.35, Max: 60})+ listingState := newState()+ listing.Handler(func(r *seed.Req) {+ st := listingState(r)+ cat := pickCategory(r)+ r.RootAttr("url.path", "/api/categories/"+cat+"/products")+ r.Gap(r.Ms(0.3, 0.9))+ preloaded := false+ for i, c := range pctCategories {+ if c == cat {+ preloaded = r.Start().After(st.started.Add(time.Duration(i+1) * 25 * time.Second))+ }+ }+ if !st.cached[cat] && !preloaded {+ connect(r, st)+ r.DB("SELECT p.id, p.slug, p.name, p.brand, p.thumbnail_url, pp.amount_cents, pp.currency, i.available FROM products p JOIN product_prices pp ON pp.product_id = p.id AND pp.price_list = $2 JOIN inventory i ON i.product_id = p.id WHERE p.category_id = (SELECT id FROM categories WHERE slug = $1) AND p.active ORDER BY p.rank LIMIT 500",+ r.LogNormal(1100*time.Millisecond, 1900*time.Millisecond), seed.Rows(380+r.IntN(121)))+ r.Compute("build category listing", r.Ms(25, 70))+ st.cached[cat] = true+ }+ r.Compute("filter and page listing", r.Ms(3, 9))+ r.Compute("encode response", r.Ms(4, 12))+ r.Gap(r.Ms(0.2, 0.6))+ })++ product := api.Endpoint("GET", "/api/products/:id")+ product.Traffic(seed.Daily(1500), seed.Diurnal(), seed.Over(window))+ productState := newState()+ product.Handler(func(r *seed.Req) {+ st := productState(r)+ r.Gap(r.Ms(0.3, 0.9))+ connect(r, st)+ r.DB("SELECT p.id, p.slug, p.name, p.brand, p.description, pp.amount_cents, pp.currency, i.available FROM products p JOIN product_prices pp ON pp.product_id = p.id AND pp.price_list = $2 JOIN inventory i ON i.product_id = p.id WHERE p.id = $1",+ r.LogNormal(3*time.Millisecond, 11*time.Millisecond), seed.Rows(1))+ r.Compute("encode response", r.Ms(0.5, 2))+ })+ s.Background(api, seed.TypicalTraffic(4))+ for _, h := range api.Hosts {+ api.HostNamed(h).CPU(seed.Flat(0.18, 0.3))+ api.HostNamed(h).Memory(seed.Flat(0.4, 0.5))+ }++ email := p.EmailChannel("Catalog team email")+ rule := p.Rule(seed.Rule{Name: "Category listing p95 above 1s", Type: "endpoint_p95_threshold", Channel: email, Cooldown: 30,+ Config: map[string]any{"endpoint": "GET /api/categories/:slug/products", "thresholdMs": 1000, "lookbackMinutes": 10}})++ s.Expose("ruleId", rule)+ s.Expose("deployAt", pctRFC3339(deployAt))+ s.Expose("prevDeployAt", pctRFC3339(prevDeploy))+ s.Expose("fireAt", pctRFC3339(fireAt))+ s.Expose("linkFrom", pctRFC3339(fireAt.Add(-time.Minute)))+ s.Expose("linkTo", pctRFC3339(fireAt.Add(2*time.Minute)))+ s.Expose("alertP95", func() any {+ return math.Round(listing.PercentileBetween(0.95, fireAt.Add(-10*time.Minute), fireAt).Seconds() * 1000)+ })+ s.Expose("p95MsLast20m", pctLazyMs(listing, 0.95, s.Ago(20*time.Minute), s.Now))+ s.Expose("p95MsBeforeDeploy", pctLazyMs(listing, 0.95, deployAt.Add(-6*time.Hour), deployAt))+ s.Expose("slowRequest", pctRef(func() *seed.Request { return pctSlowestBetween(listing, deployAt, fireAt) }))+ s.Expose("warmRequest", pctRef(func() *seed.Request {+ return pctNearestBetween(listing, s.Ago(15*time.Minute), s.Now, listing.PercentileBetween(0.5, s.Ago(15*time.Minute), s.Now))+ }))+ // Window edges for the oracle, as "<minutes> after the deploy".+ for _, m := range []int{-60, 15, 30} {+ key := fmt.Sprintf("deployPlus%dm", m)+ if m < 0 {+ key = fmt.Sprintf("deployMinus%dm", -m)+ }+ s.Expose(key, pctRFC3339(deployAt.Add(time.Duration(m)*time.Minute)))+ }+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_ctl_helpers.go b/cli/test/evals/internal/scenarios/perf/pct_ctl_helpers.gonew file mode 100644index 00000000..f6586051--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_ctl_helpers.go@@ -0,0 +1,69 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// pctSlowestBetween returns the slowest request started in [from, to) and+// forces its detail.+func pctSlowestBetween(e *seed.Endpoint, from, to time.Time) *seed.Request {+ var best *seed.Request+ for _, r := range e.Requests() {+ if r.Start.Before(from) || !r.Start.Before(to) {+ continue+ }+ if best == nil || r.Duration() > best.Duration() {+ best = r+ }+ }+ if best != nil {+ best.KeepDetail()+ }+ return best+}++// pctRef is a lazy Expose of a request picked when the scenario flushes.+func pctRef(pick func() *seed.Request) func() any {+ return func() any {+ r := pick()+ if r == nil {+ return nil+ }+ return r.Ref()+ }+}++// pctNearestBetween returns the request started in [from, to) whose duration+// is closest to target, and forces its detail.+func pctNearestBetween(e *seed.Endpoint, from, to time.Time, target time.Duration) *seed.Request {+ var best *seed.Request+ var bestD time.Duration+ for _, r := range e.Requests() {+ if r.Start.Before(from) || !r.Start.Before(to) {+ continue+ }+ d := r.Duration() - target+ if d < 0 {+ d = -d+ }+ if best == nil || d < bestD {+ best, bestD = r, d+ }+ }+ if best != nil {+ best.KeepDetail()+ }+ return best+}++// pctAgoKey names an exposed time "h" hours before now: ago54h, ago52h30m.+func pctAgoKey(h float64) string {+ d := time.Duration(h * float64(time.Hour))+ if m := int(d.Minutes()) % 60; m != 0 {+ return fmt.Sprintf("ago%dh%dm", int(d.Hours()), m)+ }+ return fmt.Sprintf("ago%dh", int(d.Hours()))+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_healthy.go b/cli/test/evals/internal/scenarios/perf/pct_healthy.gonew file mode 100644index 00000000..4c086b05--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_healthy.go@@ -0,0 +1,64 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/ctl-healthy", pctSeedHealthy) }++// pctSeedHealthy: GET /api/search is healthy and has been for eight days+// (p50 ~25ms, p95 ~45ms, no errors, no growth, flat CPU). Expected:+// none_found. Distractors: the search query uses ILIKE on name/brand, which+// looks like it wants a trigram index but takes ~15ms; checkout is the+// slowest endpoint (a payment call), unrelated to search.+func pctSeedHealthy(s *seed.Scenario) error {+ p := s.Project("shop", seed.OTel)+ window := 8 * 24 * time.Hour+ api := p.Service("storefront-api", seed.Node("express", "pg")).Versions("1.42.0", seed.Deploy{At: s.Ago(5*24*time.Hour + 5*time.Hour), Version: "1.43.0"})++ search := api.Endpoint("GET", "/api/search")+ search.Traffic(seed.Daily(600), seed.Diurnal(), seed.Over(window))+ search.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.05, Max: 20})+ search.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.6))+ r.DB("SELECT id, name, slug, brand, price_cents, thumbnail_url FROM products WHERE active AND (name ILIKE $1 OR brand ILIKE $1) ORDER BY popularity DESC LIMIT 24",+ r.LogNormal(14*time.Millisecond, 55*time.Millisecond), seed.Rows(1+r.IntN(24)))+ r.DB("SELECT facet, value, count(*) AS n FROM product_facets WHERE product_id = ANY($1) GROUP BY facet, value",+ r.LogNormal(4*time.Millisecond, 15*time.Millisecond), seed.Rows(6+r.IntN(30)))+ r.Compute("serialize results", r.Ms(1, 4))+ r.Gap(r.Ms(0.3, 0.9))+ })++ product := api.Endpoint("GET", "/api/products/:id")+ product.Traffic(seed.Daily(420), seed.Diurnal(), seed.Over(window))+ product.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.6))+ r.DB("SELECT id, name, slug, brand, description, price_cents, images FROM products WHERE id = $1 AND active", r.LogNormal(3*time.Millisecond, 10*time.Millisecond), seed.Rows(1))+ r.DB("SELECT sku, size, color, stock FROM product_variants WHERE product_id = $1", r.LogNormal(2*time.Millisecond, 8*time.Millisecond), seed.Rows(4+r.IntN(12)))+ r.Compute("serialize product", r.Ms(0.5, 2))+ })+ checkout := api.Endpoint("POST", "/api/checkout")+ checkout.Traffic(seed.Daily(80), seed.Diurnal(), seed.Over(window))+ checkout.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 2))+ r.DB("SELECT id, items, total_cents, currency FROM carts WHERE id = $1 FOR UPDATE", r.LogNormal(3*time.Millisecond, 10*time.Millisecond))+ r.HTTP("POST", "https://api.payments.example/v1/payment_intents", r.LogNormal(380*time.Millisecond, 900*time.Millisecond), 200)+ r.DB("INSERT INTO orders (cart_id, total_cents, currency, payment_ref, created_at) VALUES ($1, $2, $3, $4, now()) RETURNING id", r.LogNormal(4*time.Millisecond, 12*time.Millisecond))+ })+ s.Background(api, seed.TypicalTraffic(6))+ api.Host().CPU(seed.Flat(0.17, 0.27))+ api.Host().Memory(seed.Flat(0.46, 0.54))++ s.Expose("slowRequest", search.Slowest(24*time.Hour))+ s.Expose("typicalRequest", pctRef(func() *seed.Request {+ return pctNearestBetween(search, s.Ago(24*time.Hour), s.Now, search.Percentile(0.5, 24*time.Hour))+ }))+ s.Expose("p50Ms24h", pctLazyMs(search, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95Ms24h", pctLazyMs(search, 0.95, s.Ago(24*time.Hour), s.Now))+ s.Expose("p99Ms24h", pctLazyMs(search, 0.99, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95MsFirstDays", pctLazyMs(search, 0.95, s.Ago(window), s.Ago(window-3*24*time.Hour)))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_marked_slow_ok.go b/cli/test/evals/internal/scenarios/perf/pct_marked_slow_ok.gonew file mode 100644index 00000000..480e1d40--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_marked_slow_ok.go@@ -0,0 +1,90 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/ctl-marked-slow-ok", pctSeedMarkedSlowOK) }++// pctLedgerExport is the ledger export handler: one range query over a+// quarter of ledger entries, a CSV encode and the streamed write, all+// proportional to the row count (at most ~1.98s for 54k rows).+func pctLedgerExport(r *seed.Req, rows int) {+ n := float64(rows)+ jit := func(lo, hi float64) float64 { return lo + (hi-lo)*r.Float() }+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("SELECT id, posted_at, account_code, counterparty, description, amount_cents, currency FROM ledger_entries WHERE org_id = $1 AND posted_at >= $2 AND posted_at < $3 ORDER BY posted_at",+ time.Duration(n*0.019*jit(0.96, 1.04)*float64(time.Millisecond)), seed.Rows(rows))+ r.Compute("encode csv", time.Duration(n*0.0135*jit(0.96, 1.04)*float64(time.Millisecond)))+ r.Gap(time.Duration(n * 0.0025 * jit(0.9, 1.1) * float64(time.Millisecond)))+}++// pctLedgerRows draws the rows one export returns: most customers export+// 18-45k entries a quarter, one in ten (the large accounts) 49-54k.+func pctLedgerRows(r *seed.Req) int {+ if r.Chance(0.1) {+ return 49000 + r.IntN(5000)+ }+ return 18000 + r.IntN(27000)+}++// pctLedgerService declares the ledger API with its everyday endpoints.+func pctLedgerService(s *seed.Scenario, p *seed.Project, window time.Duration) *seed.Service {+ api := p.Service("ledger-api", seed.Go("nethttp", "sql")).Versions("3.9.2")+ bal := api.Endpoint("GET", "/api/accounts/:id/balance")+ bal.Traffic(seed.Daily(200), seed.Diurnal(), seed.Over(window))+ bal.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.0))+ r.DB("SELECT balance_cents, currency, updated_at FROM account_balances WHERE account_id = $1", r.LogNormal(2*time.Millisecond, 9*time.Millisecond))+ r.Compute("encode response", r.Ms(0.2, 0.8))+ })+ entries := api.Endpoint("GET", "/api/accounts/:id/entries")+ entries.Traffic(seed.Daily(120), seed.Diurnal(), seed.Over(window))+ entries.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.0))+ r.DB("SELECT id, posted_at, description, amount_cents FROM ledger_entries WHERE account_id = $1 ORDER BY posted_at DESC LIMIT 50", r.LogNormal(6*time.Millisecond, 24*time.Millisecond), seed.Rows(50))+ r.Compute("encode response", r.Ms(0.6, 2.0))+ })+ post := api.Endpoint("POST", "/api/journal-entries")+ post.Traffic(seed.Daily(60), seed.Diurnal(), seed.Over(window))+ post.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.4))+ r.DB("INSERT INTO ledger_entries (org_id, account_id, posted_at, description, amount_cents, currency) VALUES ($1, $2, $3, $4, $5, $6)", r.LogNormal(4*time.Millisecond, 15*time.Millisecond))+ r.DB("UPDATE account_balances SET balance_cents = balance_cents + $2, updated_at = now() WHERE account_id = $1", r.LogNormal(3*time.Millisecond, 12*time.Millisecond))+ })+ s.Background(api, seed.TypicalTraffic(6))+ return api+}++// pctSeedMarkedSlowOK: the ledger export is the slowest endpoint (p95 about+// 1.8s, no request over 2s), but an operator marked it slow with a 2000ms+// allowance and it has held that level for the whole month. Expected:+// none_found. Distractor: the export's one big range query and CSV encode+// look like an expensive query or an unbounded result set if the allowance+// is ignored.+func pctSeedMarkedSlowOK(s *seed.Scenario) error {+ p := s.Project("ledger", seed.OTel)+ window := 30 * 24 * time.Hour+ api := pctLedgerService(s, p, window)++ export := api.Endpoint("GET", "/api/reports/export")+ export.Traffic(seed.Daily(120), seed.Diurnal(), seed.Over(window))+ export.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.3, Max: 12})+ export.Handler(func(r *seed.Req) { pctLedgerExport(r, pctLedgerRows(r)) })++ api.Host().CPU(seed.Flat(0.2, 0.3))+ api.Host().Memory(seed.Flat(0.5, 0.58))++ p.MarkSlow("GET /api/reports/export", 2000, "Full ledger CSV export for the finance team: one quarter of entries per request, streamed. Up to 2s is fine.")++ s.Expose("slowRequest", export.Slowest(24*time.Hour))+ s.Expose("p50Ms24h", pctLazyMs(export, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95Ms24h", pctLazyMs(export, 0.95, s.Ago(24*time.Hour), s.Now))+ s.Expose("p99Ms24h", pctLazyMs(export, 0.99, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95MsFirstWeek", pctLazyMs(export, 0.95, s.Ago(window), s.Ago(window-7*24*time.Hour)))+ s.Expose("allowanceMs", 2000)+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_marked_slow_regressed.go b/cli/test/evals/internal/scenarios/perf/pct_marked_slow_regressed.gonew file mode 100644index 00000000..db6f8920--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_marked_slow_regressed.go@@ -0,0 +1,87 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/ctl-marked-slow-regressed", pctSeedMarkedSlowRegressed) }++// pctSeedMarkedSlowRegressed: the monthly invoice export is marked slow with+// a 2000ms allowance and sat at p95 ~1.5s until v4.8.0 (26h ago) added a tax+// breakdown that loads each invoice's tax lines with its own query. Since+// the deploy p95 is ~4s, well past the allowance. Distractors: the marking+// itself (the latency looks "accepted"), and the big invoices/lines JOIN,+// which is unchanged by the deploy.+func pctSeedMarkedSlowRegressed(s *seed.Scenario) error {+ p := s.Project("invoicing", seed.OTel)+ window := 14 * 24 * time.Hour+ deployAt := s.Ago(26 * time.Hour)+ api := p.Service("invoicing-api", seed.Go("nethttp", "sql")).Versions("4.7.3", seed.Deploy{At: deployAt, Version: "4.8.0"})++ export := api.Endpoint("GET", "/api/exports/invoices")+ export.Traffic(seed.Daily(110), seed.Diurnal(), seed.Over(window))+ export.Detail(seed.DetailPolicy{Recent: 20 * time.Hour, Rate: 0.3, Max: 14})+ export.Handler(func(r *seed.Req) {+ // Invoices in the month: most customers 60-100, large ones 112-125,+ // each with 8-12 lines.+ n := 60 + r.IntN(41)+ if r.Chance(0.12) {+ n = 112 + r.IntN(14)+ }+ lines := n * (8 + r.IntN(5))+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("SELECT i.id, i.number, i.issued_at, i.customer_name, i.total_cents, l.sku, l.description, l.qty, l.unit_price_cents FROM invoices i JOIN invoice_lines l ON l.invoice_id = i.id WHERE i.org_id = $1 AND i.issued_at >= $2 AND i.issued_at < $3 ORDER BY i.issued_at, i.id",+ time.Duration(float64(lines)*0.95*(0.95+0.1*r.Float())*float64(time.Millisecond)), seed.Rows(lines))+ if r.Version() == "4.8.0" {+ for range n {+ r.DB("SELECT rate_code, rate_bps, amount_cents FROM invoice_tax_lines WHERE invoice_id = $1", r.Ms(18, 27), seed.Rows(1+r.IntN(3)))+ }+ }+ r.Compute("encode csv", time.Duration(float64(lines)*0.28*(0.95+0.1*r.Float())*float64(time.Millisecond)))+ r.Gap(r.Ms(0.3, 0.9))+ })++ list := api.Endpoint("GET", "/api/invoices")+ list.Traffic(seed.Daily(240), seed.Diurnal(), seed.Over(window))+ list.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.0))+ r.DB("SELECT id, number, issued_at, customer_name, total_cents, status FROM invoices WHERE org_id = $1 ORDER BY issued_at DESC LIMIT 50", r.LogNormal(5*time.Millisecond, 20*time.Millisecond), seed.Rows(50))+ r.Compute("encode response", r.Ms(0.5, 1.8))+ })+ get := api.Endpoint("GET", "/api/invoices/:id")+ get.Traffic(seed.Daily(320), seed.Diurnal(), seed.Over(window))+ get.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.0))+ r.DB("SELECT id, number, issued_at, customer_name, total_cents, status FROM invoices WHERE id = $1", r.LogNormal(2*time.Millisecond, 8*time.Millisecond))+ r.DB("SELECT sku, description, qty, unit_price_cents FROM invoice_lines WHERE invoice_id = $1", r.LogNormal(2*time.Millisecond, 9*time.Millisecond))+ r.DB("SELECT rate_code, rate_bps, amount_cents FROM invoice_tax_lines WHERE invoice_id = $1", r.LogNormal(2*time.Millisecond, 8*time.Millisecond))+ r.Compute("encode response", r.Ms(0.3, 1.0))+ })+ create := api.Endpoint("POST", "/api/invoices")+ create.Traffic(seed.Daily(90), seed.Diurnal(), seed.Over(window))+ create.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.4))+ r.DB("INSERT INTO invoices (org_id, number, issued_at, customer_name, total_cents, status) VALUES ($1, $2, $3, $4, $5, 'open') RETURNING id", r.LogNormal(4*time.Millisecond, 14*time.Millisecond))+ r.DB("INSERT INTO invoice_lines (invoice_id, sku, description, qty, unit_price_cents) SELECT $1, * FROM unnest($2::text[], $3::text[], $4::int[], $5::bigint[])", r.LogNormal(4*time.Millisecond, 15*time.Millisecond))+ r.DB("INSERT INTO invoice_tax_lines (invoice_id, rate_code, rate_bps, amount_cents) SELECT $1, * FROM unnest($2::text[], $3::int[], $4::bigint[])", r.LogNormal(3*time.Millisecond, 12*time.Millisecond))+ })+ s.Background(api, seed.TypicalTraffic(6))+ api.Host().CPU(seed.Flat(0.22, 0.34))+ api.Host().Memory(seed.Flat(0.45, 0.53))++ p.MarkSlow("GET /api/exports/invoices", 2000, "Monthly invoice CSV for accounting: every invoice of the month with its lines. Up to 2s is expected.")++ s.Expose("deployAt", pctRFC3339(deployAt))+ s.Expose("slowRequest", export.Slowest(20*time.Hour))+ s.Expose("beforeRequest", pctRef(func() *seed.Request {+ return pctNearestBetween(export, deployAt.Add(-20*time.Hour), deployAt, export.PercentileBetween(0.9, s.Ago(window), deployAt))+ }))+ s.Expose("p95MsBefore", pctLazyMs(export, 0.95, s.Ago(window), deployAt))+ s.Expose("p95MsAfter", pctLazyMs(export, 0.95, deployAt, s.Now))+ s.Expose("p50MsAfter", pctLazyMs(export, 0.5, deployAt, s.Now))+ s.Expose("p95Ms24h", pctLazyMs(export, 0.95, s.Ago(24*time.Hour), s.Now))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_metric_p95_trap.go b/cli/test/evals/internal/scenarios/perf/pct_metric_p95_trap.gonew file mode 100644index 00000000..80b38741--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_metric_p95_trap.go@@ -0,0 +1,146 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/ctl-metric-p95-trap", pctSeedMetricP95Trap) }++// pctAfternoon is the afternoon traffic peak: up to 3x between 13:00 and+// 16:30 UTC, with 30-minute ramps.+func pctAfternoon(t time.Time) float64 {+ h := float64(t.Hour()) + float64(t.Minute())/60+ switch {+ case h < 12.5 || h >= 17:+ return 1+ case h < 13:+ return 1 + 2*(h-12.5)/0.5+ case h < 16.5:+ return 3+ default:+ return 3 - 2*(h-16.5)/0.5+ }+}++// pctBusy is the query hosts' busy CPU share under a load factor.+func pctBusy(load float64) float64 { return math.Min(0.985, 0.2+0.17*load) }++// pctSlowdown is how much CPU contention stretches in-process work: none+// below 60% busy, about 4x when the hosts are pegged.+func pctSlowdown(busy float64) float64 {+ if busy <= 0.6 {+ return 1+ }+ x := (busy - 0.6) / 0.38+ return 1 + 3*x*x+}++// pctSeedMetricP95Trap: every afternoon traffic triples and the two query+// hosts run at 97-100% CPU, stretching every endpoint (no single long pole).+// The user relays ops' "p95 CPU around 45%", which is what metrics query+// returns for --aggregation p95 over a day: the server has no quantiles for+// metric points and silently averages. Expected: cpu_saturation, reported+// with the max (or bucketed) CPU, not a fake percentile.+func pctSeedMetricP95Trap(s *seed.Scenario) error {+ p := s.Project("search", seed.OTel)+ window := 3 * 24 * time.Hour+ api := p.Service("query-api", seed.Go("nethttp", "pgx")).Instances("query-api-1", "query-api-2").Versions("8.2.1")++ load := func(t time.Time) float64 { return pctDiurnal(t) * pctAfternoon(t) }+ stretch := func(r *seed.Req) (cpu, db float64) {+ m := pctSlowdown(pctBusy(load(r.Start())))+ return m, 1 + (m-1)*0.25+ }+ query := api.Endpoint("GET", "/api/query")+ query.Traffic(seed.Daily(1300), seed.Diurnal(), seed.RateFunc(pctAfternoon), seed.Over(window))+ query.Detail(seed.DetailPolicy{Recent: 30 * time.Hour, Rate: 0.01, Max: 24})+ query.Handler(func(r *seed.Req) {+ cpu, db := stretch(r)+ ms := func(lo, hi, f float64) time.Duration { return time.Duration(float64(r.Ms(lo, hi)) * f) }+ r.Gap(ms(0.4, 1.2, cpu))+ r.Compute("parse query", ms(0.8, 2.5, cpu))+ r.DB("SELECT d.id, d.title, d.snippet, d.updated_at, ts_rank_cd(d.tsv, q) AS rank FROM documents d, websearch_to_tsquery('simple', $1) q WHERE d.tenant_id = $2 AND d.tsv @@ q ORDER BY rank DESC LIMIT 200",+ time.Duration(float64(r.LogNormal(18*time.Millisecond, 60*time.Millisecond))*db), seed.Rows(40+r.IntN(161)))+ r.Compute("rank results", ms(20, 55, cpu))+ r.Compute("render highlights", ms(8, 22, cpu))+ r.Gap(ms(0.3, 0.9, cpu))+ })+ suggest := api.Endpoint("GET", "/api/suggest")+ suggest.Traffic(seed.Daily(1100), seed.Diurnal(), seed.RateFunc(pctAfternoon), seed.Over(window))+ suggest.Handler(func(r *seed.Req) {+ cpu, db := stretch(r)+ ms := func(lo, hi, f float64) time.Duration { return time.Duration(float64(r.Ms(lo, hi)) * f) }+ r.Gap(ms(0.3, 0.8, cpu))+ r.DB("SELECT term, hits FROM suggest_terms WHERE tenant_id = $1 AND term LIKE $2 ORDER BY hits DESC LIMIT 10", time.Duration(float64(r.LogNormal(3*time.Millisecond, 10*time.Millisecond))*db), seed.Rows(10))+ r.Compute("encode response", ms(2, 6, cpu))+ })+ doc := api.Endpoint("GET", "/api/documents/:id")+ doc.Traffic(seed.Daily(700), seed.Diurnal(), seed.RateFunc(pctAfternoon), seed.Over(window))+ doc.Handler(func(r *seed.Req) {+ cpu, db := stretch(r)+ ms := func(lo, hi, f float64) time.Duration { return time.Duration(float64(r.Ms(lo, hi)) * f) }+ r.Gap(ms(0.3, 0.8, cpu))+ r.DB("SELECT id, title, body, updated_at FROM documents WHERE id = $1 AND tenant_id = $2", time.Duration(float64(r.LogNormal(3*time.Millisecond, 9*time.Millisecond))*db), seed.Rows(1))+ r.Compute("render document", ms(6, 18, cpu))+ })+ feedback := api.Endpoint("POST", "/api/feedback")+ feedback.Traffic(seed.Daily(240), seed.Diurnal(), seed.RateFunc(pctAfternoon), seed.Over(window))+ feedback.Handler(func(r *seed.Req) {+ cpu, db := stretch(r)+ ms := func(lo, hi, f float64) time.Duration { return time.Duration(float64(r.Ms(lo, hi)) * f) }+ r.Gap(ms(0.3, 0.8, cpu))+ r.DB("INSERT INTO search_feedback (tenant_id, query, document_id, verdict, created_at) VALUES ($1, $2, $3, $4, now())", time.Duration(float64(r.LogNormal(3*time.Millisecond, 10*time.Millisecond))*db))+ r.Compute("encode response", ms(0.3, 1, cpu))+ })++ for i, h := range api.Hosts {+ off := 0.01 * float64(i)+ busy := func(t time.Time) float64 {+ return math.Min(0.985, pctBusy(load(t))+off+0.02*(math.Sin(float64(t.Unix())/97)))+ }+ api.HostNamed(h).CPU(busy)+ api.HostNamed(h).Memory(seed.Flat(0.52, 0.6))+ }++ // What the hosts' CPU looks like to the different questions, from the+ // same busy curve the CPU series follow (one host, minute samples).+ busyAt := func(t time.Time) float64 {+ return math.Min(0.985, pctBusy(load(t))+0.02*math.Sin(float64(t.Unix())/97))+ }+ var sum, peak float64+ var mins []float64+ for t := s.Ago(24 * time.Hour); t.Before(s.Now); t = t.Add(time.Minute) {+ b := busyAt(t)+ sum += b+ peak = math.Max(peak, b)+ mins = append(mins, b)+ }+ s.Expose("busyAvg24h", math.Round(sum/float64(len(mins))*100))+ s.Expose("busyMax24h", math.Round(peak*100))++ // The most recent complete afternoon peak and a quiet morning before it.+ day := s.Now.Truncate(24 * time.Hour)+ peakFrom := day.Add(13 * time.Hour)+ if s.Now.Before(day.Add(16*time.Hour + 30*time.Minute)) {+ peakFrom = peakFrom.Add(-24 * time.Hour)+ }+ peakTo := peakFrom.Add(3*time.Hour + 30*time.Minute)+ quietFrom, quietTo := peakFrom.Add(-6*time.Hour), peakFrom.Add(-2*time.Hour)+ s.Expose("peakFrom", pctRFC3339(peakFrom))+ s.Expose("peakTo", pctRFC3339(peakTo))+ s.Expose("quietFrom", pctRFC3339(quietFrom))+ s.Expose("quietTo", pctRFC3339(quietTo))+ s.Expose("p95MsPeak", pctLazyMs(query, 0.95, peakFrom, peakTo))+ s.Expose("p95MsQuiet", pctLazyMs(query, 0.95, quietFrom, quietTo))+ s.Expose("peakRequest", pctRef(func() *seed.Request {+ return pctNearestBetween(query, peakFrom.Add(time.Hour), peakTo.Add(-time.Hour), query.PercentileBetween(0.9, peakFrom, peakTo))+ }))+ s.Expose("quietRequest", pctRef(func() *seed.Request {+ return pctNearestBetween(query, quietFrom, quietTo, query.PercentileBetween(0.9, quietFrom, quietTo))+ }))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_onset_bisect.go b/cli/test/evals/internal/scenarios/perf/pct_onset_bisect.gonew file mode 100644index 00000000..2cbb8e0a--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_onset_bisect.go@@ -0,0 +1,104 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/ctl-onset-bisect", pctSeedOnsetBisect) }++// pctSeedOnsetBisect: GET /api/account/summary picked up a monthly-spend+// aggregate (orders JOIN order_items GROUP BY month) in 2.32.0, deployed 52h+// ago, and its p95 stepped from ~20-25ms to ~450ms. Six heavier endpoints+// (imports, reports, avatar uploads, recommendations, search, login) keep it+// out of the top 5 of endpoints chart for p95 and p50, so the onset has to be+// found by comparing `endpoints list --search` windows.+// Distractor: data growth; the orders count query has been creeping up all+// along, so a coarse before/after comparison looks like gradual growth.+func pctSeedOnsetBisect(s *seed.Scenario) error {+ p := s.Project("members", seed.OTel)+ window := 10 * 24 * time.Hour+ deployAt := s.Ago(52 * time.Hour)+ api := p.Service("members-api", seed.Python("fastapi", "sqlalchemy")).Versions("2.31.4", seed.Deploy{At: deployAt, Version: "2.32.0"})++ summary := api.Endpoint("GET", "/api/account/summary")+ summary.Traffic(seed.Daily(130), seed.Diurnal(), seed.Over(window))+ summary.Detail(seed.DetailPolicy{Recent: 30 * time.Hour, Rate: 0.12, Max: 14})+ summary.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 1.8))+ r.DB("SELECT accounts.id, accounts.email, accounts.display_name, accounts.plan, accounts.created_at FROM accounts WHERE accounts.id = %(id_1)s", r.LogNormal(2*time.Millisecond, 6*time.Millisecond), seed.Rows(1))+ // orders grows every day; the count slowly gets dearer.+ r.DB("SELECT count(*) AS count_1 FROM orders WHERE orders.account_id = %(account_id_1)s",+ time.Duration(r.Lerp(4, 11)*(0.8+0.5*r.Float())*float64(time.Millisecond)), seed.Rows(1))+ r.DB("SELECT orders.id, orders.status, orders.total_cents, orders.created_at FROM orders WHERE orders.account_id = %(account_id_1)s ORDER BY orders.created_at DESC LIMIT %(param_1)s",+ r.LogNormal(3*time.Millisecond, 9*time.Millisecond), seed.Rows(5))+ if r.Version() == "2.32.0" {+ r.DB("SELECT date_trunc(%(date_trunc_1)s, orders.created_at) AS month, count(DISTINCT orders.id) AS orders, sum(order_items.qty * order_items.unit_price_cents) AS spend_cents FROM orders JOIN order_items ON order_items.order_id = orders.id WHERE orders.account_id = %(account_id_1)s GROUP BY date_trunc(%(date_trunc_1)s, orders.created_at) ORDER BY month",+ r.LogNormal(260*time.Millisecond, 520*time.Millisecond), seed.Rows(12+r.IntN(13)))+ }+ r.Compute("serialize response", r.Ms(0.8, 2.5))+ r.Gap(r.Ms(0.3, 0.9))+ })++ imports := api.Endpoint("POST", "/api/imports/contacts")+ imports.Traffic(seed.Daily(27), seed.Diurnal(), seed.Over(window))+ imports.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(1, 3))+ r.Compute("parse csv", r.LogNormal(1100*time.Millisecond, 2600*time.Millisecond))+ r.DB("INSERT INTO contacts (account_id, email, name, tags) VALUES (%(account_id_m0)s, %(email_m0)s, %(name_m0)s, %(tags_m0)s)", r.LogNormal(300*time.Millisecond, 700*time.Millisecond))+ })+ reports := api.Endpoint("GET", "/api/reports/activity")+ reports.Traffic(seed.Daily(90), seed.Diurnal(), seed.Over(window))+ reports.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 1.8))+ r.DB("SELECT activity_events.kind, count(*) AS count_1 FROM activity_events WHERE activity_events.account_id = %(account_id_1)s AND activity_events.created_at >= %(created_at_1)s GROUP BY activity_events.kind", r.LogNormal(700*time.Millisecond, 1500*time.Millisecond), seed.Rows(9))+ r.Compute("serialize response", r.Ms(1, 3))+ })+ avatar := api.Endpoint("POST", "/api/media/avatar")+ avatar.Traffic(seed.Daily(20), seed.Diurnal(), seed.Over(window))+ avatar.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(2, 6))+ r.Compute("resize image", r.LogNormal(650*time.Millisecond, 1300*time.Millisecond))+ r.DB("UPDATE accounts SET avatar_key=%(avatar_key)s WHERE accounts.id = %(id_1)s", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))+ })+ recs := api.Endpoint("GET", "/api/recommendations")+ recs.Traffic(seed.Daily(300), seed.Diurnal(), seed.Over(window))+ recs.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 1.8))+ r.HTTP("POST", "http://recs-model.internal:8501/v1/models/members:predict", r.LogNormal(430*time.Millisecond, 1100*time.Millisecond), 200)+ r.Compute("serialize response", r.Ms(0.5, 2))+ })+ search := api.Endpoint("GET", "/api/search")+ search.Traffic(seed.Daily(360), seed.Diurnal(), seed.Over(window))+ search.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 1.8))+ r.DB("SELECT contacts.id, contacts.name, contacts.email FROM contacts WHERE contacts.account_id = %(account_id_1)s AND contacts.search_vector @@ plainto_tsquery(%(plainto_tsquery_1)s) ORDER BY ts_rank(contacts.search_vector, plainto_tsquery(%(plainto_tsquery_1)s)) DESC LIMIT %(param_1)s",+ r.LogNormal(390*time.Millisecond, 900*time.Millisecond), seed.Rows(20))+ r.Compute("serialize response", r.Ms(0.5, 2))+ })+ login := api.Endpoint("POST", "/api/auth/login")+ login.Traffic(seed.Daily(260), seed.Diurnal(), seed.Over(window))+ login.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 1.8))+ r.DB("SELECT accounts.id, accounts.password_hash FROM accounts WHERE accounts.email = %(email_1)s", r.LogNormal(2*time.Millisecond, 6*time.Millisecond), seed.Rows(1))+ r.Compute("bcrypt.checkpw", r.Ms(400, 540))+ r.DB("INSERT INTO sessions (account_id, token_hash, expires_at) VALUES (%(account_id)s, %(token_hash)s, %(expires_at)s)", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))+ })+ s.Background(api, seed.TypicalTraffic(6))+ api.Host().CPU(seed.Flat(0.24, 0.36))+ api.Host().Memory(seed.Flat(0.55, 0.62))++ s.Expose("deployAt", pctRFC3339(deployAt))+ s.Expose("slowRequest", summary.Slowest(24*time.Hour))+ s.Expose("p95MsBefore24h", pctLazyMs(summary, 0.95, deployAt.Add(-24*time.Hour), deployAt))+ s.Expose("p95MsAfter24h", pctLazyMs(summary, 0.95, deployAt, deployAt.Add(24*time.Hour)))+ s.Expose("p95MsFirstDays", pctLazyMs(summary, 0.95, s.Ago(window), s.Ago(window-3*24*time.Hour)))+ // Window edges the oracle's bisection uses, as "<hours>h ago".+ for _, h := range []float64{240, 120, 60, 54, 52.5, 51, 48} {+ s.Expose(pctAgoKey(h), pctRFC3339(s.Now.Add(-time.Duration(h*float64(time.Hour)))))+ }+ s.Expose("primaryProject", "members")+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_xs_frontend_timeouts.go b/cli/test/evals/internal/scenarios/perf/pct_xs_frontend_timeouts.gonew file mode 100644index 00000000..3fd704e8--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_xs_frontend_timeouts.go@@ -0,0 +1,127 @@+package perf++import (+ "fmt"+ "net/url"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/xs-frontend-timeouts", pctSeedXSFrontendTimeouts) }++// pctSearchTerms are what shoppers type.+var pctSearchTerms = []string{"running shoes", "rain jacket", "wool socks", "trail", "linen shirt", "backpack", "kids boots", "yoga mat", "sunglasses", "denim"}++// pctSeedXSFrontendTimeouts: the web shop aborts product searches after 8s+// (AbortSignal.timeout) and reports the TimeoutError. Since ~10h ago the+// hosted search provider the API calls has been slow (median 1.5s, tail past+// 15s), so GET /api/search's p95 is ~8s. Every browser crash lines up with a+// backend request that ran past 8s; the link is the URL path and the time,+// since the browser SDK's trace id does not join OTLP traces. Expected:+// slow_third_party_upstream_api. Distractor: a frontend bug (the 8s timeout).+func pctSeedXSFrontendTimeouts(s *seed.Scenario) error {+ web := s.Project("shop-web", seed.Browser, "react")+ apiProj := s.Project("shop-api", seed.OTel)+ degradedAt := s.Ago(10*time.Hour + 20*time.Minute)+ api := apiProj.Service("shop-api", seed.Node("express", "pg")).Versions("3.12.0")++ search := api.Endpoint("GET", "/api/search")+ search.Traffic(seed.Daily(2000), seed.Diurnal(), seed.Over(3*24*time.Hour))+ search.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.03, Max: 20})+ search.Handler(func(r *seed.Req) {+ q := pctSearchTerms[r.IntN(len(pctSearchTerms))]+ r.RootAttr("url.path", "/api/search")+ r.RootAttr("url.query", "q="+url.QueryEscape(q))+ r.Gap(r.Ms(0.5, 1.5))+ provider := r.LogNormal(110*time.Millisecond, 380*time.Millisecond)+ if r.Start().After(degradedAt) {+ provider = r.LogNormal(1500*time.Millisecond, 16*time.Second)+ }+ r.HTTP("POST", "https://shop-prod.search-cloud.example/1/indexes/products/query", provider, 200,+ seed.A("http.request.body.size", 180+r.IntN(60)), seed.A("http.response.body.size", 9000+r.IntN(6000)))+ r.DB("SELECT id, stock, price_cents FROM product_stock WHERE id = ANY($1)", r.LogNormal(3*time.Millisecond, 9*time.Millisecond), seed.Rows(24))+ r.Compute("merge results", r.Ms(1, 4))+ })+ product := api.Endpoint("GET", "/api/products/:id")+ product.Traffic(seed.Daily(900), seed.Diurnal(), seed.Over(3*24*time.Hour))+ product.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.5))+ r.DB("SELECT id, name, slug, description, price_cents, images FROM products WHERE id = $1", r.LogNormal(3*time.Millisecond, 10*time.Millisecond), seed.Rows(1))+ r.Compute("serialize product", r.Ms(0.5, 2))+ })+ cart := api.Endpoint("POST", "/api/cart/items")+ cart.Traffic(seed.Daily(400), seed.Diurnal(), seed.Over(3*24*time.Hour))+ cart.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.5))+ r.DB("INSERT INTO cart_items (cart_id, product_id, qty) VALUES ($1, $2, $3) ON CONFLICT (cart_id, product_id) DO UPDATE SET qty = cart_items.qty + EXCLUDED.qty", r.LogNormal(3*time.Millisecond, 10*time.Millisecond))+ })+ s.Background(api, seed.TypicalTraffic(4))+ api.Host().CPU(seed.Flat(0.2, 0.3))+ api.Host().Memory(seed.Flat(0.5, 0.58))++ // The browser side: every search that ran past 8s on the API, for two+ // shoppers in three, ended in the web app's TimeoutError, reported with+ // the network and navigation actions the SDK captured.+ const timeout = 8 * time.Second+ browsers := []string{"Chrome 141 / macOS", "Chrome 141 / Windows", "Safari 18 / iOS", "Edge 141 / Windows", "Firefox 143 / Linux"}+ stack := "TimeoutError: signal timed out\nsearchProducts()\n src/api/search.ts:9:21\nrunSearch()\n src/pages/SearchPage.tsx:13:21"+ var newest *seed.Request+ crashes := 0+ // Crashes are derived from the generated backend requests, so they are+ // built lazily, once, when the scenario flushes.+ done := false+ build := func() {+ if done {+ return+ }+ done = true+ for i, r := range search.Requests() {+ if r.Start.Before(degradedAt) || r.Duration() < timeout+300*time.Millisecond || r.Start.After(s.Ago(2*time.Minute)) {+ continue+ }+ if i%3 == 0 {+ continue+ }+ r.KeepDetail()+ newest = r+ q, _ := r.Root.Attrs["url.query"].(string)+ at := r.Start.Add(timeout + 40*time.Millisecond)+ term, _ := url.QueryUnescape(q[2:])+ prev := r.Start.Add(-time.Duration(20+i%40) * time.Second)+ web.Crash(seed.Crash{+ At: at, Stack: stack, AppVersion: "2026.10.14", Device: browsers[i%len(browsers)],+ Attributes: map[string]string{"page": "/search", "release": "web-2026.10.14"},+ Actions: []seed.Action{+ {Type: "navigation", At: prev, From: "/", To: "/search"},+ {Type: "network", At: prev.Add(400 * time.Millisecond), Method: "GET", URL: "https://www.shop.example/api/products/" + fmt.Sprint(1000+i%900), DurationMs: float64(40 + i%60), Status: 200, RespBytes: 4200},+ {Type: "network", At: at, Method: "GET", URL: "https://www.shop.example/api/search?" + q, DurationMs: float64(timeout.Milliseconds()) + 2, Error: "TimeoutError: signal timed out"},+ },+ Logs: []seed.ActionLog{{At: at.Add(5 * time.Millisecond), Level: "warn", Message: fmt.Sprintf("search for %q gave up after 8000ms", term)}},+ })+ crashes+++ }+ }+ s.Expose("browserCrashes", func() any { build(); return crashes })+ s.Expose("slowRequest", func() any {+ build()+ if newest == nil {+ return nil+ }+ return newest.Ref()+ })+ s.Expose("degradedAt", pctRFC3339(degradedAt))+ s.Expose("p95MsBefore", pctLazyMs(search, 0.95, s.Ago(3*24*time.Hour), degradedAt))+ s.Expose("p95MsAfter", pctLazyMs(search, 0.95, degradedAt, s.Now))+ s.Expose("primaryProject", "shop-web")+ s.AfterFlush(func() error {+ hash, err := web.GroupMatching(24*time.Hour, "signal timed out")+ if err != nil {+ return err+ }+ s.Expose("hash", hash)+ return nil+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_expensive_join.go b/cli/test/evals/internal/scenarios/perf/pdb_expensive_join.gonew file mode 100644index 00000000..f2edfbeb--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_expensive_join.go@@ -0,0 +1,74 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-expensive-join", pdbSeedExpensiveJoin) }++// pdbSeedExpensiveJoin: the merchant revenue report joins 90 days of a+// store's orders to their items and products and aggregates by day and+// category on every request. That one query takes ~1.8s every time, at the+// quiet end of the night and at the daily peak alike, and has done so for+// the whole window (no growth, no deploy). Distractor: saturation, which ops+// suspects. CPU moves gently with the daily traffic and stays under 40%, and+// every other endpoint is fast at the same moments.+func pdbSeedExpensiveJoin(s *seed.Scenario) error {+ window := 14 * 24 * time.Hour+ p := s.Project("merchant-hub", seed.OTel)+ api := p.Service("merchant-api", seed.PHP("laravel")).Versions("2.41.0")++ auth := func(r *seed.Req) {+ r.Gap(r.Ms(2.5, 5.5)) // framework boot, session and auth middleware+ r.DB("select * from personal_access_tokens where token = ? limit 1", r.Ms(0.5, 1.3), seed.Rows(1))+ }++ report := api.Endpoint("GET", "/api/stores/{store}/reports/revenue")+ report.Traffic(seed.Daily(200), seed.Diurnal(), seed.Over(window))+ report.Detail(seed.DetailPolicy{Recent: 18 * time.Hour, Rate: 0.06, Max: 12})+ report.Handler(func(r *seed.Req) {+ auth(r)+ r.DB("select stores.id, stores.name, stores.currency from stores where stores.id = ? and stores.merchant_id = ? limit 1", r.Ms(0.5, 1.2), seed.Rows(1))+ r.DB("select date(orders.created_at) as day, products.category_id, count(distinct orders.customer_id) as buyers, sum(order_items.quantity * order_items.unit_price) as revenue from orders inner join order_items on order_items.order_id = orders.id inner join products on products.id = order_items.product_id where orders.store_id = ? and orders.status = ? and orders.created_at >= ? group by day, products.category_id order by day asc",+ r.Ms(1650, 1990), seed.Rows(700+r.IntN(260)))+ r.Gap(r.Ms(3, 7)) // collection mapping and the JSON response+ })++ orders := api.Endpoint("GET", "/api/stores/{store}/orders")+ orders.Traffic(seed.Daily(220), seed.Diurnal(), seed.Over(window))+ orders.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.02, Max: 6})+ orders.Handler(func(r *seed.Req) {+ auth(r)+ r.DB("select count(*) as aggregate from orders where orders.store_id = ?", r.Ms(1.5, 4), seed.Rows(1))+ r.DB("select orders.id, orders.number, orders.status, orders.total, orders.customer_id, orders.created_at from orders where orders.store_id = ? order by orders.created_at desc limit 25 offset 0", r.Ms(1.2, 3.5), seed.Rows(25))+ r.Gap(r.Ms(1.5, 4))+ })++ pdbHealthy(api, window,+ pdbRoute{method: "GET", route: "/api/user", perDay: 90, queries: []string{+ "select * from personal_access_tokens where token = ? limit 1",+ "select * from users where users.id = ? limit 1",+ }, dbMedianMs: 0.8},+ pdbRoute{method: "GET", route: "/api/stores/{store}/orders/{order}", perDay: 80, queries: []string{+ "select orders.id, orders.number, orders.status, orders.total, orders.customer_id, orders.created_at from orders where orders.id = ? and orders.store_id = ? limit 1",+ "select order_items.id, order_items.product_id, order_items.quantity, order_items.unit_price from order_items where order_items.order_id = ?",+ }, dbMedianMs: 1.2},+ pdbRoute{method: "GET", route: "/api/stores/{store}/products", perDay: 60, queries: []string{+ "select products.id, products.sku, products.name, products.price, products.category_id from products where products.store_id = ? order by products.name asc limit 50 offset 0",+ }, dbMedianMs: 2.5},+ pdbRoute{method: "PUT", route: "/api/stores/{store}/products/{product}", perDay: 15, queries: []string{+ "update products set name = ?, price = ?, updated_at = ? where id = ?",+ }, dbMedianMs: 1.6},+ )+ api.Host().CPU(seed.DiurnalSeries(seed.Flat(0.22, 0.27), 0.3))+ api.Host().Memory(seed.Flat(0.52, 0.58))++ s.Expose("slowRequest", pdbSlowest(report, s.Ago(18*time.Hour), nil))+ s.Expose("p50Ms24h", pdbMs(report, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95Ms24h", pdbMs(report, 0.95, s.Ago(24*time.Hour), s.Now))+ s.Expose("p50MsFirstDays", pdbMs(report, 0.5, s.Ago(window), s.Ago(window-3*24*time.Hour)))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_lock_contention.go b/cli/test/evals/internal/scenarios/perf/pdb_lock_contention.gonew file mode 100644index 00000000..662e5bfd--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_lock_contention.go@@ -0,0 +1,165 @@+package perf++import (+ "fmt"+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-lock-contention", pdbSeedLockContention) }++// pdbSeedLockContention: every evening at 18:00 UTC a product drop sends+// reservations for the same few SKUs. A reservation updates its cart's+// inventory rows one by one, in cart order, inside one transaction, so+// concurrent reservations queue on the hot rows' locks: the UPDATE inventory+// spans themselves run for hundreds of ms to seconds, the transaction starts+// at once (no gap before the first query), and carts that lock two hot SKUs+// in opposite orders deadlock (40P01 after deadlock_timeout) or give up at+// lock_timeout (55P03). Distractor: connection-pool exhaustion. Release+// 1.32.0 a week ago raised the pgx pool from 20 to 40 connections and the+// evening spikes did not change; there is no wait before BEGIN.+func pdbSeedLockContention(s *seed.Scenario) error {+ // Nine days, so the 42 hours before the deploy hold a full evening drop+ // whatever the hour of the run.+ window := 9 * 24 * time.Hour+ deployAt := s.Ago(7*24*time.Hour + 6*time.Hour)+ p := s.Project("fulfillment", seed.OTel)+ api := p.Service("inventory-api", seed.Go("nethttp", "pgx")).+ Versions("1.31.4", seed.Deploy{At: deployAt, Version: "1.32.0"})++ // heat is 1 during the drop (18:00-18:25 UTC) and fades out by 18:45.+ heat := func(t time.Time) float64 {+ m := float64(t.Hour()*60+t.Minute()) + float64(t.Second())/60+ switch {+ case m >= 18*60 && m < 18*60+25:+ return 1+ case m >= 18*60+25 && m < 18*60+45:+ return (18*60 + 45 - m) / 20+ }+ return 0+ }+ load := func(t time.Time) float64 { return 1 + 6*heat(t) }++ hot := []string{"SKU-DROP-1187", "SKU-DROP-1188", "SKU-DROP-1190"}+ pickSKU := func(r *seed.Req, h float64) string {+ if h > 0 && r.Chance(0.85) {+ return hot[r.IntN(len(hot))]+ }+ return fmt.Sprintf("SKU-%05d", 10000+r.IntN(80000))+ }+ const updateSQL = "UPDATE inventory SET reserved = reserved + $1, updated_at = now() WHERE sku = $2 AND warehouse_id = $3 AND on_hand - reserved >= $1"+ pid := func(r *seed.Req) int { return 41000 + r.IntN(9000) }+ tuple := func(r *seed.Req) string {+ return fmt.Sprintf("while updating tuple (%d,%d) in relation \"inventory\"", 1800+r.IntN(60), 1+r.IntN(40))+ }++ reserve := api.Endpoint("POST", "/api/reservations")+ reserve.Traffic(seed.Daily(700), seed.RateFunc(load), seed.Over(window))+ reserve.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.03, Max: 10})+ reserve.Handler(func(r *seed.Req) {+ h := heat(r.Start())+ lines := 1 + r.IntN(4)+ cart := fmt.Sprintf("c_%08x", r.IntN(1<<31))+ r.Gap(r.Ms(0.3, 0.9)) // decode the cart+ r.DB("begin", r.Ms(0.2, 0.6))+ r.DB("SET LOCAL lock_timeout = '5s'", r.Ms(0.1, 0.3))+ hotLocked := 0+ for i := 0; i < lines; i++ {+ sku := pickSKU(r, h)+ isHot := len(sku) > 8 && sku[:8] == "SKU-DROP"+ d := r.Ms(0.6, 1.8)+ if isHot && h > 0 && r.Chance(0.4+0.5*h) {+ // Waiting for the row lock held by another reservation.+ d += time.Duration(h * float64(r.LogNormal(380*time.Millisecond, 2800*time.Millisecond)))+ }+ if isHot {+ hotLocked+++ }+ switch {+ case d >= 5*time.Second:+ sp := r.DB(updateSQL, 5*time.Second+r.Ms(0.5, 2))+ pdbPgError(r, sp, "ERROR: canceling statement due to lock timeout (SQLSTATE 55P03)")+ r.DB("rollback", r.Ms(0.2, 0.6))+ r.Log(seed.SevWarn, "reserve stock: ERROR: canceling statement due to lock timeout (SQLSTATE 55P03)",+ seed.A("cart_id", cart), seed.A("sku", sku), seed.A("pg_where", tuple(r)))+ r.Status(503)+ return+ case hotLocked >= 2 && h > 0 && r.Chance(0.07):+ // Another cart holds this row and waits for one we hold.+ sp := r.DB(updateSQL, time.Second+r.Ms(1, 6))+ pdbPgError(r, sp, "ERROR: deadlock detected (SQLSTATE 40P01)")+ r.DB("rollback", r.Ms(0.2, 0.6))+ a, b := pid(r), pid(r)+ tx1, tx2 := 9917000+r.IntN(900), 9917000+r.IntN(900)+ r.Log(seed.SevError, "reserve stock: ERROR: deadlock detected (SQLSTATE 40P01)",+ seed.A("cart_id", cart), seed.A("sku", sku),+ seed.A("pg_detail", fmt.Sprintf("Process %d waits for ShareLock on transaction %d; blocked by process %d.\nProcess %d waits for ShareLock on transaction %d; blocked by process %d.", a, tx1, b, b, tx2, a)),+ seed.A("pg_where", tuple(r)))+ r.Status(500)+ return+ }+ r.DB(updateSQL, d, seed.Rows(1))+ }+ r.DB("INSERT INTO reservations (cart_id, sku, warehouse_id, qty, expires_at) SELECT $1, unnest($2::text[]), $3, unnest($4::int[]), now() + interval '15 minutes'", r.Ms(0.8, 2.2))+ r.DB("commit", r.Ms(0.8, 2.6))+ r.Gap(r.Ms(0.2, 0.5))+ r.Status(201)+ })++ release := api.Endpoint("POST", "/api/reservations/{id}/release")+ release.Traffic(seed.Daily(160), seed.RateFunc(load), seed.Over(window))+ release.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.02, Max: 3})+ release.Handler(func(r *seed.Req) {+ h := heat(r.Start())+ r.Gap(r.Ms(0.3, 0.8))+ r.DB("begin", r.Ms(0.2, 0.6))+ r.DB("DELETE FROM reservations WHERE id = $1 RETURNING sku, warehouse_id, qty", r.Ms(0.6, 1.6), seed.Rows(1))+ d := r.Ms(0.6, 1.8)+ if h > 0 && r.Chance(0.5*h) {+ d += time.Duration(h * float64(r.LogNormal(250*time.Millisecond, 2*time.Second)))+ }+ r.DB("UPDATE inventory SET reserved = reserved - $1, updated_at = now() WHERE sku = $2 AND warehouse_id = $3", time.Duration(math.Min(float64(d), float64(4900*time.Millisecond))), seed.Rows(1))+ r.DB("commit", r.Ms(0.8, 2.4))+ r.Gap(r.Ms(0.2, 0.5))+ })++ pdbHealthy(api, window,+ pdbRoute{method: "GET", route: "/api/inventory/{sku}", perDay: 220, queries: []string{+ "SELECT sku, warehouse_id, on_hand, reserved, updated_at FROM inventory WHERE sku = $1",+ }, dbMedianMs: 0.9},+ pdbRoute{method: "GET", route: "/api/warehouses", perDay: 30, queries: []string{+ "SELECT id, code, name, timezone FROM warehouses ORDER BY code",+ }, dbMedianMs: 0.8},+ )+ api.Host().CPU(seed.Plus(seed.Flat(0.16, 0.24), func(t time.Time) float64 { return 0.04 * heat(t) }))+ api.Host().Memory(seed.Flat(0.38, 0.43))++ day := s.Ago(24 * time.Hour)+ s.Expose("deployAt", deployAt.Format(time.RFC3339))+ s.Expose("slowRequest", pdbSlowest(reserve, day, func(r *seed.Request) bool { return r.Status == 201 }))+ s.Expose("p50Ms24h", pdbMs(reserve, 0.5, day, s.Now))+ s.Expose("p99Ms24h", pdbMs(reserve, 0.99, day, s.Now))+ s.Expose("failures", func() any {+ n := 0+ for _, r := range reserve.Requests() {+ if r.Status >= 500 {+ n+++ }+ }+ return n+ })+ return nil+}++// pdbPgError records the error otelpgx attaches to a failed query span.+func pdbPgError(r *seed.Req, sp *seed.Span, msg string) {+ sp.Status = seed.StatusError+ sp.StatusMsg = msg+ sp.Events = append(sp.Events, seed.Event{Name: "exception", At: sp.Dur, Attrs: map[string]any{+ "exception.type": "*pgconn.PgError", "exception.message": msg,+ }})+ r.KeepDetail()+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_missing_query_cache.go b/cli/test/evals/internal/scenarios/perf/pdb_missing_query_cache.gonew file mode 100644index 00000000..4dd4a538--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_missing_query_cache.go@@ -0,0 +1,98 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-missing-query-cache", pdbSeedMissingQueryCache) }++// pdbSeedMissingQueryCache: a global NestJS guard loads every feature flag+// from Postgres on every request, the same parameterless query returning the+// same 214 rows, before any handler runs. It costs ~15-30ms per request and+// is most of the time of the cheap endpoints (GET /api/me reads one customer+// row in ~1.5ms). Distractor: no cache / a low hit ratio. Product data is+// already behind Redis and the app's cache.hit_ratio gauge sits at 96-99%;+// the uncached thing is one specific query.+func pdbSeedMissingQueryCache(s *seed.Scenario) error {+ window := 3 * 24 * time.Hour+ p := s.Project("storefront-web", seed.OTel)+ api := p.Service("storefront-api", seed.Node("nestjs", "pg")).Versions("7.4.2")++ const flagsSQL = "SELECT key, enabled, rollout_percent, rules FROM feature_flags WHERE archived_at IS NULL ORDER BY key"+ guard := func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.9)) // session middleware+ r.DB(flagsSQL, r.LogNormal(19*time.Millisecond, 34*time.Millisecond), seed.Rows(214))+ }+ cached := func(r *seed.Req, key, missSQL string, missMs [2]float64) {+ r.Cache("GET", key, r.Ms(0.3, 0.9))+ if r.Chance(0.02) {+ r.DB(missSQL, r.Ms(missMs[0], missMs[1]), seed.Rows(1))+ r.Cache("SET", key, r.Ms(0.3, 0.8))+ }+ }+ type route struct {+ method, path, handler string+ perDay int+ body func(r *seed.Req)+ }+ routes := []route{+ {"GET", "/api/me", "MeController.getMe", 900, func(r *seed.Req) {+ r.DB("SELECT id, email, first_name, last_name, loyalty_tier FROM customers WHERE id = $1", r.Ms(0.8, 2.2), seed.Rows(1))+ }},+ {"GET", "/api/products/:id", "ProductsController.findOne", 1100, func(r *seed.Req) {+ cached(r, fmt.Sprintf("product:%d", 10000+r.IntN(50000)),+ "SELECT id, slug, name, price_cents, currency, images, attributes FROM products WHERE id = $1", [2]float64{3, 7})+ }},+ {"GET", "/api/categories/:slug/products", "ProductsController.byCategory", 700, func(r *seed.Req) {+ cached(r, fmt.Sprintf("category:%d:page:1", 100+r.IntN(300)),+ "SELECT id, slug, name, price_cents, currency, thumbnail_url FROM products WHERE category_id = $1 ORDER BY rank LIMIT 48", [2]float64{6, 14})+ }},+ {"GET", "/api/cart", "CartController.get", 650, func(r *seed.Req) {+ r.DB("SELECT l.sku, l.qty, l.unit_price_cents FROM cart_lines l WHERE l.cart_id = $1", r.Ms(1.2, 3.2), seed.Rows(1+r.IntN(5)))+ }},+ {"POST", "/api/cart/lines", "CartController.addLine", 260, func(r *seed.Req) {+ r.DB("INSERT INTO cart_lines (cart_id, sku, qty, unit_price_cents) VALUES ($1, $2, $3, $4) ON CONFLICT (cart_id, sku) DO UPDATE SET qty = cart_lines.qty + EXCLUDED.qty", r.Ms(1.5, 3.8))+ }},+ }+ var me, category *seed.Endpoint+ for _, rt := range routes {+ rt := rt+ e := api.Endpoint(rt.method, rt.path)+ e.Traffic(seed.Daily(rt.perDay), seed.Diurnal(), seed.Over(window))+ e.Detail(seed.DetailPolicy{Recent: 8 * time.Hour, Rate: 0.03, Max: 6})+ e.Handler(func(r *seed.Req) {+ guard(r)+ r.Span(rt.handler, func(r *seed.Req) {+ rt.body(r)+ r.Gap(r.Ms(0.3, 1.2)) // serialize+ })+ r.Gap(r.Ms(0.1, 0.4))+ })+ switch rt.path {+ case "/api/me":+ me = e+ case "/api/categories/:slug/products":+ category = e+ }+ }++ // Gauge adds +-1.5% jitter, so this reads about 96-99.9% and never above 1.+ api.Gauge("cache.hit_ratio", "1", map[string]any{"cache.name": "product"}, seed.Flat(0.972, 0.984))+ api.Host().CPU(seed.DiurnalSeries(seed.Flat(0.24, 0.3), 0.25))+ api.Host().Memory(seed.Flat(0.47, 0.52))++ day := s.Ago(24 * time.Hour)+ s.Expose("slowRequest", pdbSlowest(me, day, nil))+ // The newest request of these two endpoints keeps its trace and access+ // log, so a recent one is there at any hour; overnight the 3% sample+ // alone can leave the last eight hours without one.+ s.Expose("latestMeRequest", me.Pick(func(*seed.Request) bool { return true }))+ s.Expose("latestCategoryRequest", category.Pick(func(*seed.Request) bool { return true }))+ s.Expose("meP50Ms24h", pdbMs(me, 0.5, day, s.Now))+ s.Expose("meP95Ms24h", pdbMs(me, 0.95, day, s.Now))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_over_fetching.go b/cli/test/evals/internal/scenarios/perf/pdb_over_fetching.gonew file mode 100644index 00000000..9aa4613e--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_over_fetching.go@@ -0,0 +1,99 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-over-fetching", pdbSeedOverFetching) }++// pdbSeedOverFetching: the category grid loads its 48 products with SELECT *.+// Release 4.8.0 added products.supplier_payload (the supplier's raw feed,+// about 120KB per product), and the importer filled it in over the next day+// and a half. The grid query returns the same 48 rows as before but now+// drags every payload across the wire and through the driver's JSON parsing,+// so its span grows from ~12ms to ~400ms and the Node heap and host memory+// climb with it. The search endpoint reads the same table, 48 rows, with an+// explicit column list and stays fast. Distractor: JSON encoding (a+// teammate's guess): the response is 48 small cards, encoded in a couple of+// milliseconds after the query, and CPU rises only modestly.+func pdbSeedOverFetching(s *seed.Scenario) error {+ window := 14 * 24 * time.Hour+ deployAt := s.Ago(10*24*time.Hour + 3*time.Hour)+ backfill := 36 * time.Hour+ filled := func(t time.Time) float64 {+ if t.Before(deployAt) {+ return 0+ }+ return math.Min(1, float64(t.Sub(deployAt))/float64(backfill))+ }+ p := s.Project("shopfront", seed.OTel)+ api := p.Service("catalog-api", seed.Node("express", "pg")).+ Versions("4.7.2", seed.Deploy{At: deployAt, Version: "4.8.0"})++ grid := api.Endpoint("GET", "/api/categories/:categoryId/products")+ grid.Traffic(seed.Daily(260), seed.Diurnal(), seed.Over(window))+ grid.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.12, Max: 12})+ grid.Handler(func(r *seed.Req) {+ f := filled(r.Start())+ r.Gap(r.Ms(0.4, 1.1)) // session + query middleware+ r.DB("SELECT id, name, slug FROM categories WHERE id = $1", r.Ms(0.5, 1.4), seed.Rows(1))+ width := r.Ms(9, 16) + time.Duration(f*float64(r.Ms(300, 470)))+ r.DB("SELECT * FROM products WHERE category_id = $1 AND is_active = true ORDER BY sort_rank DESC LIMIT 48", width, seed.Rows(48))+ r.Gap(r.Ms(1.2, 2.8)) // map 48 rows to cards, res.json+ })++ product := api.Endpoint("GET", "/api/products/:id")+ product.Traffic(seed.Daily(180), seed.Diurnal(), seed.Over(window))+ product.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.03, Max: 4})+ product.Handler(func(r *seed.Req) {+ f := filled(r.Start())+ r.Gap(r.Ms(0.4, 1.1))+ r.DB("SELECT * FROM products WHERE id = $1", r.Ms(0.8, 2.0)+time.Duration(f*float64(r.Ms(6, 11))), seed.Rows(1))+ r.DB("SELECT id, url, alt, position FROM product_images WHERE product_id = $1 ORDER BY position", r.Ms(0.6, 1.6), seed.Rows(6))+ r.Gap(r.Ms(0.4, 1.0))+ })++ search := api.Endpoint("GET", "/api/search")+ search.Traffic(seed.Daily(150), seed.Diurnal(), seed.Over(window))+ search.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.04, Max: 4})+ search.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.1))+ r.DB("SELECT id, name, slug, price_cents, currency, thumbnail_url FROM products WHERE is_active = true AND search_vector @@ plainto_tsquery('english', $1) ORDER BY ts_rank(search_vector, plainto_tsquery('english', $1)) DESC LIMIT 48",+ r.LogNormal(11*time.Millisecond, 26*time.Millisecond), seed.Rows(48))+ r.Gap(r.Ms(0.8, 1.8))+ })++ pdbHealthy(api, window,+ pdbRoute{method: "GET", route: "/api/products/:id/price", perDay: 60, queries: []string{+ "SELECT id, price_cents, currency, compare_at_cents FROM products WHERE id = $1",+ }, dbMedianMs: 0.9},+ pdbRoute{method: "GET", route: "/api/brands", perDay: 25, queries: []string{+ "SELECT id, name, slug, logo_url FROM brands ORDER BY name",+ }, dbMedianMs: 1.5},+ pdbRoute{method: "GET", route: "/api/products/:id/reviews", perDay: 40, queries: []string{+ "SELECT id, rating, title, body, author_name, created_at FROM reviews WHERE product_id = $1 ORDER BY created_at DESC LIMIT 20",+ }, dbMedianMs: 2},+ )++ busy := func(t time.Time) float64 { return 0.04 * filled(t) }+ api.Host().CPU(seed.Plus(seed.Flat(0.21, 0.29), busy))+ api.Host().Memory(seed.Plus(seed.Flat(0.45, 0.49), func(t time.Time) float64 { return 0.19 * filled(t) }))+ heapMB := seed.Plus(seed.Sawtooth(45*time.Minute, 150, 210), func(t time.Time) float64 { return filled(t) * 340 })+ api.Gauge("v8js.memory.heap.used", "By", map[string]any{"v8js.heap.space.name": "old_space"},+ func(t time.Time) float64 { return heapMB(t) * 1024 * 1024 })++ s.Expose("slowRequest", pdbSlowest(grid, s.Ago(24*time.Hour), nil))+ // The newest grid request keeps its trace and access log, so a recent+ // one is there at any hour; overnight the sample alone can leave the+ // last six hours without one.+ s.Expose("latestRequest", grid.Pick(func(*seed.Request) bool { return true }))+ s.Expose("deployAt", deployAt.Format(time.RFC3339))+ s.Expose("gridP50Ms24h", pdbMs(grid, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("gridP50MsBeforeDeploy", pdbMs(grid, 0.5, s.Ago(window), deployAt))+ s.Expose("searchP50Ms24h", pdbMs(search, 0.5, s.Ago(24*time.Hour), s.Now))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_unbounded_result.go b/cli/test/evals/internal/scenarios/perf/pdb_unbounded_result.gonew file mode 100644index 00000000..79b95f0c--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_unbounded_result.go@@ -0,0 +1,109 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-unbounded-result", pdbSeedUnboundedResult) }++// pdbSeedUnboundedResult: the admin audit log returns every event an+// organization ever recorded, with no LIMIT. Events accrue every day, so the+// rows each request returns (db.response.returned_rows) grow week over week,+// the query span and the controller's serialization grow with them at a+// constant cost per row, and p50 creeps up with no deploy. Distractor: a+// missing index (the user's guess). The dashboard widget reads the same table+// through the same (org_id, created_at) index with LIMIT 20 in ~2ms, and the+// big query's cost per row returned stays flat while the rows grow.+func pdbSeedUnboundedResult(s *seed.Scenario) error {+ window := 29 * 24 * time.Hour+ p := s.Project("ledgerly", seed.OTel)+ api := p.Service("admin-api", seed.Java("spring")).Versions("3.18.1")++ // An organization's event count at the window start; events keep+ // accruing at a rate proportional to its size.+ orgEvents := func(r *seed.Req) float64 {+ switch x := r.Float(); {+ case x < 0.6:+ return 1500 + 4500*r.Float()+ case x < 0.9:+ return 8000 + 12000*r.Float()+ default:+ return 30000 + 30000*r.Float()+ }+ }++ audit := api.Endpoint("GET", "/api/admin/audit-log")+ audit.Traffic(seed.Daily(110), seed.Diurnal(), seed.Over(window))+ // Detail is chosen here rather than by a DetailPolicy (whose cap counts+ // forced requests): a quarter of the last 12 hours, plus a few traces+ // from every week.+ audit.Detail(seed.DetailPolicy{})+ recentCut := s.Ago(12 * time.Hour)+ audit.Handler(func(r *seed.Req) {+ n := int(orgEvents(r) * (1 + 1.3*r.Progress()))+ if r.Chance(0.012) || (!r.Start().Before(recentCut) && r.Chance(0.25)) {+ r.KeepDetail()+ }+ r.Gap(r.Ms(0.6, 1.8)) // Spring Security filter chain+ r.Span("AuditLogController.list", func(r *seed.Req) {+ r.DB("SELECT m.role FROM org_members m WHERE m.org_id = ? AND m.user_id = ?", r.Ms(0.5, 1.4), seed.Rows(1))+ perRow := 0.017 + 0.003*r.Float()+ r.DB("SELECT e.id, e.actor_id, e.action, e.target_type, e.target_id, e.ip_address, e.user_agent, e.created_at FROM audit_events e WHERE e.org_id = ? ORDER BY e.created_at DESC",+ pdbDur(1.5+perRow*float64(n)), seed.Rows(n))+ r.Gap(pdbDur(0.011 * float64(n) * (0.9 + 0.2*r.Float()))) // map rows, Jackson writes the JSON array+ })+ r.Gap(r.Ms(0.2, 0.6))+ })++ recent := api.Endpoint("GET", "/api/admin/dashboard/activity")+ recent.Traffic(seed.Daily(140), seed.Diurnal(), seed.Over(window))+ recent.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.04, Max: 4})+ recent.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.8))+ r.Span("AuditLogController.activity", func(r *seed.Req) {+ r.DB("SELECT m.role FROM org_members m WHERE m.org_id = ? AND m.user_id = ?", r.Ms(0.5, 1.4), seed.Rows(1))+ r.DB("SELECT e.id, e.actor_id, e.action, e.created_at FROM audit_events e WHERE e.org_id = ? ORDER BY e.created_at DESC LIMIT 20",+ r.Ms(1.1, 2.6), seed.Rows(20))+ r.Gap(r.Ms(0.3, 0.8))+ })+ r.Gap(r.Ms(0.2, 0.6))+ })++ settings := api.Endpoint("PUT", "/api/admin/settings")+ settings.Traffic(seed.Daily(18), seed.Diurnal(), seed.Over(window))+ settings.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.8))+ r.Span("SettingsController.update", func(r *seed.Req) {+ r.DB("UPDATE org_settings SET locale = ?, timezone = ?, invoice_prefix = ?, updated_at = now() WHERE org_id = ?", r.Ms(1.0, 2.5))+ r.DB("INSERT INTO audit_events (org_id, actor_id, action, target_type, target_id, ip_address, user_agent, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, now())", r.Ms(0.8, 2.0))+ })+ r.Gap(r.Ms(0.2, 0.6))+ })++ pdbHealthy(api, window,+ pdbRoute{method: "GET", route: "/api/admin/members", perDay: 50, queries: []string{+ "SELECT u.id, u.email, u.full_name, m.role, m.joined_at FROM org_members m JOIN users u ON u.id = m.user_id WHERE m.org_id = ? ORDER BY u.full_name",+ }, dbMedianMs: 2.5},+ pdbRoute{method: "GET", route: "/api/admin/settings", perDay: 35, queries: []string{+ "SELECT s.locale, s.timezone, s.invoice_prefix, s.updated_at FROM org_settings s WHERE s.org_id = ?",+ }, dbMedianMs: 1},+ pdbRoute{method: "GET", route: "/api/admin/invoices", perDay: 30, queries: []string{+ "SELECT i.id, i.number, i.status, i.total_cents, i.issued_at FROM invoices i WHERE i.org_id = ? ORDER BY i.issued_at DESC LIMIT 50",+ }, dbMedianMs: 2},+ )+ api.Host().CPU(seed.Flat(0.19, 0.28))+ api.Host().Memory(seed.Flat(0.56, 0.62))++ s.Expose("slowRequest", pdbSlowest(audit, s.Ago(24*time.Hour), nil))+ // The newest widget request keeps its trace and access log, so a fast+ // LIMIT 20 query to compare with is there at any hour; overnight the 4%+ // sample alone can leave the last six hours without one.+ s.Expose("widgetRequest", recent.Pick(func(*seed.Request) bool { return true }))+ s.Expose("p50Ms24h", pdbMs(audit, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95Ms24h", pdbMs(audit, 0.95, s.Ago(24*time.Hour), s.Now))+ s.Expose("p50MsFirstWeek", pdbMs(audit, 0.5, s.Ago(window), s.Ago(window-7*24*time.Hour)))+ return nil+}diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/README.md b/cli/test/evals/testdata/repos/accounts-api-dotnet/README.mdnew file mode 100644index 00000000..b83f70c9--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/README.md@@ -0,0 +1,17 @@+# accounts-api++Accounts and transactions for the finance dashboard. ASP.NET Core 9 minimal+APIs, PostgreSQL through Npgsql + Dapper. Traces, logs and metrics go to+Traceway over OTLP (`OTEL_EXPORTER_OTLP_ENDPOINT`); the OpenTelemetry setup is+in `Program.cs`.++```+dotnet test+dotnet run --project src/AccountsApi+```++Routes:++- `GET /api/accounts/{accountId}` – one account with its balance+- `GET /api/accounts/{accountId}/transactions` – a page of transactions, newest first+- `GET /api/accounts/{accountId}/transactions/export?from=&to=` – every transaction in a period as JSON (finance downloads these for reconciliation)diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Accounts/AccountEndpoints.cs b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Accounts/AccountEndpoints.csnew file mode 100644index 00000000..93abbdbd--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Accounts/AccountEndpoints.cs@@ -0,0 +1,26 @@+using System.Text.Json;+using Accounts.Api.Data;+using Accounts.Api.Serialization;++namespace Accounts.Api.Accounts;++public static class AccountEndpoints+{+ private static readonly JsonSerializerOptions PageOptions = new(JsonSerializerDefaults.Web)+ {+ Converters = { new UtcDateTimeConverter() },+ };++ public static void MapAccountEndpoints(this IEndpointRouteBuilder app)+ {+ app.MapGet("/api/accounts/{accountId}", async (long accountId, AccountRepository accounts, CancellationToken ct) =>+ await accounts.FindAsync(accountId, ct) is { } a ? Results.Ok(a) : Results.NotFound()).RequireAuthorization();++ app.MapGet("/api/accounts/{accountId}/transactions", async (long accountId, int? page, TransactionRepository transactions, CancellationToken ct) =>+ {+ const int pageSize = 50;+ var rows = await transactions.PageAsync(accountId, pageSize, ((page ?? 1) - 1) * pageSize, ct);+ return Results.Json(rows, PageOptions);+ }).RequireAuthorization();+ }+}diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/AccountsApi.csproj b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/AccountsApi.csprojnew file mode 100644index 00000000..bc97cfa8--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/AccountsApi.csproj@@ -0,0 +1,20 @@+<Project Sdk="Microsoft.NET.Sdk.Web">++ <PropertyGroup>+ <TargetFramework>net9.0</TargetFramework>+ <Nullable>enable</Nullable>+ <ImplicitUsings>enable</ImplicitUsings>+ <RootNamespace>Accounts.Api</RootNamespace>+ </PropertyGroup>++ <ItemGroup>+ <PackageReference Include="Dapper" Version="2.1.66" />+ <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="9.0.9" />+ <PackageReference Include="Npgsql" Version="9.0.3" />+ <PackageReference Include="Npgsql.OpenTelemetry" Version="9.0.3" />+ <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.13.1" />+ <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.13.1" />+ <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.13.0" />+ </ItemGroup>++</Project>diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Data/TransactionRepository.cs b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Data/TransactionRepository.csnew file mode 100644index 00000000..cfa39571--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Data/TransactionRepository.cs@@ -0,0 +1,53 @@+using Dapper;+using Npgsql;++namespace Accounts.Api.Data;++public sealed record TransactionRow(+ long Id, DateTime PostedAt, decimal Amount, string Currency, string Counterparty, string? Reference,+ string Category, decimal BalanceAfter, string? Metadata);++public sealed class TransactionRepository(NpgsqlDataSource db)+{+ private const string Columns =+ "t.id, t.posted_at, t.amount, t.currency, t.counterparty, t.reference, t.category, t.balance_after, t.metadata";++ public async Task<List<TransactionRow>> ForPeriodAsync(long accountId, DateTime from, DateTime to, CancellationToken ct)+ {+ await using var conn = await db.OpenConnectionAsync(ct);+ var rows = await conn.QueryAsync<TransactionRow>(new CommandDefinition(+ $"SELECT {Columns} FROM transactions t WHERE t.account_id = @AccountId AND t.posted_at >= @From AND t.posted_at < @To ORDER BY t.posted_at",+ new { AccountId = accountId, From = from, To = to }, cancellationToken: ct));+ return rows.AsList();+ }++ public async Task<List<TransactionRow>> PageAsync(long accountId, int pageSize, int offset, CancellationToken ct)+ {+ await using var conn = await db.OpenConnectionAsync(ct);+ var rows = await conn.QueryAsync<TransactionRow>(new CommandDefinition(+ $"SELECT {Columns} FROM transactions t WHERE t.account_id = @AccountId ORDER BY t.posted_at DESC LIMIT @PageSize OFFSET @Offset",+ new { AccountId = accountId, PageSize = pageSize, Offset = offset }, cancellationToken: ct));+ return rows.AsList();+ }++ public async Task RecordExportAsync(long accountId, string? userId, int rowCount, int byteCount, CancellationToken ct)+ {+ await using var conn = await db.OpenConnectionAsync(ct);+ await conn.ExecuteAsync(new CommandDefinition(+ "INSERT INTO export_audit (account_id, requested_by, row_count, byte_count, created_at) VALUES (@AccountId, @UserId, @RowCount, @ByteCount, now())",+ new { AccountId = accountId, UserId = userId, RowCount = rowCount, ByteCount = byteCount }, cancellationToken: ct));+ }+}++public sealed record AccountRow(long Id, string Number, string Currency, long OwnerId, decimal Balance, DateTime UpdatedAt);++public sealed class AccountRepository(NpgsqlDataSource db)+{+ public async Task<AccountRow?> FindAsync(long accountId, CancellationToken ct)+ {+ await using var conn = await db.OpenConnectionAsync(ct);+ return await conn.QuerySingleOrDefaultAsync<AccountRow>(new CommandDefinition(+ "SELECT id, number, currency, owner_id, balance, updated_at FROM accounts WHERE id = @AccountId",+ new { AccountId = accountId }, cancellationToken: ct));+ }+}diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Exports/TransactionExportEndpoints.cs b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Exports/TransactionExportEndpoints.csnew file mode 100644index 00000000..c89328ad--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Exports/TransactionExportEndpoints.cs@@ -0,0 +1,78 @@+using System.Security.Claims;+using System.Text;+using System.Text.Json;+using Accounts.Api.Data;+using Accounts.Api.Serialization;++namespace Accounts.Api.Exports;++public static class TransactionExportEndpoints+{+ public static void MapTransactionExportEndpoints(this IEndpointRouteBuilder app)+ {+ app.MapGet("/api/accounts/{accountId}/transactions/export", ExportAsync).RequireAuthorization();+ }++ // Finance downloads a period's transactions for reconciliation; the+ // dashboard saves the body as <account>-<from>-<to>.json.+ private static async Task<IResult> ExportAsync(+ long accountId, DateTime from, DateTime to, ClaimsPrincipal user,+ AccountRepository accounts, TransactionRepository transactions, ILogger<TransactionExportEndpointsLog> logger,+ CancellationToken ct)+ {+ var account = await accounts.FindAsync(accountId, ct);+ if (account is null)+ {+ return Results.NotFound();+ }+ if (to <= from)+ {+ return Results.UnprocessableEntity(new { error = "to must be after from" });+ }++ var rows = await transactions.ForPeriodAsync(accountId, from, to, ct);++ var body = new StringBuilder();+ body.Append("{\"account\":").Append(JsonSerializer.Serialize(account.Number))+ .Append(",\"currency\":").Append(JsonSerializer.Serialize(account.Currency))+ .Append(",\"transactions\":[");+ for (var i = 0; i < rows.Count; i++)+ {+ if (i > 0) body.Append(',');+ body.Append(SerializeRow(rows[i], account.Currency));+ }+ body.Append("]}");+ var json = body.ToString();+ var bytes = Encoding.UTF8.GetByteCount(json);++ await transactions.RecordExportAsync(accountId, user.FindFirstValue(ClaimTypes.NameIdentifier), rows.Count, bytes, ct);+ logger.LogInformation("Exported {RowCount} transactions ({ByteCount} bytes) for account {AccountId}", rows.Count, bytes, accountId);++ return Results.Text(json, "application/json", Encoding.UTF8);+ }++ private static string SerializeRow(TransactionRow row, string currency)+ {+ var options = new JsonSerializerOptions(JsonSerializerDefaults.Web)+ {+ WriteIndented = true,+ Converters = { new MoneyConverter(currency), new UtcDateTimeConverter() },+ };+ var export = new TransactionExport(+ row.Id,+ row.PostedAt,+ new Money(row.Amount, row.Currency),+ row.Counterparty,+ row.Reference,+ row.Category,+ new Money(row.BalanceAfter, row.Currency),+ JsonDocument.Parse(row.Metadata ?? "{}").RootElement.Clone());+ return JsonSerializer.Serialize(export, options);+ }+}++public sealed class TransactionExportEndpointsLog;++public sealed record TransactionExport(+ long Id, DateTime PostedAt, Money Amount, string Counterparty, string? Reference,+ string Category, Money BalanceAfter, JsonElement Metadata);diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Program.cs b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Program.csnew file mode 100644index 00000000..a64a4110--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Program.cs@@ -0,0 +1,39 @@+using Accounts.Api.Accounts;+using Accounts.Api.Data;+using Accounts.Api.Exports;+using Dapper;+using Npgsql;+using OpenTelemetry.Logs;+using OpenTelemetry.Resources;+using OpenTelemetry.Trace;++var builder = WebApplication.CreateBuilder(args);++DefaultTypeMap.MatchNamesWithUnderscores = true;+builder.Services.AddSingleton(_ => NpgsqlDataSource.Create(builder.Configuration.GetConnectionString("Ledger")!));+builder.Services.AddScoped<TransactionRepository>();+builder.Services.AddScoped<AccountRepository>();++builder.Services.AddOpenTelemetry()+ .ConfigureResource(r => r.AddService("accounts-api"))+ .WithTracing(t => t+ .AddAspNetCoreInstrumentation()+ .AddNpgsql()+ .AddOtlpExporter());+builder.Logging.AddOpenTelemetry(o =>+{+ o.IncludeFormattedMessage = true;+ o.AddOtlpExporter();+});++builder.Services.AddAuthentication().AddJwtBearer();+builder.Services.AddAuthorization();++var app = builder.Build();+app.UseAuthentication();+app.UseAuthorization();++app.MapAccountEndpoints();+app.MapTransactionExportEndpoints();++app.Run();diff --git a/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Serialization/MoneyConverter.cs b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Serialization/MoneyConverter.csnew file mode 100644index 00000000..5e0a8c82--- /dev/null+++ b/cli/test/evals/testdata/repos/accounts-api-dotnet/src/AccountsApi/Serialization/MoneyConverter.cs@@ -0,0 +1,34 @@+using System.Globalization;+using System.Text.Json;+using System.Text.Json.Serialization;++namespace Accounts.Api.Serialization;++public readonly record struct Money(decimal Amount, string Currency);++// Writes money as {"amount":"1234.50","currency":"EUR","display":"€1,234.50"}.+public sealed class MoneyConverter(string displayCurrency) : JsonConverter<Money>+{+ public override Money Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) =>+ throw new NotSupportedException("exports are write-only");++ public override void Write(Utf8JsonWriter writer, Money value, JsonSerializerOptions options)+ {+ var culture = CultureInfo.GetCultures(CultureTypes.SpecificCultures)+ .FirstOrDefault(c => new RegionInfo(c.Name).ISOCurrencySymbol == displayCurrency) ?? CultureInfo.InvariantCulture;+ writer.WriteStartObject();+ writer.WriteString("amount", value.Amount.ToString("F2", CultureInfo.InvariantCulture));+ writer.WriteString("currency", value.Currency);+ writer.WriteString("display", value.Amount.ToString("C", culture));+ writer.WriteEndObject();+ }+}++public sealed class UtcDateTimeConverter : JsonConverter<DateTime>+{+ public override DateTime Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) =>+ reader.GetDateTime().ToUniversalTime();++ public override void Write(Utf8JsonWriter writer, DateTime value, JsonSerializerOptions options) =>+ writer.WriteStringValue(value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture));+}diff --git a/cli/test/evals/testdata/repos/admin-api-java/README.md b/cli/test/evals/testdata/repos/admin-api-java/README.mdnew file mode 100644index 00000000..ec20b874--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/README.md@@ -0,0 +1,15 @@+# admin-api++The Ledgerly admin console backend: members, settings, invoices and the audit+log. Java 21, Spring Boot 3.5, JDBC (`JdbcTemplate`) on PostgreSQL, Flyway+migrations. Instrumented with the OpenTelemetry Java agent+(`-javaagent:opentelemetry-javaagent.jar`, `OTEL_SERVICE_NAME=admin-api`).++```+./mvnw spring-boot:run+```++Audit log routes:++- `GET /api/admin/audit-log` – the Audit log page: the organization's events, newest first+- `GET /api/admin/dashboard/activity` – the dashboard widget: the last 20 eventsdiff --git a/cli/test/evals/testdata/repos/admin-api-java/history.yaml b/cli/test/evals/testdata/repos/admin-api-java/history.yamlnew file mode 100644index 00000000..95cb5991--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/history.yaml@@ -0,0 +1,4 @@+commits: []+head:+ message: 'audit log: admin page and dashboard widget'+ age: 4300hdiff --git a/cli/test/evals/testdata/repos/admin-api-java/pom.xml b/cli/test/evals/testdata/repos/admin-api-java/pom.xmlnew file mode 100644index 00000000..49ecd5fd--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/pom.xml@@ -0,0 +1,40 @@+<?xml version="1.0" encoding="UTF-8"?>+<project xmlns="http://maven.apache.org/POM/4.0.0"+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">+ <modelVersion>4.0.0</modelVersion>+ <parent>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-parent</artifactId>+ <version>3.5.6</version>+ </parent>+ <groupId>com.ledgerly</groupId>+ <artifactId>admin-api</artifactId>+ <version>3.18.1</version>+ <properties>+ <java.version>21</java.version>+ </properties>+ <dependencies>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-web</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-jdbc</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-security</artifactId>+ </dependency>+ <dependency>+ <groupId>org.flywaydb</groupId>+ <artifactId>flyway-database-postgresql</artifactId>+ </dependency>+ <dependency>+ <groupId>org.postgresql</groupId>+ <artifactId>postgresql</artifactId>+ <scope>runtime</scope>+ </dependency>+ </dependencies>+</project>diff --git a/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditEvent.java b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditEvent.javanew file mode 100644index 00000000..b09072e3--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditEvent.java@@ -0,0 +1,14 @@+package com.ledgerly.admin.audit;++import java.time.OffsetDateTime;++public record AuditEvent(+ long id,+ long actorId,+ String action,+ String targetType,+ String targetId,+ String ipAddress,+ String userAgent,+ OffsetDateTime createdAt) {+}diff --git a/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditEventRepository.java b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditEventRepository.javanew file mode 100644index 00000000..1b7a9007--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditEventRepository.java@@ -0,0 +1,58 @@+package com.ledgerly.admin.audit;++import java.time.OffsetDateTime;+import java.util.List;++import org.springframework.jdbc.core.JdbcTemplate;+import org.springframework.jdbc.core.RowMapper;+import org.springframework.stereotype.Repository;++@Repository+public class AuditEventRepository {++ private static final RowMapper<AuditEvent> FULL = (rs, i) -> new AuditEvent(+ rs.getLong("id"),+ rs.getLong("actor_id"),+ rs.getString("action"),+ rs.getString("target_type"),+ rs.getString("target_id"),+ rs.getString("ip_address"),+ rs.getString("user_agent"),+ rs.getObject("created_at", OffsetDateTime.class));++ private static final RowMapper<AuditEvent> SUMMARY = (rs, i) -> new AuditEvent(+ rs.getLong("id"),+ rs.getLong("actor_id"),+ rs.getString("action"),+ null, null, null, null,+ rs.getObject("created_at", OffsetDateTime.class));++ private final JdbcTemplate jdbc;++ public AuditEventRepository(JdbcTemplate jdbc) {+ this.jdbc = jdbc;+ }++ /** Every event of the organization, newest first (the Audit log page). */+ public List<AuditEvent> findByOrg(long orgId) {+ return jdbc.query(+ "SELECT e.id, e.actor_id, e.action, e.target_type, e.target_id, e.ip_address, e.user_agent, e.created_at "+ + "FROM audit_events e WHERE e.org_id = ? ORDER BY e.created_at DESC",+ FULL, orgId);+ }++ /** The latest events for the dashboard widget. */+ public List<AuditEvent> findRecent(long orgId) {+ return jdbc.query(+ "SELECT e.id, e.actor_id, e.action, e.created_at "+ + "FROM audit_events e WHERE e.org_id = ? ORDER BY e.created_at DESC LIMIT 20",+ SUMMARY, orgId);+ }++ public void record(long orgId, long actorId, String action, String targetType, String targetId, String ip, String userAgent) {+ jdbc.update(+ "INSERT INTO audit_events (org_id, actor_id, action, target_type, target_id, ip_address, user_agent, created_at) "+ + "VALUES (?, ?, ?, ?, ?, ?, ?, now())",+ orgId, actorId, action, targetType, targetId, ip, userAgent);+ }+}diff --git a/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditLogController.java b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditLogController.javanew file mode 100644index 00000000..bc0b4d8d--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/audit/AuditLogController.java@@ -0,0 +1,36 @@+package com.ledgerly.admin.audit;++import java.util.List;++import org.springframework.web.bind.annotation.GetMapping;+import org.springframework.web.bind.annotation.RequestMapping;+import org.springframework.web.bind.annotation.RestController;++import com.ledgerly.admin.security.AdminContext;++@RestController+@RequestMapping("/api/admin")+public class AuditLogController {++ private final AuditEventRepository events;+ private final AdminContext admin;++ public AuditLogController(AuditEventRepository events, AdminContext admin) {+ this.events = events;+ this.admin = admin;+ }++ /** The Audit log page. The UI renders the list and filters it client side. */+ @GetMapping("/audit-log")+ public List<AuditEvent> list() {+ long orgId = admin.requireAdmin();+ return events.findByOrg(orgId);+ }++ /** The dashboard widget: the last 20 events. */+ @GetMapping("/dashboard/activity")+ public List<AuditEvent> activity() {+ long orgId = admin.requireAdmin();+ return events.findRecent(orgId);+ }+}diff --git a/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/security/AdminContext.java b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/security/AdminContext.javanew file mode 100644index 00000000..d81257b5--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/src/main/java/com/ledgerly/admin/security/AdminContext.java@@ -0,0 +1,35 @@+package com.ledgerly.admin.security;++import org.springframework.jdbc.core.JdbcTemplate;+import org.springframework.security.access.AccessDeniedException;+import org.springframework.security.core.context.SecurityContextHolder;+import org.springframework.stereotype.Component;++@Component+public class AdminContext {++ private final JdbcTemplate jdbc;++ public AdminContext(JdbcTemplate jdbc) {+ this.jdbc = jdbc;+ }++ /** Returns the signed-in user's organization, or throws unless they are an owner or admin of it. */+ public long requireAdmin() {+ var auth = SecurityContextHolder.getContext().getAuthentication();+ if (auth == null || !(auth.getPrincipal() instanceof OrgPrincipal p)) {+ throw new AccessDeniedException("admin role required");+ }+ String role = jdbc.query(+ "SELECT m.role FROM org_members m WHERE m.org_id = ? AND m.user_id = ?",+ rs -> rs.next() ? rs.getString(1) : null,+ p.orgId(), p.userId());+ if (!"owner".equals(role) && !"admin".equals(role)) {+ throw new AccessDeniedException("admin role required");+ }+ return p.orgId();+ }++ public record OrgPrincipal(long userId, long orgId) {+ }+}diff --git a/cli/test/evals/testdata/repos/admin-api-java/src/main/resources/application.yml b/cli/test/evals/testdata/repos/admin-api-java/src/main/resources/application.ymlnew file mode 100644index 00000000..610ec422--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/src/main/resources/application.yml@@ -0,0 +1,11 @@+spring:+ application:+ name: admin-api+ datasource:+ url: ${DATABASE_URL:jdbc:postgresql://localhost:5432/ledgerly}+ hikari:+ maximum-pool-size: 20+ flyway:+ locations: classpath:db/migration+server:+ port: 8080diff --git a/cli/test/evals/testdata/repos/admin-api-java/src/main/resources/db/migration/V14__audit_events.sql b/cli/test/evals/testdata/repos/admin-api-java/src/main/resources/db/migration/V14__audit_events.sqlnew file mode 100644index 00000000..4bf69092--- /dev/null+++ b/cli/test/evals/testdata/repos/admin-api-java/src/main/resources/db/migration/V14__audit_events.sql@@ -0,0 +1,14 @@+CREATE TABLE audit_events (+ id BIGSERIAL PRIMARY KEY,+ org_id BIGINT NOT NULL REFERENCES organizations (id),+ actor_id BIGINT NOT NULL REFERENCES users (id),+ action VARCHAR(64) NOT NULL,+ target_type VARCHAR(64) NOT NULL,+ target_id VARCHAR(64),+ ip_address INET,+ user_agent TEXT,+ created_at TIMESTAMPTZ NOT NULL DEFAULT now()+);++-- Every read filters by organization and lists newest first.+CREATE INDEX audit_events_org_created_idx ON audit_events (org_id, created_at DESC);diff --git a/cli/test/evals/testdata/repos/bookings-api-node/README.md b/cli/test/evals/testdata/repos/bookings-api-node/README.mdnew file mode 100644index 00000000..9294bad5--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/README.md@@ -0,0 +1,15 @@+# bookings-api++Availability search, bookings and reviews for the rentals site. Node 24,+Express 5, PostgreSQL 16 (`pg`). Traces go to Traceway through the+OpenTelemetry Node SDK (`src/tracing.js`). The database runs on its own VM+(`pg-bookings-1`, data on `/var/lib/postgresql`, an NVMe volume); the OTel+Collector there ships its host metrics to the same Traceway project as+`bookings-db`.++Batch jobs live in `jobs/` and run as Kubernetes Jobs (`deploy/k8s/`).++```+npm test+npm start+```diff --git a/cli/test/evals/testdata/repos/bookings-api-node/deploy/k8s/api.yaml b/cli/test/evals/testdata/repos/bookings-api-node/deploy/k8s/api.yamlnew file mode 100644index 00000000..cc9b684e--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/deploy/k8s/api.yaml@@ -0,0 +1,22 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: bookings-api+spec:+ replicas: 3+ selector:+ matchLabels: {app: bookings-api}+ template:+ metadata:+ labels: {app: bookings-api}+ spec:+ containers:+ - name: bookings-api+ image: registry.example.com/bookings-api:6.4.1+ env:+ - {name: OTEL_SERVICE_NAME, value: bookings-api}+ - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector.observability:4318"}+ envFrom: [{secretRef: {name: bookings-api}}]+ resources:+ requests: {cpu: 500m, memory: 512Mi}+ limits: {cpu: "1", memory: 1Gi}diff --git a/cli/test/evals/testdata/repos/bookings-api-node/deploy/k8s/rebuild-search-documents.yaml b/cli/test/evals/testdata/repos/bookings-api-node/deploy/k8s/rebuild-search-documents.yamlnew file mode 100644index 00000000..96a8221a--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/deploy/k8s/rebuild-search-documents.yaml@@ -0,0 +1,14 @@+apiVersion: batch/v1+kind: Job+metadata:+ name: rebuild-search-documents+spec:+ backoffLimit: 0+ template:+ spec:+ restartPolicy: Never+ containers:+ - name: rebuild+ image: registry.example.com/bookings-api:6.4.1+ command: ["node", "jobs/rebuild-search-documents.js"]+ envFrom: [{secretRef: {name: bookings-api}}]diff --git a/cli/test/evals/testdata/repos/bookings-api-node/history.yaml b/cli/test/evals/testdata/repos/bookings-api-node/history.yamlnew file mode 100644index 00000000..c0787368--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/history.yaml@@ -0,0 +1,11 @@+commits:+- absent:+ - deploy/k8s/rebuild-search-documents.yaml+ - jobs/rebuild-search-documents.js+ message: 'bookings: lock availability rows while booking'+ age: 260h+ author: Aiko Tanaka <[email protected]>+head:+ message: 'jobs: rebuild search_documents for the new ranking'+ age: 9h52m+ author: Sven Lindqvist <[email protected]>diff --git a/cli/test/evals/testdata/repos/bookings-api-node/jobs/rebuild-search-documents.js b/cli/test/evals/testdata/repos/bookings-api-node/jobs/rebuild-search-documents.jsnew file mode 100644index 00000000..7744055e--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/jobs/rebuild-search-documents.js@@ -0,0 +1,35 @@+// Rebuilds search_documents (one row per listing and night, with the+// denormalized fields the new ranking needs) from listings, availability and+// reviews. Runs once as a Kubernetes Job; see deploy/k8s/rebuild-search-documents.yaml.+import pg from 'pg';++const BATCH = 50_000;+const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL, max: 4 });++async function main() {+ await pool.query('CREATE TABLE IF NOT EXISTS search_documents_next (LIKE search_documents INCLUDING ALL)');+ const { rows: [{ max }] } = await pool.query('SELECT max(id) AS max FROM listings');+ for (let from = 0; from <= max; from += BATCH) {+ await pool.query(`+ INSERT INTO search_documents_next (listing_id, day, city, title, price_cents, units_free, rating, review_count, tsv)+ SELECT l.id, a.day, l.city, l.title, a.price_cents, a.units_free,+ coalesce(avg(r.rating), 0), count(r.id),+ to_tsvector('simple', l.title || ' ' || l.city || ' ' || coalesce(string_agg(r.body, ' '), ''))+ FROM listings l+ JOIN availability a ON a.listing_id = l.id+ LEFT JOIN reviews r ON r.listing_id = l.id+ WHERE l.id > $1 AND l.id <= $2+ GROUP BY l.id, a.day, l.city, l.title, a.price_cents, a.units_free`, [from, from + BATCH]);+ console.log(JSON.stringify({ msg: 'batch done', from, to: from + BATCH }));+ }+ await pool.query('BEGIN');+ await pool.query('ALTER TABLE search_documents RENAME TO search_documents_old');+ await pool.query('ALTER TABLE search_documents_next RENAME TO search_documents');+ await pool.query('COMMIT');+ await pool.end();+}++main().catch((err) => {+ console.error(err);+ process.exit(1);+});diff --git a/cli/test/evals/testdata/repos/bookings-api-node/package.json b/cli/test/evals/testdata/repos/bookings-api-node/package.jsonnew file mode 100644index 00000000..1148cfc7--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/package.json@@ -0,0 +1,16 @@+{+ "name": "bookings-api",+ "version": "6.4.1",+ "private": true,+ "type": "module",+ "scripts": {+ "start": "node --import ./src/tracing.js src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "^0.64.1",+ "@opentelemetry/sdk-node": "^0.205.0",+ "express": "^5.1.0",+ "pg": "^8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/bookings-api-node/src/dates.js b/cli/test/evals/testdata/repos/bookings-api-node/src/dates.jsnew file mode 100644index 00000000..a162e7dd--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/src/dates.js@@ -0,0 +1,8 @@+const DAY = 24 * 60 * 60 * 1000;++export function nightsBetween(checkIn, checkOut) {+ const a = Date.parse(checkIn);+ const b = Date.parse(checkOut);+ if (Number.isNaN(a) || Number.isNaN(b)) return 0;+ return Math.round((b - a) / DAY);+}diff --git a/cli/test/evals/testdata/repos/bookings-api-node/src/dates.test.js b/cli/test/evals/testdata/repos/bookings-api-node/src/dates.test.jsnew file mode 100644index 00000000..d2c316e0--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/src/dates.test.js@@ -0,0 +1,9 @@+import { test } from 'node:test';+import assert from 'node:assert/strict';+import { nightsBetween } from './dates.js';++test('nights between two dates', () => {+ assert.equal(nightsBetween('2026-11-02', '2026-11-05'), 3);+ assert.equal(nightsBetween('2026-11-05', '2026-11-02'), -3);+ assert.equal(nightsBetween('soon', '2026-11-02'), 0);+});diff --git a/cli/test/evals/testdata/repos/bookings-api-node/src/db.js b/cli/test/evals/testdata/repos/bookings-api-node/src/db.jsnew file mode 100644index 00000000..a6175b28--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/src/db.js@@ -0,0 +1,3 @@+import pg from 'pg';++export const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL, max: 20, statement_timeout: 5000 });diff --git a/cli/test/evals/testdata/repos/bookings-api-node/src/server.js b/cli/test/evals/testdata/repos/bookings-api-node/src/server.jsnew file mode 100644index 00000000..10e6c04a--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/src/server.js@@ -0,0 +1,60 @@+import express from 'express';+import { pool } from './db.js';+import { nightsBetween } from './dates.js';++const app = express();+app.use(express.json());++// Indexes: availability (listing_id, day), bookings (id), listings (id),+// reviews (listing_id, created_at DESC). See migrations in the infra repo.++app.get('/api/availability', async (req, res) => {+ const ids = String(req.query.listings ?? '').split(',').filter(Boolean).map(Number);+ const { rows } = await pool.query(+ 'SELECT listing_id, day, units_free, price_cents FROM availability WHERE listing_id = ANY($1) AND day BETWEEN $2 AND $3 ORDER BY listing_id, day',+ [ids, req.query.from, req.query.to]);+ res.json(rows);+});++app.get('/api/bookings/:id', async (req, res) => {+ const { rows: [b] } = await pool.query(+ 'SELECT id, listing_id, guest_id, check_in, check_out, status, total_cents FROM bookings WHERE id = $1', [req.params.id]);+ if (!b) return res.sendStatus(404);+ const { rows: [listing] } = await pool.query('SELECT id, title, city, host_id FROM listings WHERE id = $1', [b.listing_id]);+ res.json({ ...b, listing });+});++app.post('/api/bookings', async (req, res) => {+ const { listingId, guestId, checkIn, checkOut, totalCents } = req.body ?? {};+ if (nightsBetween(checkIn, checkOut) < 1) return res.status(422).json({ error: 'checkOut must be after checkIn' });+ const client = await pool.connect();+ try {+ await client.query('BEGIN');+ const { rows } = await client.query(+ 'SELECT units_free FROM availability WHERE listing_id = $1 AND day >= $2 AND day < $3 FOR UPDATE', [listingId, checkIn, checkOut]);+ if (rows.length === 0 || rows.some((r) => r.units_free < 1)) {+ await client.query('ROLLBACK');+ return res.status(409).json({ error: 'not available' });+ }+ await client.query('UPDATE availability SET units_free = units_free - 1 WHERE listing_id = $1 AND day >= $2 AND day < $3', [listingId, checkIn, checkOut]);+ const id = crypto.randomUUID();+ await client.query(+ "INSERT INTO bookings (id, listing_id, guest_id, check_in, check_out, status, total_cents, created_at) VALUES ($1, $2, $3, $4, $5, 'confirmed', $6, now())",+ [id, listingId, guestId, checkIn, checkOut, totalCents]);+ await client.query('COMMIT');+ res.status(201).json({ id });+ } catch (err) {+ await client.query('ROLLBACK');+ throw err;+ } finally {+ client.release();+ }+});++app.get('/api/listings/:id/reviews', async (req, res) => {+ const { rows } = await pool.query(+ 'SELECT id, author_name, rating, body, created_at FROM reviews WHERE listing_id = $1 ORDER BY created_at DESC LIMIT 20', [req.params.id]);+ res.json(rows);+});++app.listen(Number(process.env.PORT ?? 8080));diff --git a/cli/test/evals/testdata/repos/bookings-api-node/src/tracing.js b/cli/test/evals/testdata/repos/bookings-api-node/src/tracing.jsnew file mode 100644index 00000000..abe917d9--- /dev/null+++ b/cli/test/evals/testdata/repos/bookings-api-node/src/tracing.js@@ -0,0 +1,5 @@+import { NodeSDK } from '@opentelemetry/sdk-node';+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';++const sdk = new NodeSDK({ instrumentations: [getNodeAutoInstrumentations()] });+sdk.start();diff --git a/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/cache_test.go b/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/cache_test.gonew file mode 100644index 00000000..67b0fc15--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/cache_test.go@@ -0,0 +1,48 @@+package catalog++import (+ "context"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync/atomic"+ "testing"+)++type countingStore struct{ listings atomic.Int32 }++func (s *countingStore) CategoryListing(_ context.Context, slug string) ([]Product, error) {+ s.listings.Add(1)+ return []Product{{ID: 1, Slug: slug + "-1", Name: "One", Available: 3}, {ID: 2, Slug: slug + "-2", Name: "Two", Available: 40}}, nil+}++func (s *countingStore) Product(context.Context, int64) (*Product, error) { return nil, nil }++func TestCacheLoadsOnceAndServesFromMemory(t *testing.T) {+ st := &countingStore{}+ h := NewHandlers(st, NewCache(st, 0))+ for range 3 {+ req := httptest.NewRequest(http.MethodGet, "/api/categories/shoes/products", nil)+ req.SetPathValue("slug", "shoes")+ rec := httptest.NewRecorder()+ h.Listing(rec, req)+ if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"shoes-1"`) {+ t.Fatalf("status %d body %s", rec.Code, rec.Body.String())+ }+ }+ if n := st.listings.Load(); n != 1 {+ t.Fatalf("listing query ran %d times, want 1", n)+ }+}++func TestPreloadSkipsLoadedCategories(t *testing.T) {+ st := &countingStore{}+ c := NewCache(st, 0)+ if _, err := c.Get(context.Background(), "bags"); err != nil {+ t.Fatal(err)+ }+ c.Preload(context.Background(), []string{"bags", "hats"}, 0)+ if n := st.listings.Load(); n != 2 {+ t.Fatalf("listing query ran %d times, want 2", n)+ }+}diff --git a/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/handlers.go b/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/handlers.gonew file mode 100644index 00000000..b0da1fc3--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/handlers.go@@ -0,0 +1,56 @@+// Package catalog serves category listings and product pages.+package catalog++import (+ "encoding/json"+ "net/http"+ "strconv"+)++// Handlers serves the catalog routes.+type Handlers struct {+ store Store+ cache *Cache+}++// NewHandlers returns the catalog handlers.+func NewHandlers(s Store, c *Cache) *Handlers { return &Handlers{store: s, cache: c} }++const pageSize = 48++// Listing serves GET /api/categories/{slug}/products?page=N.+func (h *Handlers) Listing(w http.ResponseWriter, r *http.Request) {+ all, err := h.cache.Get(r.Context(), r.PathValue("slug"))+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ page, _ := strconv.Atoi(r.URL.Query().Get("page"))+ if page < 1 {+ page = 1+ }+ start := min((page-1)*pageSize, len(all))+ end := min(start+pageSize, len(all))+ out := all[start:end]+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(map[string]any{"page": page, "total": len(all), "products": out})+}++// Product serves GET /api/products/{id}.+func (h *Handlers) Product(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad product id", http.StatusBadRequest)+ return+ }+ p, err := h.store.Product(r.Context(), id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if p == nil {+ http.NotFound(w, r)+ return+ }+ _ = json.NewEncoder(w).Encode(p)+}diff --git a/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/store.go b/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/store.gonew file mode 100644index 00000000..436037d3--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/.history/01/internal/catalog/store.go@@ -0,0 +1,48 @@+package catalog++import (+ "context"+ "errors"+)++// Product is one listing entry.+type Product struct {+ ID int64 `json:"id"`+ Slug string `json:"slug"`+ Name string `json:"name"`+ Brand string `json:"brand"`+ ThumbnailURL string `json:"thumbnailUrl"`+ AmountCents int64 `json:"amountCents"`+ Currency string `json:"currency"`+ Available int `json:"available"`+}++// Store reads the catalog.+type Store interface {+ // CategoryListing runs the listing query: every active product of the+ // category with its price and stock, in rank order (up to 500).+ CategoryListing(ctx context.Context, slug string) ([]Product, error)+ Product(ctx context.Context, id int64) (*Product, error)+}++// PgStore implements Store on pgx (wired in production builds).+type PgStore struct{ pool any }++// NewStore returns the PostgreSQL store.+func NewStore(pool any) *PgStore { return &PgStore{pool: pool} }++var errNoPool = errors.New("catalog: no database pool configured")++// CategoryListing runs:+//+// SELECT p.id, p.slug, p.name, p.brand, p.thumbnail_url, pp.amount_cents, pp.currency, i.available+// FROM products p JOIN product_prices pp ON pp.product_id = p.id AND pp.price_list = $2+// JOIN inventory i ON i.product_id = p.id+// WHERE p.category_id = (SELECT id FROM categories WHERE slug = $1) AND p.active+// ORDER BY p.rank LIMIT 500+func (s *PgStore) CategoryListing(ctx context.Context, slug string) ([]Product, error) {+ return nil, errNoPool+}++// Product loads one product with its price and stock.+func (s *PgStore) Product(ctx context.Context, id int64) (*Product, error) { return nil, errNoPool }diff --git a/cli/test/evals/testdata/repos/catalog-api-go/README.md b/cli/test/evals/testdata/repos/catalog-api-go/README.mdnew file mode 100644index 00000000..e5b9e7d2--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/README.md@@ -0,0 +1,18 @@+# catalog-api++Product catalog for the storefront: category listings and product pages.+Go, `net/http`, PostgreSQL through pgx. Runs as three instances (catalog-api-a/b/c)+behind the load balancer; a release restarts all three. Traces, logs and+metrics go to Traceway over OTLP (`OTEL_EXPORTER_OTLP_ENDPOINT`).++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/categories/{slug}/products` – the category listing, served from an+ in-process cache that is preloaded at startup and refreshed every 10 minutes+- `GET /api/products/{id}` – one product+- `GET /healthz` – the load balancer's health checkdiff --git a/cli/test/evals/testdata/repos/catalog-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/catalog-api-go/cmd/server/main.gonew file mode 100644index 00000000..4c641446--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/cmd/server/main.go@@ -0,0 +1,45 @@+package main++import (+ "context"+ "log/slog"+ "net/http"+ "os"+ "time"++ "example.com/catalog-api/internal/catalog"+)++func main() {+ ctx := context.Background()+ // pgxpool.New(ctx, os.Getenv("DATABASE_URL")) in production builds:+ // pool_max_conns=10, pool_min_conns=0, connections are opened on demand.+ store := catalog.NewStore(nil)+ cache := catalog.NewCache(store, 10*time.Minute)++ // Fill the cache in the background so the instance can take traffic+ // straight away; categories not loaded yet are fetched on demand. One+ // category every 25s, so the three instances a release restarts together+ // do not all load the whole catalog at the same moment.+ go cache.Preload(ctx, catalog.Categories, 25*time.Second)+ go cache.RefreshLoop(ctx, catalog.Categories)++ h := catalog.NewHandlers(store, cache)+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/categories/{slug}/products", h.Listing)+ mux.HandleFunc("GET /api/products/{id}", h.Product)+ mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, mux); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/catalog-api-go/go.mod b/cli/test/evals/testdata/repos/catalog-api-go/go.modnew file mode 100644index 00000000..93931dfe--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/catalog-api++go 1.26diff --git a/cli/test/evals/testdata/repos/catalog-api-go/history.yaml b/cli/test/evals/testdata/repos/catalog-api-go/history.yamlnew file mode 100644index 00000000..792b0991--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: 'v5.0.4: brand on product pages'+ age: 30h12m+ author: Mira Holm <[email protected]>+head:+ message: 'v5.1.0: low-stock badge in category listings'+ age: 58m+ author: Jonas Becker <[email protected]>diff --git a/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/cache.go b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/cache.gonew file mode 100644index 00000000..35550e24--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/cache.go@@ -0,0 +1,96 @@+package catalog++import (+ "context"+ "log/slog"+ "sync"+ "time"+)++// Categories are the storefront's category slugs, most visited first.+var Categories = []string{+ "shoes", "jackets", "t-shirts", "jeans", "sneakers", "dresses", "hoodies", "bags", "accessories", "socks",+ "shirts", "shorts", "skirts", "sweaters", "boots", "hats", "scarves", "belts", "swimwear", "sportswear",+ "underwear", "sunglasses", "watches", "gloves",+}++// Cache keeps each category's full listing in memory. Each instance has+// its own; a restart starts it empty.+type Cache struct {+ store Store+ every time.Duration++ mu sync.RWMutex+ items map[string][]Product+}++// NewCache returns an empty cache refreshed every `every`.+func NewCache(s Store, every time.Duration) *Cache {+ return &Cache{store: s, every: every, items: map[string][]Product{}}+}++// Get returns a category's listing, loading it on a miss.+func (c *Cache) Get(ctx context.Context, slug string) ([]Product, error) {+ c.mu.RLock()+ ps, ok := c.items[slug]+ c.mu.RUnlock()+ if ok {+ return ps, nil+ }+ ps, err := c.store.CategoryListing(ctx, slug)+ if err != nil {+ return nil, err+ }+ c.put(slug, ps)+ return ps, nil+}++func (c *Cache) put(slug string, ps []Product) {+ c.mu.Lock()+ c.items[slug] = ps+ c.mu.Unlock()+}++// Preload loads every category once, one after the other, waiting pause+// before each one.+func (c *Cache) Preload(ctx context.Context, slugs []string, pause time.Duration) {+ for _, slug := range slugs {+ if pause > 0 {+ select {+ case <-ctx.Done():+ return+ case <-time.After(pause):+ }+ }+ c.mu.RLock()+ _, ok := c.items[slug]+ c.mu.RUnlock()+ if ok {+ continue+ }+ ps, err := c.store.CategoryListing(ctx, slug)+ if err != nil {+ slog.Warn("preload category", "slug", slug, "err", err)+ continue+ }+ c.put(slug, ps)+ }+}++// RefreshLoop reloads every category on a timer, off the request path.+func (c *Cache) RefreshLoop(ctx context.Context, slugs []string) {+ t := time.NewTicker(c.every)+ defer t.Stop()+ for {+ select {+ case <-ctx.Done():+ return+ case <-t.C:+ for _, slug := range slugs {+ if ps, err := c.store.CategoryListing(ctx, slug); err == nil {+ c.put(slug, ps)+ }+ }+ }+ }+}diff --git a/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/cache_test.go b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/cache_test.gonew file mode 100644index 00000000..19eac02d--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/cache_test.go@@ -0,0 +1,48 @@+package catalog++import (+ "context"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync/atomic"+ "testing"+)++type countingStore struct{ listings atomic.Int32 }++func (s *countingStore) CategoryListing(_ context.Context, slug string) ([]Product, error) {+ s.listings.Add(1)+ return []Product{{ID: 1, Slug: slug + "-1", Name: "One", Available: 3}, {ID: 2, Slug: slug + "-2", Name: "Two", Available: 40}}, nil+}++func (s *countingStore) Product(context.Context, int64) (*Product, error) { return nil, nil }++func TestCacheLoadsOnceAndServesFromMemory(t *testing.T) {+ st := &countingStore{}+ h := NewHandlers(st, NewCache(st, 0))+ for range 3 {+ req := httptest.NewRequest(http.MethodGet, "/api/categories/shoes/products", nil)+ req.SetPathValue("slug", "shoes")+ rec := httptest.NewRecorder()+ h.Listing(rec, req)+ if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"badge":"low-stock"`) {+ t.Fatalf("status %d body %s", rec.Code, rec.Body.String())+ }+ }+ if n := st.listings.Load(); n != 1 {+ t.Fatalf("listing query ran %d times, want 1", n)+ }+}++func TestPreloadSkipsLoadedCategories(t *testing.T) {+ st := &countingStore{}+ c := NewCache(st, 0)+ if _, err := c.Get(context.Background(), "bags"); err != nil {+ t.Fatal(err)+ }+ c.Preload(context.Background(), []string{"bags", "hats"}, 0)+ if n := st.listings.Load(); n != 2 {+ t.Fatalf("listing query ran %d times, want 2", n)+ }+}diff --git a/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/handlers.go b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/handlers.gonew file mode 100644index 00000000..5f515081--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/handlers.go@@ -0,0 +1,62 @@+// Package catalog serves category listings and product pages.+package catalog++import (+ "encoding/json"+ "net/http"+ "strconv"+)++// Handlers serves the catalog routes.+type Handlers struct {+ store Store+ cache *Cache+}++// NewHandlers returns the catalog handlers.+func NewHandlers(s Store, c *Cache) *Handlers { return &Handlers{store: s, cache: c} }++const pageSize = 48++// Listing serves GET /api/categories/{slug}/products?page=N.+func (h *Handlers) Listing(w http.ResponseWriter, r *http.Request) {+ all, err := h.cache.Get(r.Context(), r.PathValue("slug"))+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ page, _ := strconv.Atoi(r.URL.Query().Get("page"))+ if page < 1 {+ page = 1+ }+ start := min((page-1)*pageSize, len(all))+ end := min(start+pageSize, len(all))+ out := make([]Product, 0, end-start)+ for _, p := range all[start:end] {+ if p.Available > 0 && p.Available < 5 {+ p.Badge = "low-stock"+ }+ out = append(out, p)+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(map[string]any{"page": page, "total": len(all), "products": out})+}++// Product serves GET /api/products/{id}.+func (h *Handlers) Product(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad product id", http.StatusBadRequest)+ return+ }+ p, err := h.store.Product(r.Context(), id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if p == nil {+ http.NotFound(w, r)+ return+ }+ _ = json.NewEncoder(w).Encode(p)+}diff --git a/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/store.go b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/store.gonew file mode 100644index 00000000..cec4edbc--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-go/internal/catalog/store.go@@ -0,0 +1,49 @@+package catalog++import (+ "context"+ "errors"+)++// Product is one listing entry.+type Product struct {+ ID int64 `json:"id"`+ Slug string `json:"slug"`+ Name string `json:"name"`+ Brand string `json:"brand"`+ ThumbnailURL string `json:"thumbnailUrl"`+ AmountCents int64 `json:"amountCents"`+ Currency string `json:"currency"`+ Available int `json:"available"`+ Badge string `json:"badge,omitempty"`+}++// Store reads the catalog.+type Store interface {+ // CategoryListing runs the listing query: every active product of the+ // category with its price and stock, in rank order (up to 500).+ CategoryListing(ctx context.Context, slug string) ([]Product, error)+ Product(ctx context.Context, id int64) (*Product, error)+}++// PgStore implements Store on pgx (wired in production builds).+type PgStore struct{ pool any }++// NewStore returns the PostgreSQL store.+func NewStore(pool any) *PgStore { return &PgStore{pool: pool} }++var errNoPool = errors.New("catalog: no database pool configured")++// CategoryListing runs:+//+// SELECT p.id, p.slug, p.name, p.brand, p.thumbnail_url, pp.amount_cents, pp.currency, i.available+// FROM products p JOIN product_prices pp ON pp.product_id = p.id AND pp.price_list = $2+// JOIN inventory i ON i.product_id = p.id+// WHERE p.category_id = (SELECT id FROM categories WHERE slug = $1) AND p.active+// ORDER BY p.rank LIMIT 500+func (s *PgStore) CategoryListing(ctx context.Context, slug string) ([]Product, error) {+ return nil, errNoPool+}++// Product loads one product with its price and stock.+func (s *PgStore) Product(ctx context.Context, id int64) (*Product, error) { return nil, errNoPool }diff --git a/cli/test/evals/testdata/repos/catalog-api-node/.history/01/package.json b/cli/test/evals/testdata/repos/catalog-api-node/.history/01/package.jsonnew file mode 100644index 00000000..2e5dcf1e--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/.history/01/package.json@@ -0,0 +1,15 @@+{+ "name": "catalog-api",+ "version": "4.7.2",+ "private": true,+ "main": "src/server.js",+ "scripts": {+ "start": "node --require @opentelemetry/auto-instrumentations-node/register src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "^0.64.1",+ "express": "^5.1.0",+ "pg": "^8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/catalog-api-node/.history/01/src/importer/supplier.js b/cli/test/evals/testdata/repos/catalog-api-node/.history/01/src/importer/supplier.jsnew file mode 100644index 00000000..fd80ce88--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/.history/01/src/importer/supplier.js@@ -0,0 +1,24 @@+'use strict';++const db = require('../db');++// Upserts one product from the supplier feed.+async function upsertFromFeed(entry) {+ await db.query(+ `UPDATE products+ SET name = $2,+ price_cents = $3,+ spec_sheet = $4,+ updated_at = now()+ WHERE slug = $1`,+ [entry.slug, entry.title, Math.round(entry.price * 100), entry.specs],+ );+}++async function importFeed(entries) {+ for (const entry of entries) {+ await upsertFromFeed(entry);+ }+}++module.exports = { importFeed, upsertFromFeed };diff --git a/cli/test/evals/testdata/repos/catalog-api-node/README.md b/cli/test/evals/testdata/repos/catalog-api-node/README.mdnew file mode 100644index 00000000..de3b8f8d--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/README.md@@ -0,0 +1,13 @@+# catalog-api++The shopfront's product catalog: category grids, product pages, search and+prices. Node 24, Express 5, PostgreSQL through `pg`. Telemetry goes to+Traceway over OTLP with `@opentelemetry/auto-instrumentations-node`+(`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME=catalog-api`).++```+npm install+npm start+```++Migrations live in `migrations/` and are applied in order with `psql -f`.diff --git a/cli/test/evals/testdata/repos/catalog-api-node/history.yaml b/cli/test/evals/testdata/repos/catalog-api-node/history.yamlnew file mode 100644index 00000000..bfaab4eb--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ absent:+ - migrations/0012_products_supplier_payload.sql+ message: 'catalog: grids, product pages, search and the supplier import'+ age: 4100h+head:+ message: 'importer: keep the supplier''s raw record on each product (supplier_payload)'+ age: 245hdiff --git a/cli/test/evals/testdata/repos/catalog-api-node/migrations/0007_products.sql b/cli/test/evals/testdata/repos/catalog-api-node/migrations/0007_products.sqlnew file mode 100644index 00000000..016d4e7f--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/migrations/0007_products.sql@@ -0,0 +1,21 @@+CREATE TABLE products (+ id bigserial PRIMARY KEY,+ category_id bigint NOT NULL REFERENCES categories (id),+ brand_id bigint REFERENCES brands (id),+ name text NOT NULL,+ slug text NOT NULL UNIQUE,+ price_cents integer NOT NULL,+ compare_at_cents integer,+ currency char(3) NOT NULL DEFAULT 'EUR',+ thumbnail_url text,+ description_html text,+ spec_sheet jsonb,+ sort_rank integer NOT NULL DEFAULT 0,+ is_active boolean NOT NULL DEFAULT true,+ search_vector tsvector,+ created_at timestamptz NOT NULL DEFAULT now(),+ updated_at timestamptz NOT NULL DEFAULT now()+);++CREATE INDEX products_category_rank_idx ON products (category_id, sort_rank DESC) WHERE is_active;+CREATE INDEX products_search_idx ON products USING gin (search_vector);diff --git a/cli/test/evals/testdata/repos/catalog-api-node/migrations/0012_products_supplier_payload.sql b/cli/test/evals/testdata/repos/catalog-api-node/migrations/0012_products_supplier_payload.sqlnew file mode 100644index 00000000..00381ef4--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/migrations/0012_products_supplier_payload.sql@@ -0,0 +1,5 @@+-- Keep the supplier's raw product record (the full feed entry, with its+-- variants, media and compliance documents) so re-imports can diff against+-- it and support can see exactly what the supplier sent.+ALTER TABLE products ADD COLUMN supplier_payload jsonb;+ALTER TABLE products ADD COLUMN supplier_synced_at timestamptz;diff --git a/cli/test/evals/testdata/repos/catalog-api-node/package.json b/cli/test/evals/testdata/repos/catalog-api-node/package.jsonnew file mode 100644index 00000000..a16d4483--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/package.json@@ -0,0 +1,15 @@+{+ "name": "catalog-api",+ "version": "4.8.0",+ "private": true,+ "main": "src/server.js",+ "scripts": {+ "start": "node --require @opentelemetry/auto-instrumentations-node/register src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "^0.64.1",+ "express": "^5.1.0",+ "pg": "^8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/catalog-api-node/src/catalog/cards.js b/cli/test/evals/testdata/repos/catalog-api-node/src/catalog/cards.jsnew file mode 100644index 00000000..c394cbfe--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/src/catalog/cards.js@@ -0,0 +1,15 @@+'use strict';++// A product card is what the grid and search results render.+function toCard(p) {+ return {+ id: p.id,+ name: p.name,+ slug: p.slug,+ priceCents: p.price_cents,+ currency: p.currency,+ thumbnailUrl: p.thumbnail_url,+ };+}++module.exports = { toCard };diff --git a/cli/test/evals/testdata/repos/catalog-api-node/src/catalog/queries.js b/cli/test/evals/testdata/repos/catalog-api-node/src/catalog/queries.jsnew file mode 100644index 00000000..276fccfb--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/src/catalog/queries.js@@ -0,0 +1,44 @@+'use strict';++const db = require('../db');++const PAGE_SIZE = 48;++async function categoryProducts(categoryId) {+ const { rows } = await db.query(+ 'SELECT * FROM products WHERE category_id = $1 AND is_active = true ORDER BY sort_rank DESC LIMIT 48',+ [categoryId],+ );+ return rows;+}++async function product(id) {+ const { rows } = await db.query('SELECT * FROM products WHERE id = $1', [id]);+ return rows[0] || null;+}++async function productImages(id) {+ const { rows } = await db.query(+ 'SELECT id, url, alt, position FROM product_images WHERE product_id = $1 ORDER BY position',+ [id],+ );+ return rows;+}++async function price(id) {+ const { rows } = await db.query(+ 'SELECT id, price_cents, currency, compare_at_cents FROM products WHERE id = $1',+ [id],+ );+ return rows[0] || null;+}++async function search(q) {+ const { rows } = await db.query(+ "SELECT id, name, slug, price_cents, currency, thumbnail_url FROM products WHERE is_active = true AND search_vector @@ plainto_tsquery('english', $1) ORDER BY ts_rank(search_vector, plainto_tsquery('english', $1)) DESC LIMIT 48",+ [q],+ );+ return rows;+}++module.exports = { PAGE_SIZE, categoryProducts, product, productImages, price, search };diff --git a/cli/test/evals/testdata/repos/catalog-api-node/src/db.js b/cli/test/evals/testdata/repos/catalog-api-node/src/db.jsnew file mode 100644index 00000000..92c4fc9e--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/src/db.js@@ -0,0 +1,12 @@+'use strict';++const { Pool } = require('pg');++const pool = new Pool({+ connectionString: process.env.DATABASE_URL,+ max: 20,+});++module.exports = {+ query: (text, params) => pool.query(text, params),+};diff --git a/cli/test/evals/testdata/repos/catalog-api-node/src/importer/supplier.js b/cli/test/evals/testdata/repos/catalog-api-node/src/importer/supplier.jsnew file mode 100644index 00000000..45c9fe7b--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/src/importer/supplier.js@@ -0,0 +1,27 @@+'use strict';++const db = require('../db');++// Upserts one product from the supplier feed. The raw feed entry is stored+// as-is in supplier_payload so the next import can diff against it.+async function upsertFromFeed(entry) {+ await db.query(+ `UPDATE products+ SET name = $2,+ price_cents = $3,+ spec_sheet = $4,+ supplier_payload = $5,+ supplier_synced_at = now(),+ updated_at = now()+ WHERE slug = $1`,+ [entry.slug, entry.title, Math.round(entry.price * 100), entry.specs, entry],+ );+}++async function importFeed(entries) {+ for (const entry of entries) {+ await upsertFromFeed(entry);+ }+}++module.exports = { importFeed, upsertFromFeed };diff --git a/cli/test/evals/testdata/repos/catalog-api-node/src/server.js b/cli/test/evals/testdata/repos/catalog-api-node/src/server.jsnew file mode 100644index 00000000..0800e260--- /dev/null+++ b/cli/test/evals/testdata/repos/catalog-api-node/src/server.js@@ -0,0 +1,57 @@+'use strict';++const express = require('express');+const q = require('./catalog/queries');+const db = require('./db');+const { toCard } = require('./catalog/cards');++const app = express();++app.get('/api/categories/:categoryId/products', async (req, res) => {+ const category = await db.query('SELECT id, name, slug FROM categories WHERE id = $1', [req.params.categoryId]);+ if (category.rows.length === 0) return res.status(404).json({ error: 'category not found' });+ const products = await q.categoryProducts(category.rows[0].id);+ res.json({ category: category.rows[0], items: products.map(toCard) });+});++app.get('/api/products/:id', async (req, res) => {+ const p = await q.product(req.params.id);+ if (!p) return res.status(404).json({ error: 'not found' });+ const images = await q.productImages(p.id);+ res.json({+ id: p.id,+ name: p.name,+ slug: p.slug,+ descriptionHtml: p.description_html,+ specs: p.spec_sheet,+ priceCents: p.price_cents,+ currency: p.currency,+ images,+ });+});++app.get('/api/products/:id/price', async (req, res) => {+ const p = await q.price(req.params.id);+ if (!p) return res.status(404).json({ error: 'not found' });+ res.json(p);+});++app.get('/api/search', async (req, res) => {+ const rows = await q.search(String(req.query.q || ''));+ res.json({ items: rows.map(toCard) });+});++app.get('/api/brands', async (_req, res) => {+ const { rows } = await db.query('SELECT id, name, slug, logo_url FROM brands ORDER BY name');+ res.json(rows);+});++app.get('/api/products/:id/reviews', async (req, res) => {+ const { rows } = await db.query(+ 'SELECT id, rating, title, body, author_name, created_at FROM reviews WHERE product_id = $1 ORDER BY created_at DESC LIMIT 20',+ [req.params.id],+ );+ res.json(rows);+});++app.listen(Number(process.env.PORT || 8080));diff --git a/cli/test/evals/testdata/repos/checkout-api-go/README.md b/cli/test/evals/testdata/repos/checkout-api-go/README.mdnew file mode 100644index 00000000..a436b647--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/README.md@@ -0,0 +1,17 @@+# checkout-api++Checkout and refunds for the shop (Go, `net/http`, PostgreSQL via pgx).+`POST /api/checkout` charges the order through the payment provider+(api.payments.example) and marks it paid; `POST /api/refunds` refunds a+charge; `GET /api/orders/{id}` reads an order.++- `internal/checkout` - handlers+- `internal/payments` - payment provider client++Traces, logs and Go runtime metrics go to Traceway over OTLP (otelhttp,+otelpgx, the contrib runtime instrumentation).++```+go test ./...+go run ./cmd/server+```diff --git a/cli/test/evals/testdata/repos/checkout-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/checkout-api-go/cmd/server/main.gonew file mode 100644index 00000000..223a6a23--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/cmd/server/main.go@@ -0,0 +1,34 @@+package main++import (+ "log/slog"+ "net/http"+ "os"++ "example.com/checkout-api/internal/checkout"+ "example.com/checkout-api/internal/payments"+)++func main() {+ h := &checkout.Handler{+ Store: newStore(os.Getenv("DATABASE_URL")),+ Payments: payments.New(envOr("PAYMENTS_URL", "https://api.payments.example"), os.Getenv("PAYMENTS_API_KEY")),+ }+ mux := http.NewServeMux()+ mux.HandleFunc("POST /api/checkout", h.Checkout)+ mux.HandleFunc("POST /api/refunds", h.Refund)+ mux.HandleFunc("GET /api/orders/{id}", h.Get)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, mux); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/checkout-api-go/cmd/server/store.go b/cli/test/evals/testdata/repos/checkout-api-go/cmd/server/store.gonew file mode 100644index 00000000..951057d9--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/cmd/server/store.go@@ -0,0 +1,25 @@+package main++import (+ "context"++ "example.com/checkout-api/internal/checkout"+)++// pgStore is the pgx-backed store in production builds; this stand-in keeps+// the module dependency-free.+type pgStore struct{ dsn string }++func newStore(dsn string) *pgStore { return &pgStore{dsn: dsn} }++func (s *pgStore) Order(ctx context.Context, id, customerID int64) (*checkout.Order, error) {+ return nil, nil+}++func (s *pgStore) OrderByID(ctx context.Context, id int64) (*checkout.Order, error) {+ return nil, nil+}++func (s *pgStore) MarkPaid(ctx context.Context, id int64, chargeID string) error { return nil }++func (s *pgStore) RecordRefund(ctx context.Context, id int64) error { return nil }diff --git a/cli/test/evals/testdata/repos/checkout-api-go/go.mod b/cli/test/evals/testdata/repos/checkout-api-go/go.modnew file mode 100644index 00000000..26a520ef--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/checkout-api++go 1.26diff --git a/cli/test/evals/testdata/repos/checkout-api-go/internal/checkout/handler.go b/cli/test/evals/testdata/repos/checkout-api-go/internal/checkout/handler.gonew file mode 100644index 00000000..b9d1249f--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/internal/checkout/handler.go@@ -0,0 +1,127 @@+// Package checkout serves checkout and refunds.+package checkout++import (+ "context"+ "encoding/json"+ "fmt"+ "log/slog"+ "net/http"+ "strconv"++ "example.com/checkout-api/internal/payments"+)++// RecordError records err on the request's span+// (trace.SpanFromContext(ctx).RecordError in production builds).+var RecordError = func(ctx context.Context, err error) {}++// Store is the orders data access the handlers need.+type Store interface {+ Order(ctx context.Context, id, customerID int64) (*Order, error)+ OrderByID(ctx context.Context, id int64) (*Order, error)+ MarkPaid(ctx context.Context, id int64, chargeID string) error+ RecordRefund(ctx context.Context, id int64) error+}++// Order is an order awaiting payment.+type Order struct {+ ID int64+ Status string+ TotalCents int64+ Currency string+ PaymentMethod string+ ChargeID string+}++// Payments is the payment provider.+type Payments interface {+ Charge(ctx context.Context, req payments.ChargeRequest) (*payments.Charge, error)+ Refund(ctx context.Context, chargeID, idempotencyKey string) error+}++// Handler serves the checkout routes.+type Handler struct {+ Store Store+ Payments Payments+}++type checkoutRequest struct {+ OrderID int64 `json:"orderId"`+ CustomerID int64 `json:"customerId"`+}++// Checkout serves POST /api/checkout.+func (h *Handler) Checkout(w http.ResponseWriter, r *http.Request) {+ var in checkoutRequest+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ http.Error(w, "bad request", http.StatusBadRequest)+ return+ }+ o, err := h.Store.Order(r.Context(), in.OrderID, in.CustomerID)+ if err != nil || o == nil || o.Status != "pending" {+ http.Error(w, "order not payable", http.StatusConflict)+ return+ }+ charge, err := h.Payments.Charge(r.Context(), payments.ChargeRequest{AmountMinor: o.TotalCents, Currency: o.Currency, PaymentMethod: o.PaymentMethod, IdempotencyKey: fmt.Sprintf("order-%d", o.ID)})+ if err != nil {+ RecordError(r.Context(), err)+ slog.ErrorContext(r.Context(), "charge failed", "error", err)+ http.Error(w, "payment failed, please try again", http.StatusBadGateway)+ return+ }+ if err := h.Store.MarkPaid(r.Context(), o.ID, charge.ID); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(map[string]any{"orderId": o.ID, "status": "paid", "chargeId": charge.ID})+}++type refundRequest struct {+ OrderID int64 `json:"orderId"`+}++// Refund serves POST /api/refunds.+func (h *Handler) Refund(w http.ResponseWriter, r *http.Request) {+ var in refundRequest+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ http.Error(w, "bad request", http.StatusBadRequest)+ return+ }+ o, err := h.Store.OrderByID(r.Context(), in.OrderID)+ if err != nil || o == nil || o.Status != "paid" {+ http.Error(w, "order not refundable", http.StatusConflict)+ return+ }+ if err := h.Payments.Refund(r.Context(), o.ChargeID, fmt.Sprintf("refund-%d", o.ID)); err != nil {+ http.Error(w, "refund failed, please try again", http.StatusBadGateway)+ return+ }+ if err := h.Store.RecordRefund(r.Context(), o.ID); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(map[string]any{"orderId": o.ID, "status": "refunded"})+}++// Get serves GET /api/orders/{id}.+func (h *Handler) Get(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad order id", http.StatusBadRequest)+ return+ }+ o, err := h.Store.OrderByID(r.Context(), id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if o == nil {+ http.NotFound(w, r)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(map[string]any{"orderId": o.ID, "status": o.Status, "totalCents": o.TotalCents, "currency": o.Currency})+}diff --git a/cli/test/evals/testdata/repos/checkout-api-go/internal/checkout/handler_test.go b/cli/test/evals/testdata/repos/checkout-api-go/internal/checkout/handler_test.gonew file mode 100644index 00000000..5adc800c--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/internal/checkout/handler_test.go@@ -0,0 +1,56 @@+package checkout++import (+ "context"+ "errors"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"++ "example.com/checkout-api/internal/payments"+)++type fakeStore struct {+ order *Order+ paid string+}++func (f *fakeStore) Order(context.Context, int64, int64) (*Order, error) { return f.order, nil }+func (f *fakeStore) OrderByID(context.Context, int64) (*Order, error) { return f.order, nil }+func (f *fakeStore) RecordRefund(context.Context, int64) error { return nil }+func (f *fakeStore) MarkPaid(_ context.Context, _ int64, chargeID string) error {+ f.paid = chargeID+ return nil+}++type fakePayments struct{ err error }++func (f fakePayments) Charge(context.Context, payments.ChargeRequest) (*payments.Charge, error) {+ if f.err != nil {+ return nil, f.err+ }+ return &payments.Charge{ID: "ch_1", Status: "succeeded"}, nil+}++func (f fakePayments) Refund(context.Context, string, string) error { return f.err }++func TestCheckoutMarksOrderPaid(t *testing.T) {+ st := &fakeStore{order: &Order{ID: 7, Status: "pending", TotalCents: 4200, Currency: "EUR"}}+ h := &Handler{Store: st, Payments: fakePayments{}}+ rec := httptest.NewRecorder()+ h.Checkout(rec, httptest.NewRequest(http.MethodPost, "/api/checkout", strings.NewReader(`{"orderId":7,"customerId":1}`)))+ if rec.Code != http.StatusOK || st.paid != "ch_1" {+ t.Fatalf("status %d, paid %q", rec.Code, st.paid)+ }+}++func TestCheckoutReportsProviderFailure(t *testing.T) {+ st := &fakeStore{order: &Order{ID: 7, Status: "pending", TotalCents: 4200, Currency: "EUR"}}+ h := &Handler{Store: st, Payments: fakePayments{err: errors.New("boom")}}+ rec := httptest.NewRecorder()+ h.Checkout(rec, httptest.NewRequest(http.MethodPost, "/api/checkout", strings.NewReader(`{"orderId":7,"customerId":1}`)))+ if rec.Code != http.StatusBadGateway || st.paid != "" {+ t.Fatalf("status %d, paid %q", rec.Code, st.paid)+ }+}diff --git a/cli/test/evals/testdata/repos/checkout-api-go/internal/payments/client.go b/cli/test/evals/testdata/repos/checkout-api-go/internal/payments/client.gonew file mode 100644index 00000000..d9c0a1bf--- /dev/null+++ b/cli/test/evals/testdata/repos/checkout-api-go/internal/payments/client.go@@ -0,0 +1,83 @@+// Package payments is the client for the payment provider.+package payments++import (+ "bytes"+ "context"+ "encoding/json"+ "fmt"+ "net/http"+ "time"+)++// Instrument wraps the transport with tracing (otelhttp.NewTransport in+// production builds).+var Instrument = func(rt http.RoundTripper) http.RoundTripper { return rt }++// Client talks to the payment provider's REST API.+type Client struct {+ BaseURL string+ APIKey string+ http *http.Client+}++// New returns a client for the provider at baseURL.+func New(baseURL, apiKey string) *Client {+ return &Client{BaseURL: baseURL, APIKey: apiKey, http: &http.Client{+ Timeout: 10 * time.Second,+ Transport: Instrument(http.DefaultTransport),+ }}+}++// Charge is the provider's charge object.+type Charge struct {+ ID string `json:"id"`+ Status string `json:"status"`+}++// ChargeRequest charges an order total to a saved payment method.+type ChargeRequest struct {+ AmountMinor int64 `json:"amount"`+ Currency string `json:"currency"`+ PaymentMethod string `json:"payment_method"`+ IdempotencyKey string `json:"-"`+}++// Charge creates a charge.+func (c *Client) Charge(ctx context.Context, req ChargeRequest) (*Charge, error) {+ var out Charge+ return &out, c.post(ctx, "/v1/charges", req.IdempotencyKey, req, &out)+}++// Refund refunds a charge in full.+func (c *Client) Refund(ctx context.Context, chargeID, idempotencyKey string) error {+ return c.post(ctx, "/v1/refunds", idempotencyKey, map[string]string{"charge": chargeID}, nil)+}++func (c *Client) post(ctx context.Context, path, idem string, body, into any) error {+ raw, err := json.Marshal(body)+ if err != nil {+ return err+ }+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.BaseURL+path, bytes.NewReader(raw))+ if err != nil {+ return err+ }+ req.Header.Set("Authorization", "Bearer "+c.APIKey)+ req.Header.Set("Content-Type", "application/json")+ if idem != "" {+ req.Header.Set("Idempotency-Key", idem)+ }+ resp, err := c.http.Do(req)+ if err != nil {+ return err+ }+ defer resp.Body.Close()+ if resp.StatusCode >= 300 {+ return fmt.Errorf("payments: %s answered %d", path, resp.StatusCode)+ }+ if into == nil {+ return nil+ }+ return json.NewDecoder(resp.Body).Decode(into)+}diff --git a/cli/test/evals/testdata/repos/geo-api-go/README.md b/cli/test/evals/testdata/repos/geo-api-go/README.mdnew file mode 100644index 00000000..81c2f532--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/README.md@@ -0,0 +1,23 @@+# geo-api++Places search for the maps frontend: autocomplete, place details, reverse+geocoding and batch geocoding for imports. Go, `net/http`, PostgreSQL with+PostGIS through pgx. Traces, logs (`log/slog` through the OpenTelemetry+bridge) and the Go runtime metrics+(`go.opentelemetry.io/contrib/instrumentation/runtime`) go to Traceway over+OTLP; host metrics come from the collector DaemonSet.++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/autocomplete?q=&locale=` – the 10 best places for a prefix (memoized)+- `GET /api/places/{id}` – one place+- `GET /api/reverse?lat=&lon=` – places near a point+- `POST /api/geocode/batch` – up to 5000 addresses at once, for imports++Runs on Kubernetes (`deploy/k8s/`), one replica with a 1GiB memory limit and+`GOMEMLIMIT=900MiB`.diff --git a/cli/test/evals/testdata/repos/geo-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/geo-api-go/cmd/server/main.gonew file mode 100644index 00000000..01472a57--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/cmd/server/main.go@@ -0,0 +1,55 @@+package main++import (+ "context"+ "log/slog"+ "net/http"+ "os"+ "time"++ "github.com/exaring/otelpgx"+ "github.com/jackc/pgx/v5/pgxpool"+ "go.opentelemetry.io/contrib/bridges/otelslog"+ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"+ "go.opentelemetry.io/contrib/instrumentation/runtime"++ "github.com/acme/geo-api/internal/places"+)++var version = "dev"++func main() {+ ctx := context.Background()+ shutdown, err := setupTelemetry(ctx)+ if err != nil {+ slog.Error("telemetry setup failed", "err", err)+ os.Exit(1)+ }+ defer shutdown(ctx)+ slog.SetDefault(otelslog.NewLogger("geo-api"))+ // go.memory.*, go.goroutine.count, go.processor.limit, ...+ if err := runtime.Start(runtime.WithMinimumReadMemStatsInterval(15 * time.Second)); err != nil {+ slog.Error("runtime metrics", "err", err)+ }++ cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("database config", "err", err)+ os.Exit(1)+ }+ cfg.ConnConfig.Tracer = otelpgx.NewTracer()+ db, err := pgxpool.NewWithConfig(ctx, cfg)+ if err != nil {+ slog.Error("database connect", "err", err)+ os.Exit(1)+ }++ mux := http.NewServeMux()+ places.NewHandler(db).Routes(mux)+ slog.Info("starting geo-api", "version", version, "addr", ":8080", "gomemlimit", os.Getenv("GOMEMLIMIT"))+ srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(mux, "geo-api"), ReadHeaderTimeout: 10 * time.Second}+ if err := srv.ListenAndServe(); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}diff --git a/cli/test/evals/testdata/repos/geo-api-go/cmd/server/telemetry.go b/cli/test/evals/testdata/repos/geo-api-go/cmd/server/telemetry.gonew file mode 100644index 00000000..e6baa89e--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/cmd/server/telemetry.go@@ -0,0 +1,41 @@+package main++import (+ "context"+ "errors"++ "go.opentelemetry.io/otel"+ "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"+ "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"+ "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"+ "go.opentelemetry.io/otel/log/global"+ sdklog "go.opentelemetry.io/otel/sdk/log"+ sdkmetric "go.opentelemetry.io/otel/sdk/metric"+ sdktrace "go.opentelemetry.io/otel/sdk/trace"+)++// setupTelemetry installs the OTLP trace, metric and log pipelines; endpoint+// and resource come from the OTEL_* environment variables.+func setupTelemetry(ctx context.Context) (func(context.Context) error, error) {+ te, err := otlptracehttp.New(ctx)+ if err != nil {+ return nil, err+ }+ tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(te))+ otel.SetTracerProvider(tp)+ me, err := otlpmetrichttp.New(ctx)+ if err != nil {+ return nil, err+ }+ mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(sdkmetric.NewPeriodicReader(me)))+ otel.SetMeterProvider(mp)+ le, err := otlploghttp.New(ctx)+ if err != nil {+ return nil, err+ }+ lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(sdklog.NewBatchProcessor(le)))+ global.SetLoggerProvider(lp)+ return func(ctx context.Context) error {+ return errors.Join(tp.Shutdown(ctx), mp.Shutdown(ctx), lp.Shutdown(ctx))+ }, nil+}diff --git a/cli/test/evals/testdata/repos/geo-api-go/deploy/k8s/deployment.yaml b/cli/test/evals/testdata/repos/geo-api-go/deploy/k8s/deployment.yamlnew file mode 100644index 00000000..c2e29c9b--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/deploy/k8s/deployment.yaml@@ -0,0 +1,25 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: geo-api+ labels: {app: geo-api}+spec:+ replicas: 1+ selector:+ matchLabels: {app: geo-api}+ template:+ metadata:+ labels: {app: geo-api}+ spec:+ containers:+ - name: geo-api+ image: registry.example.com/geo-api:2.9.4+ ports: [{containerPort: 8080}]+ env:+ - {name: GOMEMLIMIT, value: 900MiB}+ - {name: OTEL_SERVICE_NAME, value: geo-api}+ - {name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector.observability:4318"}+ envFrom: [{secretRef: {name: geo-api}}]+ resources:+ requests: {cpu: 500m, memory: 1Gi}+ limits: {cpu: "2", memory: 1Gi}diff --git a/cli/test/evals/testdata/repos/geo-api-go/go.mod b/cli/test/evals/testdata/repos/geo-api-go/go.modnew file mode 100644index 00000000..74d4090c--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/go.mod@@ -0,0 +1,12 @@+module github.com/acme/geo-api++go 1.26++require (+ github.com/exaring/otelpgx v0.9.3+ github.com/jackc/pgx/v5 v5.7.6+ go.opentelemetry.io/contrib/bridges/otelslog v0.13.0+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0+ go.opentelemetry.io/contrib/instrumentation/runtime v0.63.0+ go.opentelemetry.io/otel v1.38.0+)diff --git a/cli/test/evals/testdata/repos/geo-api-go/internal/memo/memo.go b/cli/test/evals/testdata/repos/geo-api-go/internal/memo/memo.gonew file mode 100644index 00000000..529e2af6--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/internal/memo/memo.go@@ -0,0 +1,50 @@+// Package memo caches computed results in process.+package memo++import (+ "sync"+ "time"+)++type entry[V any] struct {+ value V+ expires time.Time+}++// Memo is a concurrency-safe map of results with a time to live.+type Memo[V any] struct {+ mu sync.RWMutex+ ttl time.Duration+ m map[string]entry[V]+}++// New returns a memo whose entries live for ttl.+func New[V any](ttl time.Duration) *Memo[V] {+ return &Memo[V]{ttl: ttl, m: map[string]entry[V]{}}+}++// Get returns the value stored under key if it has not expired.+func (c *Memo[V]) Get(key string) (V, bool) {+ c.mu.RLock()+ e, ok := c.m[key]+ c.mu.RUnlock()+ if !ok || time.Now().After(e.expires) {+ var zero V+ return zero, false+ }+ return e.value, true+}++// Put stores value under key.+func (c *Memo[V]) Put(key string, value V) {+ c.mu.Lock()+ c.m[key] = entry[V]{value: value, expires: time.Now().Add(c.ttl)}+ c.mu.Unlock()+}++// Len reports how many entries are stored.+func (c *Memo[V]) Len() int {+ c.mu.RLock()+ defer c.mu.RUnlock()+ return len(c.m)+}diff --git a/cli/test/evals/testdata/repos/geo-api-go/internal/memo/memo_test.go b/cli/test/evals/testdata/repos/geo-api-go/internal/memo/memo_test.gonew file mode 100644index 00000000..675fe2b1--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/internal/memo/memo_test.go@@ -0,0 +1,26 @@+package memo++import (+ "testing"+ "time"+)++func TestGetReturnsFreshValues(t *testing.T) {+ m := New[int](time.Minute)+ m.Put("a", 1)+ if v, ok := m.Get("a"); !ok || v != 1 {+ t.Fatalf("Get(a) = %d, %v", v, ok)+ }+ if _, ok := m.Get("b"); ok {+ t.Fatal("Get(b) found a value that was never stored")+ }+}++func TestExpiredValuesAreNotReturned(t *testing.T) {+ m := New[int](10 * time.Millisecond)+ m.Put("a", 1)+ time.Sleep(20 * time.Millisecond)+ if _, ok := m.Get("a"); ok {+ t.Fatal("expired value returned")+ }+}diff --git a/cli/test/evals/testdata/repos/geo-api-go/internal/places/batch.go b/cli/test/evals/testdata/repos/geo-api-go/internal/places/batch.gonew file mode 100644index 00000000..aec6f80f--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/internal/places/batch.go@@ -0,0 +1,59 @@+package places++import (+ "encoding/json"+ "net/http"+ "strings"+)++const maxBatch = 5000++type batchRequest struct {+ Addresses []string `json:"addresses"`+}++type batchResult struct {+ Index int `json:"index"`+ ID int64 `json:"id,omitempty"`+ Lat float64 `json:"lat,omitempty"`+ Lon float64 `json:"lon,omitempty"`+ Found bool `json:"found"`+}++// batch geocodes up to maxBatch addresses in one query, for CSV imports.+func (h *Handler) batch(w http.ResponseWriter, r *http.Request) {+ var req batchRequest+ if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4<<20)).Decode(&req); err != nil {+ http.Error(w, "invalid body", http.StatusBadRequest)+ return+ }+ if len(req.Addresses) == 0 || len(req.Addresses) > maxBatch {+ http.Error(w, "between 1 and 5000 addresses", http.StatusUnprocessableEntity)+ return+ }+ normalized := make([]string, len(req.Addresses))+ for i, a := range req.Addresses {+ normalized[i] = strings.Join(strings.Fields(strings.ToLower(a)), " ")+ }+ rows, err := h.db.Query(r.Context(), `SELECT q.idx, p.id, p.lat, p.lon FROM places p JOIN unnest($1::text[]) WITH ORDINALITY AS q(addr, idx) ON p.normalized_address = q.addr`, normalized)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ defer rows.Close()+ out := make([]batchResult, len(normalized))+ for i := range out {+ out[i].Index = i+ }+ for rows.Next() {+ var idx int+ var res batchResult+ if err := rows.Scan(&idx, &res.ID, &res.Lat, &res.Lon); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ res.Index, res.Found = idx-1, true+ out[idx-1] = res+ }+ writeJSON(w, out)+}diff --git a/cli/test/evals/testdata/repos/geo-api-go/internal/places/places.go b/cli/test/evals/testdata/repos/geo-api-go/internal/places/places.gonew file mode 100644index 00000000..3da8c3ed--- /dev/null+++ b/cli/test/evals/testdata/repos/geo-api-go/internal/places/places.go@@ -0,0 +1,131 @@+// Package places serves place search.+package places++import (+ "context"+ "encoding/json"+ "net/http"+ "strconv"+ "strings"+ "time"++ "github.com/jackc/pgx/v5/pgxpool"++ "github.com/acme/geo-api/internal/memo"+)++// Place is one searchable place.+type Place struct {+ ID int64 `json:"id"`+ Name string `json:"name"`+ Kind string `json:"kind"`+ Country string `json:"country"`+ Lat float64 `json:"lat"`+ Lon float64 `json:"lon"`+ Address string `json:"address,omitempty"`+ Timezone string `json:"timezone,omitempty"`+}++// Handler serves the place routes.+type Handler struct {+ db *pgxpool.Pool+ // suggestions memoizes autocomplete results per locale and prefix for+ // ten minutes; most keystrokes repeat a prefix someone typed recently.+ suggestions *memo.Memo[[]Place]+}++// NewHandler returns the handler.+func NewHandler(db *pgxpool.Pool) *Handler {+ return &Handler{db: db, suggestions: memo.New[[]Place](10 * time.Minute)}+}++// Routes registers the routes.+func (h *Handler) Routes(mux *http.ServeMux) {+ mux.HandleFunc("GET /api/autocomplete", h.autocomplete)+ mux.HandleFunc("GET /api/places/{id}", h.place)+ mux.HandleFunc("GET /api/reverse", h.reverse)+ mux.HandleFunc("POST /api/geocode/batch", h.batch)+}++func (h *Handler) autocomplete(w http.ResponseWriter, r *http.Request) {+ q := strings.ToLower(strings.TrimSpace(r.URL.Query().Get("q")))+ if len(q) < 2 {+ writeJSON(w, []Place{})+ return+ }+ key := r.URL.Query().Get("locale") + "|" + q+ if hit, ok := h.suggestions.Get(key); ok {+ writeJSON(w, hit)+ return+ }+ out, err := h.query(r.Context(), `SELECT id, name, kind, country, lat, lon FROM places WHERE name_prefix @@ to_tsquery($1) ORDER BY popularity DESC LIMIT 10`, q+":*")+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ h.suggestions.Put(key, out)+ writeJSON(w, out)+}++func (h *Handler) place(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad id", http.StatusBadRequest)+ return+ }+ var p Place+ err = h.db.QueryRow(r.Context(), `SELECT id, name, kind, country, lat, lon, address, timezone FROM places WHERE id = $1`, id).+ Scan(&p.ID, &p.Name, &p.Kind, &p.Country, &p.Lat, &p.Lon, &p.Address, &p.Timezone)+ if err != nil {+ http.NotFound(w, r)+ return+ }+ writeJSON(w, p)+}++func (h *Handler) reverse(w http.ResponseWriter, r *http.Request) {+ lat, err1 := strconv.ParseFloat(r.URL.Query().Get("lat"), 64)+ lon, err2 := strconv.ParseFloat(r.URL.Query().Get("lon"), 64)+ if err1 != nil || err2 != nil {+ http.Error(w, "lat and lon are required", http.StatusBadRequest)+ return+ }+ rows, err := h.db.Query(r.Context(), `SELECT id, name, kind, country FROM places WHERE ST_DWithin(geom, ST_SetSRID(ST_MakePoint($1, $2), 4326), 200) ORDER BY geom <-> ST_SetSRID(ST_MakePoint($1, $2), 4326) LIMIT 5`, lon, lat)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ defer rows.Close()+ var out []Place+ for rows.Next() {+ var p Place+ if err := rows.Scan(&p.ID, &p.Name, &p.Kind, &p.Country); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ out = append(out, p)+ }+ writeJSON(w, out)+}++func (h *Handler) query(ctx context.Context, sql string, args ...any) ([]Place, error) {+ rows, err := h.db.Query(ctx, sql, args...)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Place+ for rows.Next() {+ var p Place+ if err := rows.Scan(&p.ID, &p.Name, &p.Kind, &p.Country, &p.Lat, &p.Lon); err != nil {+ return nil, err+ }+ out = append(out, p)+ }+ return out, rows.Err()+}++func writeJSON(w http.ResponseWriter, v any) {+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(v)+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/Dockerfile b/cli/test/evals/testdata/repos/insights-api-java/Dockerfilenew file mode 100644index 00000000..241dd95c--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/Dockerfile@@ -0,0 +1,10 @@+FROM eclipse-temurin:21-jre+WORKDIR /app+ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.20.1/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar+COPY build/libs/insights-api-2.31.0.jar /app/app.jar+ENV OTEL_SERVICE_NAME=insights-api \+ OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \+ OTEL_METRIC_EXPORT_INTERVAL=60000 \+ JAVA_TOOL_OPTIONS="-javaagent:/app/opentelemetry-javaagent.jar -Xms4g -Xmx4g -XX:+UseG1GC"+EXPOSE 8080+ENTRYPOINT ["java", "-jar", "/app/app.jar"]diff --git a/cli/test/evals/testdata/repos/insights-api-java/README.md b/cli/test/evals/testdata/repos/insights-api-java/README.mdnew file mode 100644index 00000000..23c65654--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/README.md@@ -0,0 +1,17 @@+# insights-api++Product analytics for the insights dashboard: retention cohorts, funnels and+saved dashboards over the `events` table. Spring Boot 3.5 on Java 21,+PostgreSQL through `JdbcTemplate`. The OpenTelemetry Java agent ships traces+and the JVM runtime metrics to Traceway (see `Dockerfile`).++```+./gradlew test+./gradlew bootRun+```++Routes:++- `GET /api/insights/retention?workspace=&from=&to=&interval=week` – weekly retention cohorts+- `GET /api/insights/funnels?workspace=&funnel=` – funnel conversion per step+- `GET /api/insights/dashboards/{id}` – a saved dashboarddiff --git a/cli/test/evals/testdata/repos/insights-api-java/build.gradle.kts b/cli/test/evals/testdata/repos/insights-api-java/build.gradle.ktsnew file mode 100644index 00000000..31ffe45f--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/build.gradle.kts@@ -0,0 +1,22 @@+plugins {+ java+ id("org.springframework.boot") version "3.5.6"+ id("io.spring.dependency-management") version "1.1.7"+}++group = "com.acme"+version = "2.31.0"++java { toolchain { languageVersion = JavaLanguageVersion.of(21) } }++repositories { mavenCentral() }++dependencies {+ implementation("org.springframework.boot:spring-boot-starter-web")+ implementation("org.springframework.boot:spring-boot-starter-jdbc")+ implementation("io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:2.20.1")+ runtimeOnly("org.postgresql:postgresql")+ testImplementation("org.springframework.boot:spring-boot-starter-test")+}++tasks.test { useJUnitPlatform() }diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/InsightsApplication.java b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/InsightsApplication.javanew file mode 100644index 00000000..fd5de751--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/InsightsApplication.java@@ -0,0 +1,11 @@+package com.acme.insights;++import org.springframework.boot.SpringApplication;+import org.springframework.boot.autoconfigure.SpringBootApplication;++@SpringBootApplication+public class InsightsApplication {+ public static void main(String[] args) {+ SpringApplication.run(InsightsApplication.class, args);+ }+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/CohortTable.java b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/CohortTable.javanew file mode 100644index 00000000..9dd23b37--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/CohortTable.java@@ -0,0 +1,26 @@+package com.acme.insights.retention;++import java.time.LocalDate;+import java.util.ArrayList;+import java.util.List;+import java.util.Map;+import java.util.Set;+import java.util.TreeMap;++public record CohortTable(Interval interval, List<Row> rows) {+ public record Row(LocalDate cohort, int users, List<Double> retained) {}++ static CohortTable from(TreeMap<LocalDate, Set<Long>> cohorts,+ Map<LocalDate, Map<LocalDate, Set<Long>>> active, Interval interval) {+ List<Row> rows = new ArrayList<>();+ for (Map.Entry<LocalDate, Set<Long>> c : cohorts.entrySet()) {+ int size = c.getValue().size();+ List<Double> retained = new ArrayList<>();+ for (Map.Entry<LocalDate, Set<Long>> p : new TreeMap<>(active.getOrDefault(c.getKey(), Map.of())).entrySet()) {+ retained.add(size == 0 ? 0.0 : (double) p.getValue().size() / size);+ }+ rows.add(new Row(c.getKey(), size, retained));+ }+ return new CohortTable(interval, rows);+ }+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/EventRepository.java b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/EventRepository.javanew file mode 100644index 00000000..24d0e8b2--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/EventRepository.java@@ -0,0 +1,36 @@+package com.acme.insights.retention;++import java.sql.Timestamp;+import java.time.Instant;+import java.time.LocalDate;+import java.time.ZoneId;+import java.util.List;+import org.springframework.jdbc.core.JdbcTemplate;+import org.springframework.stereotype.Repository;++@Repository+public class EventRepository {+ private final JdbcTemplate jdbc;++ public EventRepository(JdbcTemplate jdbc) {+ this.jdbc = jdbc;+ }++ public record EventRow(long userId, String eventName, Instant occurredAt, String properties) {}++ public record Workspace(long id, String name, ZoneId timezone) {}++ public Workspace workspace(long id) {+ return jdbc.queryForObject("SELECT id, name, timezone FROM workspaces WHERE id = ?",+ (rs, n) -> new Workspace(rs.getLong(1), rs.getString(2), ZoneId.of(rs.getString(3))), id);+ }++ public List<EventRow> events(long workspaceId, LocalDate from, LocalDate to, ZoneId zone) {+ return jdbc.query(+ "SELECT user_id, event_name, occurred_at, properties FROM events WHERE workspace_id = ? AND occurred_at >= ? AND occurred_at < ? ORDER BY occurred_at",+ (rs, n) -> new EventRow(rs.getLong(1), rs.getString(2), rs.getTimestamp(3).toInstant(), rs.getString(4)),+ workspaceId,+ Timestamp.from(from.atStartOfDay(zone).toInstant()),+ Timestamp.from(to.atStartOfDay(zone).toInstant()));+ }+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/Interval.java b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/Interval.javanew file mode 100644index 00000000..086d18c9--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/Interval.java@@ -0,0 +1,17 @@+package com.acme.insights.retention;++import java.time.DayOfWeek;+import java.time.LocalDate;+import java.time.temporal.TemporalAdjusters;++public enum Interval {+ DAY, WEEK;++ public static Interval parse(String s) {+ return "day".equalsIgnoreCase(s) ? DAY : WEEK;+ }++ public LocalDate bucket(LocalDate d) {+ return this == DAY ? d : d.with(TemporalAdjusters.previousOrSame(DayOfWeek.MONDAY));+ }+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/RetentionController.java b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/RetentionController.javanew file mode 100644index 00000000..6b7b22c0--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/RetentionController.java@@ -0,0 +1,29 @@+package com.acme.insights.retention;++import java.time.LocalDate;+import org.springframework.format.annotation.DateTimeFormat;+import org.springframework.http.ResponseEntity;+import org.springframework.web.bind.annotation.GetMapping;+import org.springframework.web.bind.annotation.RequestParam;+import org.springframework.web.bind.annotation.RestController;++@RestController+public class RetentionController {+ private final RetentionService retention;++ public RetentionController(RetentionService retention) {+ this.retention = retention;+ }++ @GetMapping("/api/insights/retention")+ public ResponseEntity<CohortTable> retention(+ @RequestParam long workspace,+ @RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate from,+ @RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE) LocalDate to,+ @RequestParam(defaultValue = "week") String interval) {+ if (!to.isAfter(from)) {+ return ResponseEntity.unprocessableEntity().build();+ }+ return ResponseEntity.ok(retention.buildCohorts(workspace, from, to, Interval.parse(interval)));+ }+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/RetentionService.java b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/RetentionService.javanew file mode 100644index 00000000..34296315--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/java/com/acme/insights/retention/RetentionService.java@@ -0,0 +1,54 @@+package com.acme.insights.retention;++import io.opentelemetry.instrumentation.annotations.WithSpan;+import java.time.LocalDate;+import java.time.ZoneId;+import java.util.ArrayList;+import java.util.HashMap;+import java.util.HashSet;+import java.util.List;+import java.util.Map;+import java.util.Set;+import java.util.TreeMap;+import org.springframework.stereotype.Service;++@Service+public class RetentionService {+ private final EventRepository events;++ public RetentionService(EventRepository events) {+ this.events = events;+ }++ /**+ * Weekly (or daily) retention: users grouped by the period of their first+ * event, and for each later period the share of them who came back.+ */+ @WithSpan+ public CohortTable buildCohorts(long workspaceId, LocalDate from, LocalDate to, Interval interval) {+ EventRepository.Workspace ws = events.workspace(workspaceId);+ ZoneId zone = ws.timezone();+ List<EventRepository.EventRow> rows = events.events(workspaceId, from, to, zone);++ Map<Long, List<EventRepository.EventRow>> byUser = new HashMap<>();+ for (EventRepository.EventRow row : rows) {+ byUser.computeIfAbsent(row.userId(), k -> new ArrayList<>()).add(row);+ }++ TreeMap<LocalDate, Set<Long>> cohorts = new TreeMap<>();+ Map<LocalDate, Map<LocalDate, Set<Long>>> activeByCohort = new HashMap<>();+ for (Map.Entry<Long, List<EventRepository.EventRow>> e : byUser.entrySet()) {+ List<EventRepository.EventRow> userEvents = new ArrayList<>(e.getValue());+ userEvents.sort((a, b) -> a.occurredAt().compareTo(b.occurredAt()));+ LocalDate cohort = interval.bucket(userEvents.get(0).occurredAt().atZone(zone).toLocalDate());+ cohorts.computeIfAbsent(cohort, k -> new HashSet<>()).add(e.getKey());+ for (EventRepository.EventRow ev : userEvents) {+ LocalDate period = interval.bucket(ev.occurredAt().atZone(zone).toLocalDate());+ activeByCohort.computeIfAbsent(cohort, k -> new HashMap<>())+ .computeIfAbsent(period, k -> new HashSet<>())+ .add(e.getKey());+ }+ }+ return CohortTable.from(cohorts, activeByCohort, interval);+ }+}diff --git a/cli/test/evals/testdata/repos/insights-api-java/src/main/resources/application.yaml b/cli/test/evals/testdata/repos/insights-api-java/src/main/resources/application.yamlnew file mode 100644index 00000000..c3f2059d--- /dev/null+++ b/cli/test/evals/testdata/repos/insights-api-java/src/main/resources/application.yaml@@ -0,0 +1,11 @@+spring:+ application:+ name: insights-api+ datasource:+ url: ${DATABASE_URL:jdbc:postgresql://localhost:5432/insights}+ username: ${DATABASE_USER:insights}+ password: ${DATABASE_PASSWORD:}+ hikari:+ maximum-pool-size: 20+server:+ port: 8080diff --git a/cli/test/evals/testdata/repos/inventory-api-go/.history/01/internal/db/db.go b/cli/test/evals/testdata/repos/inventory-api-go/.history/01/internal/db/db.gonew file mode 100644index 00000000..e25d950d--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/.history/01/internal/db/db.go@@ -0,0 +1,51 @@+// Package db is the seam between the handlers and pgx: the handlers use these+// interfaces, and the pgx build (driver_pgx.go, -tags pgx) adapts+// *pgxpool.Pool and pgx.Tx to them.+package db++import (+ "context"+ "errors"+)++// Tx is the part of pgx.Tx the handlers use. Exec returns the rows affected.+type Tx interface {+ Exec(ctx context.Context, sql string, args ...any) (int64, error)+ Commit(ctx context.Context) error+ Rollback(ctx context.Context) error+}++// Pool begins transactions.+type Pool interface {+ Begin(ctx context.Context) (Tx, error)+}++// Config is the pgxpool configuration.+var Config = struct {+ MaxConns int32+ MinConns int32+ MaxConnIdleTimeMs int+}{+ MaxConns: 20,+ MinConns: 4,+ MaxConnIdleTimeMs: 300_000,+}++// PgError is the subset of *pgconn.PgError the handlers inspect.+type PgError struct {+ Code string+ Message string+ Detail string+ Where string+}++func (e *PgError) Error() string { return "ERROR: " + e.Message + " (SQLSTATE " + e.Code + ")" }++// SQLState returns the SQLSTATE of a Postgres error, or "".+func SQLState(err error) string {+ var pg *PgError+ if errors.As(err, &pg) {+ return pg.Code+ }+ return ""+}diff --git a/cli/test/evals/testdata/repos/inventory-api-go/README.md b/cli/test/evals/testdata/repos/inventory-api-go/README.mdnew file mode 100644index 00000000..eada6892--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/README.md@@ -0,0 +1,16 @@+# inventory-api++Stock levels and cart reservations for the fulfillment platform. Go,+`net/http`, PostgreSQL through pgx v5 (`pgxpool`, traced with `otelpgx`).+Traces, logs and metrics go to Traceway over OTLP.++```+go test ./...+go run -tags pgx ./cmd/server+```++Routes:++- `POST /api/reservations` – reserve stock for every line of a cart (15 minute hold)+- `POST /api/reservations/{id}/release` – release a hold+- `GET /api/inventory/{sku}`, `GET /api/warehouses` (served by the stock package, not shown)diff --git a/cli/test/evals/testdata/repos/inventory-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/inventory-api-go/cmd/server/main.gonew file mode 100644index 00000000..f75a7ae8--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/cmd/server/main.go@@ -0,0 +1,35 @@+package main++import (+ "log/slog"+ "net/http"+ "os"++ "example.com/inventory-api/internal/db"+ "example.com/inventory-api/internal/reserve"+)++func main() {+ pool, err := openPool(os.Getenv("DATABASE_URL"), db.Config.MaxConns)+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ h := &reserve.Handler{Pool: pool}+ mux := http.NewServeMux()+ mux.HandleFunc("POST /api/reservations", h.Create)+ mux.HandleFunc("POST /api/reservations/{id}/release", h.Release)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr, "max_conns", db.Config.MaxConns)+ if err := http.ListenAndServe(addr, mux); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/inventory-api-go/cmd/server/pool_stub.go b/cli/test/evals/testdata/repos/inventory-api-go/cmd/server/pool_stub.gonew file mode 100644index 00000000..26ebeeda--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/cmd/server/pool_stub.go@@ -0,0 +1,15 @@+//go:build !pgx++package main++import (+ "errors"++ "example.com/inventory-api/internal/db"+)++// openPool is provided by the pgx build (go run -tags pgx ./cmd/server),+// which wires pgxpool with the otelpgx tracer.+func openPool(string, int32) (db.Pool, error) {+ return nil, errors.New("built without -tags pgx")+}diff --git a/cli/test/evals/testdata/repos/inventory-api-go/go.mod b/cli/test/evals/testdata/repos/inventory-api-go/go.modnew file mode 100644index 00000000..90aaa3b6--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/inventory-api++go 1.26diff --git a/cli/test/evals/testdata/repos/inventory-api-go/history.yaml b/cli/test/evals/testdata/repos/inventory-api-go/history.yamlnew file mode 100644index 00000000..88c4c379--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/history.yaml@@ -0,0 +1,7 @@+commits:+- dir: .history/01+ message: 'inventory: stock levels and cart reservations'+ age: 2900h+head:+ message: 'db: raise the pgx pool to 40 connections (evening drop latency)'+ age: 175hdiff --git a/cli/test/evals/testdata/repos/inventory-api-go/internal/db/db.go b/cli/test/evals/testdata/repos/inventory-api-go/internal/db/db.gonew file mode 100644index 00000000..9b579298--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/internal/db/db.go@@ -0,0 +1,51 @@+// Package db is the seam between the handlers and pgx: the handlers use these+// interfaces, and the pgx build (driver_pgx.go, -tags pgx) adapts+// *pgxpool.Pool and pgx.Tx to them.+package db++import (+ "context"+ "errors"+)++// Tx is the part of pgx.Tx the handlers use. Exec returns the rows affected.+type Tx interface {+ Exec(ctx context.Context, sql string, args ...any) (int64, error)+ Commit(ctx context.Context) error+ Rollback(ctx context.Context) error+}++// Pool begins transactions.+type Pool interface {+ Begin(ctx context.Context) (Tx, error)+}++// Config is the pgxpool configuration.+var Config = struct {+ MaxConns int32+ MinConns int32+ MaxConnIdleTimeMs int+}{+ MaxConns: 40,+ MinConns: 4,+ MaxConnIdleTimeMs: 300_000,+}++// PgError is the subset of *pgconn.PgError the handlers inspect.+type PgError struct {+ Code string+ Message string+ Detail string+ Where string+}++func (e *PgError) Error() string { return "ERROR: " + e.Message + " (SQLSTATE " + e.Code + ")" }++// SQLState returns the SQLSTATE of a Postgres error, or "".+func SQLState(err error) string {+ var pg *PgError+ if errors.As(err, &pg) {+ return pg.Code+ }+ return ""+}diff --git a/cli/test/evals/testdata/repos/inventory-api-go/internal/reserve/reserve.go b/cli/test/evals/testdata/repos/inventory-api-go/internal/reserve/reserve.gonew file mode 100644index 00000000..b2058ed8--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/internal/reserve/reserve.go@@ -0,0 +1,118 @@+// Package reserve holds stock for carts.+package reserve++import (+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "log/slog"+ "net/http"++ "example.com/inventory-api/internal/db"+)++// Line is one cart line to reserve.+type Line struct {+ SKU string `json:"sku"`+ WarehouseID int64 `json:"warehouseId"`+ Qty int `json:"qty"`+}++// Request is the body of POST /api/reservations.+type Request struct {+ CartID string `json:"cartId"`+ Lines []Line `json:"lines"`+}++// ErrOutOfStock means a line could not be held.+var ErrOutOfStock = errors.New("out of stock")++// Handler serves the reservation routes.+type Handler struct{ Pool db.Pool }++// Create serves POST /api/reservations.+func (h *Handler) Create(w http.ResponseWriter, r *http.Request) {+ var req Request+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.CartID == "" || len(req.Lines) == 0 {+ http.Error(w, "bad request", http.StatusBadRequest)+ return+ }+ err := Reserve(r.Context(), h.Pool, req)+ switch {+ case err == nil:+ w.WriteHeader(http.StatusCreated)+ case errors.Is(err, ErrOutOfStock):+ http.Error(w, err.Error(), http.StatusConflict)+ case db.SQLState(err) == "55P03": // lock_not_available+ slog.WarnContext(r.Context(), "reserve stock: "+err.Error(), "cart_id", req.CartID)+ http.Error(w, "busy, try again", http.StatusServiceUnavailable)+ default:+ slog.ErrorContext(r.Context(), "reserve stock: "+err.Error(), "cart_id", req.CartID)+ http.Error(w, "internal error", http.StatusInternalServerError)+ }+}++// Reserve holds every line of the cart in one transaction: either all lines+// are held or none are.+func Reserve(ctx context.Context, pool db.Pool, req Request) (err error) {+ tx, err := pool.Begin(ctx)+ if err != nil {+ return err+ }+ defer func() {+ if err != nil {+ _ = tx.Rollback(ctx)+ }+ }()+ if _, err = tx.Exec(ctx, "SET LOCAL lock_timeout = '5s'"); err != nil {+ return err+ }+ skus := make([]string, len(req.Lines))+ qtys := make([]int, len(req.Lines))+ for i, l := range req.Lines {+ n, err := tx.Exec(ctx,+ "UPDATE inventory SET reserved = reserved + $1, updated_at = now() WHERE sku = $2 AND warehouse_id = $3 AND on_hand - reserved >= $1",+ l.Qty, l.SKU, l.WarehouseID)+ if err != nil {+ return err+ }+ if n == 0 {+ return fmt.Errorf("%s: %w", l.SKU, ErrOutOfStock)+ }+ skus[i], qtys[i] = l.SKU, l.Qty+ }+ if _, err = tx.Exec(ctx,+ "INSERT INTO reservations (cart_id, sku, warehouse_id, qty, expires_at) SELECT $1, unnest($2::text[]), $3, unnest($4::int[]), now() + interval '15 minutes'",+ req.CartID, skus, req.Lines[0].WarehouseID, qtys); err != nil {+ return err+ }+ return tx.Commit(ctx)+}++// Release serves POST /api/reservations/{id}/release: it drops a hold and+// gives the stock back.+func (h *Handler) Release(w http.ResponseWriter, r *http.Request) {+ ctx := r.Context()+ tx, err := h.Pool.Begin(ctx)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ defer func() { _ = tx.Rollback(ctx) }()+ // RETURNING sku, warehouse_id, qty feeds the UPDATE below (pgx scans+ // it in the production adapter).+ if _, err := tx.Exec(ctx, "DELETE FROM reservations WHERE id = $1 RETURNING sku, warehouse_id, qty", r.PathValue("id")); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if _, err := tx.Exec(ctx, "UPDATE inventory SET reserved = reserved - $1, updated_at = now() WHERE sku = $2 AND warehouse_id = $3"); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if err := tx.Commit(ctx); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.WriteHeader(http.StatusNoContent)+}diff --git a/cli/test/evals/testdata/repos/inventory-api-go/internal/reserve/reserve_test.go b/cli/test/evals/testdata/repos/inventory-api-go/internal/reserve/reserve_test.gonew file mode 100644index 00000000..81a1dc15--- /dev/null+++ b/cli/test/evals/testdata/repos/inventory-api-go/internal/reserve/reserve_test.go@@ -0,0 +1,74 @@+package reserve++import (+ "context"+ "errors"+ "strings"+ "testing"++ "example.com/inventory-api/internal/db"+)++type fakeTx struct {+ stmts []string+ args [][]any+ outOfStock map[string]bool+ committed bool+ rolledBack bool+}++func (t *fakeTx) Exec(_ context.Context, sql string, args ...any) (int64, error) {+ t.stmts = append(t.stmts, sql)+ t.args = append(t.args, args)+ if strings.HasPrefix(sql, "UPDATE inventory") && len(args) > 1 {+ if sku, ok := args[1].(string); ok && t.outOfStock[sku] {+ return 0, nil+ }+ }+ return 1, nil+}+func (t *fakeTx) Commit(context.Context) error { t.committed = true; return nil }+func (t *fakeTx) Rollback(context.Context) error { t.rolledBack = true; return nil }++type fakePool struct{ tx *fakeTx }++func (p *fakePool) Begin(context.Context) (db.Tx, error) { return p.tx, nil }++func TestReserveHoldsEveryLine(t *testing.T) {+ tx := &fakeTx{}+ err := Reserve(context.Background(), &fakePool{tx}, Request{CartID: "c_1", Lines: []Line{+ {SKU: "SKU-1", WarehouseID: 3, Qty: 1}, {SKU: "SKU-2", WarehouseID: 3, Qty: 2},+ }})+ if err != nil {+ t.Fatal(err)+ }+ if !tx.committed {+ t.Fatal("not committed")+ }+ held := map[string]bool{}+ for i, s := range tx.stmts {+ if strings.HasPrefix(s, "UPDATE inventory") {+ for _, a := range tx.args[i] {+ if v, ok := a.(string); ok {+ held[v] = true+ }+ }+ }+ }+ if !held["SKU-1"] || !held["SKU-2"] {+ t.Fatalf("lines not held: %v", tx.stmts)+ }+}++func TestReserveRollsBackWhenOutOfStock(t *testing.T) {+ tx := &fakeTx{outOfStock: map[string]bool{"SKU-2": true}}+ err := Reserve(context.Background(), &fakePool{tx}, Request{CartID: "c_1", Lines: []Line{+ {SKU: "SKU-1", WarehouseID: 3, Qty: 1}, {SKU: "SKU-2", WarehouseID: 3, Qty: 2},+ }})+ if !errors.Is(err, ErrOutOfStock) {+ t.Fatalf("err = %v", err)+ }+ if tx.committed || !tx.rolledBack {+ t.Fatal("expected a rollback")+ }+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/README.md b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/README.mdnew file mode 100644index 00000000..141d7c17--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/README.md@@ -0,0 +1,17 @@+# invoicing-api++Invoices for the billing product: create, list and show invoices, and the+monthly invoice export accounting downloads at month end. Go, `net/http`,+PostgreSQL through `database/sql`. Traces, logs and metrics go to Traceway+over OTLP (`OTEL_EXPORTER_OTLP_ENDPOINT`).++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/invoices`, `GET /api/invoices/{id}`, `POST /api/invoices`+- `GET /api/exports/invoices?month=2026-09` – every invoice of the month with+ its lines as CSVdiff --git a/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/invoices.go b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/invoices.gonew file mode 100644index 00000000..7a2274cc--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/invoices.go@@ -0,0 +1,62 @@+// Package exports serves the accounting exports.+package exports++import (+ "encoding/csv"+ "net/http"+ "strconv"+ "time"+)++// Handlers serves /api/exports.+type Handlers struct{ store Store }++// New returns the export handlers.+func New(s Store) *Handlers { return &Handlers{store: s} }++// Invoices serves GET /api/exports/invoices?month=YYYY-MM: one CSV row per+// invoice line.+func (h *Handlers) Invoices(w http.ResponseWriter, r *http.Request) {+ orgID, err := strconv.ParseInt(r.Header.Get("X-Org-Id"), 10, 64)+ if err != nil {+ http.Error(w, "unauthorized", http.StatusUnauthorized)+ return+ }+ from, err := time.Parse("2006-01", r.URL.Query().Get("month"))+ if err != nil {+ http.Error(w, "month must look like 2026-09", http.StatusBadRequest)+ return+ }+ to := from.AddDate(0, 1, 0)+ ctx := r.Context()+ invs, err := h.store.InvoicesWithLines(ctx, orgID, from, to)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.Header().Set("Content-Type", "text/csv")+ w.Header().Set("Content-Disposition", `attachment; filename="invoices-`+from.Format("2006-01")+`.csv"`)+ cw := csv.NewWriter(w)+ _ = cw.Write([]string{"invoice", "issued_at", "customer", "sku", "description", "qty", "unit_price", "invoice_total"})+ for _, inv := range invs {+ for _, l := range inv.Lines {+ _ = cw.Write([]string{+ inv.Number, inv.IssuedAt.UTC().Format(time.DateOnly), inv.CustomerName, l.SKU, l.Description,+ strconv.FormatInt(l.Qty, 10), cents(l.UnitPriceCents), cents(inv.TotalCents),+ })+ }+ }+ cw.Flush()+}++func cents(c int64) string {+ sign := ""+ if c < 0 {+ sign, c = "-", -c+ }+ frac := strconv.FormatInt(c%100, 10)+ if len(frac) == 1 {+ frac = "0" + frac+ }+ return sign + strconv.FormatInt(c/100, 10) + "." + frac+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/invoices_test.go b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/invoices_test.gonew file mode 100644index 00000000..1becd591--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/invoices_test.go@@ -0,0 +1,44 @@+package exports++import (+ "context"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+ "time"+)++type fakeStore struct {+ invoices []Invoice+}++func (f fakeStore) InvoicesWithLines(context.Context, int64, time.Time, time.Time) ([]Invoice, error) {+ out := make([]Invoice, len(f.invoices))+ copy(out, f.invoices)+ return out, nil+}++func TestInvoicesExport(t *testing.T) {+ st := fakeStore{+ invoices: []Invoice{+ {ID: 1, Number: "INV-1001", IssuedAt: time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC), CustomerName: "Acme", TotalCents: 11900,+ Lines: []Line{{SKU: "A-1", Description: "Seat", Qty: 1, UnitPriceCents: 10000}}},+ {ID: 2, Number: "INV-1002", IssuedAt: time.Date(2026, 9, 3, 0, 0, 0, 0, time.UTC), CustomerName: "Globex", TotalCents: 2380,+ Lines: []Line{{SKU: "B-1", Description: "Add-on", Qty: 2, UnitPriceCents: 1000}}},+ },+ }+ req := httptest.NewRequest(http.MethodGet, "/api/exports/invoices?month=2026-09", nil)+ req.Header.Set("X-Org-Id", "3")+ rec := httptest.NewRecorder()+ New(st).Invoices(rec, req)+ if rec.Code != http.StatusOK {+ t.Fatalf("status %d", rec.Code)+ }+ body := rec.Body.String()+ for _, want := range []string{"INV-1001,2026-09-02,Acme,A-1,Seat,1,100.00,119.00", "INV-1002,2026-09-03,Globex,B-1,Add-on,2,10.00,23.80"} {+ if !strings.Contains(body, want) {+ t.Errorf("missing row %q in:\n%s", want, body)+ }+ }+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/store.go b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/store.gonew file mode 100644index 00000000..dea0ee6f--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/.history/01/internal/exports/store.go@@ -0,0 +1,62 @@+package exports++import (+ "context"+ "database/sql"+ "time"+)++// Invoice is one exported invoice with its lines.+type Invoice struct {+ ID int64+ Number string+ IssuedAt time.Time+ CustomerName string+ TotalCents int64+ Lines []Line+}++// Line is one invoice line.+type Line struct {+ SKU string+ Description string+ Qty int64+ UnitPriceCents int64+}++// Store is the data the export reads.+type Store interface {+ // InvoicesWithLines returns the organization's invoices issued in+ // [from, to), each with its lines, in issue order.+ InvoicesWithLines(ctx context.Context, orgID int64, from, to time.Time) ([]Invoice, error)+}++// SQLStore implements Store on PostgreSQL.+type SQLStore struct{ db *sql.DB }++// NewStore returns a Store backed by db.+func NewStore(db *sql.DB) *SQLStore { return &SQLStore{db: db} }++// InvoicesWithLines loads the month in one JOIN.+func (s *SQLStore) InvoicesWithLines(ctx context.Context, orgID int64, from, to time.Time) ([]Invoice, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT i.id, i.number, i.issued_at, i.customer_name, i.total_cents, l.sku, l.description, l.qty, l.unit_price_cents FROM invoices i JOIN invoice_lines l ON l.invoice_id = i.id WHERE i.org_id = $1 AND i.issued_at >= $2 AND i.issued_at < $3 ORDER BY i.issued_at, i.id`, orgID, from, to)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Invoice+ for rows.Next() {+ var inv Invoice+ var l Line+ if err := rows.Scan(&inv.ID, &inv.Number, &inv.IssuedAt, &inv.CustomerName, &inv.TotalCents, &l.SKU, &l.Description, &l.Qty, &l.UnitPriceCents); err != nil {+ return nil, err+ }+ if n := len(out); n > 0 && out[n-1].ID == inv.ID {+ out[n-1].Lines = append(out[n-1].Lines, l)+ continue+ }+ inv.Lines = []Line{l}+ out = append(out, inv)+ }+ return out, rows.Err()+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/README.md b/cli/test/evals/testdata/repos/invoicing-api-go/README.mdnew file mode 100644index 00000000..5385e953--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/README.md@@ -0,0 +1,17 @@+# invoicing-api++Invoices for the billing product: create, list and show invoices, and the+monthly invoice export accounting downloads at month end. Go, `net/http`,+PostgreSQL through `database/sql`. Traces, logs and metrics go to Traceway+over OTLP (`OTEL_EXPORTER_OTLP_ENDPOINT`).++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/invoices`, `GET /api/invoices/{id}`, `POST /api/invoices`+- `GET /api/exports/invoices?month=2026-09` – every invoice of the month with+ its lines and tax breakdown as CSVdiff --git a/cli/test/evals/testdata/repos/invoicing-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/invoicing-api-go/cmd/server/main.gonew file mode 100644index 00000000..e6f62525--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/cmd/server/main.go@@ -0,0 +1,43 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"++ "example.com/invoicing-api/internal/exports"+ "example.com/invoicing-api/internal/invoices"+)++func main() {+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ inv := invoices.New(db)+ exp := exports.New(exports.NewStore(db))+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/invoices", inv.List)+ mux.HandleFunc("GET /api/invoices/{id}", inv.Get)+ mux.HandleFunc("POST /api/invoices", inv.Create)+ mux.HandleFunc("GET /api/exports/invoices", exp.Invoices)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, withTracing(mux)); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}++// withTracing is where the OpenTelemetry HTTP middleware is installed+// (otelhttp.NewHandler in production builds).+func withTracing(h http.Handler) http.Handler { return h }diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/go.mod b/cli/test/evals/testdata/repos/invoicing-api-go/go.modnew file mode 100644index 00000000..b0afe2c3--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/invoicing-api++go 1.26diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/history.yaml b/cli/test/evals/testdata/repos/invoicing-api-go/history.yamlnew file mode 100644index 00000000..4ba31975--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: 'v4.7.3: load the export''s invoices and lines in one query'+ age: 520h+ author: Priya Raman <[email protected]>+head:+ message: 'v4.8.0: tax breakdown per invoice in the monthly export'+ age: 26h25m+ author: Tomás Ortega <[email protected]>diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/invoices.go b/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/invoices.gonew file mode 100644index 00000000..f74d0553--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/invoices.go@@ -0,0 +1,80 @@+// Package exports serves the accounting exports.+package exports++import (+ "encoding/csv"+ "net/http"+ "strconv"+ "strings"+ "time"+)++// Handlers serves /api/exports.+type Handlers struct{ store Store }++// New returns the export handlers.+func New(s Store) *Handlers { return &Handlers{store: s} }++// Invoices serves GET /api/exports/invoices?month=YYYY-MM: one CSV row per+// invoice line, with the invoice's tax breakdown in the last column.+func (h *Handlers) Invoices(w http.ResponseWriter, r *http.Request) {+ orgID, err := strconv.ParseInt(r.Header.Get("X-Org-Id"), 10, 64)+ if err != nil {+ http.Error(w, "unauthorized", http.StatusUnauthorized)+ return+ }+ from, err := time.Parse("2006-01", r.URL.Query().Get("month"))+ if err != nil {+ http.Error(w, "month must look like 2026-09", http.StatusBadRequest)+ return+ }+ to := from.AddDate(0, 1, 0)+ ctx := r.Context()+ invs, err := h.store.InvoicesWithLines(ctx, orgID, from, to)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ for i := range invs {+ taxes, err := h.store.TaxLines(ctx, invs[i].ID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ invs[i].Taxes = taxes+ }+ w.Header().Set("Content-Type", "text/csv")+ w.Header().Set("Content-Disposition", `attachment; filename="invoices-`+from.Format("2006-01")+`.csv"`)+ cw := csv.NewWriter(w)+ _ = cw.Write([]string{"invoice", "issued_at", "customer", "sku", "description", "qty", "unit_price", "invoice_total", "taxes"})+ for _, inv := range invs {+ taxes := formatTaxes(inv.Taxes)+ for _, l := range inv.Lines {+ _ = cw.Write([]string{+ inv.Number, inv.IssuedAt.UTC().Format(time.DateOnly), inv.CustomerName, l.SKU, l.Description,+ strconv.FormatInt(l.Qty, 10), cents(l.UnitPriceCents), cents(inv.TotalCents), taxes,+ })+ }+ }+ cw.Flush()+}++func formatTaxes(ts []TaxLine) string {+ parts := make([]string, 0, len(ts))+ for _, t := range ts {+ parts = append(parts, t.RateCode+"="+cents(t.AmountCents))+ }+ return strings.Join(parts, ";")+}++func cents(c int64) string {+ sign := ""+ if c < 0 {+ sign, c = "-", -c+ }+ frac := strconv.FormatInt(c%100, 10)+ if len(frac) == 1 {+ frac = "0" + frac+ }+ return sign + strconv.FormatInt(c/100, 10) + "." + frac+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/invoices_test.go b/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/invoices_test.gonew file mode 100644index 00000000..f6dc5a61--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/invoices_test.go@@ -0,0 +1,48 @@+package exports++import (+ "context"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+ "time"+)++type fakeStore struct {+ invoices []Invoice+ taxes map[int64][]TaxLine+}++func (f fakeStore) InvoicesWithLines(context.Context, int64, time.Time, time.Time) ([]Invoice, error) {+ out := make([]Invoice, len(f.invoices))+ copy(out, f.invoices)+ return out, nil+}++func (f fakeStore) TaxLines(_ context.Context, id int64) ([]TaxLine, error) { return f.taxes[id], nil }++func TestInvoicesExport(t *testing.T) {+ st := fakeStore{+ invoices: []Invoice{+ {ID: 1, Number: "INV-1001", IssuedAt: time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC), CustomerName: "Acme", TotalCents: 11900,+ Lines: []Line{{SKU: "A-1", Description: "Seat", Qty: 1, UnitPriceCents: 10000}}},+ {ID: 2, Number: "INV-1002", IssuedAt: time.Date(2026, 9, 3, 0, 0, 0, 0, time.UTC), CustomerName: "Globex", TotalCents: 2380,+ Lines: []Line{{SKU: "B-1", Description: "Add-on", Qty: 2, UnitPriceCents: 1000}}},+ },+ taxes: map[int64][]TaxLine{1: {{RateCode: "VAT19", RateBps: 1900, AmountCents: 1900}}, 2: {{RateCode: "VAT19", RateBps: 1900, AmountCents: 380}}},+ }+ req := httptest.NewRequest(http.MethodGet, "/api/exports/invoices?month=2026-09", nil)+ req.Header.Set("X-Org-Id", "3")+ rec := httptest.NewRecorder()+ New(st).Invoices(rec, req)+ if rec.Code != http.StatusOK {+ t.Fatalf("status %d", rec.Code)+ }+ body := rec.Body.String()+ for _, want := range []string{"INV-1001,2026-09-02,Acme,A-1,Seat,1,100.00,119.00,VAT19=19.00", "INV-1002,2026-09-03,Globex,B-1,Add-on,2,10.00,23.80,VAT19=3.80"} {+ if !strings.Contains(body, want) {+ t.Errorf("missing row %q in:\n%s", want, body)+ }+ }+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/store.go b/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/store.gonew file mode 100644index 00000000..f24e203c--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/internal/exports/store.go@@ -0,0 +1,90 @@+package exports++import (+ "context"+ "database/sql"+ "time"+)++// Invoice is one exported invoice with its lines and tax breakdown.+type Invoice struct {+ ID int64+ Number string+ IssuedAt time.Time+ CustomerName string+ TotalCents int64+ Lines []Line+ Taxes []TaxLine+}++// Line is one invoice line.+type Line struct {+ SKU string+ Description string+ Qty int64+ UnitPriceCents int64+}++// TaxLine is one tax amount of an invoice.+type TaxLine struct {+ RateCode string+ RateBps int64+ AmountCents int64+}++// Store is the data the export reads.+type Store interface {+ // InvoicesWithLines returns the organization's invoices issued in+ // [from, to), each with its lines, in issue order.+ InvoicesWithLines(ctx context.Context, orgID int64, from, to time.Time) ([]Invoice, error)+ // TaxLines returns one invoice's tax breakdown.+ TaxLines(ctx context.Context, invoiceID int64) ([]TaxLine, error)+}++// SQLStore implements Store on PostgreSQL.+type SQLStore struct{ db *sql.DB }++// NewStore returns a Store backed by db.+func NewStore(db *sql.DB) *SQLStore { return &SQLStore{db: db} }++// InvoicesWithLines loads the month in one JOIN.+func (s *SQLStore) InvoicesWithLines(ctx context.Context, orgID int64, from, to time.Time) ([]Invoice, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT i.id, i.number, i.issued_at, i.customer_name, i.total_cents, l.sku, l.description, l.qty, l.unit_price_cents FROM invoices i JOIN invoice_lines l ON l.invoice_id = i.id WHERE i.org_id = $1 AND i.issued_at >= $2 AND i.issued_at < $3 ORDER BY i.issued_at, i.id`, orgID, from, to)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Invoice+ for rows.Next() {+ var inv Invoice+ var l Line+ if err := rows.Scan(&inv.ID, &inv.Number, &inv.IssuedAt, &inv.CustomerName, &inv.TotalCents, &l.SKU, &l.Description, &l.Qty, &l.UnitPriceCents); err != nil {+ return nil, err+ }+ if n := len(out); n > 0 && out[n-1].ID == inv.ID {+ out[n-1].Lines = append(out[n-1].Lines, l)+ continue+ }+ inv.Lines = []Line{l}+ out = append(out, inv)+ }+ return out, rows.Err()+}++// TaxLines loads one invoice's tax breakdown.+func (s *SQLStore) TaxLines(ctx context.Context, invoiceID int64) ([]TaxLine, error) {+ rows, err := s.db.QueryContext(ctx, `SELECT rate_code, rate_bps, amount_cents FROM invoice_tax_lines WHERE invoice_id = $1`, invoiceID)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []TaxLine+ for rows.Next() {+ var t TaxLine+ if err := rows.Scan(&t.RateCode, &t.RateBps, &t.AmountCents); err != nil {+ return nil, err+ }+ out = append(out, t)+ }+ return out, rows.Err()+}diff --git a/cli/test/evals/testdata/repos/invoicing-api-go/internal/invoices/invoices.go b/cli/test/evals/testdata/repos/invoicing-api-go/internal/invoices/invoices.gonew file mode 100644index 00000000..d550e18f--- /dev/null+++ b/cli/test/evals/testdata/repos/invoicing-api-go/internal/invoices/invoices.go@@ -0,0 +1,89 @@+// Package invoices serves the invoice CRUD routes.+package invoices++import (+ "database/sql"+ "encoding/json"+ "net/http"+ "strconv"+ "time"+)++// Handlers serves /api/invoices.+type Handlers struct{ db *sql.DB }++// New returns the invoice handlers.+func New(db *sql.DB) *Handlers { return &Handlers{db: db} }++type invoice struct {+ ID int64 `json:"id"`+ Number string `json:"number"`+ IssuedAt time.Time `json:"issuedAt"`+ CustomerName string `json:"customerName"`+ TotalCents int64 `json:"totalCents"`+ Status string `json:"status"`+}++// List serves GET /api/invoices: the latest 50 of the organization.+func (h *Handlers) List(w http.ResponseWriter, r *http.Request) {+ orgID, err := strconv.ParseInt(r.Header.Get("X-Org-Id"), 10, 64)+ if err != nil {+ http.Error(w, "unauthorized", http.StatusUnauthorized)+ return+ }+ rows, err := h.db.QueryContext(r.Context(), `SELECT id, number, issued_at, customer_name, total_cents, status FROM invoices WHERE org_id = $1 ORDER BY issued_at DESC LIMIT 50`, orgID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ defer rows.Close()+ out := []invoice{}+ for rows.Next() {+ var i invoice+ if err := rows.Scan(&i.ID, &i.Number, &i.IssuedAt, &i.CustomerName, &i.TotalCents, &i.Status); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ out = append(out, i)+ }+ _ = json.NewEncoder(w).Encode(out)+}++// Get serves GET /api/invoices/{id}.+func (h *Handlers) Get(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad invoice id", http.StatusBadRequest)+ return+ }+ var i invoice+ err = h.db.QueryRowContext(r.Context(), `SELECT id, number, issued_at, customer_name, total_cents, status FROM invoices WHERE id = $1`, id).+ Scan(&i.ID, &i.Number, &i.IssuedAt, &i.CustomerName, &i.TotalCents, &i.Status)+ if err == sql.ErrNoRows {+ http.NotFound(w, r)+ return+ }+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ _ = json.NewEncoder(w).Encode(i)+}++// Create serves POST /api/invoices.+func (h *Handlers) Create(w http.ResponseWriter, r *http.Request) {+ var in invoice+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ http.Error(w, "invalid body", http.StatusBadRequest)+ return+ }+ orgID, _ := strconv.ParseInt(r.Header.Get("X-Org-Id"), 10, 64)+ err := h.db.QueryRowContext(r.Context(), `INSERT INTO invoices (org_id, number, issued_at, customer_name, total_cents, status) VALUES ($1, $2, $3, $4, $5, 'open') RETURNING id`,+ orgID, in.Number, in.IssuedAt, in.CustomerName, in.TotalCents).Scan(&in.ID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.WriteHeader(http.StatusCreated)+ _ = json.NewEncoder(w).Encode(in)+}diff --git a/cli/test/evals/testdata/repos/ledger-api-go/README.md b/cli/test/evals/testdata/repos/ledger-api-go/README.mdnew file mode 100644index 00000000..0c720f29--- /dev/null+++ b/cli/test/evals/testdata/repos/ledger-api-go/README.md@@ -0,0 +1,19 @@+# ledger-api++General ledger service: account balances, journal entries and the finance+team's quarterly ledger export. Go, `net/http`, PostgreSQL through pgx.+Traces, logs and metrics go to Traceway over OTLP+(`OTEL_EXPORTER_OTLP_ENDPOINT`).++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/accounts/{id}/balance` – current balance of one account+- `GET /api/accounts/{id}/entries` – the latest 50 entries of one account+- `POST /api/journal-entries` – post a journal entry+- `GET /api/reports/export?from=&to=` – every ledger entry of the organization+ in the range, as CSV (finance pulls one quarter at a time)diff --git a/cli/test/evals/testdata/repos/ledger-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/ledger-api-go/cmd/server/main.gonew file mode 100644index 00000000..3ae209a5--- /dev/null+++ b/cli/test/evals/testdata/repos/ledger-api-go/cmd/server/main.go@@ -0,0 +1,43 @@+package main++import (+ "database/sql"+ "log/slog"+ "net/http"+ "os"++ "example.com/ledger-api/internal/accounts"+ "example.com/ledger-api/internal/reports"+)++func main() {+ db, err := sql.Open("pgx", os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("open database", "err", err)+ os.Exit(1)+ }+ acc := accounts.New(db)+ exp := reports.NewExporter(reports.NewStore(db))+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/accounts/{id}/balance", acc.Balance)+ mux.HandleFunc("GET /api/accounts/{id}/entries", acc.Entries)+ mux.HandleFunc("POST /api/journal-entries", acc.Post)+ mux.HandleFunc("GET /api/reports/export", exp.Export)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, withTracing(mux)); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}++// withTracing is where the OpenTelemetry HTTP middleware is installed+// (otelhttp.NewHandler in production builds).+func withTracing(h http.Handler) http.Handler { return h }diff --git a/cli/test/evals/testdata/repos/ledger-api-go/go.mod b/cli/test/evals/testdata/repos/ledger-api-go/go.modnew file mode 100644index 00000000..4e9214b9--- /dev/null+++ b/cli/test/evals/testdata/repos/ledger-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/ledger-api++go 1.26diff --git a/cli/test/evals/testdata/repos/ledger-api-go/internal/accounts/accounts.go b/cli/test/evals/testdata/repos/ledger-api-go/internal/accounts/accounts.gonew file mode 100644index 00000000..fb8bf4ab--- /dev/null+++ b/cli/test/evals/testdata/repos/ledger-api-go/internal/accounts/accounts.go@@ -0,0 +1,99 @@+// Package accounts serves balances and journal entries.+package accounts++import (+ "database/sql"+ "encoding/json"+ "net/http"+ "strconv"+ "time"+)++// Handlers serves the account routes.+type Handlers struct{ db *sql.DB }++// New returns the account handlers.+func New(db *sql.DB) *Handlers { return &Handlers{db: db} }++// Balance serves GET /api/accounts/{id}/balance.+func (h *Handlers) Balance(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad account id", http.StatusBadRequest)+ return+ }+ var out struct {+ BalanceCents int64 `json:"balanceCents"`+ Currency string `json:"currency"`+ UpdatedAt time.Time `json:"updatedAt"`+ }+ err = h.db.QueryRowContext(r.Context(), `SELECT balance_cents, currency, updated_at FROM account_balances WHERE account_id = $1`, id).+ Scan(&out.BalanceCents, &out.Currency, &out.UpdatedAt)+ if err == sql.ErrNoRows {+ http.NotFound(w, r)+ return+ }+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ _ = json.NewEncoder(w).Encode(out)+}++// Entries serves GET /api/accounts/{id}/entries: the latest 50.+func (h *Handlers) Entries(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad account id", http.StatusBadRequest)+ return+ }+ rows, err := h.db.QueryContext(r.Context(), `SELECT id, posted_at, description, amount_cents FROM ledger_entries WHERE account_id = $1 ORDER BY posted_at DESC LIMIT 50`, id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ defer rows.Close()+ type entry struct {+ ID int64 `json:"id"`+ PostedAt time.Time `json:"postedAt"`+ Description string `json:"description"`+ AmountCents int64 `json:"amountCents"`+ }+ out := []entry{}+ for rows.Next() {+ var e entry+ if err := rows.Scan(&e.ID, &e.PostedAt, &e.Description, &e.AmountCents); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ out = append(out, e)+ }+ _ = json.NewEncoder(w).Encode(out)+}++// Post serves POST /api/journal-entries.+func (h *Handlers) Post(w http.ResponseWriter, r *http.Request) {+ var in struct {+ OrgID int64 `json:"orgId"`+ AccountID int64 `json:"accountId"`+ PostedAt time.Time `json:"postedAt"`+ Description string `json:"description"`+ AmountCents int64 `json:"amountCents"`+ Currency string `json:"currency"`+ }+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ http.Error(w, "invalid body", http.StatusBadRequest)+ return+ }+ ctx := r.Context()+ if _, err := h.db.ExecContext(ctx, `INSERT INTO ledger_entries (org_id, account_id, posted_at, description, amount_cents, currency) VALUES ($1, $2, $3, $4, $5, $6)`,+ in.OrgID, in.AccountID, in.PostedAt, in.Description, in.AmountCents, in.Currency); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ if _, err := h.db.ExecContext(ctx, `UPDATE account_balances SET balance_cents = balance_cents + $2, updated_at = now() WHERE account_id = $1`, in.AccountID, in.AmountCents); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.WriteHeader(http.StatusCreated)+}diff --git a/cli/test/evals/testdata/repos/ledger-api-go/internal/reports/export.go b/cli/test/evals/testdata/repos/ledger-api-go/internal/reports/export.gonew file mode 100644index 00000000..5cfd5316--- /dev/null+++ b/cli/test/evals/testdata/repos/ledger-api-go/internal/reports/export.go@@ -0,0 +1,101 @@+// Package reports serves the finance exports.+package reports++import (+ "context"+ "database/sql"+ "encoding/csv"+ "net/http"+ "strconv"+ "time"+)++// Entry is one ledger line as exported.+type Entry struct {+ ID int64+ PostedAt time.Time+ AccountCode string+ Counterparty string+ Description string+ AmountCents int64+ Currency string+}++// Source yields the entries of an organization in a range, in posting order.+type Source interface {+ Entries(ctx context.Context, orgID int64, from, to time.Time, each func(Entry) error) error+}++// Store reads entries from PostgreSQL.+type Store struct{ db *sql.DB }++// NewStore returns a Source backed by db.+func NewStore(db *sql.DB) *Store { return &Store{db: db} }++// Entries streams the range with one query; rows are written as they are+// scanned, so memory stays flat whatever the range holds.+func (s *Store) Entries(ctx context.Context, orgID int64, from, to time.Time, each func(Entry) error) error {+ rows, err := s.db.QueryContext(ctx, `SELECT id, posted_at, account_code, counterparty, description, amount_cents, currency FROM ledger_entries WHERE org_id = $1 AND posted_at >= $2 AND posted_at < $3 ORDER BY posted_at`, orgID, from, to)+ if err != nil {+ return err+ }+ defer rows.Close()+ for rows.Next() {+ var e Entry+ if err := rows.Scan(&e.ID, &e.PostedAt, &e.AccountCode, &e.Counterparty, &e.Description, &e.AmountCents, &e.Currency); err != nil {+ return err+ }+ if err := each(e); err != nil {+ return err+ }+ }+ return rows.Err()+}++// Exporter serves GET /api/reports/export.+type Exporter struct{ src Source }++// NewExporter returns the export handler.+func NewExporter(src Source) *Exporter { return &Exporter{src: src} }++// Export writes the organization's entries between from and to as CSV.+func (x *Exporter) Export(w http.ResponseWriter, r *http.Request) {+ orgID, err := strconv.ParseInt(r.Header.Get("X-Org-Id"), 10, 64)+ if err != nil {+ http.Error(w, "unauthorized", http.StatusUnauthorized)+ return+ }+ from, err1 := time.Parse(time.DateOnly, r.URL.Query().Get("from"))+ to, err2 := time.Parse(time.DateOnly, r.URL.Query().Get("to"))+ if err1 != nil || err2 != nil || !to.After(from) {+ http.Error(w, "from and to must be dates, from before to", http.StatusBadRequest)+ return+ }+ w.Header().Set("Content-Type", "text/csv")+ w.Header().Set("Content-Disposition", `attachment; filename="ledger.csv"`)+ cw := csv.NewWriter(w)+ _ = cw.Write([]string{"id", "posted_at", "account", "counterparty", "description", "amount", "currency"})+ err = x.src.Entries(r.Context(), orgID, from, to, func(e Entry) error {+ return cw.Write([]string{+ strconv.FormatInt(e.ID, 10), e.PostedAt.UTC().Format(time.RFC3339), e.AccountCode, e.Counterparty,+ e.Description, formatCents(e.AmountCents), e.Currency,+ })+ })+ cw.Flush()+ if err != nil {+ // Headers are already sent; the truncated file fails the client's checksum.+ return+ }+}++func formatCents(c int64) string {+ sign := ""+ if c < 0 {+ sign, c = "-", -c+ }+ frac := strconv.FormatInt(c%100, 10)+ if len(frac) == 1 {+ frac = "0" + frac+ }+ return sign + strconv.FormatInt(c/100, 10) + "." + frac+}diff --git a/cli/test/evals/testdata/repos/ledger-api-go/internal/reports/export_test.go b/cli/test/evals/testdata/repos/ledger-api-go/internal/reports/export_test.gonew file mode 100644index 00000000..0e36e916--- /dev/null+++ b/cli/test/evals/testdata/repos/ledger-api-go/internal/reports/export_test.go@@ -0,0 +1,55 @@+package reports++import (+ "context"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"+ "time"+)++type fakeSource []Entry++func (f fakeSource) Entries(_ context.Context, _ int64, from, to time.Time, each func(Entry) error) error {+ for _, e := range f {+ if !e.PostedAt.Before(from) && e.PostedAt.Before(to) {+ if err := each(e); err != nil {+ return err+ }+ }+ }+ return nil+}++func TestExportWritesCSV(t *testing.T) {+ src := fakeSource{+ {ID: 1, PostedAt: time.Date(2026, 7, 2, 9, 0, 0, 0, time.UTC), AccountCode: "4000", Counterparty: "Acme", Description: "Invoice 1001", AmountCents: 125050, Currency: "EUR"},+ {ID: 2, PostedAt: time.Date(2026, 7, 3, 9, 0, 0, 0, time.UTC), AccountCode: "1200", Counterparty: "Acme", Description: "Refund", AmountCents: -999, Currency: "EUR"},+ {ID: 3, PostedAt: time.Date(2026, 10, 1, 9, 0, 0, 0, time.UTC), AccountCode: "4000", Counterparty: "Other", Description: "Next quarter", AmountCents: 100, Currency: "EUR"},+ }+ req := httptest.NewRequest(http.MethodGet, "/api/reports/export?from=2026-07-01&to=2026-10-01", nil)+ req.Header.Set("X-Org-Id", "7")+ rec := httptest.NewRecorder()+ NewExporter(src).Export(rec, req)+ if rec.Code != http.StatusOK {+ t.Fatalf("status %d", rec.Code)+ }+ lines := strings.Split(strings.TrimSpace(rec.Body.String()), "\n")+ if len(lines) != 3 {+ t.Fatalf("got %d lines, want header + 2:\n%s", len(lines), rec.Body.String())+ }+ if !strings.Contains(lines[1], "1250.50") || !strings.Contains(lines[2], "-9.99") {+ t.Errorf("amounts not formatted: %q", lines[1:])+ }+}++func TestExportRejectsBadRange(t *testing.T) {+ req := httptest.NewRequest(http.MethodGet, "/api/reports/export?from=2026-10-01&to=2026-07-01", nil)+ req.Header.Set("X-Org-Id", "7")+ rec := httptest.NewRecorder()+ NewExporter(fakeSource{}).Export(rec, req)+ if rec.Code != http.StatusBadRequest {+ t.Fatalf("status %d, want 400", rec.Code)+ }+}diff --git a/cli/test/evals/testdata/repos/listings-api-go/.history/01/deploy/production.env b/cli/test/evals/testdata/repos/listings-api-go/.history/01/deploy/production.envnew file mode 100644index 00000000..acf049b7--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/.history/01/deploy/production.env@@ -0,0 +1,10 @@+# Production configuration for listings-api (rendered into the ConfigMap).+LISTEN_ADDR=:8080+DATABASE_MAX_CONNS=20+REDIS_ADDR=listings-cache.internal:6379+SEARCH_PAGE_SIZE=44+SEARCH_CACHE_TTL=30s+LOG_LEVEL=info+LOG_FORMAT=json+OTEL_SERVICE_NAME=listings-api+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufdiff --git a/cli/test/evals/testdata/repos/listings-api-go/README.md b/cli/test/evals/testdata/repos/listings-api-go/README.mdnew file mode 100644index 00000000..37c45903--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/README.md@@ -0,0 +1,21 @@+# listings-api++Marketplace listings: category search, listing pages and view counts. Go,+`net/http`, PostgreSQL through pgx, Redis for the search cache. Traces and+logs go to Traceway over OTLP (`OTEL_EXPORTER_OTLP_ENDPOINT`); logs are+written with `log/slog` through the OpenTelemetry slog bridge.++Configuration comes from the environment; production values live in+`deploy/production.env`, which the deploy pipeline turns into the pods'+ConfigMap. A change to it rolls the deployment like any release.++```+go run ./cmd/server+```++Routes:++- `GET /api/listings?category=` – active listings of a category, ranked+- `GET /api/listings/{id}` – one listing with its images+- `POST /api/listings/{id}/views` – count a view+- `GET /api/categories` – the category tree (cached)diff --git a/cli/test/evals/testdata/repos/listings-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/listings-api-go/cmd/server/main.gonew file mode 100644index 00000000..1a1ffaf1--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/cmd/server/main.go@@ -0,0 +1,92 @@+package main++import (+ "context"+ "fmt"+ "log/slog"+ "net/http"+ "os"+ "strconv"+ "time"++ "github.com/exaring/otelpgx"+ "github.com/jackc/pgx/v5/pgxpool"+ "github.com/redis/go-redis/extra/redisotel/v9"+ "github.com/redis/go-redis/v9"+ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"++ "github.com/acme/listings-api/internal/listings"+ "github.com/acme/listings-api/internal/logging"+)++func main() {+ ctx := context.Background()+ log := logging.New(os.Getenv("LOG_LEVEL"))+ slog.SetDefault(log)++ shutdown, err := setupTelemetry(ctx)+ if err != nil {+ log.Error("telemetry setup failed", "err", err)+ os.Exit(1)+ }+ defer shutdown(ctx)++ cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))+ if err != nil {+ log.Error("database config", "err", err)+ os.Exit(1)+ }+ cfg.ConnConfig.Tracer = otelpgx.NewTracer()+ cfg.MaxConns = int32(envInt("DATABASE_MAX_CONNS", 20))+ db, err := pgxpool.NewWithConfig(ctx, cfg)+ if err != nil {+ log.Error("database connect", "err", err)+ os.Exit(1)+ }+ cache := redis.NewClient(&redis.Options{Addr: os.Getenv("REDIS_ADDR")})+ _ = redisotel.InstrumentTracing(cache)++ ttl, _ := time.ParseDuration(os.Getenv("SEARCH_CACHE_TTL"))+ repo := listings.NewRepo(db, cache, log, envInt("SEARCH_PAGE_SIZE", 44), ttl)+ mux := http.NewServeMux()+ listings.NewHandler(repo, log).Routes(mux)++ addr := os.Getenv("LISTEN_ADDR")+ log.Info("starting listings-api", "version", version, "log.level", os.Getenv("LOG_LEVEL"), "addr", addr)+ srv := &http.Server{Addr: addr, Handler: otelhttp.NewHandler(accessLog(log, mux), "listings-api"), ReadHeaderTimeout: 10 * time.Second}+ if err := srv.ListenAndServe(); err != nil {+ log.Error("serve", "err", err)+ os.Exit(1)+ }+}++var version = "dev"++// accessLog writes one line per request, inside the request's trace.+func accessLog(log *slog.Logger, next http.Handler) http.Handler {+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ start := time.Now()+ rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}+ next.ServeHTTP(rec, r)+ ms := time.Since(start).Milliseconds()+ log.InfoContext(r.Context(), fmt.Sprintf("%s %s %d %dms", r.Method, r.URL.Path, rec.status, ms),+ "http.request.method", r.Method, "url.path", r.URL.Path, "http.response.status_code", rec.status, "duration_ms", ms)+ })+}++type statusRecorder struct {+ http.ResponseWriter+ status int+}++func (s *statusRecorder) WriteHeader(code int) {+ s.status = code+ s.ResponseWriter.WriteHeader(code)+}++func envInt(name string, def int) int {+ if v, err := strconv.Atoi(os.Getenv(name)); err == nil {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/listings-api-go/cmd/server/telemetry.go b/cli/test/evals/testdata/repos/listings-api-go/cmd/server/telemetry.gonew file mode 100644index 00000000..d80adeeb--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/cmd/server/telemetry.go@@ -0,0 +1,33 @@+package main++import (+ "context"+ "errors"++ "go.opentelemetry.io/otel"+ "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"+ "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"+ "go.opentelemetry.io/otel/log/global"+ sdklog "go.opentelemetry.io/otel/sdk/log"+ sdktrace "go.opentelemetry.io/otel/sdk/trace"+)++// setupTelemetry installs the OTLP trace and log pipelines. The endpoint and+// resource come from the standard OTEL_* environment variables.+func setupTelemetry(ctx context.Context) (func(context.Context) error, error) {+ te, err := otlptracehttp.New(ctx)+ if err != nil {+ return nil, err+ }+ tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(te))+ otel.SetTracerProvider(tp)++ le, err := otlploghttp.New(ctx)+ if err != nil {+ return nil, err+ }+ lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(sdklog.NewBatchProcessor(le)))+ global.SetLoggerProvider(lp)++ return func(ctx context.Context) error { return errors.Join(tp.Shutdown(ctx), lp.Shutdown(ctx)) }, nil+}diff --git a/cli/test/evals/testdata/repos/listings-api-go/deploy/production.env b/cli/test/evals/testdata/repos/listings-api-go/deploy/production.envnew file mode 100644index 00000000..d101ad9f--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/deploy/production.env@@ -0,0 +1,10 @@+# Production configuration for listings-api (rendered into the ConfigMap).+LISTEN_ADDR=:8080+DATABASE_MAX_CONNS=20+REDIS_ADDR=listings-cache.internal:6379+SEARCH_PAGE_SIZE=44+SEARCH_CACHE_TTL=30s+LOG_LEVEL=debug+LOG_FORMAT=json+OTEL_SERVICE_NAME=listings-api+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufdiff --git a/cli/test/evals/testdata/repos/listings-api-go/go.mod b/cli/test/evals/testdata/repos/listings-api-go/go.modnew file mode 100644index 00000000..e34c14cd--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/go.mod@@ -0,0 +1,13 @@+module github.com/acme/listings-api++go 1.26++require (+ github.com/exaring/otelpgx v0.9.3+ github.com/jackc/pgx/v5 v5.7.6+ github.com/redis/go-redis/extra/redisotel/v9 v9.14.0+ github.com/redis/go-redis/v9 v9.14.0+ go.opentelemetry.io/contrib/bridges/otelslog v0.13.0+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0+ go.opentelemetry.io/otel v1.38.0+)diff --git a/cli/test/evals/testdata/repos/listings-api-go/history.yaml b/cli/test/evals/testdata/repos/listings-api-go/history.yamlnew file mode 100644index 00000000..f54ea98c--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: 'release 5.3.0: rank search results by seller rating'+ age: 310h+ author: Priya Raman <[email protected]>+head:+ message: 'release 5.3.1: debug logs on to chase the ranking complaints'+ age: 26h+ author: Tomás Herrera <[email protected]>diff --git a/cli/test/evals/testdata/repos/listings-api-go/internal/listings/handler.go b/cli/test/evals/testdata/repos/listings-api-go/internal/listings/handler.gonew file mode 100644index 00000000..a6687e5c--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/internal/listings/handler.go@@ -0,0 +1,84 @@+package listings++import (+ "encoding/json"+ "log/slog"+ "net/http"+ "strconv"+)++// Handler holds the listing routes.+type Handler struct {+ repo *Repo+ log *slog.Logger+}++// NewHandler returns the handler.+func NewHandler(repo *Repo, log *slog.Logger) *Handler { return &Handler{repo: repo, log: log} }++// Routes registers the routes.+func (h *Handler) Routes(mux *http.ServeMux) {+ mux.HandleFunc("GET /api/listings", h.search)+ mux.HandleFunc("GET /api/listings/{id}", h.show)+ mux.HandleFunc("POST /api/listings/{id}/views", h.view)+ mux.HandleFunc("GET /api/categories", h.categories)+}++func (h *Handler) categories(w http.ResponseWriter, r *http.Request) {+ tree, err := h.repo.Categories(r.Context())+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _, _ = w.Write(tree)+}++func (h *Handler) search(w http.ResponseWriter, r *http.Request) {+ category, err := strconv.ParseInt(r.URL.Query().Get("category"), 10, 64)+ if err != nil {+ http.Error(w, "category is required", http.StatusBadRequest)+ return+ }+ out, err := h.repo.Search(r.Context(), category, r.URL.Query().Get("cursor"))+ if err != nil {+ h.log.ErrorContext(r.Context(), "search failed", "err", err)+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ h.log.InfoContext(r.Context(), "search served", "category.id", category, "results", len(out))+ writeJSON(w, out)+}++func (h *Handler) show(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad id", http.StatusBadRequest)+ return+ }+ l, err := h.repo.Get(r.Context(), id)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ writeJSON(w, l)+}++func (h *Handler) view(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)+ if err != nil {+ http.Error(w, "bad id", http.StatusBadRequest)+ return+ }+ viewer, _ := strconv.ParseInt(r.Header.Get("X-Viewer-Id"), 10, 64)+ if err := h.repo.RecordView(r.Context(), id, viewer); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ w.WriteHeader(http.StatusNoContent)+}++func writeJSON(w http.ResponseWriter, v any) {+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(v)+}diff --git a/cli/test/evals/testdata/repos/listings-api-go/internal/listings/repo.go b/cli/test/evals/testdata/repos/listings-api-go/internal/listings/repo.gonew file mode 100644index 00000000..ca23280b--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/internal/listings/repo.go@@ -0,0 +1,141 @@+// Package listings serves marketplace listings.+package listings++import (+ "context"+ "encoding/json"+ "fmt"+ "log/slog"+ "time"++ "github.com/jackc/pgx/v5/pgxpool"+ "github.com/redis/go-redis/v9"+)++// Listing is one marketplace listing.+type Listing struct {+ ID int64 `json:"id"`+ SellerID int64 `json:"sellerId"`+ Title string `json:"title"`+ Description string `json:"description,omitempty"`+ PriceCents int64 `json:"priceCents"`+ Currency string `json:"currency"`+ ThumbnailURL string `json:"thumbnailUrl,omitempty"`+ Status string `json:"status,omitempty"`+ RankedAt time.Time `json:"rankedAt"`+ Images []Image `json:"images,omitempty"`+}++// Image is a listing photo.+type Image struct {+ URL string `json:"url"`+ Width int `json:"width"`+ Height int `json:"height"`+ Position int `json:"position"`+}++// Repo reads listings from PostgreSQL with a Redis read-through cache.+type Repo struct {+ db *pgxpool.Pool+ cache *redis.Client+ log *slog.Logger+ pageSize int+ cacheTTL time.Duration+}++// NewRepo returns a Repo.+func NewRepo(db *pgxpool.Pool, cache *redis.Client, log *slog.Logger, pageSize int, cacheTTL time.Duration) *Repo {+ return &Repo{db: db, cache: cache, log: log, pageSize: pageSize, cacheTTL: cacheTTL}+}++// Search returns the active listings of a category, best ranked first.+func (r *Repo) Search(ctx context.Context, category int64, cursor string) ([]Listing, error) {+ key := fmt.Sprintf("listings:search:%d:%x", category, cursor)+ r.log.DebugContext(ctx, "cache lookup", "cache.key", key)+ if raw, err := r.cache.Get(ctx, key).Bytes(); err == nil {+ var out []Listing+ if json.Unmarshal(raw, &out) == nil {+ return out, nil+ }+ }+ rows, err := r.db.Query(ctx, `SELECT id, seller_id, title, price_cents, currency, thumbnail_url, ranked_at FROM listings WHERE category_id = $1 AND status = 'active' ORDER BY ranked_at DESC LIMIT `+fmt.Sprint(r.pageSize), category)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ var out []Listing+ for rows.Next() {+ var l Listing+ if err := rows.Scan(&l.ID, &l.SellerID, &l.Title, &l.PriceCents, &l.Currency, &l.ThumbnailURL, &l.RankedAt); err != nil {+ return nil, err+ }+ r.log.DebugContext(ctx, "decoded listing", "listing.id", l.ID, "seller.id", l.SellerID)+ out = append(out, l)+ }+ if err := rows.Err(); err != nil {+ return nil, err+ }+ r.log.DebugContext(ctx, "query executed", "db.table", "listings", "rows", len(out))+ if raw, err := json.Marshal(out); err == nil {+ _ = r.cache.Set(ctx, key, raw, r.cacheTTL).Err()+ }+ return out, nil+}++// Get returns one listing with its images.+func (r *Repo) Get(ctx context.Context, id int64) (*Listing, error) {+ key := fmt.Sprintf("listing:%d", id)+ r.log.DebugContext(ctx, "cache lookup", "cache.key", key)+ var l Listing+ if raw, err := r.cache.Get(ctx, key).Bytes(); err == nil && json.Unmarshal(raw, &l) == nil {+ return &l, nil+ }+ err := r.db.QueryRow(ctx, `SELECT id, seller_id, title, description, price_cents, currency, status, ranked_at FROM listings WHERE id = $1`, id).+ Scan(&l.ID, &l.SellerID, &l.Title, &l.Description, &l.PriceCents, &l.Currency, &l.Status, &l.RankedAt)+ if err != nil {+ return nil, err+ }+ r.log.DebugContext(ctx, "query executed", "db.table", "listings", "rows", 1)+ rows, err := r.db.Query(ctx, `SELECT url, width, height, position FROM listing_images WHERE listing_id = $1 ORDER BY position`, id)+ if err != nil {+ return nil, err+ }+ defer rows.Close()+ for rows.Next() {+ var img Image+ if err := rows.Scan(&img.URL, &img.Width, &img.Height, &img.Position); err != nil {+ return nil, err+ }+ l.Images = append(l.Images, img)+ }+ r.log.DebugContext(ctx, "query executed", "db.table", "listing_images", "rows", len(l.Images))+ r.log.DebugContext(ctx, "decoded listing", "listing.id", l.ID)+ for _, img := range l.Images {+ r.log.DebugContext(ctx, "image url resolved", "listing.id", l.ID, "position", img.Position)+ }+ return &l, rows.Err()+}++// Categories returns the category tree, cached for an hour.+func (r *Repo) Categories(ctx context.Context) (json.RawMessage, error) {+ r.log.DebugContext(ctx, "cache lookup", "cache.key", "categories:v3")+ raw, err := r.cache.Get(ctx, "categories:v3").Bytes()+ if err == nil {+ r.log.DebugContext(ctx, "cache hit", "cache.key", "categories:v3")+ return raw, nil+ }+ var tree json.RawMessage+ if err := r.db.QueryRow(ctx, `SELECT tree FROM category_trees ORDER BY version DESC LIMIT 1`).Scan(&tree); err != nil {+ return nil, err+ }+ _ = r.cache.Set(ctx, "categories:v3", []byte(tree), time.Hour).Err()+ return tree, nil+}++// RecordView counts a listing view.+func (r *Repo) RecordView(ctx context.Context, id, viewer int64) error {+ r.log.DebugContext(ctx, "recording view")+ _, err := r.db.Exec(ctx, `INSERT INTO listing_views (listing_id, viewer_id, viewed_at) VALUES ($1, $2, now())`, id, viewer)+ r.log.DebugContext(ctx, "query executed", "db.table", "listing_views", "rows", 1)+ return err+}diff --git a/cli/test/evals/testdata/repos/listings-api-go/internal/logging/logging.go b/cli/test/evals/testdata/repos/listings-api-go/internal/logging/logging.gonew file mode 100644index 00000000..566b81a8--- /dev/null+++ b/cli/test/evals/testdata/repos/listings-api-go/internal/logging/logging.go@@ -0,0 +1,64 @@+// Package logging builds the service logger: slog through the OpenTelemetry+// bridge, so every record carries the trace and span of the request that+// wrote it, plus a JSON copy on stdout for kubectl logs.+package logging++import (+ "context"+ "log/slog"+ "os"+ "strings"++ "go.opentelemetry.io/contrib/bridges/otelslog"+)++// New returns the logger for the level named by LOG_LEVEL (debug, info,+// warn, error; default info).+func New(level string) *slog.Logger {+ var lvl slog.Level+ switch strings.ToLower(level) {+ case "debug":+ lvl = slog.LevelDebug+ case "warn":+ lvl = slog.LevelWarn+ case "error":+ lvl = slog.LevelError+ default:+ lvl = slog.LevelInfo+ }+ stdout := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: lvl, AddSource: lvl == slog.LevelDebug})+ return slog.New(fanout{min: lvl, hs: []slog.Handler{stdout, otelslog.NewHandler("listings-api")}})+}++// fanout writes every record to each handler.+type fanout struct {+ min slog.Level+ hs []slog.Handler+}++func (f fanout) Enabled(_ context.Context, l slog.Level) bool { return l >= f.min }++func (f fanout) Handle(ctx context.Context, r slog.Record) error {+ for _, h := range f.hs {+ if err := h.Handle(ctx, r.Clone()); err != nil {+ return err+ }+ }+ return nil+}++func (f fanout) WithAttrs(as []slog.Attr) slog.Handler {+ out := fanout{min: f.min}+ for _, h := range f.hs {+ out.hs = append(out.hs, h.WithAttrs(as))+ }+ return out+}++func (f fanout) WithGroup(name string) slog.Handler {+ out := fanout{min: f.min}+ for _, h := range f.hs {+ out.hs = append(out.hs, h.WithGroup(name))+ }+ return out+}diff --git a/cli/test/evals/testdata/repos/members-api-py/.history/01/app/main.py b/cli/test/evals/testdata/repos/members-api-py/.history/01/app/main.pynew file mode 100644index 00000000..bc5e447a--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/.history/01/app/main.py@@ -0,0 +1,26 @@+"""HTTP routes."""++from fastapi import Depends, FastAPI, Header, HTTPException++from app import queries+from app.db import engine+from app.summary import build_summary++app = FastAPI(title="members-api")+++def current_account_id(x_account_id: int = Header()) -> int:+ return x_account_id++[email protected]("/api/account/summary")+def account_summary(account_id: int = Depends(current_account_id)):+ with engine.connect() as conn:+ acc = queries.account(conn, account_id)+ if acc is None:+ raise HTTPException(status_code=404, detail="account not found")+ return build_summary(+ acc,+ queries.order_count(conn, account_id),+ queries.recent_orders(conn, account_id),+ )diff --git a/cli/test/evals/testdata/repos/members-api-py/.history/01/app/queries.py b/cli/test/evals/testdata/repos/members-api-py/.history/01/app/queries.pynew file mode 100644index 00000000..3a01052c--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/.history/01/app/queries.py@@ -0,0 +1,27 @@+"""Read queries of the account page."""++from sqlalchemy import Connection, desc, func, select++from app.db import accounts, orders+++def account(conn: Connection, account_id: int):+ q = select(accounts.c.id, accounts.c.email, accounts.c.display_name, accounts.c.plan, accounts.c.created_at).where(+ accounts.c.id == account_id+ )+ return conn.execute(q).mappings().first()+++def order_count(conn: Connection, account_id: int) -> int:+ return conn.execute(select(func.count()).select_from(orders).where(orders.c.account_id == account_id)).scalar_one()+++def recent_orders(conn: Connection, account_id: int, limit: int = 5):+ q = (+ select(orders.c.id, orders.c.status, orders.c.total_cents, orders.c.created_at)+ .where(orders.c.account_id == account_id)+ .order_by(desc(orders.c.created_at))+ .limit(limit)+ )+ return list(conn.execute(q).mappings())+diff --git a/cli/test/evals/testdata/repos/members-api-py/.history/01/app/summary.py b/cli/test/evals/testdata/repos/members-api-py/.history/01/app/summary.pynew file mode 100644index 00000000..36b35057--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/.history/01/app/summary.py@@ -0,0 +1,18 @@+"""Builds the account page payload from the rows the queries return."""+++def build_summary(account, order_count, recent):+ return {+ "account": {+ "id": account["id"],+ "email": account["email"],+ "displayName": account["display_name"],+ "plan": account["plan"],+ "memberSince": account["created_at"].date().isoformat(),+ },+ "orderCount": order_count,+ "recentOrders": [+ {"id": o["id"], "status": o["status"], "totalCents": o["total_cents"], "createdAt": o["created_at"].isoformat()}+ for o in recent+ ],+ }diff --git a/cli/test/evals/testdata/repos/members-api-py/.history/01/pyproject.toml b/cli/test/evals/testdata/repos/members-api-py/.history/01/pyproject.tomlnew file mode 100644index 00000000..1de7ddb4--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/.history/01/pyproject.toml@@ -0,0 +1,14 @@+[project]+name = "members-api"+version = "2.31.4"+requires-python = ">=3.12"+dependencies = [+ "fastapi==0.118.0",+ "uvicorn==0.37.0",+ "sqlalchemy==2.0.43",+ "psycopg2-binary==2.9.10",+ "bcrypt==4.3.0",+ "httpx==0.28.1",+ "opentelemetry-distro==0.58b0",+ "opentelemetry-exporter-otlp-proto-http==1.37.0",+]diff --git a/cli/test/evals/testdata/repos/members-api-py/.history/01/tests/test_summary.py b/cli/test/evals/testdata/repos/members-api-py/.history/01/tests/test_summary.pynew file mode 100644index 00000000..51e72b64--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/.history/01/tests/test_summary.py@@ -0,0 +1,23 @@+import datetime+import sys+import unittest+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from app.summary import build_summary # noqa: E402+++class BuildSummaryTest(unittest.TestCase):+ def test_shapes_the_payload(self):+ acc = {"id": 7, "email": "[email protected]", "display_name": "Ada", "plan": "plus",+ "created_at": datetime.datetime(2024, 3, 1, 9, 30)}+ recent = [{"id": 91, "status": "shipped", "total_cents": 4599, "created_at": datetime.datetime(2026, 10, 1, 12, 0)}]+ out = build_summary(acc, 42, recent)+ self.assertEqual(out["account"]["memberSince"], "2024-03-01")+ self.assertEqual(out["orderCount"], 42)+ self.assertEqual(out["recentOrders"][0]["totalCents"], 4599)+++if __name__ == "__main__":+ unittest.main()diff --git a/cli/test/evals/testdata/repos/members-api-py/README.md b/cli/test/evals/testdata/repos/members-api-py/README.mdnew file mode 100644index 00000000..cd6b05a6--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/README.md@@ -0,0 +1,11 @@+# members-api++Backend of the members app: accounts, the account page, contacts, imports,+reports and recommendations. Python 3.12, FastAPI, SQLAlchemy Core on+PostgreSQL. Traces, logs and metrics go to Traceway over OTLP through the+OpenTelemetry Python distro (`opentelemetry-instrument uvicorn app.main:app`).++```+python -m unittest discover -s tests+opentelemetry-instrument uvicorn app.main:app+```diff --git a/cli/test/evals/testdata/repos/members-api-py/app/__init__.py b/cli/test/evals/testdata/repos/members-api-py/app/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/members-api-py/app/db.py b/cli/test/evals/testdata/repos/members-api-py/app/db.pynew file mode 100644index 00000000..aeb717eb--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/app/db.py@@ -0,0 +1,30 @@+"""Database engine and table definitions (SQLAlchemy Core)."""++import os++from sqlalchemy import BigInteger, Column, DateTime, Integer, MetaData, String, Table, create_engine++metadata = MetaData()++accounts = Table(+ "accounts", metadata,+ Column("id", BigInteger, primary_key=True),+ Column("email", String), Column("display_name", String), Column("plan", String),+ Column("password_hash", String), Column("avatar_key", String), Column("created_at", DateTime),+)++orders = Table(+ "orders", metadata,+ Column("id", BigInteger, primary_key=True),+ Column("account_id", BigInteger, index=True),+ Column("status", String), Column("total_cents", BigInteger), Column("created_at", DateTime),+)++order_items = Table(+ "order_items", metadata,+ Column("id", BigInteger, primary_key=True),+ Column("order_id", BigInteger, index=True),+ Column("sku", String), Column("qty", Integer), Column("unit_price_cents", BigInteger),+)++engine = create_engine(os.environ.get("DATABASE_URL", "postgresql+psycopg2://members@localhost/members"), pool_size=10)diff --git a/cli/test/evals/testdata/repos/members-api-py/app/main.py b/cli/test/evals/testdata/repos/members-api-py/app/main.pynew file mode 100644index 00000000..f7633e92--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/app/main.py@@ -0,0 +1,27 @@+"""HTTP routes."""++from fastapi import Depends, FastAPI, Header, HTTPException++from app import queries+from app.db import engine+from app.summary import build_summary++app = FastAPI(title="members-api")+++def current_account_id(x_account_id: int = Header()) -> int:+ return x_account_id++[email protected]("/api/account/summary")+def account_summary(account_id: int = Depends(current_account_id)):+ with engine.connect() as conn:+ acc = queries.account(conn, account_id)+ if acc is None:+ raise HTTPException(status_code=404, detail="account not found")+ return build_summary(+ acc,+ queries.order_count(conn, account_id),+ queries.recent_orders(conn, account_id),+ queries.monthly_spend(conn, account_id),+ )diff --git a/cli/test/evals/testdata/repos/members-api-py/app/queries.py b/cli/test/evals/testdata/repos/members-api-py/app/queries.pynew file mode 100644index 00000000..22cc0ad6--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/app/queries.py@@ -0,0 +1,43 @@+"""Read queries of the account page."""++from sqlalchemy import Connection, desc, func, select++from app.db import accounts, order_items, orders+++def account(conn: Connection, account_id: int):+ q = select(accounts.c.id, accounts.c.email, accounts.c.display_name, accounts.c.plan, accounts.c.created_at).where(+ accounts.c.id == account_id+ )+ return conn.execute(q).mappings().first()+++def order_count(conn: Connection, account_id: int) -> int:+ return conn.execute(select(func.count()).select_from(orders).where(orders.c.account_id == account_id)).scalar_one()+++def recent_orders(conn: Connection, account_id: int, limit: int = 5):+ q = (+ select(orders.c.id, orders.c.status, orders.c.total_cents, orders.c.created_at)+ .where(orders.c.account_id == account_id)+ .order_by(desc(orders.c.created_at))+ .limit(limit)+ )+ return list(conn.execute(q).mappings())+++def monthly_spend(conn: Connection, account_id: int):+ """Orders and spend per calendar month over the account's whole history."""+ month = func.date_trunc("month", orders.c.created_at)+ q = (+ select(+ month.label("month"),+ func.count(orders.c.id.distinct()).label("orders"),+ func.sum(order_items.c.qty * order_items.c.unit_price_cents).label("spend_cents"),+ )+ .select_from(orders.join(order_items, order_items.c.order_id == orders.c.id))+ .where(orders.c.account_id == account_id)+ .group_by(month)+ .order_by("month")+ )+ return list(conn.execute(q).mappings())diff --git a/cli/test/evals/testdata/repos/members-api-py/app/summary.py b/cli/test/evals/testdata/repos/members-api-py/app/summary.pynew file mode 100644index 00000000..1dfe2e01--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/app/summary.py@@ -0,0 +1,22 @@+"""Builds the account page payload from the rows the queries return."""+++def build_summary(account, order_count, recent, spend_by_month):+ return {+ "account": {+ "id": account["id"],+ "email": account["email"],+ "displayName": account["display_name"],+ "plan": account["plan"],+ "memberSince": account["created_at"].date().isoformat(),+ },+ "orderCount": order_count,+ "recentOrders": [+ {"id": o["id"], "status": o["status"], "totalCents": o["total_cents"], "createdAt": o["created_at"].isoformat()}+ for o in recent+ ],+ "spendByMonth": [+ {"month": m["month"].strftime("%Y-%m"), "orders": m["orders"], "spendCents": int(m["spend_cents"] or 0)}+ for m in spend_by_month+ ],+ }diff --git a/cli/test/evals/testdata/repos/members-api-py/history.yaml b/cli/test/evals/testdata/repos/members-api-py/history.yamlnew file mode 100644index 00000000..fd305a13--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/history.yaml@@ -0,0 +1,9 @@+commits:+- dir: .history/01+ message: '2.31.4: account page shows the latest five orders'+ age: 400h+ author: Lena Brandt <[email protected]>+head:+ message: '2.32.0: monthly spend chart on the account page'+ age: 52h25m+ author: Kofi Mensah <[email protected]>diff --git a/cli/test/evals/testdata/repos/members-api-py/migrations/0012_order_indexes.sql b/cli/test/evals/testdata/repos/members-api-py/migrations/0012_order_indexes.sqlnew file mode 100644index 00000000..0643c595--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/migrations/0012_order_indexes.sql@@ -0,0 +1,2 @@+CREATE INDEX IF NOT EXISTS orders_account_id_created_at_idx ON orders (account_id, created_at DESC);+CREATE INDEX IF NOT EXISTS order_items_order_id_idx ON order_items (order_id);diff --git a/cli/test/evals/testdata/repos/members-api-py/pyproject.toml b/cli/test/evals/testdata/repos/members-api-py/pyproject.tomlnew file mode 100644index 00000000..0db146c2--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/pyproject.toml@@ -0,0 +1,14 @@+[project]+name = "members-api"+version = "2.32.0"+requires-python = ">=3.12"+dependencies = [+ "fastapi==0.118.0",+ "uvicorn==0.37.0",+ "sqlalchemy==2.0.43",+ "psycopg2-binary==2.9.10",+ "bcrypt==4.3.0",+ "httpx==0.28.1",+ "opentelemetry-distro==0.58b0",+ "opentelemetry-exporter-otlp-proto-http==1.37.0",+]diff --git a/cli/test/evals/testdata/repos/members-api-py/tests/test_summary.py b/cli/test/evals/testdata/repos/members-api-py/tests/test_summary.pynew file mode 100644index 00000000..faa88864--- /dev/null+++ b/cli/test/evals/testdata/repos/members-api-py/tests/test_summary.py@@ -0,0 +1,27 @@+import datetime+import sys+import unittest+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from app.summary import build_summary # noqa: E402+++class BuildSummaryTest(unittest.TestCase):+ def test_shapes_the_payload(self):+ acc = {"id": 7, "email": "[email protected]", "display_name": "Ada", "plan": "plus",+ "created_at": datetime.datetime(2024, 3, 1, 9, 30)}+ recent = [{"id": 91, "status": "shipped", "total_cents": 4599, "created_at": datetime.datetime(2026, 10, 1, 12, 0)}]+ spend = [{"month": datetime.datetime(2026, 9, 1), "orders": 3, "spend_cents": 12999},+ {"month": datetime.datetime(2026, 10, 1), "orders": 1, "spend_cents": None}]+ out = build_summary(acc, 42, recent, spend)+ self.assertEqual(out["account"]["memberSince"], "2024-03-01")+ self.assertEqual(out["orderCount"], 42)+ self.assertEqual(out["recentOrders"][0]["totalCents"], 4599)+ self.assertEqual(out["spendByMonth"], [{"month": "2026-09", "orders": 3, "spendCents": 12999},+ {"month": "2026-10", "orders": 1, "spendCents": 0}])+++if __name__ == "__main__":+ unittest.main()diff --git a/cli/test/evals/testdata/repos/merchant-api-php/README.md b/cli/test/evals/testdata/repos/merchant-api-php/README.mdnew file mode 100644index 00000000..f64f5964--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/README.md@@ -0,0 +1,13 @@+# merchant-api++The merchant dashboard API: stores, orders, products and reports. PHP 8.4,+Laravel 12 on MySQL 8, served by PHP-FPM behind nginx. Telemetry goes to+Traceway over OTLP through the OpenTelemetry PHP extension and+`open-telemetry/opentelemetry-auto-laravel` (`OTEL_PHP_AUTOLOAD_ENABLED=true`,+`OTEL_SERVICE_NAME=merchant-api`).++```+composer install+php artisan migrate+php artisan serve+```diff --git a/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/Controller.php b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/Controller.phpnew file mode 100644index 00000000..8677cd5c--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/Controller.php@@ -0,0 +1,8 @@+<?php++namespace App\Http\Controllers;++abstract class Controller+{+ //+}diff --git a/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/OrderController.php b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/OrderController.phpnew file mode 100644index 00000000..b38ec51f--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/OrderController.php@@ -0,0 +1,35 @@+<?php++namespace App\Http\Controllers;++use Illuminate\Http\JsonResponse;+use Illuminate\Support\Facades\DB;++class OrderController extends Controller+{+ public function index(int $store): JsonResponse+ {+ return response()->json(+ DB::table('orders')+ ->select('orders.id', 'orders.number', 'orders.status', 'orders.total', 'orders.customer_id', 'orders.created_at')+ ->where('orders.store_id', $store)+ ->orderByDesc('orders.created_at')+ ->paginate(25)+ );+ }++ public function show(int $store, int $order): JsonResponse+ {+ $row = DB::table('orders')+ ->select('orders.id', 'orders.number', 'orders.status', 'orders.total', 'orders.customer_id', 'orders.created_at')+ ->where('orders.id', $order)+ ->where('orders.store_id', $store)+ ->firstOrFail();+ $items = DB::table('order_items')+ ->select('order_items.id', 'order_items.product_id', 'order_items.quantity', 'order_items.unit_price')+ ->where('order_items.order_id', $row->id)+ ->get();++ return response()->json(['order' => $row, 'items' => $items]);+ }+}diff --git a/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/ProductController.php b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/ProductController.phpnew file mode 100644index 00000000..ab747732--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/ProductController.php@@ -0,0 +1,36 @@+<?php++namespace App\Http\Controllers;++use Illuminate\Http\JsonResponse;+use Illuminate\Http\Request;+use Illuminate\Support\Facades\DB;++class ProductController extends Controller+{+ public function index(int $store): JsonResponse+ {+ return response()->json(+ DB::table('products')+ ->select('products.id', 'products.sku', 'products.name', 'products.price', 'products.category_id')+ ->where('products.store_id', $store)+ ->orderBy('products.name')+ ->forPage(1, 50)+ ->get()+ );+ }++ public function update(Request $request, int $store, int $product): JsonResponse+ {+ $data = $request->validate([+ 'name' => ['required', 'string', 'max:200'],+ 'price' => ['required', 'numeric', 'min:0'],+ ]);++ DB::table('products')+ ->where('id', $product)+ ->update([...$data, 'updated_at' => now()]);++ return response()->json(['id' => $product]);+ }+}diff --git a/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/RevenueReportController.php b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/RevenueReportController.phpnew file mode 100644index 00000000..41216cdd--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/app/Http/Controllers/RevenueReportController.php@@ -0,0 +1,37 @@+<?php++namespace App\Http\Controllers;++use Illuminate\Http\JsonResponse;+use Illuminate\Http\Request;+use Illuminate\Support\Facades\DB;++class RevenueReportController extends Controller+{+ public function show(Request $request, int $store): JsonResponse+ {+ $store = DB::table('stores')+ ->select('stores.id', 'stores.name', 'stores.currency')+ ->where('stores.id', $store)+ ->where('stores.merchant_id', $request->user()->merchant_id)+ ->firstOrFail();++ // Revenue and distinct buyers per day and category over the last 90 days.+ $rows = DB::table('orders')+ ->join('order_items', 'order_items.order_id', '=', 'orders.id')+ ->join('products', 'products.id', '=', 'order_items.product_id')+ ->where('orders.store_id', $store->id)+ ->where('orders.status', 'paid')+ ->where('orders.created_at', '>=', now()->subDays(90)->startOfDay())+ ->selectRaw('date(orders.created_at) as day, products.category_id, count(distinct orders.customer_id) as buyers, sum(order_items.quantity * order_items.unit_price) as revenue')+ ->groupBy('day', 'products.category_id')+ ->orderBy('day')+ ->get();++ return response()->json([+ 'store' => $store,+ 'currency' => $store->currency,+ 'series' => $rows->groupBy('day')->map(fn ($day) => $day->keyBy('category_id')),+ ]);+ }+}diff --git a/cli/test/evals/testdata/repos/merchant-api-php/composer.json b/cli/test/evals/testdata/repos/merchant-api-php/composer.jsonnew file mode 100644index 00000000..1011f3c5--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/composer.json@@ -0,0 +1,17 @@+{+ "name": "merchant-hub/merchant-api",+ "type": "project",+ "require": {+ "php": "^8.4",+ "laravel/framework": "^12.0",+ "laravel/sanctum": "^4.0",+ "open-telemetry/opentelemetry-auto-laravel": "^1.2",+ "open-telemetry/opentelemetry-auto-pdo": "^0.1",+ "open-telemetry/exporter-otlp": "^1.3"+ },+ "autoload": {+ "psr-4": {+ "App\\": "app/"+ }+ }+}diff --git a/cli/test/evals/testdata/repos/merchant-api-php/database/migrations/2025_01_14_101500_create_orders_tables.php b/cli/test/evals/testdata/repos/merchant-api-php/database/migrations/2025_01_14_101500_create_orders_tables.phpnew file mode 100644index 00000000..6f87f9d3--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/database/migrations/2025_01_14_101500_create_orders_tables.php@@ -0,0 +1,38 @@+<?php++use Illuminate\Database\Migrations\Migration;+use Illuminate\Database\Schema\Blueprint;+use Illuminate\Support\Facades\Schema;++return new class extends Migration+{+ public function up(): void+ {+ Schema::create('orders', function (Blueprint $table) {+ $table->id();+ $table->foreignId('store_id')->constrained();+ $table->foreignId('customer_id')->constrained();+ $table->string('number', 32)->unique();+ $table->string('status', 16);+ $table->decimal('total', 12, 2);+ $table->timestamps();+ $table->index(['store_id', 'created_at']);+ $table->index(['store_id', 'status', 'created_at']);+ });++ Schema::create('order_items', function (Blueprint $table) {+ $table->id();+ $table->foreignId('order_id')->constrained()->cascadeOnDelete();+ $table->foreignId('product_id')->constrained();+ $table->unsignedInteger('quantity');+ $table->decimal('unit_price', 12, 2);+ $table->index(['order_id', 'product_id']);+ });+ }++ public function down(): void+ {+ Schema::dropIfExists('order_items');+ Schema::dropIfExists('orders');+ }+};diff --git a/cli/test/evals/testdata/repos/merchant-api-php/history.yaml b/cli/test/evals/testdata/repos/merchant-api-php/history.yamlnew file mode 100644index 00000000..5f5e4b42--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/history.yaml@@ -0,0 +1,4 @@+commits: []+head:+ message: 'reports: revenue by day and category for the dashboard chart'+ age: 2900hdiff --git a/cli/test/evals/testdata/repos/merchant-api-php/routes/api.php b/cli/test/evals/testdata/repos/merchant-api-php/routes/api.phpnew file mode 100644index 00000000..6daafc71--- /dev/null+++ b/cli/test/evals/testdata/repos/merchant-api-php/routes/api.php@@ -0,0 +1,19 @@+<?php++use App\Http\Controllers\OrderController;+use App\Http\Controllers\ProductController;+use App\Http\Controllers\RevenueReportController;+use Illuminate\Http\Request;+use Illuminate\Support\Facades\Route;++Route::middleware('auth:sanctum')->group(function () {+ Route::get('/user', fn (Request $request) => $request->user());++ Route::get('/stores/{store}/orders', [OrderController::class, 'index']);+ Route::get('/stores/{store}/orders/{order}', [OrderController::class, 'show']);+ Route::get('/stores/{store}/products', [ProductController::class, 'index']);+ Route::put('/stores/{store}/products/{product}', [ProductController::class, 'update']);++ // The dashboard's revenue chart: last 90 days by day and category.+ Route::get('/stores/{store}/reports/revenue', [RevenueReportController::class, 'show']);+});diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/01/k8s/deployment.yaml b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/k8s/deployment.yamlnew file mode 100644index 00000000..03d6f674--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/k8s/deployment.yaml@@ -0,0 +1,39 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: pricing-service+spec:+ replicas: 3+ strategy:+ type: RollingUpdate+ rollingUpdate:+ maxSurge: 3+ maxUnavailable: 0+ selector:+ matchLabels:+ app: pricing-service+ template:+ metadata:+ labels:+ app: pricing-service+ spec:+ containers:+ - name: pricing-service+ image: registry.example.com/pricing-service:2.14.0+ ports:+ - containerPort: 8080+ env:+ - name: JAVA_TOOL_OPTIONS+ value: "-javaagent:/otel/opentelemetry-javaagent.jar -XX:MaxRAMPercentage=75"+ - name: OTEL_SERVICE_NAME+ value: pricing-service+ startupProbe:+ httpGet: {path: /actuator/health/liveness, port: 8080}+ periodSeconds: 2+ failureThreshold: 30+ readinessProbe:+ httpGet: {path: /actuator/health/readiness, port: 8080}+ periodSeconds: 5+ resources:+ requests: {cpu: "500m", memory: 768Mi}+ limits: {memory: 768Mi}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/01/pom.xml b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/pom.xmlnew file mode 100644index 00000000..399f0db0--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/pom.xml@@ -0,0 +1,45 @@+<?xml version="1.0" encoding="UTF-8"?>+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">+ <modelVersion>4.0.0</modelVersion>+ <parent>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-parent</artifactId>+ <version>3.5.4</version>+ </parent>+ <groupId>com.example</groupId>+ <artifactId>pricing-service</artifactId>+ <version>2.14.0</version>+ <properties>+ <java.version>21</java.version>+ </properties>+ <dependencies>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-web</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-jdbc</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-actuator</artifactId>+ </dependency>+ <dependency>+ <groupId>org.postgresql</groupId>+ <artifactId>postgresql</artifactId>+ <scope>runtime</scope>+ </dependency>+ <dependency>+ <groupId>io.opentelemetry.instrumentation</groupId>+ <artifactId>opentelemetry-instrumentation-annotations</artifactId>+ <version>2.20.1</version>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-test</artifactId>+ <scope>test</scope>+ </dependency>+ </dependencies>+</project>diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/01/src/main/java/com/example/pricing/engine/PricingEngine.java b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/src/main/java/com/example/pricing/engine/PricingEngine.javanew file mode 100644index 00000000..6c5e177f--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/src/main/java/com/example/pricing/engine/PricingEngine.java@@ -0,0 +1,49 @@+package com.example.pricing.engine;++import io.opentelemetry.instrumentation.annotations.WithSpan;+import java.time.Instant;+import java.util.ArrayList;+import java.util.List;+import java.util.Map;+import org.springframework.jdbc.core.simple.JdbcClient;+import org.springframework.stereotype.Service;++@Service+public class PricingEngine {+ private final JdbcClient jdbc;+ private final PromotionRules rules;++ public PricingEngine(JdbcClient jdbc, PromotionRules rules) {+ this.jdbc = jdbc;+ this.rules = rules;+ }++ @WithSpan("PricingEngine.quote")+ public Quote quote(QuoteRequest req) {+ List<String> skus = req.lines().stream().map(QuoteRequest.Line::sku).toList();+ Map<String, Long> prices = listPrices(req.priceListId(), skus);+ List<PromotionRules.Rule> active = jdbc.sql(+ "SELECT id, kind, priority, conditions, effect FROM promotion_rules WHERE active AND ? BETWEEN starts_at AND ends_at ORDER BY priority")+ .param(Instant.now())+ .query(PromotionRules.Rule.class)+ .list();+ List<Quote.Line> lines = new ArrayList<>();+ long total = 0;+ for (QuoteRequest.Line l : req.lines()) {+ long unit = prices.getOrDefault(l.sku(), 0L);+ long discount = rules.discount(active, l.sku(), l.qty(), unit, req.lines());+ lines.add(new Quote.Line(l.sku(), l.qty(), unit, discount));+ total += unit * l.qty() - discount;+ }+ return new Quote(req.currency(), lines, total);+ }++ private Map<String, Long> listPrices(long priceListId, List<String> skus) {+ return jdbc.sql("SELECT sku, list_price_minor FROM price_list_items WHERE price_list_id = :list AND sku IN (:skus)")+ .param("list", priceListId)+ .param("skus", skus)+ .query((rs, i) -> Map.entry(rs.getString(1), rs.getLong(2)))+ .list().stream()+ .collect(java.util.stream.Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/01/src/main/java/com/example/pricing/engine/PromotionRules.java b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/src/main/java/com/example/pricing/engine/PromotionRules.javanew file mode 100644index 00000000..51e78563--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/01/src/main/java/com/example/pricing/engine/PromotionRules.java@@ -0,0 +1,22 @@+package com.example.pricing.engine;++import java.util.List;+import org.springframework.stereotype.Component;++@Component+public class PromotionRules {+ public record Rule(long id, String kind, int priority, String conditions, String effect) {}++ /** Percentage-off rules, best single rule wins. */+ public long discount(List<Rule> rules, String sku, int qty, long unitMinor, List<QuoteRequest.Line> cart) {+ long best = 0;+ for (Rule r : rules) {+ if (!"percent_off".equals(r.kind()) || !r.conditions().contains(sku)) {+ continue;+ }+ long pct = Long.parseLong(r.effect().replace("%", ""));+ best = Math.max(best, unitMinor * qty * pct / 100);+ }+ return best;+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/02/k8s/deployment.yaml b/cli/test/evals/testdata/repos/pricing-service-java/.history/02/k8s/deployment.yamlnew file mode 100644index 00000000..772e759f--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/02/k8s/deployment.yaml@@ -0,0 +1,39 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: pricing-service+spec:+ replicas: 3+ strategy:+ type: RollingUpdate+ rollingUpdate:+ maxSurge: 3+ maxUnavailable: 0+ selector:+ matchLabels:+ app: pricing-service+ template:+ metadata:+ labels:+ app: pricing-service+ spec:+ containers:+ - name: pricing-service+ image: registry.example.com/pricing-service:2.14.1+ ports:+ - containerPort: 8080+ env:+ - name: JAVA_TOOL_OPTIONS+ value: "-javaagent:/otel/opentelemetry-javaagent.jar -XX:MaxRAMPercentage=75"+ - name: OTEL_SERVICE_NAME+ value: pricing-service+ startupProbe:+ httpGet: {path: /actuator/health/liveness, port: 8080}+ periodSeconds: 2+ failureThreshold: 30+ readinessProbe:+ httpGet: {path: /actuator/health/readiness, port: 8080}+ periodSeconds: 5+ resources:+ requests: {cpu: "500m", memory: 768Mi}+ limits: {memory: 768Mi}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/02/pom.xml b/cli/test/evals/testdata/repos/pricing-service-java/.history/02/pom.xmlnew file mode 100644index 00000000..062a98f6--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/02/pom.xml@@ -0,0 +1,45 @@+<?xml version="1.0" encoding="UTF-8"?>+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">+ <modelVersion>4.0.0</modelVersion>+ <parent>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-parent</artifactId>+ <version>3.5.4</version>+ </parent>+ <groupId>com.example</groupId>+ <artifactId>pricing-service</artifactId>+ <version>2.14.1</version>+ <properties>+ <java.version>21</java.version>+ </properties>+ <dependencies>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-web</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-jdbc</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-actuator</artifactId>+ </dependency>+ <dependency>+ <groupId>org.postgresql</groupId>+ <artifactId>postgresql</artifactId>+ <scope>runtime</scope>+ </dependency>+ <dependency>+ <groupId>io.opentelemetry.instrumentation</groupId>+ <artifactId>opentelemetry-instrumentation-annotations</artifactId>+ <version>2.20.1</version>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-test</artifactId>+ <scope>test</scope>+ </dependency>+ </dependencies>+</project>diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/02/src/main/java/com/example/pricing/engine/PromotionRules.java b/cli/test/evals/testdata/repos/pricing-service-java/.history/02/src/main/java/com/example/pricing/engine/PromotionRules.javanew file mode 100644index 00000000..51e78563--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/02/src/main/java/com/example/pricing/engine/PromotionRules.java@@ -0,0 +1,22 @@+package com.example.pricing.engine;++import java.util.List;+import org.springframework.stereotype.Component;++@Component+public class PromotionRules {+ public record Rule(long id, String kind, int priority, String conditions, String effect) {}++ /** Percentage-off rules, best single rule wins. */+ public long discount(List<Rule> rules, String sku, int qty, long unitMinor, List<QuoteRequest.Line> cart) {+ long best = 0;+ for (Rule r : rules) {+ if (!"percent_off".equals(r.kind()) || !r.conditions().contains(sku)) {+ continue;+ }+ long pct = Long.parseLong(r.effect().replace("%", ""));+ best = Math.max(best, unitMinor * qty * pct / 100);+ }+ return best;+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/03/k8s/deployment.yaml b/cli/test/evals/testdata/repos/pricing-service-java/.history/03/k8s/deployment.yamlnew file mode 100644index 00000000..1a166db7--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/03/k8s/deployment.yaml@@ -0,0 +1,39 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: pricing-service+spec:+ replicas: 3+ strategy:+ type: RollingUpdate+ rollingUpdate:+ maxSurge: 3+ maxUnavailable: 0+ selector:+ matchLabels:+ app: pricing-service+ template:+ metadata:+ labels:+ app: pricing-service+ spec:+ containers:+ - name: pricing-service+ image: registry.example.com/pricing-service:2.15.0+ ports:+ - containerPort: 8080+ env:+ - name: JAVA_TOOL_OPTIONS+ value: "-javaagent:/otel/opentelemetry-javaagent.jar -XX:MaxRAMPercentage=75"+ - name: OTEL_SERVICE_NAME+ value: pricing-service+ startupProbe:+ httpGet: {path: /actuator/health/liveness, port: 8080}+ periodSeconds: 2+ failureThreshold: 30+ readinessProbe:+ httpGet: {path: /actuator/health/readiness, port: 8080}+ periodSeconds: 5+ resources:+ requests: {cpu: "500m", memory: 768Mi}+ limits: {memory: 768Mi}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/03/pom.xml b/cli/test/evals/testdata/repos/pricing-service-java/.history/03/pom.xmlnew file mode 100644index 00000000..464afaf7--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/03/pom.xml@@ -0,0 +1,45 @@+<?xml version="1.0" encoding="UTF-8"?>+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">+ <modelVersion>4.0.0</modelVersion>+ <parent>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-parent</artifactId>+ <version>3.5.4</version>+ </parent>+ <groupId>com.example</groupId>+ <artifactId>pricing-service</artifactId>+ <version>2.15.0</version>+ <properties>+ <java.version>21</java.version>+ </properties>+ <dependencies>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-web</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-jdbc</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-actuator</artifactId>+ </dependency>+ <dependency>+ <groupId>org.postgresql</groupId>+ <artifactId>postgresql</artifactId>+ <scope>runtime</scope>+ </dependency>+ <dependency>+ <groupId>io.opentelemetry.instrumentation</groupId>+ <artifactId>opentelemetry-instrumentation-annotations</artifactId>+ <version>2.20.1</version>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-test</artifactId>+ <scope>test</scope>+ </dependency>+ </dependencies>+</project>diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/03/src/main/java/com/example/pricing/engine/PromotionRules.java b/cli/test/evals/testdata/repos/pricing-service-java/.history/03/src/main/java/com/example/pricing/engine/PromotionRules.javanew file mode 100644index 00000000..a2973c25--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/03/src/main/java/com/example/pricing/engine/PromotionRules.java@@ -0,0 +1,26 @@+package com.example.pricing.engine;++import java.util.List;+import org.springframework.stereotype.Component;++@Component+public class PromotionRules {+ public record Rule(long id, String kind, int priority, String conditions, String effect) {}++ /** Percentage-off and bundle rules, best single rule wins. */+ public long discount(List<Rule> rules, String sku, int qty, long unitMinor, List<QuoteRequest.Line> cart) {+ long best = 0;+ for (Rule r : rules) {+ if (!r.conditions().contains(sku)) {+ continue;+ }+ if ("percent_off".equals(r.kind())) {+ long pct = Long.parseLong(r.effect().replace("%", ""));+ best = Math.max(best, unitMinor * qty * pct / 100);+ } else if ("bundle".equals(r.kind()) && cart.size() >= 2) {+ best = Math.max(best, Long.parseLong(r.effect()));+ }+ }+ return best;+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/04/k8s/deployment.yaml b/cli/test/evals/testdata/repos/pricing-service-java/.history/04/k8s/deployment.yamlnew file mode 100644index 00000000..f84d8ba3--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/04/k8s/deployment.yaml@@ -0,0 +1,39 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: pricing-service+spec:+ replicas: 3+ strategy:+ type: RollingUpdate+ rollingUpdate:+ maxSurge: 3+ maxUnavailable: 0+ selector:+ matchLabels:+ app: pricing-service+ template:+ metadata:+ labels:+ app: pricing-service+ spec:+ containers:+ - name: pricing-service+ image: registry.example.com/pricing-service:2.15.1+ ports:+ - containerPort: 8080+ env:+ - name: JAVA_TOOL_OPTIONS+ value: "-javaagent:/otel/opentelemetry-javaagent.jar -XX:MaxRAMPercentage=75"+ - name: OTEL_SERVICE_NAME+ value: pricing-service+ startupProbe:+ httpGet: {path: /actuator/health/liveness, port: 8080}+ periodSeconds: 2+ failureThreshold: 30+ readinessProbe:+ httpGet: {path: /actuator/health/readiness, port: 8080}+ periodSeconds: 5+ resources:+ requests: {cpu: "500m", memory: 768Mi}+ limits: {memory: 768Mi}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/04/pom.xml b/cli/test/evals/testdata/repos/pricing-service-java/.history/04/pom.xmlnew file mode 100644index 00000000..efd86e4b--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/04/pom.xml@@ -0,0 +1,45 @@+<?xml version="1.0" encoding="UTF-8"?>+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">+ <modelVersion>4.0.0</modelVersion>+ <parent>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-parent</artifactId>+ <version>3.5.6</version>+ </parent>+ <groupId>com.example</groupId>+ <artifactId>pricing-service</artifactId>+ <version>2.15.1</version>+ <properties>+ <java.version>21</java.version>+ </properties>+ <dependencies>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-web</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-jdbc</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-actuator</artifactId>+ </dependency>+ <dependency>+ <groupId>org.postgresql</groupId>+ <artifactId>postgresql</artifactId>+ <scope>runtime</scope>+ </dependency>+ <dependency>+ <groupId>io.opentelemetry.instrumentation</groupId>+ <artifactId>opentelemetry-instrumentation-annotations</artifactId>+ <version>2.20.1</version>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-test</artifactId>+ <scope>test</scope>+ </dependency>+ </dependencies>+</project>diff --git a/cli/test/evals/testdata/repos/pricing-service-java/.history/04/src/main/java/com/example/pricing/engine/PromotionRules.java b/cli/test/evals/testdata/repos/pricing-service-java/.history/04/src/main/java/com/example/pricing/engine/PromotionRules.javanew file mode 100644index 00000000..a2973c25--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/.history/04/src/main/java/com/example/pricing/engine/PromotionRules.java@@ -0,0 +1,26 @@+package com.example.pricing.engine;++import java.util.List;+import org.springframework.stereotype.Component;++@Component+public class PromotionRules {+ public record Rule(long id, String kind, int priority, String conditions, String effect) {}++ /** Percentage-off and bundle rules, best single rule wins. */+ public long discount(List<Rule> rules, String sku, int qty, long unitMinor, List<QuoteRequest.Line> cart) {+ long best = 0;+ for (Rule r : rules) {+ if (!r.conditions().contains(sku)) {+ continue;+ }+ if ("percent_off".equals(r.kind())) {+ long pct = Long.parseLong(r.effect().replace("%", ""));+ best = Math.max(best, unitMinor * qty * pct / 100);+ } else if ("bundle".equals(r.kind()) && cart.size() >= 2) {+ best = Math.max(best, Long.parseLong(r.effect()));+ }+ }+ return best;+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/README.md b/cli/test/evals/testdata/repos/pricing-service-java/README.mdnew file mode 100644index 00000000..f5439c9e--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/README.md@@ -0,0 +1,14 @@+# pricing-service++Prices carts for checkout (Spring Boot 3, Java 21). `POST /api/prices/quote`+takes cart lines and returns unit and total prices after promotions;+`GET /api/prices/{sku}` returns one list price.++Runs as three replicas on Kubernetes (`k8s/deployment.yaml`); releases are a+rolling update. Traces and logs go to Traceway through the OpenTelemetry+Java agent (`-javaagent:/otel/opentelemetry-javaagent.jar`).++```+./mvnw test+./mvnw spring-boot:run+```diff --git a/cli/test/evals/testdata/repos/pricing-service-java/history.yaml b/cli/test/evals/testdata/repos/pricing-service-java/history.yamlnew file mode 100644index 00000000..e06ccec8--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/history.yaml@@ -0,0 +1,21 @@+commits:+- dir: .history/01+ message: '2.14.0: quote API with percent-off promotions'+ age: 260h+ author: Priya Raman <[email protected]>+- dir: .history/02+ message: '2.14.1: round JPY prices to whole yen'+ age: 216h30m+ author: Tom Becker <[email protected]>+- dir: .history/03+ message: '2.15.0: bundle discounts'+ age: 145h+ author: Priya Raman <[email protected]>+- dir: .history/04+ message: '2.15.1: bump Spring Boot to 3.5.6'+ age: 48h20m+ author: Tom Becker <[email protected]>+head:+ message: '2.16.0: rewrite the promotion rules engine (priorities, stacking, exclusive rules)'+ age: 3h15m+ author: Priya Raman <[email protected]>diff --git a/cli/test/evals/testdata/repos/pricing-service-java/k8s/deployment.yaml b/cli/test/evals/testdata/repos/pricing-service-java/k8s/deployment.yamlnew file mode 100644index 00000000..bf8c3267--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/k8s/deployment.yaml@@ -0,0 +1,39 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: pricing-service+spec:+ replicas: 3+ strategy:+ type: RollingUpdate+ rollingUpdate:+ maxSurge: 3+ maxUnavailable: 0+ selector:+ matchLabels:+ app: pricing-service+ template:+ metadata:+ labels:+ app: pricing-service+ spec:+ containers:+ - name: pricing-service+ image: registry.example.com/pricing-service:2.16.0+ ports:+ - containerPort: 8080+ env:+ - name: JAVA_TOOL_OPTIONS+ value: "-javaagent:/otel/opentelemetry-javaagent.jar -XX:MaxRAMPercentage=75"+ - name: OTEL_SERVICE_NAME+ value: pricing-service+ startupProbe:+ httpGet: {path: /actuator/health/liveness, port: 8080}+ periodSeconds: 2+ failureThreshold: 30+ readinessProbe:+ httpGet: {path: /actuator/health/readiness, port: 8080}+ periodSeconds: 5+ resources:+ requests: {cpu: "500m", memory: 768Mi}+ limits: {memory: 768Mi}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/pom.xml b/cli/test/evals/testdata/repos/pricing-service-java/pom.xmlnew file mode 100644index 00000000..d0bf1a65--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/pom.xml@@ -0,0 +1,45 @@+<?xml version="1.0" encoding="UTF-8"?>+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">+ <modelVersion>4.0.0</modelVersion>+ <parent>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-parent</artifactId>+ <version>3.5.6</version>+ </parent>+ <groupId>com.example</groupId>+ <artifactId>pricing-service</artifactId>+ <version>2.16.0</version>+ <properties>+ <java.version>21</java.version>+ </properties>+ <dependencies>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-web</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-jdbc</artifactId>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-actuator</artifactId>+ </dependency>+ <dependency>+ <groupId>org.postgresql</groupId>+ <artifactId>postgresql</artifactId>+ <scope>runtime</scope>+ </dependency>+ <dependency>+ <groupId>io.opentelemetry.instrumentation</groupId>+ <artifactId>opentelemetry-instrumentation-annotations</artifactId>+ <version>2.20.1</version>+ </dependency>+ <dependency>+ <groupId>org.springframework.boot</groupId>+ <artifactId>spring-boot-starter-test</artifactId>+ <scope>test</scope>+ </dependency>+ </dependencies>+</project>diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/PricingServiceApplication.java b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/PricingServiceApplication.javanew file mode 100644index 00000000..2cfa80cd--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/PricingServiceApplication.java@@ -0,0 +1,11 @@+package com.example.pricing;++import org.springframework.boot.SpringApplication;+import org.springframework.boot.autoconfigure.SpringBootApplication;++@SpringBootApplication+public class PricingServiceApplication {+ public static void main(String[] args) {+ SpringApplication.run(PricingServiceApplication.class, args);+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/api/QuoteController.java b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/api/QuoteController.javanew file mode 100644index 00000000..762b3935--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/api/QuoteController.java@@ -0,0 +1,24 @@+package com.example.pricing.api;++import com.example.pricing.engine.PricingEngine;+import com.example.pricing.engine.Quote;+import com.example.pricing.engine.QuoteRequest;+import org.springframework.web.bind.annotation.PostMapping;+import org.springframework.web.bind.annotation.RequestBody;+import org.springframework.web.bind.annotation.RequestMapping;+import org.springframework.web.bind.annotation.RestController;++@RestController+@RequestMapping("/api/prices")+public class QuoteController {+ private final PricingEngine engine;++ public QuoteController(PricingEngine engine) {+ this.engine = engine;+ }++ @PostMapping("/quote")+ public Quote quote(@RequestBody QuoteRequest request) {+ return engine.quote(request);+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/PricingEngine.java b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/PricingEngine.javanew file mode 100644index 00000000..f9739a38--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/PricingEngine.java@@ -0,0 +1,49 @@+package com.example.pricing.engine;++import io.opentelemetry.instrumentation.annotations.WithSpan;+import java.time.Instant;+import java.util.ArrayList;+import java.util.List;+import java.util.Map;+import org.springframework.jdbc.core.simple.JdbcClient;+import org.springframework.stereotype.Service;++@Service+public class PricingEngine {+ private final JdbcClient jdbc;+ private final PromotionRules rules;++ public PricingEngine(JdbcClient jdbc, PromotionRules rules) {+ this.jdbc = jdbc;+ this.rules = rules;+ }++ @WithSpan("PricingEngine.quote")+ public Quote quote(QuoteRequest req) {+ List<String> skus = req.lines().stream().map(QuoteRequest.Line::sku).toList();+ Map<String, Long> prices = listPrices(req.priceListId(), skus);+ List<PromotionRules.Rule> active = jdbc.sql(+ "SELECT id, kind, priority, conditions, effect FROM promotion_rules WHERE active AND ? BETWEEN starts_at AND ends_at ORDER BY priority")+ .param(Instant.now())+ .query(PromotionRules.Rule.class)+ .list();+ List<Quote.Line> lines = new ArrayList<>();+ long total = 0;+ for (QuoteRequest.Line l : req.lines()) {+ long unit = "JPY".equals(req.currency()) ? Math.round(prices.getOrDefault(l.sku(), 0L) / 100.0) * 100 : prices.getOrDefault(l.sku(), 0L);+ long discount = rules.discount(active, l.sku(), l.qty(), unit, req.lines());+ lines.add(new Quote.Line(l.sku(), l.qty(), unit, discount));+ total += unit * l.qty() - discount;+ }+ return new Quote(req.currency(), lines, total);+ }++ private Map<String, Long> listPrices(long priceListId, List<String> skus) {+ return jdbc.sql("SELECT sku, list_price_minor FROM price_list_items WHERE price_list_id = :list AND sku IN (:skus)")+ .param("list", priceListId)+ .param("skus", skus)+ .query((rs, i) -> Map.entry(rs.getString(1), rs.getLong(2)))+ .list().stream()+ .collect(java.util.stream.Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/PromotionRules.java b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/PromotionRules.javanew file mode 100644index 00000000..a752d7b3--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/PromotionRules.java@@ -0,0 +1,48 @@+package com.example.pricing.engine;++import java.util.Comparator;+import java.util.List;+import org.springframework.stereotype.Component;++/**+ * Promotion rules engine: rules are evaluated in priority order; stackable+ * rules add up, the first exclusive rule that matches stops evaluation.+ */+@Component+public class PromotionRules {+ public record Rule(long id, String kind, int priority, String conditions, String effect) {+ boolean exclusive() {+ return conditions.contains("exclusive");+ }++ boolean matches(String sku, List<QuoteRequest.Line> cart) {+ if (conditions.contains("cart_min_lines:")) {+ int min = Integer.parseInt(conditions.replaceAll(".*cart_min_lines:(\\d+).*", "$1"));+ if (cart.size() < min) {+ return false;+ }+ }+ return conditions.contains("sku:" + sku) || conditions.contains("sku:*");+ }+ }++ public long discount(List<Rule> rules, String sku, int qty, long unitMinor, List<QuoteRequest.Line> cart) {+ long total = 0;+ for (Rule r : rules.stream().sorted(Comparator.comparingInt(Rule::priority)).toList()) {+ if (!r.matches(sku, cart)) {+ continue;+ }+ long d = switch (r.kind()) {+ case "percent_off" -> unitMinor * qty * Long.parseLong(r.effect().replace("%", "")) / 100;+ case "amount_off" -> Math.min(unitMinor * qty, Long.parseLong(r.effect()));+ case "bundle" -> cart.size() >= 2 ? Long.parseLong(r.effect()) : 0;+ default -> 0;+ };+ total += d;+ if (d > 0 && r.exclusive()) {+ break;+ }+ }+ return Math.min(total, unitMinor * qty);+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/Quote.java b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/Quote.javanew file mode 100644index 00000000..cb3abdf6--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/Quote.java@@ -0,0 +1,7 @@+package com.example.pricing.engine;++import java.util.List;++public record Quote(String currency, List<Line> lines, long totalMinor) {+ public record Line(String sku, int qty, long unitMinor, long discountMinor) {}+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/QuoteRequest.java b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/QuoteRequest.javanew file mode 100644index 00000000..9a4b48fd--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/java/com/example/pricing/engine/QuoteRequest.java@@ -0,0 +1,7 @@+package com.example.pricing.engine;++import java.util.List;++public record QuoteRequest(long priceListId, String currency, List<Line> lines) {+ public record Line(String sku, int qty) {}+}diff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/main/resources/application.yml b/cli/test/evals/testdata/repos/pricing-service-java/src/main/resources/application.ymlnew file mode 100644index 00000000..ed76e314--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/main/resources/application.yml@@ -0,0 +1,28 @@+spring:+ application:+ name: pricing-service+ main:+ # Beans are created on first use, so a new pod starts in a few seconds+ # and passes its startup probe quickly.+ lazy-initialization: true+ datasource:+ url: ${DATABASE_URL}+ username: ${DATABASE_USER}+ password: ${DATABASE_PASSWORD}+ hikari:+ maximum-pool-size: 10+ minimum-idle: 2++server:+ port: 8080+ shutdown: graceful++management:+ endpoints:+ web:+ exposure:+ include: health+ endpoint:+ health:+ probes:+ enabled: truediff --git a/cli/test/evals/testdata/repos/pricing-service-java/src/test/java/com/example/pricing/engine/PromotionRulesTest.java b/cli/test/evals/testdata/repos/pricing-service-java/src/test/java/com/example/pricing/engine/PromotionRulesTest.javanew file mode 100644index 00000000..d87e2439--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-java/src/test/java/com/example/pricing/engine/PromotionRulesTest.java@@ -0,0 +1,15 @@+package com.example.pricing.engine;++import static org.junit.jupiter.api.Assertions.assertEquals;++import java.util.List;+import org.junit.jupiter.api.Test;++class PromotionRulesTest {+ @Test+ void percentOff() {+ var rules = new PromotionRules();+ var rule = new PromotionRules.Rule(1, "percent_off", 1, "sku:SH-1", "10%");+ assertEquals(200, rules.discount(List.of(rule), "SH-1", 2, 1000, List.of(new QuoteRequest.Line("SH-1", 2))));+ }+}diff --git a/cli/test/evals/testdata/repos/pricing-service-py/README.md b/cli/test/evals/testdata/repos/pricing-service-py/README.mdnew file mode 100644index 00000000..9287d74f--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/README.md@@ -0,0 +1,16 @@+# pricing-service++Prices carts for the storefront and the wholesale portal: unit prices,+promotions (volume and bundle deals) and tax. FastAPI on Python 3.12,+PostgreSQL through psycopg 3. Traces, logs and metrics go to Traceway over+OTLP via `opentelemetry-instrument`.++```+python3 -m unittest discover -s tests+opentelemetry-instrument uvicorn app.main:app --port 8000+```++Routes:++- `POST /api/quotes` – price a cart `{"items": [{"sku": ..., "quantity": ...}]}`+- `GET /api/price-lists/{id}` – a price listdiff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/__init__.py b/cli/test/evals/testdata/repos/pricing-service-py/app/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/db.py b/cli/test/evals/testdata/repos/pricing-service-py/app/db.pynew file mode 100644index 00000000..de2cf863--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/app/db.py@@ -0,0 +1,5 @@+import os++from psycopg_pool import AsyncConnectionPool++pool = AsyncConnectionPool(os.environ.get("DATABASE_URL", "postgresql://pricing@localhost/pricing"), min_size=2, max_size=10, open=False)diff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/main.py b/cli/test/evals/testdata/repos/pricing-service-py/app/main.pynew file mode 100644index 00000000..8ec13aa4--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/app/main.py@@ -0,0 +1,52 @@+from fastapi import FastAPI, HTTPException+from opentelemetry import trace+from pydantic import BaseModel, Field++from .db import pool+from .pricing.models import Price, Promotion+from .pricing.quote import UnknownSku, build_quote++app = FastAPI(title="pricing-service")+++class QuoteItem(BaseModel):+ sku: str+ quantity: int = Field(gt=0)+++class QuoteRequest(BaseModel):+ items: list[QuoteItem] = Field(min_length=1)++[email protected]("/api/quotes")+async def create_quote(req: QuoteRequest):+ trace.get_current_span().set_attribute("cart.items", len(req.items))+ skus = [i.sku for i in req.items]+ async with pool.connection() as conn:+ rows = await (await conn.execute(+ "SELECT sku, family, unit_price_cents, tax_class FROM prices WHERE sku = ANY(%(skus)s)", {"skus": skus})).fetchall()+ promos = await (await conn.execute(+ "SELECT id, kind, family, sku, partner_sku, min_quantity, percent, amount_cents FROM promotions"+ " WHERE active AND starts_at <= now() AND ends_at > now()")).fetchall()+ prices = {r[0]: Price(*r) for r in rows}+ promotions = [Promotion(*p) for p in promos]+ try:+ quote = build_quote([i.model_dump() for i in req.items], prices, promotions)+ except UnknownSku as e:+ raise HTTPException(status_code=422, detail=f"unknown sku {e}")+ return {+ "lines": [l.__dict__ for l in quote.lines],+ "subtotalCents": quote.subtotal_cents,+ "discountCents": quote.discount_cents,+ "taxCents": quote.tax_cents,+ "totalCents": quote.total_cents,+ }++[email protected]("/api/price-lists/{price_list_id}")+async def price_list(price_list_id: int):+ async with pool.connection() as conn:+ rows = await (await conn.execute(+ "SELECT sku, family, unit_price_cents, tax_class FROM prices WHERE price_list_id = %(price_list_id)s ORDER BY sku LIMIT 200",+ {"price_list_id": price_list_id})).fetchall()+ return [Price(*r).__dict__ for r in rows]diff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/__init__.py b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/models.py b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/models.pynew file mode 100644index 00000000..19f7b88e--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/models.py@@ -0,0 +1,48 @@+from dataclasses import dataclass, field+++@dataclass+class Price:+ sku: str+ family: str+ unit_price_cents: int+ tax_class: str+++@dataclass+class Promotion:+ id: int+ kind: str # "volume" | "bundle"+ family: str | None = None+ sku: str | None = None+ partner_sku: str | None = None+ min_quantity: int = 0+ percent: int = 0+ amount_cents: int = 0+++@dataclass+class Line:+ sku: str+ family: str+ quantity: int+ unit_price_cents: int+ tax_class: str+ discount_cents: int = 0+ applied: list[int] = field(default_factory=list)++ @property+ def subtotal_cents(self) -> int:+ return self.unit_price_cents * self.quantity+++@dataclass+class Quote:+ lines: list[Line]+ subtotal_cents: int+ discount_cents: int+ tax_cents: int++ @property+ def total_cents(self) -> int:+ return self.subtotal_cents - self.discount_cents + self.tax_centsdiff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/promotions.py b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/promotions.pynew file mode 100644index 00000000..0f919aba--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/promotions.py@@ -0,0 +1,24 @@+from .models import Line, Promotion+++def apply_promotions(lines: list[Line], promotions: list[Promotion]) -> list[Line]:+ """Apply the best eligible promotion to every line.++ volume: a percentage off every line of a product family once the cart+ holds at least min_quantity units of that family.+ bundle: a fixed amount off each unit of `sku` when `partner_sku` is in+ the cart too.+ """+ for line in lines:+ family_quantity = sum(other.quantity for other in lines if other.family == line.family)+ for promo in promotions:+ if promo.kind == "volume" and promo.family == line.family and family_quantity >= promo.min_quantity:+ discount = line.subtotal_cents * promo.percent // 100+ elif promo.kind == "bundle" and promo.sku == line.sku and promo.partner_sku in [l.sku for l in lines]:+ discount = promo.amount_cents * line.quantity+ else:+ continue+ if discount > line.discount_cents:+ line.discount_cents = discount+ line.applied = [promo.id]+ return linesdiff --git a/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/quote.py b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/quote.pynew file mode 100644index 00000000..7ca15492--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/app/pricing/quote.py@@ -0,0 +1,34 @@+from opentelemetry import trace++from .models import Line, Price, Promotion, Quote+from .promotions import apply_promotions++TAX_RATES = {"standard": 20, "reduced": 10, "zero": 0}++tracer = trace.get_tracer("app.pricing")+++class UnknownSku(Exception):+ pass+++def build_quote(items: list[dict], prices: dict[str, Price], promotions: list[Promotion]) -> Quote:+ lines = []+ for item in items:+ price = prices.get(item["sku"])+ if price is None:+ raise UnknownSku(item["sku"])+ lines.append(Line(sku=price.sku, family=price.family, quantity=int(item["quantity"]),+ unit_price_cents=price.unit_price_cents, tax_class=price.tax_class))++ with tracer.start_as_current_span("apply_promotions") as span:+ span.set_attribute("promotions.active", len(promotions))+ apply_promotions(lines, promotions)++ with tracer.start_as_current_span("calculate_tax"):+ tax = sum((l.subtotal_cents - l.discount_cents) * TAX_RATES[l.tax_class] // 100 for l in lines)++ return Quote(lines=lines,+ subtotal_cents=sum(l.subtotal_cents for l in lines),+ discount_cents=sum(l.discount_cents for l in lines),+ tax_cents=tax)diff --git a/cli/test/evals/testdata/repos/pricing-service-py/pyproject.toml b/cli/test/evals/testdata/repos/pricing-service-py/pyproject.tomlnew file mode 100644index 00000000..9bafe8b0--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/pyproject.toml@@ -0,0 +1,14 @@+[project]+name = "pricing-service"+version = "3.8.1"+requires-python = ">=3.12"+dependencies = [+ "fastapi==0.118.0",+ "uvicorn[standard]==0.37.0",+ "psycopg[binary,pool]==3.2.10",+ "opentelemetry-distro==0.58b0",+ "opentelemetry-exporter-otlp==1.37.0",+ "opentelemetry-instrumentation-fastapi==0.58b0",+ "opentelemetry-instrumentation-psycopg==0.58b0",+ "opentelemetry-instrumentation-system-metrics==0.58b0",+]diff --git a/cli/test/evals/testdata/repos/pricing-service-py/tests/__init__.py b/cli/test/evals/testdata/repos/pricing-service-py/tests/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/pricing-service-py/tests/test_promotions.py b/cli/test/evals/testdata/repos/pricing-service-py/tests/test_promotions.pynew file mode 100644index 00000000..6e8a61a3--- /dev/null+++ b/cli/test/evals/testdata/repos/pricing-service-py/tests/test_promotions.py@@ -0,0 +1,38 @@+import unittest++from app.pricing.models import Line, Promotion+from app.pricing.promotions import apply_promotions+++def line(sku, family, qty, price=1000):+ return Line(sku=sku, family=family, quantity=qty, unit_price_cents=price, tax_class="standard")+++class ApplyPromotionsTest(unittest.TestCase):+ def test_volume_discount_needs_enough_units_of_the_family(self):+ promos = [Promotion(id=1, kind="volume", family="paper", min_quantity=10, percent=15)]+ few = apply_promotions([line("A4-500", "paper", 4), line("A3-250", "paper", 3)], promos)+ self.assertEqual([l.discount_cents for l in few], [0, 0])+ many = apply_promotions([line("A4-500", "paper", 6), line("A3-250", "paper", 5)], promos)+ self.assertEqual([l.discount_cents for l in many], [900, 750])++ def test_bundle_needs_the_partner_sku(self):+ promos = [Promotion(id=2, kind="bundle", sku="TONER-K", partner_sku="PRINTER-X1", amount_cents=500)]+ alone = apply_promotions([line("TONER-K", "toner", 2)], promos)+ self.assertEqual(alone[0].discount_cents, 0)+ both = apply_promotions([line("TONER-K", "toner", 2), line("PRINTER-X1", "printers", 1)], promos)+ self.assertEqual(both[0].discount_cents, 1000)+ self.assertEqual(both[0].applied, [2])++ def test_best_promotion_wins(self):+ promos = [+ Promotion(id=1, kind="volume", family="toner", min_quantity=2, percent=10),+ Promotion(id=2, kind="bundle", sku="TONER-K", partner_sku="PRINTER-X1", amount_cents=500),+ ]+ out = apply_promotions([line("TONER-K", "toner", 2), line("PRINTER-X1", "printers", 1)], promos)+ self.assertEqual(out[0].discount_cents, 1000)+ self.assertEqual(out[0].applied, [2])+++if __name__ == "__main__":+ unittest.main()diff --git a/cli/test/evals/testdata/repos/query-api-go/README.md b/cli/test/evals/testdata/repos/query-api-go/README.mdnew file mode 100644index 00000000..07fb8e35--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/README.md@@ -0,0 +1,12 @@+# query-api++Search for the document workspace: full-text query with in-process ranking+and highlighting, typeahead suggestions, document fetch. Go, `net/http`, PostgreSQL+(full-text index) through pgx. Two instances (`deploy/query-api.yaml`).+Traces, logs, metrics and host metrics (OTel Collector hostmetrics) go to+Traceway.++```+go test ./...+go run ./cmd/server+```diff --git a/cli/test/evals/testdata/repos/query-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/query-api-go/cmd/server/main.gonew file mode 100644index 00000000..771883aa--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/cmd/server/main.go@@ -0,0 +1,30 @@+package main++import (+ "log/slog"+ "net/http"+ "os"++ "example.com/query-api/internal/search"+)++func main() {+ h := search.NewHandlers(search.NewStore(nil))+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/query", h.Query)+ mux.HandleFunc("GET /api/suggest", h.Suggest)+ mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, mux); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/query-api-go/deploy/query-api.yaml b/cli/test/evals/testdata/repos/query-api-go/deploy/query-api.yamlnew file mode 100644index 00000000..ad840b73--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/deploy/query-api.yaml@@ -0,0 +1,11 @@+# Production deployment (applied by CI).+service: query-api+instances: 2+instance:+ type: c6i.large # 2 vCPU, 4 GiB+ goMaxProcs: 2+autoscaling:+ enabled: false+healthCheck:+ path: /healthz+ intervalSeconds: 10diff --git a/cli/test/evals/testdata/repos/query-api-go/go.mod b/cli/test/evals/testdata/repos/query-api-go/go.modnew file mode 100644index 00000000..cd71fffe--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/go.mod@@ -0,0 +1,3 @@+module example.com/query-api++go 1.26diff --git a/cli/test/evals/testdata/repos/query-api-go/internal/search/handlers.go b/cli/test/evals/testdata/repos/query-api-go/internal/search/handlers.gonew file mode 100644index 00000000..29186856--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/internal/search/handlers.go@@ -0,0 +1,55 @@+package search++import (+ "context"+ "encoding/json"+ "net/http"+ "strconv"+)++// Store runs the database side of search.+type Store interface {+ Candidates(ctx context.Context, tenantID int64, q string) ([]Hit, error)+ Suggest(ctx context.Context, tenantID int64, prefix string) ([]string, error)+}++// PgStore implements Store on pgx (wired in production builds).+type PgStore struct{ pool any }++// NewStore returns the PostgreSQL store.+func NewStore(pool any) *PgStore { return &PgStore{pool: pool} }++// Candidates runs the full-text query, top 200 by ts_rank_cd.+func (s *PgStore) Candidates(context.Context, int64, string) ([]Hit, error) { return nil, nil }++// Suggest returns the 10 most used terms with the prefix.+func (s *PgStore) Suggest(context.Context, int64, string) ([]string, error) { return nil, nil }++// Handlers serves the search routes.+type Handlers struct{ store Store }++// NewHandlers returns the handlers.+func NewHandlers(s Store) *Handlers { return &Handlers{store: s} }++// Query serves GET /api/query?q=.+func (h *Handlers) Query(w http.ResponseWriter, r *http.Request) {+ tenant, _ := strconv.ParseInt(r.Header.Get("X-Tenant-Id"), 10, 64)+ q := r.URL.Query().Get("q")+ hits, err := h.store.Candidates(r.Context(), tenant, q)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ _ = json.NewEncoder(w).Encode(Rank(q, hits))+}++// Suggest serves GET /api/suggest?prefix=.+func (h *Handlers) Suggest(w http.ResponseWriter, r *http.Request) {+ tenant, _ := strconv.ParseInt(r.Header.Get("X-Tenant-Id"), 10, 64)+ out, err := h.store.Suggest(r.Context(), tenant, r.URL.Query().Get("prefix"))+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ _ = json.NewEncoder(w).Encode(out)+}diff --git a/cli/test/evals/testdata/repos/query-api-go/internal/search/rank.go b/cli/test/evals/testdata/repos/query-api-go/internal/search/rank.gonew file mode 100644index 00000000..f6bd12f4--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/internal/search/rank.go@@ -0,0 +1,65 @@+// Package search serves the query, suggest and document routes.+package search++import (+ "math"+ "sort"+ "strings"+ "unicode"+)++// Hit is one candidate document from the full-text query.+type Hit struct {+ ID int64 `json:"id"`+ Title string `json:"title"`+ Snippet string `json:"snippet"`+ DBRank float64 `json:"-"`+ AgeDays float64 `json:"-"`+ Score float64 `json:"score"`+ Highlight string `json:"highlight"`+}++// Rank re-scores the database's top candidates: text rank blended with+// title matches and freshness, then the best 20.+func Rank(q string, hits []Hit) []Hit {+ terms := Terms(q)+ for i := range hits {+ h := &hits[i]+ title := strings.ToLower(h.Title)+ boost := 0.0+ for _, t := range terms {+ if strings.Contains(title, t) {+ boost += 0.35+ }+ }+ h.Score = h.DBRank * (1 + boost) * math.Exp(-h.AgeDays/365)+ }+ sort.SliceStable(hits, func(i, j int) bool { return hits[i].Score > hits[j].Score })+ if len(hits) > 20 {+ hits = hits[:20]+ }+ for i := range hits {+ hits[i].Highlight = Highlight(hits[i].Snippet, terms)+ }+ return hits+}++// Terms lowercases and splits a query.+func Terms(q string) []string {+ return strings.FieldsFunc(strings.ToLower(q), func(r rune) bool { return !unicode.IsLetter(r) && !unicode.IsDigit(r) })+}++// Highlight wraps query terms in <mark>.+func Highlight(s string, terms []string) string {+ words := strings.Fields(s)+ for i, w := range words {+ lw := strings.ToLower(strings.Trim(w, ".,;:!?\"'()"))+ for _, t := range terms {+ if lw == t {+ words[i] = "<mark>" + w + "</mark>"+ break+ }+ }+ }+ return strings.Join(words, " ")+}diff --git a/cli/test/evals/testdata/repos/query-api-go/internal/search/rank_test.go b/cli/test/evals/testdata/repos/query-api-go/internal/search/rank_test.gonew file mode 100644index 00000000..71325517--- /dev/null+++ b/cli/test/evals/testdata/repos/query-api-go/internal/search/rank_test.go@@ -0,0 +1,18 @@+package search++import "testing"++func TestRankPrefersTitleMatchesAndFreshness(t *testing.T) {+ hits := []Hit{+ {ID: 1, Title: "Quarterly planning", Snippet: "notes on the budget", DBRank: 0.5, AgeDays: 10},+ {ID: 2, Title: "Budget 2026", Snippet: "the budget for next year", DBRank: 0.5, AgeDays: 10},+ {ID: 3, Title: "Budget 2019", Snippet: "an old budget", DBRank: 0.5, AgeDays: 2400},+ }+ got := Rank("budget", hits)+ if got[0].ID != 2 || got[2].ID != 3 {+ t.Fatalf("order %d %d %d", got[0].ID, got[1].ID, got[2].ID)+ }+ if got[0].Highlight != "the <mark>budget</mark> for next year" {+ t.Fatalf("highlight %q", got[0].Highlight)+ }+}diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/README.md b/cli/test/evals/testdata/repos/shipping-quotes-py/README.mdnew file mode 100644index 00000000..e87a9564--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/README.md@@ -0,0 +1,12 @@+# quotes-api++Shipping quotes for checkout (FastAPI). `POST /api/shipping/quotes` loads the+parcel, asks the carrier's rate API for the available services and stores+the quote checkout shows.++Run with the OpenTelemetry distro so traces and logs reach Traceway:++```+opentelemetry-instrument uvicorn app.main:app --host 0.0.0.0 --port 8000+python -m unittest+```diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/__init__.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/__init__.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/client.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/client.pynew file mode 100644index 00000000..34d21354--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/client.py@@ -0,0 +1,41 @@+import logging++import httpx+from tenacity import (+ before_sleep_log,+ retry,+ retry_if_exception_type,+ stop_after_attempt,+ wait_exponential,+)++from app.carriers.rates import Rate, parse_rates++log = logging.getLogger("app.carriers")++RATES_URL = "https://rates.carrier.example/v2/rates"+++class CarrierClient:+ def __init__(self, api_key: str):+ self._http = httpx.AsyncClient(headers={"Authorization": f"Bearer {api_key}"})++ @retry(+ retry=retry_if_exception_type((httpx.TransportError, httpx.HTTPStatusError)),+ stop=stop_after_attempt(3),+ wait=wait_exponential(multiplier=1, min=1, max=4),+ before_sleep=before_sleep_log(log, logging.WARNING),+ )+ async def fetch_rates(self, parcel: dict) -> list[Rate]:+ payload = {+ "origin": parcel["origin_postcode"],+ "destination": {"postcode": parcel["destination_postcode"], "country": parcel["destination_country"]},+ "weight_grams": parcel["weight_grams"],+ "dimensions_mm": [parcel["length_mm"], parcel["width_mm"], parcel["height_mm"]],+ }+ response = await self._http.post(RATES_URL, json=payload)+ response.raise_for_status()+ return parse_rates(response.json())++ async def aclose(self) -> None:+ await self._http.aclose()diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/rates.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/rates.pynew file mode 100644index 00000000..436bb654--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/app/carriers/rates.py@@ -0,0 +1,27 @@+from dataclasses import dataclass+++@dataclass(frozen=True)+class Rate:+ carrier: str+ service_level: str+ amount_minor: int+ currency: str+ transit_days: int+++def parse_rates(body: dict) -> list[Rate]:+ return [+ Rate(+ carrier=r["carrier"],+ service_level=r["service"],+ amount_minor=int(r["amount"]["minor"]),+ currency=r["amount"]["currency"],+ transit_days=int(r["transit_days"]),+ )+ for r in body.get("rates", [])+ ]+++def cheapest(rates: list[Rate]) -> Rate | None:+ return min(rates, key=lambda r: (r.amount_minor, r.transit_days), default=None)diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/db.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/db.pynew file mode 100644index 00000000..5e9dbe1c--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/app/db.py@@ -0,0 +1,6 @@+import os++from psycopg.rows import dict_row+from psycopg_pool import AsyncConnectionPool++pool = AsyncConnectionPool(os.environ.get("DATABASE_URL", ""), kwargs={"row_factory": dict_row}, open=False)diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/main.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/main.pynew file mode 100644index 00000000..503eb3c1--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/app/main.py@@ -0,0 +1,22 @@+import logging+from contextlib import asynccontextmanager++from fastapi import FastAPI++from app.db import pool+from app.routes import quotes, tracking++logging.basicConfig(level=logging.INFO)+++@asynccontextmanager+async def lifespan(_: FastAPI):+ await pool.open()+ yield+ await quotes.carrier.aclose()+ await pool.close()+++app = FastAPI(title="quotes-api", lifespan=lifespan)+app.include_router(quotes.router)+app.include_router(tracking.router)diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/__init__.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/quotes.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/quotes.pynew file mode 100644index 00000000..ef7dcb49--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/quotes.py@@ -0,0 +1,36 @@+import os++from fastapi import APIRouter, HTTPException+from pydantic import BaseModel++from app.carriers.client import CarrierClient+from app.carriers.rates import cheapest+from app.db import pool++router = APIRouter()+carrier = CarrierClient(os.environ.get("CARRIER_API_KEY", ""))+++class QuoteRequest(BaseModel):+ parcel_id: int++[email protected]("/api/shipping/quotes")+async def create_quote(body: QuoteRequest):+ async with pool.connection() as conn:+ parcel = await (await conn.execute(+ "SELECT id, weight_grams, length_mm, width_mm, height_mm, origin_postcode, destination_postcode, destination_country FROM parcels WHERE id = %s",+ (body.parcel_id,),+ )).fetchone()+ if parcel is None:+ raise HTTPException(status_code=404, detail="parcel not found")+ rates = await carrier.fetch_rates(parcel)+ best = cheapest(rates)+ if best is None:+ raise HTTPException(status_code=422, detail="no carrier service for this parcel")+ async with pool.connection() as conn:+ await conn.execute(+ "INSERT INTO quotes (parcel_id, carrier, service_level, amount_minor, currency, expires_at) VALUES (%s, %s, %s, %s, %s, %s)",+ (parcel["id"], best.carrier, best.service_level, best.amount_minor, best.currency, None),+ )+ return {"carrier": best.carrier, "service": best.service_level, "amount_minor": best.amount_minor, "currency": best.currency, "rates": rates}diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/tracking.py b/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/tracking.pynew file mode 100644index 00000000..7d753d2d--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/app/routes/tracking.py@@ -0,0 +1,15 @@+from fastapi import APIRouter++from app.db import pool++router = APIRouter()++[email protected]("/api/shipments/{shipment_id}/tracking")+async def tracking(shipment_id: int):+ async with pool.connection() as conn:+ rows = await (await conn.execute(+ "SELECT status, location, occurred_at FROM tracking_events WHERE shipment_id = %s ORDER BY occurred_at DESC LIMIT 20",+ (shipment_id,),+ )).fetchall()+ return {"events": rows}diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/pyproject.toml b/cli/test/evals/testdata/repos/shipping-quotes-py/pyproject.tomlnew file mode 100644index 00000000..6ef907e6--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/pyproject.toml@@ -0,0 +1,19 @@+[project]+name = "quotes-api"+version = "2.3.0"+requires-python = ">=3.12"+dependencies = [+ "fastapi==0.118.0",+ "uvicorn[standard]==0.37.0",+ "httpx==0.28.1",+ "tenacity==9.1.2",+ "psycopg[binary,pool]==3.2.10",+ "opentelemetry-distro==0.58b0",+ "opentelemetry-exporter-otlp-proto-http==1.37.0",+ "opentelemetry-instrumentation-fastapi==0.58b0",+ "opentelemetry-instrumentation-httpx==0.58b0",+ "opentelemetry-instrumentation-psycopg==0.58b0",+]++[tool.uv]+dev-dependencies = []diff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/tests/__init__.py b/cli/test/evals/testdata/repos/shipping-quotes-py/tests/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/cli/test/evals/testdata/repos/shipping-quotes-py/tests/test_rates.py b/cli/test/evals/testdata/repos/shipping-quotes-py/tests/test_rates.pynew file mode 100644index 00000000..1bbd9128--- /dev/null+++ b/cli/test/evals/testdata/repos/shipping-quotes-py/tests/test_rates.py@@ -0,0 +1,24 @@+import unittest++from app.carriers.rates import Rate, cheapest, parse_rates+++class RatesTest(unittest.TestCase):+ def test_parse_and_pick_cheapest(self):+ body = {+ "rates": [+ {"carrier": "dhl", "service": "express", "amount": {"minor": 1490, "currency": "EUR"}, "transit_days": 1},+ {"carrier": "dpd", "service": "classic", "amount": {"minor": 690, "currency": "EUR"}, "transit_days": 3},+ {"carrier": "gls", "service": "business", "amount": {"minor": 690, "currency": "EUR"}, "transit_days": 2},+ ]+ }+ rates = parse_rates(body)+ self.assertEqual(len(rates), 3)+ self.assertEqual(cheapest(rates), Rate("gls", "business", 690, "EUR", 2))++ def test_no_rates(self):+ self.assertIsNone(cheapest(parse_rates({})))+++if __name__ == "__main__":+ unittest.main()diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/README.md b/cli/test/evals/testdata/repos/shop-web-and-api/README.mdnew file mode 100644index 00000000..e0ad1c25--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/README.md@@ -0,0 +1,11 @@+# shop++The web shop: `web/` is the React storefront (Vite, `@tracewayapp/react`,+Traceway project **shop-web**); `api/` is its backend (Node 24, Express,+OpenTelemetry, Traceway project **shop-api**). Product search goes through+the API to the hosted search index.++```+cd api && npm test+cd web && npm run build+```diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/api/package.json b/cli/test/evals/testdata/repos/shop-web-and-api/api/package.jsonnew file mode 100644index 00000000..346a4f17--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/api/package.json@@ -0,0 +1,13 @@+{+ "name": "shop-api",+ "version": "3.12.0",+ "private": true,+ "type": "module",+ "scripts": { "start": "node --import ./src/otel.js src/server.js", "test": "node --test" },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "0.64.1",+ "@opentelemetry/sdk-node": "0.205.0",+ "express": "5.1.0",+ "pg": "8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/api/src/otel.js b/cli/test/evals/testdata/repos/shop-web-and-api/api/src/otel.jsnew file mode 100644index 00000000..4e6b4cc7--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/api/src/otel.js@@ -0,0 +1,4 @@+import { NodeSDK } from '@opentelemetry/sdk-node'+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'++new NodeSDK({ instrumentations: [getNodeAutoInstrumentations()] }).start()diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/api/src/search.js b/cli/test/evals/testdata/repos/shop-web-and-api/api/src/search.jsnew file mode 100644index 00000000..676e1725--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/api/src/search.js@@ -0,0 +1,25 @@+// Product search: the hosted index answers the query, Postgres adds live+// stock and price.++const INDEX_URL = process.env.SEARCH_INDEX_URL ?? 'https://shop-prod.search-cloud.example/1/indexes/products/query'++export async function queryIndex(q, { fetchImpl = fetch } = {}) {+ const res = await fetchImpl(INDEX_URL, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json', 'X-Api-Key': process.env.SEARCH_API_KEY ?? '' },+ body: JSON.stringify({ query: q, hitsPerPage: 24 }),+ })+ if (!res.ok) throw new Error(`search index answered ${res.status}`)+ const body = await res.json()+ return body.hits.map((h) => ({ id: Number(h.objectID), name: h.name, slug: h.slug }))+}++export async function search(db, q, opts) {+ const hits = await queryIndex(q, opts)+ if (hits.length === 0) return { hits }+ const { rows } = await db.query('SELECT id, stock, price_cents FROM product_stock WHERE id = ANY($1)', [hits.map((h) => h.id)])+ const live = new Map(rows.map((r) => [r.id, r]))+ return {+ hits: hits.map((h) => ({ ...h, priceCents: live.get(h.id)?.price_cents ?? null, inStock: (live.get(h.id)?.stock ?? 0) > 0 })),+ }+}diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/api/src/server.js b/cli/test/evals/testdata/repos/shop-web-and-api/api/src/server.jsnew file mode 100644index 00000000..bfc1bebf--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/api/src/server.js@@ -0,0 +1,16 @@+import express from 'express'+import pg from 'pg'+import { search } from './search.js'++const db = new pg.Pool({ connectionString: process.env.DATABASE_URL, max: 20 })+const app = express()++app.get('/api/search', async (req, res, next) => {+ try {+ res.json(await search(db, String(req.query.q ?? '')))+ } catch (err) {+ next(err)+ }+})++app.listen(Number(process.env.PORT ?? 8080))diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/api/test/search.test.js b/cli/test/evals/testdata/repos/shop-web-and-api/api/test/search.test.jsnew file mode 100644index 00000000..02fe47c6--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/api/test/search.test.js@@ -0,0 +1,10 @@+import { test } from 'node:test'+import assert from 'node:assert/strict'+import { search } from '../src/search.js'++test('search merges index hits with live stock', async () => {+ const fetchImpl = async () => ({ ok: true, json: async () => ({ hits: [{ objectID: '7', name: 'Trail Runner', slug: 'trail-runner' }] }) })+ const db = { query: async () => ({ rows: [{ id: 7, stock: 3, price_cents: 8900 }] }) }+ const out = await search(db, 'trail', { fetchImpl })+ assert.deepEqual(out.hits, [{ id: 7, name: 'Trail Runner', slug: 'trail-runner', priceCents: 8900, inStock: true }])+})diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/web/package.json b/cli/test/evals/testdata/repos/shop-web-and-api/web/package.jsonnew file mode 100644index 00000000..06607d04--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/web/package.json@@ -0,0 +1,14 @@+{+ "name": "shop-web",+ "version": "2026.10.14",+ "private": true,+ "type": "module",+ "scripts": { "dev": "vite", "build": "vite build" },+ "dependencies": {+ "@tracewayapp/react": "1.4.2",+ "react": "19.1.1",+ "react-dom": "19.1.1",+ "react-router-dom": "7.9.3"+ },+ "devDependencies": { "vite": "7.1.7", "@vitejs/plugin-react": "5.0.4", "typescript": "5.9.3" }+}diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/web/src/App.tsx b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/App.tsxnew file mode 100644index 00000000..242d0b99--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/App.tsx@@ -0,0 +1,10 @@+import { Route, Routes } from 'react-router-dom'+import { SearchPage } from './pages/SearchPage'++export function App() {+ return (+ <Routes>+ <Route path="/search" element={<SearchPage />} />+ </Routes>+ )+}diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/web/src/api/search.ts b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/api/search.tsnew file mode 100644index 00000000..ec886236--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/api/search.ts@@ -0,0 +1,18 @@+export type SearchHit = { id: number; name: string; slug: string; priceCents: number; inStock: boolean }++// Shoppers stop waiting long before this; past it we show the+// "taking too long" message and let them retry.+export const SEARCH_TIMEOUT_MS = 8000++export async function searchProducts(q: string): Promise<SearchHit[]> {+ const params = new URLSearchParams({ q })+ const res = await fetch(`/api/search?${params}`, {+ headers: { Accept: 'application/json' },+ signal: AbortSignal.timeout(SEARCH_TIMEOUT_MS),+ })+ if (!res.ok) {+ throw new Error(`search failed: ${res.status}`)+ }+ const body = await res.json()+ return body.hits+}diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/web/src/main.tsx b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/main.tsxnew file mode 100644index 00000000..53a653ff--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/main.tsx@@ -0,0 +1,21 @@+import React from 'react'+import ReactDOM from 'react-dom/client'+import { BrowserRouter } from 'react-router-dom'+import { TracewayProvider } from '@tracewayapp/react'+import { App } from './App'++ReactDOM.createRoot(document.getElementById('root')!).render(+ <TracewayProvider+ connectionString={import.meta.env.VITE_TRACEWAY_CONNECTION_STRING}+ options={{+ version: '2026.10.14',+ // Keep the defaults except timeouts: a search that times out is a+ // shopper who gave up, and we want to see those.+ ignoreErrors: ['ResizeObserver loop', 'Failed to fetch', 'Load failed', 'NetworkError when attempting to fetch resource'],+ }}+ >+ <BrowserRouter>+ <App />+ </BrowserRouter>+ </TracewayProvider>,+)diff --git a/cli/test/evals/testdata/repos/shop-web-and-api/web/src/pages/SearchPage.tsx b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/pages/SearchPage.tsxnew file mode 100644index 00000000..5a325fb5--- /dev/null+++ b/cli/test/evals/testdata/repos/shop-web-and-api/web/src/pages/SearchPage.tsx@@ -0,0 +1,36 @@+import { useState } from 'react'+import { useTraceway } from '@tracewayapp/react'+import { searchProducts, type SearchHit } from '../api/search'++export function SearchPage() {+ const { captureException } = useTraceway()+ const [hits, setHits] = useState<SearchHit[]>([])+ const [state, setState] = useState<'idle' | 'loading' | 'slow' | 'error'>('idle')++ async function runSearch(q: string) {+ setState('loading')+ try {+ setHits(await searchProducts(q))+ setState('idle')+ } catch (err) {+ if (err instanceof DOMException && err.name === 'TimeoutError') {+ console.warn(`search for "${q}" gave up after 8000ms`)+ setState('slow')+ } else {+ setState('error')+ }+ captureException(err as Error)+ }+ }++ return (+ <main>+ <form onSubmit={(e) => { e.preventDefault(); runSearch(new FormData(e.currentTarget).get('q') as string) }}>+ <input name="q" placeholder="Search products" />+ </form>+ {state === 'slow' && <p>Search is taking too long. Please try again.</p>}+ {state === 'error' && <p>Search is unavailable right now.</p>}+ <ul>{hits.map((h) => <li key={h.id}>{h.name}</li>)}</ul>+ </main>+ )+}diff --git a/cli/test/evals/testdata/repos/store-locator-go/README.md b/cli/test/evals/testdata/repos/store-locator-go/README.mdnew file mode 100644index 00000000..22396de2--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/README.md@@ -0,0 +1,18 @@+# store-locator-api++Store finder for the shop (Go, `net/http`).+`GET /api/stores/nearby?q=<postcode or address>` geocodes the query with the+maps provider, loads the closest stores from PostGIS and asks the provider for+driving distances.++- `internal/maps`: maps provider client (geocode, distance matrix)+- `internal/inventory`: client for the internal inventory service+- `internal/stores`: HTTP handlers++Outgoing HTTP is traced with otelhttp plus otelhttptrace (connection, DNS,+TLS timings), installed through `maps.Instrument` in production builds.++```+go test ./...+go run ./cmd/server+```diff --git a/cli/test/evals/testdata/repos/store-locator-go/cmd/server/finder.go b/cli/test/evals/testdata/repos/store-locator-go/cmd/server/finder.gonew file mode 100644index 00000000..496a9c71--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/cmd/server/finder.go@@ -0,0 +1,18 @@+package main++import (+ "context"++ "example.com/store-locator/internal/maps"+ "example.com/store-locator/internal/stores"+)++// postgisFinder is the PostGIS-backed finder in production builds; this+// stand-in keeps the module dependency-free.+type postgisFinder struct{ dsn string }++func newFinder(dsn string) *postgisFinder { return &postgisFinder{dsn: dsn} }++func (f *postgisFinder) Near(ctx context.Context, p maps.Point, limit int) ([]stores.Store, error) {+ return nil, nil+}diff --git a/cli/test/evals/testdata/repos/store-locator-go/cmd/server/main.go b/cli/test/evals/testdata/repos/store-locator-go/cmd/server/main.gonew file mode 100644index 00000000..46281282--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/cmd/server/main.go@@ -0,0 +1,35 @@+package main++import (+ "log/slog"+ "net/http"+ "os"++ "example.com/store-locator/internal/inventory"+ "example.com/store-locator/internal/maps"+ "example.com/store-locator/internal/stores"+)++func main() {+ h := &stores.Handler{+ Maps: maps.New(envOr("MAPS_URL", "https://maps.geo.example"), os.Getenv("MAPS_API_KEY")),+ Inventory: inventory.New(envOr("INVENTORY_URL", "http://inventory.internal:8080"), maps.Instrument(http.DefaultTransport)),+ Stores: newFinder(os.Getenv("DATABASE_URL")),+ }+ mux := http.NewServeMux()+ mux.HandleFunc("GET /api/stores/nearby", h.Nearby)+ mux.HandleFunc("GET /api/stores/{id}/availability", h.Availability)+ addr := ":" + envOr("PORT", "8080")+ slog.Info("listening", "addr", addr)+ if err := http.ListenAndServe(addr, mux); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func envOr(k, def string) string {+ if v := os.Getenv(k); v != "" {+ return v+ }+ return def+}diff --git a/cli/test/evals/testdata/repos/store-locator-go/go.mod b/cli/test/evals/testdata/repos/store-locator-go/go.modnew file mode 100644index 00000000..01f6f7d6--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/go.mod@@ -0,0 +1,3 @@+module example.com/store-locator++go 1.26diff --git a/cli/test/evals/testdata/repos/store-locator-go/internal/inventory/client.go b/cli/test/evals/testdata/repos/store-locator-go/internal/inventory/client.gonew file mode 100644index 00000000..910d2986--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/internal/inventory/client.go@@ -0,0 +1,36 @@+// Package inventory reads store stock from the internal inventory service.+package inventory++import (+ "context"+ "encoding/json"+ "fmt"+ "net/http"+ "time"+)++// Client is the inventory client.+type Client struct {+ baseURL string+ http *http.Client+}++// New returns a client using rt (the instrumented default transport).+func New(baseURL string, rt http.RoundTripper) *Client {+ return &Client{baseURL: baseURL, http: &http.Client{Timeout: 2 * time.Second, Transport: rt}}+}++// Stock returns the units on hand per SKU for one store.+func (c *Client) Stock(ctx context.Context, storeID int) (map[string]int, error) {+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, fmt.Sprintf("%s/v2/stock?store=%d", c.baseURL, storeID), nil)+ if err != nil {+ return nil, err+ }+ resp, err := c.http.Do(req)+ if err != nil {+ return nil, err+ }+ defer resp.Body.Close()+ var out map[string]int+ return out, json.NewDecoder(resp.Body).Decode(&out)+}diff --git a/cli/test/evals/testdata/repos/store-locator-go/internal/maps/client.go b/cli/test/evals/testdata/repos/store-locator-go/internal/maps/client.gonew file mode 100644index 00000000..e640b6e1--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/internal/maps/client.go@@ -0,0 +1,96 @@+// Package maps is the client for the maps provider (geocoding and driving+// distances).+package maps++import (+ "context"+ "crypto/tls"+ "encoding/json"+ "fmt"+ "net/http"+ "net/url"+ "strings"+ "time"+)++// Instrument wraps a transport with tracing (otelhttp.NewTransport with+// otelhttptrace client traces in production builds).+var Instrument = func(rt http.RoundTripper) http.RoundTripper { return rt }++// Client calls the maps provider.+type Client struct {+ baseURL string+ apiKey string+}++// New returns a client for the provider at baseURL.+func New(baseURL, apiKey string) *Client {+ return &Client{baseURL: strings.TrimRight(baseURL, "/"), apiKey: apiKey}+}++// Point is a WGS84 coordinate.+type Point struct {+ Lat float64 `json:"lat"`+ Lng float64 `json:"lng"`+}++// Geocode resolves a free-text query (postcode, address) to a point.+func (c *Client) Geocode(ctx context.Context, q string) (Point, error) {+ var out struct {+ Results []struct {+ Location Point `json:"location"`+ } `json:"results"`+ }+ if err := c.get(ctx, "/v1/geocode?q="+url.QueryEscape(q)+"®ion=de", &out); err != nil {+ return Point{}, err+ }+ if len(out.Results) == 0 {+ return Point{}, fmt.Errorf("maps: no result for %q", q)+ }+ return out.Results[0].Location, nil+}++// Distances returns the driving distance in meters from origin to each+// destination, in order.+func (c *Client) Distances(ctx context.Context, origin Point, destinations []string) ([]int, error) {+ var out struct {+ Rows []struct {+ Meters int `json:"meters"`+ } `json:"rows"`+ }+ path := fmt.Sprintf("/v1/distancematrix?origins=%.4f,%.4f&destinations=%s&mode=driving",+ origin.Lat, origin.Lng, url.QueryEscape(strings.Join(destinations, "|")))+ if err := c.get(ctx, path, &out); err != nil {+ return nil, err+ }+ meters := make([]int, len(out.Rows))+ for i, r := range out.Rows {+ meters[i] = r.Meters+ }+ return meters, nil+}++func (c *Client) get(ctx context.Context, path string, into any) error {+ hc := &http.Client{+ Timeout: 3 * time.Second,+ Transport: Instrument(&http.Transport{+ Proxy: http.ProxyFromEnvironment,+ TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12},+ TLSHandshakeTimeout: 2 * time.Second,+ }),+ }+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+path, nil)+ if err != nil {+ return err+ }+ req.Header.Set("X-Api-Key", c.apiKey)+ resp, err := hc.Do(req)+ if err != nil {+ return err+ }+ defer resp.Body.Close()+ if resp.StatusCode != http.StatusOK {+ return fmt.Errorf("maps: %s answered %d", path, resp.StatusCode)+ }+ return json.NewDecoder(resp.Body).Decode(into)+}diff --git a/cli/test/evals/testdata/repos/store-locator-go/internal/maps/client_test.go b/cli/test/evals/testdata/repos/store-locator-go/internal/maps/client_test.gonew file mode 100644index 00000000..44d99699--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/internal/maps/client_test.go@@ -0,0 +1,32 @@+package maps++import (+ "context"+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "testing"+)++func TestGeocodeAndDistances(t *testing.T) {+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ switch r.URL.Path {+ case "/v1/geocode":+ _ = json.NewEncoder(w).Encode(map[string]any{"results": []any{map[string]any{"location": map[string]float64{"lat": 52.53, "lng": 13.38}}}})+ case "/v1/distancematrix":+ _ = json.NewEncoder(w).Encode(map[string]any{"rows": []any{map[string]int{"meters": 1200}, map[string]int{"meters": 3400}}})+ default:+ http.NotFound(w, r)+ }+ }))+ defer srv.Close()+ c := New(srv.URL, "k")+ p, err := c.Geocode(context.Background(), "10115 Berlin")+ if err != nil || p.Lat != 52.53 {+ t.Fatalf("geocode: %v %v", p, err)+ }+ m, err := c.Distances(context.Background(), p, []string{"store:1", "store:2"})+ if err != nil || len(m) != 2 || m[1] != 3400 {+ t.Fatalf("distances: %v %v", m, err)+ }+}diff --git a/cli/test/evals/testdata/repos/store-locator-go/internal/stores/handler.go b/cli/test/evals/testdata/repos/store-locator-go/internal/stores/handler.gonew file mode 100644index 00000000..ef121a1e--- /dev/null+++ b/cli/test/evals/testdata/repos/store-locator-go/internal/stores/handler.go@@ -0,0 +1,86 @@+// Package stores serves the store finder.+package stores++import (+ "context"+ "encoding/json"+ "fmt"+ "net/http"+ "strconv"++ "example.com/store-locator/internal/inventory"+ "example.com/store-locator/internal/maps"+)++// Store is one shop location.+type Store struct {+ ID int `json:"id"`+ Name string `json:"name"`+ Address string `json:"address"`+ Lat float64 `json:"lat"`+ Lng float64 `json:"lng"`+ Meters int `json:"meters"`+}++// Finder looks up stores near a point (PostGIS in production).+type Finder interface {+ Near(ctx context.Context, p maps.Point, limit int) ([]Store, error)+}++// Handler serves /api/stores.+type Handler struct {+ Maps *maps.Client+ Inventory *inventory.Client+ Stores Finder+}++// Availability serves GET /api/stores/{id}/availability.+func (h *Handler) Availability(w http.ResponseWriter, r *http.Request) {+ id, err := strconv.Atoi(r.PathValue("id"))+ if err != nil {+ http.Error(w, "bad store id", http.StatusBadRequest)+ return+ }+ stock, err := h.Inventory.Stock(r.Context(), id)+ if err != nil {+ http.Error(w, "inventory unavailable", http.StatusBadGateway)+ return+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(stock)+}++// Nearby serves GET /api/stores/nearby?q=...+func (h *Handler) Nearby(w http.ResponseWriter, r *http.Request) {+ q := r.URL.Query().Get("q")+ if q == "" {+ http.Error(w, "q is required", http.StatusBadRequest)+ return+ }+ origin, err := h.Maps.Geocode(r.Context(), q)+ if err != nil {+ http.Error(w, "could not resolve the location", http.StatusBadGateway)+ return+ }+ stores, err := h.Stores.Near(r.Context(), origin, 10)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ dest := make([]string, len(stores))+ for i, s := range stores {+ dest[i] = fmt.Sprintf("store:%d", s.ID)+ }+ if len(dest) > 0 {+ meters, err := h.Maps.Distances(r.Context(), origin, dest)+ if err == nil {+ for i := range stores {+ if i < len(meters) {+ stores[i].Meters = meters[i]+ }+ }+ }+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(stores)+}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/README.md b/cli/test/evals/testdata/repos/storefront-api-nest/README.mdnew file mode 100644index 00000000..b448e46b--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/README.md@@ -0,0 +1,14 @@+# storefront-api++The API behind the storefront web app: customer, catalog and cart. Node 24,+NestJS 11 on Express, PostgreSQL through `pg`, Redis through `ioredis`.+Telemetry goes to Traceway over OTLP with+`@opentelemetry/auto-instrumentations-node`.++```+npm install+npm run build && npm start+```++Feature flags live in the `feature_flags` table and are edited from the+back office; changes should reach the API within a minute.diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/history.yaml b/cli/test/evals/testdata/repos/storefront-api-nest/history.yamlnew file mode 100644index 00000000..cf8a61be--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/history.yaml@@ -0,0 +1,4 @@+commits: []+head:+ message: 'storefront-api: customer, catalog, cart and feature flags'+ age: 1900hdiff --git a/cli/test/evals/testdata/repos/storefront-api-nest/package.json b/cli/test/evals/testdata/repos/storefront-api-nest/package.jsonnew file mode 100644index 00000000..42d769dc--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/package.json@@ -0,0 +1,23 @@+{+ "name": "storefront-api",+ "version": "7.4.2",+ "private": true,+ "scripts": {+ "build": "nest build",+ "start": "node --require @opentelemetry/auto-instrumentations-node/register dist/main.js"+ },+ "dependencies": {+ "@nestjs/common": "^11.1.6",+ "@nestjs/core": "^11.1.6",+ "@nestjs/platform-express": "^11.1.6",+ "@opentelemetry/auto-instrumentations-node": "^0.64.1",+ "ioredis": "^5.8.0",+ "pg": "^8.16.3",+ "reflect-metadata": "^0.2.2",+ "rxjs": "^7.8.2"+ },+ "devDependencies": {+ "@nestjs/cli": "^11.0.10",+ "typescript": "^5.9.3"+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/app.module.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/app.module.tsnew file mode 100644index 00000000..4ebb34b2--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/app.module.ts@@ -0,0 +1,22 @@+import { Module } from '@nestjs/common';+import { APP_GUARD } from '@nestjs/core';+import { Pool } from 'pg';+import Redis from 'ioredis';+import { FeatureFlagsGuard } from './flags/feature-flags.guard';+import { FeatureFlagsService } from './flags/feature-flags.service';+import { MeController } from './me/me.controller';+import { ProductsController } from './catalog/products.controller';+import { CartController } from './cart/cart.controller';++@Module({+ controllers: [MeController, ProductsController, CartController],+ providers: [+ { provide: Pool, useValue: new Pool({ connectionString: process.env.DATABASE_URL, max: 20 }) },+ { provide: Redis, useValue: new Redis(process.env.REDIS_URL ?? 'redis://localhost:6379') },+ FeatureFlagsService,+ // Every request gets the current flags on req.flags, so handlers and+ // templates can branch on them.+ { provide: APP_GUARD, useClass: FeatureFlagsGuard },+ ],+})+export class AppModule {}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/cart/cart.controller.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/cart/cart.controller.tsnew file mode 100644index 00000000..d4126a2c--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/cart/cart.controller.ts@@ -0,0 +1,25 @@+import { Body, Controller, Get, Headers, Post } from '@nestjs/common';+import { Pool } from 'pg';++@Controller('api/cart')+export class CartController {+ constructor(private readonly db: Pool) {}++ @Get()+ async get(@Headers('x-cart-id') cartId: string) {+ const { rows } = await this.db.query(+ 'SELECT l.sku, l.qty, l.unit_price_cents FROM cart_lines l WHERE l.cart_id = $1',+ [cartId],+ );+ return { lines: rows, totalCents: rows.reduce((s, l) => s + l.qty * l.unit_price_cents, 0) };+ }++ @Post('lines')+ async addLine(@Headers('x-cart-id') cartId: string, @Body() body: { sku: string; qty: number; unitPriceCents: number }) {+ await this.db.query(+ 'INSERT INTO cart_lines (cart_id, sku, qty, unit_price_cents) VALUES ($1, $2, $3, $4) ON CONFLICT (cart_id, sku) DO UPDATE SET qty = cart_lines.qty + EXCLUDED.qty',+ [cartId, body.sku, body.qty, body.unitPriceCents],+ );+ return { ok: true };+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/catalog/products.controller.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/catalog/products.controller.tsnew file mode 100644index 00000000..a59be64e--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/catalog/products.controller.ts@@ -0,0 +1,40 @@+import { Controller, Get, NotFoundException, Param } from '@nestjs/common';+import Redis from 'ioredis';+import { Pool } from 'pg';++const TTL_SECONDS = 300;++@Controller('api')+export class ProductsController {+ constructor(+ private readonly db: Pool,+ private readonly redis: Redis,+ ) {}++ @Get('products/:id')+ async findOne(@Param('id') id: string) {+ const key = `product:${id}`;+ const hit = await this.redis.get(key);+ if (hit) return JSON.parse(hit);+ const { rows } = await this.db.query(+ 'SELECT id, slug, name, price_cents, currency, images, attributes FROM products WHERE id = $1',+ [id],+ );+ if (!rows[0]) throw new NotFoundException();+ await this.redis.set(key, JSON.stringify(rows[0]), 'EX', TTL_SECONDS);+ return rows[0];+ }++ @Get('categories/:slug/products')+ async byCategory(@Param('slug') slug: string) {+ const key = `category:${slug}:page:1`;+ const hit = await this.redis.get(key);+ if (hit) return JSON.parse(hit);+ const { rows } = await this.db.query(+ 'SELECT id, slug, name, price_cents, currency, thumbnail_url FROM products WHERE category_id = $1 ORDER BY rank LIMIT 48',+ [slug],+ );+ await this.redis.set(key, JSON.stringify(rows), 'EX', TTL_SECONDS);+ return rows;+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/flags/feature-flags.guard.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/flags/feature-flags.guard.tsnew file mode 100644index 00000000..e5ab301f--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/flags/feature-flags.guard.ts@@ -0,0 +1,16 @@+import { CanActivate, ExecutionContext, Injectable } from '@nestjs/common';+import { FeatureFlagsService } from './feature-flags.service';++@Injectable()+export class FeatureFlagsGuard implements CanActivate {+ constructor(private readonly flags: FeatureFlagsService) {}++ async canActivate(ctx: ExecutionContext): Promise<boolean> {+ const req = ctx.switchToHttp().getRequest();+ req.flags = await this.flags.load();+ if (req.flags.get('storefront.maintenance')?.enabled && !req.path.startsWith('/api/status')) {+ return false;+ }+ return true;+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/flags/feature-flags.service.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/flags/feature-flags.service.tsnew file mode 100644index 00000000..8871a411--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/flags/feature-flags.service.ts@@ -0,0 +1,26 @@+import { Injectable } from '@nestjs/common';+import { Pool } from 'pg';++export interface FeatureFlag {+ key: string;+ enabled: boolean;+ rolloutPercent: number;+ rules: unknown;+}++export type FlagSet = Map<string, FeatureFlag>;++@Injectable()+export class FeatureFlagsService {+ constructor(private readonly db: Pool) {}++ /** The active flags, read from the database. */+ async load(): Promise<FlagSet> {+ const { rows } = await this.db.query(+ 'SELECT key, enabled, rollout_percent, rules FROM feature_flags WHERE archived_at IS NULL ORDER BY key',+ );+ return new Map(+ rows.map((r) => [r.key, { key: r.key, enabled: r.enabled, rolloutPercent: r.rollout_percent, rules: r.rules }]),+ );+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/main.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/main.tsnew file mode 100644index 00000000..c1fabc3f--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/main.ts@@ -0,0 +1,9 @@+import 'reflect-metadata';+import { NestFactory } from '@nestjs/core';+import { AppModule } from './app.module';++async function bootstrap() {+ const app = await NestFactory.create(AppModule);+ await app.listen(Number(process.env.PORT ?? 8080));+}+bootstrap();diff --git a/cli/test/evals/testdata/repos/storefront-api-nest/src/me/me.controller.ts b/cli/test/evals/testdata/repos/storefront-api-nest/src/me/me.controller.tsnew file mode 100644index 00000000..da60ec91--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-nest/src/me/me.controller.ts@@ -0,0 +1,23 @@+import { Controller, Get, Req, UnauthorizedException } from '@nestjs/common';+import { Pool } from 'pg';++@Controller('api/me')+export class MeController {+ constructor(private readonly db: Pool) {}++ @Get()+ async getMe(@Req() req: { customerId?: number; flags: Map<string, { enabled: boolean }> }) {+ if (!req.customerId) throw new UnauthorizedException();+ const { rows } = await this.db.query(+ 'SELECT id, email, first_name, last_name, loyalty_tier FROM customers WHERE id = $1',+ [req.customerId],+ );+ const me = rows[0];+ return {+ id: me.id,+ email: me.email,+ name: `${me.first_name} ${me.last_name}`,+ loyaltyTier: req.flags.get('loyalty.tiers')?.enabled ? me.loyalty_tier : null,+ };+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-node/README.md b/cli/test/evals/testdata/repos/storefront-api-node/README.mdnew file mode 100644index 00000000..2bf92fce--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-node/README.md@@ -0,0 +1,12 @@+# storefront-api++The web shop's API: product search, product pages, cart and checkout.+Node 24, Express 5, PostgreSQL through `pg`. Traces, logs and metrics go to+Traceway over OTLP via the OpenTelemetry Node SDK (`src/otel.js`).++```+npm test+npm start+```++The products table holds about 25k active products.diff --git a/cli/test/evals/testdata/repos/storefront-api-node/package.json b/cli/test/evals/testdata/repos/storefront-api-node/package.jsonnew file mode 100644index 00000000..6a64cf12--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-node/package.json@@ -0,0 +1,17 @@+{+ "name": "storefront-api",+ "version": "1.43.0",+ "private": true,+ "type": "module",+ "main": "src/server.js",+ "scripts": {+ "start": "node --import ./src/otel.js src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "0.64.1",+ "@opentelemetry/sdk-node": "0.205.0",+ "express": "5.1.0",+ "pg": "8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-node/src/otel.js b/cli/test/evals/testdata/repos/storefront-api-node/src/otel.jsnew file mode 100644index 00000000..ddb6e34c--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-node/src/otel.js@@ -0,0 +1,5 @@+import { NodeSDK } from '@opentelemetry/sdk-node'+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'++// Endpoint and headers come from OTEL_EXPORTER_OTLP_* in the environment.+new NodeSDK({ instrumentations: [getNodeAutoInstrumentations()] }).start()diff --git a/cli/test/evals/testdata/repos/storefront-api-node/src/search.js b/cli/test/evals/testdata/repos/storefront-api-node/src/search.jsnew file mode 100644index 00000000..1c628c8e--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-node/src/search.js@@ -0,0 +1,25 @@+// Product search: a name/brand match ranked by popularity, plus the facet+// counts the filter sidebar shows.++export const PAGE_SIZE = 24++export function searchPattern(q) {+ const term = String(q ?? '').trim().slice(0, 80)+ if (term.length < 2) return null+ return '%' + term.replace(/[\\%_]/g, (c) => '\\' + c) + '%'+}++export async function search(db, q) {+ const pattern = searchPattern(q)+ if (!pattern) return { products: [], facets: [] }+ const { rows: products } = await db.query(+ 'SELECT id, name, slug, brand, price_cents, thumbnail_url FROM products WHERE active AND (name ILIKE $1 OR brand ILIKE $1) ORDER BY popularity DESC LIMIT 24',+ [pattern],+ )+ if (products.length === 0) return { products, facets: [] }+ const { rows: facets } = await db.query(+ 'SELECT facet, value, count(*) AS n FROM product_facets WHERE product_id = ANY($1) GROUP BY facet, value',+ [products.map((p) => p.id)],+ )+ return { products, facets }+}diff --git a/cli/test/evals/testdata/repos/storefront-api-node/src/server.js b/cli/test/evals/testdata/repos/storefront-api-node/src/server.jsnew file mode 100644index 00000000..e84eacb8--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-node/src/server.js@@ -0,0 +1,28 @@+import express from 'express'+import pg from 'pg'+import { search } from './search.js'++const db = new pg.Pool({ connectionString: process.env.DATABASE_URL, max: 20 })+const app = express()+app.use(express.json())++app.get('/api/search', async (req, res, next) => {+ try {+ res.json(await search(db, req.query.q))+ } catch (err) {+ next(err)+ }+})++app.get('/api/products/:id', async (req, res, next) => {+ try {+ const { rows } = await db.query('SELECT id, name, slug, brand, description, price_cents, images FROM products WHERE id = $1 AND active', [req.params.id])+ if (rows.length === 0) return res.sendStatus(404)+ const { rows: variants } = await db.query('SELECT sku, size, color, stock FROM product_variants WHERE product_id = $1', [rows[0].id])+ res.json({ ...rows[0], variants })+ } catch (err) {+ next(err)+ }+})++app.listen(Number(process.env.PORT ?? 8080))diff --git a/cli/test/evals/testdata/repos/storefront-api-node/test/search.test.js b/cli/test/evals/testdata/repos/storefront-api-node/test/search.test.jsnew file mode 100644index 00000000..7fc95e8a--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-api-node/test/search.test.js@@ -0,0 +1,23 @@+import { test } from 'node:test'+import assert from 'node:assert/strict'+import { search, searchPattern } from '../src/search.js'++test('searchPattern escapes LIKE wildcards and ignores one-letter queries', () => {+ assert.equal(searchPattern('a'), null)+ assert.equal(searchPattern(' 50%_off '), '%50\\%\\_off%')+})++test('search returns products with their facet counts', async () => {+ const calls = []+ const db = {+ async query(sql, params) {+ calls.push(sql)+ if (sql.includes('FROM products')) return { rows: [{ id: 1, name: 'Trail Runner' }, { id: 2, name: 'Trail Boot' }] }+ return { rows: [{ facet: 'size', value: '42', n: 2 }] }+ },+ }+ const out = await search(db, 'trail')+ assert.equal(out.products.length, 2)+ assert.deepEqual(out.facets, [{ facet: 'size', value: '42', n: 2 }])+ assert.equal(calls.length, 2)+})diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/README.md b/cli/test/evals/testdata/repos/storefront-gateway-node/README.mdnew file mode 100644index 00000000..09a265dc--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/README.md@@ -0,0 +1,19 @@+# storefront-gateway++The storefront's API gateway (Express). It owns the session and composes the+pages from the services behind it:++| Service | Project | Used for |+|---|---|---|+| cart-service | storefront | cart lines |+| catalog-service | catalog | product title, brand, image |+| pricing-service | catalog | current price per currency |+| inventory-service | fulfillment | stock per warehouse region |++Clients live in `src/clients`, routes in `src/routes`. Traces go to Traceway+over OTLP (`src/tracing.js`).++```+npm test+npm start+```diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/package.json b/cli/test/evals/testdata/repos/storefront-gateway-node/package.jsonnew file mode 100644index 00000000..420940fc--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/package.json@@ -0,0 +1,18 @@+{+ "name": "storefront-gateway",+ "version": "4.18.2",+ "private": true,+ "main": "src/server.js",+ "scripts": {+ "start": "node --require ./src/tracing.js src/server.js",+ "test": "node --test"+ },+ "dependencies": {+ "@opentelemetry/auto-instrumentations-node": "^0.62.0",+ "@opentelemetry/exporter-trace-otlp-proto": "^0.203.0",+ "@opentelemetry/sdk-node": "^0.203.0",+ "express": "^5.1.0",+ "ioredis": "^5.7.0",+ "pg": "^8.16.3"+ }+}diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/clients/http.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/clients/http.jsnew file mode 100644index 00000000..53bd4ed0--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/clients/http.js@@ -0,0 +1,15 @@+'use strict';++async function request(base, method, path, body) {+ const res = await fetch(base + path, {+ method,+ headers: { accept: 'application/json', ...(body ? { 'content-type': 'application/json' } : {}) },+ body: body ? JSON.stringify(body) : undefined,+ });+ if (!res.ok) {+ throw new Error(`${method} ${base}${path} answered ${res.status}`);+ }+ return res.json();+}++module.exports = { request };diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/clients/index.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/clients/index.jsnew file mode 100644index 00000000..e627a294--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/clients/index.js@@ -0,0 +1,32 @@+'use strict';+const { request } = require('./http');++const env = process.env;+const CART = env.CART_URL || 'http://cart-service.internal:8080';+const CATALOG = env.CATALOG_URL || 'http://catalog-service.internal:8080';+const PRICING = env.PRICING_URL || 'http://pricing-service.internal:8080';+const INVENTORY = env.INVENTORY_URL || 'http://inventory-service.internal:8080';++const carts = {+ get: (cartId) => request(CART, 'GET', `/carts/${encodeURIComponent(cartId)}`),+};++const catalog = {+ get: (sku) => request(CATALOG, 'GET', `/products/${encodeURIComponent(sku)}`),+ // Up to 100 SKUs per call; unknown SKUs are left out of the response.+ getMany: (skus) => request(CATALOG, 'POST', '/products/batch', { skus }),+};++const pricing = {+ get: (sku, currency) => request(PRICING, 'GET', `/prices/${encodeURIComponent(sku)}?currency=${currency}`),+ // Up to 200 SKUs per call, one currency.+ getMany: (skus, currency) => request(PRICING, 'POST', '/prices/batch', { skus, currency }),+};++const inventory = {+ get: (sku) => request(INVENTORY, 'GET', `/stock/${encodeURIComponent(sku)}`),+ // Up to 200 SKUs per call.+ getMany: (skus) => request(INVENTORY, 'POST', '/stock/batch', { skus }),+};++module.exports = { carts, catalog, pricing, inventory };diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/routes/cart.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/routes/cart.jsnew file mode 100644index 00000000..acf71b9a--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/routes/cart.js@@ -0,0 +1,37 @@+'use strict';++// buildCart returns the cart page model: every line with its product,+// price and availability.+async function buildCart(cartId, currency, { carts, catalog, pricing, inventory }) {+ const cart = await carts.get(cartId);+ const lines = [];+ for (const line of cart.lines) {+ const product = await catalog.get(line.sku);+ const price = await pricing.get(line.sku, currency);+ const stock = await inventory.get(line.sku);+ lines.push({+ sku: line.sku,+ qty: line.qty,+ title: product.title,+ brand: product.brand,+ image: product.imageUrl,+ unitPrice: price.amountMinor,+ currency: price.currency,+ inStock: stock.available >= line.qty,+ });+ }+ const total = lines.reduce((sum, l) => sum + l.unitPrice * l.qty, 0);+ return { id: cart.id, lines, total, currency };+}++function cartRoute(clients) {+ return async (req, res, next) => {+ try {+ res.json(await buildCart(req.session.cartId, req.query.currency || 'EUR', clients));+ } catch (err) {+ next(err);+ }+ };+}++module.exports = { buildCart, cartRoute };diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/routes/wishlist.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/routes/wishlist.jsnew file mode 100644index 00000000..31ceeb6c--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/routes/wishlist.js@@ -0,0 +1,22 @@+'use strict';++async function buildWishlist(customerId, { db, catalog }) {+ const { rows } = await db.query(+ 'SELECT sku FROM wishlist_items WHERE customer_id = $1 ORDER BY added_at DESC LIMIT 50',+ [customerId],+ );+ const products = await catalog.getMany(rows.map((r) => r.sku));+ return { items: products.map((p) => ({ sku: p.sku, title: p.title, image: p.imageUrl })) };+}++function wishlistRoute(deps) {+ return async (req, res, next) => {+ try {+ res.json(await buildWishlist(req.session.customerId, deps));+ } catch (err) {+ next(err);+ }+ };+}++module.exports = { buildWishlist, wishlistRoute };diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/server.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/server.jsnew file mode 100644index 00000000..42c1ee24--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/server.js@@ -0,0 +1,15 @@+'use strict';+const express = require('express');+const { Pool } = require('pg');+const clients = require('./clients');+const { session } = require('./session');+const { cartRoute } = require('./routes/cart');+const { wishlistRoute } = require('./routes/wishlist');++const db = new Pool({ connectionString: process.env.DATABASE_URL });+const app = express();+app.use(session());+app.get('/api/cart', cartRoute(clients));+app.get('/api/wishlist', wishlistRoute({ db, catalog: clients.catalog }));++app.listen(Number(process.env.PORT || 3000));diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/session.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/session.jsnew file mode 100644index 00000000..8eb16768--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/session.js@@ -0,0 +1,18 @@+'use strict';+const Redis = require('ioredis');++const redis = new Redis(process.env.REDIS_URL || 'redis://cache.internal:6379');++function session() {+ return async (req, res, next) => {+ const sid = req.get('cookie')?.match(/sid=([a-f0-9]+)/)?.[1];+ const raw = sid ? await redis.get(`sess:${sid}`) : null;+ if (!raw) {+ return res.status(401).json({ error: 'not signed in' });+ }+ req.session = JSON.parse(raw);+ next();+ };+}++module.exports = { session };diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/src/tracing.js b/cli/test/evals/testdata/repos/storefront-gateway-node/src/tracing.jsnew file mode 100644index 00000000..a33d0a76--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/src/tracing.js@@ -0,0 +1,10 @@+'use strict';+const { NodeSDK } = require('@opentelemetry/sdk-node');+const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');+const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto');++new NodeSDK({+ serviceName: 'storefront-gateway',+ traceExporter: new OTLPTraceExporter(),+ instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false } })],+}).start();diff --git a/cli/test/evals/testdata/repos/storefront-gateway-node/test/cart.test.js b/cli/test/evals/testdata/repos/storefront-gateway-node/test/cart.test.jsnew file mode 100644index 00000000..5e719b4e--- /dev/null+++ b/cli/test/evals/testdata/repos/storefront-gateway-node/test/cart.test.js@@ -0,0 +1,17 @@+'use strict';+const test = require('node:test');+const assert = require('node:assert');+const { buildCart } = require('../src/routes/cart');++test('buildCart prices every line and totals the cart', async () => {+ const clients = {+ carts: { get: async () => ({ id: 'c1', lines: [{ sku: 'A', qty: 2 }, { sku: 'B', qty: 1 }] }) },+ catalog: { get: async (sku) => ({ sku, title: `T${sku}`, brand: 'x', imageUrl: `/${sku}.jpg` }) },+ pricing: { get: async (sku, currency) => ({ sku, amountMinor: sku === 'A' ? 500 : 1200, currency }) },+ inventory: { get: async (sku) => ({ sku, available: 3 }) },+ };+ const cart = await buildCart('c1', 'EUR', clients);+ assert.equal(cart.lines.length, 2);+ assert.equal(cart.total, 2200);+ assert.equal(cart.lines[0].inStock, true);+});diff --git a/cli/test/evals/testdata/repos/tickets-api-go/README.md b/cli/test/evals/testdata/repos/tickets-api-go/README.mdnew file mode 100644index 00000000..15973123--- /dev/null+++ b/cli/test/evals/testdata/repos/tickets-api-go/README.md@@ -0,0 +1,20 @@+# tickets-api++Event listings, seat maps and ticket reservations. Go, `net/http`,+PostgreSQL through pgx. Traces, logs and the otelhttp request metrics go to+Traceway over OTLP; each VM's OTel Collector adds host metrics.++Runs on three EC2 instances (`tickets-node-a`, `-b`, `-c`) behind an+application load balancer with round-robin routing; see `deploy/terraform/`.++```+go test ./...+go run ./cmd/server+```++Routes:++- `GET /api/events?city=` – upcoming events in a city+- `GET /api/events/{id}` – one event with its seat map+- `POST /api/orders` – reserve seats+- `GET /api/orders/{id}` – one orderdiff --git a/cli/test/evals/testdata/repos/tickets-api-go/cmd/server/main.go b/cli/test/evals/testdata/repos/tickets-api-go/cmd/server/main.gonew file mode 100644index 00000000..093e26cc--- /dev/null+++ b/cli/test/evals/testdata/repos/tickets-api-go/cmd/server/main.go@@ -0,0 +1,75 @@+package main++import (+ "context"+ "encoding/json"+ "log/slog"+ "net/http"+ "os"+ "time"++ "github.com/exaring/otelpgx"+ "github.com/jackc/pgx/v5/pgxpool"+ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"++ "github.com/acme/tickets-api/internal/tickets"+)++func main() {+ ctx := context.Background()+ cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))+ if err != nil {+ slog.Error("database config", "err", err)+ os.Exit(1)+ }+ cfg.ConnConfig.Tracer = otelpgx.NewTracer()+ db, err := pgxpool.NewWithConfig(ctx, cfg)+ if err != nil {+ slog.Error("database connect", "err", err)+ os.Exit(1)+ }++ mux := http.NewServeMux()+ mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNoContent) })+ mux.HandleFunc("GET /api/events/{id}", func(w http.ResponseWriter, r *http.Request) {+ var ev tickets.Event+ err := db.QueryRow(r.Context(), `SELECT id, title, description, venue_id, starts_at FROM events WHERE id = $1`, r.PathValue("id")).+ Scan(&ev.ID, &ev.Title, &ev.Description, &ev.VenueID, &ev.StartsAt)+ if err != nil {+ http.NotFound(w, r)+ return+ }+ rows, err := db.Query(r.Context(), `SELECT section, row_label, seat_count, price_cents, available FROM seat_blocks WHERE event_id = $1`, ev.ID)+ if err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ defer rows.Close()+ var blocks []tickets.SeatBlock+ for rows.Next() {+ var b tickets.SeatBlock+ if err := rows.Scan(&b.Section, &b.Row, &b.SeatCount, &b.PriceCents, &b.Available); err != nil {+ http.Error(w, "internal error", http.StatusInternalServerError)+ return+ }+ blocks = append(blocks, b)+ }+ ev.Sections = tickets.SeatMap(blocks)+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(ev)+ })+ // The other routes are registered in internal/tickets (omitted here).++ // otelhttp records spans and the http.server.request.duration histogram.+ srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(mux, "tickets-api"), ReadHeaderTimeout: 10 * time.Second}+ slog.Info("listening", "addr", srv.Addr, "host", hostname())+ if err := srv.ListenAndServe(); err != nil {+ slog.Error("serve", "err", err)+ os.Exit(1)+ }+}++func hostname() string {+ h, _ := os.Hostname()+ return h+}diff --git a/cli/test/evals/testdata/repos/tickets-api-go/deploy/terraform/instances.tf b/cli/test/evals/testdata/repos/tickets-api-go/deploy/terraform/instances.tfnew file mode 100644index 00000000..8bac2907--- /dev/null+++ b/cli/test/evals/testdata/repos/tickets-api-go/deploy/terraform/instances.tf@@ -0,0 +1,35 @@+resource "aws_instance" "tickets" {+ for_each = toset(["a", "b", "c"])++ ami = data.aws_ami.tickets.id+ instance_type = "m6i.large"+ subnet_id = aws_subnet.private[each.key].id+ vpc_security_group_ids = [aws_security_group.tickets.id]+ iam_instance_profile = aws_iam_instance_profile.tickets.name+ user_data = templatefile("${path.module}/cloud-init.yaml", { hostname = "tickets-node-${each.key}" })++ tags = {+ Name = "tickets-node-${each.key}"+ Service = "tickets-api"+ }+}++resource "aws_lb_target_group_attachment" "tickets" {+ for_each = aws_instance.tickets+ target_group_arn = aws_lb_target_group.tickets.arn+ target_id = each.value.id+ port = 8080+}++resource "aws_lb_target_group" "tickets" {+ name = "tickets-api"+ port = 8080+ protocol = "HTTP"+ vpc_id = aws_vpc.main.id+ load_balancing_algorithm_type = "round_robin"++ health_check {+ path = "/healthz"+ interval = 15+ }+}diff --git a/cli/test/evals/testdata/repos/tickets-api-go/go.mod b/cli/test/evals/testdata/repos/tickets-api-go/go.modnew file mode 100644index 00000000..b4891c8d--- /dev/null+++ b/cli/test/evals/testdata/repos/tickets-api-go/go.mod@@ -0,0 +1,10 @@+module github.com/acme/tickets-api++go 1.26++require (+ github.com/exaring/otelpgx v0.9.3+ github.com/jackc/pgx/v5 v5.7.6+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0+ go.opentelemetry.io/otel v1.38.0+)diff --git a/cli/test/evals/testdata/repos/tickets-api-go/internal/tickets/seats.go b/cli/test/evals/testdata/repos/tickets-api-go/internal/tickets/seats.gonew file mode 100644index 00000000..fff41267--- /dev/null+++ b/cli/test/evals/testdata/repos/tickets-api-go/internal/tickets/seats.go@@ -0,0 +1,53 @@+// Package tickets serves events, seat maps and orders.+package tickets++import "time"++// Event is one event with its seat map.+type Event struct {+ ID int64 `json:"id"`+ Title string `json:"title"`+ Description string `json:"description"`+ VenueID int64 `json:"venueId"`+ StartsAt time.Time `json:"startsAt"`+ Sections []SectionSummary `json:"sections"`+}++// SeatBlock is a run of seats in one row of a section.+type SeatBlock struct {+ Section string `json:"section"`+ Row string `json:"row"`+ SeatCount int `json:"seatCount"`+ PriceCents int64 `json:"priceCents"`+ Available int `json:"available"`+}++// SectionSummary is one section of the seat map.+type SectionSummary struct {+ Section string `json:"section"`+ Available int `json:"available"`+ MinPriceCents int64 `json:"minPriceCents"`+ Rows []SeatBlock `json:"rows"`+}++// SeatMap groups an event's seat blocks into sections, keeping the order the+// blocks arrive in.+func SeatMap(blocks []SeatBlock) []SectionSummary {+ var out []SectionSummary+ index := map[string]int{}+ for _, b := range blocks {+ i, ok := index[b.Section]+ if !ok {+ i = len(out)+ index[b.Section] = i+ out = append(out, SectionSummary{Section: b.Section, MinPriceCents: b.PriceCents})+ }+ s := &out[i]+ s.Available += b.Available+ if b.Available > 0 && (b.PriceCents < s.MinPriceCents || s.MinPriceCents == 0) {+ s.MinPriceCents = b.PriceCents+ }+ s.Rows = append(s.Rows, b)+ }+ return out+}diff --git a/cli/test/evals/testdata/repos/tickets-api-go/internal/tickets/seats_test.go b/cli/test/evals/testdata/repos/tickets-api-go/internal/tickets/seats_test.gonew file mode 100644index 00000000..b3687888--- /dev/null+++ b/cli/test/evals/testdata/repos/tickets-api-go/internal/tickets/seats_test.go@@ -0,0 +1,17 @@+package tickets++import "testing"++func TestSeatMapGroupsBySection(t *testing.T) {+ got := SeatMap([]SeatBlock{+ {Section: "A", Row: "1", SeatCount: 10, PriceCents: 5000, Available: 2},+ {Section: "A", Row: "2", SeatCount: 10, PriceCents: 4500, Available: 0},+ {Section: "B", Row: "1", SeatCount: 12, PriceCents: 3000, Available: 12},+ })+ if len(got) != 2 || got[0].Available != 2 || got[1].Available != 12 {+ t.Fatalf("unexpected sections: %+v", got)+ }+ if got[0].MinPriceCents != 5000 {+ t.Fatalf("section A min price %d, want 5000 (row 2 is sold out)", got[0].MinPriceCents)+ }+}