The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
Benchmarks page (#417)
narrow-adventure21bbe0261624
diff --git a/docs/pages/learn/_meta.json b/docs/pages/learn/_meta.jsonindex 55d2d689..6b2770b3 100644--- a/docs/pages/learn/_meta.json+++ b/docs/pages/learn/_meta.json@@ -26,6 +26,7 @@"cli-auth": "CLI Authentication","mcp": "MCP Server","agent-write-access": "Agent Write Access",+ "agent-benchmarks": "Agent Benchmarks","embedded-mode": "Embedded Mode","otel-agent": "OTel Agent","kubernetes": "Kubernetes",diff --git a/docs/pages/learn/agent-benchmarks.mdx b/docs/pages/learn/agent-benchmarks.mdxnew file mode 100644index 00000000..4b0c2d75--- /dev/null+++ b/docs/pages/learn/agent-benchmarks.mdx@@ -0,0 +1,244 @@+# Agent Benchmarks++import { Callout } from "nextra/components";++## Bring your own AI++Traceway has a strict bring-your-own-AI policy. We don't ship a model, pick one for you, or send your data to an AI vendor on your behalf, and every Traceway instance works without any LLM configured. You use the AI you already trust, under your own account, and Traceway gives it the tools: an [agent skill](https://github.com/tracewayapp/traceway/tree/main/skills/traceway) for the [CLI](/learn/cli) and an [MCP server](/learn/mcp).++That raises a fair question: which models actually do well with those tools? This page exists to answer it in the open. Our eval harness, in the repository at [`cli/test/evals`](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals), hands models the kind of production problems an on-call engineer gets and grades what they do. Below is how it works and every case it runs. The results are on the [benchmarks page](https://tracewayapp.com/benchmarks). They also show us where our own skill and MCP server fall short.++### Take part++The harness is open source, and we welcome help in two ways:++- **Benchmark a new model.** Add it to `cli/test/evals/models.yaml` with an exact OpenRouter model id and the same settings as the other models, run the suites ([Run it yourself](#run-it-yourself)), and open a pull request with the generated `summary.md` and `results.json`.+- **Add a test case.** A case is a YAML file plus a seed, and it needs an oracle that passes before it can be merged. The harness [README](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals#adding-a-case) explains the format. Production failures you have debugged yourself make the best cases.++<Callout type="info">+The published results are local runs with one run per case, so any single case × setup cell is noisy. Trust the suite-level rates more than any one cell. Failures caused by our own cases or skill count as failures too; the benchmarks page lists the known ones.+</Callout>++## What an eval is++An eval is a test for an AI agent. Instead of checking a function's return value, it hands an agent a realistic task, lets it work with the same tools a person would use, and then checks the outcome with code. A **case** is one such task: a broken system plus a prompt. A **run** is one attempt at a case by one model on one setup. Running every case on every setup gives a pass rate per model that we can compare, and re-run after every change to the skill, the CLI or the MCP server.++## Models and the API++Every model call goes through [OpenRouter](https://openrouter.ai), with one API key, using its Chat Completions API. That gives every model the same request format, the same tool-calling interface and per-call cost accounting, and it lets us add a model without writing a new client. The judge goes through OpenRouter too. `models.yaml` pins each model to an exact OpenRouter id, asks OpenRouter to route it to the model's own vendor first, and records which provider actually served each run. Agent spend on the results page is the cost OpenRouter reported for each call, estimated from live prices when a response leaves it out.++## Where the results are++This page explains how we measure. The results live on the [benchmarks page](https://tracewayapp.com/benchmarks): pass rates per model, suite and setup, what the runs cost, and every case's outcome for every model.++## What one run is++Every run is a fresh, isolated Traceway instance with one broken system in it:++1. **A hermetic backend.** The harness starts the standalone backend in SQLite mode, registers an organization, and seeds it through the real ingest endpoints (OTLP and `/api/report`): weeks of traces, metrics, logs, issues, deploys and dashboard objects, ending at a fixed "now".+2. **One defect, one distractor.** The seed breaks the system in exactly one way (an N+1 query, pool exhaustion, a noisy neighbor, a cache stampede, GC pauses, a missing index, ...) and plants one named distractor that looks like a plausible cause but isn't.+3. **A prompt like a real one.** The agent gets the kind of message a person would send: "checkout got slow this week, find out why", a pasted alert, a dashboard URL, or a request to archive the errors from last week's incident.+4. **Real tools only.** The agent works through real `traceway` commands in a sandboxed shell, or through the MCP server's tools. Nothing else knows the answer.+5. **A structured answer.** The agent submits a diagnosis (or finishes a write task) with a report, citing the tool calls (`c04`, `c11`) its evidence comes from.+6. **Deterministic grading.** The graders below score the run. A model never decides whether a run passed.++## Suites++| Suite | Cases | Measures |+|---|---|---|+| perf | 35 | Diagnosing latency: every row of the skill's bottleneck checklist, plus controls (a healthy endpoint, an endpoint inside its slow allowance, one past it, an onset outside the chart's top 5, an alert hand-off, the metric "p95" trap) and cross-surface cases where browser, Android and iOS crashes have a backend cause |+| debug | 14 | Debugging an issue to its root cause and fixing the code, in Go, Python, Node, Android, iOS and Flutter. Hidden tests check the fix |+| write | 13 | The write playbooks (pages, alerts, monitors, archiving, post-mortems including a concurrent-edit conflict, slow marks) and the 403 policy (agent access off, a read-only token, a read-only member) |+| setup | 6 | Instrumenting an app with the setup skill until its telemetry arrives: Go net/http, Express, FastAPI, Next.js, a React SPA and a Go + React monorepo |++## The test cases++Every case is a YAML file in [`cli/test/evals/cases`](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals/cases). The descriptions below say what the agent is told or what it sees, not the answer. Some names give away the category, and that is fine: the agent never sees the case file.++### Performance++Each perf case seeds one bottleneck and one distractor. The control cases test judgment: a healthy endpoint, endpoints that are slow on purpose, a misleading metric, a pasted alert and an onset to pin down. Several have nothing to fix.++| Group | Case | What the agent gets |+|---|---|---|+| App code | `perf-app-cpu-serialization` | The transaction export burns seconds of CPU building its JSON body |+| App code | `perf-app-excessive-logging` | A release that only turned on debug logs slows every endpoint and doubles CPU |+| App code | `perf-app-large-alloc` | The retention report loads every event of the range into the JVM heap |+| App code | `perf-app-quadratic` | Quotes for large carts take seconds while small carts are instant |+| App code | `perf-app-sync-background` | Generating a statement holds the request open while the PDF renders |+| Caching | `perf-cache-cold-start` | Pricing times out for a few minutes after every release |+| Caching | `perf-cache-no-cache` | The catalog endpoint keeps the database busy all day |+| Caching | `perf-cache-stampede` | Home page requests stall for seconds in bursts |+| Controls and traps | `perf-ctl-alert-handoff` | A p95 latency alert pasted in from email: what happened, and does it need action? |+| Controls and traps | `perf-ctl-healthy` | Search "feels slow" but the endpoint is healthy and stable |+| Controls and traps | `perf-ctl-marked-slow-ok` | Ledger export is the slowest endpoint but sits inside its marked-slow allowance |+| Controls and traps | `perf-ctl-marked-slow-regressed` | Marked-slow invoice export regressed past its allowance after a release |+| Controls and traps | `perf-ctl-metric-p95-trap` | Search gets sluggish every afternoon, and ops says the hosts' p95 CPU looks fine |+| Controls and traps | `perf-ctl-onset-bisect` | The account page has felt slower for a few days: when did it start, and what changed? |+| Database | `perf-db-expensive-join` | The revenue report takes two seconds every time, busy or quiet |+| Database | `perf-db-lock-contention` | Stock reservations stall and fail during the evening product drop |+| Database | `perf-db-missing-index` | The user lookup gets slower every week as the users table grows |+| Database | `perf-db-missing-query-cache` | Every storefront endpoint is slower than it should be, even a single-row read |+| Database | `perf-db-n-plus-1` | Orders endpoint slows down as customers accumulate orders |+| Database | `perf-db-over-fetching` | The category grid got slow when products gained a large column |+| Database | `perf-db-pool-exhaustion` | Checkout stalls for seconds at the lunch and evening rushes |+| Database | `perf-db-unbounded-result` | The admin audit log gets slower every week as organizations accumulate events |+| Infrastructure | `perf-infra-cpu-saturation` | Traffic tripled and every endpoint slowed down with the CPU pinned |+| Infrastructure | `perf-infra-disk-io` | Every query got several times slower while the database host's disk queue filled up |+| Infrastructure | `perf-infra-memory-gc` | The heap climbs to its limit every seven hours and every endpoint stalls until the pod restarts |+| Infrastructure | `perf-infra-noisy-neighbor` | One of three VMs loses a third of its CPU to steal and serves every request slowly |+| Network and upstreams | `perf-net-chatty-fanout` | Cart page slows down with the size of the cart |+| Network and upstreams | `perf-net-large-payload` | The feed is slow, worst on phones |+| Network and upstreams | `perf-net-no-conn-reuse` | Store finder spends most of its time in calls to the maps provider |+| Network and upstreams | `perf-net-retries` | Shipping quotes occasionally take 6 to 13 seconds |+| Network and upstreams | `perf-net-sequential` | Portal dashboard is slow although every service behind it is fast |+| Network and upstreams | `perf-net-slow-third-party` | Checkout hangs every day or so, then recovers |+| Cross-stack and mobile | `perf-xs-android-timeouts` | The Android app crashes at checkout since the sale went live |+| Cross-stack and mobile | `perf-xs-frontend-timeouts` | Product search on the web shop spins, then says it is taking too long |+| Cross-stack and mobile | `perf-xs-ios-watchdog` | iOS users see the app freeze on the splash screen and close |++### Debug++Each debug case seeds an issue and a repository with the bug in it. The agent finds the root cause through Traceway and fixes the code. Hidden tests, copied in after the run, check the fix.++| Case | What the agent gets |+|---|---|+| `debug-android-npe` | Android feed crashes while scrolling since the 5.8.0 rollout |+| `debug-dashboard-url` | A pasted dashboard link to one occurrence of the invoice-page panic |+| `debug-flutter-null-check` | Flutter product pages fail with a null check on listings without photos |+| `debug-go-nil-regression` | Nil-pointer panic on the invoice page since the last release |+| `debug-go-task-panic` | A pasted new-error email for a panic inside the label worker's consumer |+| `debug-insufficient-evidence` | A rare 500 on the export endpoint |+| `debug-ios-force-unwrap` | iOS app traps when customers open live chat in Help |+| `debug-js-contract-rename` | The order history page crashes after the API renamed a field |+| `debug-log-only-error` | 500s on one endpoint with no exception attached |+| `debug-mixed-group` | One recurring issue to explain and fix |+| `debug-no-writes` | "Look into it and clean up" with a write credential: fix the code, change nothing in Traceway |+| `debug-py-keyerror-webhook` | Payment webhooks fail for guest payments since the account-linking release |+| `debug-race-duplicate` | Checkout shows an error although the order went through |+| `debug-timeout-downstream` | Cart quotes fail since the release that lowered the pricing call's timeout |++### Write++Each write case asks the agent to change something in Traceway, or to refuse when it should not. The state grader checks the backend afterwards.++| Case | What the agent gets |+|---|---|+| `write-alert-disable` | Turn off the alert for a gateway that is being decommissioned |+| `write-alert-snooze` | Quiet a flapping alert during a known third-party incident |+| `write-archive-ambiguous` | Archive last week's checkout errors |+| `write-investigate-only` | Look into a page without changing anything |+| `write-mark-slow` | Mark an endpoint that is slow by design as expected-slow |+| `write-monitor-confirm-fix` | Confirm a fix by re-running the checkout monitor |+| `write-optin-off` | Agent writes are turned off for the organization |+| `write-page-ack` | Acknowledge the checkout page the user is working on |+| `write-page-resolve-archive` | Close out a page after a rollback and archive its issue |+| `write-postmortem-conflict` | Add a follow-up to a post-mortem a teammate is editing |+| `write-postmortem-create` | Write the post-mortem for a checkout outage |+| `write-readonly-member` | Asked to archive with a readonly member's write token |+| `write-readonly-token` | Asked to archive with a read-only credential |++### Setup++Each setup case hands the agent an uninstrumented app in a container. It passes when the app builds, starts and its telemetry arrives in Traceway.++| Case | What the agent gets |+|---|---|+| `setup-go-nethttp` | Instrument a net/http Go API until its routes show up in Traceway |+| `setup-monorepo-plan` | Propose a project map for a Go API + React web monorepo, submit it as a setup plan, then instrument both |+| `setup-nextjs` | Instrument the server side of a Next.js App Router app until its API route shows up in Traceway |+| `setup-node-express` | Instrument an ES-module Express API until its routes show up in Traceway |+| `setup-python-fastapi` | Instrument a FastAPI service run by uvicorn until its routes show up in Traceway |+| `setup-react-browser` | Add the Traceway browser SDK to a React + Vite SPA, with debug IDs and source map upload in CI |++## Setups++Each case runs on several setups, so the skill and the two surfaces can be compared on the same cases:++| Setup | The agent gets |+|---|---|+| `cli+skill` | The `traceway` CLI and the `traceway` skill |+| `cli-noskill` | The CLI and nothing else: the control group for skill uplift |+| `mcp` | The MCP server's tools and knowledge |+| `cli-write+skill`, `mcp-write` | The write edition of the CLI with the `traceway-write` skill, or the `/mcp/write` server |+| `cli+setup-skill` | The setup skill and a container shell with the app's toolchain |++**Skill uplift** is the `cli+skill` pass rate minus `cli-noskill`, on the same cases. **CLI vs MCP** is `cli+skill` minus `mcp`.++## Graders++| Grader | Gates | Checks |+|---|---|---|+| `answer` | yes | The diagnosis: the right checklist row and not the distractor, the right endpoint or service, the onset within tolerance. For debug cases, the category, culprit file, symbol and line |+| `grounding` | yes | Every cited call exists, succeeded, was not truncated, and its output contains the deciding signal. An id the model invented fails the run |+| `trajectory` | yes | Required, forbidden and ordered operations, such as opening one slow request, never writing on a read-only task, or asking before an ambiguous bulk write. CLI and MCP calls map to the same operations, and many cases accept more than one route to the same evidence |+| `report` | yes | Statements the report must make, such as naming the figure that decides the case |+| `state` | yes | After a write or setup run, the backend and the workspace are exactly as expected: the right two issues archived, not three |+| `hidden_tests` | per case | Debug fixes: hidden tests copied in after the run pass, and the diff touches the culprit |+| `setup_verify` | yes | The instrumented app builds, starts against the backend and serves traffic |+| `efficiency` | no | Call counts, failed calls and repeated calls, reported in their own table |+| `judge` | no | Claude Haiku 5.5 scores the report from 0 to 1 against the case's rubric. It does not gate until it agrees with human labels on at least 85% of runs |++### What counts as a pass++A run passes when its answer is right, its evidence is grounded and its writes are safe. How the agent got there is not graded: re-running a command to check a JSON shape is not a wrong answer, so call counts are reported, never gated. Each case still has a hard budget (turns, tool calls, wall-clock time); a run that exhausts it counts as a fail.++A run that hits an infrastructure error, or stops at the spending cap, is left out of the pass rates and can be resumed.++The benchmarks page also shows **right answer and evidence**: runs whose answer, grounding, report and state graders all passed, whatever the trajectory grader said.++## Keeping it honest++- **The answer key never reaches the agent.** The cases live in `cli/test/evals`, never in a skill directory, because everything there ships to the agent under test. The seed's exposed ids, the rubric and the expected answer are shown only to the graders and the judge.+- **No hints in the data.** A leakage lint fails the build when a seeded string (a span name, an attribute, a log line, a stack trace) or a prompt names the answer: `n+1`, `stampede`, `missing index`, `gc pause` and similar. Seeds use real library messages such as SQLAlchemy's `QueuePool limit of size 5 overflow 10 reached`.+- **Every case is passable.** Each case has an oracle: a reference trajectory played by a fake model. It must pass on every setup the case lists, for $0, before the case is added, and `just eval-harness` replays every oracle after a harness change.+- **A sandbox, not a host shell.** The agent's shell is an interpreter with the setup's `traceway` binary, `jq`, `awk`, read-only text utilities confined to the workspace and read-only `git`. Tool processes get a scrubbed environment, so no key or secret reaches them.+- **The same conditions for every model.** `models.yaml` pins exact OpenRouter model ids, never a floating alias, with the same reasoning effort (medium), the same per-turn token limit (16k), the same prompts, tools and budgets. Requests go to the model's own vendor first, and every run records which provider served it.+- **Compare like with like.** Results are only compared between runs on the same commit and the same graders. When a grader changes, saved transcripts are re-scored instead of comparing old numbers with new ones.++## Spending controls++Evals cost real money, so nothing runs them automatically:++- The CLI Evals workflow has a single trigger, `workflow_dispatch`, and the paid job waits for a required reviewer. A test fails the build if the workflow gains any other trigger or anything else starts it.+- `--max-cost-usd` is required and has no default. Each run reserves its share before it starts and stops before a model call that would cross it.+- A local paid run needs `yes` typed on a terminal, after a free preflight that prints the live-price estimate.++## What is not measured yet++| Missing | Size | Why |+|---|---|---|+| The debug suite | 14 cases, 41 runs per model | Needs Docker to run the hidden tests against the model's fix with no network |+| The setup suite | 6 cases, 6 runs per model | Needs Docker containers with the app's toolchain (Go, Node, Python) |+| GPT-6 Astra | every suite | It is configured in `models.yaml` but has never run |+| Repeats | every case | Each case ran once per setup, so there is no variance estimate yet |+| Workflow baselines | every suite | The published runs were made locally, not by the CLI Evals workflow |++The judge's 0 to 1 score is reported but does not gate any run yet. It starts gating once it agrees with human labels on at least 85% of runs. The judge is Claude Haiku 5.5 for every model, including when it grades its own runs.++## Run it yourself++### Environment variables++| Variable | Needed for | What it does |+|---|---|---|+| `OPENROUTER_API_KEY` | Every paid run | Your OpenRouter API key. All model and judge calls use it. The free commands below never read it, and the harness keeps it out of the agent's tools and shell |+| `TWEVALS_HOST_TESTS` | Optional | Set to `1` to run the debug suite's hidden tests on the host when Docker is missing. Without it, and without Docker, those tests are skipped |+| `TWEVALS_ROOT` | Optional | The `cli/test/evals` directory, when you run `twevals` from somewhere else |++Export the key in your shell for a local run, for example `export OPENROUTER_API_KEY=sk-or-...`, and never put it in the repository. Check your OpenRouter credit before a long run: when it runs out, every call fails and the unfinished runs end as infrastructure errors. In GitHub the key lives only in the `evals` environment, never as a repository or organization secret.++### Commands++From `cli/` in a checkout of the repository:++| Command | Cost | What it does |+|---|---|---|+| `just eval-harness` | $0 | The harness tests, then every case's oracle on every setup, with the fake model |+| `just eval-plan --preset smoke --max-cost-usd 30` | $0 | The run matrix and a cost estimate from live OpenRouter prices |+| `just eval-dispatch -f preset=smoke -f max_cost_usd=30` | paid | Starts the CLI Evals workflow, which still waits for approval |+| `just eval-local --preset full --suites perf --models haiku --max-cost-usd 5` | paid | A local run. Needs `OPENROUTER_API_KEY` and `yes` typed on a terminal |++The debug and setup suites need Docker: it runs the hidden tests of a model's fix with no network, and it hosts the setup suite's toolchains. Each run writes a `summary.md` (pass rates, skill uplift, CLI vs MCP, model vs model, efficiency, cost) and keeps every transcript. `twevals report <dir>... --out DIR` merges runs into one comparison.++See the harness [README](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals) for the case format, the seed DSL and how to add a case.diff --git a/website/app/benchmarks/page.tsx b/website/app/benchmarks/page.tsxnew file mode 100644index 00000000..9d71b632--- /dev/null+++ b/website/app/benchmarks/page.tsx@@ -0,0 +1,1096 @@+import Link from "next/link";+import type { Metadata } from "next";+import { ArrowRight } from "lucide-react";++import { SectionHead } from "@/components/section-head";+import { StatsStrip } from "@/components/stats-strip";+import { FinalCTA } from "@/components/final-cta";+import { AuroraBackground } from "@/components/aurora-background";+import { BenchmarkCaseMatrix } from "@/components/benchmark-case-matrix";+import {+ BenchmarkComparisonTable,+ type ComparisonRow,+ type ComparisonValue,+} from "@/components/benchmark-comparison-table";+import {+ BenchmarkIndexChart,+ type IndexView,+} from "@/components/benchmark-index-chart";+import {+ BenchmarkCostChart,+ type CostPoint,+} from "@/components/benchmark-cost-chart";+import { BENCHMARK } from "@/lib/benchmark-data";+import {+ MODEL_COLOR,+ formatPp,+ formatUsd,+ numberWord,+ rate,+} from "@/lib/benchmark-format";+import type { Model, ModelKey } from "@/lib/benchmarks";++const { run, models, suites, setups, comparisons, groups, knownIssues, cases } =+ BENCHMARK;++const modelByKey = Object.fromEntries(models.map((m) => [m.key, m])) as Record<+ ModelKey,+ Model+>;+const setupLabels = Object.fromEntries(setups.map((s) => [s.arm, s.label]));+const suiteLabel = Object.fromEntries(suites.map((s) => [s.id, s.label]));+const groupById = Object.fromEntries(groups.map((g) => [g.id, g]));+const capitalize = (s: string) => s[0].toUpperCase() + s.slice(1);+const joinList = (items: string[]) =>+ items.length < 2+ ? items.join("")+ : `${items.slice(0, -1).join(", ")} and ${items[items.length - 1]}`;+const modelCount = numberWord(models.length);++export const metadata: Metadata = {+ title: "Traceway AI Benchmarks",+ description: `How well do AI agents debug production with Traceway? ${run.runsPerModel} seeded production failures, ${modelCount} models (${models.map((m) => m.label).join(", ")}), graded by deterministic code. Pass rates, costs and every case.`,+};++const CARD_STYLE = {+ background: "var(--ink-0)",+ border: "1px solid var(--hair-2)",+ boxShadow:+ "0 1px 2px rgba(10,14,24,0.04), 0 12px 30px -18px rgba(10,14,24,0.14)",+};++function totalsOf(key: ModelKey) {+ return modelByKey[key].totals;+}++function setupResult(suite: string, arm: string, key: ModelKey) {+ return setups.find((s) => s.suite === suite && s.arm === arm)!.results[key];+}++function pickBy(score: (m: Model) => number, best: "max" | "min") {+ return models.reduce((a, b) =>+ best === "max"+ ? score(b) > score(a)+ ? b+ : a+ : score(b) < score(a)+ ? b+ : a,+ );+}++const best = pickBy((m) => m.totals.pass, "max");+const value = pickBy((m) => m.totals.costUsd / m.totals.pass, "min");+const upliftOf = (key: ModelKey) =>+ comparisons.skillUplift.find((u) => u.model === key)!;+const biggestUplift = pickBy((m) => upliftOf(m.key).pp, "max");+const cheapest = pickBy((m) => m.totals.costUsd, "min");+const priciest = pickBy((m) => m.totals.costUsd, "max");++const BADGES: Partial<Record<ModelKey, string[]>> = {};+for (const [m, label] of [+ [best, "Highest pass rate"],+ [value, "Lowest cost per pass"],+ [biggestUplift, "Biggest skill uplift"],+] as const) {+ (BADGES[m.key] ??= []).push(label);+}++const controls = groupById["perf-ctl"];+const byControls = [...models].sort(+ (a, b) => controls.results[b.key].pass - controls.results[a.key].pass,+);+const mcpPerf = (key: ModelKey) =>+ comparisons.cliVsMcp.find((c) => c.suite === "perf" && c.model === key)!;+const mcpFavoured = models.filter((m) => mcpPerf(m.key).pp < 0);+const upliftOrder = [...models].sort(+ (a, b) => upliftOf(b.key).pp - upliftOf(a.key).pp,+);++const costPoints: CostPoint[] = [+ ...setups.flatMap((s) =>+ models.map((m) => ({+ id: `${s.suite}-${s.arm}-${m.key}`,+ model: m.key,+ modelLabel: m.label,+ modelShort: m.short,+ label: `${suiteLabel[s.suite]} · ${s.label}`,+ pass: s.results[m.key].pass,+ runs: s.results[m.key].runs,+ costPerRunUsd: s.results[m.key].costPerRunUsd,+ overall: false,+ })),+ ),+ ...models.map((m) => ({+ id: `all-${m.key}`,+ model: m.key,+ modelLabel: m.label,+ modelShort: m.short,+ label: `All ${m.totals.runs} runs`,+ pass: m.totals.pass,+ runs: m.totals.runs,+ costPerRunUsd: m.totals.costPerRunUsd,+ overall: true,+ })),+];++const maxCost = Math.max(...models.map((m) => m.totals.costUsd));+const issueRuns = (key: ModelKey) =>+ knownIssues.reduce((n, i) => n + i.byModel[key], 0);+const issueGroups = groups+ .filter((g) => knownIssues.some((i) => cases.find((c) => c.id === i.case)?.group === g.id))+ .map((g) => g.label);++const groupRightAnswer = (groupId: string, key: ModelKey) =>+ cases+ .filter((c) => c.group === groupId)+ .flatMap((c) => Object.values(c.runs[key]))+ .filter((r) => r.kind === "pass" || r.kind === "trajectory").length;++const indexViews: IndexView[] = [+ {+ id: "overall",+ tab: "All runs",+ title: `by model, all ${run.runsPerModel} runs`,+ bars: models.map((m) => ({+ id: m.key,+ model: m.key,+ weights: m.weights,+ label: m.label,+ detail: m.vendor,+ pass: m.totals.pass,+ rightAnswerEvidence: m.totals.rightAnswerEvidence,+ runs: m.totals.runs,+ })),+ },+ {+ id: "setup",+ tab: "By suite and setup",+ title: "by suite and setup",+ bars: setups.flatMap((s) =>+ models.map((m) => ({+ id: `${s.suite}-${s.arm}-${m.key}`,+ model: m.key,+ weights: m.weights,+ label: m.short,+ detail: s.suite === "perf" ? `Perf · ${s.label}` : s.label,+ pass: s.results[m.key].pass,+ rightAnswerEvidence: s.results[m.key].rightAnswerEvidence,+ runs: s.results[m.key].runs,+ })),+ ),+ },+ {+ id: "group",+ tab: "By case group",+ title: "by case group",+ bars: groups.flatMap((g) =>+ models.map((m) => ({+ id: `${g.id}-${m.key}`,+ model: m.key,+ weights: m.weights,+ label: m.short,+ detail: g.label,+ pass: g.results[m.key].pass,+ rightAnswerEvidence: groupRightAnswer(g.id, m.key),+ runs: g.results[m.key].runs,+ })),+ ),+ },+];++const indexSubtitle = `Traceway agent evals, run ${run.date}. ${suites+ .map((s) => `${s.label} (${s.cases} cases)`)+ .join(" and ")} on ${setups.length} setups, ${run.runsPerCase} run per case, graded by deterministic code.`;++const byPass = [...models].sort((a, b) => b.totals.pass - a.totals.pass);+const pctText = (pass: number, runs: number) =>+ `${((100 * pass) / runs).toFixed(1)}%`;+const passValue = (+ r: { pass: number; runs: number },+ note?: string,+): ComparisonValue => ({+ value: r.pass / r.runs,+ text: pctText(r.pass, r.runs),+ note,+ title: `${r.pass} of ${r.runs} runs`,+});+const perModelRow = (+ label: string,+ sub: string,+ f: (key: ModelKey) => ComparisonValue[],+ better: "higher" | "lower" = "higher",+): ComparisonRow => ({+ label,+ sub,+ better,+ values: Object.fromEntries(models.map((m) => [m.key, f(m.key)])) as Record<+ ModelKey,+ ComparisonValue[]+ >,+});+const suiteCases = (id: string) => suites.find((s) => s.id === id)!.cases;+const GROUP_ORDER = [+ "perf-ctl",+ "perf-db",+ "perf-app",+ "perf-net",+ "perf-cache",+ "perf-infra",+ "perf-xs",+];+const smallestGroup = Math.min(+ ...groups.map((g) => g.results[models[0].key].runs),+);+const comparisonRows: ComparisonRow[] = [+ perModelRow(+ "Performance diagnosis",+ `perf · ${suiteCases("perf")} cases, CLI`,+ (k) => [+ passValue(setupResult("perf", "cli+skill", k), "with the skill"),+ passValue(setupResult("perf", "cli-noskill", k), "without the skill"),+ ],+ ),+ perModelRow(+ "Performance diagnosis over MCP",+ `perf · ${suiteCases("perf")} cases, MCP server`,+ (k) => [passValue(setupResult("perf", "mcp", k))],+ ),+ perModelRow(+ "Safe write actions",+ `write · ${suiteCases("write")} cases`,+ (k) => [+ passValue(setupResult("write", "cli-write+skill", k), "write CLI + skill"),+ passValue(setupResult("write", "mcp-write", k), "write MCP server"),+ ],+ ),+ ...GROUP_ORDER.map((id) => {+ const g = groupById[id];+ return perModelRow(+ g.label,+ `${g.id}-* · ${g.cases} cases, ${g.results[models[0].key].runs} runs`,+ (k) => [passValue(g.results[k])],+ );+ }),+ perModelRow("Overall pass rate", `all ${run.runsPerModel} runs`, (k) => [+ passValue(totalsOf(k)),+ ]),+ perModelRow(+ "Right answer and evidence",+ "trajectory check not counted",+ (k) => [+ passValue({+ pass: totalsOf(k).rightAnswerEvidence,+ runs: totalsOf(k).runs,+ }),+ ],+ ),+ perModelRow(+ "Agent spend",+ `all ${run.runsPerModel} runs, lower is better`,+ (k) => [+ { value: totalsOf(k).costUsd, text: formatUsd(totalsOf(k).costUsd) },+ ],+ "lower",+ ),+ perModelRow(+ "Cost per passing run",+ "agent spend ÷ passed runs, lower is better",+ (k) => {+ const v = totalsOf(k).costUsd / totalsOf(k).pass;+ return [{ value: v, text: formatUsd(v) }];+ },+ "lower",+ ),+];++export default function BenchmarksPage() {+ return (+ <main className="relative">+ <section className="hero hero-product relative">+ <AuroraBackground variant="hero" />+ <div className="wrap relative z-10">+ <div className="flex flex-col items-center text-center">+ <h1 className="max-w-[24ch]">+ Traceway <em>AI Benchmarks</em>+ </h1>+ <p className="hero-sub text-pretty" style={{ maxWidth: 720 }}>+ We believe you should bring your own AI. Traceway isn't here+ to sell you an agent. It lets the model you already trust reach+ your production data, on your terms. These benchmarks show how+ well different models do on our eval suite, and help us make it+ better.+ </p>+ <div className="hero-cta-row justify-center">+ <Link+ href={run.methodologyUrl}+ target="_blank"+ rel="noopener noreferrer"+ className="btn btn-ghost"+ >+ How we measure+ <ArrowRight className="h-4 w-4" />+ </Link>+ </div>+ </div>+ <StatsStrip+ className="mx-auto max-w-4xl"+ stats={[+ { num: `<em>${run.totalRuns}</em>`, label: "Graded runs" },+ {+ num: `<em>${suites.reduce((n, s) => n + s.cases, 0)}</em>`,+ label: "Cases",+ },+ { num: `<em>${setups.length}</em>`, label: "Setups" },+ { num: `<em>${models.length}</em>`, label: "Models" },+ ]}+ />+ </div>+ </section>++ <div className="band-light">+ <section id="results" className="wrap scroll-mt-20 pt-16 pb-10 md:pt-20">+ <SectionHead+ eyebrow="Headline"+ title={+ <>+ Pass rate, <em>model by model.</em>+ </>+ }+ description={`Every model ran the same ${run.runsPerModel} runs. A run passes when its answer is right, its evidence is grounded and its writes are safe. Tool-call counts are reported but never gate.`}+ />+ <BenchmarkIndexChart views={indexViews} subtitle={indexSubtitle} />+ <div className="mt-8 grid gap-5 md:grid-cols-3">+ {models.map((m) => (+ <ModelCard key={m.key} model={m} badges={BADGES[m.key] ?? []} />+ ))}+ </div>+ </section>++ <section id="compare" className="wrap scroll-mt-20 py-16 md:py-20">+ <SectionHead+ eyebrow="At a glance"+ title={+ <>+ Every benchmark, <em>side by side.</em>+ </>+ }+ description="Pass rates per suite, setup and case group, then what the runs cost. The best result in each row is in bold, and a shaded cell means that model led every number in the row."+ />+ <BenchmarkComparisonTable+ models={byPass.map((m) => ({+ key: m.key,+ label: m.label,+ note: m.weights === "open" ? "open weights" : m.vendor,+ }))}+ rows={comparisonRows}+ />+ <p className="mt-4 text-[0.8125rem] leading-relaxed text-fg-3">+ One run per case and setup, so a single run moves the smallest+ groups ({smallestGroup} runs) by {Math.round(100 / smallestGroup)}{" "}+ points. Open a{" "}+ <Link+ href="https://github.com/tracewayapp/traceway/pulls"+ target="_blank"+ rel="noopener noreferrer"+ className="text-blue-bright hover:underline"+ >+ PR+ </Link>{" "}+ or email{" "}+ <a+ href="mailto:[email protected]"+ className="text-blue-bright hover:underline"+ >+ </a>{" "}+ if you'd like to see your model featured.+ </p>+ </section>++ <section id="findings" className="wrap scroll-mt-20 py-16 md:py-20">+ <SectionHead+ eyebrow="Key findings"+ title={+ <>+ What the runs say, <em>and what they don't.</em>+ </>+ }+ />+ <div className="grid gap-12 md:grid-cols-2">+ <Finding label="01 · Judgment" title={`${byControls[0].short} never false-alarms.`}>+ The control cases are traps: a healthy endpoint, endpoints marked+ slow on purpose, a misleading p95 metric, a pasted alert.{" "}+ {byControls[0].short} passed {controls.results[byControls[0].key].pass}{" "}+ of {controls.results[byControls[0].key].runs} control runs.{" "}+ {joinList(+ byControls+ .slice(1)+ .map((m) => `${m.short} passed ${controls.results[m.key].pass}`),+ )}+ . The gap between the models is in judgment, not in reading+ telemetry.+ </Finding>+ <Finding label="02 · Value" title={`${value.short} is the value pick.`}>+ {rate(value.totals.pass, value.totals.runs)}% of its runs passed for{" "}+ {formatUsd(value.totals.costUsd)} of agent spend, against{" "}+ {rate(totalsOf("haiku").pass, totalsOf("haiku").runs)}% for{" "}+ {formatUsd(totalsOf("haiku").costUsd)} with {modelByKey.haiku.short}:{" "}+ {value.totals.pass - totalsOf("haiku").pass} more passing runs at+ about the same price. {best.short} passed{" "}+ {best.totals.pass - value.totals.pass} more again, for{" "}+ {Math.round(best.totals.costUsd / value.totals.costUsd)}× the+ spend. On MCP perf, {value.short} is{" "}+ {mcpPerf(best.key).mcp - mcpPerf(value.key).mcp}pp behind{" "}+ {best.short}.+ </Finding>+ <Finding+ label="03 · The skill"+ title="The skill helps the weakest model most."+ >+ On the perf suite, CLI + skill against the same CLI without it:{" "}+ {joinList(+ upliftOrder.map((m) => `${m.short} ${formatPp(upliftOf(m.key).pp)}`),+ )}+ .{" "}+ {mcpFavoured.length === 1+ ? `${mcpFavoured[0].short} is the only model that does better on MCP (${mcpPerf(mcpFavoured[0].key).mcp}%) than on CLI + skill (${mcpPerf(mcpFavoured[0].key).cli}%).`+ : null}+ </Finding>+ <Finding label="04 · Our bugs" title="Some failures are ours.">+ {capitalize(numberWord(knownIssues.length))} problems behind+ these failures are in our own cases and skill, not in the models.+ They account for{" "}+ {joinList(+ models.map(+ (m, i) =>+ `${issueRuns(m.key)} of ${m.short}'s ${m.totals.runs - m.totals.pass}${i === 0 ? " failures" : ""}`,+ ),+ )}+ . We still count them as failures. Here they are.+ </Finding>+ </div>++ <div className="mt-14 grid gap-5 md:grid-cols-2">+ {knownIssues.map((issue) => (+ <div key={issue.id} className="rounded-2xl p-6" style={CARD_STYLE}>+ <div className="flex flex-wrap items-center gap-2">+ <span className="tag">+ {issue.kind === "skill" ? "Skill bug" : "Case bug"}+ </span>+ <code>{issue.case}</code>+ </div>+ <h3 className="mt-4 text-[1.0625rem] font-semibold text-fg-0">+ {issue.title}+ </h3>+ <p className="muted mt-2 text-[0.9375rem] text-pretty">+ {issue.detail}+ </p>+ <p className="mt-4 font-mono text-[0.75rem] text-fg-2">+ {issue.truncated+ ? `${issue.affectedRuns} of ${issue.caseRuns} runs failed, ${issue.truncated} of them on truncated output. `+ : ""}+ Runs hit:{" "}+ {models.map((m) => `${m.short} ${issue.byModel[m.key]}`).join(" · ")}+ </p>+ </div>+ ))}+ </div>+ </section>++ <section id="setups" className="wrap scroll-mt-20 py-16 md:py-20">+ <SectionHead+ eyebrow="By setup"+ title={+ <>+ Pass rate by suite <em>and setup.</em>+ </>+ }+ description="Perf cases run with the CLI and skill, the CLI alone as a control group, and the MCP server. Write cases run with the write CLI and skill, and the write MCP server."+ />+ <DataTable minWidth={760}>+ <thead>+ <tr>+ <Th>Setup</Th>+ {models.map((m) => (+ <ModelTh key={m.key} model={m} />+ ))}+ </tr>+ </thead>+ <tbody>+ {suites.map((s) => (+ <SuiteRows key={s.id} suite={s} />+ ))}+ <tr style={{ borderTop: "1px solid var(--hair-2)" }}>+ <RowHead title="All runs" sub={`${run.runsPerModel} per model`} />+ {models.map((m) => (+ <Td key={m.key}>+ <PassCell+ model={m.key}+ pass={m.totals.pass}+ runs={m.totals.runs}+ rightAnswerEvidence={m.totals.rightAnswerEvidence}+ strong+ />+ </Td>+ ))}+ </tr>+ <SubheadRow label="Differences" span={models.length + 1} />+ <DeltaRow+ title="Skill uplift, perf"+ sub="CLI + skill minus CLI without the skill"+ cells={models.map((m) => {+ const u = upliftOf(m.key);+ return { key: m.key, pp: u.pp, detail: `${u.with}% vs ${u.without}%` };+ })}+ />+ {suites.map((s) => (+ <DeltaRow+ key={s.id}+ title={`CLI vs MCP, ${s.id}`}+ sub={`${setupLabels[s.setups[0]]} minus ${setupLabels[s.setups[s.setups.length - 1]]}`}+ cells={models.map((m) => {+ const c = comparisons.cliVsMcp.find(+ (x) => x.suite === s.id && x.model === m.key,+ )!;+ return { key: m.key, pp: c.pp, detail: `${c.cli}% vs ${c.mcp}%` };+ })}+ />+ ))}+ </tbody>+ </DataTable>+ <p className="dim mt-4 text-[0.8125rem]">+ “Right answer + evidence” counts runs whose answer and+ grounding passed, whatever happened to the trajectory check. The+ differences are in percentage points, computed before rounding.+ </p>+ </section>++ <section id="groups" className="wrap scroll-mt-20 py-16 md:py-20">+ <SectionHead+ eyebrow="By case group"+ title={+ <>+ Where each model <em>wins and loses.</em>+ </>+ }+ description="Passes over every setup a case ran on. Groups follow the case names: perf-db-n-plus-1 is in Database, perf-ctl-healthy in Controls and traps."+ />+ <DataTable minWidth={820}>+ <thead>+ <tr>+ <Th>Group</Th>+ {models.map((m) => (+ <ModelTh key={m.key} model={m} />+ ))}+ </tr>+ </thead>+ <tbody>+ {groups.map((g) => {+ const top = Math.max(...models.map((m) => g.results[m.key].pass));+ return (+ <tr key={g.id} style={{ borderTop: "1px solid var(--hair)" }}>+ <RowHead+ title={g.label}+ code={g.id === "write" ? "write-*" : `${g.id}-*`}+ sub={`${g.cases} cases · ${g.results[models[0].key].runs} runs per model`}+ description={g.description}+ />+ {models.map((m) => (+ <Td key={m.key}>+ <PassCell+ model={m.key}+ pass={g.results[m.key].pass}+ runs={g.results[m.key].runs}+ strong={g.results[m.key].pass === top}+ />+ </Td>+ ))}+ </tr>+ );+ })}+ </tbody>+ </DataTable>+ <p className="dim mt-4 text-[0.8125rem]">+ The group's best count is set in bold. Groups hold{" "}+ {Math.min(...groups.map((g) => g.cases))} to{" "}+ {Math.max(...groups.map((g) => g.cases))} cases, so a single run+ moves a small group a lot. {joinList(issueGroups)} include the+ case and skill problems listed under Key findings.+ </p>+ </section>++ <section id="cost" className="wrap scroll-mt-20 py-16 md:py-20">+ <SectionHead+ eyebrow="Cost and efficiency"+ title={+ <>+ What a diagnosis <em>costs.</em>+ </>+ }+ description={`Agent spend is the model cost the harness recorded for each run, through OpenRouter. The judge's spend is separate. ${priciest.short} cost ${Math.round(priciest.totals.costUsd / cheapest.totals.costUsd)}× as much as ${cheapest.short} for the same ${run.runsPerModel} runs.`}+ />++ <div className="cost-wrap" data-cost-mount>+ {models.map((m) => (+ <div key={m.key} className="cost-bars">+ <div className="cost-name">+ <span+ aria-hidden+ className="inline-block size-2.5 shrink-0 rounded-full"+ style={{ background: MODEL_COLOR[m.key] }}+ />+ {m.short}+ </div>+ <div className="cost-bar h-6">+ <div+ className="fill"+ data-w={((100 * m.totals.costUsd) / maxCost).toFixed(2)}+ style={{+ width: `${((100 * m.totals.costUsd) / maxCost).toFixed(2)}%`,+ minWidth: 3,+ background: MODEL_COLOR[m.key],+ borderRadius: "0 4px 4px 0",+ }}+ />+ </div>+ <div className="cost-total">+ <em>{formatUsd(m.totals.costUsd)}</em>+ <span className="block text-[0.75rem] text-fg-3">+ {formatUsd(m.totals.costPerRunUsd)} per run+ </span>+ </div>+ </div>+ ))}+ </div>++ <div className="mt-14 rounded-2xl p-5 sm:p-7" style={CARD_STYLE}>+ <h3 className="text-[1.0625rem] font-semibold text-fg-0">+ Cost per run against pass rate+ </h3>+ <p className="muted mt-1 text-[0.9375rem]">+ Up is better, left is cheaper. Hover or focus a dot for its+ numbers; every value is also in the tables on this page.+ </p>+ <div className="mt-5">+ <BenchmarkCostChart+ points={costPoints}+ models={models.map((m) => ({ key: m.key, label: m.label }))}+ />+ </div>+ </div>++ <div className="mt-10">+ <DataTable minWidth={700}>+ <thead>+ <tr>+ <Th>All {run.runsPerModel} runs</Th>+ {models.map((m) => (+ <ModelTh key={m.key} model={m} />+ ))}+ </tr>+ </thead>+ <tbody>+ <MetricRow label="Agent spend" values={models.map((m) => formatUsd(m.totals.costUsd))} />+ <MetricRow label="Average cost per run" values={models.map((m) => formatUsd(m.totals.costPerRunUsd))} />+ <MetricRow label="Cost per passing run" values={models.map((m) => formatUsd(m.totals.costUsd / m.totals.pass))} />+ <MetricRow label="Average tool calls per run" values={models.map((m) => m.totals.avgToolCalls.toFixed(1))} />+ <MetricRow label="Average wall time per run" values={models.map((m) => `${m.totals.avgWallS.toFixed(1)}s`)} />+ <MetricRow label="Cache hit rate (input tokens)" values={models.map((m) => `${Math.round(100 * m.totals.cacheHitRate)}%`)} />+ <MetricRow+ label="Average judge score (0–1)"+ values={models.map((m) =>+ m.totals.avgJudge === null ? "–" : m.totals.avgJudge.toFixed(2),+ )}+ />+ <MetricRow label="Judge spend" values={models.map((m) => formatUsd(m.totals.judgeCostUsd))} />+ </tbody>+ </DataTable>+ <p className="dim mt-4 text-[0.8125rem]">+ Runs went three at a time on one machine, so wall time includes+ some contention.{" "}+ {models+ .filter((m) => m.totals.judged < m.totals.runs)+ .map(+ (m) =>+ `The judge scored ${m.totals.judged} of ${m.short}'s ${m.totals.runs} runs.`,+ )+ .join(" ")}+ </p>+ </div>+ </section>++ <section id="matrix" className="wrap scroll-mt-20 py-16 md:py-20">+ <SectionHead+ eyebrow="Every case"+ title={+ <>+ The full matrix: <em>every case, setup and model.</em>+ </>+ }+ description="One cell per run. Hover a cell for the first reason the graders gave, or click a row to read every failure of that case."+ />+ <BenchmarkCaseMatrix+ cases={cases}+ models={models.map((m) => ({ key: m.key, label: m.label, short: m.short }))}+ suites={suites.map((s) => ({ id: s.id, label: s.label, setups: s.setups }))}+ groups={groups.map((g) => ({ id: g.id, label: g.label }))}+ setupLabels={setupLabels}+ issues={knownIssues.map((i) => ({ id: i.id, kind: i.kind, title: i.title }))}+ />+ </section>+ </div>++ <section id="methodology" className="wrap scroll-mt-20 pt-20 pb-6">+ <SectionHead+ eyebrow="Methodology"+ title={+ <>+ How we <em>measure.</em>+ </>+ }+ description={`Each task is a fresh Traceway backend seeded with weeks of telemetry from a system broken in one way, plus a planted red herring. The model works with the real CLI or MCP server, and deterministic graders check its answer, its evidence and anything it changed. All runs on ${run.date}, at commit ${run.commit}, ${run.runsPerCase} run per case.`}+ cta={+ <div className="flex flex-wrap items-center gap-x-6 gap-y-3">+ <Link+ href={run.methodologyUrl}+ target="_blank"+ rel="noopener noreferrer"+ className="btn btn-accent"+ >+ Methodology and every test case+ <ArrowRight className="h-4 w-4" />+ </Link>+ <Link+ href={run.harnessUrl}+ target="_blank"+ rel="noopener noreferrer"+ className="font-mono text-[0.6875rem] uppercase tracking-[0.08em] text-fg-2 transition-colors hover:text-blue-bright"+ >+ The eval harness on GitHub →+ </Link>+ </div>+ }+ />+ </section>++ <FinalCTA+ title={+ <>+ Give your agent <em>production context.</em>+ </>+ }+ description="Install the skill or connect the MCP server, and let your agent take the next slow endpoint."+ primary={{ label: "Explore Agent Skills", href: "/product/agent-skills" }}+ secondary={{ label: "See the MCP server", href: "/product/mcp" }}+ />+ </main>+ );+}++function ModelSwatch({ model, size = 10 }: { model: ModelKey; size?: number }) {+ return (+ <span+ aria-hidden+ className="inline-block shrink-0 rounded-full"+ style={{ width: size, height: size, background: MODEL_COLOR[model] }}+ />+ );+}++function ModelCard({ model, badges }: { model: Model; badges: string[] }) {+ const t = model.totals;+ const pct = rate(t.pass, t.runs);+ return (+ <div className="flex flex-col rounded-[20px] px-6 py-7" style={CARD_STYLE}>+ <div className="flex items-center gap-2.5">+ <ModelSwatch model={model.key} size={12} />+ <span className="text-[17px] font-semibold text-fg-0">{model.label}</span>+ </div>+ <p className="mt-1.5 font-mono text-[0.75rem] break-all text-fg-3">+ {model.id}+ </p>+ <div className="mt-6 flex items-baseline gap-2">+ <span className="text-[48px] leading-none font-semibold tracking-[-0.03em] text-fg-0">+ {pct}%+ </span>+ <span className="text-[0.875rem] text-fg-2">+ {t.pass} of {t.runs} runs passed+ </span>+ </div>+ <div+ className="mt-4 h-1.5 overflow-hidden rounded-full"+ style={{ background: "var(--ink-3)" }}+ role="img"+ aria-label={`${pct}% pass rate`}+ >+ <div+ className="h-full rounded-full"+ style={{ width: `${(100 * t.pass) / t.runs}%`, background: MODEL_COLOR[model.key] }}+ />+ </div>+ <dl className="mt-6 grid gap-3 text-[0.875rem]">+ <CardStat label="Right answer with evidence" value={`${t.rightAnswerEvidence}/${t.runs}`} />+ <CardStat label="Agent cost" value={formatUsd(t.costUsd)} />+ <CardStat label="Cost per run" value={formatUsd(t.costPerRunUsd)} />+ </dl>+ {badges.length ? (+ <div className="mt-6 flex flex-wrap gap-2">+ {badges.map((b) => (+ <span key={b} className="tag">+ {b}+ </span>+ ))}+ </div>+ ) : null}+ </div>+ );+}++function CardStat({ label, value }: { label: string; value: string }) {+ return (+ <div+ className="flex items-baseline justify-between gap-4 pt-3"+ style={{ borderTop: "1px solid var(--hair)" }}+ >+ <dt className="text-fg-2">{label}</dt>+ <dd className="m-0 font-mono text-fg-0 tabular-nums">{value}</dd>+ </div>+ );+}++function Finding({+ label,+ title,+ children,+}: {+ label: string;+ title: string;+ children: React.ReactNode;+}) {+ return (+ <div className="border-t border-hair pt-8">+ <p className="font-mono text-[0.6875rem] uppercase tracking-[0.14em] text-fg-3">+ {label}+ </p>+ <h3 className="mt-3">{title}</h3>+ <p className="muted mt-3 max-w-[520px] text-pretty">{children}</p>+ </div>+ );+}++function DataTable({+ minWidth,+ children,+}: {+ minWidth: number;+ children: React.ReactNode;+}) {+ return (+ <div className="overflow-x-auto rounded-2xl" style={CARD_STYLE}>+ <table className="w-full border-collapse text-left" style={{ minWidth }}>+ {children}+ </table>+ </div>+ );+}++function Th({ children }: { children: React.ReactNode }) {+ return (+ <th+ scope="col"+ className="px-5 py-3.5 align-bottom font-mono text-[0.6875rem] font-medium uppercase tracking-[0.1em] text-fg-3"+ style={{ borderBottom: "1px solid var(--hair-2)" }}+ >+ {children}+ </th>+ );+}++function ModelTh({ model }: { model: Model }) {+ return (+ <th+ scope="col"+ className="px-5 py-3.5 align-bottom text-[0.8125rem] font-semibold text-fg-0"+ style={{ borderBottom: "1px solid var(--hair-2)" }}+ >+ <span className="inline-flex items-center gap-2 whitespace-nowrap">+ <ModelSwatch model={model.key} />+ {model.label}+ </span>+ </th>+ );+}++function Td({ children }: { children: React.ReactNode }) {+ return <td className="px-5 py-4 align-top">{children}</td>;+}++function RowHead({+ title,+ sub,+ code,+ description,+}: {+ title: string;+ sub?: string;+ code?: string;+ description?: string;+}) {+ return (+ <th scope="row" className="min-w-[190px] px-5 py-4 align-top font-normal">+ <span className="block text-[0.9375rem] font-semibold text-fg-0">+ {title}+ </span>+ {code ? (+ <span className="mt-0.5 block font-mono text-[0.75rem] text-fg-3">+ {code}+ </span>+ ) : null}+ {description ? (+ <span className="mt-1.5 block max-w-[300px] text-[0.8125rem] leading-snug text-fg-2">+ {description}+ </span>+ ) : null}+ {sub ? (+ <span className="mt-1 block font-mono text-[0.75rem] text-fg-3">{sub}</span>+ ) : null}+ </th>+ );+}++function SubheadRow({ label, span }: { label: string; span: number }) {+ return (+ <tr style={{ borderTop: "1px solid var(--hair-2)" }}>+ <th+ scope="colgroup"+ colSpan={span}+ className="px-5 pt-5 pb-1 font-mono text-[0.6875rem] font-medium uppercase tracking-[0.14em] text-fg-3"+ >+ {label}+ </th>+ </tr>+ );+}++function SuiteRows({+ suite,+}: {+ suite: (typeof suites)[number];+}) {+ return (+ <>+ <SubheadRow+ label={`${suite.label} · ${suite.cases} cases`}+ span={models.length + 1}+ />+ {suite.setups.map((arm) => (+ <tr key={arm} style={{ borderTop: "1px solid var(--hair)" }}>+ <RowHead title={setupLabels[arm]} code={arm} />+ {models.map((m) => {+ const r = setupResult(suite.id, arm, m.key);+ return (+ <Td key={m.key}>+ <PassCell+ model={m.key}+ pass={r.pass}+ runs={r.runs}+ rightAnswerEvidence={r.rightAnswerEvidence}+ />+ </Td>+ );+ })}+ </tr>+ ))}+ </>+ );+}++function PassCell({+ model,+ pass,+ runs,+ rightAnswerEvidence,+ strong,+}: {+ model: ModelKey;+ pass: number;+ runs: number;+ rightAnswerEvidence?: number;+ strong?: boolean;+}) {+ return (+ <div className="min-w-[140px]">+ <div className="flex items-baseline justify-between gap-3 font-mono tabular-nums">+ <span+ className={+ strong+ ? "text-[0.9375rem] font-semibold text-fg-0"+ : "text-[0.9375rem] text-fg-1"+ }+ >+ {pass}/{runs}+ </span>+ <span className="text-[0.8125rem] text-fg-2">{rate(pass, runs)}%</span>+ </div>+ <div+ className="mt-2 h-1.5 overflow-hidden rounded-full"+ style={{ background: "var(--ink-3)" }}+ >+ <div+ className="h-full rounded-full"+ style={{ width: `${(100 * pass) / runs}%`, background: MODEL_COLOR[model] }}+ />+ </div>+ {rightAnswerEvidence !== undefined ? (+ <p className="mt-2 font-mono text-[0.6875rem] whitespace-nowrap text-fg-3 tabular-nums">+ {rightAnswerEvidence}/{runs} right answer + evidence+ </p>+ ) : null}+ </div>+ );+}++function DeltaRow({+ title,+ sub,+ cells,+}: {+ title: string;+ sub: string;+ cells: { key: ModelKey; pp: number; detail: string }[];+}) {+ return (+ <tr style={{ borderTop: "1px solid var(--hair)" }}>+ <RowHead title={title} sub={sub} />+ {cells.map((c) => (+ <Td key={c.key}>+ <span className="block font-mono text-[0.9375rem] font-semibold text-fg-0 tabular-nums">+ {formatPp(c.pp)}+ </span>+ <span className="mt-1 block font-mono text-[0.6875rem] text-fg-3 tabular-nums">+ {c.detail}+ </span>+ </Td>+ ))}+ </tr>+ );+}++function MetricRow({ label, values }: { label: string; values: string[] }) {+ return (+ <tr style={{ borderTop: "1px solid var(--hair)" }}>+ <th scope="row" className="min-w-[190px] px-5 py-3.5 text-[0.9375rem] font-normal text-fg-1">+ {label}+ </th>+ {values.map((v, i) => (+ <td+ key={models[i].key}+ className="px-5 py-3.5 font-mono text-[0.9375rem] text-fg-0 tabular-nums"+ >+ {v}+ </td>+ ))}+ </tr>+ );+}diff --git a/website/app/product/agent-skills/page.tsx b/website/app/product/agent-skills/page.tsxindex 298e9dca..1bf6ae6c 100644--- a/website/app/product/agent-skills/page.tsx+++ b/website/app/product/agent-skills/page.tsx@@ -86,7 +86,7 @@ export default function AgentSkillsPage() {Works with Claude Code, Cursor, Codex, and any agent that readsSKILL.md.</p>- <p className="mt-3 font-mono text-[0.6875rem] uppercase tracking-[0.08em]">+ <p className="mt-3 flex flex-wrap justify-center gap-x-6 gap-y-2 font-mono text-[0.6875rem] uppercase tracking-[0.08em]"><Linkhref={GITHUB_URL}target="_blank"@@ -95,6 +95,12 @@ export default function AgentSkillsPage() {>View on GitHub →</Link>+ <Link+ href="/benchmarks"+ className="text-fg-2 hover:text-blue-bright transition-colors"+ >+ See the benchmarks →+ </Link></p></div></div>diff --git a/website/app/product/mcp/page.tsx b/website/app/product/mcp/page.tsxindex 830bf993..a6293289 100644--- a/website/app/product/mcp/page.tsx+++ b/website/app/product/mcp/page.tsx@@ -113,7 +113,7 @@ export default function McpPage() {<p className="dim mt-5 font-mono text-[0.75rem]">Works with Claude Code, Claude Desktop, Cursor, and any MCP client.</p>- <p className="mt-3 font-mono text-[0.6875rem] uppercase tracking-[0.08em]">+ <p className="mt-3 flex flex-wrap justify-center gap-x-6 gap-y-2 font-mono text-[0.6875rem] uppercase tracking-[0.08em]"><Linkhref="https://docs.tracewayapp.com/learn/mcp"target="_blank"@@ -122,6 +122,12 @@ export default function McpPage() {>Read the MCP docs →</Link>+ <Link+ href="/benchmarks"+ className="text-fg-2 hover:text-blue-bright transition-colors"+ >+ See the benchmarks →+ </Link></p></div></div>diff --git a/website/components/benchmark-case-matrix.tsx b/website/components/benchmark-case-matrix.tsxnew file mode 100644index 00000000..bee84183--- /dev/null+++ b/website/components/benchmark-case-matrix.tsx@@ -0,0 +1,636 @@+"use client";++import { Fragment, useMemo, useState } from "react";+import { Check, ChevronRight, Redo2, Timer, X } from "lucide-react";++import { cn } from "@/lib/utils";+import { KIND_LABEL, MODEL_COLOR, formatUsd } from "@/lib/benchmark-format";+import type {+ BenchmarkCase,+ CaseRun,+ ModelKey,+ RunKind,+} from "@/lib/benchmarks";++type MatrixModel = { key: ModelKey; label: string; short: string };+type MatrixSuite = { id: string; label: string; setups: string[] };+type MatrixGroup = { id: string; label: string };+type MatrixIssue = { id: string; kind: string; title: string };++type Hover = {+ caseId: string;+ model: MatrixModel;+ arm: string;+ run: CaseRun;+ x: number;+ y: number;+ above: boolean;+};++const KIND_ICON: Record<RunKind, typeof Check> = {+ pass: Check,+ trajectory: Redo2,+ wrong: X,+ budget: Timer,+};++const CELL_STYLE: Record<RunKind, React.CSSProperties> = {+ pass: {+ background: "color-mix(in oklab, var(--ok) 14%, transparent)",+ color: "var(--ok)",+ },+ trajectory: {+ border: "1px solid color-mix(in oklab, var(--crit) 55%, transparent)",+ color: "var(--crit)",+ },+ wrong: {+ background: "color-mix(in oklab, var(--crit) 16%, transparent)",+ color: "var(--crit)",+ },+ budget: {+ background: "color-mix(in oklab, var(--crit) 16%, transparent)",+ color: "var(--crit)",+ },+};++const TOOLTIP_WIDTH = 340;++export function BenchmarkCaseMatrix({+ cases,+ models,+ suites,+ groups,+ setupLabels,+ issues,+}: {+ cases: BenchmarkCase[];+ models: MatrixModel[];+ suites: MatrixSuite[];+ groups: MatrixGroup[];+ setupLabels: Record<string, string>;+ issues: MatrixIssue[];+}) {+ const [group, setGroup] = useState("all");+ const [failuresOnly, setFailuresOnly] = useState(false);+ const [expanded, setExpanded] = useState<Set<string>>(() => new Set());+ const [hover, setHover] = useState<Hover | null>(null);++ const issueById = useMemo(+ () => Object.fromEntries(issues.map((i) => [i.id, i])),+ [issues],+ );++ const hasFailure = (c: BenchmarkCase) =>+ models.some((m) =>+ Object.values(c.runs[m.key]).some((r) => r.status !== "pass"),+ );++ const visible = cases.filter(+ (c) =>+ (group === "all" || c.group === group) &&+ (!failuresOnly || hasFailure(c)),+ );+ const visibleIds = visible.map((c) => c.id);+ const allExpanded =+ visibleIds.length > 0 && visibleIds.every((id) => expanded.has(id));++ function toggle(id: string) {+ setExpanded((prev) => {+ const next = new Set(prev);+ if (next.has(id)) next.delete(id);+ else next.add(id);+ return next;+ });+ }++ function toggleAll() {+ setExpanded((prev) => {+ const next = new Set(prev);+ for (const id of visibleIds) {+ if (allExpanded) next.delete(id);+ else next.add(id);+ }+ return next;+ });+ }++ function showTooltip(+ el: HTMLElement,+ caseId: string,+ model: MatrixModel,+ arm: string,+ run: CaseRun,+ ) {+ const rect = el.getBoundingClientRect();+ const x = Math.min(+ Math.max(rect.left + rect.width / 2 - TOOLTIP_WIDTH / 2, 12),+ window.innerWidth - TOOLTIP_WIDTH - 12,+ );+ const above = rect.bottom + 220 > window.innerHeight;+ setHover({+ caseId,+ model,+ arm,+ run,+ x,+ y: above ? rect.top - 8 : rect.bottom + 8,+ above,+ });+ }++ const groupCounts = Object.fromEntries(+ groups.map((g) => [g.id, cases.filter((c) => c.group === g.id).length]),+ );++ return (+ <div>+ <div className="flex flex-wrap items-center gap-2">+ <FilterChip active={group === "all"} onClick={() => setGroup("all")}>+ All cases · {cases.length}+ </FilterChip>+ {groups.map((g) => (+ <FilterChip+ key={g.id}+ active={group === g.id}+ onClick={() => setGroup(g.id)}+ >+ {g.label} · {groupCounts[g.id]}+ </FilterChip>+ ))}+ </div>+ <div className="mt-3 flex flex-wrap items-center gap-2">+ <FilterChip+ active={failuresOnly}+ onClick={() => setFailuresOnly((v) => !v)}+ >+ Only cases with a failure+ </FilterChip>+ <button+ type="button"+ onClick={toggleAll}+ className="tag transition-colors hover:text-fg-0"+ disabled={visibleIds.length === 0}+ >+ {allExpanded ? "Collapse all" : "Expand all"}+ </button>+ </div>++ <Legend />++ {suites.map((suite) => {+ const rows = visible.filter((c) => c.suite === suite.id);+ if (!rows.length) return null;+ const suiteGroups = groups.filter((g) =>+ rows.some((c) => c.group === g.id),+ );+ const columns = suite.setups.flatMap((arm) =>+ models.map((m) => ({ arm, model: m })),+ );+ return (+ <div key={suite.id} className="mt-8">+ <p className="font-mono text-[0.6875rem] uppercase tracking-[0.14em] text-fg-3">+ {suite.label} · {rows.length}{" "}+ {rows.length === 1 ? "case" : "cases"} × {suite.setups.length}{" "}+ setups × {models.length} models+ </p>+ <div+ className="mt-3 overflow-x-auto rounded-xl"+ style={{+ background: "var(--ink-0)",+ border: "1px solid var(--hair-2)",+ }}+ >+ <table className="w-full border-collapse text-left">+ <thead>+ <tr>+ <th+ rowSpan={2}+ scope="col"+ className="sticky left-0 z-10 w-[190px] min-w-[190px] px-4 py-3 align-bottom font-mono text-[0.6875rem] font-medium uppercase tracking-[0.1em] text-fg-3 sm:w-[340px] sm:min-w-[340px]"+ style={{+ background: "var(--ink-0)",+ borderBottom: "1px solid var(--hair-2)",+ }}+ >+ Case+ </th>+ {suite.setups.map((arm) => (+ <th+ key={arm}+ colSpan={models.length}+ scope="colgroup"+ className="px-2 pt-3 pb-1 text-center font-mono text-[0.6875rem] font-medium uppercase tracking-[0.1em] whitespace-nowrap text-fg-2"+ style={{ borderLeft: "1px solid var(--hair)" }}+ >+ {setupLabels[arm] ?? arm}+ </th>+ ))}+ </tr>+ <tr>+ {columns.map(({ arm, model }, i) => (+ <th+ key={`${arm}-${model.key}`}+ scope="col"+ className="px-1 pb-2 text-center font-mono text-[0.6875rem] font-medium whitespace-nowrap text-fg-2"+ style={{+ borderBottom: "1px solid var(--hair-2)",+ borderLeft:+ i % models.length === 0+ ? "1px solid var(--hair)"+ : undefined,+ }}+ >+ <span className="inline-flex items-center gap-1">+ <span+ aria-hidden+ className="inline-block size-2 rounded-full"+ style={{ background: MODEL_COLOR[model.key] }}+ />+ {model.short}+ </span>+ </th>+ ))}+ </tr>+ </thead>+ <tbody>+ {suiteGroups.map((g) => (+ <Fragment key={g.id}>+ {suiteGroups.length > 1 || suite.id !== g.id ? (+ <tr>+ <th+ scope="colgroup"+ colSpan={columns.length + 1}+ className="px-4 pt-5 pb-2 font-mono text-[0.6875rem] font-medium uppercase tracking-[0.14em] text-fg-3"+ >+ <span className="sticky left-4 inline-block">+ {g.label}+ </span>+ </th>+ </tr>+ ) : null}+ {rows+ .filter((c) => c.group === g.id)+ .map((c) => {+ const open = expanded.has(c.id);+ return (+ <Fragment key={c.id}>+ <tr+ className="group cursor-pointer transition-colors hover:bg-[color:var(--ink-1)]"+ style={{ borderTop: "1px solid var(--hair)" }}+ onClick={() => toggle(c.id)}+ >+ <th+ scope="row"+ className="sticky left-0 z-10 bg-[color:var(--ink-0)] px-4 py-2.5 align-top font-normal transition-colors group-hover:bg-[color:var(--ink-1)]"+ >+ <button+ type="button"+ aria-expanded={open}+ onClick={(e) => {+ e.stopPropagation();+ toggle(c.id);+ }}+ className="flex w-full items-start gap-1.5 text-left"+ >+ <ChevronRight+ aria-hidden+ className={cn(+ "mt-[3px] size-3.5 shrink-0 text-fg-3 transition-transform",+ open && "rotate-90",+ )}+ />+ <span className="min-w-0">+ <span className="block font-mono text-[0.8125rem] break-words text-fg-0">+ {c.id}+ </span>+ <span className="mt-0.5 line-clamp-2 text-[0.8125rem] leading-snug text-fg-2">+ {c.title}+ </span>+ </span>+ </button>+ </th>+ {columns.map(({ arm, model }, i) => {+ const run = c.runs[model.key][arm];+ return (+ <td+ key={`${arm}-${model.key}`}+ className="px-1 py-2.5 text-center align-top"+ style={{+ borderLeft:+ i % models.length === 0+ ? "1px solid var(--hair)"+ : undefined,+ }}+ >+ {run ? (+ <RunCell+ run={run}+ label={`${model.label}, ${setupLabels[arm] ?? arm}: ${KIND_LABEL[run.kind]}`}+ onClick={(e) => {+ e.stopPropagation();+ toggle(c.id);+ }}+ onShow={(el) =>+ showTooltip(el, c.id, model, arm, run)+ }+ onHide={() => setHover(null)}+ />+ ) : null}+ </td>+ );+ })}+ </tr>+ {open ? (+ <tr>+ <td colSpan={columns.length + 1} className="p-0">+ <CaseDetails+ c={c}+ models={models}+ setups={suite.setups}+ setupLabels={setupLabels}+ issueById={issueById}+ />+ </td>+ </tr>+ ) : null}+ </Fragment>+ );+ })}+ </Fragment>+ ))}+ </tbody>+ <tfoot>+ <tr style={{ borderTop: "1px solid var(--hair-2)" }}>+ <th+ scope="row"+ className="sticky left-0 z-10 px-4 py-3 text-left font-mono text-[0.6875rem] font-medium uppercase tracking-[0.1em] text-fg-3"+ style={{ background: "var(--ink-0)" }}+ >+ Passed{rows.length !== cases.filter((c) => c.suite === suite.id).length ? " (shown)" : ""}+ </th>+ {columns.map(({ arm, model }, i) => {+ const ran = rows.filter((c) => c.runs[model.key][arm]);+ const passed = ran.filter(+ (c) => c.runs[model.key][arm].status === "pass",+ ).length;+ return (+ <td+ key={`${arm}-${model.key}`}+ className="px-1 py-3 text-center font-mono text-[0.75rem] whitespace-nowrap text-fg-1 tabular-nums"+ style={{+ borderLeft:+ i % models.length === 0+ ? "1px solid var(--hair)"+ : undefined,+ }}+ >+ {passed}/{ran.length}+ </td>+ );+ })}+ </tr>+ </tfoot>+ </table>+ </div>+ </div>+ );+ })}++ {visible.length === 0 ? (+ <p className="muted mt-8">No case matches these filters.</p>+ ) : null}++ {hover ? (+ <div+ role="tooltip"+ className="pointer-events-none fixed z-50 rounded-lg p-3.5 text-left"+ style={{+ left: hover.x,+ top: hover.y,+ width: TOOLTIP_WIDTH,+ maxWidth: "calc(100vw - 24px)",+ transform: hover.above ? "translateY(-100%)" : undefined,+ background: "var(--ink-0)",+ border: "1px solid var(--hair-2)",+ boxShadow: "0 18px 40px -20px rgba(10,14,24,0.45)",+ }}+ >+ <p className="font-mono text-[0.6875rem] uppercase tracking-[0.1em] text-fg-3">+ {hover.caseId}+ </p>+ <p className="mt-1 flex items-center gap-1.5 text-[0.875rem] font-semibold text-fg-0">+ <span+ aria-hidden+ className="inline-block size-2 rounded-full"+ style={{ background: MODEL_COLOR[hover.model.key] }}+ />+ {hover.model.label} · {setupLabels[hover.arm] ?? hover.arm}+ </p>+ <RunSummary run={hover.run} issueById={issueById} clamp />+ </div>+ ) : null}+ </div>+ );+}++function FilterChip({+ active,+ onClick,+ children,+}: {+ active: boolean;+ onClick: () => void;+ children: React.ReactNode;+}) {+ return (+ <button+ type="button"+ onClick={onClick}+ aria-pressed={active}+ className={cn("tag transition-colors", active ? "active" : "hover:text-fg-0")}+ >+ {children}+ </button>+ );+}++function Legend() {+ const items: { kind: RunKind; label: string }[] = [+ { kind: "pass", label: KIND_LABEL.pass },+ { kind: "trajectory", label: KIND_LABEL.trajectory },+ { kind: "wrong", label: KIND_LABEL.wrong },+ ];+ return (+ <div className="mt-6 flex flex-wrap items-center gap-x-6 gap-y-2 text-[0.8125rem] text-fg-2">+ {items.map((it) => (+ <span key={it.kind} className="inline-flex items-center gap-2">+ <CellGlyph kind={it.kind} />+ {it.label}+ </span>+ ))}+ <span className="inline-flex items-center gap-2">+ <span className="relative inline-flex size-[26px] items-center justify-center rounded-md border border-hair-2">+ <IssueDot />+ </span>+ Hit by a known case or skill problem+ </span>+ </div>+ );+}++function IssueDot() {+ return (+ <span+ aria-hidden+ className="absolute top-[3px] right-[3px] size-[5px] rounded-full"+ style={{ background: "var(--fg-1)" }}+ />+ );+}++function CellGlyph({ kind, issue }: { kind: RunKind; issue?: boolean }) {+ const Icon = KIND_ICON[kind];+ return (+ <span+ className="relative inline-flex size-[26px] items-center justify-center rounded-md"+ style={CELL_STYLE[kind]}+ >+ <Icon aria-hidden className="size-3.5" strokeWidth={2.5} />+ {issue ? <IssueDot /> : null}+ </span>+ );+}++function RunCell({+ run,+ label,+ onClick,+ onShow,+ onHide,+}: {+ run: CaseRun;+ label: string;+ onClick: (e: React.MouseEvent) => void;+ onShow: (el: HTMLElement) => void;+ onHide: () => void;+}) {+ return (+ <button+ type="button"+ aria-label={label}+ onClick={onClick}+ onPointerEnter={(e) => {+ if (e.pointerType === "mouse") onShow(e.currentTarget);+ }}+ onPointerLeave={onHide}+ onFocus={(e) => {+ if (e.currentTarget.matches(":focus-visible")) onShow(e.currentTarget);+ }}+ onBlur={onHide}+ className="rounded-md p-0.5 transition-transform hover:scale-110 focus-visible:outline-2 focus-visible:outline-offset-1 focus-visible:outline-[color:var(--a1)]"+ >+ <CellGlyph kind={run.kind} issue={!!run.knownIssue} />+ </button>+ );+}++function RunSummary({+ run,+ issueById,+ clamp,+}: {+ run: CaseRun;+ issueById: Record<string, MatrixIssue>;+ clamp?: boolean;+}) {+ const issue = run.knownIssue ? issueById[run.knownIssue] : null;+ return (+ <div className="mt-2 flex flex-col gap-1.5">+ <p className="flex items-center gap-2 text-[0.8125rem] text-fg-1">+ <CellGlyph kind={run.kind} />+ <span>+ {KIND_LABEL[run.kind]}+ {run.graders.length ? (+ <span className="text-fg-3"> · failed {run.graders.join(", ")}</span>+ ) : null}+ </span>+ </p>+ {run.firstFailure ? (+ <p+ className={cn(+ "font-mono text-[0.75rem] leading-relaxed break-words text-fg-1",+ clamp && "line-clamp-4",+ )}+ >+ {run.firstFailure}+ </p>+ ) : null}+ {issue ? (+ <p className="text-[0.75rem] text-fg-2">+ Known {issue.kind} problem: {issue.title}+ </p>+ ) : null}+ <p className="font-mono text-[0.6875rem] text-fg-3 tabular-nums">+ {run.toolCalls} tool calls · {formatUsd(run.costUsd)} ·{" "}+ {Math.round(run.wallS)}s+ {run.judge !== null ? ` · judge ${run.judge.toFixed(2)}` : ""}+ </p>+ </div>+ );+}++function CaseDetails({+ c,+ models,+ setups,+ setupLabels,+ issueById,+}: {+ c: BenchmarkCase;+ models: MatrixModel[];+ setups: string[];+ setupLabels: Record<string, string>;+ issueById: Record<string, MatrixIssue>;+}) {+ const all = setups.flatMap((arm) =>+ models+ .filter((m) => c.runs[m.key][arm])+ .map((m) => ({ arm, model: m, run: c.runs[m.key][arm] })),+ );+ const failed = all.filter((x) => x.run.status !== "pass");+ return (+ <div+ className="sticky left-0 w-[min(100vw_-_40px,880px)] px-4 pt-1 pb-5 sm:pl-10"+ style={{ background: "var(--ink-0)" }}+ >+ <p className="text-[0.8125rem] text-fg-2">+ {all.length - failed.length} of {all.length} runs passed.+ {failed.length ? " The failures, with the first reason the graders gave:" : ""}+ </p>+ {failed.length ? (+ <ul className="mt-3 grid gap-3 md:grid-cols-2">+ {failed.map(({ arm, model, run }) => (+ <li+ key={`${arm}-${model.key}`}+ className="rounded-lg p-3.5"+ style={{+ background: "var(--ink-1)",+ border: "1px solid var(--hair)",+ }}+ >+ <p className="flex items-center gap-1.5 text-[0.8125rem] font-semibold text-fg-0">+ <span+ aria-hidden+ className="inline-block size-2 rounded-full"+ style={{ background: MODEL_COLOR[model.key] }}+ />+ {model.label} · {setupLabels[arm] ?? arm}+ </p>+ <RunSummary run={run} issueById={issueById} />+ </li>+ ))}+ </ul>+ ) : null}+ </div>+ );+}diff --git a/website/components/benchmark-comparison-table.tsx b/website/components/benchmark-comparison-table.tsxnew file mode 100644index 00000000..f3f04852--- /dev/null+++ b/website/components/benchmark-comparison-table.tsx@@ -0,0 +1,164 @@+import { MODEL_COLOR } from "@/lib/benchmark-format";+import type { ModelKey } from "@/lib/benchmarks";++export type ComparisonValue = {+ value: number;+ text: string;+ note?: string;+ title?: string;+};++export type ComparisonRow = {+ label: string;+ sub: string;+ better: "higher" | "lower";+ values: Record<ModelKey, ComparisonValue[]>;+};++export type ComparisonModel = { key: ModelKey; label: string; note: string };++function bestOf(row: ComparisonRow, models: ComparisonModel[], i: number) {+ const vs = models.map((m) => row.values[m.key][i].value);+ return row.better === "higher" ? Math.max(...vs) : Math.min(...vs);+}++export function BenchmarkComparisonTable({+ models,+ rows,+}: {+ models: ComparisonModel[];+ rows: ComparisonRow[];+}) {+ const featured = models[0].key;+ const outline = "color-mix(in oklab, var(--a1) 70%, transparent)";++ return (+ <div className="overflow-x-auto">+ <table+ className="w-full min-w-[560px] border-separate text-left md:min-w-[680px]"+ style={{ borderSpacing: 0 }}+ >+ <thead>+ <tr>+ <th+ className="sticky left-0 z-10 w-[132px] pb-3 md:static md:w-[26%]"+ style={{ background: "var(--ink-0)" }}+ />+ {models.map((m) => {+ const isFeatured = m.key === featured;+ return (+ <th+ key={m.key}+ scope="col"+ className="px-3 pt-4 pb-3 text-center align-bottom font-normal"+ style={+ isFeatured+ ? {+ border: `1.5px solid ${outline}`,+ borderBottom: "none",+ borderRadius: "14px 14px 0 0",+ }+ : undefined+ }+ >+ <span className="inline-flex items-center gap-2 text-[0.9375rem] font-medium text-fg-0 md:text-[1.0625rem]">+ <span+ aria-hidden+ className="inline-block h-2.5 w-2.5 rounded-full"+ style={{ background: MODEL_COLOR[m.key] }}+ />+ {m.label}+ </span>+ <span className="block font-mono text-[0.6875rem] text-fg-3">+ {m.note}+ </span>+ </th>+ );+ })}+ </tr>+ </thead>+ <tbody>+ {rows.map((row, ri) => {+ const last = ri === rows.length - 1;+ const n = row.values[featured].length;+ const best = Array.from({ length: n }, (_, i) =>+ bestOf(row, models, i),+ );+ return (+ <tr key={row.label}>+ <th+ scope="row"+ className="sticky left-0 z-10 py-5 pr-3 align-middle font-normal md:static md:pr-4"+ style={{+ borderTop: "1px solid var(--hair-2)",+ background: "var(--ink-0)",+ }}+ >+ <span className="block text-[0.875rem] leading-snug font-medium text-fg-0 md:text-[1rem]">+ {row.label}+ </span>+ <span className="mt-0.5 block text-[0.75rem] text-fg-3 md:text-[0.8125rem]">+ {row.sub}+ </span>+ </th>+ {models.map((m) => {+ const values = row.values[m.key];+ const wins = values.map((v, i) => v.value === best[i]);+ const shaded = wins.every(Boolean);+ const isFeatured = m.key === featured;+ return (+ <td+ key={m.key}+ className="px-3 py-4 text-center align-middle"+ style={{+ borderTop: "1px solid var(--hair-2)",+ background: shaded+ ? isFeatured+ ? "color-mix(in oklab, var(--a1) 13%, var(--ink-0))"+ : "color-mix(in oklab, var(--fg-0) 5%, var(--ink-0))"+ : undefined,+ borderLeft: isFeatured+ ? `1.5px solid ${outline}`+ : undefined,+ borderRight: isFeatured+ ? `1.5px solid ${outline}`+ : undefined,+ borderBottom:+ isFeatured && last+ ? `1.5px solid ${outline}`+ : undefined,+ borderRadius:+ isFeatured && last ? "0 0 14px 14px" : undefined,+ }}+ >+ <div className="flex flex-col items-center gap-2">+ {values.map((v, i) => (+ <span key={i} className="block" title={v.title}>+ <span+ className="block text-[1.0625rem] tabular-nums"+ style={{+ color: "var(--fg-0)",+ fontWeight: wins[i] ? 700 : 400,+ }}+ >+ {v.text}+ </span>+ {v.note ? (+ <span className="block text-[0.6875rem] leading-tight text-fg-3">+ {v.note}+ </span>+ ) : null}+ </span>+ ))}+ </div>+ </td>+ );+ })}+ </tr>+ );+ })}+ </tbody>+ </table>+ </div>+ );+}diff --git a/website/components/benchmark-cost-chart.tsx b/website/components/benchmark-cost-chart.tsxnew file mode 100644index 00000000..9e316e35--- /dev/null+++ b/website/components/benchmark-cost-chart.tsx@@ -0,0 +1,273 @@+"use client";++import { useEffect, useRef, useState } from "react";++import { MODEL_COLOR, formatUsd, rate } from "@/lib/benchmark-format";+import type { ModelKey } from "@/lib/benchmarks";++export type CostPoint = {+ id: string;+ model: ModelKey;+ modelLabel: string;+ modelShort: string;+ label: string;+ pass: number;+ runs: number;+ costPerRunUsd: number;+ overall: boolean;+};++const HEIGHT = 340;+const MARGIN = { top: 20, right: 24, bottom: 52, left: 52 };+const LOG_TICKS = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30];++function domainFor(points: CostPoint[]) {+ const costs = points.map((p) => p.costPerRunUsd);+ const lo = Math.max(...LOG_TICKS.filter((t) => t <= Math.min(...costs)));+ const hi = Math.min(...LOG_TICKS.filter((t) => t >= Math.max(...costs)));+ const rates = points.map((p) => (100 * p.pass) / p.runs);+ const yLo = Math.max(0, Math.floor(Math.min(...rates) / 10) * 10);+ return { lo, hi, yLo, yHi: 100 };+}++export function BenchmarkCostChart({+ points,+ models,+}: {+ points: CostPoint[];+ models: { key: ModelKey; label: string }[];+}) {+ const wrapRef = useRef<HTMLDivElement>(null);+ const [width, setWidth] = useState(720);+ const [active, setActive] = useState<string | null>(null);++ useEffect(() => {+ const el = wrapRef.current;+ if (!el) return;+ const ro = new ResizeObserver(([entry]) =>+ setWidth(Math.max(280, Math.round(entry.contentRect.width))),+ );+ ro.observe(el);+ return () => ro.disconnect();+ }, []);++ const { lo, hi, yLo, yHi } = domainFor(points);+ const plotW = width - MARGIN.left - MARGIN.right;+ const plotH = HEIGHT - MARGIN.top - MARGIN.bottom;+ const x = (usd: number) =>+ MARGIN.left ++ ((Math.log10(usd) - Math.log10(lo)) / (Math.log10(hi) - Math.log10(lo))) *+ plotW;+ const y = (pct: number) =>+ MARGIN.top + (1 - (pct - yLo) / (yHi - yLo)) * plotH;++ const xTicks = LOG_TICKS.filter((t) => t >= lo && t <= hi);+ const yTicks: number[] = [];+ for (let t = yLo; t <= yHi; t += 10) yTicks.push(t);++ const ordered = [+ ...points.filter((p) => !p.overall),+ ...points.filter((p) => p.overall),+ ];+ const activePoint = points.find((p) => p.id === active) ?? null;++ return (+ <div>+ <div className="flex flex-wrap items-center gap-x-5 gap-y-2 text-[0.8125rem] text-fg-1">+ {models.map((m) => (+ <span key={m.key} className="inline-flex items-center gap-2">+ <span+ aria-hidden+ className="inline-block size-2.5 rounded-full"+ style={{ background: MODEL_COLOR[m.key] }}+ />+ {m.label}+ </span>+ ))}+ <span className="text-fg-3">+ Large dot: all runs · small dots: one suite and setup+ </span>+ </div>++ <div ref={wrapRef} className="relative mt-4">+ <svg+ width={width}+ height={HEIGHT}+ viewBox={`0 0 ${width} ${HEIGHT}`}+ role="img"+ aria-label="Pass rate against agent cost per run, one dot per model and per suite and setup"+ className="block max-w-full"+ >+ {yTicks.map((t) => (+ <g key={`y${t}`}>+ <line+ x1={MARGIN.left}+ x2={width - MARGIN.right}+ y1={y(t)}+ y2={y(t)}+ stroke="var(--hair-2)"+ strokeWidth={1}+ />+ <text+ x={MARGIN.left - 10}+ y={y(t)}+ dy="0.32em"+ textAnchor="end"+ fontSize={11}+ fill="var(--fg-3)"+ style={{ fontFamily: "var(--font-mono)" }}+ >+ {t}%+ </text>+ </g>+ ))}+ {xTicks.map((t) => (+ <g key={`x${t}`}>+ <line+ x1={x(t)}+ x2={x(t)}+ y1={MARGIN.top}+ y2={HEIGHT - MARGIN.bottom}+ stroke="var(--hair)"+ strokeWidth={1}+ />+ <text+ x={x(t)}+ y={HEIGHT - MARGIN.bottom + 18}+ textAnchor="middle"+ fontSize={11}+ fill="var(--fg-3)"+ style={{ fontFamily: "var(--font-mono)" }}+ >+ ${t}+ </text>+ </g>+ ))}+ <text+ x={MARGIN.left + plotW / 2}+ y={HEIGHT - 8}+ textAnchor="middle"+ fontSize={11}+ fill="var(--fg-2)"+ style={{ fontFamily: "var(--font-mono)" }}+ >+ Agent cost per run (log scale)+ </text>+ <text+ transform={`translate(14 ${MARGIN.top + plotH / 2}) rotate(-90)`}+ textAnchor="middle"+ fontSize={11}+ fill="var(--fg-2)"+ style={{ fontFamily: "var(--font-mono)" }}+ >+ Pass rate+ </text>++ {ordered.map((p) => {+ const cx = x(p.costPerRunUsd);+ const cy = y((100 * p.pass) / p.runs);+ const r = p.overall ? 7 : 4.5;+ const isActive = active === p.id;+ const labelLeft = cx > MARGIN.left + plotW * 0.62;+ return (+ <g key={p.id}>+ <circle+ cx={cx}+ cy={cy}+ r={isActive ? r + 1.5 : r}+ fill={MODEL_COLOR[p.model]}+ fillOpacity={p.overall ? 1 : 0.7}+ stroke="var(--ink-0)"+ strokeWidth={2}+ />+ {p.overall ? (+ <text+ x={labelLeft ? cx - 14 : cx + 14}+ y={cy}+ dy="0.32em"+ textAnchor={labelLeft ? "end" : "start"}+ fontSize={12}+ fill="var(--fg-0)"+ fontWeight={600}+ >+ {width < 560 ? p.modelShort : p.modelLabel} ·{" "}+ {rate(p.pass, p.runs)}%+ </text>+ ) : null}+ <circle+ cx={cx}+ cy={cy}+ r={12}+ fill="transparent"+ tabIndex={0}+ role="img"+ aria-label={`${p.modelLabel}, ${p.label}: ${p.pass} of ${p.runs} passed (${rate(p.pass, p.runs)}%), ${formatUsd(p.costPerRunUsd)} per run`}+ onPointerEnter={() => setActive(p.id)}+ onPointerLeave={() => setActive(null)}+ onFocus={() => setActive(p.id)}+ onBlur={() => setActive(null)}+ style={{ outline: "none", cursor: "default" }}+ />+ </g>+ );+ })}+ </svg>++ {activePoint ? (+ <Tooltip+ point={activePoint}+ left={x(activePoint.costPerRunUsd)}+ top={y((100 * activePoint.pass) / activePoint.runs)}+ width={width}+ />+ ) : null}+ </div>+ </div>+ );+}++function Tooltip({+ point,+ left,+ top,+ width,+}: {+ point: CostPoint;+ left: number;+ top: number;+ width: number;+}) {+ const boxW = 220;+ const clamped = Math.min(Math.max(left - boxW / 2, 0), width - boxW);+ const below = top < 110;+ return (+ <div+ role="tooltip"+ className="pointer-events-none absolute z-10 rounded-lg px-3 py-2.5"+ style={{+ left: clamped,+ top: below ? top + 16 : top - 16,+ width: boxW,+ transform: below ? undefined : "translateY(-100%)",+ background: "var(--ink-0)",+ border: "1px solid var(--hair-2)",+ boxShadow: "0 18px 40px -20px rgba(10,14,24,0.45)",+ }}+ >+ <p className="font-mono text-[0.9375rem] font-semibold text-fg-0 tabular-nums">+ {rate(point.pass, point.runs)}% · {formatUsd(point.costPerRunUsd)}/run+ </p>+ <p className="mt-1 flex items-center gap-1.5 text-[0.8125rem] text-fg-1">+ <span+ aria-hidden+ className="inline-block h-0.5 w-3 rounded-full"+ style={{ background: MODEL_COLOR[point.model] }}+ />+ {point.modelLabel}+ </p>+ <p className="mt-0.5 text-[0.8125rem] text-fg-2">+ {point.label} · {point.pass}/{point.runs} passed+ </p>+ </div>+ );+}diff --git a/website/components/benchmark-index-chart.tsx b/website/components/benchmark-index-chart.tsxnew file mode 100644index 00000000..d95b1dfa--- /dev/null+++ b/website/components/benchmark-index-chart.tsx@@ -0,0 +1,270 @@+"use client";++import { useState } from "react";++import { MODEL_COLOR, rate } from "@/lib/benchmark-format";+import type { ModelKey } from "@/lib/benchmarks";++export type IndexBar = {+ id: string;+ model: ModelKey;+ weights: "open" | "proprietary";+ label: string;+ detail?: string;+ pass: number;+ rightAnswerEvidence: number;+ runs: number;+};++export type IndexView = {+ id: string;+ tab: string;+ title: string;+ bars: IndexBar[];+};++type Metric = "pass" | "rightAnswerEvidence";++const METRICS: { id: Metric; label: string }[] = [+ { id: "pass", label: "Pass rate" },+ { id: "rightAnswerEvidence", label: "Right answer and evidence" },+];++export const WEIGHTS_COLOR = {+ proprietary: "var(--fg-0)",+ open: "#6aa5f8",+} as const;++const WEIGHTS_TEXT = {+ proprietary: "var(--ink-0)",+ open: "#ffffff",+} as const;++const PLOT_HEIGHT = 300;+const GRID = [0, 25, 50, 75, 100];++export function BenchmarkIndexChart({+ views,+ subtitle,+}: {+ views: IndexView[];+ subtitle: string;+}) {+ const [viewId, setViewId] = useState(views[0].id);+ const [metric, setMetric] = useState<Metric>("pass");+ const view = views.find((v) => v.id === viewId) ?? views[0];+ const value = (b: IndexBar) => rate(b[metric], b.runs);+ const bars = [...view.bars].sort((a, b) => value(b) - value(a));+ const dense = bars.length > 6;+ const minWidth = dense ? 64 + bars.length * 46 : 0;+ const metricLabel = METRICS.find((m) => m.id === metric)!.label;++ return (+ <div+ className="overflow-hidden rounded-[14px]"+ style={{+ background: "var(--ink-0)",+ border: "1px solid var(--hair-2)",+ boxShadow:+ "0 1px 2px rgba(10,14,24,0.04), 0 12px 30px -18px rgba(10,14,24,0.14)",+ }}+ >+ <div+ role="tablist"+ aria-label="Group the bars"+ className="flex gap-1 overflow-x-auto px-3 py-3 md:px-4"+ style={{+ background: "var(--ink-1)",+ borderBottom: "1px solid var(--hair-2)",+ }}+ >+ {views.map((v) => {+ const active = v.id === view.id;+ return (+ <button+ key={v.id}+ type="button"+ role="tab"+ aria-selected={active}+ onClick={() => setViewId(v.id)}+ className="shrink-0 cursor-pointer rounded-md px-4 py-2 text-[0.9375rem] transition-colors"+ style={{+ background: active ? "var(--ink-3)" : "transparent",+ color: active ? "var(--fg-0)" : "var(--fg-2)",+ fontWeight: active ? 600 : 500,+ }}+ >+ {v.tab}+ </button>+ );+ })}+ </div>++ <div className="px-5 pt-6 pb-5 md:px-8 md:pt-8">+ <div className="flex flex-col gap-5 md:flex-row md:items-start md:justify-between">+ <div className="max-w-[60ch]">+ <h3 className="text-[1.375rem] leading-tight font-semibold tracking-[-0.01em] text-fg-0 md:text-[1.75rem]">+ {metricLabel} {view.title}+ </h3>+ <p className="mt-2 text-[0.875rem] leading-relaxed text-fg-3">+ {subtitle}+ </p>+ </div>+ <div+ role="group"+ aria-label="Metric"+ className="inline-flex shrink-0 self-start rounded-lg p-1"+ style={{ border: "1px solid var(--hair-2)" }}+ >+ {METRICS.map((m) => {+ const active = m.id === metric;+ return (+ <button+ key={m.id}+ type="button"+ aria-pressed={active}+ onClick={() => setMetric(m.id)}+ className="cursor-pointer rounded-md px-3 py-1.5 text-[0.8125rem] transition-colors"+ style={{+ background: active ? "var(--ink-2)" : "transparent",+ color: active ? "var(--fg-0)" : "var(--fg-2)",+ fontWeight: active ? 600 : 500,+ }}+ >+ {m.label}+ </button>+ );+ })}+ </div>+ </div>++ <div className="mt-6 flex flex-wrap items-end justify-between gap-3">+ <div className="flex flex-wrap gap-x-5 gap-y-1 text-[0.9375rem] text-fg-1">+ {(["proprietary", "open"] as const).map((w) => (+ <span key={w} className="inline-flex items-center gap-2">+ <span+ aria-hidden+ className="inline-block h-2.5 w-2.5 rounded-full"+ style={{ background: WEIGHTS_COLOR[w] }}+ />+ {w === "open" ? "Open weights" : "Proprietary"}+ </span>+ ))}+ </div>+ <span className="inline-flex items-center gap-1.5 text-[1.0625rem] font-semibold tracking-[-0.01em] text-fg-1">+ {/* eslint-disable-next-line @next/next/no-img-element */}+ <img src="/favicon.svg" alt="" className="h-4 w-4" />+ Traceway+ </span>+ </div>++ <div className="mt-3 overflow-x-auto pb-1">+ <div style={{ minWidth, paddingLeft: dense ? 64 : 0 }}>+ <div className="relative" style={{ height: PLOT_HEIGHT }}>+ {GRID.map((g) => (+ <div+ key={g}+ aria-hidden+ className="absolute inset-x-0"+ style={{+ bottom: `${g}%`,+ borderTop:+ g === 0+ ? "1px solid var(--hair-2)"+ : "1px dashed var(--hair-2)",+ }}+ />+ ))}+ <div+ className="absolute inset-0 flex items-end justify-center"+ style={{ gap: dense ? 6 : 24 }}+ >+ {bars.map((b) => {+ const v = value(b);+ return (+ <div+ key={b.id}+ className="flex h-full flex-1 items-end justify-center"+ style={{ maxWidth: dense ? 56 : 120 }}+ >+ <div+ role="img"+ aria-label={`${b.label}${b.detail ? `, ${b.detail}` : ""}: ${v}% (${b[metric]} of ${b.runs} runs)`}+ title={`${b.label}${b.detail ? ` · ${b.detail}` : ""}: ${b[metric]}/${b.runs} runs`}+ className="flex w-full items-center justify-center rounded-t-[4px] transition-[height] duration-500"+ style={{+ height: `${Math.max(v, 1)}%`,+ background: WEIGHTS_COLOR[b.weights],+ }}+ >+ {v >= 12 ? (+ <span+ className="font-medium tabular-nums"+ style={{+ color: WEIGHTS_TEXT[b.weights],+ fontSize: dense ? "0.75rem" : "1rem",+ }}+ >+ {v}+ </span>+ ) : null}+ </div>+ </div>+ );+ })}+ </div>+ </div>++ <div+ className="flex justify-center"+ style={{ gap: dense ? 6 : 24 }}+ >+ {bars.map((b) => (+ <div+ key={b.id}+ className="flex flex-1 flex-col items-center"+ style={{ maxWidth: dense ? 56 : 120 }}+ >+ <span+ aria-hidden+ className="mt-3 inline-block h-3 w-3 rounded-[3px]"+ style={{ background: MODEL_COLOR[b.model] }}+ />+ {dense ? (+ <div+ className="relative w-full"+ style={{ height: 136 }}+ >+ <span+ className="absolute top-1.5 text-right text-[0.75rem] leading-tight whitespace-nowrap text-fg-1"+ style={{+ right: "50%",+ transform: "rotate(-45deg)",+ transformOrigin: "100% 0",+ }}+ >+ {b.label}+ {b.detail ? (+ <span className="block text-fg-3">{b.detail}</span>+ ) : null}+ </span>+ </div>+ ) : (+ <div className="mt-2 text-center text-[0.875rem] leading-snug text-fg-1">+ {b.label}+ {b.detail ? (+ <span className="block text-[0.75rem] text-fg-3">+ {b.detail}+ </span>+ ) : null}+ </div>+ )}+ </div>+ ))}+ </div>+ </div>+ </div>+ </div>+ </div>+ );+}diff --git a/website/components/mobile-nav.tsx b/website/components/mobile-nav.tsxindex ca6bc294..422957ea 100644--- a/website/components/mobile-nav.tsx+++ b/website/components/mobile-nav.tsx@@ -21,6 +21,7 @@ const SECONDARY: (NavItem & { external?: boolean })[] = [{ title: "Cloud", href: "/cloud" },{ title: "Blog", href: "/blog" },{ title: "Releases", href: "/releases" },+ { title: "Benchmarks", href: "/benchmarks" },{ title: "Docs", href: "https://docs.tracewayapp.com", external: true },{ title: "GitHub", href: GITHUB_URL, external: true },{ title: "Discord", href: DISCORD_URL, external: true },diff --git a/website/components/site-footer.tsx b/website/components/site-footer.tsxindex 255a82c6..c854cd57 100644--- a/website/components/site-footer.tsx+++ b/website/components/site-footer.tsx@@ -40,6 +40,7 @@ const COLUMNS: Column[] = [links: [{ label: "Blog", href: "/blog" },{ label: "Release Notes", href: "/releases" },+ { label: "Benchmarks", href: "/benchmarks" },{ label: "Docs", href: "https://docs.tracewayapp.com", external: true },{ label: "GitHub", href: GITHUB_URL, external: true },{ label: "Discord", href: DISCORD_URL, external: true },diff --git a/website/components/site-header.tsx b/website/components/site-header.tsxindex c0b216bd..accb980f 100644--- a/website/components/site-header.tsx+++ b/website/components/site-header.tsx@@ -334,6 +334,13 @@ export function SiteHeader() {>Blog</Link>+ <Link+ href="/benchmarks"+ className="h-9 px-3 rounded-md hidden lg:inline-flex items-center text-[14px] font-medium transition-colors text-[color:var(--fg-1)] hover:text-[color:var(--fg-0)] hover:bg-[color:var(--ink-2)]"+ style={{ fontFamily: "var(--font-display)" }}+ >+ Benchmarks+ </Link><Linkhref="https://docs.tracewayapp.com"target="_blank"diff --git a/website/content/benchmarks/2026-10-21.md b/website/content/benchmarks/2026-10-21.mdnew file mode 100644index 00000000..08039894--- /dev/null+++ b/website/content/benchmarks/2026-10-21.md@@ -0,0 +1,704 @@+---+title: "Traceway agent benchmarks: 2026-10-21"+date: "2026-10-21"+commit: "3b00b083"+cliVersion: "cli/v2.0.5"+backendVersion: "backend/v2.0.5"+harnessUrl: "https://github.com/tracewayapp/traceway/tree/main/cli/test/evals"+methodologyUrl: "https://docs.tracewayapp.com/learn/agent-benchmarks"+runsPerCase: 1+judge:+ id: "anthropic/claude-haiku-5.5"+ label: "Claude Haiku 5.5"+suites:+ - id: "perf"+ label: "Performance"+ - id: "write"+ label: "Write"+models:+ - key: "haiku"+ label: "Claude Haiku 5.5"+ short: "Haiku"+ vendor: "Anthropic"+ weights: "proprietary"+ id: "anthropic/claude-haiku-5.5"+ provider: "Anthropic"+ providers:+ - name: "Anthropic"+ runs: 131+ reasoningEffort: "medium"+ maxTokensPerTurn: 16000+ - key: "deepseek-flash"+ label: "DeepSeek V4.1 Flash"+ short: "DeepSeek"+ vendor: "DeepSeek"+ weights: "open"+ id: "deepseek/deepseek-v4.1-flash-1017"+ provider: "DeepSeek"+ providers:+ - name: "DeepSeek"+ runs: 131+ - name: "Relace"+ runs: 2+ reasoningEffort: "medium"+ maxTokensPerTurn: 16000+ - key: "opus"+ label: "Claude Opus 5.5"+ short: "Opus"+ vendor: "Anthropic"+ weights: "proprietary"+ id: "anthropic/claude-opus-5.5"+ provider: "Anthropic"+ providers:+ - name: "Anthropic"+ runs: 131+ reasoningEffort: "medium"+ maxTokensPerTurn: 16000+---++# Traceway agent benchmarks: 2026-10-21++Three models, Claude Haiku 5.5, DeepSeek V4.1 Flash and Claude Opus 5.5, each ran the same 131 eval runs on 2026-10-21, at commit `3b00b083`. Every run gets a fresh Traceway backend seeded with the telemetry of a system broken in one specific way, and deterministic code grades the result. [How we measure](https://docs.tracewayapp.com/learn/agent-benchmarks) · [the harness and the cases](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals).++This file is generated by `website/scripts/build-benchmarks-data.mjs` from the eval results and is the data behind https://tracewayapp.com/benchmarks. Change the generator, not this file.++## Summary per model++A run passes when its answer is right, its evidence is grounded and its writes are safe. "Right answer and evidence" also counts runs that failed only the trajectory check.++| Model | Runs | Passed | Pass rate | Right answer and evidence | Not graded | Agent cost | Cost per run |+|---|---:|---:|---:|---:|---:|---:|---:|+| Claude Haiku 5.5 | 131 | 93 | 71% | 98 | 0 | $1.54 | $0.01174 |+| DeepSeek V4.1 Flash | 131 | 106 | 81% | 109 | 0 | $1.51 | $0.01153 |+| Claude Opus 5.5 | 131 | 118 | 90% | 123 | 0 | $40.01 | $0.30542 |++## Pass rate by suite and setup++| Suite | Setup | Label | Description | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |+|---|---|---|---|---|---|---|+| perf | cli+skill | CLI + skill | The traceway CLI plus the Traceway agent skill. | 28/35 (80%) | 27/35 (77%) | 33/35 (94%) |+| perf | cli-noskill | CLI, no skill | The same CLI with no skill: the control group. | 19/35 (54%) | 24/35 (69%) | 28/35 (80%) |+| perf | mcp | MCP server | The Traceway MCP server: the same knowledge, plus tools. | 24/35 (69%) | 30/35 (86%) | 31/35 (89%) |+| write | cli-write+skill | Write CLI + skill | The traceway-write CLI plus the write skill. | 10/13 (77%) | 12/13 (92%) | 13/13 (100%) |+| write | mcp-write | Write MCP server | The write MCP server. | 12/13 (92%) | 13/13 (100%) | 13/13 (100%) |+| all | | All runs | | 93/131 (71%) | 106/131 (81%) | 118/131 (90%) |++### Right answer and evidence by suite and setup++| Suite | Setup | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |+|---|---|---|---|---|+| perf | cli+skill | 29/35 | 27/35 | 34/35 |+| perf | cli-noskill | 22/35 | 26/35 | 30/35 |+| perf | mcp | 24/35 | 31/35 | 33/35 |+| write | cli-write+skill | 11/13 | 12/13 | 13/13 |+| write | mcp-write | 12/13 | 13/13 | 13/13 |++## Skill uplift and CLI vs MCP++Setup A minus setup B, in percentage points, computed before rounding.++| Comparison | Suite | Setup A | Setup B | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |+|---|---|---|---|---|---|---|+| Skill uplift | perf | cli+skill | cli-noskill | +26pp (80% vs 54%) | +9pp (77% vs 69%) | +14pp (94% vs 80%) |+| CLI vs MCP | perf | cli+skill | mcp | +11pp (80% vs 69%) | -9pp (77% vs 86%) | +6pp (94% vs 89%) |+| CLI vs MCP | write | cli-write+skill | mcp-write | -15pp (77% vs 92%) | -8pp (92% vs 100%) | +0pp (100% vs 100%) |++## By case group++Passes over every setup a case ran on.++| Group | Label | Description | Cases | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |+|---|---|---|---|---|---|---|+| perf-app | App code | Slow code in the request path: serialization, quadratic loops, large allocations, debug logging, work that belongs in the background. | 5 | 11/15 (73%) | 13/15 (87%) | 13/15 (87%) |+| perf-cache | Caching | Cold starts after a release, a missing cache, a cache stampede. | 3 | 7/9 (78%) | 5/9 (56%) | 5/9 (56%) |+| perf-ctl | Controls and traps | A healthy endpoint, endpoints marked slow on purpose, a misleading metric, a pasted alert and an onset to pin down. Some have nothing to fix. | 6 | 9/18 (50%) | 11/18 (61%) | 18/18 (100%) |+| perf-db | Database | N+1 queries, a missing index, pool exhaustion, lock contention, over-fetching, unbounded results. | 8 | 20/24 (83%) | 21/24 (88%) | 23/24 (96%) |+| perf-infra | Infrastructure | CPU saturation, disk I/O, memory pressure, a noisy neighbor. | 4 | 5/12 (42%) | 8/12 (67%) | 9/12 (75%) |+| perf-net | Network and upstreams | Retries, chatty fan-out, sequential calls, large payloads, connection reuse, a slow third party. | 6 | 14/18 (78%) | 15/18 (83%) | 16/18 (89%) |+| perf-xs | Cross-stack and mobile | A browser or mobile app fails, and the agent has to follow it into the backend. | 3 | 5/9 (56%) | 8/9 (89%) | 8/9 (89%) |+| write | Write actions | Acknowledge or resolve pages, snooze or disable alerts, run a monitor, archive issues, write post-mortems, refuse what a token may not do. | 13 | 22/26 (85%) | 25/26 (96%) | 26/26 (100%) |++## Cost and efficiency++Every run per model. Agent cost is the model spend the harness recorded for each run, through OpenRouter; the judge's spend is separate. Runs went three at a time on one machine.++| Metric | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |+|---|---:|---:|---:|+| Agent spend | $1.54 | $1.51 | $40.01 |+| Average cost per run | $0.01174 | $0.01153 | $0.30542 |+| Cost per passing run | $0.01653 | $0.01425 | $0.33907 |+| Average tool calls per run | 14.4 | 21.9 | 13.3 |+| Average wall time per run | 57.1 s | 69.2 s | 73.9 s |+| Cache hit rate (input tokens) | 85% | 91% | 87% |+| Average judge score (0-1) | 0.73 | 0.89 | 0.93 |+| Runs scored by the judge | 130 | 131 | 131 |+| Judge spend | $0.1452 | $0.1866 | $0.1378 |++### Cost by suite and setup++| Suite | Setup | Model | Agent cost | Cost per run | Avg tool calls | Cache hit | Judge |+|---|---|---|---:|---:|---:|---:|---:|+| perf | cli+skill | Claude Haiku 5.5 | $0.40 | $0.01146 | 12.7 | 87% | 0.75 |+| perf | cli+skill | DeepSeek V4.1 Flash | $0.45 | $0.01286 | 24.6 | 93% | 0.87 |+| perf | cli+skill | Claude Opus 5.5 | $12.02 | $0.34340 | 14.3 | 90% | 0.93 |+| perf | cli-noskill | Claude Haiku 5.5 | $0.33 | $0.00944 | 20.2 | 90% | 0.65 |+| perf | cli-noskill | DeepSeek V4.1 Flash | $0.39 | $0.01114 | 27.7 | 92% | 0.88 |+| perf | cli-noskill | Claude Opus 5.5 | $10.07 | $0.28774 | 16.5 | 90% | 0.90 |+| perf | mcp | Claude Haiku 5.5 | $0.68 | $0.01953 | 16.1 | 82% | 0.72 |+| perf | mcp | DeepSeek V4.1 Flash | $0.55 | $0.01567 | 22.4 | 89% | 0.93 |+| perf | mcp | Claude Opus 5.5 | $13.58 | $0.38809 | 13.9 | 83% | 0.93 |+| write | cli-write+skill | Claude Haiku 5.5 | $0.06 | $0.00470 | 6.9 | 81% | 0.77 |+| write | cli-write+skill | DeepSeek V4.1 Flash | $0.05 | $0.00398 | 10.6 | 88% | 0.87 |+| write | cli-write+skill | Claude Opus 5.5 | $1.67 | $0.12847 | 7.3 | 84% | 0.95 |+| write | mcp-write | Claude Haiku 5.5 | $0.06 | $0.00472 | 5.9 | 76% | 0.92 |+| write | mcp-write | DeepSeek V4.1 Flash | $0.07 | $0.00542 | 8.8 | 84% | 0.87 |+| write | mcp-write | Claude Opus 5.5 | $2.67 | $0.20511 | 6.7 | 76% | 0.95 |++## Known issues++Problems in our own cases and skill, not in the models. The runs they hit are still counted as failures. The model columns count the runs each issue hit.++| Issue | Kind | Case | Title | Detail | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 | Runs of the case | Truncated output |+|---|---|---|---|---|---|---|---|---|---|+| iowait-recipe | skill | perf-infra-disk-io | The skill's CPU recipe counts iowait as busy CPU | On the disk case the database host's idle CPU falls while iowait rises. The skill defines busy CPU as 1 minus idle, so runs that followed it called the disk bottleneck CPU saturation, which is the case's distractor. | 2 | 1 | 0 | 9 | 0 |+| stampede-logs-query | case | perf-cache-stampede | The cache stampede case insists on a logs query | The case requires a logs.query call, but agents now find the burst of identical slow rebuilds through the newer endpoint request listing. These runs had the right answer and grounded evidence. | 2 | 1 | 3 | 9 | 0 |+| memory-gc-label | case | perf-infra-memory-gc | The memory case rejects a fair label | These runs found the memo that never evicts, but filed it under unbounded in-memory work instead of memory pressure and GC pauses. The grader treats that label as the distractor, though an unbounded memo is honestly both. | 1 | 1 | 2 | 9 | 0 |+| chatty-fanout-grounding | case | perf-net-chatty-fanout | The fan-out case fails grounding for every model | Every model fails this case's grounding check on most setups, mostly because the output it cites was truncated. | 3 | 3 | 2 | 9 | 5 |++## Full case matrix++One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure reason>`, where the kind is `wrong answer/evidence` (the answer or its evidence failed), `trajectory only` (the answer and its evidence were right, but a required call was missing or out of order) or `budget exhausted`. Descriptions summarize the case without giving the answer away.++| Case | Group | Description | Suite | Setup | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 |+|---|---|---|---|---|---|---|---|+| perf-app-cpu-serialization | perf-app | The transaction export burns seconds of CPU building its JSON body | perf | cli+skill | pass | pass | pass |+| perf-app-cpu-serialization | perf-app | The transaction export burns seconds of CPU building its JSON body | perf | cli-noskill | pass | pass | pass |+| perf-app-cpu-serialization | perf-app | The transaction export burns seconds of CPU building its JSON body | perf | mcp | pass | pass | pass |+| perf-app-excessive-logging | perf-app | A release that only turned on debug logs slows every endpoint and doubles CPU | perf | cli+skill | pass | pass | pass |+| perf-app-excessive-logging | perf-app | A release that only turned on debug logs slows every endpoint and doubles CPU | perf | cli-noskill | pass | fail (trajectory only): trajectory: required call missing: metrics.query(name=~system\.cpu) | fail (wrong answer/evidence): grounding: no cited call's output contains "LOG_LEVEL=debug" |+| perf-app-excessive-logging | perf-app | A release that only turned on debug logs slows every endpoint and doubles CPU | perf | mcp | pass | pass | pass |+| perf-app-large-alloc | perf-app | The retention report loads every event of the range into the JVM heap | perf | cli+skill | pass | fail (wrong answer/evidence): answer: checklist_item "no_pagination_unbounded_result_set", want unbounded_in_memory_work_large_allocations | pass |+| perf-app-large-alloc | perf-app | The retention report loads every event of the range into the JVM heap | perf | cli-noskill | fail (wrong answer/evidence): grounding: the submission cites no evidence | pass | pass |+| perf-app-large-alloc | perf-app | The retention report loads every event of the range into the JVM heap | perf | mcp | pass | pass | pass |+| perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | cli+skill | pass | pass | pass |+| perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | cli-noskill | fail (wrong answer/evidence): grounding: evidence[6] cites c13, which failed | pass | fail (wrong answer/evidence): grounding: evidence[2] cites c05, whose output was truncated |+| perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | mcp | pass | pass | pass |+| perf-app-sync-background | perf-app | Generating a statement holds the request open while the PDF renders | perf | cli+skill | fail (wrong answer/evidence): grounding: evidence[10] cites c14, which failed | pass | pass |+| perf-app-sync-background | perf-app | Generating a statement holds the request open while the PDF renders | perf | cli-noskill | pass | pass | pass |+| perf-app-sync-background | perf-app | Generating a statement holds the request open while the PDF renders | perf | mcp | fail (wrong answer/evidence): grounding: the submission cites no evidence | pass | pass |+| perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | cli+skill | pass | fail (wrong answer/evidence): grounding: no cited call's output contains "HikariPool-1 - Starting..." | pass |+| perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | cli-noskill | pass | pass | fail (wrong answer/evidence): grounding: evidence[1] cites c08, whose output was truncated |+| perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | mcp | pass | fail (wrong answer/evidence): grounding: no cited call's output contains "HikariPool-1 - Starting..." | pass |+| perf-cache-no-cache | perf-cache | The catalog endpoint keeps the database busy all day | perf | cli+skill | pass | fail (wrong answer/evidence): answer: checklist_item "missing_query_result_caching", want no_cache_low_hit_ratio | pass |+| perf-cache-no-cache | perf-cache | The catalog endpoint keeps the database busy all day | perf | cli-noskill | pass | pass | pass |+| perf-cache-no-cache | perf-cache | The catalog endpoint keeps the database busy all day | perf | mcp | pass | pass | pass |+| perf-cache-stampede | perf-cache | Home page requests stall for seconds in bursts | perf | cli+skill | fail (trajectory only): trajectory: required call missing: logs.query | pass | fail (trajectory only): trajectory: required call missing: logs.query |+| perf-cache-stampede | perf-cache | Home page requests stall for seconds in bursts | perf | cli-noskill | fail (trajectory only): trajectory: required call missing: logs.query | pass | fail (trajectory only): trajectory: required call missing: logs.query |+| perf-cache-stampede | perf-cache | Home page requests stall for seconds in bursts | perf | mcp | pass | fail (trajectory only): trajectory: required call missing: logs.query | fail (trajectory only): trajectory: required call missing: logs.query |+| perf-ctl-alert-handoff | perf-ctl | A p95 latency alert pasted in from email: what happened, and does it need action? | perf | cli+skill | pass | fail (wrong answer/evidence): answer: checklist_item "expensive_join_or_aggregation", want cold_start_jit_connection_warmup | pass |+| perf-ctl-alert-handoff | perf-ctl | A p95 latency alert pasted in from email: what happened, and does it need action? | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "cache_stampede_thundering_herd", want cold_start_jit_connection_warmup | fail (trajectory only): trajectory: required call missing: endpoints.slow(endpoint=GET /api/categories/{slug}/products) | pass |+| perf-ctl-alert-handoff | perf-ctl | A p95 latency alert pasted in from email: what happened, and does it need action? | perf | mcp | fail (wrong answer/evidence): answer: checklist_item "insufficient_evidence", want cold_start_jit_connection_warmup | fail (wrong answer/evidence): answer: checklist_item "cache_stampede_thundering_herd", want cold_start_jit_connection_warmup | pass |+| perf-ctl-healthy | perf-ctl | Search "feels slow" but the endpoint is healthy and stable | perf | cli+skill | pass | fail (wrong answer/evidence): answer: checklist_item "missing_or_unused_index", want none_found | pass |+| perf-ctl-healthy | perf-ctl | Search "feels slow" but the endpoint is healthy and stable | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "missing_or_unused_index", want none_found | fail (wrong answer/evidence): answer: checklist_item "missing_or_unused_index", want none_found | pass |+| perf-ctl-healthy | perf-ctl | Search "feels slow" but the endpoint is healthy and stable | perf | mcp | fail (wrong answer/evidence): answer: checklist_item "missing_or_unused_index", want none_found | fail (wrong answer/evidence): answer: checklist_item "missing_or_unused_index", want none_found | pass |+| perf-ctl-marked-slow-ok | perf-ctl | Ledger export is the slowest endpoint but sits inside its marked-slow allowance | perf | cli+skill | pass | pass | pass |+| perf-ctl-marked-slow-ok | perf-ctl | Ledger export is the slowest endpoint but sits inside its marked-slow allowance | perf | cli-noskill | pass | pass | pass |+| perf-ctl-marked-slow-ok | perf-ctl | Ledger export is the slowest endpoint but sits inside its marked-slow allowance | perf | mcp | fail (wrong answer/evidence): answer: checklist_item "no_pagination_unbounded_result_set", want none_found | pass | pass |+| perf-ctl-marked-slow-regressed | perf-ctl | Marked-slow invoice export regressed past its allowance after a release | perf | cli+skill | pass | pass | pass |+| perf-ctl-marked-slow-regressed | perf-ctl | Marked-slow invoice export regressed past its allowance after a release | perf | cli-noskill | fail (wrong answer/evidence): trajectory: required call missing: endpoints.slow(endpoint=GET /api/exports/invoices) | pass | pass |+| perf-ctl-marked-slow-regressed | perf-ctl | Marked-slow invoice export regressed past its allowance after a release | perf | mcp | pass | pass | pass |+| perf-ctl-metric-p95-trap | perf-ctl | Search gets sluggish every afternoon, and ops says the hosts' p95 CPU looks fine | perf | cli+skill | pass | pass | pass |+| perf-ctl-metric-p95-trap | perf-ctl | Search gets sluggish every afternoon, and ops says the hosts' p95 CPU looks fine | perf | cli-noskill | fail (wrong answer/evidence): report: report does not match /(9[5-9]\|100)(\.[0-9]+)?\s*%\|0\.9[5-9]/ | fail (wrong answer/evidence): report: report does not match /average\|\bavg\b\|\bmean\b/ | pass |+| perf-ctl-metric-p95-trap | perf-ctl | Search gets sluggish every afternoon, and ops says the hosts' p95 CPU looks fine | perf | mcp | fail (wrong answer/evidence): report: report does not match /average\|\bavg\b\|\bmean\b/ | pass | pass |+| perf-ctl-onset-bisect | perf-ctl | The account page has felt slower for a few days: when did it start, and what changed? | perf | cli+skill | pass | pass | pass |+| perf-ctl-onset-bisect | perf-ctl | The account page has felt slower for a few days: when did it start, and what changed? | perf | cli-noskill | fail (trajectory only): trajectory: required call missing: endpoints.list(from=present) | pass | pass |+| perf-ctl-onset-bisect | perf-ctl | The account page has felt slower for a few days: when did it start, and what changed? | perf | mcp | pass | pass | pass |+| perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | cli+skill | pass | pass | pass |+| perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | cli-noskill | pass | fail (wrong answer/evidence): grounding: evidence[4] cites c16, whose output was truncated | pass |+| perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | mcp | pass | pass | pass |+| perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | cli+skill | pass | pass | pass |+| perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | cli-noskill | fail (trajectory only): trajectory: required call missing: logs.query | fail (wrong answer/evidence): grounding: evidence[1] cites c11, whose output was truncated | pass |+| perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | mcp | pass | pass | pass |+| perf-db-missing-index | perf-db | The user lookup gets slower every week as the users table grows | perf | cli+skill | pass | pass | pass |+| perf-db-missing-index | perf-db | The user lookup gets slower every week as the users table grows | perf | cli-noskill | pass | pass | pass |+| perf-db-missing-index | perf-db | The user lookup gets slower every week as the users table grows | perf | mcp | pass | pass | pass |+| perf-db-missing-query-cache | perf-db | Every storefront endpoint is slower than it should be, even a single-row read | perf | cli+skill | pass | pass | pass |+| perf-db-missing-query-cache | perf-db | Every storefront endpoint is slower than it should be, even a single-row read | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "no_cache_low_hit_ratio", want missing_query_result_caching | pass | pass |+| perf-db-missing-query-cache | perf-db | Every storefront endpoint is slower than it should be, even a single-row read | perf | mcp | pass | pass | pass |+| perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | cli+skill | pass | pass | pass |+| perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | cli-noskill | pass | pass | pass |+| perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | mcp | fail (wrong answer/evidence): grounding: evidence[2] cites c04, whose output was truncated | pass | pass |+| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | cli+skill | pass | pass | pass |+| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | cli-noskill | pass | fail (wrong answer/evidence): grounding: evidence[6] cites c16, whose output was truncated | pass |+| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | mcp | fail (wrong answer/evidence): grounding: evidence[1] cites c03, whose output was truncated | pass | pass |+| perf-db-pool-exhaustion | perf-db | Checkout stalls for seconds at the lunch and evening rushes | perf | cli+skill | pass | pass | pass |+| perf-db-pool-exhaustion | perf-db | Checkout stalls for seconds at the lunch and evening rushes | perf | cli-noskill | pass | pass | pass |+| perf-db-pool-exhaustion | perf-db | Checkout stalls for seconds at the lunch and evening rushes | perf | mcp | pass | pass | pass |+| perf-db-unbounded-result | perf-db | The admin audit log gets slower every week as organizations accumulate events | perf | cli+skill | pass | pass | pass |+| perf-db-unbounded-result | perf-db | The admin audit log gets slower every week as organizations accumulate events | perf | cli-noskill | pass | pass | fail (trajectory only): trajectory: none of these alternatives happened: [endpoints.show(recorded_at=present) + endpoints.chart] or [traces.show(recorded_at=present) + endpoints.chart] or [endpoints.show(recorded_at=present) + endpoints.list(from=present)] or [traces.show(recorded_at=present) + endpoints.list(from=present)] or [endpoints.show(recorded_at=present) + endpoints.list(since=~^([7-9]\|[1-9][0-9]+)d$)] or [traces.show(recorded_at=present) + endpoints.list(since=~^([7-9]\|[1-9][0-9]+)d$)] |+| perf-db-unbounded-result | perf-db | The admin audit log gets slower every week as organizations accumulate events | perf | mcp | pass | pass | pass |+| perf-infra-cpu-saturation | perf-infra | Traffic tripled and every endpoint slowed down with the CPU pinned | perf | cli+skill | pass | pass | pass |+| perf-infra-cpu-saturation | perf-infra | Traffic tripled and every endpoint slowed down with the CPU pinned | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "lock_transaction_contention", want cpu_saturation | fail (wrong answer/evidence): answer: checklist_item "undersized_noisy_neighbor_instance", want cpu_saturation | pass |+| perf-infra-cpu-saturation | perf-infra | Traffic tripled and every endpoint slowed down with the CPU pinned | perf | mcp | pass | pass | pass |+| perf-infra-disk-io | perf-infra | Every query got several times slower while the database host's disk queue filled up | perf | cli+skill | fail (wrong answer/evidence): answer: checklist_item "cpu_saturation", want disk_io_bottleneck | fail (wrong answer/evidence): answer: checklist_item "cpu_saturation", want disk_io_bottleneck | pass |+| perf-infra-disk-io | perf-infra | Every query got several times slower while the database host's disk queue filled up | perf | cli-noskill | fail (wrong answer/evidence): answer: locus.host "bookings-db.internal", want "pg-bookings-1" | pass | pass |+| perf-infra-disk-io | perf-infra | Every query got several times slower while the database host's disk queue filled up | perf | mcp | fail (wrong answer/evidence): answer: checklist_item "cpu_saturation", want disk_io_bottleneck | pass | pass |+| perf-infra-memory-gc | perf-infra | The heap climbs to its limit every seven hours and every endpoint stalls until the pod restarts | perf | cli+skill | fail (wrong answer/evidence): answer: checklist_item "unbounded_in_memory_work_large_allocations", want memory_pressure_gc_pauses | fail (wrong answer/evidence): answer: checklist_item "unbounded_in_memory_work_large_allocations", want memory_pressure_gc_pauses | fail (wrong answer/evidence): answer: checklist_item "unbounded_in_memory_work_large_allocations", want memory_pressure_gc_pauses |+| perf-infra-memory-gc | perf-infra | The heap climbs to its limit every seven hours and every endpoint stalls until the pod restarts | perf | cli-noskill | pass | pass | pass |+| perf-infra-memory-gc | perf-infra | The heap climbs to its limit every seven hours and every endpoint stalls until the pod restarts | perf | mcp | fail (wrong answer/evidence): answer: checklist_item "unbounded_in_memory_work_large_allocations", want memory_pressure_gc_pauses | pass | fail (wrong answer/evidence): answer: checklist_item "unbounded_in_memory_work_large_allocations", want memory_pressure_gc_pauses |+| perf-infra-noisy-neighbor | perf-infra | One of three VMs loses a third of its CPU to steal and serves every request slowly | perf | cli+skill | pass | pass | pass |+| perf-infra-noisy-neighbor | perf-infra | One of three VMs loses a third of its CPU to steal and serves every request slowly | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "insufficient_evidence", want undersized_noisy_neighbor_instance | fail (wrong answer/evidence): answer: onset 2026-10-21T06:00:00Z is -2h15m0s from 2026-10-21T08:15:00Z (tolerance 1h0m0s) | pass |+| perf-infra-noisy-neighbor | perf-infra | One of three VMs loses a third of its CPU to steal and serves every request slowly | perf | mcp | pass | pass | fail (trajectory only): trajectory: none of these alternatives happened: [metrics.query(group_by=~host, name=~system\.cpu) + metrics.query(group_by=~host, name=~http\.server\.request\.duration)] or [metrics.query(group_by=~host, name=~system\.cpu) + logs.query] or [metrics.query(name=~system\.cpu, tag_filters=~host) + metrics.query(group_by=~host, name=~http\.server\.request\.duration)] or [metrics.query(name=~system\.cpu, tag_filters=~host) + logs.query] |+| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | cli+skill | fail (wrong answer/evidence): grounding: evidence[7] cites c11, whose output was truncated | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | pass |+| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | cli-noskill | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | fail (wrong answer/evidence): grounding: evidence[4] cites c12, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[4] cites c09, which failed |+| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | mcp | fail (wrong answer/evidence): grounding: evidence[4] cites c06, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[4] cites c08, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[2] cites c03, whose output was truncated |+| perf-net-large-payload | perf-net | The feed is slow, worst on phones | perf | cli+skill | pass | pass | pass |+| perf-net-large-payload | perf-net | The feed is slow, worst on phones | perf | cli-noskill | pass | pass | pass |+| perf-net-large-payload | perf-net | The feed is slow, worst on phones | perf | mcp | pass | pass | pass |+| perf-net-no-conn-reuse | perf-net | Store finder spends most of its time in calls to the maps provider | perf | cli+skill | pass | pass | pass |+| perf-net-no-conn-reuse | perf-net | Store finder spends most of its time in calls to the maps provider | perf | cli-noskill | pass | pass | pass |+| perf-net-no-conn-reuse | perf-net | Store finder spends most of its time in calls to the maps provider | perf | mcp | pass | pass | pass |+| perf-net-retries | perf-net | Shipping quotes occasionally take 6 to 13 seconds | perf | cli+skill | fail (wrong answer/evidence): trajectory: required call missing: logs.query | pass | pass |+| perf-net-retries | perf-net | Shipping quotes occasionally take 6 to 13 seconds | perf | cli-noskill | pass | pass | pass |+| perf-net-retries | perf-net | Shipping quotes occasionally take 6 to 13 seconds | perf | mcp | pass | pass | pass |+| perf-net-sequential | perf-net | Portal dashboard is slow although every service behind it is fast | perf | cli+skill | pass | pass | pass |+| perf-net-sequential | perf-net | Portal dashboard is slow although every service behind it is fast | perf | cli-noskill | pass | pass | pass |+| perf-net-sequential | perf-net | Portal dashboard is slow although every service behind it is fast | perf | mcp | pass | pass | pass |+| perf-net-slow-third-party | perf-net | Checkout hangs every day or so, then recovers | perf | cli+skill | pass | pass | pass |+| perf-net-slow-third-party | perf-net | Checkout hangs every day or so, then recovers | perf | cli-noskill | pass | pass | pass |+| perf-net-slow-third-party | perf-net | Checkout hangs every day or so, then recovers | perf | mcp | pass | pass | pass |+| perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | cli+skill | pass | pass | pass |+| perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | cli-noskill | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c05, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated |+| perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | mcp | pass | pass | pass |+| perf-xs-frontend-timeouts | perf-xs | Product search on the web shop spins, then says it is taking too long | perf | cli+skill | pass | pass | pass |+| perf-xs-frontend-timeouts | perf-xs | Product search on the web shop spins, then says it is taking too long | perf | cli-noskill | pass | pass | pass |+| perf-xs-frontend-timeouts | perf-xs | Product search on the web shop spins, then says it is taking too long | perf | mcp | pass | pass | pass |+| perf-xs-ios-watchdog | perf-xs | iOS users see the app freeze on the splash screen and close | perf | cli+skill | fail (wrong answer/evidence): answer: checklist_item "synchronous_work_that_belongs_in_a_background_task", want connection_pool_exhaustion | pass | pass |+| perf-xs-ios-watchdog | perf-xs | iOS users see the app freeze on the splash screen and close | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "synchronous_work_that_belongs_in_a_background_task", want connection_pool_exhaustion | pass | pass |+| perf-xs-ios-watchdog | perf-xs | iOS users see the app freeze on the splash screen and close | perf | mcp | fail (wrong answer/evidence): answer: checklist_item "synchronous_work_that_belongs_in_a_background_task", want connection_pool_exhaustion | pass | pass |+| write-alert-disable | write | Turn off the alert for a gateway that is being decommissioned | write | cli-write+skill | pass | pass | pass |+| write-alert-disable | write | Turn off the alert for a gateway that is being decommissioned | write | mcp-write | pass | pass | pass |+| write-alert-snooze | write | Quiet a flapping alert during a known third-party incident | write | cli-write+skill | pass | pass | pass |+| write-alert-snooze | write | Quiet a flapping alert during a known third-party incident | write | mcp-write | pass | pass | pass |+| write-archive-ambiguous | write | Archive last week's checkout errors | write | cli-write+skill | pass | pass | pass |+| write-archive-ambiguous | write | Archive last week's checkout errors | write | mcp-write | pass | pass | pass |+| write-investigate-only | write | Look into a page without changing anything | write | cli-write+skill | pass | pass | pass |+| write-investigate-only | write | Look into a page without changing anything | write | mcp-write | pass | pass | pass |+| write-mark-slow | write | Mark an endpoint that is slow by design as expected-slow | write | cli-write+skill | pass | pass | pass |+| write-mark-slow | write | Mark an endpoint that is slow by design as expected-slow | write | mcp-write | pass | pass | pass |+| write-monitor-confirm-fix | write | Confirm a fix by re-running the checkout monitor | write | cli-write+skill | fail (wrong answer/evidence): trajectory: required call missing: monitors.run(id=2) | fail (wrong answer/evidence): trajectory: required call missing: monitors.run(id=2) | pass |+| write-monitor-confirm-fix | write | Confirm a fix by re-running the checkout monitor | write | mcp-write | fail (wrong answer/evidence): trajectory: required call missing: monitors.run(id=2) | pass | pass |+| write-optin-off | write | Agent writes are turned off for the organization | write | cli-write+skill | pass | pass | pass |+| write-optin-off | write | Agent writes are turned off for the organization | write | mcp-write | pass | pass | pass |+| write-page-ack | write | Acknowledge the checkout page the user is working on | write | cli-write+skill | pass | pass | pass |+| write-page-ack | write | Acknowledge the checkout page the user is working on | write | mcp-write | pass | pass | pass |+| write-page-resolve-archive | write | Close out a page after a rollback and archive its issue | write | cli-write+skill | fail (trajectory only): trajectory: required call missing: ask_user | pass | pass |+| write-page-resolve-archive | write | Close out a page after a rollback and archive its issue | write | mcp-write | pass | pass | pass |+| write-postmortem-conflict | write | Add a follow-up to a post-mortem a teammate is editing | write | cli-write+skill | pass | pass | pass |+| write-postmortem-conflict | write | Add a follow-up to a post-mortem a teammate is editing | write | mcp-write | pass | pass | pass |+| write-postmortem-create | write | Write the post-mortem for a checkout outage | write | cli-write+skill | fail (wrong answer/evidence): trajectory: required call missing: ask_user | pass | pass |+| write-postmortem-create | write | Write the post-mortem for a checkout outage | write | mcp-write | pass | pass | pass |+| write-readonly-member | write | Asked to archive with a readonly member's write token | write | cli-write+skill | pass | pass | pass |+| write-readonly-member | write | Asked to archive with a readonly member's write token | write | mcp-write | pass | pass | pass |+| write-readonly-token | write | Asked to archive with a read-only credential | write | cli-write+skill | pass | pass | pass |+| write-readonly-token | write | Asked to archive with a read-only credential | write | mcp-write | pass | pass | pass |++### Run details++Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1, never gating) and the known issue it hit, if any.++| Case | Setup | Model | Failed graders | Tool calls | Agent cost | Wall time (s) | Judge | Known issue |+|---|---|---|---|---:|---:|---:|---:|---|+| perf-app-cpu-serialization | cli+skill | Claude Haiku 5.5 | | 15 | $0.01503 | 85.1 | 0.33 | |+| perf-app-cpu-serialization | cli+skill | DeepSeek V4.1 Flash | | 31 | $0.01231 | 81.9 | 0.67 | |+| perf-app-cpu-serialization | cli+skill | Claude Opus 5.5 | | 20 | $0.37265 | 102.5 | 1.00 | |+| perf-app-cpu-serialization | cli-noskill | Claude Haiku 5.5 | | 19 | $0.00828 | 55.8 | 0.33 | |+| perf-app-cpu-serialization | cli-noskill | DeepSeek V4.1 Flash | | 32 | $0.01136 | 78.8 | 1.00 | |+| perf-app-cpu-serialization | cli-noskill | Claude Opus 5.5 | | 18 | $0.26551 | 98.0 | 1.00 | |+| perf-app-cpu-serialization | mcp | Claude Haiku 5.5 | | 15 | $0.01183 | 60.2 | 0.67 | |+| perf-app-cpu-serialization | mcp | DeepSeek V4.1 Flash | | 23 | $0.01321 | 59.7 | 1.00 | |+| perf-app-cpu-serialization | mcp | Claude Opus 5.5 | | 17 | $0.35909 | 77.5 | 1.00 | |+| perf-app-excessive-logging | cli+skill | Claude Haiku 5.5 | | 11 | $0.01211 | 58.9 | 0.33 | |+| perf-app-excessive-logging | cli+skill | DeepSeek V4.1 Flash | | 25 | $0.01274 | 68.8 | 1.00 | |+| perf-app-excessive-logging | cli+skill | Claude Opus 5.5 | | 9 | $0.26692 | 66.9 | 1.00 | |+| perf-app-excessive-logging | cli-noskill | Claude Haiku 5.5 | | 32 | $0.01655 | 101.5 | 0.33 | |+| perf-app-excessive-logging | cli-noskill | DeepSeek V4.1 Flash | trajectory | 28 | $0.00988 | 70.9 | 1.00 | |+| perf-app-excessive-logging | cli-noskill | Claude Opus 5.5 | grounding | 22 | $0.33788 | 127.2 | 1.00 | |+| perf-app-excessive-logging | mcp | Claude Haiku 5.5 | | 17 | $0.01226 | 52.9 | 0.33 | |+| perf-app-excessive-logging | mcp | DeepSeek V4.1 Flash | | 29 | $0.02050 | 95.8 | 1.00 | |+| perf-app-excessive-logging | mcp | Claude Opus 5.5 | | 15 | $0.37995 | 68.0 | 0.67 | |+| perf-app-large-alloc | cli+skill | Claude Haiku 5.5 | | 11 | $0.01114 | 64.3 | 1.00 | |+| perf-app-large-alloc | cli+skill | DeepSeek V4.1 Flash | answer | 21 | $0.01293 | 78.2 | 1.00 | |+| perf-app-large-alloc | cli+skill | Claude Opus 5.5 | | 16 | $0.37144 | 91.2 | 1.00 | |+| perf-app-large-alloc | cli-noskill | Claude Haiku 5.5 | grounding | 29 | $0.01380 | 86.6 | 0.67 | |+| perf-app-large-alloc | cli-noskill | DeepSeek V4.1 Flash | | 32 | $0.01642 | 94.8 | 1.00 | |+| perf-app-large-alloc | cli-noskill | Claude Opus 5.5 | | 21 | $0.38448 | 112.5 | 1.00 | |+| perf-app-large-alloc | mcp | Claude Haiku 5.5 | | 15 | $0.01102 | 52.2 | 1.00 | |+| perf-app-large-alloc | mcp | DeepSeek V4.1 Flash | | 30 | $0.01595 | 86.5 | 1.00 | |+| perf-app-large-alloc | mcp | Claude Opus 5.5 | | 14 | $0.36444 | 65.8 | 1.00 | |+| perf-app-quadratic | cli+skill | Claude Haiku 5.5 | | 11 | $0.00955 | 42.7 | 0.67 | |+| perf-app-quadratic | cli+skill | DeepSeek V4.1 Flash | | 19 | $0.01092 | 53.8 | 0.67 | |+| perf-app-quadratic | cli+skill | Claude Opus 5.5 | | 14 | $0.33868 | 68.2 | 0.67 | |+| perf-app-quadratic | cli-noskill | Claude Haiku 5.5 | grounding | 20 | $0.00812 | 53.8 | 0.67 | |+| perf-app-quadratic | cli-noskill | DeepSeek V4.1 Flash | | 20 | $0.00668 | 48.9 | 0.67 | |+| perf-app-quadratic | cli-noskill | Claude Opus 5.5 | grounding | 9 | $0.21189 | 46.9 | 0.33 | |+| perf-app-quadratic | mcp | Claude Haiku 5.5 | | 14 | $0.00983 | 44.2 | 0.67 | |+| perf-app-quadratic | mcp | DeepSeek V4.1 Flash | | 21 | $0.01315 | 72.8 | 0.67 | |+| perf-app-quadratic | mcp | Claude Opus 5.5 | | 12 | $0.29217 | 46.7 | 0.67 | |+| perf-app-sync-background | cli+skill | Claude Haiku 5.5 | grounding | 15 | $0.01297 | 58.5 | 0.67 | |+| perf-app-sync-background | cli+skill | DeepSeek V4.1 Flash | | 25 | $0.01422 | 87.8 | 0.67 | |+| perf-app-sync-background | cli+skill | Claude Opus 5.5 | | 12 | $0.29740 | 67.5 | 0.67 | |+| perf-app-sync-background | cli-noskill | Claude Haiku 5.5 | | 25 | $0.01242 | 84.5 | 1.00 | |+| perf-app-sync-background | cli-noskill | DeepSeek V4.1 Flash | | 29 | $0.01013 | 67.3 | 1.00 | |+| perf-app-sync-background | cli-noskill | Claude Opus 5.5 | | 14 | $0.29337 | 70.7 | 1.00 | |+| perf-app-sync-background | mcp | Claude Haiku 5.5 | grounding | 25 | $0.02171 | 102.5 | 0.67 | |+| perf-app-sync-background | mcp | DeepSeek V4.1 Flash | | 26 | $0.01507 | 79.8 | 1.00 | |+| perf-app-sync-background | mcp | Claude Opus 5.5 | | 15 | $0.34381 | 60.8 | 1.00 | |+| perf-cache-cold-start | cli+skill | Claude Haiku 5.5 | | 13 | $0.01103 | 68.0 | 0.00 | |+| perf-cache-cold-start | cli+skill | DeepSeek V4.1 Flash | grounding | 26 | $0.01270 | 76.8 | 0.67 | |+| perf-cache-cold-start | cli+skill | Claude Opus 5.5 | | 15 | $0.34653 | 82.4 | 1.00 | |+| perf-cache-cold-start | cli-noskill | Claude Haiku 5.5 | | 23 | $0.01016 | 68.3 | 0.33 | |+| perf-cache-cold-start | cli-noskill | DeepSeek V4.1 Flash | | 28 | $0.01200 | 66.1 | 1.00 | |+| perf-cache-cold-start | cli-noskill | Claude Opus 5.5 | grounding | 18 | $0.38983 | 95.9 | 1.00 | |+| perf-cache-cold-start | mcp | Claude Haiku 5.5 | | 19 | $0.01683 | 70.4 | 0.33 | |+| perf-cache-cold-start | mcp | DeepSeek V4.1 Flash | grounding | 25 | $0.02134 | 92.8 | 0.67 | |+| perf-cache-cold-start | mcp | Claude Opus 5.5 | | 18 | $0.51514 | 84.1 | 1.00 | |+| perf-cache-no-cache | cli+skill | Claude Haiku 5.5 | | 13 | $0.00822 | 55.7 | 0.67 | |+| perf-cache-no-cache | cli+skill | DeepSeek V4.1 Flash | answer | 27 | $0.01285 | 93.6 | 0.00 | |+| perf-cache-no-cache | cli+skill | Claude Opus 5.5 | | 13 | $0.23237 | 62.2 | 0.67 | |+| perf-cache-no-cache | cli-noskill | Claude Haiku 5.5 | | 13 | $0.00476 | 39.3 | 0.33 | |+| perf-cache-no-cache | cli-noskill | DeepSeek V4.1 Flash | | 21 | $0.01046 | 76.2 | 0.67 | |+| perf-cache-no-cache | cli-noskill | Claude Opus 5.5 | | 11 | $0.26961 | 69.2 | 0.67 | |+| perf-cache-no-cache | mcp | Claude Haiku 5.5 | | 16 | $0.00824 | 43.5 | 0.33 | |+| perf-cache-no-cache | mcp | DeepSeek V4.1 Flash | | 21 | $0.00976 | 62.7 | 0.67 | |+| perf-cache-no-cache | mcp | Claude Opus 5.5 | | 10 | $0.20292 | 45.7 | 0.67 | |+| perf-cache-stampede | cli+skill | Claude Haiku 5.5 | trajectory | 10 | $0.01056 | 51.9 | 1.00 | stampede-logs-query |+| perf-cache-stampede | cli+skill | DeepSeek V4.1 Flash | | 26 | $0.01385 | 77.1 | 1.00 | |+| perf-cache-stampede | cli+skill | Claude Opus 5.5 | trajectory | 11 | $0.30921 | 62.3 | 1.00 | stampede-logs-query |+| perf-cache-stampede | cli-noskill | Claude Haiku 5.5 | trajectory | 15 | $0.00709 | 59.4 | 1.00 | stampede-logs-query |+| perf-cache-stampede | cli-noskill | DeepSeek V4.1 Flash | | 30 | $0.01209 | 74.1 | 1.00 | |+| perf-cache-stampede | cli-noskill | Claude Opus 5.5 | trajectory | 16 | $0.31682 | 95.4 | 0.67 | stampede-logs-query |+| perf-cache-stampede | mcp | Claude Haiku 5.5 | | 10 | $0.01477 | 50.1 | 1.00 | |+| perf-cache-stampede | mcp | DeepSeek V4.1 Flash | trajectory | 20 | $0.02134 | 81.3 | 1.00 | stampede-logs-query |+| perf-cache-stampede | mcp | Claude Opus 5.5 | trajectory | 14 | $0.39892 | 69.2 | 1.00 | stampede-logs-query |+| perf-ctl-alert-handoff | cli+skill | Claude Haiku 5.5 | | 20 | $0.01480 | 88.5 | 0.67 | |+| perf-ctl-alert-handoff | cli+skill | DeepSeek V4.1 Flash | answer | 27 | $0.01544 | 95.0 | 1.00 | |+| perf-ctl-alert-handoff | cli+skill | Claude Opus 5.5 | | 15 | $0.38503 | 83.6 | 1.00 | |+| perf-ctl-alert-handoff | cli-noskill | Claude Haiku 5.5 | answer+trajectory | 29 | $0.01376 | 104.4 | 0.67 | |+| perf-ctl-alert-handoff | cli-noskill | DeepSeek V4.1 Flash | trajectory | 29 | $0.01151 | 75.4 | 1.00 | |+| perf-ctl-alert-handoff | cli-noskill | Claude Opus 5.5 | | 23 | $0.40755 | 128.1 | 1.00 | |+| perf-ctl-alert-handoff | mcp | Claude Haiku 5.5 | answer | 17 | $0.01272 | 65.2 | 0.33 | |+| perf-ctl-alert-handoff | mcp | DeepSeek V4.1 Flash | answer | 23 | $0.02259 | 118.9 | 1.00 | |+| perf-ctl-alert-handoff | mcp | Claude Opus 5.5 | | 12 | $0.37641 | 66.3 | 1.00 | |+| perf-ctl-healthy | cli+skill | Claude Haiku 5.5 | | 7 | $0.00605 | 23.5 | 0.67 | |+| perf-ctl-healthy | cli+skill | DeepSeek V4.1 Flash | answer+trajectory | 22 | $0.01003 | 61.7 | 0.33 | |+| perf-ctl-healthy | cli+skill | Claude Opus 5.5 | | 6 | $0.17850 | 41.7 | 1.00 | |+| perf-ctl-healthy | cli-noskill | Claude Haiku 5.5 | answer | 17 | $0.00694 | 55.5 | 0.00 | |+| perf-ctl-healthy | cli-noskill | DeepSeek V4.1 Flash | answer+trajectory | 28 | $0.00749 | 59.5 | 0.00 | |+| perf-ctl-healthy | cli-noskill | Claude Opus 5.5 | | 9 | $0.15601 | 49.0 | 0.67 | |+| perf-ctl-healthy | mcp | Claude Haiku 5.5 | answer | 9 | $0.01580 | 38.2 | 0.00 | |+| perf-ctl-healthy | mcp | DeepSeek V4.1 Flash | answer | 26 | $0.02464 | 72.5 | 0.33 | |+| perf-ctl-healthy | mcp | Claude Opus 5.5 | | 9 | $0.23188 | 43.5 | 0.67 | |+| perf-ctl-marked-slow-ok | cli+skill | Claude Haiku 5.5 | | 5 | $0.00474 | 27.7 | 1.00 | |+| perf-ctl-marked-slow-ok | cli+skill | DeepSeek V4.1 Flash | | 14 | $0.00935 | 46.4 | 1.00 | |+| perf-ctl-marked-slow-ok | cli+skill | Claude Opus 5.5 | | 6 | $0.19985 | 34.2 | 1.00 | |+| perf-ctl-marked-slow-ok | cli-noskill | Claude Haiku 5.5 | | 12 | $0.00436 | 37.2 | 1.00 | |+| perf-ctl-marked-slow-ok | cli-noskill | DeepSeek V4.1 Flash | | 14 | $0.00562 | 44.8 | 1.00 | |+| perf-ctl-marked-slow-ok | cli-noskill | Claude Opus 5.5 | | 8 | $0.13831 | 50.6 | 1.00 | |+| perf-ctl-marked-slow-ok | mcp | Claude Haiku 5.5 | answer | 10 | $0.00862 | 42.3 | 0.67 | |+| perf-ctl-marked-slow-ok | mcp | DeepSeek V4.1 Flash | | 13 | $0.00761 | 36.9 | 1.00 | |+| perf-ctl-marked-slow-ok | mcp | Claude Opus 5.5 | | 7 | $0.17919 | 33.0 | 1.00 | |+| perf-ctl-marked-slow-regressed | cli+skill | Claude Haiku 5.5 | | 8 | $0.00953 | 41.3 | 0.67 | |+| perf-ctl-marked-slow-regressed | cli+skill | DeepSeek V4.1 Flash | | 21 | $0.01142 | 54.7 | 0.67 | |+| perf-ctl-marked-slow-regressed | cli+skill | Claude Opus 5.5 | | 14 | $0.37151 | 72.3 | 1.00 | |+| perf-ctl-marked-slow-regressed | cli-noskill | Claude Haiku 5.5 | trajectory+grounding+report | 15 | $0.00608 | 42.0 | 0.33 | |+| perf-ctl-marked-slow-regressed | cli-noskill | DeepSeek V4.1 Flash | | 28 | $0.01211 | 63.8 | 1.00 | |+| perf-ctl-marked-slow-regressed | cli-noskill | Claude Opus 5.5 | | 14 | $0.22603 | 67.0 | 0.67 | |+| perf-ctl-marked-slow-regressed | mcp | Claude Haiku 5.5 | | 14 | $0.12242 | 52.3 | 0.67 | |+| perf-ctl-marked-slow-regressed | mcp | DeepSeek V4.1 Flash | | 18 | $0.02043 | 56.2 | 1.00 | |+| perf-ctl-marked-slow-regressed | mcp | Claude Opus 5.5 | | 11 | $0.59859 | 56.0 | 1.00 | |+| perf-ctl-metric-p95-trap | cli+skill | Claude Haiku 5.5 | | 15 | $0.01655 | 88.1 | 0.67 | |+| perf-ctl-metric-p95-trap | cli+skill | DeepSeek V4.1 Flash | | 29 | $0.01254 | 72.9 | 1.00 | |+| perf-ctl-metric-p95-trap | cli+skill | Claude Opus 5.5 | | 17 | $0.32848 | 82.1 | 1.00 | |+| perf-ctl-metric-p95-trap | cli-noskill | Claude Haiku 5.5 | report | 30 | $0.01382 | 83.3 | 0.33 | |+| perf-ctl-metric-p95-trap | cli-noskill | DeepSeek V4.1 Flash | report | 29 | $0.01265 | 96.7 | 0.33 | |+| perf-ctl-metric-p95-trap | cli-noskill | Claude Opus 5.5 | | 25 | $0.34122 | 130.9 | 0.67 | |+| perf-ctl-metric-p95-trap | mcp | Claude Haiku 5.5 | report | 21 | $0.01819 | 76.1 | 0.67 | |+| perf-ctl-metric-p95-trap | mcp | DeepSeek V4.1 Flash | | 22 | $0.01853 | 103.5 | 1.00 | |+| perf-ctl-metric-p95-trap | mcp | Claude Opus 5.5 | | 14 | $0.43885 | 66.9 | 1.00 | |+| perf-ctl-onset-bisect | cli+skill | Claude Haiku 5.5 | | 13 | $0.01600 | 78.5 | 1.00 | |+| perf-ctl-onset-bisect | cli+skill | DeepSeek V4.1 Flash | | 30 | $0.01412 | 95.0 | 1.00 | |+| perf-ctl-onset-bisect | cli+skill | Claude Opus 5.5 | | 14 | $0.36930 | 84.1 | 1.00 | |+| perf-ctl-onset-bisect | cli-noskill | Claude Haiku 5.5 | trajectory | 22 | $0.01094 | 60.8 | 1.00 | |+| perf-ctl-onset-bisect | cli-noskill | DeepSeek V4.1 Flash | | 26 | $0.01055 | 66.5 | 1.00 | |+| perf-ctl-onset-bisect | cli-noskill | Claude Opus 5.5 | | 17 | $0.26025 | 88.5 | 1.00 | |+| perf-ctl-onset-bisect | mcp | Claude Haiku 5.5 | | 17 | $0.01812 | 70.9 | 0.67 | |+| perf-ctl-onset-bisect | mcp | DeepSeek V4.1 Flash | | 26 | $0.01667 | 68.4 | 1.00 | |+| perf-ctl-onset-bisect | mcp | Claude Opus 5.5 | | 15 | $0.42144 | 76.7 | 1.00 | |+| perf-db-expensive-join | cli+skill | Claude Haiku 5.5 | | 6 | $0.00773 | 36.6 | 1.00 | |+| perf-db-expensive-join | cli+skill | DeepSeek V4.1 Flash | | 19 | $0.01063 | 58.8 | 1.00 | |+| perf-db-expensive-join | cli+skill | Claude Opus 5.5 | | 9 | $0.23874 | 54.4 | 1.00 | |+| perf-db-expensive-join | cli-noskill | Claude Haiku 5.5 | | 16 | $0.00635 | 44.1 | 1.00 | |+| perf-db-expensive-join | cli-noskill | DeepSeek V4.1 Flash | grounding | 24 | $0.01149 | 68.5 | 1.00 | |+| perf-db-expensive-join | cli-noskill | Claude Opus 5.5 | | 12 | $0.24249 | 66.0 | 1.00 | |+| perf-db-expensive-join | mcp | Claude Haiku 5.5 | | 11 | $0.00870 | 38.6 | 1.00 | |+| perf-db-expensive-join | mcp | DeepSeek V4.1 Flash | | 19 | $0.01299 | 69.2 | 1.00 | |+| perf-db-expensive-join | mcp | Claude Opus 5.5 | | 8 | $0.21662 | 48.8 | 1.00 | |+| perf-db-lock-contention | cli+skill | Claude Haiku 5.5 | | 12 | $0.01066 | 48.9 | 1.00 | |+| perf-db-lock-contention | cli+skill | DeepSeek V4.1 Flash | | 19 | $0.01192 | 61.2 | 1.00 | |+| perf-db-lock-contention | cli+skill | Claude Opus 5.5 | | 21 | $0.45861 | 114.0 | 1.00 | |+| perf-db-lock-contention | cli-noskill | Claude Haiku 5.5 | trajectory | 17 | $0.01063 | 67.9 | 1.00 | |+| perf-db-lock-contention | cli-noskill | DeepSeek V4.1 Flash | grounding | 32 | $0.01242 | 70.9 | 1.00 | |+| perf-db-lock-contention | cli-noskill | Claude Opus 5.5 | | 17 | $0.27080 | 85.6 | 1.00 | |+| perf-db-lock-contention | mcp | Claude Haiku 5.5 | | 19 | $0.01483 | 58.9 | 1.00 | |+| perf-db-lock-contention | mcp | DeepSeek V4.1 Flash | | 20 | $0.01312 | 68.1 | 1.00 | |+| perf-db-lock-contention | mcp | Claude Opus 5.5 | | 13 | $0.34631 | 71.6 | 1.00 | |+| perf-db-missing-index | cli+skill | Claude Haiku 5.5 | | 11 | $0.00886 | 46.0 | 1.00 | |+| perf-db-missing-index | cli+skill | DeepSeek V4.1 Flash | | 18 | $0.01009 | 53.7 | 1.00 | |+| perf-db-missing-index | cli+skill | Claude Opus 5.5 | | 8 | $0.22698 | 51.7 | 0.67 | |+| perf-db-missing-index | cli-noskill | Claude Haiku 5.5 | | 14 | $0.00823 | 57.3 | 0.33 | |+| perf-db-missing-index | cli-noskill | DeepSeek V4.1 Flash | | 23 | $0.01115 | 60.7 | 1.00 | |+| perf-db-missing-index | cli-noskill | Claude Opus 5.5 | | 8 | $0.15477 | 44.4 | 1.00 | |+| perf-db-missing-index | mcp | Claude Haiku 5.5 | | 13 | $0.00847 | 37.9 | 0.67 | |+| perf-db-missing-index | mcp | DeepSeek V4.1 Flash | | 18 | $0.01083 | 49.2 | 1.00 | |+| perf-db-missing-index | mcp | Claude Opus 5.5 | | 12 | $0.32705 | 53.4 | 1.00 | |+| perf-db-missing-query-cache | cli+skill | Claude Haiku 5.5 | | 9 | $0.00959 | 48.0 | 0.67 | |+| perf-db-missing-query-cache | cli+skill | DeepSeek V4.1 Flash | | 26 | $0.01117 | 72.2 | 1.00 | |+| perf-db-missing-query-cache | cli+skill | Claude Opus 5.5 | | 13 | $0.33635 | 91.9 | 1.00 | |+| perf-db-missing-query-cache | cli-noskill | Claude Haiku 5.5 | answer | 13 | $0.00591 | 39.4 | 1.00 | |+| perf-db-missing-query-cache | cli-noskill | DeepSeek V4.1 Flash | | 29 | $0.00871 | 57.6 | 1.00 | |+| perf-db-missing-query-cache | cli-noskill | Claude Opus 5.5 | | 10 | $0.13361 | 50.7 | 1.00 | |+| perf-db-missing-query-cache | mcp | Claude Haiku 5.5 | | 18 | $0.01058 | 43.6 | 1.00 | |+| perf-db-missing-query-cache | mcp | DeepSeek V4.1 Flash | | 24 | $0.00924 | 47.5 | 1.00 | |+| perf-db-missing-query-cache | mcp | Claude Opus 5.5 | | 16 | $0.29570 | 56.0 | 1.00 | |+| perf-db-n-plus-1 | cli+skill | Claude Haiku 5.5 | | 12 | $0.01351 | 50.3 | 0.67 | |+| perf-db-n-plus-1 | cli+skill | DeepSeek V4.1 Flash | | 21 | $0.01004 | 59.5 | 1.00 | |+| perf-db-n-plus-1 | cli+skill | Claude Opus 5.5 | | 8 | $0.24074 | 54.0 | 1.00 | |+| perf-db-n-plus-1 | cli-noskill | Claude Haiku 5.5 | | 10 | $0.00588 | 35.4 | 0.67 | |+| perf-db-n-plus-1 | cli-noskill | DeepSeek V4.1 Flash | | 26 | $0.00943 | 72.4 | 0.67 | |+| perf-db-n-plus-1 | cli-noskill | Claude Opus 5.5 | | 13 | $0.20943 | 61.7 | 0.67 | |+| perf-db-n-plus-1 | mcp | Claude Haiku 5.5 | grounding | 14 | $0.06958 | 50.3 | 0.67 | |+| perf-db-n-plus-1 | mcp | DeepSeek V4.1 Flash | | 15 | $0.01259 | 45.7 | 1.00 | |+| perf-db-n-plus-1 | mcp | Claude Opus 5.5 | | 10 | $0.38496 | 47.5 | 0.67 | |+| perf-db-over-fetching | cli+skill | Claude Haiku 5.5 | | 9 | $0.00857 | 40.3 | 0.67 | |+| perf-db-over-fetching | cli+skill | DeepSeek V4.1 Flash | | 26 | $0.01092 | 64.6 | 1.00 | |+| perf-db-over-fetching | cli+skill | Claude Opus 5.5 | | 15 | $0.31148 | 79.2 | 0.67 | |+| perf-db-over-fetching | cli-noskill | Claude Haiku 5.5 | | 17 | $0.00755 | 52.0 | 0.67 | |+| perf-db-over-fetching | cli-noskill | DeepSeek V4.1 Flash | grounding | 30 | $0.00940 | 51.3 | 1.00 | |+| perf-db-over-fetching | cli-noskill | Claude Opus 5.5 | | 15 | $0.21028 | 80.7 | 1.00 | |+| perf-db-over-fetching | mcp | Claude Haiku 5.5 | grounding | 12 | $0.01724 | 48.1 | 0.67 | |+| perf-db-over-fetching | mcp | DeepSeek V4.1 Flash | | 24 | $0.01334 | 63.2 | 1.00 | |+| perf-db-over-fetching | mcp | Claude Opus 5.5 | | 15 | $0.43795 | 67.7 | 1.00 | |+| perf-db-pool-exhaustion | cli+skill | Claude Haiku 5.5 | | 9 | $0.01103 | 58.2 | 1.00 | |+| perf-db-pool-exhaustion | cli+skill | DeepSeek V4.1 Flash | | 31 | $0.01903 | 111.6 | 1.00 | |+| perf-db-pool-exhaustion | cli+skill | Claude Opus 5.5 | | 11 | $0.29402 | 64.0 | 1.00 | |+| perf-db-pool-exhaustion | cli-noskill | Claude Haiku 5.5 | | 19 | $0.00932 | 60.4 | 1.00 | |+| perf-db-pool-exhaustion | cli-noskill | DeepSeek V4.1 Flash | | 26 | $0.01605 | 92.8 | 1.00 | |+| perf-db-pool-exhaustion | cli-noskill | Claude Opus 5.5 | | 19 | $0.34420 | 102.3 | 1.00 | |+| perf-db-pool-exhaustion | mcp | Claude Haiku 5.5 | | 14 | $0.01013 | 51.6 | 1.00 | |+| perf-db-pool-exhaustion | mcp | DeepSeek V4.1 Flash | | 26 | $0.01914 | 94.8 | 1.00 | |+| perf-db-pool-exhaustion | mcp | Claude Opus 5.5 | | 12 | $0.31406 | 59.5 | 1.00 | |+| perf-db-unbounded-result | cli+skill | Claude Haiku 5.5 | | 13 | $0.00930 | 51.5 | 0.67 | |+| perf-db-unbounded-result | cli+skill | DeepSeek V4.1 Flash | | 15 | $0.00705 | 39.5 | 1.00 | |+| perf-db-unbounded-result | cli+skill | Claude Opus 5.5 | | 16 | $0.33908 | 80.4 | 1.00 | |+| perf-db-unbounded-result | cli-noskill | Claude Haiku 5.5 | | 13 | $0.00547 | 38.8 | 1.00 | |+| perf-db-unbounded-result | cli-noskill | DeepSeek V4.1 Flash | | 23 | $0.00746 | 55.2 | 1.00 | |+| perf-db-unbounded-result | cli-noskill | Claude Opus 5.5 | trajectory | 10 | $0.15222 | 49.6 | 1.00 | |+| perf-db-unbounded-result | mcp | Claude Haiku 5.5 | | 12 | $0.00892 | 40.1 | 1.00 | |+| perf-db-unbounded-result | mcp | DeepSeek V4.1 Flash | | 15 | $0.00904 | 42.5 | 1.00 | |+| perf-db-unbounded-result | mcp | Claude Opus 5.5 | | 12 | $0.25024 | 45.4 | 1.00 | |+| perf-infra-cpu-saturation | cli+skill | Claude Haiku 5.5 | | 22 | $0.01979 | 101.2 | 1.00 | |+| perf-infra-cpu-saturation | cli+skill | DeepSeek V4.1 Flash | | 33 | $0.01422 | 88.4 | 1.00 | |+| perf-infra-cpu-saturation | cli+skill | Claude Opus 5.5 | | 16 | $0.41930 | 113.9 | 1.00 | |+| perf-infra-cpu-saturation | cli-noskill | Claude Haiku 5.5 | answer+trajectory+grounding | 30 | $0.01096 | 74.4 | 0.00 | |+| perf-infra-cpu-saturation | cli-noskill | DeepSeek V4.1 Flash | answer | 32 | $0.01634 | 110.6 | 0.33 | |+| perf-infra-cpu-saturation | cli-noskill | Claude Opus 5.5 | | 28 | $0.41397 | 138.8 | 1.00 | |+| perf-infra-cpu-saturation | mcp | Claude Haiku 5.5 | | 25 | $0.02384 | 102.5 | 1.00 | |+| perf-infra-cpu-saturation | mcp | DeepSeek V4.1 Flash | | 30 | $0.02079 | 113.7 | 0.67 | |+| perf-infra-cpu-saturation | mcp | Claude Opus 5.5 | | 26 | $0.62904 | 124.6 | 1.00 | |+| perf-infra-disk-io | cli+skill | Claude Haiku 5.5 | answer+grounding | 22 | $0.01562 | 69.8 | 0.67 | iowait-recipe |+| perf-infra-disk-io | cli+skill | DeepSeek V4.1 Flash | answer+grounding | 33 | $0.01678 | 103.8 | 0.67 | iowait-recipe |+| perf-infra-disk-io | cli+skill | Claude Opus 5.5 | | 20 | $0.47247 | 103.9 | 1.00 | |+| perf-infra-disk-io | cli-noskill | Claude Haiku 5.5 | answer+grounding | 32 | $0.01351 | 91.0 | 0.67 | |+| perf-infra-disk-io | cli-noskill | DeepSeek V4.1 Flash | | 34 | $0.01011 | 68.8 | 0.67 | |+| perf-infra-disk-io | cli-noskill | Claude Opus 5.5 | | 28 | $0.40615 | 128.8 | 1.00 | |+| perf-infra-disk-io | mcp | Claude Haiku 5.5 | answer+grounding | 18 | $0.01683 | 78.4 | 0.67 | iowait-recipe |+| perf-infra-disk-io | mcp | DeepSeek V4.1 Flash | | 23 | $0.01441 | 73.0 | 1.00 | |+| perf-infra-disk-io | mcp | Claude Opus 5.5 | | 19 | $0.46923 | 82.6 | 1.00 | |+| perf-infra-memory-gc | cli+skill | Claude Haiku 5.5 | answer | 19 | $0.01449 | 65.0 | 0.67 | memory-gc-label |+| perf-infra-memory-gc | cli+skill | DeepSeek V4.1 Flash | answer | 28 | $0.02205 | 112.4 | 1.00 | memory-gc-label |+| perf-infra-memory-gc | cli+skill | Claude Opus 5.5 | answer | 15 | $0.35994 | 92.7 | 1.00 | memory-gc-label |+| perf-infra-memory-gc | cli-noskill | Claude Haiku 5.5 | | 23 | $0.01154 | 68.0 | 0.67 | |+| perf-infra-memory-gc | cli-noskill | DeepSeek V4.1 Flash | | 31 | $0.01592 | 101.6 | 1.00 | |+| perf-infra-memory-gc | cli-noskill | Claude Opus 5.5 | | 14 | $0.30630 | 89.9 | 1.00 | |+| perf-infra-memory-gc | mcp | Claude Haiku 5.5 | answer+trajectory | 21 | $0.01502 | 64.8 | 0.67 | |+| perf-infra-memory-gc | mcp | DeepSeek V4.1 Flash | | 29 | $0.02280 | 130.2 | 1.00 | |+| perf-infra-memory-gc | mcp | Claude Opus 5.5 | answer | 16 | $0.55036 | 87.5 | 1.00 | memory-gc-label |+| perf-infra-noisy-neighbor | cli+skill | Claude Haiku 5.5 | | 13 | $0.01252 | 58.8 | 1.00 | |+| perf-infra-noisy-neighbor | cli+skill | DeepSeek V4.1 Flash | | 29 | $0.01614 | 98.6 | 0.33 | |+| perf-infra-noisy-neighbor | cli+skill | Claude Opus 5.5 | | 17 | $0.38651 | 82.0 | 1.00 | |+| perf-infra-noisy-neighbor | cli-noskill | Claude Haiku 5.5 | answer+trajectory+grounding | 28 | $0.01537 | 98.6 | 0.00 | |+| perf-infra-noisy-neighbor | cli-noskill | DeepSeek V4.1 Flash | answer | 35 | $0.01600 | 105.6 | 0.67 | |+| perf-infra-noisy-neighbor | cli-noskill | Claude Opus 5.5 | | 23 | $0.29690 | 107.0 | 0.67 | |+| perf-infra-noisy-neighbor | mcp | Claude Haiku 5.5 | | 23 | $0.01523 | 71.8 | 0.00 | |+| perf-infra-noisy-neighbor | mcp | DeepSeek V4.1 Flash | | 26 | $0.01534 | 72.6 | 0.67 | |+| perf-infra-noisy-neighbor | mcp | Claude Opus 5.5 | trajectory | 13 | $0.37145 | 56.1 | 0.67 | |+| perf-net-chatty-fanout | cli+skill | Claude Haiku 5.5 | grounding | 12 | $0.01366 | 59.5 | 0.33 | chatty-fanout-grounding |+| perf-net-chatty-fanout | cli+skill | DeepSeek V4.1 Flash | grounding | 26 | $0.01536 | 93.0 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | cli+skill | Claude Opus 5.5 | | 13 | $0.36403 | 81.0 | 1.00 | |+| perf-net-chatty-fanout | cli-noskill | Claude Haiku 5.5 | grounding | 12 | $0.00711 | 53.7 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | cli-noskill | DeepSeek V4.1 Flash | grounding | 24 | $0.01154 | 63.9 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | cli-noskill | Claude Opus 5.5 | grounding | 16 | $0.22570 | 83.0 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | mcp | Claude Haiku 5.5 | grounding | 15 | $0.06257 | 65.1 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | mcp | DeepSeek V4.1 Flash | grounding | 18 | $0.01899 | 68.8 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | mcp | Claude Opus 5.5 | grounding | 10 | $0.62898 | 57.2 | 1.00 | chatty-fanout-grounding |+| perf-net-large-payload | cli+skill | Claude Haiku 5.5 | | 13 | $0.00900 | 50.7 | 0.67 | |+| perf-net-large-payload | cli+skill | DeepSeek V4.1 Flash | | 22 | $0.01193 | 68.5 | 1.00 | |+| perf-net-large-payload | cli+skill | Claude Opus 5.5 | | 14 | $0.32918 | 83.4 | 1.00 | |+| perf-net-large-payload | cli-noskill | Claude Haiku 5.5 | | 21 | $0.00815 | 61.8 | 0.67 | |+| perf-net-large-payload | cli-noskill | DeepSeek V4.1 Flash | | 29 | $0.01136 | 67.5 | 1.00 | |+| perf-net-large-payload | cli-noskill | Claude Opus 5.5 | | 11 | $0.20753 | 74.4 | 1.00 | |+| perf-net-large-payload | mcp | Claude Haiku 5.5 | | 15 | $0.01338 | 72.5 | 1.00 | |+| perf-net-large-payload | mcp | DeepSeek V4.1 Flash | | 19 | $0.01131 | 61.1 | 1.00 | |+| perf-net-large-payload | mcp | Claude Opus 5.5 | | 10 | $0.26390 | 49.5 | 1.00 | |+| perf-net-no-conn-reuse | cli+skill | Claude Haiku 5.5 | | 12 | $0.01066 | 51.3 | 0.67 | |+| perf-net-no-conn-reuse | cli+skill | DeepSeek V4.1 Flash | | 17 | $0.00958 | 46.9 | 1.00 | |+| perf-net-no-conn-reuse | cli+skill | Claude Opus 5.5 | | 13 | $0.27660 | 69.5 | 1.00 | |+| perf-net-no-conn-reuse | cli-noskill | Claude Haiku 5.5 | | 14 | $0.00764 | 46.9 | 0.67 | |+| perf-net-no-conn-reuse | cli-noskill | DeepSeek V4.1 Flash | | 22 | $0.00776 | 53.9 | 1.00 | |+| perf-net-no-conn-reuse | cli-noskill | Claude Opus 5.5 | | 12 | $0.21503 | 73.6 | 1.00 | |+| perf-net-no-conn-reuse | mcp | Claude Haiku 5.5 | | 12 | $0.01126 | 47.8 | 0.67 | |+| perf-net-no-conn-reuse | mcp | DeepSeek V4.1 Flash | | 19 | $0.01009 | 46.7 | 1.00 | |+| perf-net-no-conn-reuse | mcp | Claude Opus 5.5 | | 9 | $0.27702 | 52.8 | 0.67 | |+| perf-net-retries | cli+skill | Claude Haiku 5.5 | trajectory+grounding | 8 | $0.00945 | 46.6 | 0.67 | |+| perf-net-retries | cli+skill | DeepSeek V4.1 Flash | | 20 | $0.01255 | 78.6 | 1.00 | |+| perf-net-retries | cli+skill | Claude Opus 5.5 | | 11 | $0.27177 | 68.3 | 1.00 | |+| perf-net-retries | cli-noskill | Claude Haiku 5.5 | | 15 | $0.00577 | 42.3 | 1.00 | |+| perf-net-retries | cli-noskill | DeepSeek V4.1 Flash | | 28 | $0.00817 | 60.5 | 1.00 | |+| perf-net-retries | cli-noskill | Claude Opus 5.5 | | 20 | $0.27323 | 83.0 | 1.00 | |+| perf-net-retries | mcp | Claude Haiku 5.5 | | 9 | $0.00924 | 33.0 | 1.00 | |+| perf-net-retries | mcp | DeepSeek V4.1 Flash | | 17 | $0.01507 | 57.5 | 1.00 | |+| perf-net-retries | mcp | Claude Opus 5.5 | | 9 | $0.28747 | 48.6 | 1.00 | |+| perf-net-sequential | cli+skill | Claude Haiku 5.5 | | 13 | $0.01422 | 54.2 | 1.00 | |+| perf-net-sequential | cli+skill | DeepSeek V4.1 Flash | | 19 | $0.01273 | 61.4 | 1.00 | |+| perf-net-sequential | cli+skill | Claude Opus 5.5 | | 13 | $0.32919 | 82.7 | 0.67 | |+| perf-net-sequential | cli-noskill | Claude Haiku 5.5 | | 15 | $0.00602 | 45.9 | 0.67 | |+| perf-net-sequential | cli-noskill | DeepSeek V4.1 Flash | | 25 | $0.00854 | 57.5 | 1.00 | |+| perf-net-sequential | cli-noskill | Claude Opus 5.5 | | 12 | $0.16696 | 58.7 | 0.67 | |+| perf-net-sequential | mcp | Claude Haiku 5.5 | | 9 | $0.00795 | 36.0 | 0.67 | |+| perf-net-sequential | mcp | DeepSeek V4.1 Flash | | 15 | $0.01072 | 50.5 | 1.00 | |+| perf-net-sequential | mcp | Claude Opus 5.5 | | 10 | $0.33244 | 47.7 | 1.00 | |+| perf-net-slow-third-party | cli+skill | Claude Haiku 5.5 | | 12 | $0.00942 | 52.9 | 1.00 | |+| perf-net-slow-third-party | cli+skill | DeepSeek V4.1 Flash | | 28 | $0.01552 | 92.9 | 1.00 | |+| perf-net-slow-third-party | cli+skill | Claude Opus 5.5 | | 17 | $0.35468 | 89.5 | 0.67 | |+| perf-net-slow-third-party | cli-noskill | Claude Haiku 5.5 | | 21 | $0.00953 | 57.8 | 0.67 | |+| perf-net-slow-third-party | cli-noskill | DeepSeek V4.1 Flash | | 27 | $0.01277 | 78.7 | 1.00 | |+| perf-net-slow-third-party | cli-noskill | Claude Opus 5.5 | | 23 | $0.33016 | 103.9 | 1.00 | |+| perf-net-slow-third-party | mcp | Claude Haiku 5.5 | | 15 | $0.01231 | 46.1 | 1.00 | |+| perf-net-slow-third-party | mcp | DeepSeek V4.1 Flash | | 23 | $0.01664 | 77.8 | 1.00 | |+| perf-net-slow-third-party | mcp | Claude Opus 5.5 | | 14 | $0.46916 | 64.2 | 1.00 | |+| perf-xs-android-timeouts | cli+skill | Claude Haiku 5.5 | | 21 | $0.01327 | 75.8 | 1.00 | |+| perf-xs-android-timeouts | cli+skill | DeepSeek V4.1 Flash | | 28 | $0.01149 | 68.8 | 1.00 | |+| perf-xs-android-timeouts | cli+skill | Claude Opus 5.5 | | 17 | $0.45062 | 108.6 | 1.00 | |+| perf-xs-android-timeouts | cli-noskill | Claude Haiku 5.5 | grounding | 29 | $0.02029 | 102.0 | 0.67 | |+| perf-xs-android-timeouts | cli-noskill | DeepSeek V4.1 Flash | grounding | 31 | $0.00985 | 62.3 | 0.67 | |+| perf-xs-android-timeouts | cli-noskill | Claude Opus 5.5 | grounding | 17 | $0.46912 | 98.6 | 1.00 | |+| perf-xs-android-timeouts | mcp | Claude Haiku 5.5 | | 23 | $0.01540 | 63.6 | 1.00 | |+| perf-xs-android-timeouts | mcp | DeepSeek V4.1 Flash | | 25 | $0.01500 | 61.0 | 1.00 | |+| perf-xs-android-timeouts | mcp | Claude Opus 5.5 | | 20 | $0.63037 | 90.9 | 1.00 | |+| perf-xs-frontend-timeouts | cli+skill | Claude Haiku 5.5 | | 18 | $0.01350 | 62.8 | 1.00 | |+| perf-xs-frontend-timeouts | cli+skill | DeepSeek V4.1 Flash | | 29 | $0.01220 | 73.9 | 1.00 | |+| perf-xs-frontend-timeouts | cli+skill | Claude Opus 5.5 | | 22 | $0.50305 | 115.5 | 1.00 | |+| perf-xs-frontend-timeouts | cli-noskill | Claude Haiku 5.5 | | 30 | $0.01097 | 71.7 | 1.00 | |+| perf-xs-frontend-timeouts | cli-noskill | DeepSeek V4.1 Flash | | 31 | $0.01057 | 67.6 | 1.00 | |+| perf-xs-frontend-timeouts | cli-noskill | Claude Opus 5.5 | | 18 | $0.35105 | 87.4 | 1.00 | |+| perf-xs-frontend-timeouts | mcp | Claude Haiku 5.5 | | 23 | $0.01446 | 60.0 | 1.00 | |+| perf-xs-frontend-timeouts | mcp | DeepSeek V4.1 Flash | | 28 | $0.01310 | 64.7 | 1.00 | |+| perf-xs-frontend-timeouts | mcp | Claude Opus 5.5 | | 20 | $0.42544 | 81.7 | 1.00 | |+| perf-xs-ios-watchdog | cli+skill | Claude Haiku 5.5 | answer+trajectory+grounding | 11 | $0.00812 | 48.0 | 0.67 | |+| perf-xs-ios-watchdog | cli+skill | DeepSeek V4.1 Flash | | 32 | $0.01345 | 89.5 | 0.67 | |+| perf-xs-ios-watchdog | cli+skill | Claude Opus 5.5 | | 28 | $0.68782 | 145.2 | 1.00 | |+| perf-xs-ios-watchdog | cli-noskill | Claude Haiku 5.5 | answer | 16 | $0.00699 | 50.7 | 0.33 | |+| perf-xs-ios-watchdog | cli-noskill | DeepSeek V4.1 Flash | | 33 | $0.01581 | 103.3 | 1.00 | |+| perf-xs-ios-watchdog | cli-noskill | Claude Opus 5.5 | | 27 | $0.69231 | 163.6 | 1.00 | |+| perf-xs-ios-watchdog | mcp | Claude Haiku 5.5 | answer | 25 | $0.01532 | 72.2 | 0.67 | |+| perf-xs-ios-watchdog | mcp | DeepSeek V4.1 Flash | | 29 | $0.02300 | 123.6 | 1.00 | |+| perf-xs-ios-watchdog | mcp | Claude Opus 5.5 | | 28 | $0.57274 | 112.1 | 1.00 | |+| write-alert-disable | cli-write+skill | Claude Haiku 5.5 | | 5 | $0.00417 | 16.2 | 0.67 | |+| write-alert-disable | cli-write+skill | DeepSeek V4.1 Flash | | 7 | $0.00190 | 14.5 | 1.00 | |+| write-alert-disable | cli-write+skill | Claude Opus 5.5 | | 4 | $0.07555 | 19.6 | 1.00 | |+| write-alert-disable | mcp-write | Claude Haiku 5.5 | | 4 | $0.00354 | 14.8 | 1.00 | |+| write-alert-disable | mcp-write | DeepSeek V4.1 Flash | | 4 | $0.00301 | 14.3 | 1.00 | |+| write-alert-disable | mcp-write | Claude Opus 5.5 | | 3 | $0.11323 | 20.0 | 1.00 | |+| write-alert-snooze | cli-write+skill | Claude Haiku 5.5 | | 4 | $0.00202 | 18.8 | 1.00 | |+| write-alert-snooze | cli-write+skill | DeepSeek V4.1 Flash | | 7 | $0.00226 | 21.8 | 1.00 | |+| write-alert-snooze | cli-write+skill | Claude Opus 5.5 | | 4 | $0.08200 | 26.1 | 1.00 | |+| write-alert-snooze | mcp-write | Claude Haiku 5.5 | | 4 | $0.00340 | 17.8 | 1.00 | |+| write-alert-snooze | mcp-write | DeepSeek V4.1 Flash | | 4 | $0.00274 | 17.9 | 1.00 | |+| write-alert-snooze | mcp-write | Claude Opus 5.5 | | 4 | $0.12348 | 20.5 | 1.00 | |+| write-archive-ambiguous | cli-write+skill | Claude Haiku 5.5 | | 6 | $0.00461 | 23.5 | 1.00 | |+| write-archive-ambiguous | cli-write+skill | DeepSeek V4.1 Flash | | 10 | $0.00336 | 25.7 | 1.00 | |+| write-archive-ambiguous | cli-write+skill | Claude Opus 5.5 | | 6 | $0.10000 | 25.2 | 1.00 | |+| write-archive-ambiguous | mcp-write | Claude Haiku 5.5 | | 5 | $0.00423 | 23.4 | 1.00 | |+| write-archive-ambiguous | mcp-write | DeepSeek V4.1 Flash | | 9 | $0.00530 | 28.4 | 1.00 | |+| write-archive-ambiguous | mcp-write | Claude Opus 5.5 | | 5 | $0.14871 | 26.0 | 1.00 | |+| write-investigate-only | cli-write+skill | Claude Haiku 5.5 | | 9 | $0.00515 | 28.7 | 1.00 | |+| write-investigate-only | cli-write+skill | DeepSeek V4.1 Flash | | 15 | $0.00609 | 41.7 | 1.00 | |+| write-investigate-only | cli-write+skill | Claude Opus 5.5 | | 9 | $0.18263 | 48.7 | 1.00 | |+| write-investigate-only | mcp-write | Claude Haiku 5.5 | | 7 | $0.00467 | 23.5 | 1.00 | |+| write-investigate-only | mcp-write | DeepSeek V4.1 Flash | | 14 | $0.00621 | 28.8 | 1.00 | |+| write-investigate-only | mcp-write | Claude Opus 5.5 | | 12 | $0.27755 | 48.3 | 1.00 | |+| write-mark-slow | cli-write+skill | Claude Haiku 5.5 | | 5 | $0.00417 | 16.0 | 1.00 | |+| write-mark-slow | cli-write+skill | DeepSeek V4.1 Flash | | 11 | $0.00548 | 30.0 | 0.67 | |+| write-mark-slow | cli-write+skill | Claude Opus 5.5 | | 4 | $0.07588 | 23.8 | 1.00 | |+| write-mark-slow | mcp-write | Claude Haiku 5.5 | | 7 | $0.00421 | 22.9 | 1.00 | |+| write-mark-slow | mcp-write | DeepSeek V4.1 Flash | | 7 | $0.00372 | 23.2 | 0.67 | |+| write-mark-slow | mcp-write | Claude Opus 5.5 | | 6 | $0.13284 | 26.1 | 1.00 | |+| write-monitor-confirm-fix | cli-write+skill | Claude Haiku 5.5 | trajectory+state | 7 | $0.00475 | 40.9 | 0.33 | |+| write-monitor-confirm-fix | cli-write+skill | DeepSeek V4.1 Flash | trajectory+state | 13 | $0.00448 | 48.9 | 0.33 | |+| write-monitor-confirm-fix | cli-write+skill | Claude Opus 5.5 | | 5 | $0.08459 | 44.5 | 1.00 | |+| write-monitor-confirm-fix | mcp-write | Claude Haiku 5.5 | trajectory+state | 3 | $0.00300 | 34.3 | 0.33 | |+| write-monitor-confirm-fix | mcp-write | DeepSeek V4.1 Flash | | 10 | $0.00409 | 55.5 | 1.00 | |+| write-monitor-confirm-fix | mcp-write | Claude Opus 5.5 | | 6 | $0.12694 | 43.8 | 1.00 | |+| write-optin-off | cli-write+skill | Claude Haiku 5.5 | | 4 | $0.00192 | 16.6 | 1.00 | |+| write-optin-off | cli-write+skill | DeepSeek V4.1 Flash | | 4 | $0.00181 | 17.9 | 1.00 | |+| write-optin-off | cli-write+skill | Claude Opus 5.5 | | 4 | $0.07034 | 21.8 | 1.00 | |+| write-optin-off | mcp-write | Claude Haiku 5.5 | | 4 | $0.00340 | 17.8 | 1.00 | |+| write-optin-off | mcp-write | DeepSeek V4.1 Flash | | 4 | $0.00276 | 16.2 | 1.00 | |+| write-optin-off | mcp-write | Claude Opus 5.5 | | 3 | $0.10478 | 20.8 | 1.00 | |+| write-page-ack | cli-write+skill | Claude Haiku 5.5 | | 5 | $0.00239 | 19.8 | 1.00 | |+| write-page-ack | cli-write+skill | DeepSeek V4.1 Flash | | 5 | $0.00235 | 22.5 | 1.00 | |+| write-page-ack | cli-write+skill | Claude Opus 5.5 | | 5 | $0.08645 | 28.0 | 1.00 | |+| write-page-ack | mcp-write | Claude Haiku 5.5 | | 4 | $0.00345 | 17.5 | 1.00 | |+| write-page-ack | mcp-write | DeepSeek V4.1 Flash | | 5 | $0.00305 | 17.4 | 1.00 | |+| write-page-ack | mcp-write | Claude Opus 5.5 | | 4 | $0.12696 | 21.2 | 1.00 | |+| write-page-resolve-archive | cli-write+skill | Claude Haiku 5.5 | trajectory | 12 | $0.00838 | 196.1 | 0.33 | |+| write-page-resolve-archive | cli-write+skill | DeepSeek V4.1 Flash | | 16 | $0.00656 | 192.8 | 1.00 | |+| write-page-resolve-archive | cli-write+skill | Claude Opus 5.5 | | 14 | $0.23162 | 216.2 | 1.00 | |+| write-page-resolve-archive | mcp-write | Claude Haiku 5.5 | | 13 | $0.01115 | 200.3 | 1.00 | |+| write-page-resolve-archive | mcp-write | DeepSeek V4.1 Flash | | 14 | $0.00735 | 185.7 | 0.67 | |+| write-page-resolve-archive | mcp-write | Claude Opus 5.5 | | 11 | $0.29306 | 191.3 | 0.67 | |+| write-postmortem-conflict | cli-write+skill | Claude Haiku 5.5 | | 10 | $0.00380 | 25.1 | 1.00 | |+| write-postmortem-conflict | cli-write+skill | DeepSeek V4.1 Flash | | 15 | $0.00365 | 32.9 | 1.00 | |+| write-postmortem-conflict | cli-write+skill | Claude Opus 5.5 | | 12 | $0.14685 | 55.0 | 1.00 | |+| write-postmortem-conflict | mcp-write | Claude Haiku 5.5 | | 6 | $0.00446 | 19.9 | 1.00 | |+| write-postmortem-conflict | mcp-write | DeepSeek V4.1 Flash | | 6 | $0.00341 | 16.7 | 1.00 | |+| write-postmortem-conflict | mcp-write | Claude Opus 5.5 | | 5 | $0.13673 | 25.8 | 1.00 | |+| write-postmortem-create | cli-write+skill | Claude Haiku 5.5 | trajectory+state | 10 | $0.01048 | 209.1 | 0.00 | |+| write-postmortem-create | cli-write+skill | DeepSeek V4.1 Flash | | 19 | $0.00812 | 191.0 | 0.33 | |+| write-postmortem-create | cli-write+skill | Claude Opus 5.5 | | 15 | $0.32755 | 236.7 | 0.33 | |+| write-postmortem-create | mcp-write | Claude Haiku 5.5 | | 12 | $0.00992 | 188.3 | | |+| write-postmortem-create | mcp-write | DeepSeek V4.1 Flash | | 19 | $0.02205 | 217.5 | 0.00 | |+| write-postmortem-create | mcp-write | Claude Opus 5.5 | | 17 | $0.72760 | 218.0 | 1.00 | |+| write-readonly-member | cli-write+skill | Claude Haiku 5.5 | | 6 | $0.00451 | 17.8 | 1.00 | |+| write-readonly-member | cli-write+skill | DeepSeek V4.1 Flash | | 7 | $0.00275 | 20.3 | 1.00 | |+| write-readonly-member | cli-write+skill | Claude Opus 5.5 | | 7 | $0.10724 | 28.4 | 1.00 | |+| write-readonly-member | mcp-write | Claude Haiku 5.5 | | 6 | $0.00370 | 17.3 | 1.00 | |+| write-readonly-member | mcp-write | DeepSeek V4.1 Flash | | 8 | $0.00364 | 19.8 | 1.00 | |+| write-readonly-member | mcp-write | Claude Opus 5.5 | | 6 | $0.20475 | 31.1 | 1.00 | |+| write-readonly-token | cli-write+skill | Claude Haiku 5.5 | | 7 | $0.00470 | 30.7 | 0.67 | |+| write-readonly-token | cli-write+skill | DeepSeek V4.1 Flash | | 9 | $0.00294 | 20.5 | 1.00 | |+| write-readonly-token | cli-write+skill | Claude Opus 5.5 | | 6 | $0.09943 | 28.3 | 1.00 | |+| write-readonly-token | mcp-write | Claude Haiku 5.5 | | 2 | $0.00221 | 16.4 | 0.67 | |+| write-readonly-token | mcp-write | DeepSeek V4.1 Flash | | 10 | $0.00314 | 21.0 | 1.00 | |+| write-readonly-token | mcp-write | Claude Opus 5.5 | | 5 | $0.14987 | 35.7 | 0.67 | |diff --git a/website/lib/benchmark-data.ts b/website/lib/benchmark-data.tsnew file mode 100644index 00000000..688067e4--- /dev/null+++ b/website/lib/benchmark-data.ts@@ -0,0 +1,18 @@+import fs from "node:fs";+import path from "node:path";+import { parseBenchmark } from "@/lib/benchmarks";++const DIR = path.join(process.cwd(), "content", "benchmarks");+const LATEST = fs+ .readdirSync(DIR)+ .filter((f) => /^\d{4}-\d{2}-\d{2}\.md$/.test(f))+ .sort()+ .at(-1);++if (!LATEST) {+ throw new Error(`no YYYY-MM-DD.md benchmark file in ${DIR}`);+}++export const BENCHMARK = parseBenchmark(+ fs.readFileSync(path.join(DIR, LATEST), "utf8"),+);diff --git a/website/lib/benchmark-format.ts b/website/lib/benchmark-format.tsnew file mode 100644index 00000000..20a15e57--- /dev/null+++ b/website/lib/benchmark-format.ts@@ -0,0 +1,34 @@+import type { ModelKey, RunKind } from "@/lib/benchmarks";++export const MODEL_COLOR: Record<ModelKey, string> = {+ haiku: "var(--a3)",+ "deepseek-flash": "var(--a4)",+ opus: "var(--a1)",+};++export const KIND_LABEL: Record<RunKind, string> = {+ pass: "Pass",+ trajectory: "Right answer, missed a required step",+ wrong: "Wrong answer or ungrounded evidence",+ budget: "Ran out of budget",+};++export function rate(pass: number, runs: number) {+ return Math.round((100 * pass) / runs);+}++export function formatPp(pp: number) {+ if (pp === 0) return "±0pp";+ return `${pp > 0 ? "+" : "−"}${Math.abs(pp)}pp`;+}++export function formatUsd(usd: number) {+ if (usd >= 1) return `$${usd.toFixed(2)}`;+ if (usd >= 0.01) return `$${usd.toFixed(3)}`;+ return `$${usd.toFixed(4)}`;+}++export function numberWord(n: number) {+ const words = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten"];+ return words[n] ?? String(n);+}diff --git a/website/lib/benchmarks.ts b/website/lib/benchmarks.tsnew file mode 100644index 00000000..e545b731--- /dev/null+++ b/website/lib/benchmarks.ts@@ -0,0 +1,421 @@+import matter from "gray-matter";++export type ModelKey = "haiku" | "deepseek-flash" | "opus";++export type Totals = {+ runs: number;+ pass: number;+ rightAnswerEvidence: number;+ notGraded: number;+ costUsd: number;+ costPerRunUsd: number;+ judgeCostUsd: number;+ avgToolCalls: number;+ avgWallS: number;+ cacheHitRate: number;+ avgJudge: number | null;+ judged: number;+};++export type Model = {+ key: ModelKey;+ label: string;+ short: string;+ vendor: string;+ weights: "open" | "proprietary";+ id: string;+ provider: string;+ providers: { name: string; runs: number }[];+ reasoningEffort: string;+ maxTokensPerTurn: number;+ totals: Totals;+};++export type SetupResult = {+ runs: number;+ pass: number;+ rightAnswerEvidence: number;+ costUsd: number;+ costPerRunUsd: number;+};++export type Setup = {+ suite: string;+ arm: string;+ label: string;+ description: string;+ results: Record<ModelKey, SetupResult>;+};++export type RunKind = "pass" | "trajectory" | "wrong" | "budget";++export type CaseRun = {+ status: "pass" | "fail";+ kind: RunKind;+ graders: string[];+ firstFailure: string;+ toolCalls: number;+ costUsd: number;+ judge: number | null;+ wallS: number;+ knownIssue: string | null;+};++export type BenchmarkCase = {+ id: string;+ suite: string;+ group: string;+ title: string;+ runs: Record<ModelKey, Record<string, CaseRun>>;+};++export type KnownIssue = {+ id: string;+ kind: "skill" | "case";+ case: string;+ title: string;+ detail: string;+ caseRuns: number;+ affectedRuns: number;+ truncated: number;+ byModel: Record<ModelKey, number>;+};++export type Group = {+ id: string;+ label: string;+ description: string;+ cases: number;+ results: Record<ModelKey, { runs: number; pass: number }>;+};++export type BenchmarkData = {+ run: {+ title: string;+ date: string;+ commit: string;+ cliVersion: string;+ backendVersion: string;+ harnessUrl: string;+ methodologyUrl: string;+ runsPerCase: number;+ totalRuns: number;+ runsPerModel: number;+ allGraded: boolean;+ judge: { id: string; label: string };+ };+ models: Model[];+ suites: { id: string; label: string; cases: number; setups: string[] }[];+ setups: Setup[];+ comparisons: {+ skillUplift: {+ suite: string;+ model: ModelKey;+ with: number;+ without: number;+ pp: number;+ }[];+ cliVsMcp: {+ suite: string;+ cliArm: string;+ mcpArm: string;+ model: ModelKey;+ cli: number;+ mcp: number;+ pp: number;+ }[];+ };+ groups: Group[];+ knownIssues: KnownIssue[];+ cases: BenchmarkCase[];+};++export const SECTIONS = {+ summary: "Summary per model",+ passRate: "Pass rate by suite and setup",+ rightAnswer: "Right answer and evidence by suite and setup",+ comparisons: "Skill uplift and CLI vs MCP",+ groups: "By case group",+ cost: "Cost and efficiency",+ costBySetup: "Cost by suite and setup",+ knownIssues: "Known issues",+ matrix: "Full case matrix",+ runDetails: "Run details",+} as const;++export const MATRIX_KIND: Record<Exclude<RunKind, "pass">, string> = {+ wrong: "wrong answer/evidence",+ trajectory: "trajectory only",+ budget: "budget exhausted",+};++export const COST_METRICS = {+ spend: "Agent spend",+ perRun: "Average cost per run",+ perPass: "Cost per passing run",+ toolCalls: "Average tool calls per run",+ wall: "Average wall time per run",+ cacheHit: "Cache hit rate (input tokens)",+ judge: "Average judge score (0-1)",+ judged: "Runs scored by the judge",+ judgeSpend: "Judge spend",+} as const;++type Row = Record<string, string>;+type Table = { headers: string[]; rows: Row[] };++export function escapeCell(text: string) {+ return text.replace(/\|/g, "\\|").replace(/\n/g, " ");+}++function splitRow(line: string) {+ const inner = line.trim().replace(/^\|/, "").replace(/(?<!\\)\|$/, "");+ return inner.split(/(?<!\\)\|/).map((c) => c.trim().replace(/\\\|/g, "|"));+}++export function parseTables(markdown: string) {+ const tables = new Map<string, Table[]>();+ const lines = markdown.split("\n");+ let heading = "";+ for (let i = 0; i < lines.length; i++) {+ const line = lines[i];+ const h = line.match(/^#{1,6}\s+(.+?)\s*$/);+ if (h) {+ heading = h[1];+ continue;+ }+ if (!line.trim().startsWith("|")) continue;+ const block: string[] = [];+ while (i < lines.length && lines[i].trim().startsWith("|")) block.push(lines[i++]);+ i--;+ if (block.length < 2 || !/^\|?\s*:?-+/.test(block[1].trim())) continue;+ const headers = splitRow(block[0]);+ const rows = block.slice(2).map((l) => {+ const cells = splitRow(l);+ if (cells.length !== headers.length)+ throw new Error(`"${heading}": row has ${cells.length} cells, header ${headers.length}: ${l}`);+ return Object.fromEntries(headers.map((name, j) => [name, cells[j]]));+ });+ const list = tables.get(heading) ?? [];+ list.push({ headers, rows });+ tables.set(heading, list);+ }+ return tables;+}++function table(tables: Map<string, Table[]>, heading: string): Table {+ const t = tables.get(heading)?.[0];+ if (!t) throw new Error(`benchmark markdown: no table under "${heading}"`);+ return t;+}++function cell(row: Row, header: string) {+ const value = row[header];+ if (value === undefined) throw new Error(`benchmark markdown: no "${header}" column`);+ return value;+}++function num(text: string) {+ const n = Number(text.replace(/[$,%s\s]/g, ""));+ if (!Number.isFinite(n) || text.trim() === "") throw new Error(`benchmark markdown: not a number: "${text}"`);+ return n;+}++function fraction(text: string) {+ const m = text.match(/^(\d+)\/(\d+)/);+ if (!m) throw new Error(`benchmark markdown: not a fraction: "${text}"`);+ return { pass: Number(m[1]), runs: Number(m[2]) };+}++function delta(text: string) {+ const m = text.match(/^([+-]?\d+)pp \((\d+)% vs (\d+)%\)$/);+ if (!m) throw new Error(`benchmark markdown: not a difference: "${text}"`);+ return { pp: Number(m[1]), a: Number(m[2]), b: Number(m[3]) };+}++function matrixCell(text: string) {+ if (text === "pass") return { kind: "pass" as RunKind, firstFailure: "" };+ const m = text.match(/^fail \(([^)]+)\): ([\s\S]*)$/);+ const kind = m+ ? (Object.entries(MATRIX_KIND).find(([, label]) => label === m[1])?.[0] as RunKind | undefined)+ : undefined;+ if (!m || !kind) throw new Error(`benchmark markdown: not a matrix cell: "${text}"`);+ return { kind, firstFailure: m[2] };+}++export function parseBenchmark(source: string): BenchmarkData {+ const { data: fm, content } = matter(source);+ const tables = parseTables(content);+ const fmModels = fm.models as Omit<Model, "totals">[];+ const keys = fmModels.map((m) => m.key);+ const keyByLabel = Object.fromEntries(fmModels.map((m) => [m.label, m.key]));+ const perModel = <T,>(f: (label: string, key: ModelKey) => T) =>+ Object.fromEntries(fmModels.map((m) => [m.key, f(m.label, m.key)])) as Record<ModelKey, T>;++ const cost = table(tables, SECTIONS.cost).rows;+ const metric = (name: string, label: string) => {+ const row = cost.find((r) => r.Metric === name);+ if (!row) throw new Error(`benchmark markdown: no "${name}" cost metric`);+ return cell(row, label);+ };++ const summary = table(tables, SECTIONS.summary).rows;+ const models: Model[] = fmModels.map((m) => {+ const row = summary.find((r) => r.Model === m.label);+ if (!row) throw new Error(`benchmark markdown: no summary row for ${m.label}`);+ const judge = metric(COST_METRICS.judge, m.label);+ return {+ ...m,+ totals: {+ runs: num(cell(row, "Runs")),+ pass: num(cell(row, "Passed")),+ rightAnswerEvidence: num(cell(row, "Right answer and evidence")),+ notGraded: num(cell(row, "Not graded")),+ costUsd: num(cell(row, "Agent cost")),+ costPerRunUsd: num(cell(row, "Cost per run")),+ judgeCostUsd: num(metric(COST_METRICS.judgeSpend, m.label)),+ avgToolCalls: num(metric(COST_METRICS.toolCalls, m.label)),+ avgWallS: num(metric(COST_METRICS.wall, m.label)),+ cacheHitRate: num(metric(COST_METRICS.cacheHit, m.label)) / 100,+ avgJudge: judge === "" ? null : num(judge),+ judged: num(metric(COST_METRICS.judged, m.label)),+ },+ };+ });++ const passRows = table(tables, SECTIONS.passRate).rows.filter((r) => r.Suite !== "all");+ const raeRows = table(tables, SECTIONS.rightAnswer).rows;+ const costRows = table(tables, SECTIONS.costBySetup).rows;+ const setups: Setup[] = passRows.map((r) => {+ const suite = cell(r, "Suite");+ const arm = cell(r, "Setup");+ const rae = raeRows.find((x) => x.Suite === suite && x.Setup === arm);+ if (!rae) throw new Error(`benchmark markdown: no right-answer row for ${suite} ${arm}`);+ return {+ suite,+ arm,+ label: cell(r, "Label"),+ description: cell(r, "Description"),+ results: perModel((label, key) => {+ const c = costRows.find((x) => x.Suite === suite && x.Setup === arm && keyByLabel[x.Model] === key);+ if (!c) throw new Error(`benchmark markdown: no cost row for ${suite} ${arm} ${label}`);+ return {+ ...fraction(cell(r, label)),+ rightAnswerEvidence: fraction(cell(rae, label)).pass,+ costUsd: num(cell(c, "Agent cost")),+ costPerRunUsd: num(cell(c, "Cost per run")),+ };+ }),+ };+ });++ const comparisonRows = table(tables, SECTIONS.comparisons).rows;+ const comparisons: BenchmarkData["comparisons"] = { skillUplift: [], cliVsMcp: [] };+ for (const r of comparisonRows) {+ for (const m of fmModels) {+ const d = delta(cell(r, m.label));+ if (r.Comparison === "Skill uplift")+ comparisons.skillUplift.push({ suite: r.Suite, model: m.key, with: d.a, without: d.b, pp: d.pp });+ else if (r.Comparison === "CLI vs MCP")+ comparisons.cliVsMcp.push({ suite: r.Suite, cliArm: cell(r, "Setup A"), mcpArm: cell(r, "Setup B"), model: m.key, cli: d.a, mcp: d.b, pp: d.pp });+ else throw new Error(`benchmark markdown: unknown comparison "${r.Comparison}"`);+ }+ }++ const groups: Group[] = table(tables, SECTIONS.groups).rows.map((r) => ({+ id: cell(r, "Group"),+ label: cell(r, "Label"),+ description: cell(r, "Description"),+ cases: num(cell(r, "Cases")),+ results: perModel((label) => fraction(cell(r, label))),+ }));++ const knownIssues: KnownIssue[] = table(tables, SECTIONS.knownIssues).rows.map((r) => {+ const byModel = perModel((label) => num(cell(r, label)));+ return {+ id: cell(r, "Issue"),+ kind: cell(r, "Kind") as KnownIssue["kind"],+ case: cell(r, "Case"),+ title: cell(r, "Title"),+ detail: cell(r, "Detail"),+ caseRuns: num(cell(r, "Runs of the case")),+ affectedRuns: keys.reduce((n, k) => n + byModel[k], 0),+ truncated: num(cell(r, "Truncated output")),+ byModel,+ };+ });++ const details = new Map<string, Row>();+ for (const r of table(tables, SECTIONS.runDetails).rows)+ details.set(`${r.Case}|${r.Setup}|${keyByLabel[r.Model]}`, r);++ const cases: BenchmarkCase[] = [];+ for (const r of table(tables, SECTIONS.matrix).rows) {+ const id = cell(r, "Case");+ let c = cases.find((x) => x.id === id);+ if (!c) {+ c = {+ id,+ suite: cell(r, "Suite"),+ group: cell(r, "Group"),+ title: cell(r, "Description"),+ runs: perModel(() => ({}) as Record<string, CaseRun>),+ };+ cases.push(c);+ }+ const arm = cell(r, "Setup");+ for (const m of fmModels) {+ const text = cell(r, m.label);+ if (text === "") continue;+ const { kind, firstFailure } = matrixCell(text);+ const d = details.get(`${id}|${arm}|${m.key}`);+ if (!d) throw new Error(`benchmark markdown: no run details for ${id} ${arm} ${m.label}`);+ const judge = cell(d, "Judge");+ const graders = cell(d, "Failed graders");+ c.runs[m.key][arm] = {+ status: kind === "pass" ? "pass" : "fail",+ kind,+ graders: graders ? graders.split("+") : [],+ firstFailure,+ toolCalls: num(cell(d, "Tool calls")),+ costUsd: num(cell(d, "Agent cost")),+ judge: judge === "" ? null : num(judge),+ wallS: num(cell(d, "Wall time (s)")),+ knownIssue: cell(d, "Known issue") || null,+ };+ }+ }++ const suites = (fm.suites as { id: string; label: string }[]).map((s) => ({+ ...s,+ cases: cases.filter((c) => c.suite === s.id).length,+ setups: setups.filter((x) => x.suite === s.id).map((x) => x.arm),+ }));++ const totalRuns = models.reduce((n, m) => n + m.totals.runs, 0);+ return {+ run: {+ title: String(fm.title),+ date: String(fm.date),+ commit: String(fm.commit),+ cliVersion: String(fm.cliVersion),+ backendVersion: String(fm.backendVersion),+ harnessUrl: String(fm.harnessUrl),+ methodologyUrl: String(fm.methodologyUrl),+ runsPerCase: Number(fm.runsPerCase),+ totalRuns,+ runsPerModel: totalRuns / models.length,+ allGraded:+ models.every((m) => m.totals.notGraded === 0) &&+ cases.every((c) => keys.every((k) => Object.values(c.runs[k]).every((x) => x.kind !== "budget"))),+ judge: fm.judge as { id: string; label: string },+ },+ models,+ suites,+ setups,+ comparisons,+ groups,+ knownIssues,+ cases,+ };+}diff --git a/website/scripts/build-benchmarks-data.mjs b/website/scripts/build-benchmarks-data.mjsnew file mode 100644index 00000000..62d2148a--- /dev/null+++ b/website/scripts/build-benchmarks-data.mjs@@ -0,0 +1,966 @@+#!/usr/bin/env node+// Builds content/benchmarks/<date>.md, the only source of numbers for+// app/benchmarks, from a three-model `twevals` comparison directory+// (runs.csv, results.json, summary.md, stats.md) and the eval case files.+// Before writing, every aggregate is checked against summary.md and stats.md;+// after writing, the file is parsed back with the page's own parser and+// checked again. Any mismatch aborts.+//+// node scripts/build-benchmarks-data.mjs --source <comparison dir> \+// [--cases ../cli/test/evals/cases] [--models ../cli/test/evals/models.yaml] \+// [--out content/benchmarks/<date>.md]++import fs from "node:fs";+import path from "node:path";+import { fileURLToPath } from "node:url";++import {+ COST_METRICS,+ MATRIX_KIND,+ SECTIONS,+ escapeCell,+ parseBenchmark,+} from "../lib/benchmarks.ts";++const WEBSITE = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");+const HARNESS_URL =+ "https://github.com/tracewayapp/traceway/tree/main/cli/test/evals";+const METHODOLOGY_URL = "https://docs.tracewayapp.com/learn/agent-benchmarks";+const PAGE_URL = "https://tracewayapp.com/benchmarks";++const MODEL_ORDER = ["haiku", "deepseek-flash", "opus"];+const MODEL_SHORT = { haiku: "Haiku", "deepseek-flash": "DeepSeek", opus: "Opus" };+const MODEL_VENDOR = { haiku: "Anthropic", "deepseek-flash": "DeepSeek", opus: "Anthropic" };+const MODEL_WEIGHTS = { haiku: "proprietary", "deepseek-flash": "open", opus: "proprietary" };++const SUITES = [+ {+ id: "perf",+ label: "Performance",+ description: "Diagnose a latency problem from production telemetry.",+ },+ {+ id: "write",+ label: "Write",+ description:+ "Act on the system safely: pages, alerts, monitors, archiving and post-mortems.",+ },+];++const SETUPS = {+ "cli+skill": {+ label: "CLI + skill",+ description: "The traceway CLI plus the Traceway agent skill.",+ },+ "cli-noskill": {+ label: "CLI, no skill",+ description: "The same CLI with no skill: the control group.",+ },+ mcp: {+ label: "MCP server",+ description: "The Traceway MCP server: the same knowledge, plus tools.",+ },+ "cli-write+skill": {+ label: "Write CLI + skill",+ description: "The traceway-write CLI plus the write skill.",+ },+ "mcp-write": {+ label: "Write MCP server",+ description: "The write MCP server.",+ },+};++const GROUPS = [+ {+ id: "perf-app",+ label: "App code",+ description:+ "Slow code in the request path: serialization, quadratic loops, large allocations, debug logging, work that belongs in the background.",+ },+ {+ id: "perf-cache",+ label: "Caching",+ description: "Cold starts after a release, a missing cache, a cache stampede.",+ },+ {+ id: "perf-ctl",+ label: "Controls and traps",+ description:+ "A healthy endpoint, endpoints marked slow on purpose, a misleading metric, a pasted alert and an onset to pin down. Some have nothing to fix.",+ },+ {+ id: "perf-db",+ label: "Database",+ description:+ "N+1 queries, a missing index, pool exhaustion, lock contention, over-fetching, unbounded results.",+ },+ {+ id: "perf-infra",+ label: "Infrastructure",+ description: "CPU saturation, disk I/O, memory pressure, a noisy neighbor.",+ },+ {+ id: "perf-net",+ label: "Network and upstreams",+ description:+ "Retries, chatty fan-out, sequential calls, large payloads, connection reuse, a slow third party.",+ },+ {+ id: "perf-xs",+ label: "Cross-stack and mobile",+ description:+ "A browser or mobile app fails, and the agent has to follow it into the backend.",+ },+ {+ id: "write",+ label: "Write actions",+ description:+ "Acknowledge or resolve pages, snooze or disable alerts, run a monitor, archive issues, write post-mortems, refuse what a token may not do.",+ },+];++// Case titles that would give away the expected diagnosis are replaced by a+// summary of the case prompt, which only states the symptom.+const DESCRIPTION_OVERRIDES = {+ "perf-ctl-alert-handoff":+ "A p95 latency alert pasted in from email: what happened, and does it need action?",+ "perf-ctl-metric-p95-trap":+ "Search gets sluggish every afternoon, and ops says the hosts' p95 CPU looks fine",+ "perf-ctl-onset-bisect":+ "The account page has felt slower for a few days: when did it start, and what changed?",+ "perf-db-missing-query-cache":+ "Every storefront endpoint is slower than it should be, even a single-row read",+ "perf-xs-android-timeouts":+ "The Android app crashes at checkout since the sale went live",+ "perf-xs-frontend-timeouts":+ "Product search on the web shop spins, then says it is taking too long",+ "perf-xs-ios-watchdog":+ "iOS users see the app freeze on the splash screen and close",+ "write-archive-ambiguous": "Archive last week's checkout errors",+};++// The shared problems named in the comparison's write-up. Each matcher picks+// the affected runs out of the data, so every count on the page is computed.+const KNOWN_ISSUES = [+ {+ id: "iowait-recipe",+ kind: "skill",+ case: "perf-infra-disk-io",+ title: "The skill's CPU recipe counts iowait as busy CPU",+ detail:+ "On the disk case the database host's idle CPU falls while iowait rises. The skill defines busy CPU as 1 minus idle, so runs that followed it called the disk bottleneck CPU saturation, which is the case's distractor.",+ matches: (r) => r.firstFailure.includes('checklist_item "cpu_saturation"'),+ },+ {+ id: "stampede-logs-query",+ kind: "case",+ case: "perf-cache-stampede",+ title: "The cache stampede case insists on a logs query",+ detail:+ "The case requires a logs.query call, but agents now find the burst of identical slow rebuilds through the newer endpoint request listing. These runs had the right answer and grounded evidence.",+ matches: (r) =>+ r.kind === "trajectory" &&+ r.firstFailure.includes("required call missing: logs.query"),+ },+ {+ id: "memory-gc-label",+ kind: "case",+ case: "perf-infra-memory-gc",+ title: "The memory case rejects a fair label",+ detail:+ "These runs found the memo that never evicts, but filed it under unbounded in-memory work instead of memory pressure and GC pauses. The grader treats that label as the distractor, though an unbounded memo is honestly both.",+ matches: (r) =>+ r.graders.length === 1 &&+ r.graders[0] === "answer" &&+ r.firstFailure.includes("unbounded_in_memory_work_large_allocations"),+ },+ {+ id: "chatty-fanout-grounding",+ kind: "case",+ case: "perf-net-chatty-fanout",+ title: "The fan-out case fails grounding for every model",+ detail:+ "Every model fails this case's grounding check on most setups, mostly because the output it cites was truncated.",+ matches: (r) => r.graders.includes("grounding"),+ },+];++const KIND_BY_FAILURE = {+ pass: "pass",+ "trajectory-only (other)": "trajectory",+ "trajectory-only (repeat)": "trajectory",+ "wrong answer/evidence": "wrong",+ budget_exhausted: "budget",+};++function parseArgs(argv) {+ const args = {};+ for (let i = 0; i < argv.length; i++) {+ const a = argv[i];+ if (!a.startsWith("--")) throw new Error(`unexpected argument ${a}`);+ const value = argv[i + 1];+ if (value === undefined || value.startsWith("--"))+ throw new Error(`${a} needs a value`);+ args[a.slice(2)] = value;+ i++;+ }+ return args;+}++function parseCsv(text) {+ const rows = [];+ let row = [];+ let field = "";+ let quoted = false;+ for (let i = 0; i < text.length; i++) {+ const c = text[i];+ if (quoted) {+ if (c === '"' && text[i + 1] === '"') {+ field += '"';+ i++;+ } else if (c === '"') {+ quoted = false;+ } else {+ field += c;+ }+ } else if (c === '"') {+ quoted = true;+ } else if (c === ",") {+ row.push(field);+ field = "";+ } else if (c === "\n" || c === "\r") {+ if (c === "\r" && text[i + 1] === "\n") i++;+ row.push(field);+ rows.push(row);+ row = [];+ field = "";+ } else {+ field += c;+ }+ }+ if (field !== "" || row.length) {+ row.push(field);+ rows.push(row);+ }+ const [header, ...body] = rows.filter((r) => r.length > 1 || r[0] !== "");+ return body.map((r) => {+ if (r.length !== header.length)+ throw new Error(`runs.csv row has ${r.length} fields, want ${header.length}`);+ return Object.fromEntries(header.map((h, i) => [h, r[i]]));+ });+}++function parseModelsYaml(text) {+ const models = {};+ let section = null;+ let current = null;+ for (const line of text.split("\n")) {+ if (/^\S/.test(line)) {+ section = line.replace(/:.*$/, "").trim();+ current = section === "judge" ? (models.judge = {}) : null;+ continue;+ }+ const block = line.match(/^ {2}([\w-]+):\s*$/);+ if (section === "agents" && block) {+ current = models[block[1]] = {};+ continue;+ }+ const indent = section === "judge" ? 2 : 4;+ const kv = line.match(new RegExp(`^ {${indent}}(id|label|reasoning_effort|max_tokens_per_turn):\\s*(.+?)\\s*$`));+ if (current && kv) current[kv[1]] = kv[2];+ }+ return models;+}++function readCase(file) {+ const text = fs.readFileSync(file, "utf8");+ const field = (name) => {+ const m = text.match(new RegExp(`^${name}:\\s*(.+?)\\s*$`, "m"));+ if (!m) throw new Error(`${file}: no ${name}`);+ return m[1].replace(/^(["'])(.*)\1$/, "$2");+ };+ return { id: field("id"), suite: field("suite"), title: field("title") };+}++const pct = (n, d) => Math.round((100 * n) / d);+const round = (x, digits) => Number(x.toFixed(digits));+const sum = (xs) => xs.reduce((a, b) => a + b, 0);+const mean = (xs) => (xs.length ? sum(xs) / xs.length : null);++function summarize(runs) {+ const input = sum(runs.map((r) => r.inputTokens));+ const judged = runs.filter((r) => r.judge !== null);+ return {+ runs: runs.length,+ pass: runs.filter((r) => r.status === "pass").length,+ rightAnswerEvidence: runs.filter((r) => r.kind === "pass" || r.kind === "trajectory").length,+ costUsd: round(sum(runs.map((r) => r.costUsd)), 4),+ judgeCostUsd: round(sum(runs.map((r) => r.judgeCostUsd)), 4),+ costPerRunUsd: round(sum(runs.map((r) => r.costUsd)) / runs.length, 6),+ avgToolCalls: round(mean(runs.map((r) => r.toolCalls)), 4),+ avgFailedCalls: round(mean(runs.map((r) => r.failedCalls)), 4),+ avgTurns: round(mean(runs.map((r) => r.turns)), 4),+ avgInputTokens: Math.round(input / runs.length),+ avgOutputTokens: Math.round(mean(runs.map((r) => r.outputTokens))),+ cacheHitRate: round(sum(runs.map((r) => r.cachedTokens)) / input, 6),+ avgWallS: round(mean(runs.map((r) => r.wallS)), 4),+ avgJudge: judged.length ? round(mean(judged.map((r) => r.judge)), 6) : null,+ judged: judged.length,+ };+}++function tableRows(markdown, heading) {+ const start = markdown.indexOf(heading);+ if (start < 0) throw new Error(`no "${heading}" section`);+ const lines = markdown.slice(start).split("\n");+ const rows = [];+ let seen = false;+ for (const line of lines) {+ if (line.startsWith("|")) {+ seen = true;+ rows.push(line.split("|").slice(1, -1).map((c) => c.trim()));+ } else if (seen) break;+ }+ return rows.slice(2);+}++function main() {+ const args = parseArgs(process.argv.slice(2));+ if (!args.source) throw new Error("--source <comparison dir> is required");+ const source = path.resolve(args.source);+ const casesDir = path.resolve(WEBSITE, args.cases ?? "../cli/test/evals/cases");+ const modelsFile = path.resolve(WEBSITE, args.models ?? "../cli/test/evals/models.yaml");++ const csvRuns = parseCsv(fs.readFileSync(path.join(source, "runs.csv"), "utf8"));+ const results = JSON.parse(fs.readFileSync(path.join(source, "results.json"), "utf8"));+ const summaryMd = fs.readFileSync(path.join(source, "summary.md"), "utf8");+ const statsMd = fs.readFileSync(path.join(source, "stats.md"), "utf8");+ const modelsYaml = parseModelsYaml(fs.readFileSync(modelsFile, "utf8"));++ const runs = csvRuns.map((r) => {+ const kind = KIND_BY_FAILURE[r.failure_kind];+ if (!kind) throw new Error(`unknown failure_kind ${r.failure_kind}`);+ if ((r.status === "pass") !== (kind === "pass"))+ throw new Error(`${r.case} ${r.model} ${r.arm}: status ${r.status} with kind ${r.failure_kind}`);+ return {+ case: r.case,+ suite: r.suite,+ arm: r.arm,+ model: r.model,+ status: r.status,+ kind,+ graders: r.failed_graders ? r.failed_graders.split("+") : [],+ firstFailure: r.first_failure,+ toolCalls: Number(r.tool_calls),+ failedCalls: Number(r.failed_calls),+ turns: Number(r.turns),+ inputTokens: Number(r.input_tokens),+ cachedTokens: Number(r.cached_tokens),+ outputTokens: Number(r.output_tokens),+ costUsd: Number(r.cost_usd),+ judgeCostUsd: Number(r.judge_cost_usd || 0),+ judge: r.judge_score === "" ? null : Number(r.judge_score),+ wallS: Number(r.wall_s),+ };+ });++ const modelKeys = [...new Set(runs.map((r) => r.model))];+ if (modelKeys.length !== MODEL_ORDER.length || !MODEL_ORDER.every((m) => modelKeys.includes(m)))+ throw new Error(`models in runs.csv: ${modelKeys.join(", ")}`);++ const caseFiles = SUITES.flatMap((s) =>+ fs+ .readdirSync(path.join(casesDir, s.id))+ .filter((f) => f.endsWith(".yaml"))+ .map((f) => readCase(path.join(casesDir, s.id, f))),+ );+ const caseById = new Map(caseFiles.map((c) => [c.id, c]));++ const header = results.header;+ const runDate = header.started.slice(0, 10);+ const commit = header.gitSha.slice(0, 8);++ const providersByModel = {};+ const modelIdByKey = {};+ for (const r of results.runs) {+ modelIdByKey[r.model] ??= r.modelId;+ if (modelIdByKey[r.model] !== r.modelId)+ throw new Error(`${r.model} ran as both ${modelIdByKey[r.model]} and ${r.modelId}`);+ const counts = (providersByModel[r.model] ??= {});+ for (const p of new Set(r.provider ?? [])) counts[p] = (counts[p] ?? 0) + 1;+ }+ if (results.runs.length !== runs.length)+ throw new Error(`results.json has ${results.runs.length} runs, runs.csv ${runs.length}`);++ const models = MODEL_ORDER.map((key) => {+ const yaml = modelsYaml[key];+ if (!yaml) throw new Error(`models.yaml has no ${key}`);+ if (yaml.id !== modelIdByKey[key])+ throw new Error(`${key}: models.yaml id ${yaml.id}, results.json ${modelIdByKey[key]}`);+ const own = runs.filter((r) => r.model === key);+ const failingGraders = {};+ for (const r of own) for (const g of r.graders) failingGraders[g] = (failingGraders[g] ?? 0) + 1;+ const providers = Object.entries(providersByModel[key] ?? {})+ .sort((a, b) => b[1] - a[1])+ .map(([name, runCount]) => ({ name, runs: runCount }));+ return {+ key,+ label: yaml.label,+ short: MODEL_SHORT[key],+ vendor: MODEL_VENDOR[key],+ weights: MODEL_WEIGHTS[key],+ id: yaml.id,+ provider: providers[0]?.name ?? "",+ reasoningEffort: yaml.reasoning_effort,+ maxTokensPerTurn: Number(yaml.max_tokens_per_turn),+ providers,+ totals: {+ ...summarize(own),+ notGraded: own.filter((r) => r.status !== "pass" && r.status !== "fail").length,+ failingGraders,+ },+ };+ });++ const armsBySuite = Object.fromEntries(+ SUITES.map((s) => [+ s.id,+ Object.keys(SETUPS).filter((arm) => runs.some((r) => r.suite === s.id && r.arm === arm)),+ ]),+ );++ const suites = SUITES.map((s) => {+ const ids = [...new Set(runs.filter((r) => r.suite === s.id).map((r) => r.case))];+ return { ...s, cases: ids.length, setups: armsBySuite[s.id] };+ });++ const setups = suites.flatMap((s) =>+ s.setups.map((arm) => ({+ suite: s.id,+ arm,+ ...SETUPS[arm],+ results: Object.fromEntries(+ MODEL_ORDER.map((m) => [+ m,+ summarize(runs.filter((r) => r.suite === s.id && r.arm === arm && r.model === m)),+ ]),+ ),+ })),+ );+ const setup = (suite, arm, m) =>+ setups.find((x) => x.suite === suite && x.arm === arm).results[m];+ const rateOf = (x) => (100 * x.pass) / x.runs;++ const comparisons = {+ skillUplift: MODEL_ORDER.map((m) => {+ const a = setup("perf", "cli+skill", m);+ const b = setup("perf", "cli-noskill", m);+ return { suite: "perf", model: m, with: pct(a.pass, a.runs), without: pct(b.pass, b.runs), pp: Math.round(rateOf(a) - rateOf(b)) };+ }),+ cliVsMcp: [+ ["perf", "cli+skill", "mcp"],+ ["write", "cli-write+skill", "mcp-write"],+ ].flatMap(([suite, cli, mcp]) =>+ MODEL_ORDER.map((m) => {+ const a = setup(suite, cli, m);+ const b = setup(suite, mcp, m);+ return { suite, cliArm: cli, mcpArm: mcp, model: m, cli: pct(a.pass, a.runs), mcp: pct(b.pass, b.runs), pp: Math.round(rateOf(a) - rateOf(b)) };+ }),+ ),+ };++ const groupOf = (caseId) => GROUPS.find((g) => caseId === g.id || caseId.startsWith(`${g.id}-`)).id;+ const groups = GROUPS.map((g) => {+ const own = runs.filter((r) => groupOf(r.case) === g.id);+ return {+ ...g,+ cases: new Set(own.map((r) => r.case)).size,+ results: Object.fromEntries(+ MODEL_ORDER.map((m) => {+ const mine = own.filter((r) => r.model === m);+ return [m, { runs: mine.length, pass: mine.filter((r) => r.status === "pass").length }];+ }),+ ),+ };+ });++ const issueOf = new Map();+ const knownIssues = KNOWN_ISSUES.map((issue) => {+ const caseRuns = runs.filter((r) => r.case === issue.case);+ const hit = caseRuns.filter((r) => r.status !== "pass" && issue.matches(r));+ if (!hit.length) throw new Error(`known issue ${issue.id} matches no run`);+ for (const r of hit) issueOf.set(r, issue.id);+ return {+ id: issue.id,+ kind: issue.kind,+ case: issue.case,+ title: issue.title,+ detail: issue.detail,+ caseRuns: caseRuns.length,+ caseFailures: caseRuns.filter((r) => r.status !== "pass").length,+ affectedRuns: hit.length,+ truncated: hit.filter((r) => r.firstFailure.includes("truncated")).length,+ byModel: Object.fromEntries(MODEL_ORDER.map((m) => [m, hit.filter((r) => r.model === m).length])),+ };+ });++ const caseIds = [...new Set(runs.map((r) => r.case))].sort();+ const cases = caseIds.map((id) => {+ const meta = caseById.get(id);+ if (!meta) throw new Error(`no case file for ${id}`);+ const own = runs.filter((r) => r.case === id);+ return {+ id,+ suite: meta.suite,+ group: groupOf(id),+ title: DESCRIPTION_OVERRIDES[id] ?? meta.title,+ runs: Object.fromEntries(+ MODEL_ORDER.map((m) => [+ m,+ Object.fromEntries(+ own+ .filter((r) => r.model === m)+ .map((r) => [+ r.arm,+ {+ status: r.status,+ kind: r.kind,+ graders: r.graders,+ firstFailure: r.firstFailure,+ toolCalls: r.toolCalls,+ costUsd: round(r.costUsd, 5),+ judge: r.judge === null ? null : round(r.judge, 2),+ wallS: r.wallS,+ knownIssue: issueOf.get(r) ?? null,+ },+ ]),+ ),+ ]),+ ),+ };+ });+ for (const id of Object.keys(DESCRIPTION_OVERRIDES))+ if (!caseIds.includes(id)) throw new Error(`description override for unknown case ${id}`);++ const judge = modelsYaml.judge;+ const data = {+ run: {+ title: `Traceway agent benchmarks: ${runDate}`,+ date: runDate,+ commit,+ cliVersion: header.cliVersion,+ backendVersion: header.backendVersion,+ runsPerCase: Math.max(...results.runs.map((r) => r.runIdx)),+ totalRuns: runs.length,+ runsPerModel: runs.length / MODEL_ORDER.length,+ agentCostUsd: round(sum(runs.map((r) => r.costUsd)), 2),+ judgeCostUsd: round(sum(runs.map((r) => r.judgeCostUsd)), 2),+ judge: { id: judge.id, label: judge.label.replace(/\s*\(judge\)$/, "") },+ harnessUrl: HARNESS_URL,+ methodologyUrl: METHODOLOGY_URL,+ statuses: Object.fromEntries(+ [...new Set(runs.map((r) => r.status))].map((s) => [s, runs.filter((r) => r.status === s).length]),+ ),+ },+ models,+ suites,+ setups,+ comparisons,+ groups,+ knownIssues,+ cases,+ };++ crossCheck(data, summaryMd, statsMd);++ const out = path.resolve(WEBSITE, args.out ?? `content/benchmarks/${runDate}.md`);+ fs.mkdirSync(path.dirname(out), { recursive: true });+ fs.writeFileSync(out, renderMarkdown(data));+ console.log(`wrote ${path.relative(WEBSITE, out)}: ${data.run.totalRuns} runs, ${cases.length} cases`);++ parseBack(data, fs.readFileSync(out, "utf8"), summaryMd);+}++function crossCheck(data, summaryMd, statsMd) {+ const errors = [];+ const expect = (what, got, want) => {+ if (String(got) !== String(want)) errors.push(`${what}: generated ${got}, report says ${want}`);+ };++ const runsLine = summaryMd.match(/\*\*Runs:\*\* (\d+) \(fail (\d+), pass (\d+)\)/);+ if (!runsLine) errors.push("summary.md: no Runs line");+ else {+ expect("total runs", data.run.totalRuns, runsLine[1]);+ expect("failing runs", data.run.statuses.fail ?? 0, runsLine[2]);+ expect("passing runs", data.run.statuses.pass ?? 0, runsLine[3]);+ }++ const spend = summaryMd.match(/\*\*Spend:\*\* \$([\d.]+) \(judge \$([\d.]+)\)/);+ if (!spend) errors.push("summary.md: no Spend line");+ else {+ expect("judge spend", data.run.judgeCostUsd.toFixed(2), spend[2]);+ expect("total spend", (data.run.agentCostUsd + data.run.judgeCostUsd).toFixed(2), spend[1]);+ }++ const setupOf = (suite, model, arm) =>+ data.setups.find((s) => s.suite === suite && s.arm === arm)?.results[model];++ for (const [suite, model, arm, frac, rate] of tableRows(summaryMd, "## Pass rate by suite")) {+ const s = setupOf(suite, model, arm);+ if (!s) {+ errors.push(`summary.md row ${suite} ${model} ${arm} not generated`);+ continue;+ }+ expect(`${suite} ${model} ${arm} passes`, `${s.pass}/${s.runs}`, frac);+ expect(`${suite} ${model} ${arm} rate`, `${pct(s.pass, s.runs)}%`, rate);+ }++ for (const [, model, w, wo, uplift] of tableRows(summaryMd, "## Skill uplift")) {+ const u = data.comparisons.skillUplift.find((x) => x.model === model);+ expect(`${model} skill uplift`, `${u.with}% ${u.without}% ${u.pp >= 0 ? "+" : ""}${u.pp}pp`, `${w} ${wo} ${uplift}`);+ }++ for (const [suite, model, , , cli, mcp, delta] of tableRows(summaryMd, "## CLI vs MCP")) {+ const c = data.comparisons.cliVsMcp.find((x) => x.suite === suite && x.model === model);+ expect(`${suite} ${model} CLI vs MCP`, `${c.cli}% ${c.mcp}% ${c.pp >= 0 ? "+" : ""}${c.pp}pp`, `${cli} ${mcp} ${delta}`);+ }++ for (const row of tableRows(summaryMd, "## Cost (averages over passing runs)")) {+ const m = data.models.find((x) => x.key === row[0]);+ if (!m) continue;+ expect(`${m.key} passing runs`, m.totals.pass, row[1]);+ expect(`${m.key} cache hit`, `${Math.round(100 * m.totals.cacheHitRate)}%`, row[6]);+ expect(`${m.key} agent spend`, `$${m.totals.costUsd.toFixed(2)}`, row[7]);+ }++ for (const row of tableRows(statsMd, "| Model | Suite | Setup | Pass |")) {+ const [model, suite, arm, , rae, notGraded, cost, calls, , cache, judge] = row;+ const s = setupOf(suite, model, arm);+ if (!s) {+ errors.push(`stats.md row ${suite} ${model} ${arm} not generated`);+ continue;+ }+ expect(`${suite} ${model} ${arm} right answer and evidence`, `${s.rightAnswerEvidence}/${s.runs} (${pct(s.rightAnswerEvidence, s.runs)}%)`, rae);+ expect(`${suite} ${model} ${arm} not graded`, 0, notGraded);+ expect(`${suite} ${model} ${arm} cost`, `$${s.costUsd.toFixed(2)}`, cost);+ expect(`${suite} ${model} ${arm} avg calls`, s.avgToolCalls.toFixed(1), calls);+ expect(`${suite} ${model} ${arm} cache hit`, `${Math.round(100 * s.cacheHitRate)}%`, cache);+ expect(`${suite} ${model} ${arm} judge`, s.avgJudge.toFixed(2), judge);+ }++ if (errors.length) {+ console.error(`cross-check against summary.md and stats.md failed:\n ${errors.join("\n ")}`);+ process.exit(1);+ }+ console.log("cross-check against summary.md and stats.md: ok");+}++const usd = (x, digits) => `$${x.toFixed(digits)}`;+const frac = (pass, runs) => `${pass}/${runs} (${pct(pass, runs)}%)`;+const signedPp = (pp) => `${pp >= 0 ? "+" : "-"}${Math.abs(pp)}pp`;+const words = ["zero", "one", "two", "three", "four", "five"];++function yamlScalar(v) {+ return typeof v === "number" ? String(v) : JSON.stringify(v);+}++function yamlBlock(value, indent = "") {+ const lines = [];+ for (const [k, v] of Object.entries(value)) {+ if (Array.isArray(v)) {+ lines.push(`${indent}${k}:`);+ for (const item of v) {+ const [first, ...rest] = yamlBlock(item, `${indent} `).split("\n");+ lines.push(`${indent} - ${first.trimStart()}`, ...rest);+ }+ } else if (v && typeof v === "object") {+ lines.push(`${indent}${k}:`, yamlBlock(v, `${indent} `));+ } else {+ lines.push(`${indent}${k}: ${yamlScalar(v)}`);+ }+ }+ return lines.join("\n");+}++function mdTable(headers, rows, align) {+ const line = (cells) => `| ${cells.map((c) => escapeCell(String(c))).join(" | ")} |`;+ const sep = `|${headers.map((_, i) => (align?.[i] === "r" ? "---:" : "---")).join("|")}|`;+ return [line(headers), sep, ...rows.map(line)].join("\n");+}++function renderMarkdown(data) {+ const { run, models, suites, setups, comparisons, groups, knownIssues, cases } = data;+ const labels = models.map((m) => m.label);+ const right = (n) => Array(n).fill("r");+ const setupRow = (suite, arm) => setups.find((x) => x.suite === suite && x.arm === arm);+ const front = {+ title: run.title,+ date: run.date,+ commit: run.commit,+ cliVersion: run.cliVersion,+ backendVersion: run.backendVersion,+ harnessUrl: run.harnessUrl,+ methodologyUrl: run.methodologyUrl,+ runsPerCase: run.runsPerCase,+ judge: run.judge,+ suites: suites.map((x) => ({ id: x.id, label: x.label })),+ models: models.map((m) => ({+ key: m.key,+ label: m.label,+ short: m.short,+ vendor: m.vendor,+ weights: m.weights,+ id: m.id,+ provider: m.provider,+ providers: m.providers,+ reasoningEffort: m.reasoningEffort,+ maxTokensPerTurn: m.maxTokensPerTurn,+ })),+ };++ const comparisonRows = [+ ...comparisons.skillUplift.length+ ? [["Skill uplift", "perf", "cli+skill", "cli-noskill", ...models.map((m) => {+ const u = comparisons.skillUplift.find((x) => x.model === m.key);+ return `${signedPp(u.pp)} (${u.with}% vs ${u.without}%)`;+ })]]+ : [],+ ...[...new Set(comparisons.cliVsMcp.map((c) => c.suite))].map((suite) => {+ const first = comparisons.cliVsMcp.find((c) => c.suite === suite);+ return ["CLI vs MCP", suite, first.cliArm, first.mcpArm, ...models.map((m) => {+ const c = comparisons.cliVsMcp.find((x) => x.suite === suite && x.model === m.key);+ return `${signedPp(c.pp)} (${c.cli}% vs ${c.mcp}%)`;+ })];+ }),+ ];++ const metricRows = [+ [COST_METRICS.spend, ...models.map((m) => usd(m.totals.costUsd, 2))],+ [COST_METRICS.perRun, ...models.map((m) => usd(m.totals.costPerRunUsd, 5))],+ [COST_METRICS.perPass, ...models.map((m) => usd(m.totals.costUsd / m.totals.pass, 5))],+ [COST_METRICS.toolCalls, ...models.map((m) => m.totals.avgToolCalls.toFixed(1))],+ [COST_METRICS.wall, ...models.map((m) => `${m.totals.avgWallS.toFixed(1)} s`)],+ [COST_METRICS.cacheHit, ...models.map((m) => `${Math.round(100 * m.totals.cacheHitRate)}%`)],+ [COST_METRICS.judge, ...models.map((m) => (m.totals.avgJudge === null ? "" : m.totals.avgJudge.toFixed(2)))],+ [COST_METRICS.judged, ...models.map((m) => m.totals.judged)],+ [COST_METRICS.judgeSpend, ...models.map((m) => usd(m.totals.judgeCostUsd, 4))],+ ];++ const armsOf = (suite) => suites.find((x) => x.id === suite).setups;+ const matrixRows = cases.flatMap((c) =>+ armsOf(c.suite).map((arm) => [+ c.id,+ c.group,+ c.title,+ c.suite,+ arm,+ ...models.map((m) => {+ const r = c.runs[m.key][arm];+ if (!r) return "";+ if (r.kind === "pass") return "pass";+ if (!r.firstFailure) throw new Error(`${c.id} ${arm} ${m.key}: failed run without a first failure`);+ return `fail (${MATRIX_KIND[r.kind]}): ${r.firstFailure}`;+ }),+ ]),+ );+ const detailRows = cases.flatMap((c) =>+ armsOf(c.suite).flatMap((arm) =>+ models+ .filter((m) => c.runs[m.key][arm])+ .map((m) => {+ const r = c.runs[m.key][arm];+ return [+ c.id,+ arm,+ m.label,+ r.graders.join("+"),+ r.toolCalls,+ usd(r.costUsd, 5),+ r.wallS.toFixed(1),+ r.judge === null ? "" : r.judge.toFixed(2),+ r.knownIssue ?? "",+ ];+ }),+ ),+ );++ const modelList = `${labels.slice(0, -1).join(", ")} and ${labels[labels.length - 1]}`;+ return `---+${yamlBlock(front)}+---++# ${run.title}++${words[models.length][0].toUpperCase()}${words[models.length].slice(1)} models, ${modelList}, each ran the same ${run.runsPerModel} eval runs on ${run.date}, at commit \`${run.commit}\`. Every run gets a fresh Traceway backend seeded with the telemetry of a system broken in one specific way, and deterministic code grades the result. [How we measure](${run.methodologyUrl}) · [the harness and the cases](${run.harnessUrl}).++This file is generated by \`website/scripts/build-benchmarks-data.mjs\` from the eval results and is the data behind ${PAGE_URL}. Change the generator, not this file.++## ${SECTIONS.summary}++A run passes when its answer is right, its evidence is grounded and its writes are safe. "Right answer and evidence" also counts runs that failed only the trajectory check.++${mdTable(+ ["Model", "Runs", "Passed", "Pass rate", "Right answer and evidence", "Not graded", "Agent cost", "Cost per run"],+ models.map((m) => [m.label, m.totals.runs, m.totals.pass, `${pct(m.totals.pass, m.totals.runs)}%`, m.totals.rightAnswerEvidence, m.totals.notGraded, usd(m.totals.costUsd, 2), usd(m.totals.costPerRunUsd, 5)]),+ ["l", ...right(7)],+)}++## ${SECTIONS.passRate}++${mdTable(+ ["Suite", "Setup", "Label", "Description", ...labels],+ [+ ...setups.map((x) => [x.suite, x.arm, x.label, x.description, ...models.map((m) => frac(x.results[m.key].pass, x.results[m.key].runs))]),+ ["all", "", "All runs", "", ...models.map((m) => frac(m.totals.pass, m.totals.runs))],+ ],+)}++### ${SECTIONS.rightAnswer}++${mdTable(+ ["Suite", "Setup", ...labels],+ setups.map((x) => [x.suite, x.arm, ...models.map((m) => `${x.results[m.key].rightAnswerEvidence}/${x.results[m.key].runs}`)]),+)}++## ${SECTIONS.comparisons}++Setup A minus setup B, in percentage points, computed before rounding.++${mdTable(["Comparison", "Suite", "Setup A", "Setup B", ...labels], comparisonRows)}++## ${SECTIONS.groups}++Passes over every setup a case ran on.++${mdTable(+ ["Group", "Label", "Description", "Cases", ...labels],+ groups.map((g) => [g.id, g.label, g.description, g.cases, ...models.map((m) => frac(g.results[m.key].pass, g.results[m.key].runs))]),+)}++## ${SECTIONS.cost}++Every run per model. Agent cost is the model spend the harness recorded for each run, through OpenRouter; the judge's spend is separate. Runs went three at a time on one machine.++${mdTable(["Metric", ...labels], metricRows, ["l", ...right(labels.length)])}++### ${SECTIONS.costBySetup}++${mdTable(+ ["Suite", "Setup", "Model", "Agent cost", "Cost per run", "Avg tool calls", "Cache hit", "Judge"],+ setups.flatMap((x) =>+ models.map((m) => {+ const r = setupRow(x.suite, x.arm).results[m.key];+ return [x.suite, x.arm, m.label, usd(r.costUsd, 2), usd(r.costPerRunUsd, 5), r.avgToolCalls.toFixed(1), `${Math.round(100 * r.cacheHitRate)}%`, r.avgJudge === null ? "" : r.avgJudge.toFixed(2)];+ }),+ ),+ ["l", "l", "l", ...right(5)],+)}++## ${SECTIONS.knownIssues}++Problems in our own cases and skill, not in the models. The runs they hit are still counted as failures. The model columns count the runs each issue hit.++${mdTable(+ ["Issue", "Kind", "Case", "Title", "Detail", ...labels, "Runs of the case", "Truncated output"],+ knownIssues.map((i) => [i.id, i.kind, i.case, i.title, i.detail, ...models.map((m) => i.byModel[m.key]), i.caseRuns, i.truncated]),+)}++## ${SECTIONS.matrix}++One row per case and setup. A cell is \`pass\`, or \`fail (<kind>): <first failure reason>\`, where the kind is \`${MATRIX_KIND.wrong}\` (the answer or its evidence failed), \`${MATRIX_KIND.trajectory}\` (the answer and its evidence were right, but a required call was missing or out of order) or \`${MATRIX_KIND.budget}\`. Descriptions summarize the case without giving the answer away.++${mdTable(["Case", "Group", "Description", "Suite", "Setup", ...labels], matrixRows)}++### ${SECTIONS.runDetails}++Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1, never gating) and the known issue it hit, if any.++${mdTable(+ ["Case", "Setup", "Model", "Failed graders", "Tool calls", "Agent cost", "Wall time (s)", "Judge", "Known issue"],+ detailRows,+ ["l", "l", "l", "l", "r", "r", "r", "r", "l"],+)}+`;+}++function parseBack(data, markdown, summaryMd) {+ const parsed = parseBenchmark(markdown);+ const errors = [];+ const expect = (what, got, want) => {+ if (String(got) !== String(want)) errors.push(`${what}: parsed ${got}, generated ${want}`);+ };++ expect("total runs", parsed.run.totalRuns, data.run.totalRuns);+ expect("commit", parsed.run.commit, data.run.commit);+ expect("all graded", parsed.run.allGraded, (data.run.statuses.pass ?? 0) + (data.run.statuses.fail ?? 0) === data.run.totalRuns);+ for (const m of data.models) {+ const p = parsed.models.find((x) => x.key === m.key);+ const t = m.totals;+ expect(`${m.key} passes`, `${p.totals.pass}/${p.totals.runs}`, `${t.pass}/${t.runs}`);+ expect(`${m.key} right answer and evidence`, p.totals.rightAnswerEvidence, t.rightAnswerEvidence);+ expect(`${m.key} agent cost`, p.totals.costUsd.toFixed(2), t.costUsd.toFixed(2));+ expect(`${m.key} cost per run`, p.totals.costPerRunUsd.toFixed(5), t.costPerRunUsd.toFixed(5));+ expect(`${m.key} judge spend`, p.totals.judgeCostUsd.toFixed(4), t.judgeCostUsd.toFixed(4));+ expect(`${m.key} avg calls`, p.totals.avgToolCalls.toFixed(1), t.avgToolCalls.toFixed(1));+ expect(`${m.key} wall`, p.totals.avgWallS.toFixed(1), t.avgWallS.toFixed(1));+ expect(`${m.key} cache hit`, Math.round(100 * p.totals.cacheHitRate), Math.round(100 * t.cacheHitRate));+ expect(`${m.key} judge`, p.totals.avgJudge?.toFixed(2), t.avgJudge?.toFixed(2));+ expect(`${m.key} judged`, p.totals.judged, t.judged);+ expect(`${m.key} providers`, JSON.stringify(p.providers), JSON.stringify(m.providers));+ }+ for (const x of data.setups)+ for (const m of data.models) {+ const p = parsed.setups.find((y) => y.suite === x.suite && y.arm === x.arm)?.results[m.key];+ const r = x.results[m.key];+ expect(`${x.suite} ${x.arm} ${m.key}`, p && `${p.pass}/${p.runs} ${p.rightAnswerEvidence} ${p.costPerRunUsd.toFixed(5)}`, `${r.pass}/${r.runs} ${r.rightAnswerEvidence} ${r.costPerRunUsd.toFixed(5)}`);+ }+ expect("suites", JSON.stringify(parsed.suites.map((x) => [x.id, x.cases, x.setups])), JSON.stringify(data.suites.map((x) => [x.id, x.cases, x.setups])));+ expect("comparisons", JSON.stringify(parsed.comparisons), JSON.stringify(data.comparisons));+ for (const g of data.groups) {+ const p = parsed.groups.find((y) => y.id === g.id);+ const fold = (x) => [x.cases, ...data.models.map((m) => `${x.results[m.key].pass}/${x.results[m.key].runs}`)].join(" ");+ expect(`group ${g.id}`, p && fold(p), fold(g));+ }+ for (const i of data.knownIssues) {+ const p = parsed.knownIssues.find((y) => y.id === i.id);+ expect(`issue ${i.id}`, p && JSON.stringify([p.byModel, p.affectedRuns, p.caseRuns, p.truncated]), JSON.stringify([i.byModel, i.affectedRuns, i.caseRuns, i.truncated]));+ }+ expect("cases", parsed.cases.map((c) => c.id).join(), data.cases.map((c) => c.id).join());+ let cells = 0;+ for (const c of data.cases) {+ const p = parsed.cases.find((y) => y.id === c.id);+ expect(`${c.id} meta`, p && `${p.suite} ${p.group} ${p.title}`, `${c.suite} ${c.group} ${c.title}`);+ for (const m of data.models)+ for (const [arm, r] of Object.entries(c.runs[m.key])) {+ const q = p?.runs[m.key][arm];+ cells++;+ const flat = (x) => x && JSON.stringify([x.status, x.kind, x.graders, x.firstFailure, x.toolCalls, x.costUsd.toFixed(5), x.judge?.toFixed(2) ?? null, x.wallS, x.knownIssue]);+ expect(`${c.id} ${arm} ${m.key}`, flat(q), flat(r));+ }+ }++ for (const [suite, model, arm, frac] of tableRows(summaryMd, "## Pass rate by suite")) {+ const p = parsed.setups.find((x) => x.suite === suite && x.arm === arm)?.results[model];+ expect(`summary.md ${suite} ${model} ${arm}`, p && `${p.pass}/${p.runs}`, frac);+ }+ for (const row of tableRows(summaryMd, "## Cost (averages over passing runs)")) {+ const p = parsed.models.find((x) => x.key === row[0]);+ if (!p) continue;+ expect(`summary.md ${row[0]} passing runs`, p.totals.pass, row[1]);+ expect(`summary.md ${row[0]} agent spend`, `$${p.totals.costUsd.toFixed(2)}`, row[7]);+ }++ if (errors.length) {+ console.error(`parse-back check failed:\n ${errors.join("\n ")}`);+ process.exit(1);+ }+ console.log(`parse-back check: ok (${cells} runs compared, totals match summary.md)`);+ for (const m of parsed.models) {+ const t = m.totals;+ console.log(+ ` ${m.label}: ${t.pass}/${t.runs} (${pct(t.pass, t.runs)}%), right answer and evidence ${t.rightAnswerEvidence}, agent $${t.costUsd.toFixed(2)}`,+ );+ }+}++main();