graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff vs the Codex app server and pi; graff vs OpenCode 2 and pi
justrach755dd8176b97
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex e8ada1dd..60969393 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -12,9 +12,11 @@ grok-4.6 SuperGrok) is[graff-live-12-report.html](graff-live-12-report.html) plus[artifacts/graff-evals-live/RECEIPT.md](../artifacts/graff-evals-live/RECEIPT.md).-graff against the Codex app server on the same model and ChatGPT account (21-coding and MCP tasks, every run's data) is-[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md).+graff against the Codex app server and pi on the same model and ChatGPT account+(21 coding and MCP tasks, every run's data) is+[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md),+and graff against OpenCode 2 and pi on one model through the Codegraff gateway is+[evals/graff-vs-opencode-and-pi/RESULTS.md](../evals/graff-vs-opencode-and-pi/RESULTS.md).The same 21 tasks against the Grok CLI on the same SuperGrok account are[evals/graff-vs-grok-cli/RESULTS.md](../evals/graff-vs-grok-cli/RESULTS.md),and what both mean for an engineering budget isdiff --git a/evals/enterprise-cost.md b/evals/enterprise-cost.mdindex 87d485f3..43734f80 100644--- a/evals/enterprise-cost.md+++ b/evals/enterprise-cost.md@@ -3,7 +3,7 @@A rough guide built from two measured comparisons, each on the same modeland account in both harnesses:-- [graff vs the Codex app server](graff-vs-codex-app-server/RESULTS.md) (`gpt-6.1-sol`)+- [graff vs the Codex app server and pi](graff-vs-codex-app-server/RESULTS.md) (`gpt-6.1-sol`)- [graff vs the Grok CLI](graff-vs-grok-cli/RESULTS.md) (`grok-4.7`)Both ran the same 21 coding and MCP tasks, 3 times each. Costs are list@@ -13,8 +13,8 @@ prices of the tokens each harness used.| Compared with | graff | Other harness | graff costs || --- | ---: | ---: | ---: |-| Codex app server, its reported usage | $0.0164 | $0.0278 | 41% less |-| Codex app server, with its unreported warm-up request | $0.0164 | up to $0.0524 | up to 69% less |+| Codex app server, its reported usage | $0.0131 | $0.0426 | 69% less |+| Codex app server, with its unreported warm-up request | $0.0131 | up to $0.0672 | up to 81% less || Grok CLI, grok-4.7 public list price | $0.039 | $0.26 to $0.33 | 85% to 88% less || Grok CLI, xAI's price for its build-model route | $0.039 | $0.098 | 60% less |@@ -26,15 +26,15 @@ costs above.| Compared with | graff per month | Other per month | Saved per month | Saved per year || --- | ---: | ---: | ---: | ---: |-| Codex app server, reported usage | $1,378 | $2,335 | $958 | $11,491 |-| Codex app server, with warm-up | $1,378 | $4,402 | $3,024 | $36,288 |+| Codex app server, reported usage | $1,100 | $3,578 | $2,478 | $29,736 |+| Codex app server, with warm-up | $1,100 | $5,645 | $4,544 | $54,533 || Grok CLI, public list price | $3,284 | $22,042 to $27,770 | $18,757 to $24,486 | $225,086 to $293,832 || Grok CLI, xAI's build-model price | $3,284 | $8,240 | $4,956 | $59,472 |Time moves too. Against the Grok CLI, graff took 15.5s per task against106.9s: about 2,100 fewer hours of agent wall time a month at this volume.-Against the Codex app server graff was slower, 28.2s against 25.0s per task-(about 75 more hours a month).+Against the Codex app server graff is now faster, 22.2s against 28.6s per task+(about 149 fewer hours a month).## How far this carriesdiff --git a/evals/graff-vs-codex-app-server/RESULTS.md b/evals/graff-vs-codex-app-server/RESULTS.mdindex 6d4bdcd2..1b1312f2 100644--- a/evals/graff-vs-codex-app-server/RESULTS.md+++ b/evals/graff-vs-codex-app-server/RESULTS.md@@ -1,101 +1,103 @@-# graff vs the Codex app server+# graff vs the Codex app server and piMeasured 2026-10-01. The same model (`gpt-6.1-sol`), ChatGPT account and machine ran the same 21 tasks in each harness, side by side.| Arm | What ran || --- | --- |-| graff v0.0.302.11 | The published release asset, checked against the release's `SHA256SUMS`. |-| graff main | Main at `900e62d9`: v0.0.302.11 plus #1438, #1439 and #1440 (ADR 0226, 0227, 0228). |-| Codex app server | `codex app-server` from codex-cli 0.159.0. |+| graff | The ADR 0231 build (the `perf: drop round trips the model does not need` commit on top of v0.0.302.12), as a scripted `graff repl` over the Codex Responses WebSocket. |+| Codex app server | `codex app-server` from codex-cli 0.159.0, through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py). |+| pi | pi 0.84.2 on the ChatGPT plan (`--provider openai-codex`). pi has no MCP client, so it ran the 11 coding tasks only. |-graff runs as a scripted `graff repl` over the Codex Responses WebSocket, the transport the app server uses (the `graff-codex-repl` harness). The app server runs through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py), which drives its JSON-RPC and reads the token usage it reports. Each harness uses its own defaults. For this model that means `low` effort and `low` verbosity for graff main and for the app server (the model catalog's defaults), and `medium` effort for v0.0.302.11.+Each harness runs at the model catalog's default, `low` effort (pi's `--thinking low`). graff sends the prompt as one bracketed-paste frame, so a task with a line break is one turn, as it is for the other two.## Results-All 21 tasks, mean per task over 3 runs of each:+All 21 tasks (pi: the 11 coding tasks), mean per task over 3 runs of each:| | Pass | Wall per task | List cost per task | Input read from cache | Output tokens per task | Model calls per task || --- | ---: | ---: | ---: | ---: | ---: | ---: |-| graff v0.0.302.11 | 63/63 | 31.2s | $0.0189 | 92% | 598 | 4.7 |-| graff main | 63/63 | 28.2s | $0.0164 | 93% | 490 | 4.7 |-| Codex app server | 63/63 | 25.0s | $0.0278 | 86% | 382 | 4.3 |+| graff (ADR 0231 build) | 63/63 | 22.2s | $0.0131 | 90% | 348 | 3.1 |+| Codex app server | 63/63 | 28.6s | $0.0426 | 86% | 409 | 4.4 |+| pi | 33/33 | 17.0s | $0.0067 | 33% | 107 | 3.2 |By suite:-| Suite | graff v0.0.302.11 | graff main | Codex app server |+| Suite | graff (ADR 0231 build) | Codex app server | pi || --- | ---: | ---: | ---: |-| core (11 coding tasks), wall | 17.3s | 17.5s | 17.6s |-| core (11 coding tasks), cost | $0.0078 | $0.0081 | $0.0148 |+| core (11 coding tasks), wall | 16.0s | 20.4s | 17.0s |+| core (11 coding tasks), cost | $0.0093 | $0.0156 | $0.0067 || core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 |-| mcp (10 MCP tasks), wall | 46.5s | 40.0s | 33.0s |-| mcp (10 MCP tasks), cost | $0.0311 | $0.0256 | $0.0436 |-| mcp (10 MCP tasks), pass | 30/30 | 30/30 | 30/30 |+| mcp (10 MCP tasks), wall | 29.0s | 37.8s | — |+| mcp (10 MCP tasks), cost | $0.0173 | $0.0722 | — |+| mcp (10 MCP tasks), pass | 30/30 | 30/30 | — |-**Against the release.** graff main is 10% faster and 13% cheaper per task than v0.0.302.11. The gain is in the MCP suite (46.5s to 40.0s per task, $0.0311 to $0.0256), most of it on `linear-split` (114.6s to 65.7s), where sub-agents now call the tools their root loaded (ADR 0227, 0228). The core suite is unchanged.+**Against the Codex app server.** graff is 22% faster per task over all 21 tasks (22.2s against 28.6s): 22% on the coding tasks (16.0s against 20.4s) and 23% on the MCP tasks (29.0s against 37.8s). It makes 3.1 model calls per task to the app server's 4.4, and 3.3 to its 5.8 on the MCP tasks, where a single `rlm` script often fetches, reduces and writes the report. It costs 69% less per task on the app server's own reported usage ($0.0131 against $0.0426). Every arm passed every run.-**Against the Codex app server.** graff main ties on the core suite (17.5s against 17.6s) and is slower on the MCP suite (40.0s against 33.0s). Per MCP task it makes 6.1 model calls to the app server's 5.8 and writes 789 output tokens to its 575; `linear-sidecar` shows it most, 6 to 7 calls against 4. Over all 21 tasks the app server takes 11% less time per task (25.0s against 28.2s).+**Against pi.** On the coding tasks graff is 6% faster (16.0s against 17.0s). pi's system prompt is about a tenth of graff's, so it sends far fewer tokens and costs less per coding task ($0.0067 against $0.0093); it reads little from cache.-graff main costs 41% less per task on the app server's own reported usage ($0.0164 against $0.0278), and 43% to 69% less once the app server's unreported warm-up request is counted (see Caveats). graff reads more of its input from cache (93% against 86%); part of that gap is the same warm-up, which caches the app server's first real request without appearing in its usage.--Every arm passed all 63 runs.+**Against the release.** Earlier the same afternoon, in a run interleaved the same way, v0.0.302.12 took 27.1s per task and the app server 27.4s; on the MCP tasks the release was slower (38.1s against 36.6s, 5.9 calls to 5.8). The ADR 0231 build is 18% faster than the release (22.2s against 27.1s) and 23% cheaper ($0.0131 against $0.0171). That run is [`release-baseline.json`](release-baseline.json).## Per taskWall time is the mean of 3 runs.-| Task | Suite | graff v0.0.302.11 | graff main | Codex app server |+| Task | Suite | graff (ADR 0231 build) | Codex app server | pi || --- | --- | ---: | ---: | ---: |-| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 16.8s | 15.2s | 22.1s |-| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 8.9s | 9.0s | 18.9s |-| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 4.4s | 4.3s | 6.4s |-| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 22.4s | 21.2s | 11.8s |-| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 20.9s | 20.5s | 21.6s |-| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 24.4s | 32.7s | 17.9s |-| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 20.0s | 20.8s | 20.5s |-| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 33.7s | 36.7s | 27.1s |-| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 42.6s | 43.7s | 31.1s |-| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 43.3s | 39.5s | 31.1s |-| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 35.5s | 33.8s | 29.6s |-| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 34.0s | 33.6s | 29.1s |-| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 38.2s | 36.9s | 34.6s |-| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 39.5s | 34.5s | 36.2s |-| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 49.0s | 46.8s | 27.1s |-| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 114.6s | 65.7s | 58.0s |-| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 35.0s | 28.9s | 26.2s |-| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.2s | 7.4s | 10.2s |-| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 24.3s | 20.7s | 26.9s |-| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 18.8s | 19.1s | 14.1s |-| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 22.0s | 21.6s | 23.6s |+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 16.0s | 21.5s | 13.8s |+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 10.6s | 21.0s | 18.2s |+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 5.0s | 8.3s | 8.5s |+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 14.5s | 13.1s | 23.0s |+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 21.4s | 23.5s | 22.3s |+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 21.5s | 20.4s | 15.5s |+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 16.8s | 30.6s | 15.2s |+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 22.8s | 26.8s | — |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 30.7s | 28.5s | — |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 27.8s | 34.2s | — |+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 22.0s | 36.6s | — |+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 24.0s | 33.4s | — |+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 22.5s | 39.2s | — |+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 26.6s | 37.7s | — |+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 38.3s | 35.4s | — |+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 52.5s | 66.4s | — |+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 23.0s | 39.4s | — |+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 9.9s | 10.5s | 10.3s |+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 23.0s | 29.0s | 23.4s |+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 17.2s | 18.0s | 13.3s |+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 20.4s | 28.1s | 23.8s |`results.json` has every run: pass, wall time, model calls, input, cached input and output tokens, and list cost.## Method-- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The fixture declares its two list tools read-only (`readOnlyHint`), as the hosted server it imitates does. Each task passes or fails on its own check.-- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 8 runs at a time. Each arm made one unscored request first, so no arm started with a prompt cache the others had warmed.+- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The fixture declares its two list tools read-only (`readOnlyHint`). Each task passes or fails on its own check.+- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 8 runs at a time; the MCP suite ran first, then the coding suite with pi added. Each arm made one unscored request first, so no arm started with a prompt cache the others had warmed.- **Wall time.** The runner's clock around each process, start to exit.-- **Cost.** List price of the tokens each harness reports: $2 per 1M input, $0.10 per 1M cached input and $10 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). A ChatGPT plan is not billed per token; this makes the two harnesses comparable, not a bill.+- **Cost.** List price of the tokens each harness reports: $2 per 1M input, $0.10 per 1M cached input and $10 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). The app server's three `linear-split` runs total more than 272k input tokens, the model's higher-price threshold, across 16 to 18 requests averaging 22k to 25k each; no single request comes near it, so they are priced at the standard rates. A ChatGPT plan is not billed per token; this makes the harnesses comparable, not a bill.## Caveats-- **Codex's reported usage leaves out one request.** Before a thread's first turn the app server sends a warm-up request (`generate: false`) with its instructions and tools, about 12k tokens. Its usage is not reported, so the Codex cost above is a floor. Counting it adds between about $0.001 per task (fully cached) and $0.025 (uncached).-- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in either harness.-- **`file-ops` has a newline in its prompt.** The scripted repl sends each line as a turn, so graff runs that task as two turns and the app server as one.-- **Small sample.** 3 runs per task, one machine, one account, one 12-minute run. A few seconds per task is within run-to-run noise.+- **Codex's reported usage leaves out one request.** Before a thread's first turn the app server sends a warm-up request (`generate: false`) with its instructions and tools, about 12k tokens. Its usage is not reported, so the Codex cost above is a floor.+- **pi runs only the coding tasks.** It has no MCP client.+- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in every harness.+- **Small sample.** 3 runs per task, one machine, one account. A few seconds per task is within run-to-run noise.## Reproduce```shcd graff-evals-# graff-codex-repl runs {repo}/zig-out/bin/graff; --binary points it elsewhere.-# The app server needs `codex` on PATH and CODEX_HOME at a signed-in Codex home.-# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py-# copied to <parent of output-root>/scripts/ for the mcp suite.-python3 run.py --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol \- --suite core,mcp --reps 3 -j 8 --interleave --output-root /path/outside/repo/run \- --task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \+# graff-repl runs {repo}/zig-out/bin/graff (it needs the piped paste of ADR 0231);+# --binary points it elsewhere. The app server needs `codex` on PATH and CODEX_HOME at a+# signed-in Codex home. pi-codex needs pi signed in on the ChatGPT plan, with gpt-6.1-sol+# declared under providers.openai-codex.models in ~/.pi/agent/models.json.+# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py copied to+# <parent of output-root>/scripts/ for the mcp suite.+TASKS="--task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \--task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \--task dead-code --task linear-issues --task linear-warm --task linear-sidecar \--task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \- --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm"+python3 run.py --harness graff-repl,codex-app-server --suite mcp $TASKS \+ --reps 3 -j 8 --interleave --output-root /path/outside/repo/mcp+python3 run.py --harness graff-repl,codex-app-server,pi-codex --suite core $TASKS \+ --reps 3 -j 8 --interleave --output-root /path/outside/repo/core```diff --git a/evals/graff-vs-codex-app-server/release-baseline.json b/evals/graff-vs-codex-app-server/release-baseline.jsonnew file mode 100644index 00000000..4468b982--- /dev/null+++ b/evals/graff-vs-codex-app-server/release-baseline.json@@ -0,0 +1,2196 @@+{+ "model": "gpt-6.1-sol",+ "tasks": 21,+ "reps": 3,+ "arms": {+ "graff-v302-12": "graff v0.0.302.12 (release)",+ "codex-app-server": "Codex app server"+ },+ "summary": {+ "all": {+ "graff-v302-12": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 27.1,+ "list_usd_per_task": 0.0171,+ "cached_input_share": 0.915,+ "output_tokens_per_task": 431,+ "model_calls_per_task": 4.5+ },+ "codex-app-server": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 27.4,+ "list_usd_per_task": 0.0408,+ "cached_input_share": 0.866,+ "output_tokens_per_task": 401,+ "model_calls_per_task": 4.3+ }+ },+ "core": {+ "graff-v302-12": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 17.1,+ "list_usd_per_task": 0.009,+ "cached_input_share": 0.942,+ "output_tokens_per_task": 186,+ "model_calls_per_task": 3.2+ },+ "codex-app-server": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 19.1,+ "list_usd_per_task": 0.0142,+ "cached_input_share": 0.908,+ "output_tokens_per_task": 206,+ "model_calls_per_task": 3.1+ }+ },+ "mcp": {+ "graff-v302-12": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 38.1,+ "list_usd_per_task": 0.0261,+ "cached_input_share": 0.9,+ "output_tokens_per_task": 701,+ "model_calls_per_task": 5.9+ },+ "codex-app-server": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 36.6,+ "list_usd_per_task": 0.0701,+ "cached_input_share": 0.849,+ "output_tokens_per_task": 616,+ "model_calls_per_task": 5.8+ }+ }+ },+ "per_task": {+ "csv-sum": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 16.8,+ "list_usd": 0.0071,+ "model_calls": 3+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.2,+ "list_usd": 0.0167,+ "model_calls": 3.3+ }+ },+ "dead-code": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.2,+ "list_usd": 0.0085,+ "model_calls": 2+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.6,+ "list_usd": 0.0143,+ "model_calls": 4+ }+ },+ "exact-reply": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.3,+ "list_usd": 0.0039,+ "model_calls": 1+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.3,+ "list_usd": 0.0052,+ "model_calls": 1+ }+ },+ "file-ops": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 25.8,+ "list_usd": 0.0101,+ "model_calls": 5+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 15.3,+ "list_usd": 0.0093,+ "model_calls": 2+ }+ },+ "fix-fib": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.7,+ "list_usd": 0.0137,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 23.0,+ "list_usd": 0.0185,+ "model_calls": 4+ }+ },+ "git-ops": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.4,+ "list_usd": 0.0108,+ "model_calls": 3+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.4,+ "list_usd": 0.0146,+ "model_calls": 3+ }+ },+ "json-transform": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.8,+ "list_usd": 0.0092,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.5,+ "list_usd": 0.0168,+ "model_calls": 4+ }+ },+ "linear-issues": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 33.2,+ "list_usd": 0.0235,+ "model_calls": 4.7+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 30.9,+ "list_usd": 0.0404,+ "model_calls": 4.3+ }+ },+ "linear-nohint": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 45.7,+ "list_usd": 0.0246,+ "model_calls": 6.3+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 36.1,+ "list_usd": 0.0596,+ "model_calls": 4.3+ }+ },+ "linear-nohint-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 43.1,+ "list_usd": 0.0283,+ "model_calls": 6.3+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 28.1,+ "list_usd": 0.0726,+ "model_calls": 4+ }+ },+ "linear-quiet": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 25.5,+ "list_usd": 0.0137,+ "model_calls": 3+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 36.2,+ "list_usd": 0.0351,+ "model_calls": 4.3+ }+ },+ "linear-quiet-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 32.7,+ "list_usd": 0.0204,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 28.5,+ "list_usd": 0.0212,+ "model_calls": 4+ }+ },+ "linear-reduce": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 30.3,+ "list_usd": 0.0168,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 30.0,+ "list_usd": 0.0239,+ "model_calls": 4.3+ }+ },+ "linear-reduce-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 30.4,+ "list_usd": 0.0181,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 40.8,+ "list_usd": 0.042,+ "model_calls": 5.3+ }+ },+ "linear-sidecar": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 44.4,+ "list_usd": 0.0258,+ "model_calls": 6+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 31.7,+ "list_usd": 0.0533,+ "model_calls": 4.7+ }+ },+ "linear-split": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 65.2,+ "list_usd": 0.072,+ "model_calls": 17+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 67.2,+ "list_usd": 0.3055,+ "model_calls": 17.7+ }+ },+ "linear-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 30.4,+ "list_usd": 0.0182,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 36.9,+ "list_usd": 0.047,+ "model_calls": 4.7+ }+ },+ "recall-noise": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.2,+ "list_usd": 0.0031,+ "model_calls": 2+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.0,+ "list_usd": 0.0091,+ "model_calls": 2+ }+ },+ "refactor-rename": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.7,+ "list_usd": 0.0101,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 28.4,+ "list_usd": 0.0239,+ "model_calls": 4+ }+ },+ "regex-count": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.0,+ "list_usd": 0.0102,+ "model_calls": 3.3+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 15.7,+ "list_usd": 0.0094,+ "model_calls": 2.3+ }+ },+ "write-tests": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.7,+ "list_usd": 0.0118,+ "model_calls": 4+ },+ "codex-app-server": {+ "passed": 3,+ "runs": 3,+ "wall_s": 27.4,+ "list_usd": 0.0189,+ "model_calls": 4+ }+ }+ },+ "runs": [+ {+ "arm": "codex-app-server",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.12,+ "model_calls": 4,+ "input_tokens": 57898,+ "cached_input_tokens": 52992,+ "output_tokens": 250,+ "list_usd": 0.017611+ },+ {+ "arm": "codex-app-server",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.1,+ "model_calls": 3,+ "input_tokens": 43326,+ "cached_input_tokens": 38656,+ "output_tokens": 311,+ "list_usd": 0.016316+ },+ {+ "arm": "codex-app-server",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.53,+ "model_calls": 3,+ "input_tokens": 43430,+ "cached_input_tokens": 38656,+ "output_tokens": 267,+ "list_usd": 0.016084+ },+ {+ "arm": "codex-app-server",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.14,+ "model_calls": 4,+ "input_tokens": 58049,+ "cached_input_tokens": 53120,+ "output_tokens": 157,+ "list_usd": 0.01674+ },+ {+ "arm": "codex-app-server",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.52,+ "model_calls": 4,+ "input_tokens": 58063,+ "cached_input_tokens": 55040,+ "output_tokens": 159,+ "list_usd": 0.01314+ },+ {+ "arm": "codex-app-server",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.22,+ "model_calls": 4,+ "input_tokens": 58064,+ "cached_input_tokens": 55040,+ "output_tokens": 151,+ "list_usd": 0.013062+ },+ {+ "arm": "codex-app-server",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.44,+ "model_calls": 1,+ "input_tokens": 14231,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005165+ },+ {+ "arm": "codex-app-server",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.35,+ "model_calls": 1,+ "input_tokens": 14231,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005165+ },+ {+ "arm": "codex-app-server",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.08,+ "model_calls": 1,+ "input_tokens": 14231,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005165+ },+ {+ "arm": "codex-app-server",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.77,+ "model_calls": 2,+ "input_tokens": 28642,+ "cached_input_tokens": 26368,+ "output_tokens": 97,+ "list_usd": 0.008155+ },+ {+ "arm": "codex-app-server",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.59,+ "model_calls": 2,+ "input_tokens": 28639,+ "cached_input_tokens": 26368,+ "output_tokens": 104,+ "list_usd": 0.008219+ },+ {+ "arm": "codex-app-server",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.43,+ "model_calls": 2,+ "input_tokens": 28645,+ "cached_input_tokens": 24576,+ "output_tokens": 106,+ "list_usd": 0.011656+ },+ {+ "arm": "codex-app-server",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.5,+ "model_calls": 4,+ "input_tokens": 58544,+ "cached_input_tokens": 53504,+ "output_tokens": 278,+ "list_usd": 0.01821+ },+ {+ "arm": "codex-app-server",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.73,+ "model_calls": 4,+ "input_tokens": 58552,+ "cached_input_tokens": 52992,+ "output_tokens": 275,+ "list_usd": 0.019169+ },+ {+ "arm": "codex-app-server",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.9,+ "model_calls": 4,+ "input_tokens": 58570,+ "cached_input_tokens": 53632,+ "output_tokens": 285,+ "list_usd": 0.018089+ },+ {+ "arm": "codex-app-server",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.61,+ "model_calls": 3,+ "input_tokens": 43583,+ "cached_input_tokens": 38656,+ "output_tokens": 206,+ "list_usd": 0.01578+ },+ {+ "arm": "codex-app-server",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.0,+ "model_calls": 3,+ "input_tokens": 43683,+ "cached_input_tokens": 38912,+ "output_tokens": 239,+ "list_usd": 0.015823+ },+ {+ "arm": "codex-app-server",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.54,+ "model_calls": 3,+ "input_tokens": 43552,+ "cached_input_tokens": 40704,+ "output_tokens": 230,+ "list_usd": 0.012066+ },+ {+ "arm": "codex-app-server",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.3,+ "model_calls": 4,+ "input_tokens": 57990,+ "cached_input_tokens": 52992,+ "output_tokens": 266,+ "list_usd": 0.017955+ },+ {+ "arm": "codex-app-server",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.99,+ "model_calls": 4,+ "input_tokens": 57987,+ "cached_input_tokens": 52864,+ "output_tokens": 278,+ "list_usd": 0.018312+ },+ {+ "arm": "codex-app-server",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.32,+ "model_calls": 4,+ "input_tokens": 57978,+ "cached_input_tokens": 54912,+ "output_tokens": 252,+ "list_usd": 0.014143+ },+ {+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.56,+ "model_calls": 4,+ "input_tokens": 89389,+ "cached_input_tokens": 75648,+ "output_tokens": 544,+ "list_usd": 0.040487+ },+ {+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.36,+ "model_calls": 4,+ "input_tokens": 78740,+ "cached_input_tokens": 66176,+ "output_tokens": 410,+ "list_usd": 0.035846+ },+ {+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.73,+ "model_calls": 5,+ "input_tokens": 108142,+ "cached_input_tokens": 93312,+ "output_tokens": 576,+ "list_usd": 0.044751+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 45.77,+ "model_calls": 4,+ "input_tokens": 104971,+ "cached_input_tokens": 83456,+ "output_tokens": 492,+ "list_usd": 0.056296+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.82,+ "model_calls": 4,+ "input_tokens": 105723,+ "cached_input_tokens": 82176,+ "output_tokens": 466,+ "list_usd": 0.059972+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 33.6,+ "model_calls": 5,+ "input_tokens": 135068,+ "cached_input_tokens": 112256,+ "output_tokens": 581,+ "list_usd": 0.06266+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.41,+ "model_calls": 4,+ "input_tokens": 105522,+ "cached_input_tokens": 71552,+ "output_tokens": 440,+ "list_usd": 0.079495+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.44,+ "model_calls": 4,+ "input_tokens": 104968,+ "cached_input_tokens": 81664,+ "output_tokens": 455,+ "list_usd": 0.059324+ },+ {+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.4,+ "model_calls": 4,+ "input_tokens": 104946,+ "cached_input_tokens": 71296,+ "output_tokens": 442,+ "list_usd": 0.07885+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.78,+ "model_calls": 3,+ "input_tokens": 63893,+ "cached_input_tokens": 50816,+ "output_tokens": 471,+ "list_usd": 0.035946+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 45.12,+ "model_calls": 5,+ "input_tokens": 84578,+ "cached_input_tokens": 71296,+ "output_tokens": 572,+ "list_usd": 0.039414+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 33.66,+ "model_calls": 5,+ "input_tokens": 79561,+ "cached_input_tokens": 70656,+ "output_tokens": 507,+ "list_usd": 0.029946+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.38,+ "model_calls": 4,+ "input_tokens": 62546,+ "cached_input_tokens": 56064,+ "output_tokens": 449,+ "list_usd": 0.02306+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.9,+ "model_calls": 4,+ "input_tokens": 63311,+ "cached_input_tokens": 58496,+ "output_tokens": 522,+ "list_usd": 0.0207+ },+ {+ "arm": "codex-app-server",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.15,+ "model_calls": 4,+ "input_tokens": 62599,+ "cached_input_tokens": 57984,+ "output_tokens": 485,+ "list_usd": 0.019878+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 31.93,+ "model_calls": 5,+ "input_tokens": 80670,+ "cached_input_tokens": 75264,+ "output_tokens": 539,+ "list_usd": 0.023728+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.77,+ "model_calls": 4,+ "input_tokens": 63279,+ "cached_input_tokens": 58368,+ "output_tokens": 488,+ "list_usd": 0.020539+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.37,+ "model_calls": 4,+ "input_tokens": 63280,+ "cached_input_tokens": 54656,+ "output_tokens": 480,+ "list_usd": 0.027514+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 42.59,+ "model_calls": 6,+ "input_tokens": 133335,+ "cached_input_tokens": 116608,+ "output_tokens": 666,+ "list_usd": 0.051775+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.59,+ "model_calls": 6,+ "input_tokens": 133104,+ "cached_input_tokens": 116480,+ "output_tokens": 658,+ "list_usd": 0.051476+ },+ {+ "arm": "codex-app-server",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.21,+ "model_calls": 4,+ "input_tokens": 66953,+ "cached_input_tokens": 60928,+ "output_tokens": 452,+ "list_usd": 0.022663+ },+ {+ "arm": "codex-app-server",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.41,+ "model_calls": 5,+ "input_tokens": 139070,+ "cached_input_tokens": 117120,+ "output_tokens": 571,+ "list_usd": 0.061322+ },+ {+ "arm": "codex-app-server",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 31.89,+ "model_calls": 4,+ "input_tokens": 77986,+ "cached_input_tokens": 63744,+ "output_tokens": 440,+ "list_usd": 0.039258+ },+ {+ "arm": "codex-app-server",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.69,+ "model_calls": 5,+ "input_tokens": 138841,+ "cached_input_tokens": 117120,+ "output_tokens": 426,+ "list_usd": 0.059414+ },+ {+ "arm": "codex-app-server",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 70.35,+ "model_calls": 18,+ "input_tokens": 411883,+ "cached_input_tokens": 360960,+ "output_tokens": 1618,+ "list_usd": 0.300154+ },+ {+ "arm": "codex-app-server",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 66.0,+ "model_calls": 17,+ "input_tokens": 373663,+ "cached_input_tokens": 338816,+ "output_tokens": 1561,+ "list_usd": 0.230566+ },+ {+ "arm": "codex-app-server",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 65.11,+ "model_calls": 18,+ "input_tokens": 460984,+ "cached_input_tokens": 390144,+ "output_tokens": 1630,+ "list_usd": 0.385839+ },+ {+ "arm": "codex-app-server",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.54,+ "model_calls": 4,+ "input_tokens": 78747,+ "cached_input_tokens": 66176,+ "output_tokens": 419,+ "list_usd": 0.03595+ },+ {+ "arm": "codex-app-server",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 49.04,+ "model_calls": 5,+ "input_tokens": 105075,+ "cached_input_tokens": 89472,+ "output_tokens": 595,+ "list_usd": 0.046103+ },+ {+ "arm": "codex-app-server",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.07,+ "model_calls": 5,+ "input_tokens": 129028,+ "cached_input_tokens": 107520,+ "output_tokens": 526,+ "list_usd": 0.059028+ },+ {+ "arm": "codex-app-server",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.16,+ "model_calls": 2,+ "input_tokens": 28796,+ "cached_input_tokens": 26368,+ "output_tokens": 38,+ "list_usd": 0.007873+ },+ {+ "arm": "codex-app-server",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.74,+ "model_calls": 2,+ "input_tokens": 28801,+ "cached_input_tokens": 24576,+ "output_tokens": 52,+ "list_usd": 0.011428+ },+ {+ "arm": "codex-app-server",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.0,+ "model_calls": 2,+ "input_tokens": 28805,+ "cached_input_tokens": 26368,+ "output_tokens": 52,+ "list_usd": 0.008031+ },+ {+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.18,+ "model_calls": 4,+ "input_tokens": 58438,+ "cached_input_tokens": 50944,+ "output_tokens": 379,+ "list_usd": 0.023872+ },+ {+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.04,+ "model_calls": 4,+ "input_tokens": 58840,+ "cached_input_tokens": 51584,+ "output_tokens": 440,+ "list_usd": 0.02407+ },+ {+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.98,+ "model_calls": 4,+ "input_tokens": 58534,+ "cached_input_tokens": 51328,+ "output_tokens": 425,+ "list_usd": 0.023795+ },+ {+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.01,+ "model_calls": 2,+ "input_tokens": 28673,+ "cached_input_tokens": 26368,+ "output_tokens": 101,+ "list_usd": 0.008257+ },+ {+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 13.66,+ "model_calls": 2,+ "input_tokens": 28700,+ "cached_input_tokens": 26368,+ "output_tokens": 130,+ "list_usd": 0.008601+ },+ {+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.42,+ "model_calls": 3,+ "input_tokens": 43278,+ "cached_input_tokens": 40576,+ "output_tokens": 190,+ "list_usd": 0.011362+ },+ {+ "arm": "codex-app-server",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.47,+ "model_calls": 4,+ "input_tokens": 58002,+ "cached_input_tokens": 52864,+ "output_tokens": 357,+ "list_usd": 0.019132+ },+ {+ "arm": "codex-app-server",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.91,+ "model_calls": 4,+ "input_tokens": 57968,+ "cached_input_tokens": 52992,+ "output_tokens": 331,+ "list_usd": 0.018561+ },+ {+ "arm": "codex-app-server",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 31.78,+ "model_calls": 4,+ "input_tokens": 58016,+ "cached_input_tokens": 52992,+ "output_tokens": 378,+ "list_usd": 0.019127+ },+ {+ "arm": "graff-v302-12",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.29,+ "model_calls": 3,+ "input_tokens": 31280,+ "cached_input_tokens": 30080,+ "output_tokens": 216,+ "list_usd": 0.007568+ },+ {+ "arm": "graff-v302-12",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.07,+ "model_calls": 3,+ "input_tokens": 31248,+ "cached_input_tokens": 30464,+ "output_tokens": 215,+ "list_usd": 0.006764+ },+ {+ "arm": "graff-v302-12",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.0,+ "model_calls": 3,+ "input_tokens": 31276,+ "cached_input_tokens": 30336,+ "output_tokens": 214,+ "list_usd": 0.007054+ },+ {+ "arm": "graff-v302-12",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.07,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 16512,+ "output_tokens": 56,+ "list_usd": 0.010809+ },+ {+ "arm": "graff-v302-12",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.72,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 16512,+ "output_tokens": 56,+ "list_usd": 0.010809+ },+ {+ "arm": "graff-v302-12",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.71,+ "model_calls": 2,+ "input_tokens": 20811,+ "cached_input_tokens": 20224,+ "output_tokens": 70,+ "list_usd": 0.003896+ },+ {+ "arm": "graff-v302-12",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.33,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 9984,+ "output_tokens": 5,+ "list_usd": 0.001622+ },+ {+ "arm": "graff-v302-12",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.41,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 9984,+ "output_tokens": 5,+ "list_usd": 0.001622+ },+ {+ "arm": "graff-v302-12",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.29,+ "model_calls": 1,+ "input_tokens": 10271,+ "cached_input_tokens": 6400,+ "output_tokens": 5,+ "list_usd": 0.008432+ },+ {+ "arm": "graff-v302-12",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.59,+ "model_calls": 5,+ "input_tokens": 51995,+ "cached_input_tokens": 50432,+ "output_tokens": 219,+ "list_usd": 0.010359+ },+ {+ "arm": "graff-v302-12",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.41,+ "model_calls": 5,+ "input_tokens": 51990,+ "cached_input_tokens": 50816,+ "output_tokens": 221,+ "list_usd": 0.00964+ },+ {+ "arm": "graff-v302-12",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.29,+ "model_calls": 5,+ "input_tokens": 52018,+ "cached_input_tokens": 50688,+ "output_tokens": 249,+ "list_usd": 0.010219+ },+ {+ "arm": "graff-v302-12",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.32,+ "model_calls": 4,+ "input_tokens": 42502,+ "cached_input_tokens": 40064,+ "output_tokens": 251,+ "list_usd": 0.011392+ },+ {+ "arm": "graff-v302-12",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.3,+ "model_calls": 4,+ "input_tokens": 42509,+ "cached_input_tokens": 36352,+ "output_tokens": 258,+ "list_usd": 0.018529+ },+ {+ "arm": "graff-v302-12",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.52,+ "model_calls": 4,+ "input_tokens": 42513,+ "cached_input_tokens": 40192,+ "output_tokens": 257,+ "list_usd": 0.011231+ },+ {+ "arm": "graff-v302-12",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.32,+ "model_calls": 3,+ "input_tokens": 31616,+ "cached_input_tokens": 30336,+ "output_tokens": 233,+ "list_usd": 0.007924+ },+ {+ "arm": "graff-v302-12",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.51,+ "model_calls": 3,+ "input_tokens": 31626,+ "cached_input_tokens": 26624,+ "output_tokens": 249,+ "list_usd": 0.015156+ },+ {+ "arm": "graff-v302-12",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.35,+ "model_calls": 3,+ "input_tokens": 31749,+ "cached_input_tokens": 30080,+ "output_tokens": 300,+ "list_usd": 0.009346+ },+ {+ "arm": "graff-v302-12",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.18,+ "model_calls": 4,+ "input_tokens": 41886,+ "cached_input_tokens": 40576,+ "output_tokens": 262,+ "list_usd": 0.009298+ },+ {+ "arm": "graff-v302-12",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.82,+ "model_calls": 4,+ "input_tokens": 41856,+ "cached_input_tokens": 40448,+ "output_tokens": 235,+ "list_usd": 0.009211+ },+ {+ "arm": "graff-v302-12",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.46,+ "model_calls": 4,+ "input_tokens": 41872,+ "cached_input_tokens": 40576,+ "output_tokens": 252,+ "list_usd": 0.00917+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.7,+ "model_calls": 4,+ "input_tokens": 45434,+ "cached_input_tokens": 35072,+ "output_tokens": 501,+ "list_usd": 0.029241+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.83,+ "model_calls": 5,+ "input_tokens": 57527,+ "cached_input_tokens": 51200,+ "output_tokens": 616,+ "list_usd": 0.023934+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 34.19,+ "model_calls": 5,+ "input_tokens": 57109,+ "cached_input_tokens": 53632,+ "output_tokens": 515,+ "list_usd": 0.017467+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.62,+ "model_calls": 6,+ "input_tokens": 67899,+ "cached_input_tokens": 61568,+ "output_tokens": 734,+ "list_usd": 0.026159+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 44.91,+ "model_calls": 6,+ "input_tokens": 67896,+ "cached_input_tokens": 63616,+ "output_tokens": 753,+ "list_usd": 0.022452+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 52.49,+ "model_calls": 7,+ "input_tokens": 80116,+ "cached_input_tokens": 75648,+ "output_tokens": 872,+ "list_usd": 0.025221+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.11,+ "model_calls": 6,+ "input_tokens": 69188,+ "cached_input_tokens": 63488,+ "output_tokens": 737,+ "list_usd": 0.025119+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 51.97,+ "model_calls": 7,+ "input_tokens": 81844,+ "cached_input_tokens": 74496,+ "output_tokens": 922,+ "list_usd": 0.031366+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.11,+ "model_calls": 6,+ "input_tokens": 69081,+ "cached_input_tokens": 61568,+ "output_tokens": 727,+ "list_usd": 0.028453+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.03,+ "model_calls": 3,+ "input_tokens": 33479,+ "cached_input_tokens": 30720,+ "output_tokens": 476,+ "list_usd": 0.01335+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.63,+ "model_calls": 3,+ "input_tokens": 33533,+ "cached_input_tokens": 30720,+ "output_tokens": 507,+ "list_usd": 0.013768+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.88,+ "model_calls": 3,+ "input_tokens": 33552,+ "cached_input_tokens": 30720,+ "output_tokens": 513,+ "list_usd": 0.013866+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 38.22,+ "model_calls": 4,+ "input_tokens": 46459,+ "cached_input_tokens": 40960,+ "output_tokens": 828,+ "list_usd": 0.023374+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.54,+ "model_calls": 4,+ "input_tokens": 46362,+ "cached_input_tokens": 42496,+ "output_tokens": 529,+ "list_usd": 0.017272+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.31,+ "model_calls": 4,+ "input_tokens": 46392,+ "cached_input_tokens": 40960,+ "output_tokens": 548,+ "list_usd": 0.02044+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 28.84,+ "model_calls": 4,+ "input_tokens": 45083,+ "cached_input_tokens": 40960,+ "output_tokens": 541,+ "list_usd": 0.017752+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.24,+ "model_calls": 4,+ "input_tokens": 45068,+ "cached_input_tokens": 41984,+ "output_tokens": 529,+ "list_usd": 0.015656+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.86,+ "model_calls": 4,+ "input_tokens": 45092,+ "cached_input_tokens": 41472,+ "output_tokens": 553,+ "list_usd": 0.016917+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 31.87,+ "model_calls": 4,+ "input_tokens": 45837,+ "cached_input_tokens": 40832,+ "output_tokens": 519,+ "list_usd": 0.019283+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.62,+ "model_calls": 4,+ "input_tokens": 45815,+ "cached_input_tokens": 40960,+ "output_tokens": 499,+ "list_usd": 0.018796+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.61,+ "model_calls": 4,+ "input_tokens": 45813,+ "cached_input_tokens": 42240,+ "output_tokens": 492,+ "list_usd": 0.01629+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.34,+ "model_calls": 6,+ "input_tokens": 68055,+ "cached_input_tokens": 61056,+ "output_tokens": 732,+ "list_usd": 0.027424+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 44.16,+ "model_calls": 6,+ "input_tokens": 68136,+ "cached_input_tokens": 63104,+ "output_tokens": 766,+ "list_usd": 0.024034+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 48.84,+ "model_calls": 6,+ "input_tokens": 68077,+ "cached_input_tokens": 61824,+ "output_tokens": 737,+ "list_usd": 0.026058+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 59.78,+ "model_calls": 16,+ "input_tokens": 161480,+ "cached_input_tokens": 132736,+ "output_tokens": 1380,+ "list_usd": 0.084562+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 65.9,+ "model_calls": 17,+ "input_tokens": 174216,+ "cached_input_tokens": 155392,+ "output_tokens": 1483,+ "list_usd": 0.068017+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 69.82,+ "model_calls": 18,+ "input_tokens": 190135,+ "cached_input_tokens": 175104,+ "output_tokens": 1585,+ "list_usd": 0.063422+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 30.0,+ "model_calls": 4,+ "input_tokens": 46123,+ "cached_input_tokens": 40960,+ "output_tokens": 547,+ "list_usd": 0.019892+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 32.05,+ "model_calls": 4,+ "input_tokens": 46015,+ "cached_input_tokens": 40960,+ "output_tokens": 449,+ "list_usd": 0.018696+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.1,+ "model_calls": 4,+ "input_tokens": 46016,+ "cached_input_tokens": 42368,+ "output_tokens": 442,+ "list_usd": 0.015953+ },+ {+ "arm": "graff-v302-12",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.19,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-v302-12",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.52,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-v302-12",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.82,+ "model_calls": 2,+ "input_tokens": 20859,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003096+ },+ {+ "arm": "graff-v302-12",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.44,+ "model_calls": 4,+ "input_tokens": 42122,+ "cached_input_tokens": 40832,+ "output_tokens": 288,+ "list_usd": 0.009543+ },+ {+ "arm": "graff-v302-12",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.93,+ "model_calls": 4,+ "input_tokens": 42103,+ "cached_input_tokens": 40320,+ "output_tokens": 276,+ "list_usd": 0.010358+ },+ {+ "arm": "graff-v302-12",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.85,+ "model_calls": 4,+ "input_tokens": 42106,+ "cached_input_tokens": 40320,+ "output_tokens": 274,+ "list_usd": 0.010344+ },+ {+ "arm": "graff-v302-12",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.22,+ "model_calls": 3,+ "input_tokens": 31264,+ "cached_input_tokens": 30336,+ "output_tokens": 135,+ "list_usd": 0.00624+ },+ {+ "arm": "graff-v302-12",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.7,+ "model_calls": 4,+ "input_tokens": 42009,+ "cached_input_tokens": 36736,+ "output_tokens": 287,+ "list_usd": 0.01709+ },+ {+ "arm": "graff-v302-12",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.18,+ "model_calls": 3,+ "input_tokens": 31337,+ "cached_input_tokens": 30336,+ "output_tokens": 237,+ "list_usd": 0.007406+ },+ {+ "arm": "graff-v302-12",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.04,+ "model_calls": 4,+ "input_tokens": 41876,+ "cached_input_tokens": 40448,+ "output_tokens": 248,+ "list_usd": 0.009381+ },+ {+ "arm": "graff-v302-12",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.65,+ "model_calls": 4,+ "input_tokens": 41794,+ "cached_input_tokens": 36736,+ "output_tokens": 218,+ "list_usd": 0.01597+ },+ {+ "arm": "graff-v302-12",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.4,+ "model_calls": 4,+ "input_tokens": 41919,+ "cached_input_tokens": 40320,+ "output_tokens": 272,+ "list_usd": 0.00995+ }+ ]+}\ No newline at end of filediff --git a/evals/graff-vs-codex-app-server/results.json b/evals/graff-vs-codex-app-server/results.jsonindex 6d8c1090..a0b10921 100644--- a/evals/graff-vs-codex-app-server/results.json+++ b/evals/graff-vs-codex-app-server/results.json@@ -3,95 +3,86 @@"tasks": 21,"reps": 3,"arms": {- "graff-v302-11": "graff v0.0.302.11 (release)",- "graff-next": "graff main (ADR 0226, 0227, 0228)",- "codex-app-server": "Codex app server"+ "graff": "graff (ADR 0231 build)",+ "codex-app-server": "Codex app server",+ "pi": "pi"},"summary": {"all": {- "graff-v302-11": {+ "graff": {"runs": 63,"passed": 63,- "wall_s_per_task": 31.2,- "list_usd_per_task": 0.0189,- "cached_input_share": 0.922,- "output_tokens_per_task": 598,- "model_calls_per_task": 4.7- },- "graff-next": {- "runs": 63,- "passed": 63,- "wall_s_per_task": 28.2,- "list_usd_per_task": 0.0164,- "cached_input_share": 0.933,- "output_tokens_per_task": 490,- "model_calls_per_task": 4.7+ "wall_s_per_task": 22.2,+ "list_usd_per_task": 0.0131,+ "cached_input_share": 0.904,+ "output_tokens_per_task": 348,+ "model_calls_per_task": 3.1},"codex-app-server": {"runs": 63,"passed": 63,- "wall_s_per_task": 25.0,- "list_usd_per_task": 0.0278,- "cached_input_share": 0.857,- "output_tokens_per_task": 382,- "model_calls_per_task": 4.3+ "wall_s_per_task": 28.6,+ "list_usd_per_task": 0.0426,+ "cached_input_share": 0.855,+ "output_tokens_per_task": 409,+ "model_calls_per_task": 4.4+ },+ "pi": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 17.0,+ "list_usd_per_task": 0.0067,+ "cached_input_share": 0.334,+ "output_tokens_per_task": 107,+ "model_calls_per_task": 3.2}},"core": {- "graff-v302-11": {+ "graff": {"runs": 33,"passed": 33,- "wall_s_per_task": 17.3,- "list_usd_per_task": 0.0078,- "cached_input_share": 0.964,- "output_tokens_per_task": 218,- "model_calls_per_task": 3.2+ "wall_s_per_task": 16.0,+ "list_usd_per_task": 0.0093,+ "cached_input_share": 0.923,+ "output_tokens_per_task": 164,+ "model_calls_per_task": 3.0},- "graff-next": {+ "codex-app-server": {"runs": 33,"passed": 33,- "wall_s_per_task": 17.5,- "list_usd_per_task": 0.0081,- "cached_input_share": 0.964,- "output_tokens_per_task": 219,- "model_calls_per_task": 3.3+ "wall_s_per_task": 20.4,+ "list_usd_per_task": 0.0156,+ "cached_input_share": 0.896,+ "output_tokens_per_task": 209,+ "model_calls_per_task": 3.1},- "codex-app-server": {+ "pi": {"runs": 33,"passed": 33,- "wall_s_per_task": 17.6,- "list_usd_per_task": 0.0148,- "cached_input_share": 0.9,- "output_tokens_per_task": 206,- "model_calls_per_task": 3.0+ "wall_s_per_task": 17.0,+ "list_usd_per_task": 0.0067,+ "cached_input_share": 0.334,+ "output_tokens_per_task": 107,+ "model_calls_per_task": 3.2}},"mcp": {- "graff-v302-11": {- "runs": 30,- "passed": 30,- "wall_s_per_task": 46.5,- "list_usd_per_task": 0.0311,- "cached_input_share": 0.901,- "output_tokens_per_task": 1015,- "model_calls_per_task": 6.3- },- "graff-next": {+ "graff": {"runs": 30,"passed": 30,- "wall_s_per_task": 40.0,- "list_usd_per_task": 0.0256,- "cached_input_share": 0.917,- "output_tokens_per_task": 789,- "model_calls_per_task": 6.1+ "wall_s_per_task": 29.0,+ "list_usd_per_task": 0.0173,+ "cached_input_share": 0.887,+ "output_tokens_per_task": 550,+ "model_calls_per_task": 3.3},"codex-app-server": {"runs": 30,"passed": 30,- "wall_s_per_task": 33.0,- "list_usd_per_task": 0.0436,- "cached_input_share": 0.84,- "output_tokens_per_task": 575,+ "wall_s_per_task": 37.8,+ "list_usd_per_task": 0.0722,+ "cached_input_share": 0.839,+ "output_tokens_per_task": 628,"model_calls_per_task": 5.8}}@@ -99,443 +90,436 @@"per_task": {"csv-sum": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 16.8,- "list_usd": 0.0071+ "wall_s": 16.0,+ "list_usd": 0.0116,+ "model_calls": 3.3},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 15.2,- "list_usd": 0.007+ "wall_s": 21.5,+ "list_usd": 0.018,+ "model_calls": 3.3},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 22.1,- "list_usd": 0.0153+ "wall_s": 13.8,+ "list_usd": 0.0051,+ "model_calls": 3}},"dead-code": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 8.9,- "list_usd": 0.0036+ "wall_s": 10.6,+ "list_usd": 0.0059,+ "model_calls": 2},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 9.0,- "list_usd": 0.0035+ "wall_s": 21.0,+ "list_usd": 0.0171,+ "model_calls": 4},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 18.9,- "list_usd": 0.0146+ "wall_s": 18.2,+ "list_usd": 0.0068,+ "model_calls": 4}},"exact-reply": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 4.4,- "list_usd": 0.0037+ "wall_s": 5.0,+ "list_usd": 0.0014,+ "model_calls": 1},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 4.3,- "list_usd": 0.0014+ "wall_s": 8.3,+ "list_usd": 0.0052,+ "model_calls": 1},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 6.4,- "list_usd": 0.0052+ "wall_s": 8.5,+ "list_usd": 0.0023,+ "model_calls": 1}},"file-ops": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 22.4,- "list_usd": 0.0098+ "wall_s": 14.5,+ "list_usd": 0.0107,+ "model_calls": 3},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 21.2,- "list_usd": 0.0089+ "wall_s": 13.1,+ "list_usd": 0.0116,+ "model_calls": 2},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 11.8,- "list_usd": 0.0105+ "wall_s": 23.0,+ "list_usd": 0.0059,+ "model_calls": 3}},"fix-fib": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 20.9,- "list_usd": 0.0108+ "wall_s": 21.4,+ "list_usd": 0.0143,+ "model_calls": 4},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 20.5,- "list_usd": 0.0106+ "wall_s": 23.5,+ "list_usd": 0.0202,+ "model_calls": 4},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 21.6,- "list_usd": 0.0183+ "wall_s": 22.3,+ "list_usd": 0.0103,+ "model_calls": 5}},"git-ops": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 24.4,- "list_usd": 0.0106+ "wall_s": 21.5,+ "list_usd": 0.0115,+ "model_calls": 3},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 32.7,- "list_usd": 0.0151+ "wall_s": 20.4,+ "list_usd": 0.0159,+ "model_calls": 3},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 17.9,- "list_usd": 0.0166+ "wall_s": 15.5,+ "list_usd": 0.0059,+ "model_calls": 3}},"json-transform": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 20.0,- "list_usd": 0.0095+ "wall_s": 16.8,+ "list_usd": 0.0089,+ "model_calls": 3},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 20.8,- "list_usd": 0.0096+ "wall_s": 30.6,+ "list_usd": 0.0169,+ "model_calls": 4},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 20.5,- "list_usd": 0.0181+ "wall_s": 15.2,+ "list_usd": 0.0063,+ "model_calls": 3}},"linear-issues": {"suite": "mcp",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 33.7,- "list_usd": 0.0206- },- "graff-next": {- "passed": 3,- "runs": 3,- "wall_s": 36.7,- "list_usd": 0.0207+ "wall_s": 22.8,+ "list_usd": 0.0175,+ "model_calls": 2},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 27.1,- "list_usd": 0.0532+ "wall_s": 26.8,+ "list_usd": 0.0436,+ "model_calls": 4}},"linear-nohint": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 42.6,- "list_usd": 0.0244- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 43.7,- "list_usd": 0.0245+ "wall_s": 30.7,+ "list_usd": 0.0196,+ "model_calls": 4},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 31.1,- "list_usd": 0.0524+ "wall_s": 28.5,+ "list_usd": 0.062,+ "model_calls": 4.3}},"linear-nohint-warm": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 43.3,- "list_usd": 0.0233- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 39.5,- "list_usd": 0.0251+ "wall_s": 27.8,+ "list_usd": 0.0173,+ "model_calls": 3},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 31.1,- "list_usd": 0.0583+ "wall_s": 34.2,+ "list_usd": 0.0702,+ "model_calls": 5}},"linear-quiet": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 35.5,- "list_usd": 0.0164- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 33.8,- "list_usd": 0.0163+ "wall_s": 22.0,+ "list_usd": 0.0107,+ "model_calls": 2},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 29.6,- "list_usd": 0.0281+ "wall_s": 36.6,+ "list_usd": 0.0389,+ "model_calls": 4.3}},"linear-quiet-warm": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 34.0,- "list_usd": 0.0173- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 33.6,- "list_usd": 0.0167+ "wall_s": 24.0,+ "list_usd": 0.0132,+ "model_calls": 2.3},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 29.1,- "list_usd": 0.0282+ "wall_s": 33.4,+ "list_usd": 0.0456,+ "model_calls": 4}},"linear-reduce": {"suite": "mcp",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 38.2,- "list_usd": 0.0207- },- "graff-next": {- "passed": 3,- "runs": 3,- "wall_s": 36.9,- "list_usd": 0.0188+ "wall_s": 22.5,+ "list_usd": 0.0108,+ "model_calls": 2},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 34.6,- "list_usd": 0.0489+ "wall_s": 39.2,+ "list_usd": 0.0386,+ "model_calls": 5.3}},"linear-reduce-warm": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 39.5,- "list_usd": 0.0204- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 34.5,- "list_usd": 0.0207+ "wall_s": 26.6,+ "list_usd": 0.0145,+ "model_calls": 3},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 36.2,- "list_usd": 0.033+ "wall_s": 37.7,+ "list_usd": 0.0483,+ "model_calls": 5.3}},"linear-sidecar": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 49.0,- "list_usd": 0.0293- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 46.8,- "list_usd": 0.0251+ "wall_s": 38.3,+ "list_usd": 0.0252,+ "model_calls": 4.3},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 27.1,- "list_usd": 0.0499+ "wall_s": 35.4,+ "list_usd": 0.05,+ "model_calls": 4.7}},"linear-split": {"suite": "mcp",- "graff-v302-11": {- "passed": 3,- "runs": 3,- "wall_s": 114.6,- "list_usd": 0.1186- },- "graff-next": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 65.7,- "list_usd": 0.0705+ "wall_s": 52.5,+ "list_usd": 0.0322,+ "model_calls": 8},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 58.0,- "list_usd": 0+ "wall_s": 66.4,+ "list_usd": 0.2772,+ "model_calls": 17}},"linear-warm": {"suite": "mcp",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 35.0,- "list_usd": 0.0204- },- "graff-next": {- "passed": 3,- "runs": 3,- "wall_s": 28.9,- "list_usd": 0.0178+ "wall_s": 23.0,+ "list_usd": 0.012,+ "model_calls": 2},"codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 26.2,- "list_usd": 0.0404+ "wall_s": 39.4,+ "list_usd": 0.0477,+ "model_calls": 4.3}},"recall-noise": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 7.2,- "list_usd": 0.0033+ "wall_s": 9.9,+ "list_usd": 0.0081,+ "model_calls": 2},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 7.4,- "list_usd": 0.0054+ "wall_s": 10.5,+ "list_usd": 0.0092,+ "model_calls": 2},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 10.2,- "list_usd": 0.0104+ "wall_s": 10.3,+ "list_usd": 0.0053,+ "model_calls": 2}},"refactor-rename": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 24.3,- "list_usd": 0.0099+ "wall_s": 23.0,+ "list_usd": 0.0141,+ "model_calls": 4},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 20.7,- "list_usd": 0.0101+ "wall_s": 29.0,+ "list_usd": 0.0224,+ "model_calls": 4},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 26.9,- "list_usd": 0.0224+ "wall_s": 23.4,+ "list_usd": 0.0132,+ "model_calls": 5}},"regex-count": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 18.8,- "list_usd": 0.0076+ "wall_s": 17.2,+ "list_usd": 0.0071,+ "model_calls": 3.3},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 19.1,- "list_usd": 0.0074+ "wall_s": 18.0,+ "list_usd": 0.0158,+ "model_calls": 3},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 14.1,- "list_usd": 0.0136+ "wall_s": 13.3,+ "list_usd": 0.0033,+ "model_calls": 2}},"write-tests": {"suite": "core",- "graff-v302-11": {+ "graff": {"passed": 3,"runs": 3,- "wall_s": 22.0,- "list_usd": 0.0101+ "wall_s": 20.4,+ "list_usd": 0.0087,+ "model_calls": 4},- "graff-next": {+ "codex-app-server": {"passed": 3,"runs": 3,- "wall_s": 21.6,- "list_usd": 0.0097+ "wall_s": 28.1,+ "list_usd": 0.019,+ "model_calls": 4},- "codex-app-server": {+ "pi": {"passed": 3,"runs": 3,- "wall_s": 23.6,- "list_usd": 0.0177+ "wall_s": 23.8,+ "list_usd": 0.0093,+ "model_calls": 4.7}}},@@ -547,12 +531,12 @@"rep": 1,"pass": true,"timed_out": false,- "wall_s": 20.08,+ "wall_s": 21.1,"model_calls": 3,- "input_tokens": 43326,- "cached_input_tokens": 40576,- "output_tokens": 308,- "list_usd": 0.012638+ "input_tokens": 43494,+ "cached_input_tokens": 36864,+ "output_tokens": 265,+ "list_usd": 0.019596},{"arm": "codex-app-server",@@ -561,12 +545,12 @@"rep": 2,"pass": true,"timed_out": false,- "wall_s": 26.12,+ "wall_s": 22.6,"model_calls": 3,- "input_tokens": 43418,- "cached_input_tokens": 40576,- "output_tokens": 404,- "list_usd": 0.013782+ "input_tokens": 43342,+ "cached_input_tokens": 38784,+ "output_tokens": 326,+ "list_usd": 0.016254},{"arm": "codex-app-server",@@ -575,2616 +559,2196 @@"rep": 3,"pass": true,"timed_out": false,- "wall_s": 19.99,- "model_calls": 3,- "input_tokens": 43332,- "cached_input_tokens": 36864,- "output_tokens": 286,- "list_usd": 0.019482+ "wall_s": 20.91,+ "model_calls": 4,+ "input_tokens": 57931,+ "cached_input_tokens": 52992,+ "output_tokens": 288,+ "list_usd": 0.018057},{- "arm": "graff-next",- "task": "csv-sum",+ "arm": "codex-app-server",+ "task": "dead-code","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 16.06,- "model_calls": 3,- "input_tokens": 31281,- "cached_input_tokens": 30336,- "output_tokens": 216,- "list_usd": 0.007084+ "wall_s": 22.78,+ "model_calls": 4,+ "input_tokens": 57946,+ "cached_input_tokens": 52864,+ "output_tokens": 150,+ "list_usd": 0.01695},{- "arm": "graff-next",- "task": "csv-sum",+ "arm": "codex-app-server",+ "task": "dead-code","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 14.83,- "model_calls": 3,- "input_tokens": 31270,- "cached_input_tokens": 30464,- "output_tokens": 203,- "list_usd": 0.006688+ "wall_s": 21.11,+ "model_calls": 4,+ "input_tokens": 58070,+ "cached_input_tokens": 52864,+ "output_tokens": 150,+ "list_usd": 0.017198},{- "arm": "graff-next",- "task": "csv-sum",+ "arm": "codex-app-server",+ "task": "dead-code","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 14.83,- "model_calls": 3,- "input_tokens": 31286,- "cached_input_tokens": 30336,- "output_tokens": 219,- "list_usd": 0.007124+ "wall_s": 19.15,+ "model_calls": 4,+ "input_tokens": 58043,+ "cached_input_tokens": 52864,+ "output_tokens": 165,+ "list_usd": 0.017294},{- "arm": "graff-v302-11",- "task": "csv-sum",+ "arm": "codex-app-server",+ "task": "exact-reply","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 15.82,- "model_calls": 3,- "input_tokens": 31286,- "cached_input_tokens": 30720,- "output_tokens": 223,- "list_usd": 0.006434+ "wall_s": 6.37,+ "model_calls": 1,+ "input_tokens": 14234,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005171},{- "arm": "graff-v302-11",- "task": "csv-sum",+ "arm": "codex-app-server",+ "task": "exact-reply","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 16.23,- "model_calls": 3,- "input_tokens": 31286,- "cached_input_tokens": 30336,- "output_tokens": 223,- "list_usd": 0.007164+ "wall_s": 9.23,+ "model_calls": 1,+ "input_tokens": 14235,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005173},{- "arm": "graff-v302-11",- "task": "csv-sum",+ "arm": "codex-app-server",+ "task": "exact-reply","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 18.44,- "model_calls": 3,- "input_tokens": 31301,- "cached_input_tokens": 30336,- "output_tokens": 272,- "list_usd": 0.007684+ "wall_s": 9.37,+ "model_calls": 1,+ "input_tokens": 14234,+ "cached_input_tokens": 12288,+ "output_tokens": 5,+ "list_usd": 0.005171},{"arm": "codex-app-server",- "task": "dead-code",+ "task": "file-ops","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 19.97,- "model_calls": 4,- "input_tokens": 58039,- "cached_input_tokens": 55040,- "output_tokens": 156,- "list_usd": 0.013062+ "wall_s": 11.53,+ "model_calls": 2,+ "input_tokens": 28646,+ "cached_input_tokens": 24576,+ "output_tokens": 97,+ "list_usd": 0.011568},{"arm": "codex-app-server",- "task": "dead-code",+ "task": "file-ops","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 16.93,- "model_calls": 4,- "input_tokens": 57949,- "cached_input_tokens": 54912,- "output_tokens": 150,- "list_usd": 0.013065+ "wall_s": 15.1,+ "model_calls": 2,+ "input_tokens": 28646,+ "cached_input_tokens": 24576,+ "output_tokens": 99,+ "list_usd": 0.011588},{"arm": "codex-app-server",- "task": "dead-code",+ "task": "file-ops","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 19.81,- "model_calls": 4,- "input_tokens": 58134,- "cached_input_tokens": 52864,- "output_tokens": 171,- "list_usd": 0.017536+ "wall_s": 12.7,+ "model_calls": 2,+ "input_tokens": 28646,+ "cached_input_tokens": 24576,+ "output_tokens": 101,+ "list_usd": 0.011608},{- "arm": "graff-next",- "task": "dead-code",+ "arm": "codex-app-server",+ "task": "fix-fib","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 9.81,- "model_calls": 2,- "input_tokens": 20811,- "cached_input_tokens": 20352,- "output_tokens": 56,- "list_usd": 0.003513+ "wall_s": 22.94,+ "model_calls": 4,+ "input_tokens": 58587,+ "cached_input_tokens": 53120,+ "output_tokens": 284,+ "list_usd": 0.019086},{- "arm": "graff-next",- "task": "dead-code",+ "arm": "codex-app-server",+ "task": "fix-fib","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 8.12,- "model_calls": 2,- "input_tokens": 20811,- "cached_input_tokens": 20352,- "output_tokens": 56,- "list_usd": 0.003513+ "wall_s": 23.54,+ "model_calls": 4,+ "input_tokens": 58579,+ "cached_input_tokens": 53120,+ "output_tokens": 285,+ "list_usd": 0.01908},{- "arm": "graff-next",- "task": "dead-code",+ "arm": "codex-app-server",+ "task": "fix-fib","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 8.99,- "model_calls": 2,- "input_tokens": 20811,- "cached_input_tokens": 20352,- "output_tokens": 56,- "list_usd": 0.003513+ "wall_s": 24.12,+ "model_calls": 4,+ "input_tokens": 58566,+ "cached_input_tokens": 51328,+ "output_tokens": 279,+ "list_usd": 0.022399},{- "arm": "graff-v302-11",- "task": "dead-code",+ "arm": "codex-app-server",+ "task": "git-ops","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 9.63,- "model_calls": 2,- "input_tokens": 20811,- "cached_input_tokens": 20352,- "output_tokens": 70,- "list_usd": 0.003653+ "wall_s": 22.45,+ "model_calls": 3,+ "input_tokens": 43610,+ "cached_input_tokens": 38656,+ "output_tokens": 218,+ "list_usd": 0.015954},{- "arm": "graff-v302-11",- "task": "dead-code",+ "arm": "codex-app-server",+ "task": "git-ops","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 7.58,- "model_calls": 2,- "input_tokens": 20811,- "cached_input_tokens": 20352,- "output_tokens": 56,- "list_usd": 0.003513+ "wall_s": 17.47,+ "model_calls": 3,+ "input_tokens": 43653,+ "cached_input_tokens": 38656,+ "output_tokens": 215,+ "list_usd": 0.01601},{- "arm": "graff-v302-11",- "task": "dead-code",+ "arm": "codex-app-server",+ "task": "git-ops","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 9.56,- "model_calls": 2,- "input_tokens": 20811,- "cached_input_tokens": 20352,- "output_tokens": 56,- "list_usd": 0.003513+ "wall_s": 21.17,+ "model_calls": 3,+ "input_tokens": 43532,+ "cached_input_tokens": 38656,+ "output_tokens": 221,+ "list_usd": 0.015828},{"arm": "codex-app-server",- "task": "exact-reply",+ "task": "json-transform","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 6.6,- "model_calls": 1,- "input_tokens": 14234,- "cached_input_tokens": 12288,- "output_tokens": 5,- "list_usd": 0.005171+ "wall_s": 28.19,+ "model_calls": 4,+ "input_tokens": 58004,+ "cached_input_tokens": 52992,+ "output_tokens": 252,+ "list_usd": 0.017843},{"arm": "codex-app-server",- "task": "exact-reply",+ "task": "json-transform","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 6.4,- "model_calls": 1,- "input_tokens": 14233,- "cached_input_tokens": 12288,- "output_tokens": 5,- "list_usd": 0.005169+ "wall_s": 40.09,+ "model_calls": 4,+ "input_tokens": 58013,+ "cached_input_tokens": 54912,+ "output_tokens": 287,+ "list_usd": 0.014563},{"arm": "codex-app-server",- "task": "exact-reply",+ "task": "json-transform","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 6.12,- "model_calls": 1,- "input_tokens": 14233,- "cached_input_tokens": 12288,- "output_tokens": 5,- "list_usd": 0.005169+ "wall_s": 23.55,+ "model_calls": 4,+ "input_tokens": 58008,+ "cached_input_tokens": 52864,+ "output_tokens": 274,+ "list_usd": 0.018314},{- "arm": "graff-next",- "task": "exact-reply",- "suite": "core",+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 4.46,- "model_calls": 1,- "input_tokens": 10271,- "cached_input_tokens": 10112,- "output_tokens": 5,- "list_usd": 0.001379+ "wall_s": 26.65,+ "model_calls": 4,+ "input_tokens": 78780,+ "cached_input_tokens": 66176,+ "output_tokens": 428,+ "list_usd": 0.036106},{- "arm": "graff-next",- "task": "exact-reply",- "suite": "core",+ "arm": "codex-app-server",+ "task": "linear-issues",+ "suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 5.19,- "model_calls": 1,- "input_tokens": 10271,- "cached_input_tokens": 10112,- "output_tokens": 5,- "list_usd": 0.001379- },- {- "arm": "graff-next",- "task": "exact-reply",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 3.32,- "model_calls": 1,- "input_tokens": 10271,- "cached_input_tokens": 10112,- "output_tokens": 5,- "list_usd": 0.001379- },- {- "arm": "graff-v302-11",- "task": "exact-reply",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 5.29,- "model_calls": 1,- "input_tokens": 10271,- "cached_input_tokens": 6400,- "output_tokens": 5,- "list_usd": 0.008432- },- {- "arm": "graff-v302-11",- "task": "exact-reply",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 3.54,- "model_calls": 1,- "input_tokens": 10271,- "cached_input_tokens": 10112,- "output_tokens": 5,- "list_usd": 0.001379- },- {- "arm": "graff-v302-11",- "task": "exact-reply",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 4.28,- "model_calls": 1,- "input_tokens": 10271,- "cached_input_tokens": 10112,- "output_tokens": 5,- "list_usd": 0.001379+ "wall_s": 28.72,+ "model_calls": 4,+ "input_tokens": 78019,+ "cached_input_tokens": 60416,+ "output_tokens": 448,+ "list_usd": 0.045728},{"arm": "codex-app-server",- "task": "file-ops",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 10.73,- "model_calls": 2,- "input_tokens": 28647,- "cached_input_tokens": 24576,- "output_tokens": 99,- "list_usd": 0.01159- },- {- "arm": "codex-app-server",- "task": "file-ops",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 11.56,- "model_calls": 2,- "input_tokens": 28653,- "cached_input_tokens": 24576,- "output_tokens": 104,- "list_usd": 0.011652- },- {- "arm": "codex-app-server",- "task": "file-ops",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 13.23,- "model_calls": 2,- "input_tokens": 28649,- "cached_input_tokens": 26368,- "output_tokens": 104,- "list_usd": 0.008239- },- {- "arm": "graff-next",- "task": "file-ops",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 24.16,- "model_calls": 5,- "input_tokens": 51990,- "cached_input_tokens": 50688,- "output_tokens": 219,- "list_usd": 0.009863- },- {- "arm": "graff-next",- "task": "file-ops",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 21.6,- "model_calls": 5,- "input_tokens": 51990,- "cached_input_tokens": 50944,- "output_tokens": 217,- "list_usd": 0.009356- },- {- "arm": "graff-next",- "task": "file-ops",- "suite": "core",+ "task": "linear-issues",+ "suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 17.95,+ "wall_s": 25.05,"model_calls": 4,- "input_tokens": 41443,- "cached_input_tokens": 40576,- "output_tokens": 158,- "list_usd": 0.007372- },- {- "arm": "graff-v302-11",- "task": "file-ops",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 17.16,- "model_calls": 4,- "input_tokens": 41391,- "cached_input_tokens": 40192,- "output_tokens": 125,- "list_usd": 0.007667- },- {- "arm": "graff-v302-11",- "task": "file-ops",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 24.82,- "model_calls": 5,- "input_tokens": 52064,- "cached_input_tokens": 50560,- "output_tokens": 288,- "list_usd": 0.010944- },- {- "arm": "graff-v302-11",- "task": "file-ops",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 25.23,- "model_calls": 5,- "input_tokens": 52073,- "cached_input_tokens": 50688,- "output_tokens": 299,- "list_usd": 0.010829+ "input_tokens": 91803,+ "cached_input_tokens": 72960,+ "output_tokens": 402,+ "list_usd": 0.049002},{"arm": "codex-app-server",- "task": "fix-fib",- "suite": "core",+ "task": "linear-nohint",+ "suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 19.37,- "model_calls": 3,- "input_tokens": 43831,- "cached_input_tokens": 38656,- "output_tokens": 259,- "list_usd": 0.016806- },- {- "arm": "codex-app-server",- "task": "fix-fib",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 20.93,+ "wall_s": 27.89,"model_calls": 4,- "input_tokens": 58587,- "cached_input_tokens": 53120,- "output_tokens": 287,- "list_usd": 0.019116+ "input_tokens": 104953,+ "cached_input_tokens": 81664,+ "output_tokens": 462,+ "list_usd": 0.059364},{"arm": "codex-app-server",- "task": "fix-fib",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 24.64,- "model_calls": 4,- "input_tokens": 58583,- "cached_input_tokens": 53248,- "output_tokens": 286,- "list_usd": 0.018855- },- {- "arm": "graff-next",- "task": "fix-fib",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 21.24,- "model_calls": 4,- "input_tokens": 42525,- "cached_input_tokens": 40448,- "output_tokens": 261,- "list_usd": 0.010809- },- {- "arm": "graff-next",- "task": "fix-fib",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 20.3,- "model_calls": 4,- "input_tokens": 42277,- "cached_input_tokens": 40704,- "output_tokens": 247,- "list_usd": 0.009686- },- {- "arm": "graff-next",- "task": "fix-fib",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 20.05,- "model_calls": 4,- "input_tokens": 42561,- "cached_input_tokens": 40448,- "output_tokens": 289,- "list_usd": 0.011161- },- {- "arm": "graff-v302-11",- "task": "fix-fib",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 21.05,- "model_calls": 4,- "input_tokens": 42551,- "cached_input_tokens": 40832,- "output_tokens": 283,- "list_usd": 0.010351- },- {- "arm": "graff-v302-11",- "task": "fix-fib",- "suite": "core",+ "task": "linear-nohint",+ "suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 20.92,- "model_calls": 4,- "input_tokens": 42547,- "cached_input_tokens": 40448,- "output_tokens": 277,- "list_usd": 0.011013+ "wall_s": 31.32,+ "model_calls": 5,+ "input_tokens": 138111,+ "cached_input_tokens": 116480,+ "output_tokens": 456,+ "list_usd": 0.05947},{- "arm": "graff-v302-11",- "task": "fix-fib",- "suite": "core",+ "arm": "codex-app-server",+ "task": "linear-nohint",+ "suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 20.69,+ "wall_s": 26.24,"model_calls": 4,- "input_tokens": 42549,- "cached_input_tokens": 40448,- "output_tokens": 276,- "list_usd": 0.011007+ "input_tokens": 105177,+ "cached_input_tokens": 77696,+ "output_tokens": 442,+ "list_usd": 0.067152},{"arm": "codex-app-server",- "task": "git-ops",- "suite": "core",+ "task": "linear-nohint-warm",+ "suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 18.11,- "model_calls": 3,- "input_tokens": 43523,- "cached_input_tokens": 38912,- "output_tokens": 206,- "list_usd": 0.015173+ "wall_s": 30.78,+ "model_calls": 4,+ "input_tokens": 104995,+ "cached_input_tokens": 83584,+ "output_tokens": 471,+ "list_usd": 0.05589},{"arm": "codex-app-server",- "task": "git-ops",- "suite": "core",+ "task": "linear-nohint-warm",+ "suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 17.36,- "model_calls": 3,- "input_tokens": 43519,- "cached_input_tokens": 38656,- "output_tokens": 203,- "list_usd": 0.015622- },- {- "arm": "codex-app-server",- "task": "git-ops",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 18.25,- "model_calls": 3,- "input_tokens": 43551,- "cached_input_tokens": 36864,- "output_tokens": 193,- "list_usd": 0.01899- },- {- "arm": "graff-next",- "task": "git-ops",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 38.24,+ "wall_s": 38.38,"model_calls": 6,- "input_tokens": 64159,- "cached_input_tokens": 61056,- "output_tokens": 573,- "list_usd": 0.018042+ "input_tokens": 167525,+ "cached_input_tokens": 130304,+ "output_tokens": 612,+ "list_usd": 0.093592},{- "arm": "graff-next",- "task": "git-ops",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 24.31,- "model_calls": 3,- "input_tokens": 31714,- "cached_input_tokens": 30336,- "output_tokens": 344,- "list_usd": 0.00923- },- {- "arm": "graff-next",- "task": "git-ops",- "suite": "core",+ "arm": "codex-app-server",+ "task": "linear-nohint-warm",+ "suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 35.56,+ "wall_s": 33.44,"model_calls": 5,- "input_tokens": 53790,- "cached_input_tokens": 50560,- "output_tokens": 648,- "list_usd": 0.017996- },- {- "arm": "graff-v302-11",- "task": "git-ops",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 24.04,- "model_calls": 3,- "input_tokens": 31814,- "cached_input_tokens": 30336,- "output_tokens": 448,- "list_usd": 0.01047- },- {- "arm": "graff-v302-11",- "task": "git-ops",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 29.79,- "model_calls": 4,- "input_tokens": 42655,- "cached_input_tokens": 40832,+ "input_tokens": 134923,+ "cached_input_tokens": 112256,"output_tokens": 467,- "list_usd": 0.012399- },- {- "arm": "graff-v302-11",- "task": "git-ops",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 19.45,- "model_calls": 3,- "input_tokens": 31706,- "cached_input_tokens": 30336,- "output_tokens": 316,- "list_usd": 0.008934+ "list_usd": 0.06123},{"arm": "codex-app-server",- "task": "json-transform",- "suite": "core",+ "task": "linear-quiet",+ "suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 19.71,- "model_calls": 4,- "input_tokens": 57997,- "cached_input_tokens": 52864,- "output_tokens": 277,- "list_usd": 0.018322- },- {- "arm": "codex-app-server",- "task": "json-transform",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 20.08,- "model_calls": 4,- "input_tokens": 58013,- "cached_input_tokens": 52992,- "output_tokens": 275,- "list_usd": 0.018091+ "wall_s": 45.11,+ "model_calls": 5,+ "input_tokens": 107148,+ "cached_input_tokens": 92800,+ "output_tokens": 671,+ "list_usd": 0.044686},{"arm": "codex-app-server",- "task": "json-transform",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 21.67,- "model_calls": 4,- "input_tokens": 57990,- "cached_input_tokens": 52992,- "output_tokens": 247,- "list_usd": 0.017765- },- {- "arm": "graff-next",- "task": "json-transform",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 19.79,- "model_calls": 4,- "input_tokens": 41937,- "cached_input_tokens": 40704,- "output_tokens": 271,- "list_usd": 0.009246- },- {- "arm": "graff-next",- "task": "json-transform",- "suite": "core",- "rep": 2,- "pass": true,- "timed_out": false,- "wall_s": 21.3,- "model_calls": 4,- "input_tokens": 41975,- "cached_input_tokens": 40704,- "output_tokens": 295,- "list_usd": 0.009562- },- {- "arm": "graff-next",- "task": "json-transform",- "suite": "core",- "rep": 3,- "pass": true,- "timed_out": false,- "wall_s": 21.35,- "model_calls": 4,- "input_tokens": 41976,- "cached_input_tokens": 40448,- "output_tokens": 294,- "list_usd": 0.010041- },- {- "arm": "graff-v302-11",- "task": "json-transform",- "suite": "core",- "rep": 1,- "pass": true,- "timed_out": false,- "wall_s": 19.67,- "model_calls": 4,- "input_tokens": 41919,- "cached_input_tokens": 40576,- "output_tokens": 290,- "list_usd": 0.009644- },- {- "arm": "graff-v302-11",- "task": "json-transform",- "suite": "core",+ "task": "linear-quiet",+ "suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 20.69,+ "wall_s": 29.19,"model_calls": 4,- "input_tokens": 41889,- "cached_input_tokens": 40576,- "output_tokens": 270,- "list_usd": 0.009384+ "input_tokens": 63325,+ "cached_input_tokens": 52736,+ "output_tokens": 494,+ "list_usd": 0.031392},{- "arm": "graff-v302-11",- "task": "json-transform",- "suite": "core",+ "arm": "codex-app-server",+ "task": "linear-quiet",+ "suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 19.75,+ "wall_s": 35.47,"model_calls": 4,- "input_tokens": 41907,- "cached_input_tokens": 40576,- "output_tokens": 280,- "list_usd": 0.00952+ "input_tokens": 81625,+ "cached_input_tokens": 67584,+ "output_tokens": 575,+ "list_usd": 0.04059},{"arm": "codex-app-server",- "task": "linear-issues",+ "task": "linear-quiet-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 26.29,+ "wall_s": 27.33,"model_calls": 4,- "input_tokens": 82398,- "cached_input_tokens": 66688,- "output_tokens": 388,- "list_usd": 0.041969+ "input_tokens": 63408,+ "cached_input_tokens": 52992,+ "output_tokens": 466,+ "list_usd": 0.030791},{"arm": "codex-app-server",- "task": "linear-issues",+ "task": "linear-quiet-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 29.73,- "model_calls": 5,- "input_tokens": 131306,- "cached_input_tokens": 109312,- "output_tokens": 506,- "list_usd": 0.059979+ "wall_s": 35.83,+ "model_calls": 4,+ "input_tokens": 81586,+ "cached_input_tokens": 54656,+ "output_tokens": 547,+ "list_usd": 0.064796},{"arm": "codex-app-server",- "task": "linear-issues",+ "task": "linear-quiet-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 25.31,+ "wall_s": 37.12,"model_calls": 4,- "input_tokens": 78755,- "cached_input_tokens": 54656,- "output_tokens": 410,- "list_usd": 0.057764+ "input_tokens": 81885,+ "cached_input_tokens": 67712,+ "output_tokens": 612,+ "list_usd": 0.041237},{- "arm": "graff-next",- "task": "linear-issues",+ "arm": "codex-app-server",+ "task": "linear-reduce","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 37.44,- "model_calls": 5,- "input_tokens": 57133,- "cached_input_tokens": 54144,- "output_tokens": 599,- "list_usd": 0.017382+ "wall_s": 30.24,+ "model_calls": 4,+ "input_tokens": 63323,+ "cached_input_tokens": 54656,+ "output_tokens": 519,+ "list_usd": 0.02799},{- "arm": "graff-next",- "task": "linear-issues",+ "arm": "codex-app-server",+ "task": "linear-reduce","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 35.36,- "model_calls": 5,- "input_tokens": 57541,- "cached_input_tokens": 52736,- "output_tokens": 613,- "list_usd": 0.021014+ "wall_s": 48.78,+ "model_calls": 6,+ "input_tokens": 132981,+ "cached_input_tokens": 114944,+ "output_tokens": 674,+ "list_usd": 0.054308},{- "arm": "graff-next",- "task": "linear-issues",+ "arm": "codex-app-server",+ "task": "linear-reduce","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 37.27,- "model_calls": 5,- "input_tokens": 57293,- "cached_input_tokens": 51200,- "output_tokens": 631,- "list_usd": 0.023616+ "wall_s": 38.67,+ "model_calls": 6,+ "input_tokens": 96523,+ "cached_input_tokens": 87424,+ "output_tokens": 655,+ "list_usd": 0.03349},{- "arm": "graff-v302-11",- "task": "linear-issues",+ "arm": "codex-app-server",+ "task": "linear-reduce-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 35.54,- "model_calls": 5,- "input_tokens": 57185,- "cached_input_tokens": 51200,- "output_tokens": 575,- "list_usd": 0.02284+ "wall_s": 45.0,+ "model_calls": 6,+ "input_tokens": 138736,+ "cached_input_tokens": 112384,+ "output_tokens": 658,+ "list_usd": 0.070522},{- "arm": "graff-v302-11",- "task": "linear-issues",+ "arm": "codex-app-server",+ "task": "linear-reduce-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 31.41,- "model_calls": 4,- "input_tokens": 45746,- "cached_input_tokens": 40960,- "output_tokens": 580,- "list_usd": 0.019468+ "wall_s": 41.32,+ "model_calls": 6,+ "input_tokens": 132919,+ "cached_input_tokens": 118400,+ "output_tokens": 649,+ "list_usd": 0.047368},{- "arm": "graff-v302-11",- "task": "linear-issues",+ "arm": "codex-app-server",+ "task": "linear-reduce-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 34.06,+ "wall_s": 26.82,"model_calls": 4,- "input_tokens": 45436,- "cached_input_tokens": 40960,- "output_tokens": 632,- "list_usd": 0.019368+ "input_tokens": 63275,+ "cached_input_tokens": 54656,+ "output_tokens": 436,+ "list_usd": 0.027064},{"arm": "codex-app-server",- "task": "linear-nohint",+ "task": "linear-sidecar","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 27.64,- "model_calls": 4,- "input_tokens": 102531,- "cached_input_tokens": 80128,- "output_tokens": 443,- "list_usd": 0.057249+ "wall_s": 41.51,+ "model_calls": 5,+ "input_tokens": 139088,+ "cached_input_tokens": 115200,+ "output_tokens": 585,+ "list_usd": 0.065146},{"arm": "codex-app-server",- "task": "linear-nohint",+ "task": "linear-sidecar","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 26.92,- "model_calls": 4,- "input_tokens": 105527,- "cached_input_tokens": 83840,- "output_tokens": 442,- "list_usd": 0.056178+ "wall_s": 36.42,+ "model_calls": 5,+ "input_tokens": 96819,+ "cached_input_tokens": 79232,+ "output_tokens": 559,+ "list_usd": 0.048687},{"arm": "codex-app-server",- "task": "linear-nohint",+ "task": "linear-sidecar","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 38.74,- "model_calls": 5,- "input_tokens": 115163,- "cached_input_tokens": 100864,- "output_tokens": 504,- "list_usd": 0.043724+ "wall_s": 28.14,+ "model_calls": 4,+ "input_tokens": 78076,+ "cached_input_tokens": 65664,+ "output_tokens": 474,+ "list_usd": 0.03613},{- "arm": "graff-next",- "task": "linear-nohint",+ "arm": "codex-app-server",+ "task": "linear-split","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 50.7,- "model_calls": 7,- "input_tokens": 80135,- "cached_input_tokens": 75136,- "output_tokens": 873,- "list_usd": 0.026242+ "wall_s": 68.37,+ "model_calls": 17,+ "input_tokens": 376049,+ "cached_input_tokens": 341632,+ "output_tokens": 1479,+ "list_usd": 0.228179},{- "arm": "graff-next",- "task": "linear-nohint",+ "arm": "codex-app-server",+ "task": "linear-split","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 39.2,- "model_calls": 6,- "input_tokens": 67932,- "cached_input_tokens": 64384,- "output_tokens": 752,- "list_usd": 0.021054+ "wall_s": 67.48,+ "model_calls": 16,+ "input_tokens": 353717,+ "cached_input_tokens": 295936,+ "output_tokens": 1460,+ "list_usd": 0.312211},{- "arm": "graff-next",- "task": "linear-nohint",+ "arm": "codex-app-server",+ "task": "linear-split","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 41.31,- "model_calls": 6,- "input_tokens": 67909,- "cached_input_tokens": 61568,- "output_tokens": 739,- "list_usd": 0.026229+ "wall_s": 63.22,+ "model_calls": 18,+ "input_tokens": 440631,+ "cached_input_tokens": 392832,+ "output_tokens": 1423,+ "list_usd": 0.291107},{- "arm": "graff-v302-11",- "task": "linear-nohint",+ "arm": "codex-app-server",+ "task": "linear-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 40.35,- "model_calls": 6,- "input_tokens": 67928,- "cached_input_tokens": 62080,- "output_tokens": 753,- "list_usd": 0.025434+ "wall_s": 46.12,+ "model_calls": 5,+ "input_tokens": 105284,+ "cached_input_tokens": 87936,+ "output_tokens": 580,+ "list_usd": 0.04929},{- "arm": "graff-v302-11",- "task": "linear-nohint",+ "arm": "codex-app-server",+ "task": "linear-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 41.26,- "model_calls": 6,- "input_tokens": 67928,- "cached_input_tokens": 63616,- "output_tokens": 754,- "list_usd": 0.022526+ "wall_s": 33.35,+ "model_calls": 4,+ "input_tokens": 89282,+ "cached_input_tokens": 73600,+ "output_tokens": 569,+ "list_usd": 0.044414},{- "arm": "graff-v302-11",- "task": "linear-nohint",+ "arm": "codex-app-server",+ "task": "linear-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 46.34,- "model_calls": 6,- "input_tokens": 68431,- "cached_input_tokens": 63872,- "output_tokens": 982,- "list_usd": 0.025325+ "wall_s": 38.58,+ "model_calls": 4,+ "input_tokens": 80241,+ "cached_input_tokens": 61312,+ "output_tokens": 553,+ "list_usd": 0.049519},{"arm": "codex-app-server",- "task": "linear-nohint-warm",- "suite": "mcp",+ "task": "recall-noise",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 30.35,- "model_calls": 4,- "input_tokens": 102526,- "cached_input_tokens": 80128,- "output_tokens": 428,- "list_usd": 0.057089+ "wall_s": 10.44,+ "model_calls": 2,+ "input_tokens": 28812,+ "cached_input_tokens": 26368,+ "output_tokens": 52,+ "list_usd": 0.008045},{"arm": "codex-app-server",- "task": "linear-nohint-warm",- "suite": "mcp",+ "task": "recall-noise",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 36.49,- "model_calls": 4,- "input_tokens": 104964,- "cached_input_tokens": 81664,- "output_tokens": 449,- "list_usd": 0.059256+ "wall_s": 9.8,+ "model_calls": 2,+ "input_tokens": 28807,+ "cached_input_tokens": 26368,+ "output_tokens": 51,+ "list_usd": 0.008025},{"arm": "codex-app-server",- "task": "linear-nohint-warm",- "suite": "mcp",+ "task": "recall-noise",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 26.41,- "model_calls": 4,- "input_tokens": 103865,- "cached_input_tokens": 80896,- "output_tokens": 456,- "list_usd": 0.058588+ "wall_s": 11.18,+ "model_calls": 2,+ "input_tokens": 28817,+ "cached_input_tokens": 24576,+ "output_tokens": 57,+ "list_usd": 0.01151},{- "arm": "graff-next",- "task": "linear-nohint-warm",- "suite": "mcp",+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 38.95,- "model_calls": 6,- "input_tokens": 69113,- "cached_input_tokens": 63616,- "output_tokens": 721,- "list_usd": 0.024566+ "wall_s": 26.91,+ "model_calls": 4,+ "input_tokens": 58485,+ "cached_input_tokens": 52992,+ "output_tokens": 429,+ "list_usd": 0.020575},{- "arm": "graff-next",- "task": "linear-nohint-warm",- "suite": "mcp",+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 40.71,- "model_calls": 6,- "input_tokens": 69164,- "cached_input_tokens": 63616,- "output_tokens": 795,- "list_usd": 0.025408+ "wall_s": 32.06,+ "model_calls": 4,+ "input_tokens": 58307,+ "cached_input_tokens": 51328,+ "output_tokens": 332,+ "list_usd": 0.022411},{- "arm": "graff-next",- "task": "linear-nohint-warm",- "suite": "mcp",+ "arm": "codex-app-server",+ "task": "refactor-rename",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 38.69,- "model_calls": 6,- "input_tokens": 69249,- "cached_input_tokens": 63616,- "output_tokens": 767,- "list_usd": 0.025298+ "wall_s": 28.13,+ "model_calls": 4,+ "input_tokens": 58435,+ "cached_input_tokens": 50944,+ "output_tokens": 420,+ "list_usd": 0.024276},{- "arm": "graff-v302-11",- "task": "linear-nohint-warm",- "suite": "mcp",+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 43.51,- "model_calls": 6,- "input_tokens": 69306,- "cached_input_tokens": 63360,- "output_tokens": 796,- "list_usd": 0.026188+ "wall_s": 18.3,+ "model_calls": 3,+ "input_tokens": 43226,+ "cached_input_tokens": 36864,+ "output_tokens": 175,+ "list_usd": 0.01816},{- "arm": "graff-v302-11",- "task": "linear-nohint-warm",- "suite": "mcp",+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 46.2,- "model_calls": 6,- "input_tokens": 69673,- "cached_input_tokens": 65664,- "output_tokens": 923,- "list_usd": 0.023814+ "wall_s": 17.35,+ "model_calls": 3,+ "input_tokens": 43195,+ "cached_input_tokens": 38784,+ "output_tokens": 170,+ "list_usd": 0.0144},{- "arm": "graff-v302-11",- "task": "linear-nohint-warm",- "suite": "mcp",+ "arm": "codex-app-server",+ "task": "regex-count",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 40.09,- "model_calls": 6,- "input_tokens": 69145,- "cached_input_tokens": 66304,- "output_tokens": 760,- "list_usd": 0.019912+ "wall_s": 18.26,+ "model_calls": 3,+ "input_tokens": 43262,+ "cached_input_tokens": 38656,+ "output_tokens": 190,+ "list_usd": 0.014978},{"arm": "codex-app-server",- "task": "linear-quiet",- "suite": "mcp",+ "task": "write-tests",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 28.32,+ "wall_s": 31.21,"model_calls": 4,- "input_tokens": 63308,+ "input_tokens": 58018,"cached_input_tokens": 52864,- "output_tokens": 479,- "list_usd": 0.030964+ "output_tokens": 361,+ "list_usd": 0.019204},{"arm": "codex-app-server",- "task": "linear-quiet",- "suite": "mcp",+ "task": "write-tests",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 32.73,- "model_calls": 5,- "input_tokens": 79606,- "cached_input_tokens": 68864,- "output_tokens": 514,- "list_usd": 0.03351+ "wall_s": 24.56,+ "model_calls": 4,+ "input_tokens": 58006,+ "cached_input_tokens": 52992,+ "output_tokens": 356,+ "list_usd": 0.018887},{"arm": "codex-app-server",- "task": "linear-quiet",- "suite": "mcp",+ "task": "write-tests",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 27.88,+ "wall_s": 28.53,"model_calls": 4,- "input_tokens": 63293,- "cached_input_tokens": 58496,- "output_tokens": 435,- "list_usd": 0.019794+ "input_tokens": 57998,+ "cached_input_tokens": 52864,+ "output_tokens": 349,+ "list_usd": 0.019044},{- "arm": "graff-next",- "task": "linear-quiet",- "suite": "mcp",+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 28.02,+ "wall_s": 13.5,"model_calls": 3,- "input_tokens": 33539,- "cached_input_tokens": 31232,- "output_tokens": 508,- "list_usd": 0.012817+ "input_tokens": 31280,+ "cached_input_tokens": 26624,+ "output_tokens": 107,+ "list_usd": 0.013044},{- "arm": "graff-next",- "task": "linear-quiet",- "suite": "mcp",+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 44.37,- "model_calls": 4,- "input_tokens": 45830,- "cached_input_tokens": 41728,- "output_tokens": 893,- "list_usd": 0.021307+ "wall_s": 14.83,+ "model_calls": 3,+ "input_tokens": 31243,+ "cached_input_tokens": 26624,+ "output_tokens": 101,+ "list_usd": 0.01291},{- "arm": "graff-next",- "task": "linear-quiet",- "suite": "mcp",+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 29.12,- "model_calls": 3,- "input_tokens": 33616,- "cached_input_tokens": 30720,- "output_tokens": 606,- "list_usd": 0.014924+ "wall_s": 19.63,+ "model_calls": 4,+ "input_tokens": 41912,+ "cached_input_tokens": 40448,+ "output_tokens": 186,+ "list_usd": 0.008833},{- "arm": "graff-v302-11",- "task": "linear-quiet",- "suite": "mcp",+ "arm": "graff",+ "task": "dead-code",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 37.88,- "model_calls": 3,- "input_tokens": 33819,- "cached_input_tokens": 30720,- "output_tokens": 764,- "list_usd": 0.01691+ "wall_s": 9.57,+ "model_calls": 2,+ "input_tokens": 20881,+ "cached_input_tokens": 20352,+ "output_tokens": 56,+ "list_usd": 0.003653},{- "arm": "graff-v302-11",- "task": "linear-quiet",- "suite": "mcp",+ "arm": "graff",+ "task": "dead-code",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 39.42,- "model_calls": 3,- "input_tokens": 33831,- "cached_input_tokens": 30720,- "output_tokens": 841,- "list_usd": 0.017704+ "wall_s": 9.3,+ "model_calls": 2,+ "input_tokens": 20881,+ "cached_input_tokens": 16512,+ "output_tokens": 56,+ "list_usd": 0.010949},{- "arm": "graff-v302-11",- "task": "linear-quiet",- "suite": "mcp",+ "arm": "graff",+ "task": "dead-code",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 29.18,- "model_calls": 3,- "input_tokens": 33635,- "cached_input_tokens": 30720,- "output_tokens": 564,- "list_usd": 0.014542+ "wall_s": 12.97,+ "model_calls": 2,+ "input_tokens": 20881,+ "cached_input_tokens": 20608,+ "output_tokens": 56,+ "list_usd": 0.003167},{- "arm": "codex-app-server",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 30.01,- "model_calls": 5,- "input_tokens": 79608,- "cached_input_tokens": 70656,- "output_tokens": 460,- "list_usd": 0.02957+ "wall_s": 4.66,+ "model_calls": 1,+ "input_tokens": 10306,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001449},- {- "arm": "codex-app-server",- "task": "linear-quiet-warm",- "suite": "mcp",+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 30.91,- "model_calls": 5,- "input_tokens": 84598,- "cached_input_tokens": 73344,- "output_tokens": 540,- "list_usd": 0.035242+ "wall_s": 3.89,+ "model_calls": 1,+ "input_tokens": 10306,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001449},{- "arm": "codex-app-server",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 26.39,- "model_calls": 4,- "input_tokens": 63291,- "cached_input_tokens": 58496,- "output_tokens": 443,- "list_usd": 0.01987+ "wall_s": 6.55,+ "model_calls": 1,+ "input_tokens": 10306,+ "cached_input_tokens": 10112,+ "output_tokens": 5,+ "list_usd": 0.001449},{- "arm": "graff-next",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "file-ops",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 35.13,- "model_calls": 4,- "input_tokens": 46171,- "cached_input_tokens": 42496,- "output_tokens": 656,- "list_usd": 0.01816+ "wall_s": 14.98,+ "model_calls": 3,+ "input_tokens": 31156,+ "cached_input_tokens": 26624,+ "output_tokens": 125,+ "list_usd": 0.012976},{- "arm": "graff-next",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "file-ops",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 28.18,+ "wall_s": 15.58,"model_calls": 3,- "input_tokens": 34078,- "cached_input_tokens": 30720,- "output_tokens": 585,- "list_usd": 0.015638+ "input_tokens": 31192,+ "cached_input_tokens": 30336,+ "output_tokens": 135,+ "list_usd": 0.006096},{- "arm": "graff-next",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "file-ops",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 37.51,+ "wall_s": 13.01,"model_calls": 3,- "input_tokens": 34235,- "cached_input_tokens": 31488,- "output_tokens": 764,- "list_usd": 0.016283+ "input_tokens": 31158,+ "cached_input_tokens": 26624,+ "output_tokens": 131,+ "list_usd": 0.01304},{- "arm": "graff-v302-11",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 41.66,+ "wall_s": 22.95,"model_calls": 4,- "input_tokens": 46759,- "cached_input_tokens": 42624,- "output_tokens": 742,- "list_usd": 0.019952+ "input_tokens": 42668,+ "cached_input_tokens": 41344,+ "output_tokens": 260,+ "list_usd": 0.009382},{- "arm": "graff-v302-11",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 32.96,+ "wall_s": 20.02,"model_calls": 4,- "input_tokens": 46403,- "cached_input_tokens": 43264,- "output_tokens": 561,- "list_usd": 0.016214+ "input_tokens": 42674,+ "cached_input_tokens": 41344,+ "output_tokens": 272,+ "list_usd": 0.009514},{- "arm": "graff-v302-11",- "task": "linear-quiet-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 27.51,- "model_calls": 3,- "input_tokens": 34085,- "cached_input_tokens": 30720,- "output_tokens": 581,- "list_usd": 0.015612+ "wall_s": 21.31,+ "model_calls": 4,+ "input_tokens": 42407,+ "cached_input_tokens": 33280,+ "output_tokens": 239,+ "list_usd": 0.023972},{- "arm": "codex-app-server",- "task": "linear-reduce",- "suite": "mcp",+ "arm": "graff",+ "task": "git-ops",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 32.76,- "model_calls": 5,- "input_tokens": 107419,- "cached_input_tokens": 89344,- "output_tokens": 608,- "list_usd": 0.051164+ "wall_s": 21.0,+ "model_calls": 3,+ "input_tokens": 31815,+ "cached_input_tokens": 30464,+ "output_tokens": 326,+ "list_usd": 0.009008},{- "arm": "codex-app-server",- "task": "linear-reduce",- "suite": "mcp",+ "arm": "graff",+ "task": "git-ops",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 33.79,- "model_calls": 5,- "input_tokens": 107311,- "cached_input_tokens": 92928,- "output_tokens": 626,- "list_usd": 0.044319+ "wall_s": 21.12,+ "model_calls": 3,+ "input_tokens": 31791,+ "cached_input_tokens": 30464,+ "output_tokens": 307,+ "list_usd": 0.00877},{- "arm": "codex-app-server",- "task": "linear-reduce",- "suite": "mcp",+ "arm": "graff",+ "task": "git-ops",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 37.32,- "model_calls": 6,- "input_tokens": 133180,- "cached_input_tokens": 116480,- "output_tokens": 626,- "list_usd": 0.051308+ "wall_s": 22.3,+ "model_calls": 3,+ "input_tokens": 31852,+ "cached_input_tokens": 26624,+ "output_tokens": 367,+ "list_usd": 0.016788},{- "arm": "graff-next",- "task": "linear-reduce",- "suite": "mcp",+ "arm": "graff",+ "task": "json-transform",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 37.1,- "model_calls": 5,- "input_tokens": 56886,- "cached_input_tokens": 52224,- "output_tokens": 653,- "list_usd": 0.021076+ "wall_s": 18.13,+ "model_calls": 3,+ "input_tokens": 31395,+ "cached_input_tokens": 30592,+ "output_tokens": 175,+ "list_usd": 0.006415},{- "arm": "graff-next",- "task": "linear-reduce",- "suite": "mcp",+ "arm": "graff",+ "task": "json-transform",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 36.92,- "model_calls": 4,- "input_tokens": 45136,- "cached_input_tokens": 41984,- "output_tokens": 614,- "list_usd": 0.016642+ "wall_s": 16.53,+ "model_calls": 3,+ "input_tokens": 31400,+ "cached_input_tokens": 30592,+ "output_tokens": 180,+ "list_usd": 0.006475},{- "arm": "graff-next",- "task": "linear-reduce",- "suite": "mcp",+ "arm": "graff",+ "task": "json-transform",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 36.74,- "model_calls": 5,- "input_tokens": 56686,- "cached_input_tokens": 53248,- "output_tokens": 645,- "list_usd": 0.018651+ "wall_s": 15.79,+ "model_calls": 3,+ "input_tokens": 31401,+ "cached_input_tokens": 26752,+ "output_tokens": 181,+ "list_usd": 0.013783},{- "arm": "graff-v302-11",- "task": "linear-reduce",+ "arm": "graff",+ "task": "linear-issues","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 29.49,- "model_calls": 4,- "input_tokens": 45075,- "cached_input_tokens": 41984,- "output_tokens": 539,- "list_usd": 0.01577+ "wall_s": 23.02,+ "model_calls": 2,+ "input_tokens": 22866,+ "cached_input_tokens": 20224,+ "output_tokens": 422,+ "list_usd": 0.011526},{- "arm": "graff-v302-11",- "task": "linear-reduce",+ "arm": "graff",+ "task": "linear-issues","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 33.75,- "model_calls": 4,- "input_tokens": 45113,- "cached_input_tokens": 40960,- "output_tokens": 581,- "list_usd": 0.018212+ "wall_s": 23.65,+ "model_calls": 2,+ "input_tokens": 22836,+ "cached_input_tokens": 20736,+ "output_tokens": 434,+ "list_usd": 0.010614},{- "arm": "graff-v302-11",- "task": "linear-reduce",+ "arm": "graff",+ "task": "linear-issues","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 51.42,- "model_calls": 6,- "input_tokens": 69541,- "cached_input_tokens": 63488,- "output_tokens": 966,- "list_usd": 0.028115+ "wall_s": 21.87,+ "model_calls": 2,+ "input_tokens": 22789,+ "cached_input_tokens": 10112,+ "output_tokens": 386,+ "list_usd": 0.030225},{- "arm": "codex-app-server",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-nohint","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 35.38,- "model_calls": 6,- "input_tokens": 96202,- "cached_input_tokens": 86784,- "output_tokens": 590,- "list_usd": 0.033414+ "wall_s": 20.44,+ "model_calls": 2,+ "input_tokens": 22777,+ "cached_input_tokens": 20224,+ "output_tokens": 399,+ "list_usd": 0.011118},{- "arm": "codex-app-server",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-nohint","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 34.92,- "model_calls": 6,- "input_tokens": 96157,- "cached_input_tokens": 87040,- "output_tokens": 549,- "list_usd": 0.032428+ "wall_s": 37.99,+ "model_calls": 5,+ "input_tokens": 58041,+ "cached_input_tokens": 52352,+ "output_tokens": 561,+ "list_usd": 0.022223},{- "arm": "codex-app-server",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-nohint","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 38.31,- "model_calls": 6,- "input_tokens": 96155,- "cached_input_tokens": 86784,- "output_tokens": 568,- "list_usd": 0.0331+ "wall_s": 33.81,+ "model_calls": 5,+ "input_tokens": 58107,+ "cached_input_tokens": 50560,+ "output_tokens": 525,+ "list_usd": 0.0254},{- "arm": "graff-next",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-nohint-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 32.54,- "model_calls": 5,- "input_tokens": 57658,- "cached_input_tokens": 52480,- "output_tokens": 559,- "list_usd": 0.021194+ "wall_s": 34.92,+ "model_calls": 4,+ "input_tokens": 47810,+ "cached_input_tokens": 40448,+ "output_tokens": 611,+ "list_usd": 0.024879},{- "arm": "graff-next",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-nohint-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 38.58,- "model_calls": 5,- "input_tokens": 57972,- "cached_input_tokens": 52480,- "output_tokens": 707,- "list_usd": 0.023302+ "wall_s": 26.5,+ "model_calls": 3,+ "input_tokens": 35402,+ "cached_input_tokens": 31104,+ "output_tokens": 451,+ "list_usd": 0.016216},{- "arm": "graff-next",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-nohint-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 32.26,- "model_calls": 4,- "input_tokens": 45909,- "cached_input_tokens": 42240,- "output_tokens": 594,- "list_usd": 0.017502+ "wall_s": 21.9,+ "model_calls": 2,+ "input_tokens": 23300,+ "cached_input_tokens": 20992,+ "output_tokens": 420,+ "list_usd": 0.010915},{- "arm": "graff-v302-11",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-quiet","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 32.23,- "model_calls": 4,- "input_tokens": 45833,- "cached_input_tokens": 42240,- "output_tokens": 561,- "list_usd": 0.01702+ "wall_s": 22.5,+ "model_calls": 2,+ "input_tokens": 22933,+ "cached_input_tokens": 20736,+ "output_tokens": 421,+ "list_usd": 0.010678},{- "arm": "graff-v302-11",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-quiet","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 40.59,- "model_calls": 5,- "input_tokens": 57909,- "cached_input_tokens": 52608,- "output_tokens": 656,- "list_usd": 0.022423+ "wall_s": 23.89,+ "model_calls": 2,+ "input_tokens": 22916,+ "cached_input_tokens": 20736,+ "output_tokens": 409,+ "list_usd": 0.010524},{- "arm": "graff-v302-11",- "task": "linear-reduce-warm",+ "arm": "graff",+ "task": "linear-quiet","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 45.54,- "model_calls": 6,- "input_tokens": 69948,- "cached_input_tokens": 66176,- "output_tokens": 747,- "list_usd": 0.021632+ "wall_s": 19.72,+ "model_calls": 2,+ "input_tokens": 22815,+ "cached_input_tokens": 20224,+ "output_tokens": 357,+ "list_usd": 0.010774},{- "arm": "codex-app-server",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-quiet-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 27.2,- "model_calls": 4,- "input_tokens": 78016,- "cached_input_tokens": 65664,- "output_tokens": 420,- "list_usd": 0.03547+ "wall_s": 21.03,+ "model_calls": 2,+ "input_tokens": 23413,+ "cached_input_tokens": 20224,+ "output_tokens": 385,+ "list_usd": 0.01225},{- "arm": "codex-app-server",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-quiet-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 27.82,- "model_calls": 4,- "input_tokens": 105531,- "cached_input_tokens": 82048,- "output_tokens": 417,- "list_usd": 0.059341+ "wall_s": 21.77,+ "model_calls": 2,+ "input_tokens": 23480,+ "cached_input_tokens": 20992,+ "output_tokens": 442,+ "list_usd": 0.011495},{- "arm": "codex-app-server",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-quiet-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 26.4,- "model_calls": 4,- "input_tokens": 104169,- "cached_input_tokens": 83072,- "output_tokens": 432,- "list_usd": 0.054821+ "wall_s": 29.15,+ "model_calls": 3,+ "input_tokens": 35478,+ "cached_input_tokens": 31872,+ "output_tokens": 555,+ "list_usd": 0.015949},{- "arm": "graff-next",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-reduce","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 49.54,- "model_calls": 7,- "input_tokens": 80485,- "cached_input_tokens": 76160,- "output_tokens": 967,- "list_usd": 0.025936+ "wall_s": 21.84,+ "model_calls": 2,+ "input_tokens": 22900,+ "cached_input_tokens": 20736,+ "output_tokens": 425,+ "list_usd": 0.010652},{- "arm": "graff-next",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-reduce","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 52.58,- "model_calls": 7,- "input_tokens": 80592,- "cached_input_tokens": 75776,- "output_tokens": 979,- "list_usd": 0.027+ "wall_s": 25.31,+ "model_calls": 2,+ "input_tokens": 23013,+ "cached_input_tokens": 20736,+ "output_tokens": 487,+ "list_usd": 0.011498},{- "arm": "graff-next",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-reduce","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 38.4,- "model_calls": 6,- "input_tokens": 68164,- "cached_input_tokens": 64000,- "output_tokens": 763,- "list_usd": 0.022358+ "wall_s": 20.31,+ "model_calls": 2,+ "input_tokens": 22917,+ "cached_input_tokens": 20736,+ "output_tokens": 395,+ "list_usd": 0.010386},{- "arm": "graff-v302-11",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-reduce-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 47.53,- "model_calls": 6,- "input_tokens": 68696,- "cached_input_tokens": 63360,- "output_tokens": 986,- "list_usd": 0.026868+ "wall_s": 23.4,+ "model_calls": 3,+ "input_tokens": 35190,+ "cached_input_tokens": 31104,+ "output_tokens": 442,+ "list_usd": 0.015702},{- "arm": "graff-v302-11",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-reduce-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 49.91,- "model_calls": 7,- "input_tokens": 80552,- "cached_input_tokens": 73472,- "output_tokens": 1000,- "list_usd": 0.031507+ "wall_s": 31.58,+ "model_calls": 3,+ "input_tokens": 35247,+ "cached_input_tokens": 31872,+ "output_tokens": 491,+ "list_usd": 0.014847},{- "arm": "graff-v302-11",- "task": "linear-sidecar",+ "arm": "graff",+ "task": "linear-reduce-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 49.5,- "model_calls": 7,- "input_tokens": 80632,- "cached_input_tokens": 74368,- "output_tokens": 960,- "list_usd": 0.029565+ "wall_s": 24.86,+ "model_calls": 3,+ "input_tokens": 35204,+ "cached_input_tokens": 32640,+ "output_tokens": 455,+ "list_usd": 0.012942},{- "arm": "codex-app-server",- "task": "linear-split",+ "arm": "graff",+ "task": "linear-sidecar","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 60.23,- "model_calls": 16,- "input_tokens": 366660,- "cached_input_tokens": 309632,- "output_tokens": 1384,- "list_usd": null+ "wall_s": 36.66,+ "model_calls": 4,+ "input_tokens": 50303,+ "cached_input_tokens": 42112,+ "output_tokens": 690,+ "list_usd": 0.027493},{- "arm": "codex-app-server",- "task": "linear-split",+ "arm": "graff",+ "task": "linear-sidecar","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 57.52,- "model_calls": 16,- "input_tokens": 367431,- "cached_input_tokens": 312704,- "output_tokens": 1416,- "list_usd": null+ "wall_s": 41.79,+ "model_calls": 5,+ "input_tokens": 65453,+ "cached_input_tokens": 59392,+ "output_tokens": 853,+ "list_usd": 0.026591},{- "arm": "codex-app-server",- "task": "linear-split",+ "arm": "graff",+ "task": "linear-sidecar","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 56.25,- "model_calls": 18,- "input_tokens": 439318,- "cached_input_tokens": 377344,- "output_tokens": 1495,- "list_usd": null+ "wall_s": 36.38,+ "model_calls": 4,+ "input_tokens": 51164,+ "cached_input_tokens": 45440,+ "output_tokens": 553,+ "list_usd": 0.021522},{- "arm": "graff-next",+ "arm": "graff","task": "linear-split","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 67.91,- "model_calls": 18,- "input_tokens": 190207,- "cached_input_tokens": 167680,- "output_tokens": 1688,- "list_usd": 0.078702+ "wall_s": 46.43,+ "model_calls": 8,+ "input_tokens": 83470,+ "cached_input_tokens": 77312,+ "output_tokens": 1175,+ "list_usd": 0.031797},{- "arm": "graff-next",+ "arm": "graff","task": "linear-split","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 58.66,- "model_calls": 17,- "input_tokens": 176523,- "cached_input_tokens": 160384,- "output_tokens": 1628,- "list_usd": 0.064596+ "wall_s": 63.62,+ "model_calls": 8,+ "input_tokens": 83506,+ "cached_input_tokens": 76672,+ "output_tokens": 1257,+ "list_usd": 0.033905},{- "arm": "graff-next",+ "arm": "graff","task": "linear-split","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 70.64,- "model_calls": 18,- "input_tokens": 190968,- "cached_input_tokens": 174720,- "output_tokens": 1823,- "list_usd": 0.068198+ "wall_s": 47.37,+ "model_calls": 8,+ "input_tokens": 83664,+ "cached_input_tokens": 78336,+ "output_tokens": 1255,+ "list_usd": 0.03104},{- "arm": "graff-v302-11",- "task": "linear-split",+ "arm": "graff",+ "task": "linear-warm","suite": "mcp","rep": 1,"pass": true,"timed_out": false,- "wall_s": 140.58,- "model_calls": 21,- "input_tokens": 256931,- "cached_input_tokens": 213760,- "output_tokens": 4107,- "list_usd": 0.148788+ "wall_s": 22.62,+ "model_calls": 2,+ "input_tokens": 23364,+ "cached_input_tokens": 20224,+ "output_tokens": 418,+ "list_usd": 0.012482},{- "arm": "graff-v302-11",- "task": "linear-split",+ "arm": "graff",+ "task": "linear-warm","suite": "mcp","rep": 2,"pass": true,"timed_out": false,- "wall_s": 107.78,- "model_calls": 21,- "input_tokens": 226902,- "cached_input_tokens": 199808,- "output_tokens": 3823,- "list_usd": 0.112399+ "wall_s": 21.14,+ "model_calls": 2,+ "input_tokens": 23372,+ "cached_input_tokens": 20224,+ "output_tokens": 421,+ "list_usd": 0.012528},{- "arm": "graff-v302-11",- "task": "linear-split",+ "arm": "graff",+ "task": "linear-warm","suite": "mcp","rep": 3,"pass": true,"timed_out": false,- "wall_s": 95.44,- "model_calls": 17,- "input_tokens": 202662,- "cached_input_tokens": 177664,- "output_tokens": 2686,- "list_usd": 0.094622+ "wall_s": 25.22,+ "model_calls": 2,+ "input_tokens": 23334,+ "cached_input_tokens": 20992,+ "output_tokens": 415,+ "list_usd": 0.010933},{- "arm": "codex-app-server",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 26.83,- "model_calls": 4,- "input_tokens": 82420,- "cached_input_tokens": 66688,- "output_tokens": 407,- "list_usd": 0.042203+ "wall_s": 10.96,+ "model_calls": 2,+ "input_tokens": 20929,+ "cached_input_tokens": 20480,+ "output_tokens": 29,+ "list_usd": 0.003236},{- "arm": "codex-app-server",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 27.12,- "model_calls": 4,- "input_tokens": 78746,- "cached_input_tokens": 64256,- "output_tokens": 409,- "list_usd": 0.039496- },- {- "arm": "codex-app-server",- "task": "linear-warm",- "suite": "mcp",+ "wall_s": 8.48,+ "model_calls": 2,+ "input_tokens": 20929,+ "cached_input_tokens": 16768,+ "output_tokens": 29,+ "list_usd": 0.010289+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 24.51,- "model_calls": 4,- "input_tokens": 78756,- "cached_input_tokens": 64256,- "output_tokens": 416,- "list_usd": 0.039586+ "wall_s": 10.2,+ "model_calls": 2,+ "input_tokens": 20929,+ "cached_input_tokens": 16512,+ "output_tokens": 29,+ "list_usd": 0.010775},{- "arm": "graff-next",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 26.57,- "model_calls": 3,- "input_tokens": 33957,- "cached_input_tokens": 30720,- "output_tokens": 513,- "list_usd": 0.014676+ "wall_s": 25.3,+ "model_calls": 4,+ "input_tokens": 42255,+ "cached_input_tokens": 34688,+ "output_tokens": 280,+ "list_usd": 0.021403},{- "arm": "graff-next",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 29.41,+ "wall_s": 22.34,"model_calls": 4,- "input_tokens": 46236,- "cached_input_tokens": 40960,- "output_tokens": 537,- "list_usd": 0.020018+ "input_tokens": 42253,+ "cached_input_tokens": 40448,+ "output_tokens": 280,+ "list_usd": 0.010455},{- "arm": "graff-next",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 30.59,+ "wall_s": 21.46,"model_calls": 4,- "input_tokens": 46376,- "cached_input_tokens": 41728,- "output_tokens": 511,- "list_usd": 0.018579+ "input_tokens": 42240,+ "cached_input_tokens": 40448,+ "output_tokens": 275,+ "list_usd": 0.010379},{- "arm": "graff-v302-11",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "regex-count",+ "suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 29.73,- "model_calls": 4,- "input_tokens": 46192,- "cached_input_tokens": 40960,- "output_tokens": 523,- "list_usd": 0.01979+ "wall_s": 15.54,+ "model_calls": 3,+ "input_tokens": 31356,+ "cached_input_tokens": 30336,+ "output_tokens": 153,+ "list_usd": 0.006604},{- "arm": "graff-v302-11",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "regex-count",+ "suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 35.51,- "model_calls": 5,- "input_tokens": 58221,- "cached_input_tokens": 54016,- "output_tokens": 599,- "list_usd": 0.019802+ "wall_s": 20.09,+ "model_calls": 4,+ "input_tokens": 41931,+ "cached_input_tokens": 40704,+ "output_tokens": 179,+ "list_usd": 0.008314},{- "arm": "graff-v302-11",- "task": "linear-warm",- "suite": "mcp",+ "arm": "graff",+ "task": "regex-count",+ "suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 39.91,- "model_calls": 4,- "input_tokens": 46634,- "cached_input_tokens": 42496,- "output_tokens": 914,- "list_usd": 0.021666+ "wall_s": 15.89,+ "model_calls": 3,+ "input_tokens": 31358,+ "cached_input_tokens": 30464,+ "output_tokens": 154,+ "list_usd": 0.006374},{- "arm": "codex-app-server",- "task": "recall-noise",+ "arm": "graff",+ "task": "write-tests","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 10.81,- "model_calls": 2,- "input_tokens": 28817,- "cached_input_tokens": 24576,- "output_tokens": 57,- "list_usd": 0.01151+ "wall_s": 21.6,+ "model_calls": 4,+ "input_tokens": 42032,+ "cached_input_tokens": 40960,+ "output_tokens": 255,+ "list_usd": 0.00879},{- "arm": "codex-app-server",- "task": "recall-noise",+ "arm": "graff",+ "task": "write-tests","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 10.09,- "model_calls": 2,- "input_tokens": 28814,- "cached_input_tokens": 24576,- "output_tokens": 57,- "list_usd": 0.011504+ "wall_s": 19.43,+ "model_calls": 4,+ "input_tokens": 41997,+ "cached_input_tokens": 40832,+ "output_tokens": 236,+ "list_usd": 0.008773},{- "arm": "codex-app-server",- "task": "recall-noise",+ "arm": "graff",+ "task": "write-tests","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 9.69,- "model_calls": 2,- "input_tokens": 28808,- "cached_input_tokens": 26368,- "output_tokens": 52,- "list_usd": 0.008037+ "wall_s": 20.05,+ "model_calls": 4,+ "input_tokens": 41989,+ "cached_input_tokens": 40960,+ "output_tokens": 233,+ "list_usd": 0.008484},{- "arm": "graff-next",- "task": "recall-noise",+ "arm": "pi",+ "task": "csv-sum","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 7.4,- "model_calls": 2,- "input_tokens": 20859,- "cached_input_tokens": 20480,- "output_tokens": 29,- "list_usd": 0.003096+ "wall_s": 15.0,+ "model_calls": 3,+ "input_tokens": 3573,+ "cached_input_tokens": 1024,+ "output_tokens": 58,+ "list_usd": 0.00578},{- "arm": "graff-next",- "task": "recall-noise",+ "arm": "pi",+ "task": "csv-sum","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 6.69,- "model_calls": 2,- "input_tokens": 20859,- "cached_input_tokens": 16768,- "output_tokens": 29,- "list_usd": 0.010149+ "wall_s": 13.29,+ "model_calls": 3,+ "input_tokens": 3573,+ "cached_input_tokens": 2048,+ "output_tokens": 58,+ "list_usd": 0.003835},{- "arm": "graff-next",- "task": "recall-noise",+ "arm": "pi",+ "task": "csv-sum","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 8.1,- "model_calls": 2,- "input_tokens": 20859,- "cached_input_tokens": 20480,- "output_tokens": 29,- "list_usd": 0.003096+ "wall_s": 13.23,+ "model_calls": 3,+ "input_tokens": 3573,+ "cached_input_tokens": 1024,+ "output_tokens": 58,+ "list_usd": 0.00578},{- "arm": "graff-v302-11",- "task": "recall-noise",+ "arm": "pi",+ "task": "dead-code","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 7.56,- "model_calls": 2,- "input_tokens": 20859,- "cached_input_tokens": 20480,- "output_tokens": 29,- "list_usd": 0.003096+ "wall_s": 18.01,+ "model_calls": 4,+ "input_tokens": 4977,+ "cached_input_tokens": 3200,+ "output_tokens": 73,+ "list_usd": 0.004604},{- "arm": "graff-v302-11",- "task": "recall-noise",+ "arm": "pi",+ "task": "dead-code","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 6.68,- "model_calls": 2,- "input_tokens": 20859,- "cached_input_tokens": 20480,- "output_tokens": 29,- "list_usd": 0.003096+ "wall_s": 16.44,+ "model_calls": 4,+ "input_tokens": 4965,+ "cached_input_tokens": 2048,+ "output_tokens": 69,+ "list_usd": 0.006729},{- "arm": "graff-v302-11",- "task": "recall-noise",+ "arm": "pi",+ "task": "dead-code","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 7.38,- "model_calls": 2,- "input_tokens": 20859,- "cached_input_tokens": 20224,- "output_tokens": 43,- "list_usd": 0.003722+ "wall_s": 20.18,+ "model_calls": 4,+ "input_tokens": 5082,+ "cached_input_tokens": 1024,+ "output_tokens": 93,+ "list_usd": 0.009148},{- "arm": "codex-app-server",- "task": "refactor-rename",+ "arm": "pi",+ "task": "exact-reply","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 28.72,- "model_calls": 4,- "input_tokens": 58425,- "cached_input_tokens": 50944,- "output_tokens": 365,- "list_usd": 0.023706+ "wall_s": 7.38,+ "model_calls": 1,+ "input_tokens": 1115,+ "cached_input_tokens": 0,+ "output_tokens": 5,+ "list_usd": 0.00228},{- "arm": "codex-app-server",- "task": "refactor-rename",+ "arm": "pi",+ "task": "exact-reply","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 26.76,- "model_calls": 4,- "input_tokens": 58340,- "cached_input_tokens": 50944,- "output_tokens": 374,- "list_usd": 0.023626+ "wall_s": 6.55,+ "model_calls": 1,+ "input_tokens": 1115,+ "cached_input_tokens": 0,+ "output_tokens": 5,+ "list_usd": 0.00228},{- "arm": "codex-app-server",- "task": "refactor-rename",+ "arm": "pi",+ "task": "exact-reply","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 25.34,- "model_calls": 4,- "input_tokens": 58471,- "cached_input_tokens": 52992,- "output_tokens": 375,- "list_usd": 0.020007+ "wall_s": 11.46,+ "model_calls": 1,+ "input_tokens": 1115,+ "cached_input_tokens": 0,+ "output_tokens": 5,+ "list_usd": 0.00228},{- "arm": "graff-next",- "task": "refactor-rename",+ "arm": "pi",+ "task": "file-ops","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 20.81,- "model_calls": 4,- "input_tokens": 42164,- "cached_input_tokens": 40576,- "output_tokens": 304,- "list_usd": 0.010274+ "wall_s": 25.16,+ "model_calls": 3,+ "input_tokens": 3540,+ "cached_input_tokens": 2048,+ "output_tokens": 78,+ "list_usd": 0.003969},{- "arm": "graff-next",- "task": "refactor-rename",+ "arm": "pi",+ "task": "file-ops","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 20.8,- "model_calls": 4,- "input_tokens": 42152,- "cached_input_tokens": 40832,- "output_tokens": 310,- "list_usd": 0.009823+ "wall_s": 27.22,+ "model_calls": 3,+ "input_tokens": 3540,+ "cached_input_tokens": 0,+ "output_tokens": 78,+ "list_usd": 0.00786},{- "arm": "graff-next",- "task": "refactor-rename",+ "arm": "pi",+ "task": "file-ops","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 20.5,- "model_calls": 4,- "input_tokens": 42164,- "cached_input_tokens": 40704,- "output_tokens": 316,- "list_usd": 0.01015+ "wall_s": 16.71,+ "model_calls": 3,+ "input_tokens": 3540,+ "cached_input_tokens": 1024,+ "output_tokens": 78,+ "list_usd": 0.005914},{- "arm": "graff-v302-11",- "task": "refactor-rename",+ "arm": "pi",+ "task": "fix-fib","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 21.19,- "model_calls": 4,- "input_tokens": 42142,- "cached_input_tokens": 40704,- "output_tokens": 305,- "list_usd": 0.009996+ "wall_s": 22.03,+ "model_calls": 5,+ "input_tokens": 6612,+ "cached_input_tokens": 2176,+ "output_tokens": 149,+ "list_usd": 0.01058},{- "arm": "graff-v302-11",- "task": "refactor-rename",+ "arm": "pi",+ "task": "fix-fib","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 30.0,- "model_calls": 4,- "input_tokens": 42190,- "cached_input_tokens": 40832,- "output_tokens": 332,- "list_usd": 0.010119+ "wall_s": 22.07,+ "model_calls": 5,+ "input_tokens": 6612,+ "cached_input_tokens": 2304,+ "output_tokens": 149,+ "list_usd": 0.010336},{- "arm": "graff-v302-11",- "task": "refactor-rename",+ "arm": "pi",+ "task": "fix-fib","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 21.71,- "model_calls": 4,- "input_tokens": 42186,- "cached_input_tokens": 41088,- "output_tokens": 327,- "list_usd": 0.009575+ "wall_s": 22.87,+ "model_calls": 5,+ "input_tokens": 6612,+ "cached_input_tokens": 2432,+ "output_tokens": 149,+ "list_usd": 0.010093},{- "arm": "codex-app-server",- "task": "regex-count",+ "arm": "pi",+ "task": "git-ops","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 11.33,- "model_calls": 2,- "input_tokens": 28686,- "cached_input_tokens": 26368,- "output_tokens": 111,- "list_usd": 0.008383+ "wall_s": 14.13,+ "model_calls": 3,+ "input_tokens": 3956,+ "cached_input_tokens": 2176,+ "output_tokens": 137,+ "list_usd": 0.005148},{- "arm": "codex-app-server",- "task": "regex-count",+ "arm": "pi",+ "task": "git-ops","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 16.72,+ "wall_s": 15.88,"model_calls": 3,- "input_tokens": 43218,- "cached_input_tokens": 36864,- "output_tokens": 172,- "list_usd": 0.018114+ "input_tokens": 3971,+ "cached_input_tokens": 2176,+ "output_tokens": 143,+ "list_usd": 0.005238},{- "arm": "codex-app-server",- "task": "regex-count",+ "arm": "pi",+ "task": "git-ops","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 14.27,+ "wall_s": 16.54,"model_calls": 3,- "input_tokens": 43241,- "cached_input_tokens": 38656,- "output_tokens": 128,- "list_usd": 0.014316+ "input_tokens": 3957,+ "cached_input_tokens": 1024,+ "output_tokens": 142,+ "list_usd": 0.007388},{- "arm": "graff-next",- "task": "regex-count",+ "arm": "pi",+ "task": "json-transform","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 20.4,- "model_calls": 4,- "input_tokens": 41918,- "cached_input_tokens": 40704,- "output_tokens": 230,- "list_usd": 0.008798+ "wall_s": 15.03,+ "model_calls": 3,+ "input_tokens": 3663,+ "cached_input_tokens": 1024,+ "output_tokens": 94,+ "list_usd": 0.00632},{- "arm": "graff-next",- "task": "regex-count",+ "arm": "pi",+ "task": "json-transform","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 17.32,- "model_calls": 2,- "input_tokens": 20863,- "cached_input_tokens": 20224,- "output_tokens": 224,- "list_usd": 0.00554+ "wall_s": 15.51,+ "model_calls": 3,+ "input_tokens": 3663,+ "cached_input_tokens": 2048,+ "output_tokens": 94,+ "list_usd": 0.004375},{- "arm": "graff-next",- "task": "regex-count",+ "arm": "pi",+ "task": "json-transform","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 19.68,+ "wall_s": 15.01,"model_calls": 3,- "input_tokens": 31380,- "cached_input_tokens": 30336,- "output_tokens": 280,- "list_usd": 0.007922+ "input_tokens": 3663,+ "cached_input_tokens": 0,+ "output_tokens": 95,+ "list_usd": 0.008276},{- "arm": "graff-v302-11",- "task": "regex-count",+ "arm": "pi",+ "task": "recall-noise","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 17.81,- "model_calls": 3,- "input_tokens": 31390,- "cached_input_tokens": 30208,- "output_tokens": 206,- "list_usd": 0.007445+ "wall_s": 9.56,+ "model_calls": 2,+ "input_tokens": 2499,+ "cached_input_tokens": 0,+ "output_tokens": 28,+ "list_usd": 0.005278},{- "arm": "graff-v302-11",- "task": "regex-count",+ "arm": "pi",+ "task": "recall-noise","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 17.79,- "model_calls": 3,- "input_tokens": 31340,- "cached_input_tokens": 30464,- "output_tokens": 240,- "list_usd": 0.007198+ "wall_s": 9.73,+ "model_calls": 2,+ "input_tokens": 2499,+ "cached_input_tokens": 0,+ "output_tokens": 28,+ "list_usd": 0.005278},{- "arm": "graff-v302-11",- "task": "regex-count",+ "arm": "pi",+ "task": "recall-noise","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 20.94,- "model_calls": 3,- "input_tokens": 31387,- "cached_input_tokens": 30208,- "output_tokens": 287,- "list_usd": 0.008249+ "wall_s": 11.65,+ "model_calls": 2,+ "input_tokens": 2499,+ "cached_input_tokens": 0,+ "output_tokens": 28,+ "list_usd": 0.005278},{- "arm": "codex-app-server",- "task": "write-tests",+ "arm": "pi",+ "task": "refactor-rename","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 22.95,- "model_calls": 4,- "input_tokens": 57983,- "cached_input_tokens": 54912,- "output_tokens": 348,- "list_usd": 0.015113+ "wall_s": 23.26,+ "model_calls": 5,+ "input_tokens": 7583,+ "cached_input_tokens": 2688,+ "output_tokens": 289,+ "list_usd": 0.012949},{- "arm": "codex-app-server",- "task": "write-tests",+ "arm": "pi",+ "task": "refactor-rename","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 22.81,- "model_calls": 4,- "input_tokens": 58008,- "cached_input_tokens": 52992,- "output_tokens": 356,- "list_usd": 0.018891+ "wall_s": 24.13,+ "model_calls": 5,+ "input_tokens": 6947,+ "cached_input_tokens": 1152,+ "output_tokens": 281,+ "list_usd": 0.014515},{- "arm": "codex-app-server",- "task": "write-tests",+ "arm": "pi",+ "task": "refactor-rename","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 25.03,- "model_calls": 4,- "input_tokens": 58015,- "cached_input_tokens": 52864,- "output_tokens": 361,- "list_usd": 0.019198+ "wall_s": 22.81,+ "model_calls": 5,+ "input_tokens": 7008,+ "cached_input_tokens": 2432,+ "output_tokens": 279,+ "list_usd": 0.012185},{- "arm": "graff-next",- "task": "write-tests",+ "arm": "pi",+ "task": "regex-count","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 20.07,- "model_calls": 4,- "input_tokens": 41938,- "cached_input_tokens": 40320,- "output_tokens": 280,- "list_usd": 0.010068+ "wall_s": 11.5,+ "model_calls": 2,+ "input_tokens": 2361,+ "cached_input_tokens": 1024,+ "output_tokens": 55,+ "list_usd": 0.003326},{- "arm": "graff-next",- "task": "write-tests",+ "arm": "pi",+ "task": "regex-count","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 23.39,- "model_calls": 4,- "input_tokens": 41909,- "cached_input_tokens": 40576,- "output_tokens": 269,- "list_usd": 0.009414+ "wall_s": 12.91,+ "model_calls": 2,+ "input_tokens": 2357,+ "cached_input_tokens": 1024,+ "output_tokens": 51,+ "list_usd": 0.003278},{- "arm": "graff-next",- "task": "write-tests",+ "arm": "pi",+ "task": "regex-count","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 21.42,- "model_calls": 4,- "input_tokens": 41933,- "cached_input_tokens": 40576,- "output_tokens": 280,- "list_usd": 0.009572+ "wall_s": 15.61,+ "model_calls": 2,+ "input_tokens": 2357,+ "cached_input_tokens": 1024,+ "output_tokens": 51,+ "list_usd": 0.003278},{- "arm": "graff-v302-11",+ "arm": "pi","task": "write-tests","suite": "core","rep": 1,"pass": true,"timed_out": false,- "wall_s": 23.26,- "model_calls": 4,- "input_tokens": 41950,- "cached_input_tokens": 40192,- "output_tokens": 298,- "list_usd": 0.010515+ "wall_s": 29.83,+ "model_calls": 5,+ "input_tokens": 6414,+ "cached_input_tokens": 2048,+ "output_tokens": 266,+ "list_usd": 0.011597},{- "arm": "graff-v302-11",+ "arm": "pi","task": "write-tests","suite": "core","rep": 2,"pass": true,"timed_out": false,- "wall_s": 22.2,- "model_calls": 4,- "input_tokens": 41960,- "cached_input_tokens": 40320,- "output_tokens": 297,- "list_usd": 0.010282+ "wall_s": 22.82,+ "model_calls": 5,+ "input_tokens": 6278,+ "cached_input_tokens": 3200,+ "output_tokens": 196,+ "list_usd": 0.008436},{- "arm": "graff-v302-11",+ "arm": "pi","task": "write-tests","suite": "core","rep": 3,"pass": true,"timed_out": false,- "wall_s": 20.44,+ "wall_s": 18.68,"model_calls": 4,- "input_tokens": 41912,- "cached_input_tokens": 40576,- "output_tokens": 270,- "list_usd": 0.00943+ "input_tokens": 5003,+ "cached_input_tokens": 2048,+ "output_tokens": 179,+ "list_usd": 0.007905}]-}+}\ No newline at end of filediff --git a/evals/graff-vs-opencode-and-pi/RESULTS.md b/evals/graff-vs-opencode-and-pi/RESULTS.mdnew file mode 100644index 00000000..36ac2cfd--- /dev/null+++ b/evals/graff-vs-opencode-and-pi/RESULTS.md@@ -0,0 +1,99 @@+# graff vs OpenCode 2 and pi on a gateway model++Measured 2026-10-01. One model, served through the Codegraff gateway's chat-completions endpoint, ran the same 21 tasks in each harness, side by side, on one machine and one gateway key.++| Arm | What ran |+| --- | --- |+| graff | The ADR 0231 build as a scripted `graff repl` (the `graff-gateway-repl` harness). |+| OpenCode 2 | `@opencode/cli` 2.0.21: `opencode run --standalone --auto --format json`, through [`graff-evals/opencode2_run.py`](../../graff-evals/opencode2_run.py). |+| pi | pi 0.84.2 (the `pi-codegraff` harness). pi has no MCP client, so it ran the 11 coding tasks only. |++All three send the same model id to the same endpoint, and each runs at its own defaults.++## Results++All 21 tasks (pi: the 11 coding tasks), mean per task over 3 runs of each:++| | Pass | Wall per task | Input read from cache | Output tokens per task | Model calls per task |+| --- | ---: | ---: | ---: | ---: | ---: |+| graff (ADR 0231 build) | 63/63 | 9.1s | 94% | 582 | 3.6 |+| OpenCode 2 | 62/63 | 14.4s | 86% | 980 | 4.2 |+| pi | 33/33 | 5.4s | 78% | 258 | 2.7 |++By suite:++| Suite | graff (ADR 0231 build) | OpenCode 2 | pi |+| --- | ---: | ---: | ---: |+| core (11 coding tasks), wall | 7.3s | 6.9s | 5.4s |+| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 |+| mcp (10 MCP tasks), wall | 11.0s | 22.6s | — |+| mcp (10 MCP tasks), pass | 30/30 | 29/30 | — |++**Against OpenCode 2.** graff is 37% faster per task over all 21 tasks (9.1s against 14.4s). The gap is the MCP tasks, which graff finishes in about half the time (11.0s against 22.6s), with 4.4 model calls to 5.3 and 917 output tokens to 1,655: graff slims large list results and often fetches, reduces and writes the report in one `rlm` script. On the coding tasks OpenCode 2 is slightly faster (6.9s against 7.3s). OpenCode 2 failed one run: on one `linear-split` run it ended while still waiting for its two sub-agents, without writing `report.json`.++**Against pi.** pi is the fastest on the coding tasks (5.4s against graff's 7.3s), with a much smaller prompt and the fewest calls (2.7 per task). It has no MCP client, so it has no MCP result.++graff reads the most of its input from cache (94%, against 86% for OpenCode 2 and 78% for pi).++## Per task++Wall time is the mean of 3 runs.++| Task | Suite | graff (ADR 0231 build) | OpenCode 2 | pi |+| --- | --- | ---: | ---: | ---: |+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 9.5s | 7.8s | 5.0s |+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 6.9s | 6.8s | 6.0s |+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 1.8s | 2.1s | 1.7s |+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 4.2s | 5.9s | 4.6s |+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 8.1s | 7.6s | 7.6s |+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 6.1s | 6.2s | 4.6s |+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 5.5s | 6.0s | 5.6s |+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 7.5s | 30.4s | — |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 10.0s | 20.2s | — |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 9.9s | 20.4s | — |+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 11.3s | 20.2s | — |+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 15.7s | 21.5s | — |+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 9.5s | 20.9s | — |+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 8.8s | 19.2s | — |+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 8.9s | 22.5s | — |+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 18.9s | 30.0s (2/3) | — |+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 9.1s | 20.6s | — |+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.8s | 7.6s | 4.4s |+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 10.3s | 8.6s | 7.8s |+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 14.1s | 8.6s | 6.0s |+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 6.6s | 8.4s | 6.1s |++`results.json` has every run: pass, wall time, model calls, and input, cached input and output tokens.++## Method++- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). OpenCode 2 does not read `.mcp.json`; its wrapper writes the same server into the task's `opencode.json`. Each task passes or fails on its own check.+- **Order.** 3 runs per task and arm. graff and OpenCode 2 were interleaved task by task (`run.py --interleave`), 6 runs at a time; pi ran its coding tasks afterwards. Each arm made one unscored request first.+- **Wall time.** The runner's clock around each process, start to exit.+- **Tokens.** Prompt tokens include cached ones in every arm. OpenCode 2's come from its own session database (`session_v2`), whose input count leaves out cache reads and writes, so the wrapper adds them back. No list cost: the gateway's prices are not public list prices.++## Caveats++- **pi runs only the coding tasks.** It has no MCP client.+- **One model.** Harness differences can look different on another model; the [Codex comparison](../graff-vs-codex-app-server/RESULTS.md) runs graff and pi on a different one.+- **Small sample.** 3 runs per task, one machine, one key. A second or two per task is within run-to-run noise.++## Reproduce++```sh+cd graff-evals+# CODEGRAFF_API_KEY set; OpenCode 2 needs a `codegraff` provider in its config+# (openai-compatible, baseURL https://gateway.codegraff.com/v1, apiKey {env:CODEGRAFF_API_KEY})+# and OPENCODE2_BIN when `opencode` on PATH is 1.x; pi needs the `codegraff`+# provider in ~/.pi/agent/models.json. Keep --output-root outside any repository,+# with scripts/linear_fixture_mcp.py copied to <parent of output-root>/scripts/.+TASKS="--task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm"+python3 run.py --harness graff-gateway-repl,opencode2 --model <gateway model> --suite core,mcp $TASKS \+ --reps 3 -j 6 --interleave --output-root /path/outside/repo/gateway+python3 run.py --harness pi-codegraff --model <gateway model> --suite core $TASKS \+ --reps 3 -j 6 --output-root /path/outside/repo/gateway-pi+```diff --git a/evals/graff-vs-opencode-and-pi/results.json b/evals/graff-vs-opencode-and-pi/results.jsonnew file mode 100644index 00000000..e75e5b52--- /dev/null+++ b/evals/graff-vs-opencode-and-pi/results.json@@ -0,0 +1,2534 @@+{+ "model": "one gateway model (same id in every arm)",+ "tasks": 21,+ "reps": 3,+ "arms": {+ "graff": "graff (ADR 0231 build)",+ "opencode2": "OpenCode 2",+ "pi": "pi"+ },+ "summary": {+ "all": {+ "graff": {+ "runs": 63,+ "passed": 63,+ "wall_s_per_task": 9.1,+ "cached_input_share": 0.945,+ "output_tokens_per_task": 582,+ "model_calls_per_task": 3.6+ },+ "opencode2": {+ "runs": 63,+ "passed": 62,+ "wall_s_per_task": 14.4,+ "cached_input_share": 0.862,+ "output_tokens_per_task": 980,+ "model_calls_per_task": 4.2+ },+ "pi": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 5.4,+ "cached_input_share": 0.785,+ "output_tokens_per_task": 258,+ "model_calls_per_task": 2.7+ }+ },+ "core": {+ "graff": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 7.3,+ "cached_input_share": 0.937,+ "output_tokens_per_task": 277,+ "model_calls_per_task": 2.8+ },+ "opencode2": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 6.9,+ "cached_input_share": 0.868,+ "output_tokens_per_task": 366,+ "model_calls_per_task": 3.2+ },+ "pi": {+ "runs": 33,+ "passed": 33,+ "wall_s_per_task": 5.4,+ "cached_input_share": 0.785,+ "output_tokens_per_task": 258,+ "model_calls_per_task": 2.7+ }+ },+ "mcp": {+ "graff": {+ "runs": 30,+ "passed": 30,+ "wall_s_per_task": 11.0,+ "cached_input_share": 0.95,+ "output_tokens_per_task": 917,+ "model_calls_per_task": 4.4+ },+ "opencode2": {+ "runs": 30,+ "passed": 29,+ "wall_s_per_task": 22.6,+ "cached_input_share": 0.858,+ "output_tokens_per_task": 1655,+ "model_calls_per_task": 5.3+ }+ }+ },+ "per_task": {+ "csv-sum": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.5,+ "model_calls": 3.3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.8,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.0,+ "model_calls": 3+ }+ },+ "dead-code": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.9,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.8,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.0,+ "model_calls": 3+ }+ },+ "exact-reply": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 1.8,+ "model_calls": 1+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 2.1,+ "model_calls": 1+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 1.7,+ "model_calls": 1+ }+ },+ "file-ops": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.2,+ "model_calls": 2+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.9,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.6,+ "model_calls": 2+ }+ },+ "fix-fib": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.1,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.6,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.6,+ "model_calls": 4+ }+ },+ "git-ops": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.1,+ "model_calls": 2+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.2,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.6,+ "model_calls": 2+ }+ },+ "json-transform": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.5,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.0,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.6,+ "model_calls": 3+ }+ },+ "linear-issues": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.5,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 30.4,+ "model_calls": 6+ }+ },+ "linear-nohint": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.0,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.2,+ "model_calls": 5.3+ }+ },+ "linear-nohint-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.9,+ "model_calls": 3.3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.4,+ "model_calls": 5.7+ }+ },+ "linear-quiet": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 11.3,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.2,+ "model_calls": 5.7+ }+ },+ "linear-quiet-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 15.7,+ "model_calls": 5+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.5,+ "model_calls": 5+ }+ },+ "linear-reduce": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.5,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.9,+ "model_calls": 5.7+ }+ },+ "linear-reduce-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.8,+ "model_calls": 4+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.2,+ "model_calls": 5.7+ }+ },+ "linear-sidecar": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.9,+ "model_calls": 3.7+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 22.5,+ "model_calls": 6+ }+ },+ "linear-split": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.9,+ "model_calls": 9+ },+ "opencode2": {+ "passed": 2,+ "runs": 3,+ "wall_s": 30.0,+ "model_calls": 2.7+ }+ },+ "linear-warm": {+ "suite": "mcp",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.1,+ "model_calls": 3.7+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.6,+ "model_calls": 5.7+ }+ },+ "recall-noise": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.8,+ "model_calls": 2+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.6,+ "model_calls": 3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.4,+ "model_calls": 2+ }+ },+ "refactor-rename": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.3,+ "model_calls": 5+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.6,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.8,+ "model_calls": 4+ }+ },+ "regex-count": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 14.1,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.6,+ "model_calls": 3.3+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.0,+ "model_calls": 3+ }+ },+ "write-tests": {+ "suite": "core",+ "graff": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.6,+ "model_calls": 3+ },+ "opencode2": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.4,+ "model_calls": 4+ },+ "pi": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.1,+ "model_calls": 3+ }+ }+ },+ "runs": [+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.9,+ "model_calls": 3,+ "input_tokens": 27883,+ "cached_input_tokens": 22546,+ "output_tokens": 311+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.23,+ "model_calls": 3,+ "input_tokens": 27912,+ "cached_input_tokens": 27756,+ "output_tokens": 355+ },+ {+ "arm": "graff",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.24,+ "model_calls": 4,+ "input_tokens": 37348,+ "cached_input_tokens": 37159,+ "output_tokens": 471+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.92,+ "model_calls": 3,+ "input_tokens": 28218,+ "cached_input_tokens": 22632,+ "output_tokens": 160+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.44,+ "model_calls": 3,+ "input_tokens": 28218,+ "cached_input_tokens": 28210,+ "output_tokens": 160+ },+ {+ "arm": "graff",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.36,+ "model_calls": 3,+ "input_tokens": 28228,+ "cached_input_tokens": 27748,+ "output_tokens": 165+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.89,+ "model_calls": 1,+ "input_tokens": 9121,+ "cached_input_tokens": 9117,+ "output_tokens": 51+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.75,+ "model_calls": 1,+ "input_tokens": 9121,+ "cached_input_tokens": 9117,+ "output_tokens": 51+ },+ {+ "arm": "graff",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.76,+ "model_calls": 1,+ "input_tokens": 9121,+ "cached_input_tokens": 9117,+ "output_tokens": 51+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.93,+ "model_calls": 2,+ "input_tokens": 18395,+ "cached_input_tokens": 18186,+ "output_tokens": 143+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.22,+ "model_calls": 2,+ "input_tokens": 18395,+ "cached_input_tokens": 18389,+ "output_tokens": 141+ },+ {+ "arm": "graff",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.39,+ "model_calls": 2,+ "input_tokens": 18395,+ "cached_input_tokens": 18389,+ "output_tokens": 158+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.41,+ "model_calls": 4,+ "input_tokens": 38412,+ "cached_input_tokens": 32526,+ "output_tokens": 394+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.11,+ "model_calls": 4,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null+ },+ {+ "arm": "graff",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.65,+ "model_calls": 4,+ "input_tokens": 38412,+ "cached_input_tokens": 37647,+ "output_tokens": 407+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.24,+ "model_calls": 2,+ "input_tokens": 18596,+ "cached_input_tokens": 13279,+ "output_tokens": 258+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.27,+ "model_calls": 2,+ "input_tokens": 18590,+ "cached_input_tokens": 18398,+ "output_tokens": 240+ },+ {+ "arm": "graff",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.77,+ "model_calls": 2,+ "input_tokens": 18595,+ "cached_input_tokens": 18398,+ "output_tokens": 264+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.44,+ "model_calls": 3,+ "input_tokens": 27943,+ "cached_input_tokens": 22578,+ "output_tokens": 254+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.77,+ "model_calls": 3,+ "input_tokens": 27943,+ "cached_input_tokens": 27935,+ "output_tokens": 261+ },+ {+ "arm": "graff",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.41,+ "model_calls": 3,+ "input_tokens": 27943,+ "cached_input_tokens": 27935,+ "output_tokens": 256+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.63,+ "model_calls": 3,+ "input_tokens": 30744,+ "cached_input_tokens": 20170,+ "output_tokens": 636+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.05,+ "model_calls": 3,+ "input_tokens": 30882,+ "cached_input_tokens": 30536,+ "output_tokens": 611+ },+ {+ "arm": "graff",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.81,+ "model_calls": 3,+ "input_tokens": 30934,+ "cached_input_tokens": 30536,+ "output_tokens": 647+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.38,+ "model_calls": 4,+ "input_tokens": 41204,+ "cached_input_tokens": 39485,+ "output_tokens": 907+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.07,+ "model_calls": 4,+ "input_tokens": 41193,+ "cached_input_tokens": 40898,+ "output_tokens": 851+ },+ {+ "arm": "graff",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.6,+ "model_calls": 4,+ "input_tokens": 41231,+ "cached_input_tokens": 40166,+ "output_tokens": 853+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.69,+ "model_calls": 4,+ "input_tokens": 42833,+ "cached_input_tokens": 40824,+ "output_tokens": 676+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.45,+ "model_calls": 3,+ "input_tokens": 32072,+ "cached_input_tokens": 31503,+ "output_tokens": 993+ },+ {+ "arm": "graff",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.64,+ "model_calls": 3,+ "input_tokens": 32072,+ "cached_input_tokens": 32064,+ "output_tokens": 922+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.98,+ "model_calls": 4,+ "input_tokens": 41013,+ "cached_input_tokens": 39412,+ "output_tokens": 594+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.67,+ "model_calls": 4,+ "input_tokens": 41013,+ "cached_input_tokens": 40808,+ "output_tokens": 617+ },+ {+ "arm": "graff",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.12,+ "model_calls": 4,+ "input_tokens": 41013,+ "cached_input_tokens": 41003,+ "output_tokens": 580+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.89,+ "model_calls": 7,+ "input_tokens": 78474,+ "cached_input_tokens": 75338,+ "output_tokens": 1305+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.06,+ "model_calls": 4,+ "input_tokens": 43098,+ "cached_input_tokens": 42796,+ "output_tokens": 653+ },+ {+ "arm": "graff",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.08,+ "model_calls": 4,+ "input_tokens": 43040,+ "cached_input_tokens": 42395,+ "output_tokens": 679+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.39,+ "model_calls": 4,+ "input_tokens": 41229,+ "cached_input_tokens": 39545,+ "output_tokens": 774+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.94,+ "model_calls": 4,+ "input_tokens": 41245,+ "cached_input_tokens": 40485,+ "output_tokens": 616+ },+ {+ "arm": "graff",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.18,+ "model_calls": 4,+ "input_tokens": 41252,+ "cached_input_tokens": 40488,+ "output_tokens": 596+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.74,+ "model_calls": 4,+ "input_tokens": 42823,+ "cached_input_tokens": 40749,+ "output_tokens": 639+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.15,+ "model_calls": 4,+ "input_tokens": 42815,+ "cached_input_tokens": 41901,+ "output_tokens": 548+ },+ {+ "arm": "graff",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.54,+ "model_calls": 4,+ "input_tokens": 42815,+ "cached_input_tokens": 42805,+ "output_tokens": 622+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.51,+ "model_calls": 4,+ "input_tokens": 49605,+ "cached_input_tokens": 45782,+ "output_tokens": 912+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.25,+ "model_calls": 3,+ "input_tokens": 37300,+ "cached_input_tokens": 36266,+ "output_tokens": 896+ },+ {+ "arm": "graff",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.99,+ "model_calls": 4,+ "input_tokens": 49603,+ "cached_input_tokens": 48530,+ "output_tokens": 875+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.16,+ "model_calls": 8,+ "input_tokens": 77647,+ "cached_input_tokens": 59430,+ "output_tokens": 2062+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.19,+ "model_calls": 8,+ "input_tokens": 77930,+ "cached_input_tokens": 74039,+ "output_tokens": 2168+ },+ {+ "arm": "graff",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.48,+ "model_calls": 11,+ "input_tokens": 104516,+ "cached_input_tokens": 95787,+ "output_tokens": 2901+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.33,+ "model_calls": 4,+ "input_tokens": 43066,+ "cached_input_tokens": 40995,+ "output_tokens": 698+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.63,+ "model_calls": 4,+ "input_tokens": 43097,+ "cached_input_tokens": 42654,+ "output_tokens": 719+ },+ {+ "arm": "graff",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.39,+ "model_calls": 3,+ "input_tokens": 32244,+ "cached_input_tokens": 31674,+ "output_tokens": 953+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.2,+ "model_calls": 2,+ "input_tokens": 18747,+ "cached_input_tokens": 13253,+ "output_tokens": 63+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.56,+ "model_calls": 2,+ "input_tokens": 18747,+ "cached_input_tokens": 18741,+ "output_tokens": 110+ },+ {+ "arm": "graff",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.58,+ "model_calls": 2,+ "input_tokens": 18747,+ "cached_input_tokens": 18741,+ "output_tokens": 110+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.61,+ "model_calls": 5,+ "input_tokens": 47904,+ "cached_input_tokens": 42043,+ "output_tokens": 555+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.28,+ "model_calls": 5,+ "input_tokens": 48174,+ "cached_input_tokens": 47635,+ "output_tokens": 639+ },+ {+ "arm": "graff",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.99,+ "model_calls": 5,+ "input_tokens": 47904,+ "cached_input_tokens": 47892,+ "output_tokens": 525+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.91,+ "model_calls": 3,+ "input_tokens": 28150,+ "cached_input_tokens": 22704,+ "output_tokens": 505+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.03,+ "model_calls": 3,+ "input_tokens": 28165,+ "cached_input_tokens": 27851,+ "output_tokens": 365+ },+ {+ "arm": "graff",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.32,+ "model_calls": 3,+ "input_tokens": 28061,+ "cached_input_tokens": 27784,+ "output_tokens": 404+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.49,+ "model_calls": 3,+ "input_tokens": 28080,+ "cached_input_tokens": 22556,+ "output_tokens": 446+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.22,+ "model_calls": 3,+ "input_tokens": 28080,+ "cached_input_tokens": 28072,+ "output_tokens": 420+ },+ {+ "arm": "graff",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.13,+ "model_calls": 3,+ "input_tokens": 28080,+ "cached_input_tokens": 28072,+ "output_tokens": 444+ },+ {+ "arm": "opencode2",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.68,+ "model_calls": 3,+ "input_tokens": 23865,+ "cached_input_tokens": 20630,+ "output_tokens": 311+ },+ {+ "arm": "opencode2",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.96,+ "model_calls": 3,+ "input_tokens": 23865,+ "cached_input_tokens": 20671,+ "output_tokens": 319+ },+ {+ "arm": "opencode2",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.9,+ "model_calls": 3,+ "input_tokens": 23865,+ "cached_input_tokens": 20671,+ "output_tokens": 304+ },+ {+ "arm": "opencode2",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.91,+ "model_calls": 4,+ "input_tokens": 32597,+ "cached_input_tokens": 28619,+ "output_tokens": 345+ },+ {+ "arm": "opencode2",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.53,+ "model_calls": 4,+ "input_tokens": 32305,+ "cached_input_tokens": 28671,+ "output_tokens": 289+ },+ {+ "arm": "opencode2",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.97,+ "model_calls": 4,+ "input_tokens": 32305,+ "cached_input_tokens": 28671,+ "output_tokens": 289+ },+ {+ "arm": "opencode2",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 2.49,+ "model_calls": 1,+ "input_tokens": 8280,+ "cached_input_tokens": 5436,+ "output_tokens": 13+ },+ {+ "arm": "opencode2",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.75,+ "model_calls": 1,+ "input_tokens": 7484,+ "cached_input_tokens": 4644,+ "output_tokens": 4+ },+ {+ "arm": "opencode2",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.96,+ "model_calls": 1,+ "input_tokens": 7484,+ "cached_input_tokens": 4644,+ "output_tokens": 4+ },+ {+ "arm": "opencode2",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.73,+ "model_calls": 3,+ "input_tokens": 23691,+ "cached_input_tokens": 20568,+ "output_tokens": 255+ },+ {+ "arm": "opencode2",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.77,+ "model_calls": 3,+ "input_tokens": 23744,+ "cached_input_tokens": 20614,+ "output_tokens": 282+ },+ {+ "arm": "opencode2",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.21,+ "model_calls": 3,+ "input_tokens": 23857,+ "cached_input_tokens": 20614,+ "output_tokens": 295+ },+ {+ "arm": "opencode2",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.21,+ "model_calls": 4,+ "input_tokens": 32294,+ "cached_input_tokens": 28798,+ "output_tokens": 345+ },+ {+ "arm": "opencode2",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.74,+ "model_calls": 4,+ "input_tokens": 32294,+ "cached_input_tokens": 28857,+ "output_tokens": 358+ },+ {+ "arm": "opencode2",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.86,+ "model_calls": 4,+ "input_tokens": 32294,+ "cached_input_tokens": 28857,+ "output_tokens": 358+ },+ {+ "arm": "opencode2",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.65,+ "model_calls": 3,+ "input_tokens": 23234,+ "cached_input_tokens": 19849,+ "output_tokens": 319+ },+ {+ "arm": "opencode2",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.05,+ "model_calls": 3,+ "input_tokens": 24079,+ "cached_input_tokens": 20691,+ "output_tokens": 353+ },+ {+ "arm": "opencode2",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.79,+ "model_calls": 3,+ "input_tokens": 24083,+ "cached_input_tokens": 20687,+ "output_tokens": 330+ },+ {+ "arm": "opencode2",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.75,+ "model_calls": 3,+ "input_tokens": 23966,+ "cached_input_tokens": 20665,+ "output_tokens": 378+ },+ {+ "arm": "opencode2",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.12,+ "model_calls": 3,+ "input_tokens": 23966,+ "cached_input_tokens": 20719,+ "output_tokens": 349+ },+ {+ "arm": "opencode2",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.07,+ "model_calls": 3,+ "input_tokens": 23966,+ "cached_input_tokens": 20719,+ "output_tokens": 391+ },+ {+ "arm": "opencode2",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 39.11,+ "model_calls": 6,+ "input_tokens": 64184,+ "cached_input_tokens": 55752,+ "output_tokens": 1632+ },+ {+ "arm": "opencode2",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.91,+ "model_calls": 6,+ "input_tokens": 66506,+ "cached_input_tokens": 57714,+ "output_tokens": 1849+ },+ {+ "arm": "opencode2",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.19,+ "model_calls": 6,+ "input_tokens": 67866,+ "cached_input_tokens": 58508,+ "output_tokens": 1966+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.28,+ "model_calls": 5,+ "input_tokens": 55723,+ "cached_input_tokens": 46086,+ "output_tokens": 1773+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.57,+ "model_calls": 6,+ "input_tokens": 67456,+ "cached_input_tokens": 58096,+ "output_tokens": 1759+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.79,+ "model_calls": 5,+ "input_tokens": 55608,+ "cached_input_tokens": 46175,+ "output_tokens": 1693+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.38,+ "model_calls": 6,+ "input_tokens": 67190,+ "cached_input_tokens": 57961,+ "output_tokens": 1782+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.84,+ "model_calls": 5,+ "input_tokens": 53445,+ "cached_input_tokens": 45015,+ "output_tokens": 1547+ },+ {+ "arm": "opencode2",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.92,+ "model_calls": 6,+ "input_tokens": 66914,+ "cached_input_tokens": 57976,+ "output_tokens": 1719+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.99,+ "model_calls": 6,+ "input_tokens": 58611,+ "cached_input_tokens": 51656,+ "output_tokens": 1652+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.81,+ "model_calls": 5,+ "input_tokens": 47567,+ "cached_input_tokens": 41313,+ "output_tokens": 1653+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.95,+ "model_calls": 6,+ "input_tokens": 58287,+ "cached_input_tokens": 51569,+ "output_tokens": 1867+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.55,+ "model_calls": 4,+ "input_tokens": 36901,+ "cached_input_tokens": 31489,+ "output_tokens": 1737+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.03,+ "model_calls": 5,+ "input_tokens": 54234,+ "cached_input_tokens": 44605,+ "output_tokens": 1786+ },+ {+ "arm": "opencode2",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.01,+ "model_calls": 6,+ "input_tokens": 59377,+ "cached_input_tokens": 52389,+ "output_tokens": 1848+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.74,+ "model_calls": 6,+ "input_tokens": 60410,+ "cached_input_tokens": 52832,+ "output_tokens": 1888+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.63,+ "model_calls": 6,+ "input_tokens": 66525,+ "cached_input_tokens": 57121,+ "output_tokens": 1891+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.28,+ "model_calls": 5,+ "input_tokens": 48244,+ "cached_input_tokens": 41797,+ "output_tokens": 2266+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.98,+ "model_calls": 5,+ "input_tokens": 47381,+ "cached_input_tokens": 40951,+ "output_tokens": 1725+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.83,+ "model_calls": 6,+ "input_tokens": 57363,+ "cached_input_tokens": 50959,+ "output_tokens": 1764+ },+ {+ "arm": "opencode2",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.87,+ "model_calls": 6,+ "input_tokens": 58147,+ "cached_input_tokens": 51586,+ "output_tokens": 1819+ },+ {+ "arm": "opencode2",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.1,+ "model_calls": 6,+ "input_tokens": 65921,+ "cached_input_tokens": 57064,+ "output_tokens": 1761+ },+ {+ "arm": "opencode2",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.24,+ "model_calls": 6,+ "input_tokens": 65529,+ "cached_input_tokens": 57447,+ "output_tokens": 1833+ },+ {+ "arm": "opencode2",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.05,+ "model_calls": 6,+ "input_tokens": 65650,+ "cached_input_tokens": 57132,+ "output_tokens": 1712+ },+ {+ "arm": "opencode2",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": false,+ "timed_out": false,+ "wall_s": 42.55,+ "model_calls": 3,+ "input_tokens": 13805,+ "cached_input_tokens": 6430,+ "output_tokens": 291+ },+ {+ "arm": "opencode2",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.9,+ "model_calls": 3,+ "input_tokens": 21079,+ "cached_input_tokens": 16530,+ "output_tokens": 583+ },+ {+ "arm": "opencode2",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.6,+ "model_calls": 2,+ "input_tokens": 13327,+ "cached_input_tokens": 9691,+ "output_tokens": 355+ },+ {+ "arm": "opencode2",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.22,+ "model_calls": 6,+ "input_tokens": 67825,+ "cached_input_tokens": 58324,+ "output_tokens": 1803+ },+ {+ "arm": "opencode2",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.32,+ "model_calls": 6,+ "input_tokens": 66147,+ "cached_input_tokens": 57536,+ "output_tokens": 1984+ },+ {+ "arm": "opencode2",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.27,+ "model_calls": 5,+ "input_tokens": 53869,+ "cached_input_tokens": 45387,+ "output_tokens": 1714+ },+ {+ "arm": "opencode2",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.66,+ "model_calls": 3,+ "input_tokens": 24029,+ "cached_input_tokens": 20531,+ "output_tokens": 167+ },+ {+ "arm": "opencode2",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.56,+ "model_calls": 3,+ "input_tokens": 24029,+ "cached_input_tokens": 20567,+ "output_tokens": 167+ },+ {+ "arm": "opencode2",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.48,+ "model_calls": 3,+ "input_tokens": 23211,+ "cached_input_tokens": 19753,+ "output_tokens": 153+ },+ {+ "arm": "opencode2",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.12,+ "model_calls": 4,+ "input_tokens": 32966,+ "cached_input_tokens": 29034,+ "output_tokens": 730+ },+ {+ "arm": "opencode2",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.01,+ "model_calls": 4,+ "input_tokens": 32969,+ "cached_input_tokens": 29076,+ "output_tokens": 801+ },+ {+ "arm": "opencode2",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.68,+ "model_calls": 4,+ "input_tokens": 32998,+ "cached_input_tokens": 29120,+ "output_tokens": 737+ },+ {+ "arm": "opencode2",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.25,+ "model_calls": 3,+ "input_tokens": 24358,+ "cached_input_tokens": 20847,+ "output_tokens": 496+ },+ {+ "arm": "opencode2",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.45,+ "model_calls": 3,+ "input_tokens": 24207,+ "cached_input_tokens": 20897,+ "output_tokens": 401+ },+ {+ "arm": "opencode2",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.99,+ "model_calls": 4,+ "input_tokens": 32621,+ "cached_input_tokens": 29005,+ "output_tokens": 619+ },+ {+ "arm": "opencode2",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.5,+ "model_calls": 4,+ "input_tokens": 32008,+ "cached_input_tokens": 28378,+ "output_tokens": 655+ },+ {+ "arm": "opencode2",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.73,+ "model_calls": 4,+ "input_tokens": 31962,+ "cached_input_tokens": 28438,+ "output_tokens": 605+ },+ {+ "arm": "opencode2",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.01,+ "model_calls": 4,+ "input_tokens": 32008,+ "cached_input_tokens": 28438,+ "output_tokens": 640+ },+ {+ "arm": "pi",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.9,+ "model_calls": 3,+ "input_tokens": 7244,+ "cached_input_tokens": 5896,+ "output_tokens": 262+ },+ {+ "arm": "pi",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.53,+ "model_calls": 3,+ "input_tokens": 7223,+ "cached_input_tokens": 5896,+ "output_tokens": 215+ },+ {+ "arm": "pi",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.54,+ "model_calls": 3,+ "input_tokens": 7223,+ "cached_input_tokens": 5896,+ "output_tokens": 235+ },+ {+ "arm": "pi",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.25,+ "model_calls": 3,+ "input_tokens": 7488,+ "cached_input_tokens": 5970,+ "output_tokens": 175+ },+ {+ "arm": "pi",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.43,+ "model_calls": 3,+ "input_tokens": 8379,+ "cached_input_tokens": 5999,+ "output_tokens": 291+ },+ {+ "arm": "pi",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.19,+ "model_calls": 3,+ "input_tokens": 7572,+ "cached_input_tokens": 5991,+ "output_tokens": 238+ },+ {+ "arm": "pi",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.77,+ "model_calls": 1,+ "input_tokens": 2271,+ "cached_input_tokens": 1190,+ "output_tokens": 4+ },+ {+ "arm": "pi",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.64,+ "model_calls": 1,+ "input_tokens": 2271,+ "cached_input_tokens": 1190,+ "output_tokens": 4+ },+ {+ "arm": "pi",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.82,+ "model_calls": 1,+ "input_tokens": 2271,+ "cached_input_tokens": 1190,+ "output_tokens": 4+ },+ {+ "arm": "pi",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.38,+ "model_calls": 2,+ "input_tokens": 4746,+ "cached_input_tokens": 3489,+ "output_tokens": 163+ },+ {+ "arm": "pi",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.95,+ "model_calls": 2,+ "input_tokens": 4746,+ "cached_input_tokens": 3489,+ "output_tokens": 169+ },+ {+ "arm": "pi",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.57,+ "model_calls": 2,+ "input_tokens": 4746,+ "cached_input_tokens": 3489,+ "output_tokens": 168+ },+ {+ "arm": "pi",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.84,+ "model_calls": 4,+ "input_tokens": 10457,+ "cached_input_tokens": 8722,+ "output_tokens": 434+ },+ {+ "arm": "pi",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.13,+ "model_calls": 4,+ "input_tokens": 10704,+ "cached_input_tokens": 8892,+ "output_tokens": 417+ },+ {+ "arm": "pi",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.78,+ "model_calls": 4,+ "input_tokens": 10710,+ "cached_input_tokens": 8892,+ "output_tokens": 428+ },+ {+ "arm": "pi",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.73,+ "model_calls": 2,+ "input_tokens": 5009,+ "cached_input_tokens": 3507,+ "output_tokens": 289+ },+ {+ "arm": "pi",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.53,+ "model_calls": 2,+ "input_tokens": 5009,+ "cached_input_tokens": 3507,+ "output_tokens": 257+ },+ {+ "arm": "pi",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.51,+ "model_calls": 2,+ "input_tokens": 5009,+ "cached_input_tokens": 3507,+ "output_tokens": 254+ },+ {+ "arm": "pi",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.42,+ "model_calls": 3,+ "input_tokens": 7314,+ "cached_input_tokens": 5927,+ "output_tokens": 205+ },+ {+ "arm": "pi",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.51,+ "model_calls": 3,+ "input_tokens": 7298,+ "cached_input_tokens": 5927,+ "output_tokens": 249+ },+ {+ "arm": "pi",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.76,+ "model_calls": 3,+ "input_tokens": 7341,+ "cached_input_tokens": 5927,+ "output_tokens": 261+ },+ {+ "arm": "pi",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.43,+ "model_calls": 2,+ "input_tokens": 4979,+ "cached_input_tokens": 3480,+ "output_tokens": 61+ },+ {+ "arm": "pi",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 2.89,+ "model_calls": 2,+ "input_tokens": 4979,+ "cached_input_tokens": 3480,+ "output_tokens": 61+ },+ {+ "arm": "pi",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 2.91,+ "model_calls": 2,+ "input_tokens": 4979,+ "cached_input_tokens": 3480,+ "output_tokens": 61+ },+ {+ "arm": "pi",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.98,+ "model_calls": 4,+ "input_tokens": 10202,+ "cached_input_tokens": 8555,+ "output_tokens": 535+ },+ {+ "arm": "pi",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.92,+ "model_calls": 4,+ "input_tokens": 10185,+ "cached_input_tokens": 8555,+ "output_tokens": 501+ },+ {+ "arm": "pi",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.41,+ "model_calls": 4,+ "input_tokens": 10202,+ "cached_input_tokens": 8589,+ "output_tokens": 420+ },+ {+ "arm": "pi",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.3,+ "model_calls": 3,+ "input_tokens": 7414,+ "cached_input_tokens": 6012,+ "output_tokens": 343+ },+ {+ "arm": "pi",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.5,+ "model_calls": 3,+ "input_tokens": 7433,+ "cached_input_tokens": 6029,+ "output_tokens": 311+ },+ {+ "arm": "pi",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.24,+ "model_calls": 3,+ "input_tokens": 7429,+ "cached_input_tokens": 6027,+ "output_tokens": 349+ },+ {+ "arm": "pi",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.24,+ "model_calls": 3,+ "input_tokens": 7324,+ "cached_input_tokens": 5903,+ "output_tokens": 428+ },+ {+ "arm": "pi",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.32,+ "model_calls": 3,+ "input_tokens": 7324,+ "cached_input_tokens": 5903,+ "output_tokens": 324+ },+ {+ "arm": "pi",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.68,+ "model_calls": 3,+ "input_tokens": 7324,+ "cached_input_tokens": 5903,+ "output_tokens": 401+ }+ ]+}\ No newline at end of filediff --git a/graff-evals/README.md b/graff-evals/README.mdindex 078fda44..e1af4f1c 100644--- a/graff-evals/README.md+++ b/graff-evals/README.md@@ -48,12 +48,18 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co./run.py --suite live --harness graff-dev --reps 3 -j 1# same SuperGrok seat, other harnesses (OpenCode / Pi / exo local-process):./run.py --suite live --harness opencode,pi-xai,exo --reps 3 -j 1-# ChatGPT login, 1:1 with the Codex app server (needs `codex`; the driver copies the-# login without its refresh token). Keep --output-root outside any repo so neither-# harness reads repo docs; the mcp suite then needs scripts/linear_fixture_mcp.py-# copied next to that root (the setup looks for it in a parent directory).+# ChatGPT login, 1:1 with the Codex app server and pi (needs `codex`; the driver copies+# the login without its refresh token; pi-codex needs pi signed in on the ChatGPT plan).+# Keep --output-root outside any repo so no harness reads repo docs; the mcp suite then+# needs scripts/linear_fixture_mcp.py copied next to that root (the setup looks for it+# in a parent directory). pi has no MCP client, so it runs the core suite.# Results: evals/graff-vs-codex-app-server/RESULTS.md-./run.py --suite core,mcp --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol -j 6+./run.py --suite mcp --harness graff-repl,codex-app-server -j 8 --interleave+./run.py --suite core --harness graff-repl,codex-app-server,pi-codex -j 8 --interleave+# one gateway model in graff, OpenCode 2 and pi (set CODEGRAFF_API_KEY; OPENCODE2_BIN+# when `opencode` on PATH is 1.x). Results: evals/graff-vs-opencode-and-pi/RESULTS.md+./run.py --suite core,mcp --harness graff-gateway-repl,opencode2 --model <gateway model> -j 6 --interleave+./run.py --suite core --harness pi-codegraff --model <gateway model> -j 6```## Run itdiff --git a/graff-evals/codex_app_server.py b/graff-evals/codex_app_server.pyindex d1b71236..5da71afd 100755--- a/graff-evals/codex_app_server.py+++ b/graff-evals/codex_app_server.py@@ -33,6 +33,9 @@ import timefrom datetime import datetime, timezone+T0 = time.monotonic()++def trace(ev: dict) -> None:"""EVAL_TRACE_DIR: one JSONL per sandbox of per-call usage and item sizes(what each output token went to), for output and cache analysis."""@@ -40,6 +43,7 @@ def trace(ev: dict) -> None:if not root:returnos.makedirs(root, exist_ok=True)+ ev = dict(ev, t=round((time.monotonic() - T0) * 1000)) # ms since the driver startedwith open(os.path.join(root, os.path.basename(os.getcwd()) + ".jsonl"), "a") as fh:fh.write(json.dumps(ev) + "\n")diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.jsonindex b61aaf53..803e44a7 100644--- a/graff-evals/harnesses.json+++ b/graff-evals/harnesses.json@@ -333,5 +333,39 @@"answer": "stdout","usage": "graff-stderr","default_model": "gpt-6.1-sol"+ },+ "graff-repl": {+ "cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"],+ "stdin": "\u001b[200~{prompt}\u001b[201~\n",+ "answer": "stdout",+ "usage": "graff-stderr",+ "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"},+ "model_prefix": "codex",+ "default_model": "gpt-6.1-sol",+ "note": "scripted graff repl over the Codex WebSocket. The prompt rides one bracketed-paste frame, so a multi-line task is one turn as in the other harnesses (needs the piped paste of ADR 0231)."+ },+ "graff-gateway-repl": {+ "cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"],+ "stdin": "\u001b[200~{prompt}\u001b[201~\n",+ "answer": "stdout",+ "usage": "graff-stderr",+ "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"},+ "model_prefix": "codegraff",+ "note": "scripted graff repl on the Codegraff gateway (CODEGRAFF_API_KEY); pass --model"+ },+ "pi-codex": {+ "cmd": ["pi", "-p", "--mode", "json", "--provider", "openai-codex", "--model", "{model}", "--thinking", "low", "--no-session", "{prompt}"],+ "answer": "pi-json",+ "usage": "pi-json",+ "capabilities": [],+ "default_model": "gpt-6.1-sol",+ "note": "pi on the ChatGPT plan (provider openai-codex, signed in through pi; declare gpt-6.1-sol in ~/.pi/agent/models.json until pi lists it). Low effort, the catalog default graff and the Codex app server use. No MCP client: core suite only."+ },+ "opencode2": {+ "cmd": ["python3", "{repo}/graff-evals/opencode2_run.py", "{model}", "{prompt}"],+ "answer": "stdout",+ "usage": "graff-stderr",+ "model_prefix": "codegraff",+ "note": "OpenCode 2 (@opencode/cli 2.x; OPENCODE2_BIN when `opencode` on PATH is 1.x) with a codegraff provider in its config; pass --model. The wrapper maps .mcp.json into opencode.json and reads usage from OpenCode's session database."}}diff --git a/graff-evals/opencode2_run.py b/graff-evals/opencode2_run.pynew file mode 100755index 00000000..f27d064c--- /dev/null+++ b/graff-evals/opencode2_run.py@@ -0,0 +1,53 @@+#!/usr/bin/env python3+"""Run OpenCode 2 headless on one eval task and report like graff does.++usage: opencode2_run.py PROVIDER/MODEL PROMPT (cwd = the task sandbox)++The task's `.mcp.json` becomes this directory's `opencode.json` `mcp` block+(OpenCode does not read `.mcp.json`). The final assistant text goes to stdout;+a graff-format `[usage]` line goes to stderr, read back from OpenCode's own+session database (input there excludes cache reads and writes, so they are+added back to make the total prompt tokens). stdin is empty: OpenCode appends+piped stdin to the prompt.+"""+import json, os, sqlite3, subprocess, sys++# OpenCode 2 is the @opencode/cli package; an `opencode` on PATH may be a 1.x install.+OC2 = os.environ.get("OPENCODE2_BIN", "opencode")+model, prompt = sys.argv[1], sys.argv[2]+cwd = os.getcwd()++if os.path.exists(".mcp.json"):+ servers = json.load(open(".mcp.json")).get("mcpServers", {})+ mcp = {name: {"type": "local", "command": [spec["command"], *spec.get("args", [])], "enabled": True}+ for name, spec in servers.items()}+ json.dump({"$schema": "https://opencode.ai/config.json", "mcp": mcp}, open("opencode.json", "w"))++r = subprocess.run([OC2, "run", "--standalone", "--auto", "--format", "json", "-m", model, prompt],+ stdin=subprocess.DEVNULL, capture_output=True, text=True)+texts = []+for line in r.stdout.splitlines():+ try:+ ev = json.loads(line)+ except ValueError:+ continue+ if ev.get("type") == "text":+ texts.append((ev.get("part") or {}).get("text", ""))+print(texts[-1] if texts else "")+sys.stderr.write(r.stderr)++db_path = os.path.join(os.environ["HOME"], ".local/share/opencode/opencode.db")+try:+ db = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)+ dirs = {cwd, os.path.realpath(cwd), "/private" + cwd if cwd.startswith("/tmp/") else cwd}+ q = ",".join("?" * len(dirs))+ row = db.execute(f"select id, tokens_input, tokens_output, tokens_reasoning, tokens_cache_read, tokens_cache_write "+ f"from session_v2 where directory in ({q}) order by time_created desc limit 1", tuple(dirs)).fetchone()+ if row:+ sid, tin, tout, treason, tread, twrite = row+ calls = db.execute("select count(*) from session_message where session_id=? and type='assistant'", (sid,)).fetchone()[0]+ total_in = (tin or 0) + (tread or 0) + (twrite or 0)+ sys.stderr.write(f"\n[usage] {calls} api call(s) · {total_in} in ({tread or 0} cached, {twrite or 0} cache writes) + {(tout or 0) + (treason or 0)} out tokens\n")+except sqlite3.Error as e:+ sys.stderr.write(f"opencode2_run: usage unavailable: {e}\n")+sys.exit(r.returncode)diff --git a/graff-evals/run.py b/graff-evals/run.pyindex 3bad96b6..888e111d 100755--- a/graff-evals/run.py+++ b/graff-evals/run.py@@ -32,7 +32,7 @@ RESULTS_DIR = os.path.join(ROOT, "results")SANDBOX_DIR = os.path.join(ROOT, ".sandboxes")GRAFF_USAGE_RE = re.compile(- r"\[usage\] (\d+) api call\(s\) · (\d+) in \((\d+) cached(?:, (\d+) cache writes)?\) \+ (\d+) out tokens")+ r"\[usage\] (?:known subtotal: )?(\d+) api call\(s\) · (\d+) in \((\d+) cached(?:, (\d+) cache writes)?\) \+ (\d+) out tokens")def load_tasks():@@ -371,7 +371,7 @@ def one_run(hname, harness, task, model, rep, live=False):request_capture.configure(env, sandbox)try:p = subprocess.Popen(cmd, cwd=sandbox, stdout=subprocess.PIPE,- stderr=subprocess.PIPE, stdin=subprocess.PIPE if stdin_body is not None else None,+ stderr=subprocess.PIPE, stdin=subprocess.PIPE if stdin_body is not None else subprocess.DEVNULL,text=True, start_new_session=True,env=env)if stdin_body is not None and p.stdin is not None: