Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

evals: graff vs Claude Code, OpenCode 2 and pi on a gateway model

justrachfb5ab10b139d
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 60969393..a3ecd5dc 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -15,8 +15,10 @@ grok-4.6 SuperGrok) is
graff against the Codex app server and pi on the same model and ChatGPT account
(21 coding and MCP tasks, every run's data) is
[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md),
-and graff against OpenCode 2 and pi on one model through the Codegraff gateway is
-[evals/graff-vs-opencode-and-pi/RESULTS.md](../evals/graff-vs-opencode-and-pi/RESULTS.md).
+graff against Claude Code, OpenCode 2 and pi on one model through the Codegraff
+gateway is [evals/graff-vs-claude-code/RESULTS.md](../evals/graff-vs-claude-code/RESULTS.md)
+(an earlier run without Claude Code:
+[evals/graff-vs-opencode-and-pi/RESULTS.md](../evals/graff-vs-opencode-and-pi/RESULTS.md)).
The same 21 tasks against the Grok CLI on the same SuperGrok account are
[evals/graff-vs-grok-cli/RESULTS.md](../evals/graff-vs-grok-cli/RESULTS.md),
and what both mean for an engineering budget is
diff --git a/evals/graff-vs-claude-code/RESULTS.md b/evals/graff-vs-claude-code/RESULTS.md
new file mode 100644
index 00000000..6eeb8139
--- /dev/null
+++ b/evals/graff-vs-claude-code/RESULTS.md
@@ -0,0 +1,106 @@
+# graff vs Claude Code, OpenCode 2 and pi on a gateway model
+
+Measured 2026-10-01. One model, served through the Codegraff gateway, ran the same 21 tasks in each harness, side by side, on one machine and one gateway key.
+
+| Arm | What ran |
+| --- | --- |
+| graff | The ADR 0231 build (#1449) as a scripted `graff repl` (the `graff-gateway-repl` harness). |
+| Claude Code | Claude Code 2.1.286: `claude -p --output-format stream-json --verbose --dangerously-skip-permissions`, with the task's `.mcp.json` as its only MCP config, through [`graff-evals/claude_code_run.py`](../../graff-evals/claude_code_run.py). The gateway takes chat completions only, so Claude Code reached it through a local LiteLLM 1.103.2 proxy that turns Anthropic Messages requests into chat completions. |
+| OpenCode 2 | `@opencode/cli` 2.0.21: `opencode run --standalone --auto --format json`, through [`graff-evals/opencode2_run.py`](../../graff-evals/opencode2_run.py). |
+| pi | pi 0.84.2 (the `pi-codegraff` harness). pi has no MCP client, so it ran the 11 coding tasks only. |
+
+All four send the same model id to the same gateway with the same key, and each runs at its own defaults. Claude Code's background requests for a small model went to the same model.
+
+## Results
+
+All 21 tasks (pi: the 11 coding tasks), mean per task over 3 runs of each:
+
+| | Pass | Wall per task | Input read from cache | Output tokens per task | Model calls per task |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| graff (ADR 0231 build) | 63/63 | 8.7s | 95% | 573 | 3.5 |
+| Claude Code | 63/63 | 13.2s | 60% | 926 | 3.5 |
+| OpenCode 2 | 63/63 | 14.1s | 86% | 978 | 4.3 |
+| pi | 33/33 | 6.0s | 76% | 262 | 2.8 |
+
+By suite:
+
+| Suite | graff (ADR 0231 build) | Claude Code | OpenCode 2 | pi |
+| --- | ---: | ---: | ---: | ---: |
+| core (11 coding tasks), wall | 7.0s | 6.0s | 6.8s | 6.0s |
+| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 | 33/33 |
+| mcp (10 MCP tasks), wall | 10.6s | 21.2s | 22.1s | — |
+| mcp (10 MCP tasks), pass | 30/30 | 30/30 | 30/30 | — |
+
+**Against Claude Code.** graff is 34% faster per task over all 21 tasks (8.7s against 13.2s). The gap is the MCP tasks, which graff finishes in half the time (10.6s against 21.2s). Claude Code keeps every MCP result whole in its context: 179k prompt tokens per MCP task against graff's 44k, about 40k per call against 10k, and it writes nearly twice the output (1,645 tokens against 882). graff slims large list results and often fetches, reduces and writes the report in one `rlm` script. On the coding tasks Claude Code is faster (6.0s against 7.0s); one 11.7-second gateway response on graff's one-call `exact-reply` accounts for about 0.3s of that.
+
+**Against OpenCode 2 and pi.** graff takes 38% less time than OpenCode 2 over all tasks (8.7s against 14.1s) and about half on the MCP tasks (10.6s against 22.1s); OpenCode 2 is slightly faster on the coding tasks (6.8s against 7.0s). pi ties Claude Code as the fastest on the coding tasks (6.0s), with the smallest prompt.
+
+## Per task
+
+Wall time is the mean of 3 runs.
+
+| Task | Suite | graff (ADR 0231 build) | Claude Code | OpenCode 2 | pi |
+| --- | --- | ---: | ---: | ---: | ---: |
+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 8.4s | 6.9s | 5.9s | 4.7s |
+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 5.3s | 6.0s | 6.8s | 7.8s |
+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 6.2s | 1.7s | 1.7s | 1.9s |
+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 6.8s | 6.0s | 5.8s | 3.5s |
+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 8.7s | 6.4s | 7.9s | 12.0s |
+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 4.1s | 4.6s | 6.3s | 7.1s |
+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 5.8s | 6.7s | 6.3s | 5.0s |
+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 8.8s | 18.6s | 22.9s | — |
+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 11.1s | 20.1s | 18.7s | — |
+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 10.2s | 19.3s | 23.0s | — |
+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 9.2s | 28.0s | 21.2s | — |
+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 9.8s | 19.3s | 21.4s | — |
+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 11.5s | 21.7s | 22.8s | — |
+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 10.1s | 23.5s | 19.2s | — |
+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 11.2s | 21.5s | 22.3s | — |
+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 15.6s | 20.3s | 26.5s | — |
+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 8.8s | 19.8s | 23.4s | — |
+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 3.5s | 3.7s | 5.7s | 3.5s |
+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 10.7s | 6.5s | 10.6s | 7.9s |
+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 10.5s | 7.8s | 8.7s | 6.6s |
+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 7.0s | 9.3s | 9.2s | 6.4s |
+
+`results.json` has every run: pass, wall time, model calls, and input, cached input and output tokens. An earlier run of graff, OpenCode 2 and pi on the same model is [graff-vs-opencode-and-pi](../graff-vs-opencode-and-pi/RESULTS.md).
+
+## Method
+
+- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). Each task passes or fails on its own check.
+- **Order.** 3 runs per task and arm. graff, Claude Code and OpenCode 2 were interleaved task by task (`run.py --interleave`), 6 runs at a time; pi ran its coding tasks afterwards. Each arm made one unscored request first.
+- **Wall time.** The runner's clock around each process, start to exit.
+- **Tokens.** Prompt tokens include cached ones in every arm. Claude Code's come from its stream's final result (session totals; Anthropic's `input_tokens` leaves out cache reads and writes, so the wrapper adds them back); its model calls are the distinct API responses. No list cost: the gateway's prices are not public list prices.
+
+## Caveats
+
+- **Claude Code ran through a translation proxy.** The proxy itself added no measurable time: a small request took 1.10s median through it against 1.61s sent straight to the gateway, because it keeps its connection to the gateway open. But Claude Code's own prompt-cache breakpoints do not survive the translation; the gateway applies its own caching, as it does for the other three. Only 49% of Claude Code's MCP input was read from cache (graff's: 96%), so against an endpoint that takes Anthropic Messages directly, Claude Code's MCP times would likely be better. The MCP gap here is an upper bound. Claude Code's token-count requests also fail through the proxy; it carried on without them.
+- **pi runs only the coding tasks.** It has no MCP client.
+- **One model.** Harness differences can look different on another model; the [Codex comparison](../graff-vs-codex-app-server/RESULTS.md) runs graff and pi on a different one.
+- **Small sample.** 3 runs per task, one machine, one key. A second or two per task is within run-to-run noise.
+
+## Reproduce
+
+```sh
+cd graff-evals
+# CODEGRAFF_API_KEY set. Claude Code needs an Anthropic Messages endpoint; with LiteLLM in front
+# of the gateway's chat completions:
+# model_list:
+# - model_name: "*"
+# litellm_params: {model: "openai/<gateway model>", api_base: "https://gateway.codegraff.com/v1",
+# api_key: "os.environ/CODEGRAFF_API_KEY"}
+# litellm_settings: {use_chat_completions_url_for_anthropic_messages: true}
+# litellm --config config.yaml --host 127.0.0.1 --port 4000
+# then ANTHROPIC_BASE_URL=http://127.0.0.1:4000 and any ANTHROPIC_AUTH_TOKEN for the claude-code arm.
+# OpenCode 2 and pi set up as in evals/graff-vs-opencode-and-pi. Keep --output-root outside any
+# repository, with scripts/linear_fixture_mcp.py copied to <parent of output-root>/scripts/.
+TASKS="--task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \
+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \
+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \
+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \
+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm"
+python3 run.py --harness graff-gateway-repl,claude-code,opencode2 --model <gateway model> --suite core,mcp $TASKS \
+ --reps 3 -j 6 --interleave --output-root /path/outside/repo/gateway
+python3 run.py --harness pi-codegraff --model <gateway model> --suite core $TASKS \
+ --reps 3 -j 6 --output-root /path/outside/repo/gateway-pi
+```
diff --git a/evals/graff-vs-claude-code/results.json b/evals/graff-vs-claude-code/results.json
new file mode 100644
index 00000000..01b2d1d9
--- /dev/null
+++ b/evals/graff-vs-claude-code/results.json
@@ -0,0 +1,3504 @@
+{
+ "model": "one gateway model (same id in every arm)",
+ "tasks": 21,
+ "reps": 3,
+ "arms": {
+ "graff": "graff (ADR 0231 build)",
+ "claude-code": "Claude Code",
+ "opencode2": "OpenCode 2",
+ "pi": "pi"
+ },
+ "summary": {
+ "all": {
+ "graff": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 8.7,
+ "cached_input_share": 0.953,
+ "output_tokens_per_task": 573,
+ "model_calls_per_task": 3.5
+ },
+ "claude-code": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 13.2,
+ "cached_input_share": 0.599,
+ "output_tokens_per_task": 926,
+ "model_calls_per_task": 3.5
+ },
+ "opencode2": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 14.1,
+ "cached_input_share": 0.86,
+ "output_tokens_per_task": 978,
+ "model_calls_per_task": 4.3
+ },
+ "pi": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 6.0,
+ "cached_input_share": 0.765,
+ "output_tokens_per_task": 262,
+ "model_calls_per_task": 2.8
+ }
+ },
+ "core": {
+ "graff": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 7.0,
+ "cached_input_share": 0.944,
+ "output_tokens_per_task": 293,
+ "model_calls_per_task": 2.8
+ },
+ "claude-code": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 6.0,
+ "cached_input_share": 0.888,
+ "output_tokens_per_task": 272,
+ "model_calls_per_task": 2.6
+ },
+ "opencode2": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 6.8,
+ "cached_input_share": 0.868,
+ "output_tokens_per_task": 363,
+ "model_calls_per_task": 3.2
+ },
+ "pi": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 6.0,
+ "cached_input_share": 0.765,
+ "output_tokens_per_task": 262,
+ "model_calls_per_task": 2.8
+ }
+ },
+ "mcp": {
+ "graff": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 10.6,
+ "cached_input_share": 0.96,
+ "output_tokens_per_task": 882,
+ "model_calls_per_task": 4.2
+ },
+ "claude-code": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 21.2,
+ "cached_input_share": 0.493,
+ "output_tokens_per_task": 1645,
+ "model_calls_per_task": 4.5
+ },
+ "opencode2": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 22.1,
+ "cached_input_share": 0.856,
+ "output_tokens_per_task": 1654,
+ "model_calls_per_task": 5.5
+ }
+ }
+ },
+ "per_task": {
+ "csv-sum": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.4,
+ "model_calls": 3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.9,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.9,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.7,
+ "model_calls": 3
+ }
+ },
+ "dead-code": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.3,
+ "model_calls": 3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.0,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.8,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.8,
+ "model_calls": 3
+ }
+ },
+ "exact-reply": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.2,
+ "model_calls": 1
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 1.7,
+ "model_calls": 1
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 1.7,
+ "model_calls": 1
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 1.9,
+ "model_calls": 1
+ }
+ },
+ "file-ops": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.8,
+ "model_calls": 2
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.0,
+ "model_calls": 2.7
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.8,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 3.5,
+ "model_calls": 2
+ }
+ },
+ "fix-fib": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.7,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.4,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.9,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 12.0,
+ "model_calls": 4
+ }
+ },
+ "git-ops": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.1,
+ "model_calls": 2
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.6,
+ "model_calls": 2
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.3,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.1,
+ "model_calls": 2.3
+ }
+ },
+ "json-transform": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.8,
+ "model_calls": 3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.7,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.3,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.0,
+ "model_calls": 3
+ }
+ },
+ "linear-issues": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.8,
+ "model_calls": 3.3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.6,
+ "model_calls": 3.7
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.9,
+ "model_calls": 5.7
+ }
+ },
+ "linear-nohint": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 11.1,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.1,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.7,
+ "model_calls": 5.7
+ }
+ },
+ "linear-nohint-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.2,
+ "model_calls": 3.3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.3,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 23.0,
+ "model_calls": 6
+ }
+ },
+ "linear-quiet": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.2,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 28.0,
+ "model_calls": 5
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.2,
+ "model_calls": 5.7
+ }
+ },
+ "linear-quiet-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.8,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.3,
+ "model_calls": 4.3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.4,
+ "model_calls": 5.3
+ }
+ },
+ "linear-reduce": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 11.5,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.7,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.8,
+ "model_calls": 6
+ }
+ },
+ "linear-reduce-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.1,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 23.5,
+ "model_calls": 4.7
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.2,
+ "model_calls": 5
+ }
+ },
+ "linear-sidecar": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 11.2,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.5,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.3,
+ "model_calls": 6
+ }
+ },
+ "linear-split": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 15.6,
+ "model_calls": 7.3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.3,
+ "model_calls": 7
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 26.5,
+ "model_calls": 3
+ }
+ },
+ "linear-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.8,
+ "model_calls": 4
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.8,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 23.4,
+ "model_calls": 6.3
+ }
+ },
+ "recall-noise": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 3.5,
+ "model_calls": 2
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 3.7,
+ "model_calls": 2
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.7,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 3.5,
+ "model_calls": 2
+ }
+ },
+ "refactor-rename": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.7,
+ "model_calls": 5
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.5,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.6,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.9,
+ "model_calls": 4
+ }
+ },
+ "regex-count": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.5,
+ "model_calls": 3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.8,
+ "model_calls": 3.3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.7,
+ "model_calls": 3.3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.6,
+ "model_calls": 3.3
+ }
+ },
+ "write-tests": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.0,
+ "model_calls": 3
+ },
+ "claude-code": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.3,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.2,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.4,
+ "model_calls": 3
+ }
+ }
+ },
+ "runs": [
+ {
+ "arm": "claude-code",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.73,
+ "model_calls": 3,
+ "input_tokens": 67670,
+ "cached_input_tokens": 60999,
+ "output_tokens": 230
+ },
+ {
+ "arm": "claude-code",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.09,
+ "model_calls": 3,
+ "input_tokens": 67671,
+ "cached_input_tokens": 60999,
+ "output_tokens": 236
+ },
+ {
+ "arm": "claude-code",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.02,
+ "model_calls": 3,
+ "input_tokens": 67663,
+ "cached_input_tokens": 60999,
+ "output_tokens": 227
+ },
+ {
+ "arm": "claude-code",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.32,
+ "model_calls": 3,
+ "input_tokens": 67857,
+ "cached_input_tokens": 60993,
+ "output_tokens": 176
+ },
+ {
+ "arm": "claude-code",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.27,
+ "model_calls": 3,
+ "input_tokens": 67851,
+ "cached_input_tokens": 60993,
+ "output_tokens": 170
+ },
+ {
+ "arm": "claude-code",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.43,
+ "model_calls": 3,
+ "input_tokens": 67980,
+ "cached_input_tokens": 60993,
+ "output_tokens": 213
+ },
+ {
+ "arm": "claude-code",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.79,
+ "model_calls": 1,
+ "input_tokens": 22408,
+ "cached_input_tokens": 17105,
+ "output_tokens": 4
+ },
+ {
+ "arm": "claude-code",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.81,
+ "model_calls": 1,
+ "input_tokens": 22409,
+ "cached_input_tokens": 17105,
+ "output_tokens": 4
+ },
+ {
+ "arm": "claude-code",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.62,
+ "model_calls": 1,
+ "input_tokens": 22408,
+ "cached_input_tokens": 17105,
+ "output_tokens": 4
+ },
+ {
+ "arm": "claude-code",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.44,
+ "model_calls": 3,
+ "input_tokens": 67824,
+ "cached_input_tokens": 60993,
+ "output_tokens": 278
+ },
+ {
+ "arm": "claude-code",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.18,
+ "model_calls": 3,
+ "input_tokens": 67818,
+ "cached_input_tokens": 60993,
+ "output_tokens": 277
+ },
+ {
+ "arm": "claude-code",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.45,
+ "model_calls": 2,
+ "input_tokens": 45024,
+ "cached_input_tokens": 39038,
+ "output_tokens": 227
+ },
+ {
+ "arm": "claude-code",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.29,
+ "model_calls": 3,
+ "input_tokens": 68239,
+ "cached_input_tokens": 60999,
+ "output_tokens": 371
+ },
+ {
+ "arm": "claude-code",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.64,
+ "model_calls": 3,
+ "input_tokens": 68239,
+ "cached_input_tokens": 60999,
+ "output_tokens": 372
+ },
+ {
+ "arm": "claude-code",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.35,
+ "model_calls": 3,
+ "input_tokens": 68233,
+ "cached_input_tokens": 60999,
+ "output_tokens": 371
+ },
+ {
+ "arm": "claude-code",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.69,
+ "model_calls": 2,
+ "input_tokens": 45117,
+ "cached_input_tokens": 39038,
+ "output_tokens": 226
+ },
+ {
+ "arm": "claude-code",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.95,
+ "model_calls": 2,
+ "input_tokens": 45098,
+ "cached_input_tokens": 39038,
+ "output_tokens": 276
+ },
+ {
+ "arm": "claude-code",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.16,
+ "model_calls": 2,
+ "input_tokens": 45100,
+ "cached_input_tokens": 39038,
+ "output_tokens": 239
+ },
+ {
+ "arm": "claude-code",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.31,
+ "model_calls": 3,
+ "input_tokens": 67944,
+ "cached_input_tokens": 60993,
+ "output_tokens": 339
+ },
+ {
+ "arm": "claude-code",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.74,
+ "model_calls": 3,
+ "input_tokens": 67953,
+ "cached_input_tokens": 60993,
+ "output_tokens": 318
+ },
+ {
+ "arm": "claude-code",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.94,
+ "model_calls": 3,
+ "input_tokens": 67944,
+ "cached_input_tokens": 60993,
+ "output_tokens": 325
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.82,
+ "model_calls": 4,
+ "input_tokens": 169851,
+ "cached_input_tokens": 83758,
+ "output_tokens": 1556
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.87,
+ "model_calls": 3,
+ "input_tokens": 144910,
+ "cached_input_tokens": 61584,
+ "output_tokens": 1366
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.25,
+ "model_calls": 4,
+ "input_tokens": 170065,
+ "cached_input_tokens": 83758,
+ "output_tokens": 1678
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.68,
+ "model_calls": 4,
+ "input_tokens": 169849,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1674
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.8,
+ "model_calls": 4,
+ "input_tokens": 169408,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1382
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.79,
+ "model_calls": 4,
+ "input_tokens": 205410,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1417
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.48,
+ "model_calls": 4,
+ "input_tokens": 169460,
+ "cached_input_tokens": 83794,
+ "output_tokens": 1503
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.62,
+ "model_calls": 4,
+ "input_tokens": 206396,
+ "cached_input_tokens": 83794,
+ "output_tokens": 1824
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.79,
+ "model_calls": 4,
+ "input_tokens": 205770,
+ "cached_input_tokens": 83794,
+ "output_tokens": 1473
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.49,
+ "model_calls": 4,
+ "input_tokens": 169392,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1526
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.45,
+ "model_calls": 6,
+ "input_tokens": 293424,
+ "cached_input_tokens": 128184,
+ "output_tokens": 2361
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.17,
+ "model_calls": 5,
+ "input_tokens": 231177,
+ "cached_input_tokens": 105965,
+ "output_tokens": 1863
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.64,
+ "model_calls": 5,
+ "input_tokens": 231129,
+ "cached_input_tokens": 106001,
+ "output_tokens": 1933
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.69,
+ "model_calls": 4,
+ "input_tokens": 169413,
+ "cached_input_tokens": 83794,
+ "output_tokens": 1402
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.51,
+ "model_calls": 4,
+ "input_tokens": 169464,
+ "cached_input_tokens": 83794,
+ "output_tokens": 1492
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.47,
+ "model_calls": 4,
+ "input_tokens": 170042,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1456
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.0,
+ "model_calls": 4,
+ "input_tokens": 170234,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1748
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.49,
+ "model_calls": 4,
+ "input_tokens": 169913,
+ "cached_input_tokens": 83767,
+ "output_tokens": 1555
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.94,
+ "model_calls": 5,
+ "input_tokens": 232235,
+ "cached_input_tokens": 106001,
+ "output_tokens": 2258
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.32,
+ "model_calls": 5,
+ "input_tokens": 231251,
+ "cached_input_tokens": 106001,
+ "output_tokens": 1915
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.24,
+ "model_calls": 4,
+ "input_tokens": 170289,
+ "cached_input_tokens": 83794,
+ "output_tokens": 1792
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.81,
+ "model_calls": 4,
+ "input_tokens": 170071,
+ "cached_input_tokens": 83776,
+ "output_tokens": 1629
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.52,
+ "model_calls": 4,
+ "input_tokens": 169813,
+ "cached_input_tokens": 83776,
+ "output_tokens": 1443
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.23,
+ "model_calls": 4,
+ "input_tokens": 170174,
+ "cached_input_tokens": 83776,
+ "output_tokens": 1679
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.53,
+ "model_calls": 6,
+ "input_tokens": 100734,
+ "cached_input_tokens": 83749,
+ "output_tokens": 1528
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.18,
+ "model_calls": 9,
+ "input_tokens": 131230,
+ "cached_input_tokens": 107157,
+ "output_tokens": 1627
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.33,
+ "model_calls": 6,
+ "input_tokens": 100546,
+ "cached_input_tokens": 83749,
+ "output_tokens": 1494
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.95,
+ "model_calls": 4,
+ "input_tokens": 169963,
+ "cached_input_tokens": 83758,
+ "output_tokens": 1534
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.16,
+ "model_calls": 4,
+ "input_tokens": 169806,
+ "cached_input_tokens": 83758,
+ "output_tokens": 1394
+ },
+ {
+ "arm": "claude-code",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.18,
+ "model_calls": 4,
+ "input_tokens": 170101,
+ "cached_input_tokens": 83758,
+ "output_tokens": 1847
+ },
+ {
+ "arm": "claude-code",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.76,
+ "model_calls": 2,
+ "input_tokens": 45330,
+ "cached_input_tokens": 39041,
+ "output_tokens": 104
+ },
+ {
+ "arm": "claude-code",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.15,
+ "model_calls": 2,
+ "input_tokens": 45330,
+ "cached_input_tokens": 39041,
+ "output_tokens": 104
+ },
+ {
+ "arm": "claude-code",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.25,
+ "model_calls": 2,
+ "input_tokens": 45328,
+ "cached_input_tokens": 39041,
+ "output_tokens": 104
+ },
+ {
+ "arm": "claude-code",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.14,
+ "model_calls": 3,
+ "input_tokens": 67833,
+ "cached_input_tokens": 61005,
+ "output_tokens": 254
+ },
+ {
+ "arm": "claude-code",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.98,
+ "model_calls": 3,
+ "input_tokens": 67827,
+ "cached_input_tokens": 61005,
+ "output_tokens": 248
+ },
+ {
+ "arm": "claude-code",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.3,
+ "model_calls": 3,
+ "input_tokens": 67827,
+ "cached_input_tokens": 61005,
+ "output_tokens": 251
+ },
+ {
+ "arm": "claude-code",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.11,
+ "model_calls": 3,
+ "input_tokens": 67999,
+ "cached_input_tokens": 60993,
+ "output_tokens": 399
+ },
+ {
+ "arm": "claude-code",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.21,
+ "model_calls": 4,
+ "input_tokens": 90726,
+ "cached_input_tokens": 82969,
+ "output_tokens": 396
+ },
+ {
+ "arm": "claude-code",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.19,
+ "model_calls": 3,
+ "input_tokens": 67999,
+ "cached_input_tokens": 60993,
+ "output_tokens": 358
+ },
+ {
+ "arm": "claude-code",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.28,
+ "model_calls": 3,
+ "input_tokens": 68157,
+ "cached_input_tokens": 60993,
+ "output_tokens": 636
+ },
+ {
+ "arm": "claude-code",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.78,
+ "model_calls": 3,
+ "input_tokens": 68166,
+ "cached_input_tokens": 60993,
+ "output_tokens": 618
+ },
+ {
+ "arm": "claude-code",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.88,
+ "model_calls": 3,
+ "input_tokens": 68160,
+ "cached_input_tokens": 60993,
+ "output_tokens": 637
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.05,
+ "model_calls": 3,
+ "input_tokens": 27862,
+ "cached_input_tokens": 22546,
+ "output_tokens": 393
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.82,
+ "model_calls": 3,
+ "input_tokens": 27839,
+ "cached_input_tokens": 27756,
+ "output_tokens": 303
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.27,
+ "model_calls": 3,
+ "input_tokens": 27927,
+ "cached_input_tokens": 27756,
+ "output_tokens": 341
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.31,
+ "model_calls": 3,
+ "input_tokens": 28218,
+ "cached_input_tokens": 22632,
+ "output_tokens": 162
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.86,
+ "model_calls": 3,
+ "input_tokens": 28228,
+ "cached_input_tokens": 27748,
+ "output_tokens": 165
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.6,
+ "model_calls": 3,
+ "input_tokens": 28218,
+ "cached_input_tokens": 28210,
+ "output_tokens": 160
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.27,
+ "model_calls": 1,
+ "input_tokens": 9121,
+ "cached_input_tokens": 9117,
+ "output_tokens": 51
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.74,
+ "model_calls": 1,
+ "input_tokens": 9121,
+ "cached_input_tokens": 9117,
+ "output_tokens": 51
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.73,
+ "model_calls": 1,
+ "input_tokens": 9121,
+ "cached_input_tokens": 9117,
+ "output_tokens": 51
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.5,
+ "model_calls": 2,
+ "input_tokens": 18395,
+ "cached_input_tokens": 18186,
+ "output_tokens": 162
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.2,
+ "model_calls": 2,
+ "input_tokens": 18395,
+ "cached_input_tokens": 18389,
+ "output_tokens": 145
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 13.55,
+ "model_calls": 2,
+ "input_tokens": 18395,
+ "cached_input_tokens": 18389,
+ "output_tokens": 148
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.33,
+ "model_calls": 4,
+ "input_tokens": 38157,
+ "cached_input_tokens": 32356,
+ "output_tokens": 386
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.79,
+ "model_calls": 4,
+ "input_tokens": 38157,
+ "cached_input_tokens": 37477,
+ "output_tokens": 388
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.91,
+ "model_calls": 4,
+ "input_tokens": 38157,
+ "cached_input_tokens": 37477,
+ "output_tokens": 388
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.47,
+ "model_calls": 2,
+ "input_tokens": 18597,
+ "cached_input_tokens": 13279,
+ "output_tokens": 311
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.13,
+ "model_calls": 2,
+ "input_tokens": 18594,
+ "cached_input_tokens": 18398,
+ "output_tokens": 266
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.71,
+ "model_calls": 2,
+ "input_tokens": 18590,
+ "cached_input_tokens": 18398,
+ "output_tokens": 240
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.04,
+ "model_calls": 3,
+ "input_tokens": 27943,
+ "cached_input_tokens": 27935,
+ "output_tokens": 240
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.62,
+ "model_calls": 3,
+ "input_tokens": 27943,
+ "cached_input_tokens": 27935,
+ "output_tokens": 273
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.63,
+ "model_calls": 3,
+ "input_tokens": 27943,
+ "cached_input_tokens": 27935,
+ "output_tokens": 252
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.63,
+ "model_calls": 3,
+ "input_tokens": 30825,
+ "cached_input_tokens": 29264,
+ "output_tokens": 500
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.04,
+ "model_calls": 4,
+ "input_tokens": 41510,
+ "cached_input_tokens": 40531,
+ "output_tokens": 723
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.73,
+ "model_calls": 3,
+ "input_tokens": 30744,
+ "cached_input_tokens": 30536,
+ "output_tokens": 637
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.56,
+ "model_calls": 4,
+ "input_tokens": 41221,
+ "cached_input_tokens": 39503,
+ "output_tokens": 856
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.37,
+ "model_calls": 4,
+ "input_tokens": 41244,
+ "cached_input_tokens": 40545,
+ "output_tokens": 868
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.3,
+ "model_calls": 4,
+ "input_tokens": 41221,
+ "cached_input_tokens": 40925,
+ "output_tokens": 846
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.75,
+ "model_calls": 4,
+ "input_tokens": 42475,
+ "cached_input_tokens": 40513,
+ "output_tokens": 798
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.19,
+ "model_calls": 3,
+ "input_tokens": 32072,
+ "cached_input_tokens": 30981,
+ "output_tokens": 959
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.52,
+ "model_calls": 3,
+ "input_tokens": 31797,
+ "cached_input_tokens": 31503,
+ "output_tokens": 683
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.71,
+ "model_calls": 4,
+ "input_tokens": 41002,
+ "cached_input_tokens": 40808,
+ "output_tokens": 613
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.96,
+ "model_calls": 4,
+ "input_tokens": 41013,
+ "cached_input_tokens": 40808,
+ "output_tokens": 587
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.84,
+ "model_calls": 4,
+ "input_tokens": 41002,
+ "cached_input_tokens": 40299,
+ "output_tokens": 514
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.61,
+ "model_calls": 4,
+ "input_tokens": 43108,
+ "cached_input_tokens": 40900,
+ "output_tokens": 744
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.05,
+ "model_calls": 4,
+ "input_tokens": 43108,
+ "cached_input_tokens": 43098,
+ "output_tokens": 683
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.86,
+ "model_calls": 4,
+ "input_tokens": 43108,
+ "cached_input_tokens": 43098,
+ "output_tokens": 645
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.76,
+ "model_calls": 4,
+ "input_tokens": 41266,
+ "cached_input_tokens": 39561,
+ "output_tokens": 765
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.79,
+ "model_calls": 4,
+ "input_tokens": 41308,
+ "cached_input_tokens": 40370,
+ "output_tokens": 869
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.84,
+ "model_calls": 4,
+ "input_tokens": 41229,
+ "cached_input_tokens": 40477,
+ "output_tokens": 701
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.32,
+ "model_calls": 4,
+ "input_tokens": 42800,
+ "cached_input_tokens": 40737,
+ "output_tokens": 729
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.21,
+ "model_calls": 4,
+ "input_tokens": 42819,
+ "cached_input_tokens": 41903,
+ "output_tokens": 751
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.7,
+ "model_calls": 4,
+ "input_tokens": 42818,
+ "cached_input_tokens": 42613,
+ "output_tokens": 588
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.15,
+ "model_calls": 5,
+ "input_tokens": 63150,
+ "cached_input_tokens": 58966,
+ "output_tokens": 1302
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.74,
+ "model_calls": 4,
+ "input_tokens": 49882,
+ "cached_input_tokens": 48906,
+ "output_tokens": 1173
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.66,
+ "model_calls": 3,
+ "input_tokens": 37282,
+ "cached_input_tokens": 36266,
+ "output_tokens": 833
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.27,
+ "model_calls": 7,
+ "input_tokens": 63761,
+ "cached_input_tokens": 46131,
+ "output_tokens": 1874
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.49,
+ "model_calls": 7,
+ "input_tokens": 63887,
+ "cached_input_tokens": 60554,
+ "output_tokens": 2014
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.96,
+ "model_calls": 8,
+ "input_tokens": 77784,
+ "cached_input_tokens": 73972,
+ "output_tokens": 2184
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.36,
+ "model_calls": 4,
+ "input_tokens": 43475,
+ "cached_input_tokens": 41215,
+ "output_tokens": 741
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.46,
+ "model_calls": 4,
+ "input_tokens": 43085,
+ "cached_input_tokens": 42111,
+ "output_tokens": 590
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.55,
+ "model_calls": 4,
+ "input_tokens": 43098,
+ "cached_input_tokens": 42654,
+ "output_tokens": 684
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.74,
+ "model_calls": 2,
+ "input_tokens": 18747,
+ "cached_input_tokens": 13253,
+ "output_tokens": 110
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.27,
+ "model_calls": 2,
+ "input_tokens": 18747,
+ "cached_input_tokens": 18741,
+ "output_tokens": 110
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.4,
+ "model_calls": 2,
+ "input_tokens": 18747,
+ "cached_input_tokens": 18741,
+ "output_tokens": 110
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.42,
+ "model_calls": 5,
+ "input_tokens": 48198,
+ "cached_input_tokens": 42190,
+ "output_tokens": 665
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.7,
+ "model_calls": 5,
+ "input_tokens": 48174,
+ "cached_input_tokens": 47635,
+ "output_tokens": 614
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.1,
+ "model_calls": 5,
+ "input_tokens": 48194,
+ "cached_input_tokens": 47316,
+ "output_tokens": 656
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.27,
+ "model_calls": 3,
+ "input_tokens": 28160,
+ "cached_input_tokens": 22709,
+ "output_tokens": 423
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.78,
+ "model_calls": 3,
+ "input_tokens": 28085,
+ "cached_input_tokens": 27811,
+ "output_tokens": 327
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.54,
+ "model_calls": 3,
+ "input_tokens": 28146,
+ "cached_input_tokens": 27829,
+ "output_tokens": 435
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.09,
+ "model_calls": 3,
+ "input_tokens": 28080,
+ "cached_input_tokens": 22556,
+ "output_tokens": 440
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.75,
+ "model_calls": 3,
+ "input_tokens": 28080,
+ "cached_input_tokens": 28072,
+ "output_tokens": 482
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.03,
+ "model_calls": 3,
+ "input_tokens": 28080,
+ "cached_input_tokens": 28072,
+ "output_tokens": 415
+ },
+ {
+ "arm": "opencode2",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.36,
+ "model_calls": 3,
+ "input_tokens": 23865,
+ "cached_input_tokens": 20630,
+ "output_tokens": 282
+ },
+ {
+ "arm": "opencode2",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.79,
+ "model_calls": 3,
+ "input_tokens": 23865,
+ "cached_input_tokens": 20671,
+ "output_tokens": 321
+ },
+ {
+ "arm": "opencode2",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.48,
+ "model_calls": 3,
+ "input_tokens": 23865,
+ "cached_input_tokens": 20671,
+ "output_tokens": 324
+ },
+ {
+ "arm": "opencode2",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.29,
+ "model_calls": 4,
+ "input_tokens": 32305,
+ "cached_input_tokens": 28619,
+ "output_tokens": 289
+ },
+ {
+ "arm": "opencode2",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.09,
+ "model_calls": 4,
+ "input_tokens": 32305,
+ "cached_input_tokens": 28671,
+ "output_tokens": 289
+ },
+ {
+ "arm": "opencode2",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.03,
+ "model_calls": 4,
+ "input_tokens": 32301,
+ "cached_input_tokens": 28669,
+ "output_tokens": 287
+ },
+ {
+ "arm": "opencode2",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.87,
+ "model_calls": 1,
+ "input_tokens": 7484,
+ "cached_input_tokens": 4644,
+ "output_tokens": 4
+ },
+ {
+ "arm": "opencode2",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.57,
+ "model_calls": 1,
+ "input_tokens": 7484,
+ "cached_input_tokens": 4644,
+ "output_tokens": 4
+ },
+ {
+ "arm": "opencode2",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.79,
+ "model_calls": 1,
+ "input_tokens": 7484,
+ "cached_input_tokens": 4644,
+ "output_tokens": 4
+ },
+ {
+ "arm": "opencode2",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.24,
+ "model_calls": 3,
+ "input_tokens": 23744,
+ "cached_input_tokens": 20568,
+ "output_tokens": 286
+ },
+ {
+ "arm": "opencode2",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.35,
+ "model_calls": 3,
+ "input_tokens": 23744,
+ "cached_input_tokens": 20614,
+ "output_tokens": 276
+ },
+ {
+ "arm": "opencode2",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.7,
+ "model_calls": 3,
+ "input_tokens": 23744,
+ "cached_input_tokens": 20614,
+ "output_tokens": 311
+ },
+ {
+ "arm": "opencode2",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.52,
+ "model_calls": 4,
+ "input_tokens": 32294,
+ "cached_input_tokens": 28798,
+ "output_tokens": 355
+ },
+ {
+ "arm": "opencode2",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.35,
+ "model_calls": 4,
+ "input_tokens": 32294,
+ "cached_input_tokens": 28857,
+ "output_tokens": 344
+ },
+ {
+ "arm": "opencode2",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.76,
+ "model_calls": 4,
+ "input_tokens": 32294,
+ "cached_input_tokens": 28857,
+ "output_tokens": 358
+ },
+ {
+ "arm": "opencode2",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.21,
+ "model_calls": 3,
+ "input_tokens": 24080,
+ "cached_input_tokens": 20623,
+ "output_tokens": 357
+ },
+ {
+ "arm": "opencode2",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.95,
+ "model_calls": 3,
+ "input_tokens": 24075,
+ "cached_input_tokens": 20691,
+ "output_tokens": 340
+ },
+ {
+ "arm": "opencode2",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.76,
+ "model_calls": 3,
+ "input_tokens": 24084,
+ "cached_input_tokens": 20691,
+ "output_tokens": 332
+ },
+ {
+ "arm": "opencode2",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.34,
+ "model_calls": 3,
+ "input_tokens": 23966,
+ "cached_input_tokens": 20665,
+ "output_tokens": 383
+ },
+ {
+ "arm": "opencode2",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.81,
+ "model_calls": 3,
+ "input_tokens": 23966,
+ "cached_input_tokens": 20719,
+ "output_tokens": 344
+ },
+ {
+ "arm": "opencode2",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.66,
+ "model_calls": 3,
+ "input_tokens": 23966,
+ "cached_input_tokens": 20719,
+ "output_tokens": 384
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.4,
+ "model_calls": 6,
+ "input_tokens": 65814,
+ "cached_input_tokens": 57075,
+ "output_tokens": 1818
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.99,
+ "model_calls": 6,
+ "input_tokens": 65767,
+ "cached_input_tokens": 57326,
+ "output_tokens": 1745
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.43,
+ "model_calls": 5,
+ "input_tokens": 53853,
+ "cached_input_tokens": 45358,
+ "output_tokens": 1783
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.09,
+ "model_calls": 5,
+ "input_tokens": 57856,
+ "cached_input_tokens": 46884,
+ "output_tokens": 1634
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.06,
+ "model_calls": 6,
+ "input_tokens": 65561,
+ "cached_input_tokens": 56903,
+ "output_tokens": 1742
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.06,
+ "model_calls": 6,
+ "input_tokens": 68449,
+ "cached_input_tokens": 58710,
+ "output_tokens": 1726
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.95,
+ "model_calls": 6,
+ "input_tokens": 66382,
+ "cached_input_tokens": 57487,
+ "output_tokens": 1624
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.19,
+ "model_calls": 6,
+ "input_tokens": 65088,
+ "cached_input_tokens": 56446,
+ "output_tokens": 1658
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.98,
+ "model_calls": 6,
+ "input_tokens": 66345,
+ "cached_input_tokens": 57604,
+ "output_tokens": 1644
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.61,
+ "model_calls": 5,
+ "input_tokens": 45917,
+ "cached_input_tokens": 40113,
+ "output_tokens": 1443
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.39,
+ "model_calls": 6,
+ "input_tokens": 58273,
+ "cached_input_tokens": 51693,
+ "output_tokens": 1867
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.64,
+ "model_calls": 6,
+ "input_tokens": 57350,
+ "cached_input_tokens": 50964,
+ "output_tokens": 1849
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.65,
+ "model_calls": 5,
+ "input_tokens": 47358,
+ "cached_input_tokens": 41341,
+ "output_tokens": 1761
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.65,
+ "model_calls": 5,
+ "input_tokens": 47471,
+ "cached_input_tokens": 41447,
+ "output_tokens": 1974
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.93,
+ "model_calls": 6,
+ "input_tokens": 58086,
+ "cached_input_tokens": 51508,
+ "output_tokens": 1783
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.34,
+ "model_calls": 6,
+ "input_tokens": 56123,
+ "cached_input_tokens": 49789,
+ "output_tokens": 1798
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.47,
+ "model_calls": 6,
+ "input_tokens": 59512,
+ "cached_input_tokens": 52199,
+ "output_tokens": 1901
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.54,
+ "model_calls": 6,
+ "input_tokens": 57388,
+ "cached_input_tokens": 50882,
+ "output_tokens": 1990
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.16,
+ "model_calls": 6,
+ "input_tokens": 57297,
+ "cached_input_tokens": 51036,
+ "output_tokens": 1853
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.99,
+ "model_calls": 4,
+ "input_tokens": 36639,
+ "cached_input_tokens": 31328,
+ "output_tokens": 1681
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.47,
+ "model_calls": 5,
+ "input_tokens": 47319,
+ "cached_input_tokens": 41186,
+ "output_tokens": 1897
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.03,
+ "model_calls": 6,
+ "input_tokens": 67258,
+ "cached_input_tokens": 57985,
+ "output_tokens": 1889
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.89,
+ "model_calls": 6,
+ "input_tokens": 67495,
+ "cached_input_tokens": 58046,
+ "output_tokens": 1789
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.86,
+ "model_calls": 6,
+ "input_tokens": 66217,
+ "cached_input_tokens": 57489,
+ "output_tokens": 1838
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.72,
+ "model_calls": 3,
+ "input_tokens": 21547,
+ "cached_input_tokens": 13793,
+ "output_tokens": 594
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.44,
+ "model_calls": 3,
+ "input_tokens": 20988,
+ "cached_input_tokens": 16802,
+ "output_tokens": 406
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.32,
+ "model_calls": 3,
+ "input_tokens": 32210,
+ "cached_input_tokens": 22190,
+ "output_tokens": 666
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.66,
+ "model_calls": 6,
+ "input_tokens": 69118,
+ "cached_input_tokens": 59552,
+ "output_tokens": 1774
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.66,
+ "model_calls": 6,
+ "input_tokens": 66240,
+ "cached_input_tokens": 57633,
+ "output_tokens": 1691
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.87,
+ "model_calls": 7,
+ "input_tokens": 271673,
+ "cached_input_tokens": 224144,
+ "output_tokens": 1799
+ },
+ {
+ "arm": "opencode2",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.36,
+ "model_calls": 3,
+ "input_tokens": 24029,
+ "cached_input_tokens": 20531,
+ "output_tokens": 167
+ },
+ {
+ "arm": "opencode2",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.66,
+ "model_calls": 3,
+ "input_tokens": 24029,
+ "cached_input_tokens": 20567,
+ "output_tokens": 167
+ },
+ {
+ "arm": "opencode2",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.97,
+ "model_calls": 3,
+ "input_tokens": 24029,
+ "cached_input_tokens": 20567,
+ "output_tokens": 167
+ },
+ {
+ "arm": "opencode2",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.86,
+ "model_calls": 4,
+ "input_tokens": 32910,
+ "cached_input_tokens": 29006,
+ "output_tokens": 692
+ },
+ {
+ "arm": "opencode2",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.67,
+ "model_calls": 4,
+ "input_tokens": 32878,
+ "cached_input_tokens": 29060,
+ "output_tokens": 679
+ },
+ {
+ "arm": "opencode2",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.22,
+ "model_calls": 4,
+ "input_tokens": 32966,
+ "cached_input_tokens": 29104,
+ "output_tokens": 716
+ },
+ {
+ "arm": "opencode2",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.44,
+ "model_calls": 3,
+ "input_tokens": 24257,
+ "cached_input_tokens": 20810,
+ "output_tokens": 519
+ },
+ {
+ "arm": "opencode2",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.48,
+ "model_calls": 4,
+ "input_tokens": 32509,
+ "cached_input_tokens": 28973,
+ "output_tokens": 539
+ },
+ {
+ "arm": "opencode2",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.06,
+ "model_calls": 3,
+ "input_tokens": 24324,
+ "cached_input_tokens": 20935,
+ "output_tokens": 454
+ },
+ {
+ "arm": "opencode2",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.76,
+ "model_calls": 4,
+ "input_tokens": 32026,
+ "cached_input_tokens": 28378,
+ "output_tokens": 710
+ },
+ {
+ "arm": "opencode2",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.63,
+ "model_calls": 4,
+ "input_tokens": 31983,
+ "cached_input_tokens": 28440,
+ "output_tokens": 628
+ },
+ {
+ "arm": "opencode2",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.35,
+ "model_calls": 4,
+ "input_tokens": 32019,
+ "cached_input_tokens": 28438,
+ "output_tokens": 671
+ },
+ {
+ "arm": "pi",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.57,
+ "model_calls": 3,
+ "input_tokens": 7198,
+ "cached_input_tokens": 5896,
+ "output_tokens": 224
+ },
+ {
+ "arm": "pi",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.53,
+ "model_calls": 3,
+ "input_tokens": 7223,
+ "cached_input_tokens": 5896,
+ "output_tokens": 230
+ },
+ {
+ "arm": "pi",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.99,
+ "model_calls": 3,
+ "input_tokens": 7223,
+ "cached_input_tokens": 5896,
+ "output_tokens": 204
+ },
+ {
+ "arm": "pi",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.56,
+ "model_calls": 3,
+ "input_tokens": 7558,
+ "cached_input_tokens": 5984,
+ "output_tokens": 231
+ },
+ {
+ "arm": "pi",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.37,
+ "model_calls": 3,
+ "input_tokens": 7536,
+ "cached_input_tokens": 5973,
+ "output_tokens": 220
+ },
+ {
+ "arm": "pi",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.41,
+ "model_calls": 3,
+ "input_tokens": 7572,
+ "cached_input_tokens": 5991,
+ "output_tokens": 238
+ },
+ {
+ "arm": "pi",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 2.13,
+ "model_calls": 1,
+ "input_tokens": 2271,
+ "cached_input_tokens": 0,
+ "output_tokens": 4
+ },
+ {
+ "arm": "pi",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.73,
+ "model_calls": 1,
+ "input_tokens": 2271,
+ "cached_input_tokens": 0,
+ "output_tokens": 4
+ },
+ {
+ "arm": "pi",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.73,
+ "model_calls": 1,
+ "input_tokens": 2271,
+ "cached_input_tokens": 0,
+ "output_tokens": 4
+ },
+ {
+ "arm": "pi",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.34,
+ "model_calls": 2,
+ "input_tokens": 4746,
+ "cached_input_tokens": 3489,
+ "output_tokens": 168
+ },
+ {
+ "arm": "pi",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.71,
+ "model_calls": 2,
+ "input_tokens": 4746,
+ "cached_input_tokens": 3489,
+ "output_tokens": 164
+ },
+ {
+ "arm": "pi",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.37,
+ "model_calls": 2,
+ "input_tokens": 4746,
+ "cached_input_tokens": 3489,
+ "output_tokens": 168
+ },
+ {
+ "arm": "pi",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.83,
+ "model_calls": 4,
+ "input_tokens": 10457,
+ "cached_input_tokens": 8722,
+ "output_tokens": 432
+ },
+ {
+ "arm": "pi",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.32,
+ "model_calls": 4,
+ "input_tokens": 10449,
+ "cached_input_tokens": 7532,
+ "output_tokens": 417
+ },
+ {
+ "arm": "pi",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.91,
+ "model_calls": 4,
+ "input_tokens": 10434,
+ "cached_input_tokens": 7532,
+ "output_tokens": 464
+ },
+ {
+ "arm": "pi",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.9,
+ "model_calls": 2,
+ "input_tokens": 5016,
+ "cached_input_tokens": 3507,
+ "output_tokens": 286
+ },
+ {
+ "arm": "pi",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.01,
+ "model_calls": 3,
+ "input_tokens": 7451,
+ "cached_input_tokens": 5897,
+ "output_tokens": 330
+ },
+ {
+ "arm": "pi",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.27,
+ "model_calls": 2,
+ "input_tokens": 5009,
+ "cached_input_tokens": 3507,
+ "output_tokens": 274
+ },
+ {
+ "arm": "pi",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.74,
+ "model_calls": 3,
+ "input_tokens": 7341,
+ "cached_input_tokens": 5927,
+ "output_tokens": 281
+ },
+ {
+ "arm": "pi",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.47,
+ "model_calls": 3,
+ "input_tokens": 7298,
+ "cached_input_tokens": 5927,
+ "output_tokens": 238
+ },
+ {
+ "arm": "pi",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.7,
+ "model_calls": 3,
+ "input_tokens": 7357,
+ "cached_input_tokens": 5934,
+ "output_tokens": 240
+ },
+ {
+ "arm": "pi",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.6,
+ "model_calls": 2,
+ "input_tokens": 4979,
+ "cached_input_tokens": 3480,
+ "output_tokens": 61
+ },
+ {
+ "arm": "pi",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.68,
+ "model_calls": 2,
+ "input_tokens": 4979,
+ "cached_input_tokens": 3480,
+ "output_tokens": 61
+ },
+ {
+ "arm": "pi",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.19,
+ "model_calls": 2,
+ "input_tokens": 4979,
+ "cached_input_tokens": 3480,
+ "output_tokens": 61
+ },
+ {
+ "arm": "pi",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.65,
+ "model_calls": 4,
+ "input_tokens": 10136,
+ "cached_input_tokens": 8537,
+ "output_tokens": 538
+ },
+ {
+ "arm": "pi",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.25,
+ "model_calls": 4,
+ "input_tokens": 10226,
+ "cached_input_tokens": 8601,
+ "output_tokens": 384
+ },
+ {
+ "arm": "pi",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.75,
+ "model_calls": 4,
+ "input_tokens": 10230,
+ "cached_input_tokens": 8579,
+ "output_tokens": 541
+ },
+ {
+ "arm": "pi",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.21,
+ "model_calls": 3,
+ "input_tokens": 7399,
+ "cached_input_tokens": 6012,
+ "output_tokens": 257
+ },
+ {
+ "arm": "pi",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.97,
+ "model_calls": 4,
+ "input_tokens": 10428,
+ "cached_input_tokens": 8756,
+ "output_tokens": 518
+ },
+ {
+ "arm": "pi",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.51,
+ "model_calls": 3,
+ "input_tokens": 7389,
+ "cached_input_tokens": 5993,
+ "output_tokens": 316
+ },
+ {
+ "arm": "pi",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.82,
+ "model_calls": 3,
+ "input_tokens": 7324,
+ "cached_input_tokens": 5903,
+ "output_tokens": 386
+ },
+ {
+ "arm": "pi",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.09,
+ "model_calls": 3,
+ "input_tokens": 7324,
+ "cached_input_tokens": 5903,
+ "output_tokens": 357
+ },
+ {
+ "arm": "pi",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.22,
+ "model_calls": 3,
+ "input_tokens": 7324,
+ "cached_input_tokens": 5903,
+ "output_tokens": 350
+ }
+ ]
+}
\ No newline at end of file
diff --git a/graff-evals/README.md b/graff-evals/README.md
index e1af4f1c..19e0d2a4 100644
--- a/graff-evals/README.md
+++ b/graff-evals/README.md
@@ -56,9 +56,10 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co
# Results: evals/graff-vs-codex-app-server/RESULTS.md
./run.py --suite mcp --harness graff-repl,codex-app-server -j 8 --interleave
./run.py --suite core --harness graff-repl,codex-app-server,pi-codex -j 8 --interleave
-# one gateway model in graff, OpenCode 2 and pi (set CODEGRAFF_API_KEY; OPENCODE2_BIN
-# when `opencode` on PATH is 1.x). Results: evals/graff-vs-opencode-and-pi/RESULTS.md
-./run.py --suite core,mcp --harness graff-gateway-repl,opencode2 --model <gateway model> -j 6 --interleave
+# one gateway model in graff, Claude Code, OpenCode 2 and pi (set CODEGRAFF_API_KEY; OPENCODE2_BIN
+# when `opencode` on PATH is 1.x; Claude Code needs an Anthropic-format proxy in front of the
+# gateway, see the write-up). Results: evals/graff-vs-claude-code/RESULTS.md
+./run.py --suite core,mcp --harness graff-gateway-repl,claude-code,opencode2 --model <gateway model> -j 6 --interleave
./run.py --suite core --harness pi-codegraff --model <gateway model> -j 6
```
diff --git a/graff-evals/claude_code_run.py b/graff-evals/claude_code_run.py
new file mode 100644
index 00000000..eef9b14c
--- /dev/null
+++ b/graff-evals/claude_code_run.py
@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+"""Run Claude Code headless on one eval task and report like graff does.
+
+usage: claude_code_run.py MODEL PROMPT (cwd = the task sandbox)
+
+`claude -p --output-format stream-json --verbose`, permissions skipped, with
+the task's `.mcp.json` as its only MCP config. The final result text goes to
+stdout; a graff-format `[usage]` line goes to stderr. Calls are the API
+responses (one message id each); tokens are the result event's session totals.
+Anthropic's `input_tokens` leaves out cache reads and writes, so they are
+added back to make the total prompt tokens. stdin is empty. Point Claude Code at an endpoint with the
+usual environment (ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN).
+"""
+import json, os, subprocess, sys
+
+CLAUDE = os.environ.get("CLAUDE_CODE_BIN", "claude")
+model, prompt = sys.argv[1], sys.argv[2]
+
+cmd = [CLAUDE, "-p", prompt, "--model", model, "--dangerously-skip-permissions",
+ "--output-format", "stream-json", "--verbose"]
+if os.path.exists(".mcp.json"):
+ cmd += ["--mcp-config", ".mcp.json", "--strict-mcp-config"]
+
+r = subprocess.run(cmd, stdin=subprocess.DEVNULL, capture_output=True, text=True)
+per_message, answer, result = {}, "", None
+for line in r.stdout.splitlines():
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "assistant":
+ msg = ev.get("message") or {}
+ u = msg.get("usage") or {}
+ key = msg.get("id") or f"anon-{len(per_message)}"
+ prev = per_message.get(key, {})
+ # Content blocks of one response arrive as separate events with the same id.
+ per_message[key] = {k: max(int(u.get(k) or 0), int(prev.get(k) or 0))
+ for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens", "output_tokens")}
+ elif ev.get("type") == "result":
+ result = ev
+ answer = ev.get("result") or answer
+print(answer)
+sys.stderr.write(r.stderr)
+if result is not None and result.get("is_error"):
+ sys.stderr.write(f"claude_code_run: result is_error ({result.get('subtype')})\n")
+calls = len(per_message) or int((result or {}).get("num_turns") or 0)
+# The result event carries the session's totals; per-message usage can be empty
+# when a gateway streams it only at the end.
+totals = (result or {}).get("usage") or {}
+if totals:
+ tin, tread, twrite, tout = (int(totals.get(k) or 0) for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens", "output_tokens"))
+else:
+ tin = sum(m["input_tokens"] for m in per_message.values())
+ tread = sum(m["cache_read_input_tokens"] for m in per_message.values())
+ twrite = sum(m["cache_creation_input_tokens"] for m in per_message.values())
+ tout = sum(m["output_tokens"] for m in per_message.values())
+sys.stderr.write(f"\n[usage] {calls} api call(s) · {tin + tread + twrite} in ({tread} cached, {twrite} cache writes) + {tout} out tokens\n")
+sys.exit(r.returncode)
diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.json
index 803e44a7..96da5052 100644
--- a/graff-evals/harnesses.json
+++ b/graff-evals/harnesses.json
@@ -367,5 +367,12 @@
"usage": "graff-stderr",
"model_prefix": "codegraff",
"note": "OpenCode 2 (@opencode/cli 2.x; OPENCODE2_BIN when `opencode` on PATH is 1.x) with a codegraff provider in its config; pass --model. The wrapper maps .mcp.json into opencode.json and reads usage from OpenCode's session database."
+ },
+ "claude-code": {
+ "cmd": ["python3", "{repo}/graff-evals/claude_code_run.py", "{model}", "{prompt}"],
+ "answer": "stdout",
+ "usage": "graff-stderr",
+ "env": {"DISABLE_TELEMETRY": "1", "DISABLE_AUTOUPDATER": "1", "DISABLE_ERROR_REPORTING": "1", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"},
+ "note": "Claude Code (`claude -p`, stream-json) with the task's .mcp.json as its only MCP config. Set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN for the endpoint; a chat-completions-only endpoint needs an Anthropic-format proxy in front (evals/graff-vs-claude-code). Pass --model."
}
}