Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

evals: graff vs the Codex app server and pi; graff vs OpenCode 2 and pi

justrach755dd8176b97
diff --git a/benchmarks/README.md b/benchmarks/README.md
index e8ada1dd..60969393 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -12,9 +12,11 @@ grok-4.6 SuperGrok) is
[graff-live-12-report.html](graff-live-12-report.html) plus
[artifacts/graff-evals-live/RECEIPT.md](../artifacts/graff-evals-live/RECEIPT.md).
-graff against the Codex app server on the same model and ChatGPT account (21
-coding and MCP tasks, every run's data) is
-[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md).
+graff against the Codex app server and pi on the same model and ChatGPT account
+(21 coding and MCP tasks, every run's data) is
+[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md),
+and graff against OpenCode 2 and pi on one model through the Codegraff gateway is
+[evals/graff-vs-opencode-and-pi/RESULTS.md](../evals/graff-vs-opencode-and-pi/RESULTS.md).
The same 21 tasks against the Grok CLI on the same SuperGrok account are
[evals/graff-vs-grok-cli/RESULTS.md](../evals/graff-vs-grok-cli/RESULTS.md),
and what both mean for an engineering budget is
diff --git a/evals/enterprise-cost.md b/evals/enterprise-cost.md
index 87d485f3..43734f80 100644
--- a/evals/enterprise-cost.md
+++ b/evals/enterprise-cost.md
@@ -3,7 +3,7 @@
A rough guide built from two measured comparisons, each on the same model
and account in both harnesses:
-- [graff vs the Codex app server](graff-vs-codex-app-server/RESULTS.md) (`gpt-6.1-sol`)
+- [graff vs the Codex app server and pi](graff-vs-codex-app-server/RESULTS.md) (`gpt-6.1-sol`)
- [graff vs the Grok CLI](graff-vs-grok-cli/RESULTS.md) (`grok-4.7`)
Both ran the same 21 coding and MCP tasks, 3 times each. Costs are list
@@ -13,8 +13,8 @@ prices of the tokens each harness used.
| Compared with | graff | Other harness | graff costs |
| --- | ---: | ---: | ---: |
-| Codex app server, its reported usage | $0.0164 | $0.0278 | 41% less |
-| Codex app server, with its unreported warm-up request | $0.0164 | up to $0.0524 | up to 69% less |
+| Codex app server, its reported usage | $0.0131 | $0.0426 | 69% less |
+| Codex app server, with its unreported warm-up request | $0.0131 | up to $0.0672 | up to 81% less |
| Grok CLI, grok-4.7 public list price | $0.039 | $0.26 to $0.33 | 85% to 88% less |
| Grok CLI, xAI's price for its build-model route | $0.039 | $0.098 | 60% less |
@@ -26,15 +26,15 @@ costs above.
| Compared with | graff per month | Other per month | Saved per month | Saved per year |
| --- | ---: | ---: | ---: | ---: |
-| Codex app server, reported usage | $1,378 | $2,335 | $958 | $11,491 |
-| Codex app server, with warm-up | $1,378 | $4,402 | $3,024 | $36,288 |
+| Codex app server, reported usage | $1,100 | $3,578 | $2,478 | $29,736 |
+| Codex app server, with warm-up | $1,100 | $5,645 | $4,544 | $54,533 |
| Grok CLI, public list price | $3,284 | $22,042 to $27,770 | $18,757 to $24,486 | $225,086 to $293,832 |
| Grok CLI, xAI's build-model price | $3,284 | $8,240 | $4,956 | $59,472 |
Time moves too. Against the Grok CLI, graff took 15.5s per task against
106.9s: about 2,100 fewer hours of agent wall time a month at this volume.
-Against the Codex app server graff was slower, 28.2s against 25.0s per task
-(about 75 more hours a month).
+Against the Codex app server graff is now faster, 22.2s against 28.6s per task
+(about 149 fewer hours a month).
## How far this carries
diff --git a/evals/graff-vs-codex-app-server/RESULTS.md b/evals/graff-vs-codex-app-server/RESULTS.md
index 6d4bdcd2..1b1312f2 100644
--- a/evals/graff-vs-codex-app-server/RESULTS.md
+++ b/evals/graff-vs-codex-app-server/RESULTS.md
@@ -1,101 +1,103 @@
-# graff vs the Codex app server
+# graff vs the Codex app server and pi
Measured 2026-10-01. The same model (`gpt-6.1-sol`), ChatGPT account and machine ran the same 21 tasks in each harness, side by side.
| Arm | What ran |
| --- | --- |
-| graff v0.0.302.11 | The published release asset, checked against the release's `SHA256SUMS`. |
-| graff main | Main at `900e62d9`: v0.0.302.11 plus #1438, #1439 and #1440 (ADR 0226, 0227, 0228). |
-| Codex app server | `codex app-server` from codex-cli 0.159.0. |
+| graff | The ADR 0231 build (the `perf: drop round trips the model does not need` commit on top of v0.0.302.12), as a scripted `graff repl` over the Codex Responses WebSocket. |
+| Codex app server | `codex app-server` from codex-cli 0.159.0, through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py). |
+| pi | pi 0.84.2 on the ChatGPT plan (`--provider openai-codex`). pi has no MCP client, so it ran the 11 coding tasks only. |
-graff runs as a scripted `graff repl` over the Codex Responses WebSocket, the transport the app server uses (the `graff-codex-repl` harness). The app server runs through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py), which drives its JSON-RPC and reads the token usage it reports. Each harness uses its own defaults. For this model that means `low` effort and `low` verbosity for graff main and for the app server (the model catalog's defaults), and `medium` effort for v0.0.302.11.
+Each harness runs at the model catalog's default, `low` effort (pi's `--thinking low`). graff sends the prompt as one bracketed-paste frame, so a task with a line break is one turn, as it is for the other two.
## Results
-All 21 tasks, mean per task over 3 runs of each:
+All 21 tasks (pi: the 11 coding tasks), mean per task over 3 runs of each:
| | Pass | Wall per task | List cost per task | Input read from cache | Output tokens per task | Model calls per task |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
-| graff v0.0.302.11 | 63/63 | 31.2s | $0.0189 | 92% | 598 | 4.7 |
-| graff main | 63/63 | 28.2s | $0.0164 | 93% | 490 | 4.7 |
-| Codex app server | 63/63 | 25.0s | $0.0278 | 86% | 382 | 4.3 |
+| graff (ADR 0231 build) | 63/63 | 22.2s | $0.0131 | 90% | 348 | 3.1 |
+| Codex app server | 63/63 | 28.6s | $0.0426 | 86% | 409 | 4.4 |
+| pi | 33/33 | 17.0s | $0.0067 | 33% | 107 | 3.2 |
By suite:
-| Suite | graff v0.0.302.11 | graff main | Codex app server |
+| Suite | graff (ADR 0231 build) | Codex app server | pi |
| --- | ---: | ---: | ---: |
-| core (11 coding tasks), wall | 17.3s | 17.5s | 17.6s |
-| core (11 coding tasks), cost | $0.0078 | $0.0081 | $0.0148 |
+| core (11 coding tasks), wall | 16.0s | 20.4s | 17.0s |
+| core (11 coding tasks), cost | $0.0093 | $0.0156 | $0.0067 |
| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 |
-| mcp (10 MCP tasks), wall | 46.5s | 40.0s | 33.0s |
-| mcp (10 MCP tasks), cost | $0.0311 | $0.0256 | $0.0436 |
-| mcp (10 MCP tasks), pass | 30/30 | 30/30 | 30/30 |
+| mcp (10 MCP tasks), wall | 29.0s | 37.8s | — |
+| mcp (10 MCP tasks), cost | $0.0173 | $0.0722 | — |
+| mcp (10 MCP tasks), pass | 30/30 | 30/30 | — |
-**Against the release.** graff main is 10% faster and 13% cheaper per task than v0.0.302.11. The gain is in the MCP suite (46.5s to 40.0s per task, $0.0311 to $0.0256), most of it on `linear-split` (114.6s to 65.7s), where sub-agents now call the tools their root loaded (ADR 0227, 0228). The core suite is unchanged.
+**Against the Codex app server.** graff is 22% faster per task over all 21 tasks (22.2s against 28.6s): 22% on the coding tasks (16.0s against 20.4s) and 23% on the MCP tasks (29.0s against 37.8s). It makes 3.1 model calls per task to the app server's 4.4, and 3.3 to its 5.8 on the MCP tasks, where a single `rlm` script often fetches, reduces and writes the report. It costs 69% less per task on the app server's own reported usage ($0.0131 against $0.0426). Every arm passed every run.
-**Against the Codex app server.** graff main ties on the core suite (17.5s against 17.6s) and is slower on the MCP suite (40.0s against 33.0s). Per MCP task it makes 6.1 model calls to the app server's 5.8 and writes 789 output tokens to its 575; `linear-sidecar` shows it most, 6 to 7 calls against 4. Over all 21 tasks the app server takes 11% less time per task (25.0s against 28.2s).
+**Against pi.** On the coding tasks graff is 6% faster (16.0s against 17.0s). pi's system prompt is about a tenth of graff's, so it sends far fewer tokens and costs less per coding task ($0.0067 against $0.0093); it reads little from cache.
-graff main costs 41% less per task on the app server's own reported usage ($0.0164 against $0.0278), and 43% to 69% less once the app server's unreported warm-up request is counted (see Caveats). graff reads more of its input from cache (93% against 86%); part of that gap is the same warm-up, which caches the app server's first real request without appearing in its usage.
-
-Every arm passed all 63 runs.
+**Against the release.** Earlier the same afternoon, in a run interleaved the same way, v0.0.302.12 took 27.1s per task and the app server 27.4s; on the MCP tasks the release was slower (38.1s against 36.6s, 5.9 calls to 5.8). The ADR 0231 build is 18% faster than the release (22.2s against 27.1s) and 23% cheaper ($0.0131 against $0.0171). That run is [`release-baseline.json`](release-baseline.json).
## Per task
Wall time is the mean of 3 runs.
-| Task | Suite | graff v0.0.302.11 | graff main | Codex app server |
+| Task | Suite | graff (ADR 0231 build) | Codex app server | pi |
| --- | --- | ---: | ---: | ---: |
-| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 16.8s | 15.2s | 22.1s |
-| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 8.9s | 9.0s | 18.9s |
-| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 4.4s | 4.3s | 6.4s |
-| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 22.4s | 21.2s | 11.8s |
-| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 20.9s | 20.5s | 21.6s |
-| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 24.4s | 32.7s | 17.9s |
-| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 20.0s | 20.8s | 20.5s |
-| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 33.7s | 36.7s | 27.1s |
-| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 42.6s | 43.7s | 31.1s |
-| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 43.3s | 39.5s | 31.1s |
-| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 35.5s | 33.8s | 29.6s |
-| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 34.0s | 33.6s | 29.1s |
-| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 38.2s | 36.9s | 34.6s |
-| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 39.5s | 34.5s | 36.2s |
-| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 49.0s | 46.8s | 27.1s |
-| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 114.6s | 65.7s | 58.0s |
-| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 35.0s | 28.9s | 26.2s |
-| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.2s | 7.4s | 10.2s |
-| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 24.3s | 20.7s | 26.9s |
-| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 18.8s | 19.1s | 14.1s |
-| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 22.0s | 21.6s | 23.6s |
+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 16.0s | 21.5s | 13.8s |
+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 10.6s | 21.0s | 18.2s |
+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 5.0s | 8.3s | 8.5s |
+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 14.5s | 13.1s | 23.0s |
+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 21.4s | 23.5s | 22.3s |
+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 21.5s | 20.4s | 15.5s |
+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 16.8s | 30.6s | 15.2s |
+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 22.8s | 26.8s | — |
+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 30.7s | 28.5s | — |
+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 27.8s | 34.2s | — |
+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 22.0s | 36.6s | — |
+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 24.0s | 33.4s | — |
+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 22.5s | 39.2s | — |
+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 26.6s | 37.7s | — |
+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 38.3s | 35.4s | — |
+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 52.5s | 66.4s | — |
+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 23.0s | 39.4s | — |
+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 9.9s | 10.5s | 10.3s |
+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 23.0s | 29.0s | 23.4s |
+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 17.2s | 18.0s | 13.3s |
+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 20.4s | 28.1s | 23.8s |
`results.json` has every run: pass, wall time, model calls, input, cached input and output tokens, and list cost.
## Method
-- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The fixture declares its two list tools read-only (`readOnlyHint`), as the hosted server it imitates does. Each task passes or fails on its own check.
-- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 8 runs at a time. Each arm made one unscored request first, so no arm started with a prompt cache the others had warmed.
+- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The fixture declares its two list tools read-only (`readOnlyHint`). Each task passes or fails on its own check.
+- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 8 runs at a time; the MCP suite ran first, then the coding suite with pi added. Each arm made one unscored request first, so no arm started with a prompt cache the others had warmed.
- **Wall time.** The runner's clock around each process, start to exit.
-- **Cost.** List price of the tokens each harness reports: $2 per 1M input, $0.10 per 1M cached input and $10 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). A ChatGPT plan is not billed per token; this makes the two harnesses comparable, not a bill.
+- **Cost.** List price of the tokens each harness reports: $2 per 1M input, $0.10 per 1M cached input and $10 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). The app server's three `linear-split` runs total more than 272k input tokens, the model's higher-price threshold, across 16 to 18 requests averaging 22k to 25k each; no single request comes near it, so they are priced at the standard rates. A ChatGPT plan is not billed per token; this makes the harnesses comparable, not a bill.
## Caveats
-- **Codex's reported usage leaves out one request.** Before a thread's first turn the app server sends a warm-up request (`generate: false`) with its instructions and tools, about 12k tokens. Its usage is not reported, so the Codex cost above is a floor. Counting it adds between about $0.001 per task (fully cached) and $0.025 (uncached).
-- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in either harness.
-- **`file-ops` has a newline in its prompt.** The scripted repl sends each line as a turn, so graff runs that task as two turns and the app server as one.
-- **Small sample.** 3 runs per task, one machine, one account, one 12-minute run. A few seconds per task is within run-to-run noise.
+- **Codex's reported usage leaves out one request.** Before a thread's first turn the app server sends a warm-up request (`generate: false`) with its instructions and tools, about 12k tokens. Its usage is not reported, so the Codex cost above is a floor.
+- **pi runs only the coding tasks.** It has no MCP client.
+- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in every harness.
+- **Small sample.** 3 runs per task, one machine, one account. A few seconds per task is within run-to-run noise.
## Reproduce
```sh
cd graff-evals
-# graff-codex-repl runs {repo}/zig-out/bin/graff; --binary points it elsewhere.
-# The app server needs `codex` on PATH and CODEX_HOME at a signed-in Codex home.
-# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py
-# copied to <parent of output-root>/scripts/ for the mcp suite.
-python3 run.py --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol \
- --suite core,mcp --reps 3 -j 8 --interleave --output-root /path/outside/repo/run \
- --task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \
+# graff-repl runs {repo}/zig-out/bin/graff (it needs the piped paste of ADR 0231);
+# --binary points it elsewhere. The app server needs `codex` on PATH and CODEX_HOME at a
+# signed-in Codex home. pi-codex needs pi signed in on the ChatGPT plan, with gpt-6.1-sol
+# declared under providers.openai-codex.models in ~/.pi/agent/models.json.
+# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py copied to
+# <parent of output-root>/scripts/ for the mcp suite.
+TASKS="--task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \
--task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \
--task dead-code --task linear-issues --task linear-warm --task linear-sidecar \
--task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \
- --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm
+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm"
+python3 run.py --harness graff-repl,codex-app-server --suite mcp $TASKS \
+ --reps 3 -j 8 --interleave --output-root /path/outside/repo/mcp
+python3 run.py --harness graff-repl,codex-app-server,pi-codex --suite core $TASKS \
+ --reps 3 -j 8 --interleave --output-root /path/outside/repo/core
```
diff --git a/evals/graff-vs-codex-app-server/release-baseline.json b/evals/graff-vs-codex-app-server/release-baseline.json
new file mode 100644
index 00000000..4468b982
--- /dev/null
+++ b/evals/graff-vs-codex-app-server/release-baseline.json
@@ -0,0 +1,2196 @@
+{
+ "model": "gpt-6.1-sol",
+ "tasks": 21,
+ "reps": 3,
+ "arms": {
+ "graff-v302-12": "graff v0.0.302.12 (release)",
+ "codex-app-server": "Codex app server"
+ },
+ "summary": {
+ "all": {
+ "graff-v302-12": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 27.1,
+ "list_usd_per_task": 0.0171,
+ "cached_input_share": 0.915,
+ "output_tokens_per_task": 431,
+ "model_calls_per_task": 4.5
+ },
+ "codex-app-server": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 27.4,
+ "list_usd_per_task": 0.0408,
+ "cached_input_share": 0.866,
+ "output_tokens_per_task": 401,
+ "model_calls_per_task": 4.3
+ }
+ },
+ "core": {
+ "graff-v302-12": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 17.1,
+ "list_usd_per_task": 0.009,
+ "cached_input_share": 0.942,
+ "output_tokens_per_task": 186,
+ "model_calls_per_task": 3.2
+ },
+ "codex-app-server": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 19.1,
+ "list_usd_per_task": 0.0142,
+ "cached_input_share": 0.908,
+ "output_tokens_per_task": 206,
+ "model_calls_per_task": 3.1
+ }
+ },
+ "mcp": {
+ "graff-v302-12": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 38.1,
+ "list_usd_per_task": 0.0261,
+ "cached_input_share": 0.9,
+ "output_tokens_per_task": 701,
+ "model_calls_per_task": 5.9
+ },
+ "codex-app-server": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 36.6,
+ "list_usd_per_task": 0.0701,
+ "cached_input_share": 0.849,
+ "output_tokens_per_task": 616,
+ "model_calls_per_task": 5.8
+ }
+ }
+ },
+ "per_task": {
+ "csv-sum": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 16.8,
+ "list_usd": 0.0071,
+ "model_calls": 3
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.2,
+ "list_usd": 0.0167,
+ "model_calls": 3.3
+ }
+ },
+ "dead-code": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.2,
+ "list_usd": 0.0085,
+ "model_calls": 2
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.6,
+ "list_usd": 0.0143,
+ "model_calls": 4
+ }
+ },
+ "exact-reply": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.3,
+ "list_usd": 0.0039,
+ "model_calls": 1
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.3,
+ "list_usd": 0.0052,
+ "model_calls": 1
+ }
+ },
+ "file-ops": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 25.8,
+ "list_usd": 0.0101,
+ "model_calls": 5
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 15.3,
+ "list_usd": 0.0093,
+ "model_calls": 2
+ }
+ },
+ "fix-fib": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.7,
+ "list_usd": 0.0137,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 23.0,
+ "list_usd": 0.0185,
+ "model_calls": 4
+ }
+ },
+ "git-ops": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.4,
+ "list_usd": 0.0108,
+ "model_calls": 3
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.4,
+ "list_usd": 0.0146,
+ "model_calls": 3
+ }
+ },
+ "json-transform": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.8,
+ "list_usd": 0.0092,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.5,
+ "list_usd": 0.0168,
+ "model_calls": 4
+ }
+ },
+ "linear-issues": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 33.2,
+ "list_usd": 0.0235,
+ "model_calls": 4.7
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 30.9,
+ "list_usd": 0.0404,
+ "model_calls": 4.3
+ }
+ },
+ "linear-nohint": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 45.7,
+ "list_usd": 0.0246,
+ "model_calls": 6.3
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 36.1,
+ "list_usd": 0.0596,
+ "model_calls": 4.3
+ }
+ },
+ "linear-nohint-warm": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 43.1,
+ "list_usd": 0.0283,
+ "model_calls": 6.3
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 28.1,
+ "list_usd": 0.0726,
+ "model_calls": 4
+ }
+ },
+ "linear-quiet": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 25.5,
+ "list_usd": 0.0137,
+ "model_calls": 3
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 36.2,
+ "list_usd": 0.0351,
+ "model_calls": 4.3
+ }
+ },
+ "linear-quiet-warm": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 32.7,
+ "list_usd": 0.0204,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 28.5,
+ "list_usd": 0.0212,
+ "model_calls": 4
+ }
+ },
+ "linear-reduce": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 30.3,
+ "list_usd": 0.0168,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 30.0,
+ "list_usd": 0.0239,
+ "model_calls": 4.3
+ }
+ },
+ "linear-reduce-warm": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 30.4,
+ "list_usd": 0.0181,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 40.8,
+ "list_usd": 0.042,
+ "model_calls": 5.3
+ }
+ },
+ "linear-sidecar": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 44.4,
+ "list_usd": 0.0258,
+ "model_calls": 6
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 31.7,
+ "list_usd": 0.0533,
+ "model_calls": 4.7
+ }
+ },
+ "linear-split": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 65.2,
+ "list_usd": 0.072,
+ "model_calls": 17
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 67.2,
+ "list_usd": 0.3055,
+ "model_calls": 17.7
+ }
+ },
+ "linear-warm": {
+ "suite": "mcp",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 30.4,
+ "list_usd": 0.0182,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 36.9,
+ "list_usd": 0.047,
+ "model_calls": 4.7
+ }
+ },
+ "recall-noise": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.2,
+ "list_usd": 0.0031,
+ "model_calls": 2
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.0,
+ "list_usd": 0.0091,
+ "model_calls": 2
+ }
+ },
+ "refactor-rename": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.7,
+ "list_usd": 0.0101,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 28.4,
+ "list_usd": 0.0239,
+ "model_calls": 4
+ }
+ },
+ "regex-count": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.0,
+ "list_usd": 0.0102,
+ "model_calls": 3.3
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 15.7,
+ "list_usd": 0.0094,
+ "model_calls": 2.3
+ }
+ },
+ "write-tests": {
+ "suite": "core",
+ "graff-v302-12": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.7,
+ "list_usd": 0.0118,
+ "model_calls": 4
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 27.4,
+ "list_usd": 0.0189,
+ "model_calls": 4
+ }
+ }
+ },
+ "runs": [
+ {
+ "arm": "codex-app-server",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.12,
+ "model_calls": 4,
+ "input_tokens": 57898,
+ "cached_input_tokens": 52992,
+ "output_tokens": 250,
+ "list_usd": 0.017611
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.1,
+ "model_calls": 3,
+ "input_tokens": 43326,
+ "cached_input_tokens": 38656,
+ "output_tokens": 311,
+ "list_usd": 0.016316
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.53,
+ "model_calls": 3,
+ "input_tokens": 43430,
+ "cached_input_tokens": 38656,
+ "output_tokens": 267,
+ "list_usd": 0.016084
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.14,
+ "model_calls": 4,
+ "input_tokens": 58049,
+ "cached_input_tokens": 53120,
+ "output_tokens": 157,
+ "list_usd": 0.01674
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.52,
+ "model_calls": 4,
+ "input_tokens": 58063,
+ "cached_input_tokens": 55040,
+ "output_tokens": 159,
+ "list_usd": 0.01314
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.22,
+ "model_calls": 4,
+ "input_tokens": 58064,
+ "cached_input_tokens": 55040,
+ "output_tokens": 151,
+ "list_usd": 0.013062
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.44,
+ "model_calls": 1,
+ "input_tokens": 14231,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005165
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.35,
+ "model_calls": 1,
+ "input_tokens": 14231,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005165
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.08,
+ "model_calls": 1,
+ "input_tokens": 14231,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005165
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.77,
+ "model_calls": 2,
+ "input_tokens": 28642,
+ "cached_input_tokens": 26368,
+ "output_tokens": 97,
+ "list_usd": 0.008155
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.59,
+ "model_calls": 2,
+ "input_tokens": 28639,
+ "cached_input_tokens": 26368,
+ "output_tokens": 104,
+ "list_usd": 0.008219
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.43,
+ "model_calls": 2,
+ "input_tokens": 28645,
+ "cached_input_tokens": 24576,
+ "output_tokens": 106,
+ "list_usd": 0.011656
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.5,
+ "model_calls": 4,
+ "input_tokens": 58544,
+ "cached_input_tokens": 53504,
+ "output_tokens": 278,
+ "list_usd": 0.01821
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.73,
+ "model_calls": 4,
+ "input_tokens": 58552,
+ "cached_input_tokens": 52992,
+ "output_tokens": 275,
+ "list_usd": 0.019169
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.9,
+ "model_calls": 4,
+ "input_tokens": 58570,
+ "cached_input_tokens": 53632,
+ "output_tokens": 285,
+ "list_usd": 0.018089
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.61,
+ "model_calls": 3,
+ "input_tokens": 43583,
+ "cached_input_tokens": 38656,
+ "output_tokens": 206,
+ "list_usd": 0.01578
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.0,
+ "model_calls": 3,
+ "input_tokens": 43683,
+ "cached_input_tokens": 38912,
+ "output_tokens": 239,
+ "list_usd": 0.015823
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.54,
+ "model_calls": 3,
+ "input_tokens": 43552,
+ "cached_input_tokens": 40704,
+ "output_tokens": 230,
+ "list_usd": 0.012066
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.3,
+ "model_calls": 4,
+ "input_tokens": 57990,
+ "cached_input_tokens": 52992,
+ "output_tokens": 266,
+ "list_usd": 0.017955
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.99,
+ "model_calls": 4,
+ "input_tokens": 57987,
+ "cached_input_tokens": 52864,
+ "output_tokens": 278,
+ "list_usd": 0.018312
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.32,
+ "model_calls": 4,
+ "input_tokens": 57978,
+ "cached_input_tokens": 54912,
+ "output_tokens": 252,
+ "list_usd": 0.014143
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.56,
+ "model_calls": 4,
+ "input_tokens": 89389,
+ "cached_input_tokens": 75648,
+ "output_tokens": 544,
+ "list_usd": 0.040487
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.36,
+ "model_calls": 4,
+ "input_tokens": 78740,
+ "cached_input_tokens": 66176,
+ "output_tokens": 410,
+ "list_usd": 0.035846
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 36.73,
+ "model_calls": 5,
+ "input_tokens": 108142,
+ "cached_input_tokens": 93312,
+ "output_tokens": 576,
+ "list_usd": 0.044751
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 45.77,
+ "model_calls": 4,
+ "input_tokens": 104971,
+ "cached_input_tokens": 83456,
+ "output_tokens": 492,
+ "list_usd": 0.056296
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.82,
+ "model_calls": 4,
+ "input_tokens": 105723,
+ "cached_input_tokens": 82176,
+ "output_tokens": 466,
+ "list_usd": 0.059972
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 33.6,
+ "model_calls": 5,
+ "input_tokens": 135068,
+ "cached_input_tokens": 112256,
+ "output_tokens": 581,
+ "list_usd": 0.06266
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.41,
+ "model_calls": 4,
+ "input_tokens": 105522,
+ "cached_input_tokens": 71552,
+ "output_tokens": 440,
+ "list_usd": 0.079495
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.44,
+ "model_calls": 4,
+ "input_tokens": 104968,
+ "cached_input_tokens": 81664,
+ "output_tokens": 455,
+ "list_usd": 0.059324
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.4,
+ "model_calls": 4,
+ "input_tokens": 104946,
+ "cached_input_tokens": 71296,
+ "output_tokens": 442,
+ "list_usd": 0.07885
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.78,
+ "model_calls": 3,
+ "input_tokens": 63893,
+ "cached_input_tokens": 50816,
+ "output_tokens": 471,
+ "list_usd": 0.035946
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 45.12,
+ "model_calls": 5,
+ "input_tokens": 84578,
+ "cached_input_tokens": 71296,
+ "output_tokens": 572,
+ "list_usd": 0.039414
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 33.66,
+ "model_calls": 5,
+ "input_tokens": 79561,
+ "cached_input_tokens": 70656,
+ "output_tokens": 507,
+ "list_usd": 0.029946
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.38,
+ "model_calls": 4,
+ "input_tokens": 62546,
+ "cached_input_tokens": 56064,
+ "output_tokens": 449,
+ "list_usd": 0.02306
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.9,
+ "model_calls": 4,
+ "input_tokens": 63311,
+ "cached_input_tokens": 58496,
+ "output_tokens": 522,
+ "list_usd": 0.0207
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.15,
+ "model_calls": 4,
+ "input_tokens": 62599,
+ "cached_input_tokens": 57984,
+ "output_tokens": 485,
+ "list_usd": 0.019878
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 31.93,
+ "model_calls": 5,
+ "input_tokens": 80670,
+ "cached_input_tokens": 75264,
+ "output_tokens": 539,
+ "list_usd": 0.023728
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.77,
+ "model_calls": 4,
+ "input_tokens": 63279,
+ "cached_input_tokens": 58368,
+ "output_tokens": 488,
+ "list_usd": 0.020539
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.37,
+ "model_calls": 4,
+ "input_tokens": 63280,
+ "cached_input_tokens": 54656,
+ "output_tokens": 480,
+ "list_usd": 0.027514
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 42.59,
+ "model_calls": 6,
+ "input_tokens": 133335,
+ "cached_input_tokens": 116608,
+ "output_tokens": 666,
+ "list_usd": 0.051775
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.59,
+ "model_calls": 6,
+ "input_tokens": 133104,
+ "cached_input_tokens": 116480,
+ "output_tokens": 658,
+ "list_usd": 0.051476
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 40.21,
+ "model_calls": 4,
+ "input_tokens": 66953,
+ "cached_input_tokens": 60928,
+ "output_tokens": 452,
+ "list_usd": 0.022663
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.41,
+ "model_calls": 5,
+ "input_tokens": 139070,
+ "cached_input_tokens": 117120,
+ "output_tokens": 571,
+ "list_usd": 0.061322
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 31.89,
+ "model_calls": 4,
+ "input_tokens": 77986,
+ "cached_input_tokens": 63744,
+ "output_tokens": 440,
+ "list_usd": 0.039258
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.69,
+ "model_calls": 5,
+ "input_tokens": 138841,
+ "cached_input_tokens": 117120,
+ "output_tokens": 426,
+ "list_usd": 0.059414
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 70.35,
+ "model_calls": 18,
+ "input_tokens": 411883,
+ "cached_input_tokens": 360960,
+ "output_tokens": 1618,
+ "list_usd": 0.300154
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 66.0,
+ "model_calls": 17,
+ "input_tokens": 373663,
+ "cached_input_tokens": 338816,
+ "output_tokens": 1561,
+ "list_usd": 0.230566
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 65.11,
+ "model_calls": 18,
+ "input_tokens": 460984,
+ "cached_input_tokens": 390144,
+ "output_tokens": 1630,
+ "list_usd": 0.385839
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.54,
+ "model_calls": 4,
+ "input_tokens": 78747,
+ "cached_input_tokens": 66176,
+ "output_tokens": 419,
+ "list_usd": 0.03595
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 49.04,
+ "model_calls": 5,
+ "input_tokens": 105075,
+ "cached_input_tokens": 89472,
+ "output_tokens": 595,
+ "list_usd": 0.046103
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 36.07,
+ "model_calls": 5,
+ "input_tokens": 129028,
+ "cached_input_tokens": 107520,
+ "output_tokens": 526,
+ "list_usd": 0.059028
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.16,
+ "model_calls": 2,
+ "input_tokens": 28796,
+ "cached_input_tokens": 26368,
+ "output_tokens": 38,
+ "list_usd": 0.007873
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.74,
+ "model_calls": 2,
+ "input_tokens": 28801,
+ "cached_input_tokens": 24576,
+ "output_tokens": 52,
+ "list_usd": 0.011428
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.0,
+ "model_calls": 2,
+ "input_tokens": 28805,
+ "cached_input_tokens": 26368,
+ "output_tokens": 52,
+ "list_usd": 0.008031
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.18,
+ "model_calls": 4,
+ "input_tokens": 58438,
+ "cached_input_tokens": 50944,
+ "output_tokens": 379,
+ "list_usd": 0.023872
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.04,
+ "model_calls": 4,
+ "input_tokens": 58840,
+ "cached_input_tokens": 51584,
+ "output_tokens": 440,
+ "list_usd": 0.02407
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.98,
+ "model_calls": 4,
+ "input_tokens": 58534,
+ "cached_input_tokens": 51328,
+ "output_tokens": 425,
+ "list_usd": 0.023795
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.01,
+ "model_calls": 2,
+ "input_tokens": 28673,
+ "cached_input_tokens": 26368,
+ "output_tokens": 101,
+ "list_usd": 0.008257
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 13.66,
+ "model_calls": 2,
+ "input_tokens": 28700,
+ "cached_input_tokens": 26368,
+ "output_tokens": 130,
+ "list_usd": 0.008601
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.42,
+ "model_calls": 3,
+ "input_tokens": 43278,
+ "cached_input_tokens": 40576,
+ "output_tokens": 190,
+ "list_usd": 0.011362
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.47,
+ "model_calls": 4,
+ "input_tokens": 58002,
+ "cached_input_tokens": 52864,
+ "output_tokens": 357,
+ "list_usd": 0.019132
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.91,
+ "model_calls": 4,
+ "input_tokens": 57968,
+ "cached_input_tokens": 52992,
+ "output_tokens": 331,
+ "list_usd": 0.018561
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 31.78,
+ "model_calls": 4,
+ "input_tokens": 58016,
+ "cached_input_tokens": 52992,
+ "output_tokens": 378,
+ "list_usd": 0.019127
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.29,
+ "model_calls": 3,
+ "input_tokens": 31280,
+ "cached_input_tokens": 30080,
+ "output_tokens": 216,
+ "list_usd": 0.007568
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.07,
+ "model_calls": 3,
+ "input_tokens": 31248,
+ "cached_input_tokens": 30464,
+ "output_tokens": 215,
+ "list_usd": 0.006764
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.0,
+ "model_calls": 3,
+ "input_tokens": 31276,
+ "cached_input_tokens": 30336,
+ "output_tokens": 214,
+ "list_usd": 0.007054
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.07,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 16512,
+ "output_tokens": 56,
+ "list_usd": 0.010809
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.72,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 16512,
+ "output_tokens": 56,
+ "list_usd": 0.010809
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.71,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20224,
+ "output_tokens": 70,
+ "list_usd": 0.003896
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.33,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 9984,
+ "output_tokens": 5,
+ "list_usd": 0.001622
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.41,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 9984,
+ "output_tokens": 5,
+ "list_usd": 0.001622
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.29,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 6400,
+ "output_tokens": 5,
+ "list_usd": 0.008432
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.59,
+ "model_calls": 5,
+ "input_tokens": 51995,
+ "cached_input_tokens": 50432,
+ "output_tokens": 219,
+ "list_usd": 0.010359
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.41,
+ "model_calls": 5,
+ "input_tokens": 51990,
+ "cached_input_tokens": 50816,
+ "output_tokens": 221,
+ "list_usd": 0.00964
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.29,
+ "model_calls": 5,
+ "input_tokens": 52018,
+ "cached_input_tokens": 50688,
+ "output_tokens": 249,
+ "list_usd": 0.010219
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.32,
+ "model_calls": 4,
+ "input_tokens": 42502,
+ "cached_input_tokens": 40064,
+ "output_tokens": 251,
+ "list_usd": 0.011392
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.3,
+ "model_calls": 4,
+ "input_tokens": 42509,
+ "cached_input_tokens": 36352,
+ "output_tokens": 258,
+ "list_usd": 0.018529
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.52,
+ "model_calls": 4,
+ "input_tokens": 42513,
+ "cached_input_tokens": 40192,
+ "output_tokens": 257,
+ "list_usd": 0.011231
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.32,
+ "model_calls": 3,
+ "input_tokens": 31616,
+ "cached_input_tokens": 30336,
+ "output_tokens": 233,
+ "list_usd": 0.007924
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.51,
+ "model_calls": 3,
+ "input_tokens": 31626,
+ "cached_input_tokens": 26624,
+ "output_tokens": 249,
+ "list_usd": 0.015156
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.35,
+ "model_calls": 3,
+ "input_tokens": 31749,
+ "cached_input_tokens": 30080,
+ "output_tokens": 300,
+ "list_usd": 0.009346
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.18,
+ "model_calls": 4,
+ "input_tokens": 41886,
+ "cached_input_tokens": 40576,
+ "output_tokens": 262,
+ "list_usd": 0.009298
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.82,
+ "model_calls": 4,
+ "input_tokens": 41856,
+ "cached_input_tokens": 40448,
+ "output_tokens": 235,
+ "list_usd": 0.009211
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.46,
+ "model_calls": 4,
+ "input_tokens": 41872,
+ "cached_input_tokens": 40576,
+ "output_tokens": 252,
+ "list_usd": 0.00917
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.7,
+ "model_calls": 4,
+ "input_tokens": 45434,
+ "cached_input_tokens": 35072,
+ "output_tokens": 501,
+ "list_usd": 0.029241
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 36.83,
+ "model_calls": 5,
+ "input_tokens": 57527,
+ "cached_input_tokens": 51200,
+ "output_tokens": 616,
+ "list_usd": 0.023934
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 34.19,
+ "model_calls": 5,
+ "input_tokens": 57109,
+ "cached_input_tokens": 53632,
+ "output_tokens": 515,
+ "list_usd": 0.017467
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.62,
+ "model_calls": 6,
+ "input_tokens": 67899,
+ "cached_input_tokens": 61568,
+ "output_tokens": 734,
+ "list_usd": 0.026159
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 44.91,
+ "model_calls": 6,
+ "input_tokens": 67896,
+ "cached_input_tokens": 63616,
+ "output_tokens": 753,
+ "list_usd": 0.022452
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 52.49,
+ "model_calls": 7,
+ "input_tokens": 80116,
+ "cached_input_tokens": 75648,
+ "output_tokens": 872,
+ "list_usd": 0.025221
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.11,
+ "model_calls": 6,
+ "input_tokens": 69188,
+ "cached_input_tokens": 63488,
+ "output_tokens": 737,
+ "list_usd": 0.025119
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 51.97,
+ "model_calls": 7,
+ "input_tokens": 81844,
+ "cached_input_tokens": 74496,
+ "output_tokens": 922,
+ "list_usd": 0.031366
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.11,
+ "model_calls": 6,
+ "input_tokens": 69081,
+ "cached_input_tokens": 61568,
+ "output_tokens": 727,
+ "list_usd": 0.028453
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.03,
+ "model_calls": 3,
+ "input_tokens": 33479,
+ "cached_input_tokens": 30720,
+ "output_tokens": 476,
+ "list_usd": 0.01335
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.63,
+ "model_calls": 3,
+ "input_tokens": 33533,
+ "cached_input_tokens": 30720,
+ "output_tokens": 507,
+ "list_usd": 0.013768
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.88,
+ "model_calls": 3,
+ "input_tokens": 33552,
+ "cached_input_tokens": 30720,
+ "output_tokens": 513,
+ "list_usd": 0.013866
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.22,
+ "model_calls": 4,
+ "input_tokens": 46459,
+ "cached_input_tokens": 40960,
+ "output_tokens": 828,
+ "list_usd": 0.023374
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.54,
+ "model_calls": 4,
+ "input_tokens": 46362,
+ "cached_input_tokens": 42496,
+ "output_tokens": 529,
+ "list_usd": 0.017272
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.31,
+ "model_calls": 4,
+ "input_tokens": 46392,
+ "cached_input_tokens": 40960,
+ "output_tokens": 548,
+ "list_usd": 0.02044
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.84,
+ "model_calls": 4,
+ "input_tokens": 45083,
+ "cached_input_tokens": 40960,
+ "output_tokens": 541,
+ "list_usd": 0.017752
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.24,
+ "model_calls": 4,
+ "input_tokens": 45068,
+ "cached_input_tokens": 41984,
+ "output_tokens": 529,
+ "list_usd": 0.015656
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.86,
+ "model_calls": 4,
+ "input_tokens": 45092,
+ "cached_input_tokens": 41472,
+ "output_tokens": 553,
+ "list_usd": 0.016917
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 31.87,
+ "model_calls": 4,
+ "input_tokens": 45837,
+ "cached_input_tokens": 40832,
+ "output_tokens": 519,
+ "list_usd": 0.019283
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.62,
+ "model_calls": 4,
+ "input_tokens": 45815,
+ "cached_input_tokens": 40960,
+ "output_tokens": 499,
+ "list_usd": 0.018796
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.61,
+ "model_calls": 4,
+ "input_tokens": 45813,
+ "cached_input_tokens": 42240,
+ "output_tokens": 492,
+ "list_usd": 0.01629
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 40.34,
+ "model_calls": 6,
+ "input_tokens": 68055,
+ "cached_input_tokens": 61056,
+ "output_tokens": 732,
+ "list_usd": 0.027424
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 44.16,
+ "model_calls": 6,
+ "input_tokens": 68136,
+ "cached_input_tokens": 63104,
+ "output_tokens": 766,
+ "list_usd": 0.024034
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 48.84,
+ "model_calls": 6,
+ "input_tokens": 68077,
+ "cached_input_tokens": 61824,
+ "output_tokens": 737,
+ "list_usd": 0.026058
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 59.78,
+ "model_calls": 16,
+ "input_tokens": 161480,
+ "cached_input_tokens": 132736,
+ "output_tokens": 1380,
+ "list_usd": 0.084562
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 65.9,
+ "model_calls": 17,
+ "input_tokens": 174216,
+ "cached_input_tokens": 155392,
+ "output_tokens": 1483,
+ "list_usd": 0.068017
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 69.82,
+ "model_calls": 18,
+ "input_tokens": 190135,
+ "cached_input_tokens": 175104,
+ "output_tokens": 1585,
+ "list_usd": 0.063422
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.0,
+ "model_calls": 4,
+ "input_tokens": 46123,
+ "cached_input_tokens": 40960,
+ "output_tokens": 547,
+ "list_usd": 0.019892
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.05,
+ "model_calls": 4,
+ "input_tokens": 46015,
+ "cached_input_tokens": 40960,
+ "output_tokens": 449,
+ "list_usd": 0.018696
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.1,
+ "model_calls": 4,
+ "input_tokens": 46016,
+ "cached_input_tokens": 42368,
+ "output_tokens": 442,
+ "list_usd": 0.015953
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.19,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.52,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.82,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.44,
+ "model_calls": 4,
+ "input_tokens": 42122,
+ "cached_input_tokens": 40832,
+ "output_tokens": 288,
+ "list_usd": 0.009543
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.93,
+ "model_calls": 4,
+ "input_tokens": 42103,
+ "cached_input_tokens": 40320,
+ "output_tokens": 276,
+ "list_usd": 0.010358
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.85,
+ "model_calls": 4,
+ "input_tokens": 42106,
+ "cached_input_tokens": 40320,
+ "output_tokens": 274,
+ "list_usd": 0.010344
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.22,
+ "model_calls": 3,
+ "input_tokens": 31264,
+ "cached_input_tokens": 30336,
+ "output_tokens": 135,
+ "list_usd": 0.00624
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.7,
+ "model_calls": 4,
+ "input_tokens": 42009,
+ "cached_input_tokens": 36736,
+ "output_tokens": 287,
+ "list_usd": 0.01709
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.18,
+ "model_calls": 3,
+ "input_tokens": 31337,
+ "cached_input_tokens": 30336,
+ "output_tokens": 237,
+ "list_usd": 0.007406
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.04,
+ "model_calls": 4,
+ "input_tokens": 41876,
+ "cached_input_tokens": 40448,
+ "output_tokens": 248,
+ "list_usd": 0.009381
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.65,
+ "model_calls": 4,
+ "input_tokens": 41794,
+ "cached_input_tokens": 36736,
+ "output_tokens": 218,
+ "list_usd": 0.01597
+ },
+ {
+ "arm": "graff-v302-12",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.4,
+ "model_calls": 4,
+ "input_tokens": 41919,
+ "cached_input_tokens": 40320,
+ "output_tokens": 272,
+ "list_usd": 0.00995
+ }
+ ]
+}
\ No newline at end of file
diff --git a/evals/graff-vs-codex-app-server/results.json b/evals/graff-vs-codex-app-server/results.json
index 6d8c1090..a0b10921 100644
--- a/evals/graff-vs-codex-app-server/results.json
+++ b/evals/graff-vs-codex-app-server/results.json
@@ -3,95 +3,86 @@
"tasks": 21,
"reps": 3,
"arms": {
- "graff-v302-11": "graff v0.0.302.11 (release)",
- "graff-next": "graff main (ADR 0226, 0227, 0228)",
- "codex-app-server": "Codex app server"
+ "graff": "graff (ADR 0231 build)",
+ "codex-app-server": "Codex app server",
+ "pi": "pi"
},
"summary": {
"all": {
- "graff-v302-11": {
+ "graff": {
"runs": 63,
"passed": 63,
- "wall_s_per_task": 31.2,
- "list_usd_per_task": 0.0189,
- "cached_input_share": 0.922,
- "output_tokens_per_task": 598,
- "model_calls_per_task": 4.7
- },
- "graff-next": {
- "runs": 63,
- "passed": 63,
- "wall_s_per_task": 28.2,
- "list_usd_per_task": 0.0164,
- "cached_input_share": 0.933,
- "output_tokens_per_task": 490,
- "model_calls_per_task": 4.7
+ "wall_s_per_task": 22.2,
+ "list_usd_per_task": 0.0131,
+ "cached_input_share": 0.904,
+ "output_tokens_per_task": 348,
+ "model_calls_per_task": 3.1
},
"codex-app-server": {
"runs": 63,
"passed": 63,
- "wall_s_per_task": 25.0,
- "list_usd_per_task": 0.0278,
- "cached_input_share": 0.857,
- "output_tokens_per_task": 382,
- "model_calls_per_task": 4.3
+ "wall_s_per_task": 28.6,
+ "list_usd_per_task": 0.0426,
+ "cached_input_share": 0.855,
+ "output_tokens_per_task": 409,
+ "model_calls_per_task": 4.4
+ },
+ "pi": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 17.0,
+ "list_usd_per_task": 0.0067,
+ "cached_input_share": 0.334,
+ "output_tokens_per_task": 107,
+ "model_calls_per_task": 3.2
}
},
"core": {
- "graff-v302-11": {
+ "graff": {
"runs": 33,
"passed": 33,
- "wall_s_per_task": 17.3,
- "list_usd_per_task": 0.0078,
- "cached_input_share": 0.964,
- "output_tokens_per_task": 218,
- "model_calls_per_task": 3.2
+ "wall_s_per_task": 16.0,
+ "list_usd_per_task": 0.0093,
+ "cached_input_share": 0.923,
+ "output_tokens_per_task": 164,
+ "model_calls_per_task": 3.0
},
- "graff-next": {
+ "codex-app-server": {
"runs": 33,
"passed": 33,
- "wall_s_per_task": 17.5,
- "list_usd_per_task": 0.0081,
- "cached_input_share": 0.964,
- "output_tokens_per_task": 219,
- "model_calls_per_task": 3.3
+ "wall_s_per_task": 20.4,
+ "list_usd_per_task": 0.0156,
+ "cached_input_share": 0.896,
+ "output_tokens_per_task": 209,
+ "model_calls_per_task": 3.1
},
- "codex-app-server": {
+ "pi": {
"runs": 33,
"passed": 33,
- "wall_s_per_task": 17.6,
- "list_usd_per_task": 0.0148,
- "cached_input_share": 0.9,
- "output_tokens_per_task": 206,
- "model_calls_per_task": 3.0
+ "wall_s_per_task": 17.0,
+ "list_usd_per_task": 0.0067,
+ "cached_input_share": 0.334,
+ "output_tokens_per_task": 107,
+ "model_calls_per_task": 3.2
}
},
"mcp": {
- "graff-v302-11": {
- "runs": 30,
- "passed": 30,
- "wall_s_per_task": 46.5,
- "list_usd_per_task": 0.0311,
- "cached_input_share": 0.901,
- "output_tokens_per_task": 1015,
- "model_calls_per_task": 6.3
- },
- "graff-next": {
+ "graff": {
"runs": 30,
"passed": 30,
- "wall_s_per_task": 40.0,
- "list_usd_per_task": 0.0256,
- "cached_input_share": 0.917,
- "output_tokens_per_task": 789,
- "model_calls_per_task": 6.1
+ "wall_s_per_task": 29.0,
+ "list_usd_per_task": 0.0173,
+ "cached_input_share": 0.887,
+ "output_tokens_per_task": 550,
+ "model_calls_per_task": 3.3
},
"codex-app-server": {
"runs": 30,
"passed": 30,
- "wall_s_per_task": 33.0,
- "list_usd_per_task": 0.0436,
- "cached_input_share": 0.84,
- "output_tokens_per_task": 575,
+ "wall_s_per_task": 37.8,
+ "list_usd_per_task": 0.0722,
+ "cached_input_share": 0.839,
+ "output_tokens_per_task": 628,
"model_calls_per_task": 5.8
}
}
@@ -99,443 +90,436 @@
"per_task": {
"csv-sum": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 16.8,
- "list_usd": 0.0071
+ "wall_s": 16.0,
+ "list_usd": 0.0116,
+ "model_calls": 3.3
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 15.2,
- "list_usd": 0.007
+ "wall_s": 21.5,
+ "list_usd": 0.018,
+ "model_calls": 3.3
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 22.1,
- "list_usd": 0.0153
+ "wall_s": 13.8,
+ "list_usd": 0.0051,
+ "model_calls": 3
}
},
"dead-code": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 8.9,
- "list_usd": 0.0036
+ "wall_s": 10.6,
+ "list_usd": 0.0059,
+ "model_calls": 2
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 9.0,
- "list_usd": 0.0035
+ "wall_s": 21.0,
+ "list_usd": 0.0171,
+ "model_calls": 4
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 18.9,
- "list_usd": 0.0146
+ "wall_s": 18.2,
+ "list_usd": 0.0068,
+ "model_calls": 4
}
},
"exact-reply": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 4.4,
- "list_usd": 0.0037
+ "wall_s": 5.0,
+ "list_usd": 0.0014,
+ "model_calls": 1
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 4.3,
- "list_usd": 0.0014
+ "wall_s": 8.3,
+ "list_usd": 0.0052,
+ "model_calls": 1
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 6.4,
- "list_usd": 0.0052
+ "wall_s": 8.5,
+ "list_usd": 0.0023,
+ "model_calls": 1
}
},
"file-ops": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 22.4,
- "list_usd": 0.0098
+ "wall_s": 14.5,
+ "list_usd": 0.0107,
+ "model_calls": 3
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 21.2,
- "list_usd": 0.0089
+ "wall_s": 13.1,
+ "list_usd": 0.0116,
+ "model_calls": 2
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 11.8,
- "list_usd": 0.0105
+ "wall_s": 23.0,
+ "list_usd": 0.0059,
+ "model_calls": 3
}
},
"fix-fib": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 20.9,
- "list_usd": 0.0108
+ "wall_s": 21.4,
+ "list_usd": 0.0143,
+ "model_calls": 4
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 20.5,
- "list_usd": 0.0106
+ "wall_s": 23.5,
+ "list_usd": 0.0202,
+ "model_calls": 4
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 21.6,
- "list_usd": 0.0183
+ "wall_s": 22.3,
+ "list_usd": 0.0103,
+ "model_calls": 5
}
},
"git-ops": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 24.4,
- "list_usd": 0.0106
+ "wall_s": 21.5,
+ "list_usd": 0.0115,
+ "model_calls": 3
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 32.7,
- "list_usd": 0.0151
+ "wall_s": 20.4,
+ "list_usd": 0.0159,
+ "model_calls": 3
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 17.9,
- "list_usd": 0.0166
+ "wall_s": 15.5,
+ "list_usd": 0.0059,
+ "model_calls": 3
}
},
"json-transform": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 20.0,
- "list_usd": 0.0095
+ "wall_s": 16.8,
+ "list_usd": 0.0089,
+ "model_calls": 3
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 20.8,
- "list_usd": 0.0096
+ "wall_s": 30.6,
+ "list_usd": 0.0169,
+ "model_calls": 4
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 20.5,
- "list_usd": 0.0181
+ "wall_s": 15.2,
+ "list_usd": 0.0063,
+ "model_calls": 3
}
},
"linear-issues": {
"suite": "mcp",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 33.7,
- "list_usd": 0.0206
- },
- "graff-next": {
- "passed": 3,
- "runs": 3,
- "wall_s": 36.7,
- "list_usd": 0.0207
+ "wall_s": 22.8,
+ "list_usd": 0.0175,
+ "model_calls": 2
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 27.1,
- "list_usd": 0.0532
+ "wall_s": 26.8,
+ "list_usd": 0.0436,
+ "model_calls": 4
}
},
"linear-nohint": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 42.6,
- "list_usd": 0.0244
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 43.7,
- "list_usd": 0.0245
+ "wall_s": 30.7,
+ "list_usd": 0.0196,
+ "model_calls": 4
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 31.1,
- "list_usd": 0.0524
+ "wall_s": 28.5,
+ "list_usd": 0.062,
+ "model_calls": 4.3
}
},
"linear-nohint-warm": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 43.3,
- "list_usd": 0.0233
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 39.5,
- "list_usd": 0.0251
+ "wall_s": 27.8,
+ "list_usd": 0.0173,
+ "model_calls": 3
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 31.1,
- "list_usd": 0.0583
+ "wall_s": 34.2,
+ "list_usd": 0.0702,
+ "model_calls": 5
}
},
"linear-quiet": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 35.5,
- "list_usd": 0.0164
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 33.8,
- "list_usd": 0.0163
+ "wall_s": 22.0,
+ "list_usd": 0.0107,
+ "model_calls": 2
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 29.6,
- "list_usd": 0.0281
+ "wall_s": 36.6,
+ "list_usd": 0.0389,
+ "model_calls": 4.3
}
},
"linear-quiet-warm": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 34.0,
- "list_usd": 0.0173
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 33.6,
- "list_usd": 0.0167
+ "wall_s": 24.0,
+ "list_usd": 0.0132,
+ "model_calls": 2.3
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 29.1,
- "list_usd": 0.0282
+ "wall_s": 33.4,
+ "list_usd": 0.0456,
+ "model_calls": 4
}
},
"linear-reduce": {
"suite": "mcp",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 38.2,
- "list_usd": 0.0207
- },
- "graff-next": {
- "passed": 3,
- "runs": 3,
- "wall_s": 36.9,
- "list_usd": 0.0188
+ "wall_s": 22.5,
+ "list_usd": 0.0108,
+ "model_calls": 2
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 34.6,
- "list_usd": 0.0489
+ "wall_s": 39.2,
+ "list_usd": 0.0386,
+ "model_calls": 5.3
}
},
"linear-reduce-warm": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 39.5,
- "list_usd": 0.0204
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 34.5,
- "list_usd": 0.0207
+ "wall_s": 26.6,
+ "list_usd": 0.0145,
+ "model_calls": 3
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 36.2,
- "list_usd": 0.033
+ "wall_s": 37.7,
+ "list_usd": 0.0483,
+ "model_calls": 5.3
}
},
"linear-sidecar": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 49.0,
- "list_usd": 0.0293
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 46.8,
- "list_usd": 0.0251
+ "wall_s": 38.3,
+ "list_usd": 0.0252,
+ "model_calls": 4.3
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 27.1,
- "list_usd": 0.0499
+ "wall_s": 35.4,
+ "list_usd": 0.05,
+ "model_calls": 4.7
}
},
"linear-split": {
"suite": "mcp",
- "graff-v302-11": {
- "passed": 3,
- "runs": 3,
- "wall_s": 114.6,
- "list_usd": 0.1186
- },
- "graff-next": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 65.7,
- "list_usd": 0.0705
+ "wall_s": 52.5,
+ "list_usd": 0.0322,
+ "model_calls": 8
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 58.0,
- "list_usd": 0
+ "wall_s": 66.4,
+ "list_usd": 0.2772,
+ "model_calls": 17
}
},
"linear-warm": {
"suite": "mcp",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 35.0,
- "list_usd": 0.0204
- },
- "graff-next": {
- "passed": 3,
- "runs": 3,
- "wall_s": 28.9,
- "list_usd": 0.0178
+ "wall_s": 23.0,
+ "list_usd": 0.012,
+ "model_calls": 2
},
"codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 26.2,
- "list_usd": 0.0404
+ "wall_s": 39.4,
+ "list_usd": 0.0477,
+ "model_calls": 4.3
}
},
"recall-noise": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 7.2,
- "list_usd": 0.0033
+ "wall_s": 9.9,
+ "list_usd": 0.0081,
+ "model_calls": 2
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 7.4,
- "list_usd": 0.0054
+ "wall_s": 10.5,
+ "list_usd": 0.0092,
+ "model_calls": 2
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 10.2,
- "list_usd": 0.0104
+ "wall_s": 10.3,
+ "list_usd": 0.0053,
+ "model_calls": 2
}
},
"refactor-rename": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 24.3,
- "list_usd": 0.0099
+ "wall_s": 23.0,
+ "list_usd": 0.0141,
+ "model_calls": 4
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 20.7,
- "list_usd": 0.0101
+ "wall_s": 29.0,
+ "list_usd": 0.0224,
+ "model_calls": 4
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 26.9,
- "list_usd": 0.0224
+ "wall_s": 23.4,
+ "list_usd": 0.0132,
+ "model_calls": 5
}
},
"regex-count": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 18.8,
- "list_usd": 0.0076
+ "wall_s": 17.2,
+ "list_usd": 0.0071,
+ "model_calls": 3.3
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 19.1,
- "list_usd": 0.0074
+ "wall_s": 18.0,
+ "list_usd": 0.0158,
+ "model_calls": 3
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 14.1,
- "list_usd": 0.0136
+ "wall_s": 13.3,
+ "list_usd": 0.0033,
+ "model_calls": 2
}
},
"write-tests": {
"suite": "core",
- "graff-v302-11": {
+ "graff": {
"passed": 3,
"runs": 3,
- "wall_s": 22.0,
- "list_usd": 0.0101
+ "wall_s": 20.4,
+ "list_usd": 0.0087,
+ "model_calls": 4
},
- "graff-next": {
+ "codex-app-server": {
"passed": 3,
"runs": 3,
- "wall_s": 21.6,
- "list_usd": 0.0097
+ "wall_s": 28.1,
+ "list_usd": 0.019,
+ "model_calls": 4
},
- "codex-app-server": {
+ "pi": {
"passed": 3,
"runs": 3,
- "wall_s": 23.6,
- "list_usd": 0.0177
+ "wall_s": 23.8,
+ "list_usd": 0.0093,
+ "model_calls": 4.7
}
}
},
@@ -547,12 +531,12 @@
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 20.08,
+ "wall_s": 21.1,
"model_calls": 3,
- "input_tokens": 43326,
- "cached_input_tokens": 40576,
- "output_tokens": 308,
- "list_usd": 0.012638
+ "input_tokens": 43494,
+ "cached_input_tokens": 36864,
+ "output_tokens": 265,
+ "list_usd": 0.019596
},
{
"arm": "codex-app-server",
@@ -561,12 +545,12 @@
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 26.12,
+ "wall_s": 22.6,
"model_calls": 3,
- "input_tokens": 43418,
- "cached_input_tokens": 40576,
- "output_tokens": 404,
- "list_usd": 0.013782
+ "input_tokens": 43342,
+ "cached_input_tokens": 38784,
+ "output_tokens": 326,
+ "list_usd": 0.016254
},
{
"arm": "codex-app-server",
@@ -575,2616 +559,2196 @@
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 19.99,
- "model_calls": 3,
- "input_tokens": 43332,
- "cached_input_tokens": 36864,
- "output_tokens": 286,
- "list_usd": 0.019482
+ "wall_s": 20.91,
+ "model_calls": 4,
+ "input_tokens": 57931,
+ "cached_input_tokens": 52992,
+ "output_tokens": 288,
+ "list_usd": 0.018057
},
{
- "arm": "graff-next",
- "task": "csv-sum",
+ "arm": "codex-app-server",
+ "task": "dead-code",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 16.06,
- "model_calls": 3,
- "input_tokens": 31281,
- "cached_input_tokens": 30336,
- "output_tokens": 216,
- "list_usd": 0.007084
+ "wall_s": 22.78,
+ "model_calls": 4,
+ "input_tokens": 57946,
+ "cached_input_tokens": 52864,
+ "output_tokens": 150,
+ "list_usd": 0.01695
},
{
- "arm": "graff-next",
- "task": "csv-sum",
+ "arm": "codex-app-server",
+ "task": "dead-code",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 14.83,
- "model_calls": 3,
- "input_tokens": 31270,
- "cached_input_tokens": 30464,
- "output_tokens": 203,
- "list_usd": 0.006688
+ "wall_s": 21.11,
+ "model_calls": 4,
+ "input_tokens": 58070,
+ "cached_input_tokens": 52864,
+ "output_tokens": 150,
+ "list_usd": 0.017198
},
{
- "arm": "graff-next",
- "task": "csv-sum",
+ "arm": "codex-app-server",
+ "task": "dead-code",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 14.83,
- "model_calls": 3,
- "input_tokens": 31286,
- "cached_input_tokens": 30336,
- "output_tokens": 219,
- "list_usd": 0.007124
+ "wall_s": 19.15,
+ "model_calls": 4,
+ "input_tokens": 58043,
+ "cached_input_tokens": 52864,
+ "output_tokens": 165,
+ "list_usd": 0.017294
},
{
- "arm": "graff-v302-11",
- "task": "csv-sum",
+ "arm": "codex-app-server",
+ "task": "exact-reply",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 15.82,
- "model_calls": 3,
- "input_tokens": 31286,
- "cached_input_tokens": 30720,
- "output_tokens": 223,
- "list_usd": 0.006434
+ "wall_s": 6.37,
+ "model_calls": 1,
+ "input_tokens": 14234,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005171
},
{
- "arm": "graff-v302-11",
- "task": "csv-sum",
+ "arm": "codex-app-server",
+ "task": "exact-reply",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 16.23,
- "model_calls": 3,
- "input_tokens": 31286,
- "cached_input_tokens": 30336,
- "output_tokens": 223,
- "list_usd": 0.007164
+ "wall_s": 9.23,
+ "model_calls": 1,
+ "input_tokens": 14235,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005173
},
{
- "arm": "graff-v302-11",
- "task": "csv-sum",
+ "arm": "codex-app-server",
+ "task": "exact-reply",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 18.44,
- "model_calls": 3,
- "input_tokens": 31301,
- "cached_input_tokens": 30336,
- "output_tokens": 272,
- "list_usd": 0.007684
+ "wall_s": 9.37,
+ "model_calls": 1,
+ "input_tokens": 14234,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005171
},
{
"arm": "codex-app-server",
- "task": "dead-code",
+ "task": "file-ops",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 19.97,
- "model_calls": 4,
- "input_tokens": 58039,
- "cached_input_tokens": 55040,
- "output_tokens": 156,
- "list_usd": 0.013062
+ "wall_s": 11.53,
+ "model_calls": 2,
+ "input_tokens": 28646,
+ "cached_input_tokens": 24576,
+ "output_tokens": 97,
+ "list_usd": 0.011568
},
{
"arm": "codex-app-server",
- "task": "dead-code",
+ "task": "file-ops",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 16.93,
- "model_calls": 4,
- "input_tokens": 57949,
- "cached_input_tokens": 54912,
- "output_tokens": 150,
- "list_usd": 0.013065
+ "wall_s": 15.1,
+ "model_calls": 2,
+ "input_tokens": 28646,
+ "cached_input_tokens": 24576,
+ "output_tokens": 99,
+ "list_usd": 0.011588
},
{
"arm": "codex-app-server",
- "task": "dead-code",
+ "task": "file-ops",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 19.81,
- "model_calls": 4,
- "input_tokens": 58134,
- "cached_input_tokens": 52864,
- "output_tokens": 171,
- "list_usd": 0.017536
+ "wall_s": 12.7,
+ "model_calls": 2,
+ "input_tokens": 28646,
+ "cached_input_tokens": 24576,
+ "output_tokens": 101,
+ "list_usd": 0.011608
},
{
- "arm": "graff-next",
- "task": "dead-code",
+ "arm": "codex-app-server",
+ "task": "fix-fib",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 9.81,
- "model_calls": 2,
- "input_tokens": 20811,
- "cached_input_tokens": 20352,
- "output_tokens": 56,
- "list_usd": 0.003513
+ "wall_s": 22.94,
+ "model_calls": 4,
+ "input_tokens": 58587,
+ "cached_input_tokens": 53120,
+ "output_tokens": 284,
+ "list_usd": 0.019086
},
{
- "arm": "graff-next",
- "task": "dead-code",
+ "arm": "codex-app-server",
+ "task": "fix-fib",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 8.12,
- "model_calls": 2,
- "input_tokens": 20811,
- "cached_input_tokens": 20352,
- "output_tokens": 56,
- "list_usd": 0.003513
+ "wall_s": 23.54,
+ "model_calls": 4,
+ "input_tokens": 58579,
+ "cached_input_tokens": 53120,
+ "output_tokens": 285,
+ "list_usd": 0.01908
},
{
- "arm": "graff-next",
- "task": "dead-code",
+ "arm": "codex-app-server",
+ "task": "fix-fib",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 8.99,
- "model_calls": 2,
- "input_tokens": 20811,
- "cached_input_tokens": 20352,
- "output_tokens": 56,
- "list_usd": 0.003513
+ "wall_s": 24.12,
+ "model_calls": 4,
+ "input_tokens": 58566,
+ "cached_input_tokens": 51328,
+ "output_tokens": 279,
+ "list_usd": 0.022399
},
{
- "arm": "graff-v302-11",
- "task": "dead-code",
+ "arm": "codex-app-server",
+ "task": "git-ops",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 9.63,
- "model_calls": 2,
- "input_tokens": 20811,
- "cached_input_tokens": 20352,
- "output_tokens": 70,
- "list_usd": 0.003653
+ "wall_s": 22.45,
+ "model_calls": 3,
+ "input_tokens": 43610,
+ "cached_input_tokens": 38656,
+ "output_tokens": 218,
+ "list_usd": 0.015954
},
{
- "arm": "graff-v302-11",
- "task": "dead-code",
+ "arm": "codex-app-server",
+ "task": "git-ops",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 7.58,
- "model_calls": 2,
- "input_tokens": 20811,
- "cached_input_tokens": 20352,
- "output_tokens": 56,
- "list_usd": 0.003513
+ "wall_s": 17.47,
+ "model_calls": 3,
+ "input_tokens": 43653,
+ "cached_input_tokens": 38656,
+ "output_tokens": 215,
+ "list_usd": 0.01601
},
{
- "arm": "graff-v302-11",
- "task": "dead-code",
+ "arm": "codex-app-server",
+ "task": "git-ops",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 9.56,
- "model_calls": 2,
- "input_tokens": 20811,
- "cached_input_tokens": 20352,
- "output_tokens": 56,
- "list_usd": 0.003513
+ "wall_s": 21.17,
+ "model_calls": 3,
+ "input_tokens": 43532,
+ "cached_input_tokens": 38656,
+ "output_tokens": 221,
+ "list_usd": 0.015828
},
{
"arm": "codex-app-server",
- "task": "exact-reply",
+ "task": "json-transform",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 6.6,
- "model_calls": 1,
- "input_tokens": 14234,
- "cached_input_tokens": 12288,
- "output_tokens": 5,
- "list_usd": 0.005171
+ "wall_s": 28.19,
+ "model_calls": 4,
+ "input_tokens": 58004,
+ "cached_input_tokens": 52992,
+ "output_tokens": 252,
+ "list_usd": 0.017843
},
{
"arm": "codex-app-server",
- "task": "exact-reply",
+ "task": "json-transform",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 6.4,
- "model_calls": 1,
- "input_tokens": 14233,
- "cached_input_tokens": 12288,
- "output_tokens": 5,
- "list_usd": 0.005169
+ "wall_s": 40.09,
+ "model_calls": 4,
+ "input_tokens": 58013,
+ "cached_input_tokens": 54912,
+ "output_tokens": 287,
+ "list_usd": 0.014563
},
{
"arm": "codex-app-server",
- "task": "exact-reply",
+ "task": "json-transform",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 6.12,
- "model_calls": 1,
- "input_tokens": 14233,
- "cached_input_tokens": 12288,
- "output_tokens": 5,
- "list_usd": 0.005169
+ "wall_s": 23.55,
+ "model_calls": 4,
+ "input_tokens": 58008,
+ "cached_input_tokens": 52864,
+ "output_tokens": 274,
+ "list_usd": 0.018314
},
{
- "arm": "graff-next",
- "task": "exact-reply",
- "suite": "core",
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 4.46,
- "model_calls": 1,
- "input_tokens": 10271,
- "cached_input_tokens": 10112,
- "output_tokens": 5,
- "list_usd": 0.001379
+ "wall_s": 26.65,
+ "model_calls": 4,
+ "input_tokens": 78780,
+ "cached_input_tokens": 66176,
+ "output_tokens": 428,
+ "list_usd": 0.036106
},
{
- "arm": "graff-next",
- "task": "exact-reply",
- "suite": "core",
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 5.19,
- "model_calls": 1,
- "input_tokens": 10271,
- "cached_input_tokens": 10112,
- "output_tokens": 5,
- "list_usd": 0.001379
- },
- {
- "arm": "graff-next",
- "task": "exact-reply",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 3.32,
- "model_calls": 1,
- "input_tokens": 10271,
- "cached_input_tokens": 10112,
- "output_tokens": 5,
- "list_usd": 0.001379
- },
- {
- "arm": "graff-v302-11",
- "task": "exact-reply",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 5.29,
- "model_calls": 1,
- "input_tokens": 10271,
- "cached_input_tokens": 6400,
- "output_tokens": 5,
- "list_usd": 0.008432
- },
- {
- "arm": "graff-v302-11",
- "task": "exact-reply",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 3.54,
- "model_calls": 1,
- "input_tokens": 10271,
- "cached_input_tokens": 10112,
- "output_tokens": 5,
- "list_usd": 0.001379
- },
- {
- "arm": "graff-v302-11",
- "task": "exact-reply",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 4.28,
- "model_calls": 1,
- "input_tokens": 10271,
- "cached_input_tokens": 10112,
- "output_tokens": 5,
- "list_usd": 0.001379
+ "wall_s": 28.72,
+ "model_calls": 4,
+ "input_tokens": 78019,
+ "cached_input_tokens": 60416,
+ "output_tokens": 448,
+ "list_usd": 0.045728
},
{
"arm": "codex-app-server",
- "task": "file-ops",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 10.73,
- "model_calls": 2,
- "input_tokens": 28647,
- "cached_input_tokens": 24576,
- "output_tokens": 99,
- "list_usd": 0.01159
- },
- {
- "arm": "codex-app-server",
- "task": "file-ops",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 11.56,
- "model_calls": 2,
- "input_tokens": 28653,
- "cached_input_tokens": 24576,
- "output_tokens": 104,
- "list_usd": 0.011652
- },
- {
- "arm": "codex-app-server",
- "task": "file-ops",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 13.23,
- "model_calls": 2,
- "input_tokens": 28649,
- "cached_input_tokens": 26368,
- "output_tokens": 104,
- "list_usd": 0.008239
- },
- {
- "arm": "graff-next",
- "task": "file-ops",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 24.16,
- "model_calls": 5,
- "input_tokens": 51990,
- "cached_input_tokens": 50688,
- "output_tokens": 219,
- "list_usd": 0.009863
- },
- {
- "arm": "graff-next",
- "task": "file-ops",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 21.6,
- "model_calls": 5,
- "input_tokens": 51990,
- "cached_input_tokens": 50944,
- "output_tokens": 217,
- "list_usd": 0.009356
- },
- {
- "arm": "graff-next",
- "task": "file-ops",
- "suite": "core",
+ "task": "linear-issues",
+ "suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 17.95,
+ "wall_s": 25.05,
"model_calls": 4,
- "input_tokens": 41443,
- "cached_input_tokens": 40576,
- "output_tokens": 158,
- "list_usd": 0.007372
- },
- {
- "arm": "graff-v302-11",
- "task": "file-ops",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 17.16,
- "model_calls": 4,
- "input_tokens": 41391,
- "cached_input_tokens": 40192,
- "output_tokens": 125,
- "list_usd": 0.007667
- },
- {
- "arm": "graff-v302-11",
- "task": "file-ops",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 24.82,
- "model_calls": 5,
- "input_tokens": 52064,
- "cached_input_tokens": 50560,
- "output_tokens": 288,
- "list_usd": 0.010944
- },
- {
- "arm": "graff-v302-11",
- "task": "file-ops",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 25.23,
- "model_calls": 5,
- "input_tokens": 52073,
- "cached_input_tokens": 50688,
- "output_tokens": 299,
- "list_usd": 0.010829
+ "input_tokens": 91803,
+ "cached_input_tokens": 72960,
+ "output_tokens": 402,
+ "list_usd": 0.049002
},
{
"arm": "codex-app-server",
- "task": "fix-fib",
- "suite": "core",
+ "task": "linear-nohint",
+ "suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 19.37,
- "model_calls": 3,
- "input_tokens": 43831,
- "cached_input_tokens": 38656,
- "output_tokens": 259,
- "list_usd": 0.016806
- },
- {
- "arm": "codex-app-server",
- "task": "fix-fib",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 20.93,
+ "wall_s": 27.89,
"model_calls": 4,
- "input_tokens": 58587,
- "cached_input_tokens": 53120,
- "output_tokens": 287,
- "list_usd": 0.019116
+ "input_tokens": 104953,
+ "cached_input_tokens": 81664,
+ "output_tokens": 462,
+ "list_usd": 0.059364
},
{
"arm": "codex-app-server",
- "task": "fix-fib",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 24.64,
- "model_calls": 4,
- "input_tokens": 58583,
- "cached_input_tokens": 53248,
- "output_tokens": 286,
- "list_usd": 0.018855
- },
- {
- "arm": "graff-next",
- "task": "fix-fib",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 21.24,
- "model_calls": 4,
- "input_tokens": 42525,
- "cached_input_tokens": 40448,
- "output_tokens": 261,
- "list_usd": 0.010809
- },
- {
- "arm": "graff-next",
- "task": "fix-fib",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 20.3,
- "model_calls": 4,
- "input_tokens": 42277,
- "cached_input_tokens": 40704,
- "output_tokens": 247,
- "list_usd": 0.009686
- },
- {
- "arm": "graff-next",
- "task": "fix-fib",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 20.05,
- "model_calls": 4,
- "input_tokens": 42561,
- "cached_input_tokens": 40448,
- "output_tokens": 289,
- "list_usd": 0.011161
- },
- {
- "arm": "graff-v302-11",
- "task": "fix-fib",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 21.05,
- "model_calls": 4,
- "input_tokens": 42551,
- "cached_input_tokens": 40832,
- "output_tokens": 283,
- "list_usd": 0.010351
- },
- {
- "arm": "graff-v302-11",
- "task": "fix-fib",
- "suite": "core",
+ "task": "linear-nohint",
+ "suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 20.92,
- "model_calls": 4,
- "input_tokens": 42547,
- "cached_input_tokens": 40448,
- "output_tokens": 277,
- "list_usd": 0.011013
+ "wall_s": 31.32,
+ "model_calls": 5,
+ "input_tokens": 138111,
+ "cached_input_tokens": 116480,
+ "output_tokens": 456,
+ "list_usd": 0.05947
},
{
- "arm": "graff-v302-11",
- "task": "fix-fib",
- "suite": "core",
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 20.69,
+ "wall_s": 26.24,
"model_calls": 4,
- "input_tokens": 42549,
- "cached_input_tokens": 40448,
- "output_tokens": 276,
- "list_usd": 0.011007
+ "input_tokens": 105177,
+ "cached_input_tokens": 77696,
+ "output_tokens": 442,
+ "list_usd": 0.067152
},
{
"arm": "codex-app-server",
- "task": "git-ops",
- "suite": "core",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 18.11,
- "model_calls": 3,
- "input_tokens": 43523,
- "cached_input_tokens": 38912,
- "output_tokens": 206,
- "list_usd": 0.015173
+ "wall_s": 30.78,
+ "model_calls": 4,
+ "input_tokens": 104995,
+ "cached_input_tokens": 83584,
+ "output_tokens": 471,
+ "list_usd": 0.05589
},
{
"arm": "codex-app-server",
- "task": "git-ops",
- "suite": "core",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 17.36,
- "model_calls": 3,
- "input_tokens": 43519,
- "cached_input_tokens": 38656,
- "output_tokens": 203,
- "list_usd": 0.015622
- },
- {
- "arm": "codex-app-server",
- "task": "git-ops",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 18.25,
- "model_calls": 3,
- "input_tokens": 43551,
- "cached_input_tokens": 36864,
- "output_tokens": 193,
- "list_usd": 0.01899
- },
- {
- "arm": "graff-next",
- "task": "git-ops",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 38.24,
+ "wall_s": 38.38,
"model_calls": 6,
- "input_tokens": 64159,
- "cached_input_tokens": 61056,
- "output_tokens": 573,
- "list_usd": 0.018042
+ "input_tokens": 167525,
+ "cached_input_tokens": 130304,
+ "output_tokens": 612,
+ "list_usd": 0.093592
},
{
- "arm": "graff-next",
- "task": "git-ops",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 24.31,
- "model_calls": 3,
- "input_tokens": 31714,
- "cached_input_tokens": 30336,
- "output_tokens": 344,
- "list_usd": 0.00923
- },
- {
- "arm": "graff-next",
- "task": "git-ops",
- "suite": "core",
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 35.56,
+ "wall_s": 33.44,
"model_calls": 5,
- "input_tokens": 53790,
- "cached_input_tokens": 50560,
- "output_tokens": 648,
- "list_usd": 0.017996
- },
- {
- "arm": "graff-v302-11",
- "task": "git-ops",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 24.04,
- "model_calls": 3,
- "input_tokens": 31814,
- "cached_input_tokens": 30336,
- "output_tokens": 448,
- "list_usd": 0.01047
- },
- {
- "arm": "graff-v302-11",
- "task": "git-ops",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 29.79,
- "model_calls": 4,
- "input_tokens": 42655,
- "cached_input_tokens": 40832,
+ "input_tokens": 134923,
+ "cached_input_tokens": 112256,
"output_tokens": 467,
- "list_usd": 0.012399
- },
- {
- "arm": "graff-v302-11",
- "task": "git-ops",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 19.45,
- "model_calls": 3,
- "input_tokens": 31706,
- "cached_input_tokens": 30336,
- "output_tokens": 316,
- "list_usd": 0.008934
+ "list_usd": 0.06123
},
{
"arm": "codex-app-server",
- "task": "json-transform",
- "suite": "core",
+ "task": "linear-quiet",
+ "suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 19.71,
- "model_calls": 4,
- "input_tokens": 57997,
- "cached_input_tokens": 52864,
- "output_tokens": 277,
- "list_usd": 0.018322
- },
- {
- "arm": "codex-app-server",
- "task": "json-transform",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 20.08,
- "model_calls": 4,
- "input_tokens": 58013,
- "cached_input_tokens": 52992,
- "output_tokens": 275,
- "list_usd": 0.018091
+ "wall_s": 45.11,
+ "model_calls": 5,
+ "input_tokens": 107148,
+ "cached_input_tokens": 92800,
+ "output_tokens": 671,
+ "list_usd": 0.044686
},
{
"arm": "codex-app-server",
- "task": "json-transform",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 21.67,
- "model_calls": 4,
- "input_tokens": 57990,
- "cached_input_tokens": 52992,
- "output_tokens": 247,
- "list_usd": 0.017765
- },
- {
- "arm": "graff-next",
- "task": "json-transform",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 19.79,
- "model_calls": 4,
- "input_tokens": 41937,
- "cached_input_tokens": 40704,
- "output_tokens": 271,
- "list_usd": 0.009246
- },
- {
- "arm": "graff-next",
- "task": "json-transform",
- "suite": "core",
- "rep": 2,
- "pass": true,
- "timed_out": false,
- "wall_s": 21.3,
- "model_calls": 4,
- "input_tokens": 41975,
- "cached_input_tokens": 40704,
- "output_tokens": 295,
- "list_usd": 0.009562
- },
- {
- "arm": "graff-next",
- "task": "json-transform",
- "suite": "core",
- "rep": 3,
- "pass": true,
- "timed_out": false,
- "wall_s": 21.35,
- "model_calls": 4,
- "input_tokens": 41976,
- "cached_input_tokens": 40448,
- "output_tokens": 294,
- "list_usd": 0.010041
- },
- {
- "arm": "graff-v302-11",
- "task": "json-transform",
- "suite": "core",
- "rep": 1,
- "pass": true,
- "timed_out": false,
- "wall_s": 19.67,
- "model_calls": 4,
- "input_tokens": 41919,
- "cached_input_tokens": 40576,
- "output_tokens": 290,
- "list_usd": 0.009644
- },
- {
- "arm": "graff-v302-11",
- "task": "json-transform",
- "suite": "core",
+ "task": "linear-quiet",
+ "suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 20.69,
+ "wall_s": 29.19,
"model_calls": 4,
- "input_tokens": 41889,
- "cached_input_tokens": 40576,
- "output_tokens": 270,
- "list_usd": 0.009384
+ "input_tokens": 63325,
+ "cached_input_tokens": 52736,
+ "output_tokens": 494,
+ "list_usd": 0.031392
},
{
- "arm": "graff-v302-11",
- "task": "json-transform",
- "suite": "core",
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 19.75,
+ "wall_s": 35.47,
"model_calls": 4,
- "input_tokens": 41907,
- "cached_input_tokens": 40576,
- "output_tokens": 280,
- "list_usd": 0.00952
+ "input_tokens": 81625,
+ "cached_input_tokens": 67584,
+ "output_tokens": 575,
+ "list_usd": 0.04059
},
{
"arm": "codex-app-server",
- "task": "linear-issues",
+ "task": "linear-quiet-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 26.29,
+ "wall_s": 27.33,
"model_calls": 4,
- "input_tokens": 82398,
- "cached_input_tokens": 66688,
- "output_tokens": 388,
- "list_usd": 0.041969
+ "input_tokens": 63408,
+ "cached_input_tokens": 52992,
+ "output_tokens": 466,
+ "list_usd": 0.030791
},
{
"arm": "codex-app-server",
- "task": "linear-issues",
+ "task": "linear-quiet-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 29.73,
- "model_calls": 5,
- "input_tokens": 131306,
- "cached_input_tokens": 109312,
- "output_tokens": 506,
- "list_usd": 0.059979
+ "wall_s": 35.83,
+ "model_calls": 4,
+ "input_tokens": 81586,
+ "cached_input_tokens": 54656,
+ "output_tokens": 547,
+ "list_usd": 0.064796
},
{
"arm": "codex-app-server",
- "task": "linear-issues",
+ "task": "linear-quiet-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 25.31,
+ "wall_s": 37.12,
"model_calls": 4,
- "input_tokens": 78755,
- "cached_input_tokens": 54656,
- "output_tokens": 410,
- "list_usd": 0.057764
+ "input_tokens": 81885,
+ "cached_input_tokens": 67712,
+ "output_tokens": 612,
+ "list_usd": 0.041237
},
{
- "arm": "graff-next",
- "task": "linear-issues",
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 37.44,
- "model_calls": 5,
- "input_tokens": 57133,
- "cached_input_tokens": 54144,
- "output_tokens": 599,
- "list_usd": 0.017382
+ "wall_s": 30.24,
+ "model_calls": 4,
+ "input_tokens": 63323,
+ "cached_input_tokens": 54656,
+ "output_tokens": 519,
+ "list_usd": 0.02799
},
{
- "arm": "graff-next",
- "task": "linear-issues",
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 35.36,
- "model_calls": 5,
- "input_tokens": 57541,
- "cached_input_tokens": 52736,
- "output_tokens": 613,
- "list_usd": 0.021014
+ "wall_s": 48.78,
+ "model_calls": 6,
+ "input_tokens": 132981,
+ "cached_input_tokens": 114944,
+ "output_tokens": 674,
+ "list_usd": 0.054308
},
{
- "arm": "graff-next",
- "task": "linear-issues",
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 37.27,
- "model_calls": 5,
- "input_tokens": 57293,
- "cached_input_tokens": 51200,
- "output_tokens": 631,
- "list_usd": 0.023616
+ "wall_s": 38.67,
+ "model_calls": 6,
+ "input_tokens": 96523,
+ "cached_input_tokens": 87424,
+ "output_tokens": 655,
+ "list_usd": 0.03349
},
{
- "arm": "graff-v302-11",
- "task": "linear-issues",
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 35.54,
- "model_calls": 5,
- "input_tokens": 57185,
- "cached_input_tokens": 51200,
- "output_tokens": 575,
- "list_usd": 0.02284
+ "wall_s": 45.0,
+ "model_calls": 6,
+ "input_tokens": 138736,
+ "cached_input_tokens": 112384,
+ "output_tokens": 658,
+ "list_usd": 0.070522
},
{
- "arm": "graff-v302-11",
- "task": "linear-issues",
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 31.41,
- "model_calls": 4,
- "input_tokens": 45746,
- "cached_input_tokens": 40960,
- "output_tokens": 580,
- "list_usd": 0.019468
+ "wall_s": 41.32,
+ "model_calls": 6,
+ "input_tokens": 132919,
+ "cached_input_tokens": 118400,
+ "output_tokens": 649,
+ "list_usd": 0.047368
},
{
- "arm": "graff-v302-11",
- "task": "linear-issues",
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 34.06,
+ "wall_s": 26.82,
"model_calls": 4,
- "input_tokens": 45436,
- "cached_input_tokens": 40960,
- "output_tokens": 632,
- "list_usd": 0.019368
+ "input_tokens": 63275,
+ "cached_input_tokens": 54656,
+ "output_tokens": 436,
+ "list_usd": 0.027064
},
{
"arm": "codex-app-server",
- "task": "linear-nohint",
+ "task": "linear-sidecar",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 27.64,
- "model_calls": 4,
- "input_tokens": 102531,
- "cached_input_tokens": 80128,
- "output_tokens": 443,
- "list_usd": 0.057249
+ "wall_s": 41.51,
+ "model_calls": 5,
+ "input_tokens": 139088,
+ "cached_input_tokens": 115200,
+ "output_tokens": 585,
+ "list_usd": 0.065146
},
{
"arm": "codex-app-server",
- "task": "linear-nohint",
+ "task": "linear-sidecar",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 26.92,
- "model_calls": 4,
- "input_tokens": 105527,
- "cached_input_tokens": 83840,
- "output_tokens": 442,
- "list_usd": 0.056178
+ "wall_s": 36.42,
+ "model_calls": 5,
+ "input_tokens": 96819,
+ "cached_input_tokens": 79232,
+ "output_tokens": 559,
+ "list_usd": 0.048687
},
{
"arm": "codex-app-server",
- "task": "linear-nohint",
+ "task": "linear-sidecar",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 38.74,
- "model_calls": 5,
- "input_tokens": 115163,
- "cached_input_tokens": 100864,
- "output_tokens": 504,
- "list_usd": 0.043724
+ "wall_s": 28.14,
+ "model_calls": 4,
+ "input_tokens": 78076,
+ "cached_input_tokens": 65664,
+ "output_tokens": 474,
+ "list_usd": 0.03613
},
{
- "arm": "graff-next",
- "task": "linear-nohint",
+ "arm": "codex-app-server",
+ "task": "linear-split",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 50.7,
- "model_calls": 7,
- "input_tokens": 80135,
- "cached_input_tokens": 75136,
- "output_tokens": 873,
- "list_usd": 0.026242
+ "wall_s": 68.37,
+ "model_calls": 17,
+ "input_tokens": 376049,
+ "cached_input_tokens": 341632,
+ "output_tokens": 1479,
+ "list_usd": 0.228179
},
{
- "arm": "graff-next",
- "task": "linear-nohint",
+ "arm": "codex-app-server",
+ "task": "linear-split",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 39.2,
- "model_calls": 6,
- "input_tokens": 67932,
- "cached_input_tokens": 64384,
- "output_tokens": 752,
- "list_usd": 0.021054
+ "wall_s": 67.48,
+ "model_calls": 16,
+ "input_tokens": 353717,
+ "cached_input_tokens": 295936,
+ "output_tokens": 1460,
+ "list_usd": 0.312211
},
{
- "arm": "graff-next",
- "task": "linear-nohint",
+ "arm": "codex-app-server",
+ "task": "linear-split",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 41.31,
- "model_calls": 6,
- "input_tokens": 67909,
- "cached_input_tokens": 61568,
- "output_tokens": 739,
- "list_usd": 0.026229
+ "wall_s": 63.22,
+ "model_calls": 18,
+ "input_tokens": 440631,
+ "cached_input_tokens": 392832,
+ "output_tokens": 1423,
+ "list_usd": 0.291107
},
{
- "arm": "graff-v302-11",
- "task": "linear-nohint",
+ "arm": "codex-app-server",
+ "task": "linear-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 40.35,
- "model_calls": 6,
- "input_tokens": 67928,
- "cached_input_tokens": 62080,
- "output_tokens": 753,
- "list_usd": 0.025434
+ "wall_s": 46.12,
+ "model_calls": 5,
+ "input_tokens": 105284,
+ "cached_input_tokens": 87936,
+ "output_tokens": 580,
+ "list_usd": 0.04929
},
{
- "arm": "graff-v302-11",
- "task": "linear-nohint",
+ "arm": "codex-app-server",
+ "task": "linear-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 41.26,
- "model_calls": 6,
- "input_tokens": 67928,
- "cached_input_tokens": 63616,
- "output_tokens": 754,
- "list_usd": 0.022526
+ "wall_s": 33.35,
+ "model_calls": 4,
+ "input_tokens": 89282,
+ "cached_input_tokens": 73600,
+ "output_tokens": 569,
+ "list_usd": 0.044414
},
{
- "arm": "graff-v302-11",
- "task": "linear-nohint",
+ "arm": "codex-app-server",
+ "task": "linear-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 46.34,
- "model_calls": 6,
- "input_tokens": 68431,
- "cached_input_tokens": 63872,
- "output_tokens": 982,
- "list_usd": 0.025325
+ "wall_s": 38.58,
+ "model_calls": 4,
+ "input_tokens": 80241,
+ "cached_input_tokens": 61312,
+ "output_tokens": 553,
+ "list_usd": 0.049519
},
{
"arm": "codex-app-server",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "task": "recall-noise",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 30.35,
- "model_calls": 4,
- "input_tokens": 102526,
- "cached_input_tokens": 80128,
- "output_tokens": 428,
- "list_usd": 0.057089
+ "wall_s": 10.44,
+ "model_calls": 2,
+ "input_tokens": 28812,
+ "cached_input_tokens": 26368,
+ "output_tokens": 52,
+ "list_usd": 0.008045
},
{
"arm": "codex-app-server",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "task": "recall-noise",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 36.49,
- "model_calls": 4,
- "input_tokens": 104964,
- "cached_input_tokens": 81664,
- "output_tokens": 449,
- "list_usd": 0.059256
+ "wall_s": 9.8,
+ "model_calls": 2,
+ "input_tokens": 28807,
+ "cached_input_tokens": 26368,
+ "output_tokens": 51,
+ "list_usd": 0.008025
},
{
"arm": "codex-app-server",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "task": "recall-noise",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 26.41,
- "model_calls": 4,
- "input_tokens": 103865,
- "cached_input_tokens": 80896,
- "output_tokens": 456,
- "list_usd": 0.058588
+ "wall_s": 11.18,
+ "model_calls": 2,
+ "input_tokens": 28817,
+ "cached_input_tokens": 24576,
+ "output_tokens": 57,
+ "list_usd": 0.01151
},
{
- "arm": "graff-next",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 38.95,
- "model_calls": 6,
- "input_tokens": 69113,
- "cached_input_tokens": 63616,
- "output_tokens": 721,
- "list_usd": 0.024566
+ "wall_s": 26.91,
+ "model_calls": 4,
+ "input_tokens": 58485,
+ "cached_input_tokens": 52992,
+ "output_tokens": 429,
+ "list_usd": 0.020575
},
{
- "arm": "graff-next",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 40.71,
- "model_calls": 6,
- "input_tokens": 69164,
- "cached_input_tokens": 63616,
- "output_tokens": 795,
- "list_usd": 0.025408
+ "wall_s": 32.06,
+ "model_calls": 4,
+ "input_tokens": 58307,
+ "cached_input_tokens": 51328,
+ "output_tokens": 332,
+ "list_usd": 0.022411
},
{
- "arm": "graff-next",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 38.69,
- "model_calls": 6,
- "input_tokens": 69249,
- "cached_input_tokens": 63616,
- "output_tokens": 767,
- "list_usd": 0.025298
+ "wall_s": 28.13,
+ "model_calls": 4,
+ "input_tokens": 58435,
+ "cached_input_tokens": 50944,
+ "output_tokens": 420,
+ "list_usd": 0.024276
},
{
- "arm": "graff-v302-11",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 43.51,
- "model_calls": 6,
- "input_tokens": 69306,
- "cached_input_tokens": 63360,
- "output_tokens": 796,
- "list_usd": 0.026188
+ "wall_s": 18.3,
+ "model_calls": 3,
+ "input_tokens": 43226,
+ "cached_input_tokens": 36864,
+ "output_tokens": 175,
+ "list_usd": 0.01816
},
{
- "arm": "graff-v302-11",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 46.2,
- "model_calls": 6,
- "input_tokens": 69673,
- "cached_input_tokens": 65664,
- "output_tokens": 923,
- "list_usd": 0.023814
+ "wall_s": 17.35,
+ "model_calls": 3,
+ "input_tokens": 43195,
+ "cached_input_tokens": 38784,
+ "output_tokens": 170,
+ "list_usd": 0.0144
},
{
- "arm": "graff-v302-11",
- "task": "linear-nohint-warm",
- "suite": "mcp",
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 40.09,
- "model_calls": 6,
- "input_tokens": 69145,
- "cached_input_tokens": 66304,
- "output_tokens": 760,
- "list_usd": 0.019912
+ "wall_s": 18.26,
+ "model_calls": 3,
+ "input_tokens": 43262,
+ "cached_input_tokens": 38656,
+ "output_tokens": 190,
+ "list_usd": 0.014978
},
{
"arm": "codex-app-server",
- "task": "linear-quiet",
- "suite": "mcp",
+ "task": "write-tests",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 28.32,
+ "wall_s": 31.21,
"model_calls": 4,
- "input_tokens": 63308,
+ "input_tokens": 58018,
"cached_input_tokens": 52864,
- "output_tokens": 479,
- "list_usd": 0.030964
+ "output_tokens": 361,
+ "list_usd": 0.019204
},
{
"arm": "codex-app-server",
- "task": "linear-quiet",
- "suite": "mcp",
+ "task": "write-tests",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 32.73,
- "model_calls": 5,
- "input_tokens": 79606,
- "cached_input_tokens": 68864,
- "output_tokens": 514,
- "list_usd": 0.03351
+ "wall_s": 24.56,
+ "model_calls": 4,
+ "input_tokens": 58006,
+ "cached_input_tokens": 52992,
+ "output_tokens": 356,
+ "list_usd": 0.018887
},
{
"arm": "codex-app-server",
- "task": "linear-quiet",
- "suite": "mcp",
+ "task": "write-tests",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 27.88,
+ "wall_s": 28.53,
"model_calls": 4,
- "input_tokens": 63293,
- "cached_input_tokens": 58496,
- "output_tokens": 435,
- "list_usd": 0.019794
+ "input_tokens": 57998,
+ "cached_input_tokens": 52864,
+ "output_tokens": 349,
+ "list_usd": 0.019044
},
{
- "arm": "graff-next",
- "task": "linear-quiet",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 28.02,
+ "wall_s": 13.5,
"model_calls": 3,
- "input_tokens": 33539,
- "cached_input_tokens": 31232,
- "output_tokens": 508,
- "list_usd": 0.012817
+ "input_tokens": 31280,
+ "cached_input_tokens": 26624,
+ "output_tokens": 107,
+ "list_usd": 0.013044
},
{
- "arm": "graff-next",
- "task": "linear-quiet",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 44.37,
- "model_calls": 4,
- "input_tokens": 45830,
- "cached_input_tokens": 41728,
- "output_tokens": 893,
- "list_usd": 0.021307
+ "wall_s": 14.83,
+ "model_calls": 3,
+ "input_tokens": 31243,
+ "cached_input_tokens": 26624,
+ "output_tokens": 101,
+ "list_usd": 0.01291
},
{
- "arm": "graff-next",
- "task": "linear-quiet",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 29.12,
- "model_calls": 3,
- "input_tokens": 33616,
- "cached_input_tokens": 30720,
- "output_tokens": 606,
- "list_usd": 0.014924
+ "wall_s": 19.63,
+ "model_calls": 4,
+ "input_tokens": 41912,
+ "cached_input_tokens": 40448,
+ "output_tokens": 186,
+ "list_usd": 0.008833
},
{
- "arm": "graff-v302-11",
- "task": "linear-quiet",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 37.88,
- "model_calls": 3,
- "input_tokens": 33819,
- "cached_input_tokens": 30720,
- "output_tokens": 764,
- "list_usd": 0.01691
+ "wall_s": 9.57,
+ "model_calls": 2,
+ "input_tokens": 20881,
+ "cached_input_tokens": 20352,
+ "output_tokens": 56,
+ "list_usd": 0.003653
},
{
- "arm": "graff-v302-11",
- "task": "linear-quiet",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 39.42,
- "model_calls": 3,
- "input_tokens": 33831,
- "cached_input_tokens": 30720,
- "output_tokens": 841,
- "list_usd": 0.017704
+ "wall_s": 9.3,
+ "model_calls": 2,
+ "input_tokens": 20881,
+ "cached_input_tokens": 16512,
+ "output_tokens": 56,
+ "list_usd": 0.010949
},
{
- "arm": "graff-v302-11",
- "task": "linear-quiet",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 29.18,
- "model_calls": 3,
- "input_tokens": 33635,
- "cached_input_tokens": 30720,
- "output_tokens": 564,
- "list_usd": 0.014542
+ "wall_s": 12.97,
+ "model_calls": 2,
+ "input_tokens": 20881,
+ "cached_input_tokens": 20608,
+ "output_tokens": 56,
+ "list_usd": 0.003167
},
{
- "arm": "codex-app-server",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 30.01,
- "model_calls": 5,
- "input_tokens": 79608,
- "cached_input_tokens": 70656,
- "output_tokens": 460,
- "list_usd": 0.02957
+ "wall_s": 4.66,
+ "model_calls": 1,
+ "input_tokens": 10306,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001449
},
- {
- "arm": "codex-app-server",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 30.91,
- "model_calls": 5,
- "input_tokens": 84598,
- "cached_input_tokens": 73344,
- "output_tokens": 540,
- "list_usd": 0.035242
+ "wall_s": 3.89,
+ "model_calls": 1,
+ "input_tokens": 10306,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001449
},
{
- "arm": "codex-app-server",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 26.39,
- "model_calls": 4,
- "input_tokens": 63291,
- "cached_input_tokens": 58496,
- "output_tokens": 443,
- "list_usd": 0.01987
+ "wall_s": 6.55,
+ "model_calls": 1,
+ "input_tokens": 10306,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001449
},
{
- "arm": "graff-next",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 35.13,
- "model_calls": 4,
- "input_tokens": 46171,
- "cached_input_tokens": 42496,
- "output_tokens": 656,
- "list_usd": 0.01816
+ "wall_s": 14.98,
+ "model_calls": 3,
+ "input_tokens": 31156,
+ "cached_input_tokens": 26624,
+ "output_tokens": 125,
+ "list_usd": 0.012976
},
{
- "arm": "graff-next",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 28.18,
+ "wall_s": 15.58,
"model_calls": 3,
- "input_tokens": 34078,
- "cached_input_tokens": 30720,
- "output_tokens": 585,
- "list_usd": 0.015638
+ "input_tokens": 31192,
+ "cached_input_tokens": 30336,
+ "output_tokens": 135,
+ "list_usd": 0.006096
},
{
- "arm": "graff-next",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 37.51,
+ "wall_s": 13.01,
"model_calls": 3,
- "input_tokens": 34235,
- "cached_input_tokens": 31488,
- "output_tokens": 764,
- "list_usd": 0.016283
+ "input_tokens": 31158,
+ "cached_input_tokens": 26624,
+ "output_tokens": 131,
+ "list_usd": 0.01304
},
{
- "arm": "graff-v302-11",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 41.66,
+ "wall_s": 22.95,
"model_calls": 4,
- "input_tokens": 46759,
- "cached_input_tokens": 42624,
- "output_tokens": 742,
- "list_usd": 0.019952
+ "input_tokens": 42668,
+ "cached_input_tokens": 41344,
+ "output_tokens": 260,
+ "list_usd": 0.009382
},
{
- "arm": "graff-v302-11",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 32.96,
+ "wall_s": 20.02,
"model_calls": 4,
- "input_tokens": 46403,
- "cached_input_tokens": 43264,
- "output_tokens": 561,
- "list_usd": 0.016214
+ "input_tokens": 42674,
+ "cached_input_tokens": 41344,
+ "output_tokens": 272,
+ "list_usd": 0.009514
},
{
- "arm": "graff-v302-11",
- "task": "linear-quiet-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 27.51,
- "model_calls": 3,
- "input_tokens": 34085,
- "cached_input_tokens": 30720,
- "output_tokens": 581,
- "list_usd": 0.015612
+ "wall_s": 21.31,
+ "model_calls": 4,
+ "input_tokens": 42407,
+ "cached_input_tokens": 33280,
+ "output_tokens": 239,
+ "list_usd": 0.023972
},
{
- "arm": "codex-app-server",
- "task": "linear-reduce",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 32.76,
- "model_calls": 5,
- "input_tokens": 107419,
- "cached_input_tokens": 89344,
- "output_tokens": 608,
- "list_usd": 0.051164
+ "wall_s": 21.0,
+ "model_calls": 3,
+ "input_tokens": 31815,
+ "cached_input_tokens": 30464,
+ "output_tokens": 326,
+ "list_usd": 0.009008
},
{
- "arm": "codex-app-server",
- "task": "linear-reduce",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 33.79,
- "model_calls": 5,
- "input_tokens": 107311,
- "cached_input_tokens": 92928,
- "output_tokens": 626,
- "list_usd": 0.044319
+ "wall_s": 21.12,
+ "model_calls": 3,
+ "input_tokens": 31791,
+ "cached_input_tokens": 30464,
+ "output_tokens": 307,
+ "list_usd": 0.00877
},
{
- "arm": "codex-app-server",
- "task": "linear-reduce",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 37.32,
- "model_calls": 6,
- "input_tokens": 133180,
- "cached_input_tokens": 116480,
- "output_tokens": 626,
- "list_usd": 0.051308
+ "wall_s": 22.3,
+ "model_calls": 3,
+ "input_tokens": 31852,
+ "cached_input_tokens": 26624,
+ "output_tokens": 367,
+ "list_usd": 0.016788
},
{
- "arm": "graff-next",
- "task": "linear-reduce",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 37.1,
- "model_calls": 5,
- "input_tokens": 56886,
- "cached_input_tokens": 52224,
- "output_tokens": 653,
- "list_usd": 0.021076
+ "wall_s": 18.13,
+ "model_calls": 3,
+ "input_tokens": 31395,
+ "cached_input_tokens": 30592,
+ "output_tokens": 175,
+ "list_usd": 0.006415
},
{
- "arm": "graff-next",
- "task": "linear-reduce",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 36.92,
- "model_calls": 4,
- "input_tokens": 45136,
- "cached_input_tokens": 41984,
- "output_tokens": 614,
- "list_usd": 0.016642
+ "wall_s": 16.53,
+ "model_calls": 3,
+ "input_tokens": 31400,
+ "cached_input_tokens": 30592,
+ "output_tokens": 180,
+ "list_usd": 0.006475
},
{
- "arm": "graff-next",
- "task": "linear-reduce",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 36.74,
- "model_calls": 5,
- "input_tokens": 56686,
- "cached_input_tokens": 53248,
- "output_tokens": 645,
- "list_usd": 0.018651
+ "wall_s": 15.79,
+ "model_calls": 3,
+ "input_tokens": 31401,
+ "cached_input_tokens": 26752,
+ "output_tokens": 181,
+ "list_usd": 0.013783
},
{
- "arm": "graff-v302-11",
- "task": "linear-reduce",
+ "arm": "graff",
+ "task": "linear-issues",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 29.49,
- "model_calls": 4,
- "input_tokens": 45075,
- "cached_input_tokens": 41984,
- "output_tokens": 539,
- "list_usd": 0.01577
+ "wall_s": 23.02,
+ "model_calls": 2,
+ "input_tokens": 22866,
+ "cached_input_tokens": 20224,
+ "output_tokens": 422,
+ "list_usd": 0.011526
},
{
- "arm": "graff-v302-11",
- "task": "linear-reduce",
+ "arm": "graff",
+ "task": "linear-issues",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 33.75,
- "model_calls": 4,
- "input_tokens": 45113,
- "cached_input_tokens": 40960,
- "output_tokens": 581,
- "list_usd": 0.018212
+ "wall_s": 23.65,
+ "model_calls": 2,
+ "input_tokens": 22836,
+ "cached_input_tokens": 20736,
+ "output_tokens": 434,
+ "list_usd": 0.010614
},
{
- "arm": "graff-v302-11",
- "task": "linear-reduce",
+ "arm": "graff",
+ "task": "linear-issues",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 51.42,
- "model_calls": 6,
- "input_tokens": 69541,
- "cached_input_tokens": 63488,
- "output_tokens": 966,
- "list_usd": 0.028115
+ "wall_s": 21.87,
+ "model_calls": 2,
+ "input_tokens": 22789,
+ "cached_input_tokens": 10112,
+ "output_tokens": 386,
+ "list_usd": 0.030225
},
{
- "arm": "codex-app-server",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-nohint",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 35.38,
- "model_calls": 6,
- "input_tokens": 96202,
- "cached_input_tokens": 86784,
- "output_tokens": 590,
- "list_usd": 0.033414
+ "wall_s": 20.44,
+ "model_calls": 2,
+ "input_tokens": 22777,
+ "cached_input_tokens": 20224,
+ "output_tokens": 399,
+ "list_usd": 0.011118
},
{
- "arm": "codex-app-server",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-nohint",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 34.92,
- "model_calls": 6,
- "input_tokens": 96157,
- "cached_input_tokens": 87040,
- "output_tokens": 549,
- "list_usd": 0.032428
+ "wall_s": 37.99,
+ "model_calls": 5,
+ "input_tokens": 58041,
+ "cached_input_tokens": 52352,
+ "output_tokens": 561,
+ "list_usd": 0.022223
},
{
- "arm": "codex-app-server",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-nohint",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 38.31,
- "model_calls": 6,
- "input_tokens": 96155,
- "cached_input_tokens": 86784,
- "output_tokens": 568,
- "list_usd": 0.0331
+ "wall_s": 33.81,
+ "model_calls": 5,
+ "input_tokens": 58107,
+ "cached_input_tokens": 50560,
+ "output_tokens": 525,
+ "list_usd": 0.0254
},
{
- "arm": "graff-next",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-nohint-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 32.54,
- "model_calls": 5,
- "input_tokens": 57658,
- "cached_input_tokens": 52480,
- "output_tokens": 559,
- "list_usd": 0.021194
+ "wall_s": 34.92,
+ "model_calls": 4,
+ "input_tokens": 47810,
+ "cached_input_tokens": 40448,
+ "output_tokens": 611,
+ "list_usd": 0.024879
},
{
- "arm": "graff-next",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-nohint-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 38.58,
- "model_calls": 5,
- "input_tokens": 57972,
- "cached_input_tokens": 52480,
- "output_tokens": 707,
- "list_usd": 0.023302
+ "wall_s": 26.5,
+ "model_calls": 3,
+ "input_tokens": 35402,
+ "cached_input_tokens": 31104,
+ "output_tokens": 451,
+ "list_usd": 0.016216
},
{
- "arm": "graff-next",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-nohint-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 32.26,
- "model_calls": 4,
- "input_tokens": 45909,
- "cached_input_tokens": 42240,
- "output_tokens": 594,
- "list_usd": 0.017502
+ "wall_s": 21.9,
+ "model_calls": 2,
+ "input_tokens": 23300,
+ "cached_input_tokens": 20992,
+ "output_tokens": 420,
+ "list_usd": 0.010915
},
{
- "arm": "graff-v302-11",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-quiet",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 32.23,
- "model_calls": 4,
- "input_tokens": 45833,
- "cached_input_tokens": 42240,
- "output_tokens": 561,
- "list_usd": 0.01702
+ "wall_s": 22.5,
+ "model_calls": 2,
+ "input_tokens": 22933,
+ "cached_input_tokens": 20736,
+ "output_tokens": 421,
+ "list_usd": 0.010678
},
{
- "arm": "graff-v302-11",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-quiet",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 40.59,
- "model_calls": 5,
- "input_tokens": 57909,
- "cached_input_tokens": 52608,
- "output_tokens": 656,
- "list_usd": 0.022423
+ "wall_s": 23.89,
+ "model_calls": 2,
+ "input_tokens": 22916,
+ "cached_input_tokens": 20736,
+ "output_tokens": 409,
+ "list_usd": 0.010524
},
{
- "arm": "graff-v302-11",
- "task": "linear-reduce-warm",
+ "arm": "graff",
+ "task": "linear-quiet",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 45.54,
- "model_calls": 6,
- "input_tokens": 69948,
- "cached_input_tokens": 66176,
- "output_tokens": 747,
- "list_usd": 0.021632
+ "wall_s": 19.72,
+ "model_calls": 2,
+ "input_tokens": 22815,
+ "cached_input_tokens": 20224,
+ "output_tokens": 357,
+ "list_usd": 0.010774
},
{
- "arm": "codex-app-server",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-quiet-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 27.2,
- "model_calls": 4,
- "input_tokens": 78016,
- "cached_input_tokens": 65664,
- "output_tokens": 420,
- "list_usd": 0.03547
+ "wall_s": 21.03,
+ "model_calls": 2,
+ "input_tokens": 23413,
+ "cached_input_tokens": 20224,
+ "output_tokens": 385,
+ "list_usd": 0.01225
},
{
- "arm": "codex-app-server",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-quiet-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 27.82,
- "model_calls": 4,
- "input_tokens": 105531,
- "cached_input_tokens": 82048,
- "output_tokens": 417,
- "list_usd": 0.059341
+ "wall_s": 21.77,
+ "model_calls": 2,
+ "input_tokens": 23480,
+ "cached_input_tokens": 20992,
+ "output_tokens": 442,
+ "list_usd": 0.011495
},
{
- "arm": "codex-app-server",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-quiet-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 26.4,
- "model_calls": 4,
- "input_tokens": 104169,
- "cached_input_tokens": 83072,
- "output_tokens": 432,
- "list_usd": 0.054821
+ "wall_s": 29.15,
+ "model_calls": 3,
+ "input_tokens": 35478,
+ "cached_input_tokens": 31872,
+ "output_tokens": 555,
+ "list_usd": 0.015949
},
{
- "arm": "graff-next",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-reduce",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 49.54,
- "model_calls": 7,
- "input_tokens": 80485,
- "cached_input_tokens": 76160,
- "output_tokens": 967,
- "list_usd": 0.025936
+ "wall_s": 21.84,
+ "model_calls": 2,
+ "input_tokens": 22900,
+ "cached_input_tokens": 20736,
+ "output_tokens": 425,
+ "list_usd": 0.010652
},
{
- "arm": "graff-next",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-reduce",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 52.58,
- "model_calls": 7,
- "input_tokens": 80592,
- "cached_input_tokens": 75776,
- "output_tokens": 979,
- "list_usd": 0.027
+ "wall_s": 25.31,
+ "model_calls": 2,
+ "input_tokens": 23013,
+ "cached_input_tokens": 20736,
+ "output_tokens": 487,
+ "list_usd": 0.011498
},
{
- "arm": "graff-next",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-reduce",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 38.4,
- "model_calls": 6,
- "input_tokens": 68164,
- "cached_input_tokens": 64000,
- "output_tokens": 763,
- "list_usd": 0.022358
+ "wall_s": 20.31,
+ "model_calls": 2,
+ "input_tokens": 22917,
+ "cached_input_tokens": 20736,
+ "output_tokens": 395,
+ "list_usd": 0.010386
},
{
- "arm": "graff-v302-11",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-reduce-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 47.53,
- "model_calls": 6,
- "input_tokens": 68696,
- "cached_input_tokens": 63360,
- "output_tokens": 986,
- "list_usd": 0.026868
+ "wall_s": 23.4,
+ "model_calls": 3,
+ "input_tokens": 35190,
+ "cached_input_tokens": 31104,
+ "output_tokens": 442,
+ "list_usd": 0.015702
},
{
- "arm": "graff-v302-11",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-reduce-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 49.91,
- "model_calls": 7,
- "input_tokens": 80552,
- "cached_input_tokens": 73472,
- "output_tokens": 1000,
- "list_usd": 0.031507
+ "wall_s": 31.58,
+ "model_calls": 3,
+ "input_tokens": 35247,
+ "cached_input_tokens": 31872,
+ "output_tokens": 491,
+ "list_usd": 0.014847
},
{
- "arm": "graff-v302-11",
- "task": "linear-sidecar",
+ "arm": "graff",
+ "task": "linear-reduce-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 49.5,
- "model_calls": 7,
- "input_tokens": 80632,
- "cached_input_tokens": 74368,
- "output_tokens": 960,
- "list_usd": 0.029565
+ "wall_s": 24.86,
+ "model_calls": 3,
+ "input_tokens": 35204,
+ "cached_input_tokens": 32640,
+ "output_tokens": 455,
+ "list_usd": 0.012942
},
{
- "arm": "codex-app-server",
- "task": "linear-split",
+ "arm": "graff",
+ "task": "linear-sidecar",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 60.23,
- "model_calls": 16,
- "input_tokens": 366660,
- "cached_input_tokens": 309632,
- "output_tokens": 1384,
- "list_usd": null
+ "wall_s": 36.66,
+ "model_calls": 4,
+ "input_tokens": 50303,
+ "cached_input_tokens": 42112,
+ "output_tokens": 690,
+ "list_usd": 0.027493
},
{
- "arm": "codex-app-server",
- "task": "linear-split",
+ "arm": "graff",
+ "task": "linear-sidecar",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 57.52,
- "model_calls": 16,
- "input_tokens": 367431,
- "cached_input_tokens": 312704,
- "output_tokens": 1416,
- "list_usd": null
+ "wall_s": 41.79,
+ "model_calls": 5,
+ "input_tokens": 65453,
+ "cached_input_tokens": 59392,
+ "output_tokens": 853,
+ "list_usd": 0.026591
},
{
- "arm": "codex-app-server",
- "task": "linear-split",
+ "arm": "graff",
+ "task": "linear-sidecar",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 56.25,
- "model_calls": 18,
- "input_tokens": 439318,
- "cached_input_tokens": 377344,
- "output_tokens": 1495,
- "list_usd": null
+ "wall_s": 36.38,
+ "model_calls": 4,
+ "input_tokens": 51164,
+ "cached_input_tokens": 45440,
+ "output_tokens": 553,
+ "list_usd": 0.021522
},
{
- "arm": "graff-next",
+ "arm": "graff",
"task": "linear-split",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 67.91,
- "model_calls": 18,
- "input_tokens": 190207,
- "cached_input_tokens": 167680,
- "output_tokens": 1688,
- "list_usd": 0.078702
+ "wall_s": 46.43,
+ "model_calls": 8,
+ "input_tokens": 83470,
+ "cached_input_tokens": 77312,
+ "output_tokens": 1175,
+ "list_usd": 0.031797
},
{
- "arm": "graff-next",
+ "arm": "graff",
"task": "linear-split",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 58.66,
- "model_calls": 17,
- "input_tokens": 176523,
- "cached_input_tokens": 160384,
- "output_tokens": 1628,
- "list_usd": 0.064596
+ "wall_s": 63.62,
+ "model_calls": 8,
+ "input_tokens": 83506,
+ "cached_input_tokens": 76672,
+ "output_tokens": 1257,
+ "list_usd": 0.033905
},
{
- "arm": "graff-next",
+ "arm": "graff",
"task": "linear-split",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 70.64,
- "model_calls": 18,
- "input_tokens": 190968,
- "cached_input_tokens": 174720,
- "output_tokens": 1823,
- "list_usd": 0.068198
+ "wall_s": 47.37,
+ "model_calls": 8,
+ "input_tokens": 83664,
+ "cached_input_tokens": 78336,
+ "output_tokens": 1255,
+ "list_usd": 0.03104
},
{
- "arm": "graff-v302-11",
- "task": "linear-split",
+ "arm": "graff",
+ "task": "linear-warm",
"suite": "mcp",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 140.58,
- "model_calls": 21,
- "input_tokens": 256931,
- "cached_input_tokens": 213760,
- "output_tokens": 4107,
- "list_usd": 0.148788
+ "wall_s": 22.62,
+ "model_calls": 2,
+ "input_tokens": 23364,
+ "cached_input_tokens": 20224,
+ "output_tokens": 418,
+ "list_usd": 0.012482
},
{
- "arm": "graff-v302-11",
- "task": "linear-split",
+ "arm": "graff",
+ "task": "linear-warm",
"suite": "mcp",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 107.78,
- "model_calls": 21,
- "input_tokens": 226902,
- "cached_input_tokens": 199808,
- "output_tokens": 3823,
- "list_usd": 0.112399
+ "wall_s": 21.14,
+ "model_calls": 2,
+ "input_tokens": 23372,
+ "cached_input_tokens": 20224,
+ "output_tokens": 421,
+ "list_usd": 0.012528
},
{
- "arm": "graff-v302-11",
- "task": "linear-split",
+ "arm": "graff",
+ "task": "linear-warm",
"suite": "mcp",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 95.44,
- "model_calls": 17,
- "input_tokens": 202662,
- "cached_input_tokens": 177664,
- "output_tokens": 2686,
- "list_usd": 0.094622
+ "wall_s": 25.22,
+ "model_calls": 2,
+ "input_tokens": 23334,
+ "cached_input_tokens": 20992,
+ "output_tokens": 415,
+ "list_usd": 0.010933
},
{
- "arm": "codex-app-server",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 26.83,
- "model_calls": 4,
- "input_tokens": 82420,
- "cached_input_tokens": 66688,
- "output_tokens": 407,
- "list_usd": 0.042203
+ "wall_s": 10.96,
+ "model_calls": 2,
+ "input_tokens": 20929,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003236
},
{
- "arm": "codex-app-server",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 27.12,
- "model_calls": 4,
- "input_tokens": 78746,
- "cached_input_tokens": 64256,
- "output_tokens": 409,
- "list_usd": 0.039496
- },
- {
- "arm": "codex-app-server",
- "task": "linear-warm",
- "suite": "mcp",
+ "wall_s": 8.48,
+ "model_calls": 2,
+ "input_tokens": 20929,
+ "cached_input_tokens": 16768,
+ "output_tokens": 29,
+ "list_usd": 0.010289
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 24.51,
- "model_calls": 4,
- "input_tokens": 78756,
- "cached_input_tokens": 64256,
- "output_tokens": 416,
- "list_usd": 0.039586
+ "wall_s": 10.2,
+ "model_calls": 2,
+ "input_tokens": 20929,
+ "cached_input_tokens": 16512,
+ "output_tokens": 29,
+ "list_usd": 0.010775
},
{
- "arm": "graff-next",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 26.57,
- "model_calls": 3,
- "input_tokens": 33957,
- "cached_input_tokens": 30720,
- "output_tokens": 513,
- "list_usd": 0.014676
+ "wall_s": 25.3,
+ "model_calls": 4,
+ "input_tokens": 42255,
+ "cached_input_tokens": 34688,
+ "output_tokens": 280,
+ "list_usd": 0.021403
},
{
- "arm": "graff-next",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 29.41,
+ "wall_s": 22.34,
"model_calls": 4,
- "input_tokens": 46236,
- "cached_input_tokens": 40960,
- "output_tokens": 537,
- "list_usd": 0.020018
+ "input_tokens": 42253,
+ "cached_input_tokens": 40448,
+ "output_tokens": 280,
+ "list_usd": 0.010455
},
{
- "arm": "graff-next",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 30.59,
+ "wall_s": 21.46,
"model_calls": 4,
- "input_tokens": 46376,
- "cached_input_tokens": 41728,
- "output_tokens": 511,
- "list_usd": 0.018579
+ "input_tokens": 42240,
+ "cached_input_tokens": 40448,
+ "output_tokens": 275,
+ "list_usd": 0.010379
},
{
- "arm": "graff-v302-11",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 29.73,
- "model_calls": 4,
- "input_tokens": 46192,
- "cached_input_tokens": 40960,
- "output_tokens": 523,
- "list_usd": 0.01979
+ "wall_s": 15.54,
+ "model_calls": 3,
+ "input_tokens": 31356,
+ "cached_input_tokens": 30336,
+ "output_tokens": 153,
+ "list_usd": 0.006604
},
{
- "arm": "graff-v302-11",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 35.51,
- "model_calls": 5,
- "input_tokens": 58221,
- "cached_input_tokens": 54016,
- "output_tokens": 599,
- "list_usd": 0.019802
+ "wall_s": 20.09,
+ "model_calls": 4,
+ "input_tokens": 41931,
+ "cached_input_tokens": 40704,
+ "output_tokens": 179,
+ "list_usd": 0.008314
},
{
- "arm": "graff-v302-11",
- "task": "linear-warm",
- "suite": "mcp",
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 39.91,
- "model_calls": 4,
- "input_tokens": 46634,
- "cached_input_tokens": 42496,
- "output_tokens": 914,
- "list_usd": 0.021666
+ "wall_s": 15.89,
+ "model_calls": 3,
+ "input_tokens": 31358,
+ "cached_input_tokens": 30464,
+ "output_tokens": 154,
+ "list_usd": 0.006374
},
{
- "arm": "codex-app-server",
- "task": "recall-noise",
+ "arm": "graff",
+ "task": "write-tests",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 10.81,
- "model_calls": 2,
- "input_tokens": 28817,
- "cached_input_tokens": 24576,
- "output_tokens": 57,
- "list_usd": 0.01151
+ "wall_s": 21.6,
+ "model_calls": 4,
+ "input_tokens": 42032,
+ "cached_input_tokens": 40960,
+ "output_tokens": 255,
+ "list_usd": 0.00879
},
{
- "arm": "codex-app-server",
- "task": "recall-noise",
+ "arm": "graff",
+ "task": "write-tests",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 10.09,
- "model_calls": 2,
- "input_tokens": 28814,
- "cached_input_tokens": 24576,
- "output_tokens": 57,
- "list_usd": 0.011504
+ "wall_s": 19.43,
+ "model_calls": 4,
+ "input_tokens": 41997,
+ "cached_input_tokens": 40832,
+ "output_tokens": 236,
+ "list_usd": 0.008773
},
{
- "arm": "codex-app-server",
- "task": "recall-noise",
+ "arm": "graff",
+ "task": "write-tests",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 9.69,
- "model_calls": 2,
- "input_tokens": 28808,
- "cached_input_tokens": 26368,
- "output_tokens": 52,
- "list_usd": 0.008037
+ "wall_s": 20.05,
+ "model_calls": 4,
+ "input_tokens": 41989,
+ "cached_input_tokens": 40960,
+ "output_tokens": 233,
+ "list_usd": 0.008484
},
{
- "arm": "graff-next",
- "task": "recall-noise",
+ "arm": "pi",
+ "task": "csv-sum",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 7.4,
- "model_calls": 2,
- "input_tokens": 20859,
- "cached_input_tokens": 20480,
- "output_tokens": 29,
- "list_usd": 0.003096
+ "wall_s": 15.0,
+ "model_calls": 3,
+ "input_tokens": 3573,
+ "cached_input_tokens": 1024,
+ "output_tokens": 58,
+ "list_usd": 0.00578
},
{
- "arm": "graff-next",
- "task": "recall-noise",
+ "arm": "pi",
+ "task": "csv-sum",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 6.69,
- "model_calls": 2,
- "input_tokens": 20859,
- "cached_input_tokens": 16768,
- "output_tokens": 29,
- "list_usd": 0.010149
+ "wall_s": 13.29,
+ "model_calls": 3,
+ "input_tokens": 3573,
+ "cached_input_tokens": 2048,
+ "output_tokens": 58,
+ "list_usd": 0.003835
},
{
- "arm": "graff-next",
- "task": "recall-noise",
+ "arm": "pi",
+ "task": "csv-sum",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 8.1,
- "model_calls": 2,
- "input_tokens": 20859,
- "cached_input_tokens": 20480,
- "output_tokens": 29,
- "list_usd": 0.003096
+ "wall_s": 13.23,
+ "model_calls": 3,
+ "input_tokens": 3573,
+ "cached_input_tokens": 1024,
+ "output_tokens": 58,
+ "list_usd": 0.00578
},
{
- "arm": "graff-v302-11",
- "task": "recall-noise",
+ "arm": "pi",
+ "task": "dead-code",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 7.56,
- "model_calls": 2,
- "input_tokens": 20859,
- "cached_input_tokens": 20480,
- "output_tokens": 29,
- "list_usd": 0.003096
+ "wall_s": 18.01,
+ "model_calls": 4,
+ "input_tokens": 4977,
+ "cached_input_tokens": 3200,
+ "output_tokens": 73,
+ "list_usd": 0.004604
},
{
- "arm": "graff-v302-11",
- "task": "recall-noise",
+ "arm": "pi",
+ "task": "dead-code",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 6.68,
- "model_calls": 2,
- "input_tokens": 20859,
- "cached_input_tokens": 20480,
- "output_tokens": 29,
- "list_usd": 0.003096
+ "wall_s": 16.44,
+ "model_calls": 4,
+ "input_tokens": 4965,
+ "cached_input_tokens": 2048,
+ "output_tokens": 69,
+ "list_usd": 0.006729
},
{
- "arm": "graff-v302-11",
- "task": "recall-noise",
+ "arm": "pi",
+ "task": "dead-code",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 7.38,
- "model_calls": 2,
- "input_tokens": 20859,
- "cached_input_tokens": 20224,
- "output_tokens": 43,
- "list_usd": 0.003722
+ "wall_s": 20.18,
+ "model_calls": 4,
+ "input_tokens": 5082,
+ "cached_input_tokens": 1024,
+ "output_tokens": 93,
+ "list_usd": 0.009148
},
{
- "arm": "codex-app-server",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "exact-reply",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 28.72,
- "model_calls": 4,
- "input_tokens": 58425,
- "cached_input_tokens": 50944,
- "output_tokens": 365,
- "list_usd": 0.023706
+ "wall_s": 7.38,
+ "model_calls": 1,
+ "input_tokens": 1115,
+ "cached_input_tokens": 0,
+ "output_tokens": 5,
+ "list_usd": 0.00228
},
{
- "arm": "codex-app-server",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "exact-reply",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 26.76,
- "model_calls": 4,
- "input_tokens": 58340,
- "cached_input_tokens": 50944,
- "output_tokens": 374,
- "list_usd": 0.023626
+ "wall_s": 6.55,
+ "model_calls": 1,
+ "input_tokens": 1115,
+ "cached_input_tokens": 0,
+ "output_tokens": 5,
+ "list_usd": 0.00228
},
{
- "arm": "codex-app-server",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "exact-reply",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 25.34,
- "model_calls": 4,
- "input_tokens": 58471,
- "cached_input_tokens": 52992,
- "output_tokens": 375,
- "list_usd": 0.020007
+ "wall_s": 11.46,
+ "model_calls": 1,
+ "input_tokens": 1115,
+ "cached_input_tokens": 0,
+ "output_tokens": 5,
+ "list_usd": 0.00228
},
{
- "arm": "graff-next",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "file-ops",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 20.81,
- "model_calls": 4,
- "input_tokens": 42164,
- "cached_input_tokens": 40576,
- "output_tokens": 304,
- "list_usd": 0.010274
+ "wall_s": 25.16,
+ "model_calls": 3,
+ "input_tokens": 3540,
+ "cached_input_tokens": 2048,
+ "output_tokens": 78,
+ "list_usd": 0.003969
},
{
- "arm": "graff-next",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "file-ops",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 20.8,
- "model_calls": 4,
- "input_tokens": 42152,
- "cached_input_tokens": 40832,
- "output_tokens": 310,
- "list_usd": 0.009823
+ "wall_s": 27.22,
+ "model_calls": 3,
+ "input_tokens": 3540,
+ "cached_input_tokens": 0,
+ "output_tokens": 78,
+ "list_usd": 0.00786
},
{
- "arm": "graff-next",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "file-ops",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 20.5,
- "model_calls": 4,
- "input_tokens": 42164,
- "cached_input_tokens": 40704,
- "output_tokens": 316,
- "list_usd": 0.01015
+ "wall_s": 16.71,
+ "model_calls": 3,
+ "input_tokens": 3540,
+ "cached_input_tokens": 1024,
+ "output_tokens": 78,
+ "list_usd": 0.005914
},
{
- "arm": "graff-v302-11",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "fix-fib",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 21.19,
- "model_calls": 4,
- "input_tokens": 42142,
- "cached_input_tokens": 40704,
- "output_tokens": 305,
- "list_usd": 0.009996
+ "wall_s": 22.03,
+ "model_calls": 5,
+ "input_tokens": 6612,
+ "cached_input_tokens": 2176,
+ "output_tokens": 149,
+ "list_usd": 0.01058
},
{
- "arm": "graff-v302-11",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "fix-fib",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 30.0,
- "model_calls": 4,
- "input_tokens": 42190,
- "cached_input_tokens": 40832,
- "output_tokens": 332,
- "list_usd": 0.010119
+ "wall_s": 22.07,
+ "model_calls": 5,
+ "input_tokens": 6612,
+ "cached_input_tokens": 2304,
+ "output_tokens": 149,
+ "list_usd": 0.010336
},
{
- "arm": "graff-v302-11",
- "task": "refactor-rename",
+ "arm": "pi",
+ "task": "fix-fib",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 21.71,
- "model_calls": 4,
- "input_tokens": 42186,
- "cached_input_tokens": 41088,
- "output_tokens": 327,
- "list_usd": 0.009575
+ "wall_s": 22.87,
+ "model_calls": 5,
+ "input_tokens": 6612,
+ "cached_input_tokens": 2432,
+ "output_tokens": 149,
+ "list_usd": 0.010093
},
{
- "arm": "codex-app-server",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "git-ops",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 11.33,
- "model_calls": 2,
- "input_tokens": 28686,
- "cached_input_tokens": 26368,
- "output_tokens": 111,
- "list_usd": 0.008383
+ "wall_s": 14.13,
+ "model_calls": 3,
+ "input_tokens": 3956,
+ "cached_input_tokens": 2176,
+ "output_tokens": 137,
+ "list_usd": 0.005148
},
{
- "arm": "codex-app-server",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "git-ops",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 16.72,
+ "wall_s": 15.88,
"model_calls": 3,
- "input_tokens": 43218,
- "cached_input_tokens": 36864,
- "output_tokens": 172,
- "list_usd": 0.018114
+ "input_tokens": 3971,
+ "cached_input_tokens": 2176,
+ "output_tokens": 143,
+ "list_usd": 0.005238
},
{
- "arm": "codex-app-server",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "git-ops",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 14.27,
+ "wall_s": 16.54,
"model_calls": 3,
- "input_tokens": 43241,
- "cached_input_tokens": 38656,
- "output_tokens": 128,
- "list_usd": 0.014316
+ "input_tokens": 3957,
+ "cached_input_tokens": 1024,
+ "output_tokens": 142,
+ "list_usd": 0.007388
},
{
- "arm": "graff-next",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "json-transform",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 20.4,
- "model_calls": 4,
- "input_tokens": 41918,
- "cached_input_tokens": 40704,
- "output_tokens": 230,
- "list_usd": 0.008798
+ "wall_s": 15.03,
+ "model_calls": 3,
+ "input_tokens": 3663,
+ "cached_input_tokens": 1024,
+ "output_tokens": 94,
+ "list_usd": 0.00632
},
{
- "arm": "graff-next",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "json-transform",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 17.32,
- "model_calls": 2,
- "input_tokens": 20863,
- "cached_input_tokens": 20224,
- "output_tokens": 224,
- "list_usd": 0.00554
+ "wall_s": 15.51,
+ "model_calls": 3,
+ "input_tokens": 3663,
+ "cached_input_tokens": 2048,
+ "output_tokens": 94,
+ "list_usd": 0.004375
},
{
- "arm": "graff-next",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "json-transform",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 19.68,
+ "wall_s": 15.01,
"model_calls": 3,
- "input_tokens": 31380,
- "cached_input_tokens": 30336,
- "output_tokens": 280,
- "list_usd": 0.007922
+ "input_tokens": 3663,
+ "cached_input_tokens": 0,
+ "output_tokens": 95,
+ "list_usd": 0.008276
},
{
- "arm": "graff-v302-11",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "recall-noise",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 17.81,
- "model_calls": 3,
- "input_tokens": 31390,
- "cached_input_tokens": 30208,
- "output_tokens": 206,
- "list_usd": 0.007445
+ "wall_s": 9.56,
+ "model_calls": 2,
+ "input_tokens": 2499,
+ "cached_input_tokens": 0,
+ "output_tokens": 28,
+ "list_usd": 0.005278
},
{
- "arm": "graff-v302-11",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "recall-noise",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 17.79,
- "model_calls": 3,
- "input_tokens": 31340,
- "cached_input_tokens": 30464,
- "output_tokens": 240,
- "list_usd": 0.007198
+ "wall_s": 9.73,
+ "model_calls": 2,
+ "input_tokens": 2499,
+ "cached_input_tokens": 0,
+ "output_tokens": 28,
+ "list_usd": 0.005278
},
{
- "arm": "graff-v302-11",
- "task": "regex-count",
+ "arm": "pi",
+ "task": "recall-noise",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 20.94,
- "model_calls": 3,
- "input_tokens": 31387,
- "cached_input_tokens": 30208,
- "output_tokens": 287,
- "list_usd": 0.008249
+ "wall_s": 11.65,
+ "model_calls": 2,
+ "input_tokens": 2499,
+ "cached_input_tokens": 0,
+ "output_tokens": 28,
+ "list_usd": 0.005278
},
{
- "arm": "codex-app-server",
- "task": "write-tests",
+ "arm": "pi",
+ "task": "refactor-rename",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 22.95,
- "model_calls": 4,
- "input_tokens": 57983,
- "cached_input_tokens": 54912,
- "output_tokens": 348,
- "list_usd": 0.015113
+ "wall_s": 23.26,
+ "model_calls": 5,
+ "input_tokens": 7583,
+ "cached_input_tokens": 2688,
+ "output_tokens": 289,
+ "list_usd": 0.012949
},
{
- "arm": "codex-app-server",
- "task": "write-tests",
+ "arm": "pi",
+ "task": "refactor-rename",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 22.81,
- "model_calls": 4,
- "input_tokens": 58008,
- "cached_input_tokens": 52992,
- "output_tokens": 356,
- "list_usd": 0.018891
+ "wall_s": 24.13,
+ "model_calls": 5,
+ "input_tokens": 6947,
+ "cached_input_tokens": 1152,
+ "output_tokens": 281,
+ "list_usd": 0.014515
},
{
- "arm": "codex-app-server",
- "task": "write-tests",
+ "arm": "pi",
+ "task": "refactor-rename",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 25.03,
- "model_calls": 4,
- "input_tokens": 58015,
- "cached_input_tokens": 52864,
- "output_tokens": 361,
- "list_usd": 0.019198
+ "wall_s": 22.81,
+ "model_calls": 5,
+ "input_tokens": 7008,
+ "cached_input_tokens": 2432,
+ "output_tokens": 279,
+ "list_usd": 0.012185
},
{
- "arm": "graff-next",
- "task": "write-tests",
+ "arm": "pi",
+ "task": "regex-count",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 20.07,
- "model_calls": 4,
- "input_tokens": 41938,
- "cached_input_tokens": 40320,
- "output_tokens": 280,
- "list_usd": 0.010068
+ "wall_s": 11.5,
+ "model_calls": 2,
+ "input_tokens": 2361,
+ "cached_input_tokens": 1024,
+ "output_tokens": 55,
+ "list_usd": 0.003326
},
{
- "arm": "graff-next",
- "task": "write-tests",
+ "arm": "pi",
+ "task": "regex-count",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 23.39,
- "model_calls": 4,
- "input_tokens": 41909,
- "cached_input_tokens": 40576,
- "output_tokens": 269,
- "list_usd": 0.009414
+ "wall_s": 12.91,
+ "model_calls": 2,
+ "input_tokens": 2357,
+ "cached_input_tokens": 1024,
+ "output_tokens": 51,
+ "list_usd": 0.003278
},
{
- "arm": "graff-next",
- "task": "write-tests",
+ "arm": "pi",
+ "task": "regex-count",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 21.42,
- "model_calls": 4,
- "input_tokens": 41933,
- "cached_input_tokens": 40576,
- "output_tokens": 280,
- "list_usd": 0.009572
+ "wall_s": 15.61,
+ "model_calls": 2,
+ "input_tokens": 2357,
+ "cached_input_tokens": 1024,
+ "output_tokens": 51,
+ "list_usd": 0.003278
},
{
- "arm": "graff-v302-11",
+ "arm": "pi",
"task": "write-tests",
"suite": "core",
"rep": 1,
"pass": true,
"timed_out": false,
- "wall_s": 23.26,
- "model_calls": 4,
- "input_tokens": 41950,
- "cached_input_tokens": 40192,
- "output_tokens": 298,
- "list_usd": 0.010515
+ "wall_s": 29.83,
+ "model_calls": 5,
+ "input_tokens": 6414,
+ "cached_input_tokens": 2048,
+ "output_tokens": 266,
+ "list_usd": 0.011597
},
{
- "arm": "graff-v302-11",
+ "arm": "pi",
"task": "write-tests",
"suite": "core",
"rep": 2,
"pass": true,
"timed_out": false,
- "wall_s": 22.2,
- "model_calls": 4,
- "input_tokens": 41960,
- "cached_input_tokens": 40320,
- "output_tokens": 297,
- "list_usd": 0.010282
+ "wall_s": 22.82,
+ "model_calls": 5,
+ "input_tokens": 6278,
+ "cached_input_tokens": 3200,
+ "output_tokens": 196,
+ "list_usd": 0.008436
},
{
- "arm": "graff-v302-11",
+ "arm": "pi",
"task": "write-tests",
"suite": "core",
"rep": 3,
"pass": true,
"timed_out": false,
- "wall_s": 20.44,
+ "wall_s": 18.68,
"model_calls": 4,
- "input_tokens": 41912,
- "cached_input_tokens": 40576,
- "output_tokens": 270,
- "list_usd": 0.00943
+ "input_tokens": 5003,
+ "cached_input_tokens": 2048,
+ "output_tokens": 179,
+ "list_usd": 0.007905
}
]
-}
+}
\ No newline at end of file
diff --git a/evals/graff-vs-opencode-and-pi/RESULTS.md b/evals/graff-vs-opencode-and-pi/RESULTS.md
new file mode 100644
index 00000000..36ac2cfd
--- /dev/null
+++ b/evals/graff-vs-opencode-and-pi/RESULTS.md
@@ -0,0 +1,99 @@
+# graff vs OpenCode 2 and pi on a gateway model
+
+Measured 2026-10-01. One model, served through the Codegraff gateway's chat-completions endpoint, ran the same 21 tasks in each harness, side by side, on one machine and one gateway key.
+
+| Arm | What ran |
+| --- | --- |
+| graff | The ADR 0231 build as a scripted `graff repl` (the `graff-gateway-repl` harness). |
+| OpenCode 2 | `@opencode/cli` 2.0.21: `opencode run --standalone --auto --format json`, through [`graff-evals/opencode2_run.py`](../../graff-evals/opencode2_run.py). |
+| pi | pi 0.84.2 (the `pi-codegraff` harness). pi has no MCP client, so it ran the 11 coding tasks only. |
+
+All three send the same model id to the same endpoint, and each runs at its own defaults.
+
+## Results
+
+All 21 tasks (pi: the 11 coding tasks), mean per task over 3 runs of each:
+
+| | Pass | Wall per task | Input read from cache | Output tokens per task | Model calls per task |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| graff (ADR 0231 build) | 63/63 | 9.1s | 94% | 582 | 3.6 |
+| OpenCode 2 | 62/63 | 14.4s | 86% | 980 | 4.2 |
+| pi | 33/33 | 5.4s | 78% | 258 | 2.7 |
+
+By suite:
+
+| Suite | graff (ADR 0231 build) | OpenCode 2 | pi |
+| --- | ---: | ---: | ---: |
+| core (11 coding tasks), wall | 7.3s | 6.9s | 5.4s |
+| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 |
+| mcp (10 MCP tasks), wall | 11.0s | 22.6s | — |
+| mcp (10 MCP tasks), pass | 30/30 | 29/30 | — |
+
+**Against OpenCode 2.** graff is 37% faster per task over all 21 tasks (9.1s against 14.4s). The gap is the MCP tasks, which graff finishes in about half the time (11.0s against 22.6s), with 4.4 model calls to 5.3 and 917 output tokens to 1,655: graff slims large list results and often fetches, reduces and writes the report in one `rlm` script. On the coding tasks OpenCode 2 is slightly faster (6.9s against 7.3s). OpenCode 2 failed one run: on one `linear-split` run it ended while still waiting for its two sub-agents, without writing `report.json`.
+
+**Against pi.** pi is the fastest on the coding tasks (5.4s against graff's 7.3s), with a much smaller prompt and the fewest calls (2.7 per task). It has no MCP client, so it has no MCP result.
+
+graff reads the most of its input from cache (94%, against 86% for OpenCode 2 and 78% for pi).
+
+## Per task
+
+Wall time is the mean of 3 runs.
+
+| Task | Suite | graff (ADR 0231 build) | OpenCode 2 | pi |
+| --- | --- | ---: | ---: | ---: |
+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 9.5s | 7.8s | 5.0s |
+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 6.9s | 6.8s | 6.0s |
+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 1.8s | 2.1s | 1.7s |
+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 4.2s | 5.9s | 4.6s |
+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 8.1s | 7.6s | 7.6s |
+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 6.1s | 6.2s | 4.6s |
+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 5.5s | 6.0s | 5.6s |
+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 7.5s | 30.4s | — |
+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 10.0s | 20.2s | — |
+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 9.9s | 20.4s | — |
+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 11.3s | 20.2s | — |
+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 15.7s | 21.5s | — |
+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 9.5s | 20.9s | — |
+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 8.8s | 19.2s | — |
+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 8.9s | 22.5s | — |
+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 18.9s | 30.0s (2/3) | — |
+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 9.1s | 20.6s | — |
+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.8s | 7.6s | 4.4s |
+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 10.3s | 8.6s | 7.8s |
+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 14.1s | 8.6s | 6.0s |
+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 6.6s | 8.4s | 6.1s |
+
+`results.json` has every run: pass, wall time, model calls, and input, cached input and output tokens.
+
+## Method
+
+- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). OpenCode 2 does not read `.mcp.json`; its wrapper writes the same server into the task's `opencode.json`. Each task passes or fails on its own check.
+- **Order.** 3 runs per task and arm. graff and OpenCode 2 were interleaved task by task (`run.py --interleave`), 6 runs at a time; pi ran its coding tasks afterwards. Each arm made one unscored request first.
+- **Wall time.** The runner's clock around each process, start to exit.
+- **Tokens.** Prompt tokens include cached ones in every arm. OpenCode 2's come from its own session database (`session_v2`), whose input count leaves out cache reads and writes, so the wrapper adds them back. No list cost: the gateway's prices are not public list prices.
+
+## Caveats
+
+- **pi runs only the coding tasks.** It has no MCP client.
+- **One model.** Harness differences can look different on another model; the [Codex comparison](../graff-vs-codex-app-server/RESULTS.md) runs graff and pi on a different one.
+- **Small sample.** 3 runs per task, one machine, one key. A second or two per task is within run-to-run noise.
+
+## Reproduce
+
+```sh
+cd graff-evals
+# CODEGRAFF_API_KEY set; OpenCode 2 needs a `codegraff` provider in its config
+# (openai-compatible, baseURL https://gateway.codegraff.com/v1, apiKey {env:CODEGRAFF_API_KEY})
+# and OPENCODE2_BIN when `opencode` on PATH is 1.x; pi needs the `codegraff`
+# provider in ~/.pi/agent/models.json. Keep --output-root outside any repository,
+# with scripts/linear_fixture_mcp.py copied to <parent of output-root>/scripts/.
+TASKS="--task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \
+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \
+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \
+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \
+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm"
+python3 run.py --harness graff-gateway-repl,opencode2 --model <gateway model> --suite core,mcp $TASKS \
+ --reps 3 -j 6 --interleave --output-root /path/outside/repo/gateway
+python3 run.py --harness pi-codegraff --model <gateway model> --suite core $TASKS \
+ --reps 3 -j 6 --output-root /path/outside/repo/gateway-pi
+```
diff --git a/evals/graff-vs-opencode-and-pi/results.json b/evals/graff-vs-opencode-and-pi/results.json
new file mode 100644
index 00000000..e75e5b52
--- /dev/null
+++ b/evals/graff-vs-opencode-and-pi/results.json
@@ -0,0 +1,2534 @@
+{
+ "model": "one gateway model (same id in every arm)",
+ "tasks": 21,
+ "reps": 3,
+ "arms": {
+ "graff": "graff (ADR 0231 build)",
+ "opencode2": "OpenCode 2",
+ "pi": "pi"
+ },
+ "summary": {
+ "all": {
+ "graff": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 9.1,
+ "cached_input_share": 0.945,
+ "output_tokens_per_task": 582,
+ "model_calls_per_task": 3.6
+ },
+ "opencode2": {
+ "runs": 63,
+ "passed": 62,
+ "wall_s_per_task": 14.4,
+ "cached_input_share": 0.862,
+ "output_tokens_per_task": 980,
+ "model_calls_per_task": 4.2
+ },
+ "pi": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 5.4,
+ "cached_input_share": 0.785,
+ "output_tokens_per_task": 258,
+ "model_calls_per_task": 2.7
+ }
+ },
+ "core": {
+ "graff": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 7.3,
+ "cached_input_share": 0.937,
+ "output_tokens_per_task": 277,
+ "model_calls_per_task": 2.8
+ },
+ "opencode2": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 6.9,
+ "cached_input_share": 0.868,
+ "output_tokens_per_task": 366,
+ "model_calls_per_task": 3.2
+ },
+ "pi": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 5.4,
+ "cached_input_share": 0.785,
+ "output_tokens_per_task": 258,
+ "model_calls_per_task": 2.7
+ }
+ },
+ "mcp": {
+ "graff": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 11.0,
+ "cached_input_share": 0.95,
+ "output_tokens_per_task": 917,
+ "model_calls_per_task": 4.4
+ },
+ "opencode2": {
+ "runs": 30,
+ "passed": 29,
+ "wall_s_per_task": 22.6,
+ "cached_input_share": 0.858,
+ "output_tokens_per_task": 1655,
+ "model_calls_per_task": 5.3
+ }
+ }
+ },
+ "per_task": {
+ "csv-sum": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.5,
+ "model_calls": 3.3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.8,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.0,
+ "model_calls": 3
+ }
+ },
+ "dead-code": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.9,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.8,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.0,
+ "model_calls": 3
+ }
+ },
+ "exact-reply": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 1.8,
+ "model_calls": 1
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 2.1,
+ "model_calls": 1
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 1.7,
+ "model_calls": 1
+ }
+ },
+ "file-ops": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.2,
+ "model_calls": 2
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.9,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.6,
+ "model_calls": 2
+ }
+ },
+ "fix-fib": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.1,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.6,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.6,
+ "model_calls": 4
+ }
+ },
+ "git-ops": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.1,
+ "model_calls": 2
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.2,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.6,
+ "model_calls": 2
+ }
+ },
+ "json-transform": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.5,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.0,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 5.6,
+ "model_calls": 3
+ }
+ },
+ "linear-issues": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.5,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 30.4,
+ "model_calls": 6
+ }
+ },
+ "linear-nohint": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.0,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.2,
+ "model_calls": 5.3
+ }
+ },
+ "linear-nohint-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.9,
+ "model_calls": 3.3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.4,
+ "model_calls": 5.7
+ }
+ },
+ "linear-quiet": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 11.3,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.2,
+ "model_calls": 5.7
+ }
+ },
+ "linear-quiet-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 15.7,
+ "model_calls": 5
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.5,
+ "model_calls": 5
+ }
+ },
+ "linear-reduce": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.5,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.9,
+ "model_calls": 5.7
+ }
+ },
+ "linear-reduce-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.8,
+ "model_calls": 4
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.2,
+ "model_calls": 5.7
+ }
+ },
+ "linear-sidecar": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.9,
+ "model_calls": 3.7
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.5,
+ "model_calls": 6
+ }
+ },
+ "linear-split": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.9,
+ "model_calls": 9
+ },
+ "opencode2": {
+ "passed": 2,
+ "runs": 3,
+ "wall_s": 30.0,
+ "model_calls": 2.7
+ }
+ },
+ "linear-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.1,
+ "model_calls": 3.7
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.6,
+ "model_calls": 5.7
+ }
+ },
+ "recall-noise": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.8,
+ "model_calls": 2
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.6,
+ "model_calls": 3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.4,
+ "model_calls": 2
+ }
+ },
+ "refactor-rename": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.3,
+ "model_calls": 5
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.6,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.8,
+ "model_calls": 4
+ }
+ },
+ "regex-count": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 14.1,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.6,
+ "model_calls": 3.3
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.0,
+ "model_calls": 3
+ }
+ },
+ "write-tests": {
+ "suite": "core",
+ "graff": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.6,
+ "model_calls": 3
+ },
+ "opencode2": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.4,
+ "model_calls": 4
+ },
+ "pi": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.1,
+ "model_calls": 3
+ }
+ }
+ },
+ "runs": [
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.9,
+ "model_calls": 3,
+ "input_tokens": 27883,
+ "cached_input_tokens": 22546,
+ "output_tokens": 311
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.23,
+ "model_calls": 3,
+ "input_tokens": 27912,
+ "cached_input_tokens": 27756,
+ "output_tokens": 355
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.24,
+ "model_calls": 4,
+ "input_tokens": 37348,
+ "cached_input_tokens": 37159,
+ "output_tokens": 471
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.92,
+ "model_calls": 3,
+ "input_tokens": 28218,
+ "cached_input_tokens": 22632,
+ "output_tokens": 160
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.44,
+ "model_calls": 3,
+ "input_tokens": 28218,
+ "cached_input_tokens": 28210,
+ "output_tokens": 160
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.36,
+ "model_calls": 3,
+ "input_tokens": 28228,
+ "cached_input_tokens": 27748,
+ "output_tokens": 165
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.89,
+ "model_calls": 1,
+ "input_tokens": 9121,
+ "cached_input_tokens": 9117,
+ "output_tokens": 51
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.75,
+ "model_calls": 1,
+ "input_tokens": 9121,
+ "cached_input_tokens": 9117,
+ "output_tokens": 51
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.76,
+ "model_calls": 1,
+ "input_tokens": 9121,
+ "cached_input_tokens": 9117,
+ "output_tokens": 51
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.93,
+ "model_calls": 2,
+ "input_tokens": 18395,
+ "cached_input_tokens": 18186,
+ "output_tokens": 143
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.22,
+ "model_calls": 2,
+ "input_tokens": 18395,
+ "cached_input_tokens": 18389,
+ "output_tokens": 141
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.39,
+ "model_calls": 2,
+ "input_tokens": 18395,
+ "cached_input_tokens": 18389,
+ "output_tokens": 158
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.41,
+ "model_calls": 4,
+ "input_tokens": 38412,
+ "cached_input_tokens": 32526,
+ "output_tokens": 394
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.11,
+ "model_calls": 4,
+ "input_tokens": null,
+ "cached_input_tokens": null,
+ "output_tokens": null
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.65,
+ "model_calls": 4,
+ "input_tokens": 38412,
+ "cached_input_tokens": 37647,
+ "output_tokens": 407
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.24,
+ "model_calls": 2,
+ "input_tokens": 18596,
+ "cached_input_tokens": 13279,
+ "output_tokens": 258
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.27,
+ "model_calls": 2,
+ "input_tokens": 18590,
+ "cached_input_tokens": 18398,
+ "output_tokens": 240
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.77,
+ "model_calls": 2,
+ "input_tokens": 18595,
+ "cached_input_tokens": 18398,
+ "output_tokens": 264
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.44,
+ "model_calls": 3,
+ "input_tokens": 27943,
+ "cached_input_tokens": 22578,
+ "output_tokens": 254
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.77,
+ "model_calls": 3,
+ "input_tokens": 27943,
+ "cached_input_tokens": 27935,
+ "output_tokens": 261
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.41,
+ "model_calls": 3,
+ "input_tokens": 27943,
+ "cached_input_tokens": 27935,
+ "output_tokens": 256
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.63,
+ "model_calls": 3,
+ "input_tokens": 30744,
+ "cached_input_tokens": 20170,
+ "output_tokens": 636
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.05,
+ "model_calls": 3,
+ "input_tokens": 30882,
+ "cached_input_tokens": 30536,
+ "output_tokens": 611
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.81,
+ "model_calls": 3,
+ "input_tokens": 30934,
+ "cached_input_tokens": 30536,
+ "output_tokens": 647
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.38,
+ "model_calls": 4,
+ "input_tokens": 41204,
+ "cached_input_tokens": 39485,
+ "output_tokens": 907
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.07,
+ "model_calls": 4,
+ "input_tokens": 41193,
+ "cached_input_tokens": 40898,
+ "output_tokens": 851
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.6,
+ "model_calls": 4,
+ "input_tokens": 41231,
+ "cached_input_tokens": 40166,
+ "output_tokens": 853
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.69,
+ "model_calls": 4,
+ "input_tokens": 42833,
+ "cached_input_tokens": 40824,
+ "output_tokens": 676
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.45,
+ "model_calls": 3,
+ "input_tokens": 32072,
+ "cached_input_tokens": 31503,
+ "output_tokens": 993
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.64,
+ "model_calls": 3,
+ "input_tokens": 32072,
+ "cached_input_tokens": 32064,
+ "output_tokens": 922
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.98,
+ "model_calls": 4,
+ "input_tokens": 41013,
+ "cached_input_tokens": 39412,
+ "output_tokens": 594
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.67,
+ "model_calls": 4,
+ "input_tokens": 41013,
+ "cached_input_tokens": 40808,
+ "output_tokens": 617
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.12,
+ "model_calls": 4,
+ "input_tokens": 41013,
+ "cached_input_tokens": 41003,
+ "output_tokens": 580
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.89,
+ "model_calls": 7,
+ "input_tokens": 78474,
+ "cached_input_tokens": 75338,
+ "output_tokens": 1305
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.06,
+ "model_calls": 4,
+ "input_tokens": 43098,
+ "cached_input_tokens": 42796,
+ "output_tokens": 653
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.08,
+ "model_calls": 4,
+ "input_tokens": 43040,
+ "cached_input_tokens": 42395,
+ "output_tokens": 679
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.39,
+ "model_calls": 4,
+ "input_tokens": 41229,
+ "cached_input_tokens": 39545,
+ "output_tokens": 774
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.94,
+ "model_calls": 4,
+ "input_tokens": 41245,
+ "cached_input_tokens": 40485,
+ "output_tokens": 616
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.18,
+ "model_calls": 4,
+ "input_tokens": 41252,
+ "cached_input_tokens": 40488,
+ "output_tokens": 596
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.74,
+ "model_calls": 4,
+ "input_tokens": 42823,
+ "cached_input_tokens": 40749,
+ "output_tokens": 639
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.15,
+ "model_calls": 4,
+ "input_tokens": 42815,
+ "cached_input_tokens": 41901,
+ "output_tokens": 548
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.54,
+ "model_calls": 4,
+ "input_tokens": 42815,
+ "cached_input_tokens": 42805,
+ "output_tokens": 622
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.51,
+ "model_calls": 4,
+ "input_tokens": 49605,
+ "cached_input_tokens": 45782,
+ "output_tokens": 912
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.25,
+ "model_calls": 3,
+ "input_tokens": 37300,
+ "cached_input_tokens": 36266,
+ "output_tokens": 896
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.99,
+ "model_calls": 4,
+ "input_tokens": 49603,
+ "cached_input_tokens": 48530,
+ "output_tokens": 875
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.16,
+ "model_calls": 8,
+ "input_tokens": 77647,
+ "cached_input_tokens": 59430,
+ "output_tokens": 2062
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.19,
+ "model_calls": 8,
+ "input_tokens": 77930,
+ "cached_input_tokens": 74039,
+ "output_tokens": 2168
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.48,
+ "model_calls": 11,
+ "input_tokens": 104516,
+ "cached_input_tokens": 95787,
+ "output_tokens": 2901
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.33,
+ "model_calls": 4,
+ "input_tokens": 43066,
+ "cached_input_tokens": 40995,
+ "output_tokens": 698
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.63,
+ "model_calls": 4,
+ "input_tokens": 43097,
+ "cached_input_tokens": 42654,
+ "output_tokens": 719
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.39,
+ "model_calls": 3,
+ "input_tokens": 32244,
+ "cached_input_tokens": 31674,
+ "output_tokens": 953
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.2,
+ "model_calls": 2,
+ "input_tokens": 18747,
+ "cached_input_tokens": 13253,
+ "output_tokens": 63
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.56,
+ "model_calls": 2,
+ "input_tokens": 18747,
+ "cached_input_tokens": 18741,
+ "output_tokens": 110
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.58,
+ "model_calls": 2,
+ "input_tokens": 18747,
+ "cached_input_tokens": 18741,
+ "output_tokens": 110
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.61,
+ "model_calls": 5,
+ "input_tokens": 47904,
+ "cached_input_tokens": 42043,
+ "output_tokens": 555
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.28,
+ "model_calls": 5,
+ "input_tokens": 48174,
+ "cached_input_tokens": 47635,
+ "output_tokens": 639
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.99,
+ "model_calls": 5,
+ "input_tokens": 47904,
+ "cached_input_tokens": 47892,
+ "output_tokens": 525
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.91,
+ "model_calls": 3,
+ "input_tokens": 28150,
+ "cached_input_tokens": 22704,
+ "output_tokens": 505
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.03,
+ "model_calls": 3,
+ "input_tokens": 28165,
+ "cached_input_tokens": 27851,
+ "output_tokens": 365
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.32,
+ "model_calls": 3,
+ "input_tokens": 28061,
+ "cached_input_tokens": 27784,
+ "output_tokens": 404
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.49,
+ "model_calls": 3,
+ "input_tokens": 28080,
+ "cached_input_tokens": 22556,
+ "output_tokens": 446
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.22,
+ "model_calls": 3,
+ "input_tokens": 28080,
+ "cached_input_tokens": 28072,
+ "output_tokens": 420
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.13,
+ "model_calls": 3,
+ "input_tokens": 28080,
+ "cached_input_tokens": 28072,
+ "output_tokens": 444
+ },
+ {
+ "arm": "opencode2",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.68,
+ "model_calls": 3,
+ "input_tokens": 23865,
+ "cached_input_tokens": 20630,
+ "output_tokens": 311
+ },
+ {
+ "arm": "opencode2",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.96,
+ "model_calls": 3,
+ "input_tokens": 23865,
+ "cached_input_tokens": 20671,
+ "output_tokens": 319
+ },
+ {
+ "arm": "opencode2",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.9,
+ "model_calls": 3,
+ "input_tokens": 23865,
+ "cached_input_tokens": 20671,
+ "output_tokens": 304
+ },
+ {
+ "arm": "opencode2",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.91,
+ "model_calls": 4,
+ "input_tokens": 32597,
+ "cached_input_tokens": 28619,
+ "output_tokens": 345
+ },
+ {
+ "arm": "opencode2",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.53,
+ "model_calls": 4,
+ "input_tokens": 32305,
+ "cached_input_tokens": 28671,
+ "output_tokens": 289
+ },
+ {
+ "arm": "opencode2",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.97,
+ "model_calls": 4,
+ "input_tokens": 32305,
+ "cached_input_tokens": 28671,
+ "output_tokens": 289
+ },
+ {
+ "arm": "opencode2",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 2.49,
+ "model_calls": 1,
+ "input_tokens": 8280,
+ "cached_input_tokens": 5436,
+ "output_tokens": 13
+ },
+ {
+ "arm": "opencode2",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.75,
+ "model_calls": 1,
+ "input_tokens": 7484,
+ "cached_input_tokens": 4644,
+ "output_tokens": 4
+ },
+ {
+ "arm": "opencode2",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.96,
+ "model_calls": 1,
+ "input_tokens": 7484,
+ "cached_input_tokens": 4644,
+ "output_tokens": 4
+ },
+ {
+ "arm": "opencode2",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.73,
+ "model_calls": 3,
+ "input_tokens": 23691,
+ "cached_input_tokens": 20568,
+ "output_tokens": 255
+ },
+ {
+ "arm": "opencode2",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.77,
+ "model_calls": 3,
+ "input_tokens": 23744,
+ "cached_input_tokens": 20614,
+ "output_tokens": 282
+ },
+ {
+ "arm": "opencode2",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.21,
+ "model_calls": 3,
+ "input_tokens": 23857,
+ "cached_input_tokens": 20614,
+ "output_tokens": 295
+ },
+ {
+ "arm": "opencode2",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.21,
+ "model_calls": 4,
+ "input_tokens": 32294,
+ "cached_input_tokens": 28798,
+ "output_tokens": 345
+ },
+ {
+ "arm": "opencode2",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.74,
+ "model_calls": 4,
+ "input_tokens": 32294,
+ "cached_input_tokens": 28857,
+ "output_tokens": 358
+ },
+ {
+ "arm": "opencode2",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.86,
+ "model_calls": 4,
+ "input_tokens": 32294,
+ "cached_input_tokens": 28857,
+ "output_tokens": 358
+ },
+ {
+ "arm": "opencode2",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.65,
+ "model_calls": 3,
+ "input_tokens": 23234,
+ "cached_input_tokens": 19849,
+ "output_tokens": 319
+ },
+ {
+ "arm": "opencode2",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.05,
+ "model_calls": 3,
+ "input_tokens": 24079,
+ "cached_input_tokens": 20691,
+ "output_tokens": 353
+ },
+ {
+ "arm": "opencode2",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.79,
+ "model_calls": 3,
+ "input_tokens": 24083,
+ "cached_input_tokens": 20687,
+ "output_tokens": 330
+ },
+ {
+ "arm": "opencode2",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.75,
+ "model_calls": 3,
+ "input_tokens": 23966,
+ "cached_input_tokens": 20665,
+ "output_tokens": 378
+ },
+ {
+ "arm": "opencode2",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.12,
+ "model_calls": 3,
+ "input_tokens": 23966,
+ "cached_input_tokens": 20719,
+ "output_tokens": 349
+ },
+ {
+ "arm": "opencode2",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.07,
+ "model_calls": 3,
+ "input_tokens": 23966,
+ "cached_input_tokens": 20719,
+ "output_tokens": 391
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.11,
+ "model_calls": 6,
+ "input_tokens": 64184,
+ "cached_input_tokens": 55752,
+ "output_tokens": 1632
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.91,
+ "model_calls": 6,
+ "input_tokens": 66506,
+ "cached_input_tokens": 57714,
+ "output_tokens": 1849
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.19,
+ "model_calls": 6,
+ "input_tokens": 67866,
+ "cached_input_tokens": 58508,
+ "output_tokens": 1966
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.28,
+ "model_calls": 5,
+ "input_tokens": 55723,
+ "cached_input_tokens": 46086,
+ "output_tokens": 1773
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.57,
+ "model_calls": 6,
+ "input_tokens": 67456,
+ "cached_input_tokens": 58096,
+ "output_tokens": 1759
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.79,
+ "model_calls": 5,
+ "input_tokens": 55608,
+ "cached_input_tokens": 46175,
+ "output_tokens": 1693
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.38,
+ "model_calls": 6,
+ "input_tokens": 67190,
+ "cached_input_tokens": 57961,
+ "output_tokens": 1782
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.84,
+ "model_calls": 5,
+ "input_tokens": 53445,
+ "cached_input_tokens": 45015,
+ "output_tokens": 1547
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.92,
+ "model_calls": 6,
+ "input_tokens": 66914,
+ "cached_input_tokens": 57976,
+ "output_tokens": 1719
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.99,
+ "model_calls": 6,
+ "input_tokens": 58611,
+ "cached_input_tokens": 51656,
+ "output_tokens": 1652
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.81,
+ "model_calls": 5,
+ "input_tokens": 47567,
+ "cached_input_tokens": 41313,
+ "output_tokens": 1653
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.95,
+ "model_calls": 6,
+ "input_tokens": 58287,
+ "cached_input_tokens": 51569,
+ "output_tokens": 1867
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.55,
+ "model_calls": 4,
+ "input_tokens": 36901,
+ "cached_input_tokens": 31489,
+ "output_tokens": 1737
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.03,
+ "model_calls": 5,
+ "input_tokens": 54234,
+ "cached_input_tokens": 44605,
+ "output_tokens": 1786
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.01,
+ "model_calls": 6,
+ "input_tokens": 59377,
+ "cached_input_tokens": 52389,
+ "output_tokens": 1848
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.74,
+ "model_calls": 6,
+ "input_tokens": 60410,
+ "cached_input_tokens": 52832,
+ "output_tokens": 1888
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.63,
+ "model_calls": 6,
+ "input_tokens": 66525,
+ "cached_input_tokens": 57121,
+ "output_tokens": 1891
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.28,
+ "model_calls": 5,
+ "input_tokens": 48244,
+ "cached_input_tokens": 41797,
+ "output_tokens": 2266
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.98,
+ "model_calls": 5,
+ "input_tokens": 47381,
+ "cached_input_tokens": 40951,
+ "output_tokens": 1725
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.83,
+ "model_calls": 6,
+ "input_tokens": 57363,
+ "cached_input_tokens": 50959,
+ "output_tokens": 1764
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.87,
+ "model_calls": 6,
+ "input_tokens": 58147,
+ "cached_input_tokens": 51586,
+ "output_tokens": 1819
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.1,
+ "model_calls": 6,
+ "input_tokens": 65921,
+ "cached_input_tokens": 57064,
+ "output_tokens": 1761
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.24,
+ "model_calls": 6,
+ "input_tokens": 65529,
+ "cached_input_tokens": 57447,
+ "output_tokens": 1833
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.05,
+ "model_calls": 6,
+ "input_tokens": 65650,
+ "cached_input_tokens": 57132,
+ "output_tokens": 1712
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": false,
+ "timed_out": false,
+ "wall_s": 42.55,
+ "model_calls": 3,
+ "input_tokens": 13805,
+ "cached_input_tokens": 6430,
+ "output_tokens": 291
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.9,
+ "model_calls": 3,
+ "input_tokens": 21079,
+ "cached_input_tokens": 16530,
+ "output_tokens": 583
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.6,
+ "model_calls": 2,
+ "input_tokens": 13327,
+ "cached_input_tokens": 9691,
+ "output_tokens": 355
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.22,
+ "model_calls": 6,
+ "input_tokens": 67825,
+ "cached_input_tokens": 58324,
+ "output_tokens": 1803
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.32,
+ "model_calls": 6,
+ "input_tokens": 66147,
+ "cached_input_tokens": 57536,
+ "output_tokens": 1984
+ },
+ {
+ "arm": "opencode2",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.27,
+ "model_calls": 5,
+ "input_tokens": 53869,
+ "cached_input_tokens": 45387,
+ "output_tokens": 1714
+ },
+ {
+ "arm": "opencode2",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.66,
+ "model_calls": 3,
+ "input_tokens": 24029,
+ "cached_input_tokens": 20531,
+ "output_tokens": 167
+ },
+ {
+ "arm": "opencode2",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.56,
+ "model_calls": 3,
+ "input_tokens": 24029,
+ "cached_input_tokens": 20567,
+ "output_tokens": 167
+ },
+ {
+ "arm": "opencode2",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.48,
+ "model_calls": 3,
+ "input_tokens": 23211,
+ "cached_input_tokens": 19753,
+ "output_tokens": 153
+ },
+ {
+ "arm": "opencode2",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.12,
+ "model_calls": 4,
+ "input_tokens": 32966,
+ "cached_input_tokens": 29034,
+ "output_tokens": 730
+ },
+ {
+ "arm": "opencode2",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.01,
+ "model_calls": 4,
+ "input_tokens": 32969,
+ "cached_input_tokens": 29076,
+ "output_tokens": 801
+ },
+ {
+ "arm": "opencode2",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.68,
+ "model_calls": 4,
+ "input_tokens": 32998,
+ "cached_input_tokens": 29120,
+ "output_tokens": 737
+ },
+ {
+ "arm": "opencode2",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.25,
+ "model_calls": 3,
+ "input_tokens": 24358,
+ "cached_input_tokens": 20847,
+ "output_tokens": 496
+ },
+ {
+ "arm": "opencode2",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.45,
+ "model_calls": 3,
+ "input_tokens": 24207,
+ "cached_input_tokens": 20897,
+ "output_tokens": 401
+ },
+ {
+ "arm": "opencode2",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.99,
+ "model_calls": 4,
+ "input_tokens": 32621,
+ "cached_input_tokens": 29005,
+ "output_tokens": 619
+ },
+ {
+ "arm": "opencode2",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.5,
+ "model_calls": 4,
+ "input_tokens": 32008,
+ "cached_input_tokens": 28378,
+ "output_tokens": 655
+ },
+ {
+ "arm": "opencode2",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.73,
+ "model_calls": 4,
+ "input_tokens": 31962,
+ "cached_input_tokens": 28438,
+ "output_tokens": 605
+ },
+ {
+ "arm": "opencode2",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.01,
+ "model_calls": 4,
+ "input_tokens": 32008,
+ "cached_input_tokens": 28438,
+ "output_tokens": 640
+ },
+ {
+ "arm": "pi",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.9,
+ "model_calls": 3,
+ "input_tokens": 7244,
+ "cached_input_tokens": 5896,
+ "output_tokens": 262
+ },
+ {
+ "arm": "pi",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.53,
+ "model_calls": 3,
+ "input_tokens": 7223,
+ "cached_input_tokens": 5896,
+ "output_tokens": 215
+ },
+ {
+ "arm": "pi",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.54,
+ "model_calls": 3,
+ "input_tokens": 7223,
+ "cached_input_tokens": 5896,
+ "output_tokens": 235
+ },
+ {
+ "arm": "pi",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.25,
+ "model_calls": 3,
+ "input_tokens": 7488,
+ "cached_input_tokens": 5970,
+ "output_tokens": 175
+ },
+ {
+ "arm": "pi",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.43,
+ "model_calls": 3,
+ "input_tokens": 8379,
+ "cached_input_tokens": 5999,
+ "output_tokens": 291
+ },
+ {
+ "arm": "pi",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.19,
+ "model_calls": 3,
+ "input_tokens": 7572,
+ "cached_input_tokens": 5991,
+ "output_tokens": 238
+ },
+ {
+ "arm": "pi",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.77,
+ "model_calls": 1,
+ "input_tokens": 2271,
+ "cached_input_tokens": 1190,
+ "output_tokens": 4
+ },
+ {
+ "arm": "pi",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.64,
+ "model_calls": 1,
+ "input_tokens": 2271,
+ "cached_input_tokens": 1190,
+ "output_tokens": 4
+ },
+ {
+ "arm": "pi",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.82,
+ "model_calls": 1,
+ "input_tokens": 2271,
+ "cached_input_tokens": 1190,
+ "output_tokens": 4
+ },
+ {
+ "arm": "pi",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.38,
+ "model_calls": 2,
+ "input_tokens": 4746,
+ "cached_input_tokens": 3489,
+ "output_tokens": 163
+ },
+ {
+ "arm": "pi",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.95,
+ "model_calls": 2,
+ "input_tokens": 4746,
+ "cached_input_tokens": 3489,
+ "output_tokens": 169
+ },
+ {
+ "arm": "pi",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.57,
+ "model_calls": 2,
+ "input_tokens": 4746,
+ "cached_input_tokens": 3489,
+ "output_tokens": 168
+ },
+ {
+ "arm": "pi",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.84,
+ "model_calls": 4,
+ "input_tokens": 10457,
+ "cached_input_tokens": 8722,
+ "output_tokens": 434
+ },
+ {
+ "arm": "pi",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.13,
+ "model_calls": 4,
+ "input_tokens": 10704,
+ "cached_input_tokens": 8892,
+ "output_tokens": 417
+ },
+ {
+ "arm": "pi",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.78,
+ "model_calls": 4,
+ "input_tokens": 10710,
+ "cached_input_tokens": 8892,
+ "output_tokens": 428
+ },
+ {
+ "arm": "pi",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.73,
+ "model_calls": 2,
+ "input_tokens": 5009,
+ "cached_input_tokens": 3507,
+ "output_tokens": 289
+ },
+ {
+ "arm": "pi",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.53,
+ "model_calls": 2,
+ "input_tokens": 5009,
+ "cached_input_tokens": 3507,
+ "output_tokens": 257
+ },
+ {
+ "arm": "pi",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.51,
+ "model_calls": 2,
+ "input_tokens": 5009,
+ "cached_input_tokens": 3507,
+ "output_tokens": 254
+ },
+ {
+ "arm": "pi",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.42,
+ "model_calls": 3,
+ "input_tokens": 7314,
+ "cached_input_tokens": 5927,
+ "output_tokens": 205
+ },
+ {
+ "arm": "pi",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.51,
+ "model_calls": 3,
+ "input_tokens": 7298,
+ "cached_input_tokens": 5927,
+ "output_tokens": 249
+ },
+ {
+ "arm": "pi",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.76,
+ "model_calls": 3,
+ "input_tokens": 7341,
+ "cached_input_tokens": 5927,
+ "output_tokens": 261
+ },
+ {
+ "arm": "pi",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.43,
+ "model_calls": 2,
+ "input_tokens": 4979,
+ "cached_input_tokens": 3480,
+ "output_tokens": 61
+ },
+ {
+ "arm": "pi",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 2.89,
+ "model_calls": 2,
+ "input_tokens": 4979,
+ "cached_input_tokens": 3480,
+ "output_tokens": 61
+ },
+ {
+ "arm": "pi",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 2.91,
+ "model_calls": 2,
+ "input_tokens": 4979,
+ "cached_input_tokens": 3480,
+ "output_tokens": 61
+ },
+ {
+ "arm": "pi",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.98,
+ "model_calls": 4,
+ "input_tokens": 10202,
+ "cached_input_tokens": 8555,
+ "output_tokens": 535
+ },
+ {
+ "arm": "pi",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.92,
+ "model_calls": 4,
+ "input_tokens": 10185,
+ "cached_input_tokens": 8555,
+ "output_tokens": 501
+ },
+ {
+ "arm": "pi",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.41,
+ "model_calls": 4,
+ "input_tokens": 10202,
+ "cached_input_tokens": 8589,
+ "output_tokens": 420
+ },
+ {
+ "arm": "pi",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.3,
+ "model_calls": 3,
+ "input_tokens": 7414,
+ "cached_input_tokens": 6012,
+ "output_tokens": 343
+ },
+ {
+ "arm": "pi",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.5,
+ "model_calls": 3,
+ "input_tokens": 7433,
+ "cached_input_tokens": 6029,
+ "output_tokens": 311
+ },
+ {
+ "arm": "pi",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.24,
+ "model_calls": 3,
+ "input_tokens": 7429,
+ "cached_input_tokens": 6027,
+ "output_tokens": 349
+ },
+ {
+ "arm": "pi",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.24,
+ "model_calls": 3,
+ "input_tokens": 7324,
+ "cached_input_tokens": 5903,
+ "output_tokens": 428
+ },
+ {
+ "arm": "pi",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.32,
+ "model_calls": 3,
+ "input_tokens": 7324,
+ "cached_input_tokens": 5903,
+ "output_tokens": 324
+ },
+ {
+ "arm": "pi",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.68,
+ "model_calls": 3,
+ "input_tokens": 7324,
+ "cached_input_tokens": 5903,
+ "output_tokens": 401
+ }
+ ]
+}
\ No newline at end of file
diff --git a/graff-evals/README.md b/graff-evals/README.md
index 078fda44..e1af4f1c 100644
--- a/graff-evals/README.md
+++ b/graff-evals/README.md
@@ -48,12 +48,18 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co
./run.py --suite live --harness graff-dev --reps 3 -j 1
# same SuperGrok seat, other harnesses (OpenCode / Pi / exo local-process):
./run.py --suite live --harness opencode,pi-xai,exo --reps 3 -j 1
-# ChatGPT login, 1:1 with the Codex app server (needs `codex`; the driver copies the
-# login without its refresh token). Keep --output-root outside any repo so neither
-# harness reads repo docs; the mcp suite then needs scripts/linear_fixture_mcp.py
-# copied next to that root (the setup looks for it in a parent directory).
+# ChatGPT login, 1:1 with the Codex app server and pi (needs `codex`; the driver copies
+# the login without its refresh token; pi-codex needs pi signed in on the ChatGPT plan).
+# Keep --output-root outside any repo so no harness reads repo docs; the mcp suite then
+# needs scripts/linear_fixture_mcp.py copied next to that root (the setup looks for it
+# in a parent directory). pi has no MCP client, so it runs the core suite.
# Results: evals/graff-vs-codex-app-server/RESULTS.md
-./run.py --suite core,mcp --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol -j 6
+./run.py --suite mcp --harness graff-repl,codex-app-server -j 8 --interleave
+./run.py --suite core --harness graff-repl,codex-app-server,pi-codex -j 8 --interleave
+# one gateway model in graff, OpenCode 2 and pi (set CODEGRAFF_API_KEY; OPENCODE2_BIN
+# when `opencode` on PATH is 1.x). Results: evals/graff-vs-opencode-and-pi/RESULTS.md
+./run.py --suite core,mcp --harness graff-gateway-repl,opencode2 --model <gateway model> -j 6 --interleave
+./run.py --suite core --harness pi-codegraff --model <gateway model> -j 6
```
## Run it
diff --git a/graff-evals/codex_app_server.py b/graff-evals/codex_app_server.py
index d1b71236..5da71afd 100755
--- a/graff-evals/codex_app_server.py
+++ b/graff-evals/codex_app_server.py
@@ -33,6 +33,9 @@ import time
from datetime import datetime, timezone
+T0 = time.monotonic()
+
+
def trace(ev: dict) -> None:
"""EVAL_TRACE_DIR: one JSONL per sandbox of per-call usage and item sizes
(what each output token went to), for output and cache analysis."""
@@ -40,6 +43,7 @@ def trace(ev: dict) -> None:
if not root:
return
os.makedirs(root, exist_ok=True)
+ ev = dict(ev, t=round((time.monotonic() - T0) * 1000)) # ms since the driver started
with open(os.path.join(root, os.path.basename(os.getcwd()) + ".jsonl"), "a") as fh:
fh.write(json.dumps(ev) + "\n")
diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.json
index b61aaf53..803e44a7 100644
--- a/graff-evals/harnesses.json
+++ b/graff-evals/harnesses.json
@@ -333,5 +333,39 @@
"answer": "stdout",
"usage": "graff-stderr",
"default_model": "gpt-6.1-sol"
+ },
+ "graff-repl": {
+ "cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"],
+ "stdin": "\u001b[200~{prompt}\u001b[201~\n",
+ "answer": "stdout",
+ "usage": "graff-stderr",
+ "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"},
+ "model_prefix": "codex",
+ "default_model": "gpt-6.1-sol",
+ "note": "scripted graff repl over the Codex WebSocket. The prompt rides one bracketed-paste frame, so a multi-line task is one turn as in the other harnesses (needs the piped paste of ADR 0231)."
+ },
+ "graff-gateway-repl": {
+ "cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"],
+ "stdin": "\u001b[200~{prompt}\u001b[201~\n",
+ "answer": "stdout",
+ "usage": "graff-stderr",
+ "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"},
+ "model_prefix": "codegraff",
+ "note": "scripted graff repl on the Codegraff gateway (CODEGRAFF_API_KEY); pass --model"
+ },
+ "pi-codex": {
+ "cmd": ["pi", "-p", "--mode", "json", "--provider", "openai-codex", "--model", "{model}", "--thinking", "low", "--no-session", "{prompt}"],
+ "answer": "pi-json",
+ "usage": "pi-json",
+ "capabilities": [],
+ "default_model": "gpt-6.1-sol",
+ "note": "pi on the ChatGPT plan (provider openai-codex, signed in through pi; declare gpt-6.1-sol in ~/.pi/agent/models.json until pi lists it). Low effort, the catalog default graff and the Codex app server use. No MCP client: core suite only."
+ },
+ "opencode2": {
+ "cmd": ["python3", "{repo}/graff-evals/opencode2_run.py", "{model}", "{prompt}"],
+ "answer": "stdout",
+ "usage": "graff-stderr",
+ "model_prefix": "codegraff",
+ "note": "OpenCode 2 (@opencode/cli 2.x; OPENCODE2_BIN when `opencode` on PATH is 1.x) with a codegraff provider in its config; pass --model. The wrapper maps .mcp.json into opencode.json and reads usage from OpenCode's session database."
}
}
diff --git a/graff-evals/opencode2_run.py b/graff-evals/opencode2_run.py
new file mode 100755
index 00000000..f27d064c
--- /dev/null
+++ b/graff-evals/opencode2_run.py
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+"""Run OpenCode 2 headless on one eval task and report like graff does.
+
+usage: opencode2_run.py PROVIDER/MODEL PROMPT (cwd = the task sandbox)
+
+The task's `.mcp.json` becomes this directory's `opencode.json` `mcp` block
+(OpenCode does not read `.mcp.json`). The final assistant text goes to stdout;
+a graff-format `[usage]` line goes to stderr, read back from OpenCode's own
+session database (input there excludes cache reads and writes, so they are
+added back to make the total prompt tokens). stdin is empty: OpenCode appends
+piped stdin to the prompt.
+"""
+import json, os, sqlite3, subprocess, sys
+
+# OpenCode 2 is the @opencode/cli package; an `opencode` on PATH may be a 1.x install.
+OC2 = os.environ.get("OPENCODE2_BIN", "opencode")
+model, prompt = sys.argv[1], sys.argv[2]
+cwd = os.getcwd()
+
+if os.path.exists(".mcp.json"):
+ servers = json.load(open(".mcp.json")).get("mcpServers", {})
+ mcp = {name: {"type": "local", "command": [spec["command"], *spec.get("args", [])], "enabled": True}
+ for name, spec in servers.items()}
+ json.dump({"$schema": "https://opencode.ai/config.json", "mcp": mcp}, open("opencode.json", "w"))
+
+r = subprocess.run([OC2, "run", "--standalone", "--auto", "--format", "json", "-m", model, prompt],
+ stdin=subprocess.DEVNULL, capture_output=True, text=True)
+texts = []
+for line in r.stdout.splitlines():
+ try:
+ ev = json.loads(line)
+ except ValueError:
+ continue
+ if ev.get("type") == "text":
+ texts.append((ev.get("part") or {}).get("text", ""))
+print(texts[-1] if texts else "")
+sys.stderr.write(r.stderr)
+
+db_path = os.path.join(os.environ["HOME"], ".local/share/opencode/opencode.db")
+try:
+ db = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
+ dirs = {cwd, os.path.realpath(cwd), "/private" + cwd if cwd.startswith("/tmp/") else cwd}
+ q = ",".join("?" * len(dirs))
+ row = db.execute(f"select id, tokens_input, tokens_output, tokens_reasoning, tokens_cache_read, tokens_cache_write "
+ f"from session_v2 where directory in ({q}) order by time_created desc limit 1", tuple(dirs)).fetchone()
+ if row:
+ sid, tin, tout, treason, tread, twrite = row
+ calls = db.execute("select count(*) from session_message where session_id=? and type='assistant'", (sid,)).fetchone()[0]
+ total_in = (tin or 0) + (tread or 0) + (twrite or 0)
+ sys.stderr.write(f"\n[usage] {calls} api call(s) · {total_in} in ({tread or 0} cached, {twrite or 0} cache writes) + {(tout or 0) + (treason or 0)} out tokens\n")
+except sqlite3.Error as e:
+ sys.stderr.write(f"opencode2_run: usage unavailable: {e}\n")
+sys.exit(r.returncode)
diff --git a/graff-evals/run.py b/graff-evals/run.py
index 3bad96b6..888e111d 100755
--- a/graff-evals/run.py
+++ b/graff-evals/run.py
@@ -32,7 +32,7 @@ RESULTS_DIR = os.path.join(ROOT, "results")
SANDBOX_DIR = os.path.join(ROOT, ".sandboxes")
GRAFF_USAGE_RE = re.compile(
- r"\[usage\] (\d+) api call\(s\) · (\d+) in \((\d+) cached(?:, (\d+) cache writes)?\) \+ (\d+) out tokens")
+ r"\[usage\] (?:known subtotal: )?(\d+) api call\(s\) · (\d+) in \((\d+) cached(?:, (\d+) cache writes)?\) \+ (\d+) out tokens")
def load_tasks():
@@ -371,7 +371,7 @@ def one_run(hname, harness, task, model, rep, live=False):
request_capture.configure(env, sandbox)
try:
p = subprocess.Popen(cmd, cwd=sandbox, stdout=subprocess.PIPE,
- stderr=subprocess.PIPE, stdin=subprocess.PIPE if stdin_body is not None else None,
+ stderr=subprocess.PIPE, stdin=subprocess.PIPE if stdin_body is not None else subprocess.DEVNULL,
text=True, start_new_session=True,
env=env)
if stdin_body is not None and p.stdin is not None: