Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

evals: graff vs the Codex app server, same model and account

justrach45c0e4b0d4fb
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 9b5d63fe..379ba320 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -12,6 +12,10 @@ grok-4.6 SuperGrok) is
[graff-live-12-report.html](graff-live-12-report.html) plus
[artifacts/graff-evals-live/RECEIPT.md](../artifacts/graff-evals-live/RECEIPT.md).
+graff against the Codex app server on the same model and ChatGPT account (21
+coding and MCP tasks, every run's data) is
+[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md).
+
## Prerequisites
- `graff`, `claude`, and `codex` on your PATH, each authenticated.
diff --git a/evals/graff-vs-codex-app-server/RESULTS.md b/evals/graff-vs-codex-app-server/RESULTS.md
new file mode 100644
index 00000000..6d4bdcd2
--- /dev/null
+++ b/evals/graff-vs-codex-app-server/RESULTS.md
@@ -0,0 +1,101 @@
+# graff vs the Codex app server
+
+Measured 2026-10-01. The same model (`gpt-6.1-sol`), ChatGPT account and machine ran the same 21 tasks in each harness, side by side.
+
+| Arm | What ran |
+| --- | --- |
+| graff v0.0.302.11 | The published release asset, checked against the release's `SHA256SUMS`. |
+| graff main | Main at `900e62d9`: v0.0.302.11 plus #1438, #1439 and #1440 (ADR 0226, 0227, 0228). |
+| Codex app server | `codex app-server` from codex-cli 0.159.0. |
+
+graff runs as a scripted `graff repl` over the Codex Responses WebSocket, the transport the app server uses (the `graff-codex-repl` harness). The app server runs through [`graff-evals/codex_app_server.py`](../../graff-evals/codex_app_server.py), which drives its JSON-RPC and reads the token usage it reports. Each harness uses its own defaults. For this model that means `low` effort and `low` verbosity for graff main and for the app server (the model catalog's defaults), and `medium` effort for v0.0.302.11.
+
+## Results
+
+All 21 tasks, mean per task over 3 runs of each:
+
+| | Pass | Wall per task | List cost per task | Input read from cache | Output tokens per task | Model calls per task |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| graff v0.0.302.11 | 63/63 | 31.2s | $0.0189 | 92% | 598 | 4.7 |
+| graff main | 63/63 | 28.2s | $0.0164 | 93% | 490 | 4.7 |
+| Codex app server | 63/63 | 25.0s | $0.0278 | 86% | 382 | 4.3 |
+
+By suite:
+
+| Suite | graff v0.0.302.11 | graff main | Codex app server |
+| --- | ---: | ---: | ---: |
+| core (11 coding tasks), wall | 17.3s | 17.5s | 17.6s |
+| core (11 coding tasks), cost | $0.0078 | $0.0081 | $0.0148 |
+| core (11 coding tasks), pass | 33/33 | 33/33 | 33/33 |
+| mcp (10 MCP tasks), wall | 46.5s | 40.0s | 33.0s |
+| mcp (10 MCP tasks), cost | $0.0311 | $0.0256 | $0.0436 |
+| mcp (10 MCP tasks), pass | 30/30 | 30/30 | 30/30 |
+
+**Against the release.** graff main is 10% faster and 13% cheaper per task than v0.0.302.11. The gain is in the MCP suite (46.5s to 40.0s per task, $0.0311 to $0.0256), most of it on `linear-split` (114.6s to 65.7s), where sub-agents now call the tools their root loaded (ADR 0227, 0228). The core suite is unchanged.
+
+**Against the Codex app server.** graff main ties on the core suite (17.5s against 17.6s) and is slower on the MCP suite (40.0s against 33.0s). Per MCP task it makes 6.1 model calls to the app server's 5.8 and writes 789 output tokens to its 575; `linear-sidecar` shows it most, 6 to 7 calls against 4. Over all 21 tasks the app server takes 11% less time per task (25.0s against 28.2s).
+
+graff main costs 41% less per task on the app server's own reported usage ($0.0164 against $0.0278), and 43% to 69% less once the app server's unreported warm-up request is counted (see Caveats). graff reads more of its input from cache (93% against 86%); part of that gap is the same warm-up, which caches the app server's first real request without appearing in its usage.
+
+Every arm passed all 63 runs.
+
+## Per task
+
+Wall time is the mean of 3 runs.
+
+| Task | Suite | graff v0.0.302.11 | graff main | Codex app server |
+| --- | --- | ---: | ---: | ---: |
+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 16.8s | 15.2s | 22.1s |
+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 8.9s | 9.0s | 18.9s |
+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 4.4s | 4.3s | 6.4s |
+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 22.4s | 21.2s | 11.8s |
+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 20.9s | 20.5s | 21.6s |
+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 24.4s | 32.7s | 17.9s |
+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 20.0s | 20.8s | 20.5s |
+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 33.7s | 36.7s | 27.1s |
+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 42.6s | 43.7s | 31.1s |
+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 43.3s | 39.5s | 31.1s |
+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 35.5s | 33.8s | 29.6s |
+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 34.0s | 33.6s | 29.1s |
+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 38.2s | 36.9s | 34.6s |
+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 39.5s | 34.5s | 36.2s |
+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 49.0s | 46.8s | 27.1s |
+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 114.6s | 65.7s | 58.0s |
+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 35.0s | 28.9s | 26.2s |
+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.2s | 7.4s | 10.2s |
+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 24.3s | 20.7s | 26.9s |
+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 18.8s | 19.1s | 14.1s |
+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 22.0s | 21.6s | 23.6s |
+
+`results.json` has every run: pass, wall time, model calls, input, cached input and output tokens, and list cost.
+
+## Method
+
+- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The fixture declares its two list tools read-only (`readOnlyHint`), as the hosted server it imitates does. Each task passes or fails on its own check.
+- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 8 runs at a time. Each arm made one unscored request first, so no arm started with a prompt cache the others had warmed.
+- **Wall time.** The runner's clock around each process, start to exit.
+- **Cost.** List price of the tokens each harness reports: $2 per 1M input, $0.10 per 1M cached input and $10 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). A ChatGPT plan is not billed per token; this makes the two harnesses comparable, not a bill.
+
+## Caveats
+
+- **Codex's reported usage leaves out one request.** Before a thread's first turn the app server sends a warm-up request (`generate: false`) with its instructions and tools, about 12k tokens. Its usage is not reported, so the Codex cost above is a floor. Counting it adds between about $0.001 per task (fully cached) and $0.025 (uncached).
+- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in either harness.
+- **`file-ops` has a newline in its prompt.** The scripted repl sends each line as a turn, so graff runs that task as two turns and the app server as one.
+- **Small sample.** 3 runs per task, one machine, one account, one 12-minute run. A few seconds per task is within run-to-run noise.
+
+## Reproduce
+
+```sh
+cd graff-evals
+# graff-codex-repl runs {repo}/zig-out/bin/graff; --binary points it elsewhere.
+# The app server needs `codex` on PATH and CODEX_HOME at a signed-in Codex home.
+# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py
+# copied to <parent of output-root>/scripts/ for the mcp suite.
+python3 run.py --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol \
+ --suite core,mcp --reps 3 -j 8 --interleave --output-root /path/outside/repo/run \
+ --task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \
+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \
+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \
+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \
+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm
+```
diff --git a/evals/graff-vs-codex-app-server/results.json b/evals/graff-vs-codex-app-server/results.json
new file mode 100644
index 00000000..6d8c1090
--- /dev/null
+++ b/evals/graff-vs-codex-app-server/results.json
@@ -0,0 +1,3190 @@
+{
+ "model": "gpt-6.1-sol",
+ "tasks": 21,
+ "reps": 3,
+ "arms": {
+ "graff-v302-11": "graff v0.0.302.11 (release)",
+ "graff-next": "graff main (ADR 0226, 0227, 0228)",
+ "codex-app-server": "Codex app server"
+ },
+ "summary": {
+ "all": {
+ "graff-v302-11": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 31.2,
+ "list_usd_per_task": 0.0189,
+ "cached_input_share": 0.922,
+ "output_tokens_per_task": 598,
+ "model_calls_per_task": 4.7
+ },
+ "graff-next": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 28.2,
+ "list_usd_per_task": 0.0164,
+ "cached_input_share": 0.933,
+ "output_tokens_per_task": 490,
+ "model_calls_per_task": 4.7
+ },
+ "codex-app-server": {
+ "runs": 63,
+ "passed": 63,
+ "wall_s_per_task": 25.0,
+ "list_usd_per_task": 0.0278,
+ "cached_input_share": 0.857,
+ "output_tokens_per_task": 382,
+ "model_calls_per_task": 4.3
+ }
+ },
+ "core": {
+ "graff-v302-11": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 17.3,
+ "list_usd_per_task": 0.0078,
+ "cached_input_share": 0.964,
+ "output_tokens_per_task": 218,
+ "model_calls_per_task": 3.2
+ },
+ "graff-next": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 17.5,
+ "list_usd_per_task": 0.0081,
+ "cached_input_share": 0.964,
+ "output_tokens_per_task": 219,
+ "model_calls_per_task": 3.3
+ },
+ "codex-app-server": {
+ "runs": 33,
+ "passed": 33,
+ "wall_s_per_task": 17.6,
+ "list_usd_per_task": 0.0148,
+ "cached_input_share": 0.9,
+ "output_tokens_per_task": 206,
+ "model_calls_per_task": 3.0
+ }
+ },
+ "mcp": {
+ "graff-v302-11": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 46.5,
+ "list_usd_per_task": 0.0311,
+ "cached_input_share": 0.901,
+ "output_tokens_per_task": 1015,
+ "model_calls_per_task": 6.3
+ },
+ "graff-next": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 40.0,
+ "list_usd_per_task": 0.0256,
+ "cached_input_share": 0.917,
+ "output_tokens_per_task": 789,
+ "model_calls_per_task": 6.1
+ },
+ "codex-app-server": {
+ "runs": 30,
+ "passed": 30,
+ "wall_s_per_task": 33.0,
+ "list_usd_per_task": 0.0436,
+ "cached_input_share": 0.84,
+ "output_tokens_per_task": 575,
+ "model_calls_per_task": 5.8
+ }
+ }
+ },
+ "per_task": {
+ "csv-sum": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 16.8,
+ "list_usd": 0.0071
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 15.2,
+ "list_usd": 0.007
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.1,
+ "list_usd": 0.0153
+ }
+ },
+ "dead-code": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 8.9,
+ "list_usd": 0.0036
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 9.0,
+ "list_usd": 0.0035
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.9,
+ "list_usd": 0.0146
+ }
+ },
+ "exact-reply": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.4,
+ "list_usd": 0.0037
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 4.3,
+ "list_usd": 0.0014
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 6.4,
+ "list_usd": 0.0052
+ }
+ },
+ "file-ops": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.4,
+ "list_usd": 0.0098
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.2,
+ "list_usd": 0.0089
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 11.8,
+ "list_usd": 0.0105
+ }
+ },
+ "fix-fib": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.9,
+ "list_usd": 0.0108
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.5,
+ "list_usd": 0.0106
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.6,
+ "list_usd": 0.0183
+ }
+ },
+ "git-ops": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 24.4,
+ "list_usd": 0.0106
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 32.7,
+ "list_usd": 0.0151
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 17.9,
+ "list_usd": 0.0166
+ }
+ },
+ "json-transform": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.0,
+ "list_usd": 0.0095
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.8,
+ "list_usd": 0.0096
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.5,
+ "list_usd": 0.0181
+ }
+ },
+ "linear-issues": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 33.7,
+ "list_usd": 0.0206
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 36.7,
+ "list_usd": 0.0207
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 27.1,
+ "list_usd": 0.0532
+ }
+ },
+ "linear-nohint": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 42.6,
+ "list_usd": 0.0244
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 43.7,
+ "list_usd": 0.0245
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 31.1,
+ "list_usd": 0.0524
+ }
+ },
+ "linear-nohint-warm": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 43.3,
+ "list_usd": 0.0233
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 39.5,
+ "list_usd": 0.0251
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 31.1,
+ "list_usd": 0.0583
+ }
+ },
+ "linear-quiet": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 35.5,
+ "list_usd": 0.0164
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 33.8,
+ "list_usd": 0.0163
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 29.6,
+ "list_usd": 0.0281
+ }
+ },
+ "linear-quiet-warm": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 34.0,
+ "list_usd": 0.0173
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 33.6,
+ "list_usd": 0.0167
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 29.1,
+ "list_usd": 0.0282
+ }
+ },
+ "linear-reduce": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 38.2,
+ "list_usd": 0.0207
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 36.9,
+ "list_usd": 0.0188
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 34.6,
+ "list_usd": 0.0489
+ }
+ },
+ "linear-reduce-warm": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 39.5,
+ "list_usd": 0.0204
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 34.5,
+ "list_usd": 0.0207
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 36.2,
+ "list_usd": 0.033
+ }
+ },
+ "linear-sidecar": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 49.0,
+ "list_usd": 0.0293
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 46.8,
+ "list_usd": 0.0251
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 27.1,
+ "list_usd": 0.0499
+ }
+ },
+ "linear-split": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 114.6,
+ "list_usd": 0.1186
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 65.7,
+ "list_usd": 0.0705
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 58.0,
+ "list_usd": 0
+ }
+ },
+ "linear-warm": {
+ "suite": "mcp",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 35.0,
+ "list_usd": 0.0204
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 28.9,
+ "list_usd": 0.0178
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 26.2,
+ "list_usd": 0.0404
+ }
+ },
+ "recall-noise": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.2,
+ "list_usd": 0.0033
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 7.4,
+ "list_usd": 0.0054
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 10.2,
+ "list_usd": 0.0104
+ }
+ },
+ "refactor-rename": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 24.3,
+ "list_usd": 0.0099
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 20.7,
+ "list_usd": 0.0101
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 26.9,
+ "list_usd": 0.0224
+ }
+ },
+ "regex-count": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 18.8,
+ "list_usd": 0.0076
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 19.1,
+ "list_usd": 0.0074
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 14.1,
+ "list_usd": 0.0136
+ }
+ },
+ "write-tests": {
+ "suite": "core",
+ "graff-v302-11": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 22.0,
+ "list_usd": 0.0101
+ },
+ "graff-next": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 21.6,
+ "list_usd": 0.0097
+ },
+ "codex-app-server": {
+ "passed": 3,
+ "runs": 3,
+ "wall_s": 23.6,
+ "list_usd": 0.0177
+ }
+ }
+ },
+ "runs": [
+ {
+ "arm": "codex-app-server",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.08,
+ "model_calls": 3,
+ "input_tokens": 43326,
+ "cached_input_tokens": 40576,
+ "output_tokens": 308,
+ "list_usd": 0.012638
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.12,
+ "model_calls": 3,
+ "input_tokens": 43418,
+ "cached_input_tokens": 40576,
+ "output_tokens": 404,
+ "list_usd": 0.013782
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.99,
+ "model_calls": 3,
+ "input_tokens": 43332,
+ "cached_input_tokens": 36864,
+ "output_tokens": 286,
+ "list_usd": 0.019482
+ },
+ {
+ "arm": "graff-next",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.06,
+ "model_calls": 3,
+ "input_tokens": 31281,
+ "cached_input_tokens": 30336,
+ "output_tokens": 216,
+ "list_usd": 0.007084
+ },
+ {
+ "arm": "graff-next",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.83,
+ "model_calls": 3,
+ "input_tokens": 31270,
+ "cached_input_tokens": 30464,
+ "output_tokens": 203,
+ "list_usd": 0.006688
+ },
+ {
+ "arm": "graff-next",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.83,
+ "model_calls": 3,
+ "input_tokens": 31286,
+ "cached_input_tokens": 30336,
+ "output_tokens": 219,
+ "list_usd": 0.007124
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.82,
+ "model_calls": 3,
+ "input_tokens": 31286,
+ "cached_input_tokens": 30720,
+ "output_tokens": 223,
+ "list_usd": 0.006434
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.23,
+ "model_calls": 3,
+ "input_tokens": 31286,
+ "cached_input_tokens": 30336,
+ "output_tokens": 223,
+ "list_usd": 0.007164
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.44,
+ "model_calls": 3,
+ "input_tokens": 31301,
+ "cached_input_tokens": 30336,
+ "output_tokens": 272,
+ "list_usd": 0.007684
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.97,
+ "model_calls": 4,
+ "input_tokens": 58039,
+ "cached_input_tokens": 55040,
+ "output_tokens": 156,
+ "list_usd": 0.013062
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.93,
+ "model_calls": 4,
+ "input_tokens": 57949,
+ "cached_input_tokens": 54912,
+ "output_tokens": 150,
+ "list_usd": 0.013065
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.81,
+ "model_calls": 4,
+ "input_tokens": 58134,
+ "cached_input_tokens": 52864,
+ "output_tokens": 171,
+ "list_usd": 0.017536
+ },
+ {
+ "arm": "graff-next",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.81,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20352,
+ "output_tokens": 56,
+ "list_usd": 0.003513
+ },
+ {
+ "arm": "graff-next",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.12,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20352,
+ "output_tokens": 56,
+ "list_usd": 0.003513
+ },
+ {
+ "arm": "graff-next",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.99,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20352,
+ "output_tokens": 56,
+ "list_usd": 0.003513
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.63,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20352,
+ "output_tokens": 70,
+ "list_usd": 0.003653
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.58,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20352,
+ "output_tokens": 56,
+ "list_usd": 0.003513
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.56,
+ "model_calls": 2,
+ "input_tokens": 20811,
+ "cached_input_tokens": 20352,
+ "output_tokens": 56,
+ "list_usd": 0.003513
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.6,
+ "model_calls": 1,
+ "input_tokens": 14234,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005171
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.4,
+ "model_calls": 1,
+ "input_tokens": 14233,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005169
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.12,
+ "model_calls": 1,
+ "input_tokens": 14233,
+ "cached_input_tokens": 12288,
+ "output_tokens": 5,
+ "list_usd": 0.005169
+ },
+ {
+ "arm": "graff-next",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.46,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001379
+ },
+ {
+ "arm": "graff-next",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.19,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001379
+ },
+ {
+ "arm": "graff-next",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.32,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001379
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.29,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 6400,
+ "output_tokens": 5,
+ "list_usd": 0.008432
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 3.54,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001379
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 4.28,
+ "model_calls": 1,
+ "input_tokens": 10271,
+ "cached_input_tokens": 10112,
+ "output_tokens": 5,
+ "list_usd": 0.001379
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.73,
+ "model_calls": 2,
+ "input_tokens": 28647,
+ "cached_input_tokens": 24576,
+ "output_tokens": 99,
+ "list_usd": 0.01159
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.56,
+ "model_calls": 2,
+ "input_tokens": 28653,
+ "cached_input_tokens": 24576,
+ "output_tokens": 104,
+ "list_usd": 0.011652
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 13.23,
+ "model_calls": 2,
+ "input_tokens": 28649,
+ "cached_input_tokens": 26368,
+ "output_tokens": 104,
+ "list_usd": 0.008239
+ },
+ {
+ "arm": "graff-next",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.16,
+ "model_calls": 5,
+ "input_tokens": 51990,
+ "cached_input_tokens": 50688,
+ "output_tokens": 219,
+ "list_usd": 0.009863
+ },
+ {
+ "arm": "graff-next",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.6,
+ "model_calls": 5,
+ "input_tokens": 51990,
+ "cached_input_tokens": 50944,
+ "output_tokens": 217,
+ "list_usd": 0.009356
+ },
+ {
+ "arm": "graff-next",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.95,
+ "model_calls": 4,
+ "input_tokens": 41443,
+ "cached_input_tokens": 40576,
+ "output_tokens": 158,
+ "list_usd": 0.007372
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.16,
+ "model_calls": 4,
+ "input_tokens": 41391,
+ "cached_input_tokens": 40192,
+ "output_tokens": 125,
+ "list_usd": 0.007667
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.82,
+ "model_calls": 5,
+ "input_tokens": 52064,
+ "cached_input_tokens": 50560,
+ "output_tokens": 288,
+ "list_usd": 0.010944
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.23,
+ "model_calls": 5,
+ "input_tokens": 52073,
+ "cached_input_tokens": 50688,
+ "output_tokens": 299,
+ "list_usd": 0.010829
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.37,
+ "model_calls": 3,
+ "input_tokens": 43831,
+ "cached_input_tokens": 38656,
+ "output_tokens": 259,
+ "list_usd": 0.016806
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.93,
+ "model_calls": 4,
+ "input_tokens": 58587,
+ "cached_input_tokens": 53120,
+ "output_tokens": 287,
+ "list_usd": 0.019116
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.64,
+ "model_calls": 4,
+ "input_tokens": 58583,
+ "cached_input_tokens": 53248,
+ "output_tokens": 286,
+ "list_usd": 0.018855
+ },
+ {
+ "arm": "graff-next",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.24,
+ "model_calls": 4,
+ "input_tokens": 42525,
+ "cached_input_tokens": 40448,
+ "output_tokens": 261,
+ "list_usd": 0.010809
+ },
+ {
+ "arm": "graff-next",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.3,
+ "model_calls": 4,
+ "input_tokens": 42277,
+ "cached_input_tokens": 40704,
+ "output_tokens": 247,
+ "list_usd": 0.009686
+ },
+ {
+ "arm": "graff-next",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.05,
+ "model_calls": 4,
+ "input_tokens": 42561,
+ "cached_input_tokens": 40448,
+ "output_tokens": 289,
+ "list_usd": 0.011161
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.05,
+ "model_calls": 4,
+ "input_tokens": 42551,
+ "cached_input_tokens": 40832,
+ "output_tokens": 283,
+ "list_usd": 0.010351
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.92,
+ "model_calls": 4,
+ "input_tokens": 42547,
+ "cached_input_tokens": 40448,
+ "output_tokens": 277,
+ "list_usd": 0.011013
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.69,
+ "model_calls": 4,
+ "input_tokens": 42549,
+ "cached_input_tokens": 40448,
+ "output_tokens": 276,
+ "list_usd": 0.011007
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.11,
+ "model_calls": 3,
+ "input_tokens": 43523,
+ "cached_input_tokens": 38912,
+ "output_tokens": 206,
+ "list_usd": 0.015173
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.36,
+ "model_calls": 3,
+ "input_tokens": 43519,
+ "cached_input_tokens": 38656,
+ "output_tokens": 203,
+ "list_usd": 0.015622
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.25,
+ "model_calls": 3,
+ "input_tokens": 43551,
+ "cached_input_tokens": 36864,
+ "output_tokens": 193,
+ "list_usd": 0.01899
+ },
+ {
+ "arm": "graff-next",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.24,
+ "model_calls": 6,
+ "input_tokens": 64159,
+ "cached_input_tokens": 61056,
+ "output_tokens": 573,
+ "list_usd": 0.018042
+ },
+ {
+ "arm": "graff-next",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.31,
+ "model_calls": 3,
+ "input_tokens": 31714,
+ "cached_input_tokens": 30336,
+ "output_tokens": 344,
+ "list_usd": 0.00923
+ },
+ {
+ "arm": "graff-next",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 35.56,
+ "model_calls": 5,
+ "input_tokens": 53790,
+ "cached_input_tokens": 50560,
+ "output_tokens": 648,
+ "list_usd": 0.017996
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.04,
+ "model_calls": 3,
+ "input_tokens": 31814,
+ "cached_input_tokens": 30336,
+ "output_tokens": 448,
+ "list_usd": 0.01047
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.79,
+ "model_calls": 4,
+ "input_tokens": 42655,
+ "cached_input_tokens": 40832,
+ "output_tokens": 467,
+ "list_usd": 0.012399
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.45,
+ "model_calls": 3,
+ "input_tokens": 31706,
+ "cached_input_tokens": 30336,
+ "output_tokens": 316,
+ "list_usd": 0.008934
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.71,
+ "model_calls": 4,
+ "input_tokens": 57997,
+ "cached_input_tokens": 52864,
+ "output_tokens": 277,
+ "list_usd": 0.018322
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.08,
+ "model_calls": 4,
+ "input_tokens": 58013,
+ "cached_input_tokens": 52992,
+ "output_tokens": 275,
+ "list_usd": 0.018091
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.67,
+ "model_calls": 4,
+ "input_tokens": 57990,
+ "cached_input_tokens": 52992,
+ "output_tokens": 247,
+ "list_usd": 0.017765
+ },
+ {
+ "arm": "graff-next",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.79,
+ "model_calls": 4,
+ "input_tokens": 41937,
+ "cached_input_tokens": 40704,
+ "output_tokens": 271,
+ "list_usd": 0.009246
+ },
+ {
+ "arm": "graff-next",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.3,
+ "model_calls": 4,
+ "input_tokens": 41975,
+ "cached_input_tokens": 40704,
+ "output_tokens": 295,
+ "list_usd": 0.009562
+ },
+ {
+ "arm": "graff-next",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.35,
+ "model_calls": 4,
+ "input_tokens": 41976,
+ "cached_input_tokens": 40448,
+ "output_tokens": 294,
+ "list_usd": 0.010041
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.67,
+ "model_calls": 4,
+ "input_tokens": 41919,
+ "cached_input_tokens": 40576,
+ "output_tokens": 290,
+ "list_usd": 0.009644
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.69,
+ "model_calls": 4,
+ "input_tokens": 41889,
+ "cached_input_tokens": 40576,
+ "output_tokens": 270,
+ "list_usd": 0.009384
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.75,
+ "model_calls": 4,
+ "input_tokens": 41907,
+ "cached_input_tokens": 40576,
+ "output_tokens": 280,
+ "list_usd": 0.00952
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.29,
+ "model_calls": 4,
+ "input_tokens": 82398,
+ "cached_input_tokens": 66688,
+ "output_tokens": 388,
+ "list_usd": 0.041969
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.73,
+ "model_calls": 5,
+ "input_tokens": 131306,
+ "cached_input_tokens": 109312,
+ "output_tokens": 506,
+ "list_usd": 0.059979
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.31,
+ "model_calls": 4,
+ "input_tokens": 78755,
+ "cached_input_tokens": 54656,
+ "output_tokens": 410,
+ "list_usd": 0.057764
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 37.44,
+ "model_calls": 5,
+ "input_tokens": 57133,
+ "cached_input_tokens": 54144,
+ "output_tokens": 599,
+ "list_usd": 0.017382
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 35.36,
+ "model_calls": 5,
+ "input_tokens": 57541,
+ "cached_input_tokens": 52736,
+ "output_tokens": 613,
+ "list_usd": 0.021014
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 37.27,
+ "model_calls": 5,
+ "input_tokens": 57293,
+ "cached_input_tokens": 51200,
+ "output_tokens": 631,
+ "list_usd": 0.023616
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 35.54,
+ "model_calls": 5,
+ "input_tokens": 57185,
+ "cached_input_tokens": 51200,
+ "output_tokens": 575,
+ "list_usd": 0.02284
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 31.41,
+ "model_calls": 4,
+ "input_tokens": 45746,
+ "cached_input_tokens": 40960,
+ "output_tokens": 580,
+ "list_usd": 0.019468
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 34.06,
+ "model_calls": 4,
+ "input_tokens": 45436,
+ "cached_input_tokens": 40960,
+ "output_tokens": 632,
+ "list_usd": 0.019368
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.64,
+ "model_calls": 4,
+ "input_tokens": 102531,
+ "cached_input_tokens": 80128,
+ "output_tokens": 443,
+ "list_usd": 0.057249
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.92,
+ "model_calls": 4,
+ "input_tokens": 105527,
+ "cached_input_tokens": 83840,
+ "output_tokens": 442,
+ "list_usd": 0.056178
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.74,
+ "model_calls": 5,
+ "input_tokens": 115163,
+ "cached_input_tokens": 100864,
+ "output_tokens": 504,
+ "list_usd": 0.043724
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 50.7,
+ "model_calls": 7,
+ "input_tokens": 80135,
+ "cached_input_tokens": 75136,
+ "output_tokens": 873,
+ "list_usd": 0.026242
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.2,
+ "model_calls": 6,
+ "input_tokens": 67932,
+ "cached_input_tokens": 64384,
+ "output_tokens": 752,
+ "list_usd": 0.021054
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 41.31,
+ "model_calls": 6,
+ "input_tokens": 67909,
+ "cached_input_tokens": 61568,
+ "output_tokens": 739,
+ "list_usd": 0.026229
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 40.35,
+ "model_calls": 6,
+ "input_tokens": 67928,
+ "cached_input_tokens": 62080,
+ "output_tokens": 753,
+ "list_usd": 0.025434
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 41.26,
+ "model_calls": 6,
+ "input_tokens": 67928,
+ "cached_input_tokens": 63616,
+ "output_tokens": 754,
+ "list_usd": 0.022526
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 46.34,
+ "model_calls": 6,
+ "input_tokens": 68431,
+ "cached_input_tokens": 63872,
+ "output_tokens": 982,
+ "list_usd": 0.025325
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.35,
+ "model_calls": 4,
+ "input_tokens": 102526,
+ "cached_input_tokens": 80128,
+ "output_tokens": 428,
+ "list_usd": 0.057089
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 36.49,
+ "model_calls": 4,
+ "input_tokens": 104964,
+ "cached_input_tokens": 81664,
+ "output_tokens": 449,
+ "list_usd": 0.059256
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.41,
+ "model_calls": 4,
+ "input_tokens": 103865,
+ "cached_input_tokens": 80896,
+ "output_tokens": 456,
+ "list_usd": 0.058588
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.95,
+ "model_calls": 6,
+ "input_tokens": 69113,
+ "cached_input_tokens": 63616,
+ "output_tokens": 721,
+ "list_usd": 0.024566
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 40.71,
+ "model_calls": 6,
+ "input_tokens": 69164,
+ "cached_input_tokens": 63616,
+ "output_tokens": 795,
+ "list_usd": 0.025408
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.69,
+ "model_calls": 6,
+ "input_tokens": 69249,
+ "cached_input_tokens": 63616,
+ "output_tokens": 767,
+ "list_usd": 0.025298
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 43.51,
+ "model_calls": 6,
+ "input_tokens": 69306,
+ "cached_input_tokens": 63360,
+ "output_tokens": 796,
+ "list_usd": 0.026188
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 46.2,
+ "model_calls": 6,
+ "input_tokens": 69673,
+ "cached_input_tokens": 65664,
+ "output_tokens": 923,
+ "list_usd": 0.023814
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 40.09,
+ "model_calls": 6,
+ "input_tokens": 69145,
+ "cached_input_tokens": 66304,
+ "output_tokens": 760,
+ "list_usd": 0.019912
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.32,
+ "model_calls": 4,
+ "input_tokens": 63308,
+ "cached_input_tokens": 52864,
+ "output_tokens": 479,
+ "list_usd": 0.030964
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.73,
+ "model_calls": 5,
+ "input_tokens": 79606,
+ "cached_input_tokens": 68864,
+ "output_tokens": 514,
+ "list_usd": 0.03351
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.88,
+ "model_calls": 4,
+ "input_tokens": 63293,
+ "cached_input_tokens": 58496,
+ "output_tokens": 435,
+ "list_usd": 0.019794
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.02,
+ "model_calls": 3,
+ "input_tokens": 33539,
+ "cached_input_tokens": 31232,
+ "output_tokens": 508,
+ "list_usd": 0.012817
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 44.37,
+ "model_calls": 4,
+ "input_tokens": 45830,
+ "cached_input_tokens": 41728,
+ "output_tokens": 893,
+ "list_usd": 0.021307
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.12,
+ "model_calls": 3,
+ "input_tokens": 33616,
+ "cached_input_tokens": 30720,
+ "output_tokens": 606,
+ "list_usd": 0.014924
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 37.88,
+ "model_calls": 3,
+ "input_tokens": 33819,
+ "cached_input_tokens": 30720,
+ "output_tokens": 764,
+ "list_usd": 0.01691
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.42,
+ "model_calls": 3,
+ "input_tokens": 33831,
+ "cached_input_tokens": 30720,
+ "output_tokens": 841,
+ "list_usd": 0.017704
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.18,
+ "model_calls": 3,
+ "input_tokens": 33635,
+ "cached_input_tokens": 30720,
+ "output_tokens": 564,
+ "list_usd": 0.014542
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.01,
+ "model_calls": 5,
+ "input_tokens": 79608,
+ "cached_input_tokens": 70656,
+ "output_tokens": 460,
+ "list_usd": 0.02957
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.91,
+ "model_calls": 5,
+ "input_tokens": 84598,
+ "cached_input_tokens": 73344,
+ "output_tokens": 540,
+ "list_usd": 0.035242
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.39,
+ "model_calls": 4,
+ "input_tokens": 63291,
+ "cached_input_tokens": 58496,
+ "output_tokens": 443,
+ "list_usd": 0.01987
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 35.13,
+ "model_calls": 4,
+ "input_tokens": 46171,
+ "cached_input_tokens": 42496,
+ "output_tokens": 656,
+ "list_usd": 0.01816
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.18,
+ "model_calls": 3,
+ "input_tokens": 34078,
+ "cached_input_tokens": 30720,
+ "output_tokens": 585,
+ "list_usd": 0.015638
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 37.51,
+ "model_calls": 3,
+ "input_tokens": 34235,
+ "cached_input_tokens": 31488,
+ "output_tokens": 764,
+ "list_usd": 0.016283
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 41.66,
+ "model_calls": 4,
+ "input_tokens": 46759,
+ "cached_input_tokens": 42624,
+ "output_tokens": 742,
+ "list_usd": 0.019952
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.96,
+ "model_calls": 4,
+ "input_tokens": 46403,
+ "cached_input_tokens": 43264,
+ "output_tokens": 561,
+ "list_usd": 0.016214
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.51,
+ "model_calls": 3,
+ "input_tokens": 34085,
+ "cached_input_tokens": 30720,
+ "output_tokens": 581,
+ "list_usd": 0.015612
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.76,
+ "model_calls": 5,
+ "input_tokens": 107419,
+ "cached_input_tokens": 89344,
+ "output_tokens": 608,
+ "list_usd": 0.051164
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 33.79,
+ "model_calls": 5,
+ "input_tokens": 107311,
+ "cached_input_tokens": 92928,
+ "output_tokens": 626,
+ "list_usd": 0.044319
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 37.32,
+ "model_calls": 6,
+ "input_tokens": 133180,
+ "cached_input_tokens": 116480,
+ "output_tokens": 626,
+ "list_usd": 0.051308
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 37.1,
+ "model_calls": 5,
+ "input_tokens": 56886,
+ "cached_input_tokens": 52224,
+ "output_tokens": 653,
+ "list_usd": 0.021076
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 36.92,
+ "model_calls": 4,
+ "input_tokens": 45136,
+ "cached_input_tokens": 41984,
+ "output_tokens": 614,
+ "list_usd": 0.016642
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 36.74,
+ "model_calls": 5,
+ "input_tokens": 56686,
+ "cached_input_tokens": 53248,
+ "output_tokens": 645,
+ "list_usd": 0.018651
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.49,
+ "model_calls": 4,
+ "input_tokens": 45075,
+ "cached_input_tokens": 41984,
+ "output_tokens": 539,
+ "list_usd": 0.01577
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 33.75,
+ "model_calls": 4,
+ "input_tokens": 45113,
+ "cached_input_tokens": 40960,
+ "output_tokens": 581,
+ "list_usd": 0.018212
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 51.42,
+ "model_calls": 6,
+ "input_tokens": 69541,
+ "cached_input_tokens": 63488,
+ "output_tokens": 966,
+ "list_usd": 0.028115
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 35.38,
+ "model_calls": 6,
+ "input_tokens": 96202,
+ "cached_input_tokens": 86784,
+ "output_tokens": 590,
+ "list_usd": 0.033414
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 34.92,
+ "model_calls": 6,
+ "input_tokens": 96157,
+ "cached_input_tokens": 87040,
+ "output_tokens": 549,
+ "list_usd": 0.032428
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.31,
+ "model_calls": 6,
+ "input_tokens": 96155,
+ "cached_input_tokens": 86784,
+ "output_tokens": 568,
+ "list_usd": 0.0331
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.54,
+ "model_calls": 5,
+ "input_tokens": 57658,
+ "cached_input_tokens": 52480,
+ "output_tokens": 559,
+ "list_usd": 0.021194
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.58,
+ "model_calls": 5,
+ "input_tokens": 57972,
+ "cached_input_tokens": 52480,
+ "output_tokens": 707,
+ "list_usd": 0.023302
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.26,
+ "model_calls": 4,
+ "input_tokens": 45909,
+ "cached_input_tokens": 42240,
+ "output_tokens": 594,
+ "list_usd": 0.017502
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 32.23,
+ "model_calls": 4,
+ "input_tokens": 45833,
+ "cached_input_tokens": 42240,
+ "output_tokens": 561,
+ "list_usd": 0.01702
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 40.59,
+ "model_calls": 5,
+ "input_tokens": 57909,
+ "cached_input_tokens": 52608,
+ "output_tokens": 656,
+ "list_usd": 0.022423
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 45.54,
+ "model_calls": 6,
+ "input_tokens": 69948,
+ "cached_input_tokens": 66176,
+ "output_tokens": 747,
+ "list_usd": 0.021632
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.2,
+ "model_calls": 4,
+ "input_tokens": 78016,
+ "cached_input_tokens": 65664,
+ "output_tokens": 420,
+ "list_usd": 0.03547
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.82,
+ "model_calls": 4,
+ "input_tokens": 105531,
+ "cached_input_tokens": 82048,
+ "output_tokens": 417,
+ "list_usd": 0.059341
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.4,
+ "model_calls": 4,
+ "input_tokens": 104169,
+ "cached_input_tokens": 83072,
+ "output_tokens": 432,
+ "list_usd": 0.054821
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 49.54,
+ "model_calls": 7,
+ "input_tokens": 80485,
+ "cached_input_tokens": 76160,
+ "output_tokens": 967,
+ "list_usd": 0.025936
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 52.58,
+ "model_calls": 7,
+ "input_tokens": 80592,
+ "cached_input_tokens": 75776,
+ "output_tokens": 979,
+ "list_usd": 0.027
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 38.4,
+ "model_calls": 6,
+ "input_tokens": 68164,
+ "cached_input_tokens": 64000,
+ "output_tokens": 763,
+ "list_usd": 0.022358
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 47.53,
+ "model_calls": 6,
+ "input_tokens": 68696,
+ "cached_input_tokens": 63360,
+ "output_tokens": 986,
+ "list_usd": 0.026868
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 49.91,
+ "model_calls": 7,
+ "input_tokens": 80552,
+ "cached_input_tokens": 73472,
+ "output_tokens": 1000,
+ "list_usd": 0.031507
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 49.5,
+ "model_calls": 7,
+ "input_tokens": 80632,
+ "cached_input_tokens": 74368,
+ "output_tokens": 960,
+ "list_usd": 0.029565
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 60.23,
+ "model_calls": 16,
+ "input_tokens": 366660,
+ "cached_input_tokens": 309632,
+ "output_tokens": 1384,
+ "list_usd": null
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 57.52,
+ "model_calls": 16,
+ "input_tokens": 367431,
+ "cached_input_tokens": 312704,
+ "output_tokens": 1416,
+ "list_usd": null
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 56.25,
+ "model_calls": 18,
+ "input_tokens": 439318,
+ "cached_input_tokens": 377344,
+ "output_tokens": 1495,
+ "list_usd": null
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 67.91,
+ "model_calls": 18,
+ "input_tokens": 190207,
+ "cached_input_tokens": 167680,
+ "output_tokens": 1688,
+ "list_usd": 0.078702
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 58.66,
+ "model_calls": 17,
+ "input_tokens": 176523,
+ "cached_input_tokens": 160384,
+ "output_tokens": 1628,
+ "list_usd": 0.064596
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 70.64,
+ "model_calls": 18,
+ "input_tokens": 190968,
+ "cached_input_tokens": 174720,
+ "output_tokens": 1823,
+ "list_usd": 0.068198
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 140.58,
+ "model_calls": 21,
+ "input_tokens": 256931,
+ "cached_input_tokens": 213760,
+ "output_tokens": 4107,
+ "list_usd": 0.148788
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 107.78,
+ "model_calls": 21,
+ "input_tokens": 226902,
+ "cached_input_tokens": 199808,
+ "output_tokens": 3823,
+ "list_usd": 0.112399
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 95.44,
+ "model_calls": 17,
+ "input_tokens": 202662,
+ "cached_input_tokens": 177664,
+ "output_tokens": 2686,
+ "list_usd": 0.094622
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.83,
+ "model_calls": 4,
+ "input_tokens": 82420,
+ "cached_input_tokens": 66688,
+ "output_tokens": 407,
+ "list_usd": 0.042203
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 27.12,
+ "model_calls": 4,
+ "input_tokens": 78746,
+ "cached_input_tokens": 64256,
+ "output_tokens": 409,
+ "list_usd": 0.039496
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.51,
+ "model_calls": 4,
+ "input_tokens": 78756,
+ "cached_input_tokens": 64256,
+ "output_tokens": 416,
+ "list_usd": 0.039586
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.57,
+ "model_calls": 3,
+ "input_tokens": 33957,
+ "cached_input_tokens": 30720,
+ "output_tokens": 513,
+ "list_usd": 0.014676
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.41,
+ "model_calls": 4,
+ "input_tokens": 46236,
+ "cached_input_tokens": 40960,
+ "output_tokens": 537,
+ "list_usd": 0.020018
+ },
+ {
+ "arm": "graff-next",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.59,
+ "model_calls": 4,
+ "input_tokens": 46376,
+ "cached_input_tokens": 41728,
+ "output_tokens": 511,
+ "list_usd": 0.018579
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 29.73,
+ "model_calls": 4,
+ "input_tokens": 46192,
+ "cached_input_tokens": 40960,
+ "output_tokens": 523,
+ "list_usd": 0.01979
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 35.51,
+ "model_calls": 5,
+ "input_tokens": 58221,
+ "cached_input_tokens": 54016,
+ "output_tokens": 599,
+ "list_usd": 0.019802
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 39.91,
+ "model_calls": 4,
+ "input_tokens": 46634,
+ "cached_input_tokens": 42496,
+ "output_tokens": 914,
+ "list_usd": 0.021666
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.81,
+ "model_calls": 2,
+ "input_tokens": 28817,
+ "cached_input_tokens": 24576,
+ "output_tokens": 57,
+ "list_usd": 0.01151
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 10.09,
+ "model_calls": 2,
+ "input_tokens": 28814,
+ "cached_input_tokens": 24576,
+ "output_tokens": 57,
+ "list_usd": 0.011504
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 9.69,
+ "model_calls": 2,
+ "input_tokens": 28808,
+ "cached_input_tokens": 26368,
+ "output_tokens": 52,
+ "list_usd": 0.008037
+ },
+ {
+ "arm": "graff-next",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.4,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-next",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.69,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 16768,
+ "output_tokens": 29,
+ "list_usd": 0.010149
+ },
+ {
+ "arm": "graff-next",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 8.1,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.56,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.68,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20480,
+ "output_tokens": 29,
+ "list_usd": 0.003096
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 7.38,
+ "model_calls": 2,
+ "input_tokens": 20859,
+ "cached_input_tokens": 20224,
+ "output_tokens": 43,
+ "list_usd": 0.003722
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.72,
+ "model_calls": 4,
+ "input_tokens": 58425,
+ "cached_input_tokens": 50944,
+ "output_tokens": 365,
+ "list_usd": 0.023706
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 26.76,
+ "model_calls": 4,
+ "input_tokens": 58340,
+ "cached_input_tokens": 50944,
+ "output_tokens": 374,
+ "list_usd": 0.023626
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.34,
+ "model_calls": 4,
+ "input_tokens": 58471,
+ "cached_input_tokens": 52992,
+ "output_tokens": 375,
+ "list_usd": 0.020007
+ },
+ {
+ "arm": "graff-next",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.81,
+ "model_calls": 4,
+ "input_tokens": 42164,
+ "cached_input_tokens": 40576,
+ "output_tokens": 304,
+ "list_usd": 0.010274
+ },
+ {
+ "arm": "graff-next",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.8,
+ "model_calls": 4,
+ "input_tokens": 42152,
+ "cached_input_tokens": 40832,
+ "output_tokens": 310,
+ "list_usd": 0.009823
+ },
+ {
+ "arm": "graff-next",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.5,
+ "model_calls": 4,
+ "input_tokens": 42164,
+ "cached_input_tokens": 40704,
+ "output_tokens": 316,
+ "list_usd": 0.01015
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.19,
+ "model_calls": 4,
+ "input_tokens": 42142,
+ "cached_input_tokens": 40704,
+ "output_tokens": 305,
+ "list_usd": 0.009996
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.0,
+ "model_calls": 4,
+ "input_tokens": 42190,
+ "cached_input_tokens": 40832,
+ "output_tokens": 332,
+ "list_usd": 0.010119
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.71,
+ "model_calls": 4,
+ "input_tokens": 42186,
+ "cached_input_tokens": 41088,
+ "output_tokens": 327,
+ "list_usd": 0.009575
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.33,
+ "model_calls": 2,
+ "input_tokens": 28686,
+ "cached_input_tokens": 26368,
+ "output_tokens": 111,
+ "list_usd": 0.008383
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.72,
+ "model_calls": 3,
+ "input_tokens": 43218,
+ "cached_input_tokens": 36864,
+ "output_tokens": 172,
+ "list_usd": 0.018114
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 14.27,
+ "model_calls": 3,
+ "input_tokens": 43241,
+ "cached_input_tokens": 38656,
+ "output_tokens": 128,
+ "list_usd": 0.014316
+ },
+ {
+ "arm": "graff-next",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.4,
+ "model_calls": 4,
+ "input_tokens": 41918,
+ "cached_input_tokens": 40704,
+ "output_tokens": 230,
+ "list_usd": 0.008798
+ },
+ {
+ "arm": "graff-next",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.32,
+ "model_calls": 2,
+ "input_tokens": 20863,
+ "cached_input_tokens": 20224,
+ "output_tokens": 224,
+ "list_usd": 0.00554
+ },
+ {
+ "arm": "graff-next",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.68,
+ "model_calls": 3,
+ "input_tokens": 31380,
+ "cached_input_tokens": 30336,
+ "output_tokens": 280,
+ "list_usd": 0.007922
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.81,
+ "model_calls": 3,
+ "input_tokens": 31390,
+ "cached_input_tokens": 30208,
+ "output_tokens": 206,
+ "list_usd": 0.007445
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 17.79,
+ "model_calls": 3,
+ "input_tokens": 31340,
+ "cached_input_tokens": 30464,
+ "output_tokens": 240,
+ "list_usd": 0.007198
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.94,
+ "model_calls": 3,
+ "input_tokens": 31387,
+ "cached_input_tokens": 30208,
+ "output_tokens": 287,
+ "list_usd": 0.008249
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.95,
+ "model_calls": 4,
+ "input_tokens": 57983,
+ "cached_input_tokens": 54912,
+ "output_tokens": 348,
+ "list_usd": 0.015113
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.81,
+ "model_calls": 4,
+ "input_tokens": 58008,
+ "cached_input_tokens": 52992,
+ "output_tokens": 356,
+ "list_usd": 0.018891
+ },
+ {
+ "arm": "codex-app-server",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.03,
+ "model_calls": 4,
+ "input_tokens": 58015,
+ "cached_input_tokens": 52864,
+ "output_tokens": 361,
+ "list_usd": 0.019198
+ },
+ {
+ "arm": "graff-next",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.07,
+ "model_calls": 4,
+ "input_tokens": 41938,
+ "cached_input_tokens": 40320,
+ "output_tokens": 280,
+ "list_usd": 0.010068
+ },
+ {
+ "arm": "graff-next",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.39,
+ "model_calls": 4,
+ "input_tokens": 41909,
+ "cached_input_tokens": 40576,
+ "output_tokens": 269,
+ "list_usd": 0.009414
+ },
+ {
+ "arm": "graff-next",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.42,
+ "model_calls": 4,
+ "input_tokens": 41933,
+ "cached_input_tokens": 40576,
+ "output_tokens": 280,
+ "list_usd": 0.009572
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 23.26,
+ "model_calls": 4,
+ "input_tokens": 41950,
+ "cached_input_tokens": 40192,
+ "output_tokens": 298,
+ "list_usd": 0.010515
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.2,
+ "model_calls": 4,
+ "input_tokens": 41960,
+ "cached_input_tokens": 40320,
+ "output_tokens": 297,
+ "list_usd": 0.010282
+ },
+ {
+ "arm": "graff-v302-11",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 3,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 20.44,
+ "model_calls": 4,
+ "input_tokens": 41912,
+ "cached_input_tokens": 40576,
+ "output_tokens": 270,
+ "list_usd": 0.00943
+ }
+ ]
+}
diff --git a/graff-evals/README.md b/graff-evals/README.md
index b9a66cb4..8eabb2a3 100644
--- a/graff-evals/README.md
+++ b/graff-evals/README.md
@@ -52,6 +52,7 @@ CODEGRAFF_API_KEY=cg_sk_… ./run.py --suite swe --harness graff-dev,opencode-co
# login without its refresh token). Keep --output-root outside any repo so neither
# harness reads repo docs; the mcp suite then needs scripts/linear_fixture_mcp.py
# copied next to that root (the setup looks for it in a parent directory).
+# Results: evals/graff-vs-codex-app-server/RESULTS.md
./run.py --suite core,mcp --harness graff-codex-repl,codex-app-server --model gpt-6.1-sol -j 6
```