graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff vs the Grok CLI; what the comparisons mean for a budget
justracha8b32b310a53
diff --git a/README.md b/README.mdindex 9f037130..4a324a26 100644--- a/README.md+++ b/README.md@@ -408,6 +408,15 @@ Tier 1 is `zig fmt`, the 600-line ceiling, test reachability, `zig build test`Docs-only pushes skip it. In-house PR fixtures: `graff-evals/`(`--suite inhouse`).+## Enterprise++Companies that have raised more than US$500k, or that have meaningful revenue+or valuation, should contact+[[email protected]](mailto:[email protected]) about enterprise+offerings: commercial licensing, support, and deployment help. What the+measured harness comparisons mean for a budget:+[evals/enterprise-cost.md](evals/enterprise-cost.md).+## License**Modified GNU AGPL-3.0** ([`LICENSE`](LICENSE)). Network use triggersdiff --git a/benchmarks/README.md b/benchmarks/README.mdindex 379ba320..e8ada1dd 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -15,12 +15,18 @@ grok-4.6 SuperGrok) isgraff against the Codex app server on the same model and ChatGPT account (21coding and MCP tasks, every run's data) is[evals/graff-vs-codex-app-server/RESULTS.md](../evals/graff-vs-codex-app-server/RESULTS.md).+The same 21 tasks against the Grok CLI on the same SuperGrok account are+[evals/graff-vs-grok-cli/RESULTS.md](../evals/graff-vs-grok-cli/RESULTS.md),+and what both mean for an engineering budget is+[evals/enterprise-cost.md](../evals/enterprise-cost.md).## Prerequisites- `graff`, `claude`, and `codex` on your PATH, each authenticated.-- For `latency.py`, graff must be logged into Codex (`graff login codex`) so that- `graff --model codex` hits the same ChatGPT endpoint as `codex exec` (no gateway).+- For `latency.py`, graff needs the Codex CLI's sign-in (`codex login`, read from+ `~/.codex/auth.json`) so that `graff --model codex` hits the same ChatGPT+ endpoint as `codex exec` (no gateway). `graff login codex` now runs the+ ChatGPT sign-in for open-source apps (ADR 0229).- macOS for `memory.py` (it uses BSD `/usr/bin/time -l`).- Python 3.diff --git a/evals/enterprise-cost.md b/evals/enterprise-cost.mdnew file mode 100644index 00000000..87d485f3--- /dev/null+++ b/evals/enterprise-cost.md@@ -0,0 +1,62 @@+# What the harness comparisons mean for an engineering budget++A rough guide built from two measured comparisons, each on the same model+and account in both harnesses:++- [graff vs the Codex app server](graff-vs-codex-app-server/RESULTS.md) (`gpt-6.1-sol`)+- [graff vs the Grok CLI](graff-vs-grok-cli/RESULTS.md) (`grok-4.7`)++Both ran the same 21 coding and MCP tasks, 3 times each. Costs are list+prices of the tokens each harness used.++## Per task++| Compared with | graff | Other harness | graff costs |+| --- | ---: | ---: | ---: |+| Codex app server, its reported usage | $0.0164 | $0.0278 | 41% less |+| Codex app server, with its unreported warm-up request | $0.0164 | up to $0.0524 | up to 69% less |+| Grok CLI, grok-4.7 public list price | $0.039 | $0.26 to $0.33 | 85% to 88% less |+| Grok CLI, xAI's price for its build-model route | $0.039 | $0.098 | 60% less |++## At company scale++An illustrative organization: 100 engineers, each running 40 agent tasks per+working day, 21 working days a month, so 84,000 tasks a month at the per-task+costs above.++| Compared with | graff per month | Other per month | Saved per month | Saved per year |+| --- | ---: | ---: | ---: | ---: |+| Codex app server, reported usage | $1,378 | $2,335 | $958 | $11,491 |+| Codex app server, with warm-up | $1,378 | $4,402 | $3,024 | $36,288 |+| Grok CLI, public list price | $3,284 | $22,042 to $27,770 | $18,757 to $24,486 | $225,086 to $293,832 |+| Grok CLI, xAI's build-model price | $3,284 | $8,240 | $4,956 | $59,472 |++Time moves too. Against the Grok CLI, graff took 15.5s per task against+106.9s: about 2,100 fewer hours of agent wall time a month at this volume.+Against the Codex app server graff was slower, 28.2s against 25.0s per task+(about 75 more hours a month).++## How far this carries++- **Real tasks are bigger.** These tasks average about 30k prompt tokens in+ graff. Cost grows with tokens, so the percentage is the part that+ transfers: an organization spending $X a month on agent runs in one of+ these harnesses would spend roughly the "graff costs" share of $X for the+ same work.+- **MCP-heavy work saves the most.** Most of the gap against the Grok CLI is+ on MCP tasks, where it re-sends every tool result on every turn.+- **Plans are not per-token bills.** Both comparisons ran on subscription+ sign-ins; the dollars are list-price equivalents, which is what API-key+ usage pays.+- **Small samples.** 3 runs per task on one machine and one account per+ comparison. Run [`graff-evals`](../graff-evals) on your own tasks before+ budgeting from these numbers.++## Enterprise++graff is open source under a modified AGPL-3.0 ([LICENSE](../LICENSE)).+Companies that have raised more than US$500k, or that have meaningful+revenue or valuation, should contact+[[email protected]](mailto:[email protected]) about enterprise+offerings: commercial licensing under the Licensors' reserved rights,+support, and deployment help.diff --git a/evals/graff-vs-grok-cli/RESULTS.md b/evals/graff-vs-grok-cli/RESULTS.mdnew file mode 100644index 00000000..ddd9d8a4--- /dev/null+++ b/evals/graff-vs-grok-cli/RESULTS.md@@ -0,0 +1,130 @@+# graff vs the Grok CLI++Measured 2026-10-01. One machine and one SuperGrok account ran the same 21+tasks in each harness, side by side.++| Arm | What ran |+| --- | --- |+| graff v0.0.302.12 | The published release asset, checked against the release's `SHA256SUMS`, with `--model xai/grok-4.7` on the `graff login xai` sign-in. |+| Grok CLI | `grok` 1.0.41 (Grok Build) with `-m grok-4.7`, headless: `-p … --always-approve --output-format streaming-messages-json`. |++Both sign in through the same xAI OAuth client to the same account, and both+run at their own defaults. The model is not quite the same: asked for+`grok-4.7`, the Grok CLI's route serves `grok-4.7-build`, a variant only that+route offers. graff uses xAI's public API, which serves `grok-4.7` and has no+build variant.++## Results++All 21 tasks, mean per task over 3 runs of each:++| | Pass | Wall per task | List cost per task | Prompt tokens per task | Input read from cache | Output tokens per task | Model calls per task |+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |+| graff v0.0.302.12 | 63/63 | 15.5s | $0.039 | 29,696 | 57% | 823 | 4.3 |+| Grok CLI | 54/63 | 106.9s | $0.33\* | 466,465\* | 90%\* | 4,007\* | 10.4\* |++\* Over the 50 Grok runs that reported usage. The 13 runs stopped at the+240-second time limit left no usage record; they ran longest, so these+figures are likely low. Counting those runs as free gives $0.26 per task.++By suite:++| Suite | graff v0.0.302.12 | Grok CLI |+| --- | ---: | ---: |+| core (11 coding tasks), wall | 6.6s | 14.8s |+| core (11 coding tasks), cost | $0.0215 | $0.0578 |+| core (11 coding tasks), pass | 33/33 | 33/33 |+| mcp (10 MCP tasks), wall | 25.3s | 208.4s |+| mcp (10 MCP tasks), cost | $0.058 | $0.86\* |+| mcp (10 MCP tasks), pass | 30/30 | 21/30 |++**Coding tasks.** Both pass every run. graff takes less than half the time+per task (6.6s against 14.8s) and costs 63% less.++**MCP tasks.** graff is about 8 times faster (25.3s against 208.4s per task).+The Grok CLI failed 9 runs, all at the time limit, including every+`linear-split` run, where its two sub-agents never finished. Four more Grok+runs hit the limit after writing a correct report; they count as passes. Its+MCP runs that reported usage averaged 23.5 model calls and 1.24M prompt+tokens per task, because every turn re-sends the full MCP results. graff+slims list results, batches the calls in one script, and averaged 5.7 calls+and 41k prompt tokens.++**Cost.** At grok-4.7's public list price, graff costs 85% to 88% less per+task ($0.039 against $0.26 to $0.33). xAI's own figure for the Grok CLI's+build-model route is $0.098 per task; graff is 60% below that even at the+higher public price.++**Where the Grok CLI does better.** It reads more of its input from cache (90%+against graff's 57%). graff still sends about a quarter as many uncached+tokens per task (12.9k against 48.9k).++## Per task++Wall time is the mean of 3 runs.++| Task | Suite | graff v0.0.302.12 | Grok CLI |+| --- | --- | ---: | ---: |+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 6.2s | 18.5s |+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 5.0s | 12.3s |+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 2.0s | 5.0s |+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 5.2s | 7.1s |+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 9.6s | 17.0s |+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 8.5s | 16.7s |+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 5.8s | 16.4s |+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 19.7s | 173.0s |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 24.5s | 214.5s (2/3) |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 20.0s | 189.9s |+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 21.3s | 208.1s (2/3) |+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 23.1s | 203.9s (2/3) |+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 21.3s | 214.5s |+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 14.9s | 210.9s (2/3) |+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 46.7s | 233.3s (2/3) |+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 42.5s | 240.5s (0/3) |+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 18.7s | 194.9s (2/3) |+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 4.0s | 7.3s |+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 8.4s | 18.8s |+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 10.8s | 26.4s |+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 6.6s | 16.8s |++graff passed every run; Grok's passes are 3/3 unless shown. `results.json` has+every run: pass, wall time, model calls, prompt, cached and output tokens,+list cost, and xAI's reported cost for the Grok runs.++## Method++- **Tasks.** 11 coding tasks (`core`) and 10 tasks against a Linear-shaped MCP server (`mcp`, [`scripts/linear_fixture_mcp.py`](../../scripts/linear_fixture_mcp.py)), from [`graff-evals/tasks`](../../graff-evals/tasks). The Grok CLI reads the task's `.mcp.json` itself. Each task passes or fails on its own check, within its 240-second limit.+- **Order.** 3 runs per task and arm, arms interleaved task by task (`run.py --interleave`), 6 runs at a time. Each arm made one unscored request first.+- **Isolation.** Both ran with a fresh `HOME`, so neither saw user-level MCP servers or rules. The Grok CLI's `GROK_HOME` pointed at its signed-in config directory.+- **Wall time.** The runner's clock around each process, start to exit.+- **Tokens.** Prompt tokens include cached tokens for both arms (the Grok CLI's stream reports them separately). A Grok run stopped at the time limit has no result event; its tokens come from the Grok session's own usage record when one exists.+- **Cost.** List price of each run's tokens at grok-4.7's public rates: $2 per 1M input, $0.50 per 1M cached input, $6 per 1M output ([`graff-evals/list_price.py`](../../graff-evals/list_price.py)). No request in either arm reached the 200k-token higher price band (the largest Grok context was 93.6k). A SuperGrok plan is not billed per token; this makes the harnesses comparable, not a bill.++## Caveats++- **Different model variants.** The Grok CLI ran `grok-4.7-build` on its own route, graff ran `grok-4.7` on the public API.+- **Missing Grok usage.** 13 Grok runs, all MCP, were stopped at the time limit without a usage record, so Grok's token and cost means cover the other 50 runs.+- **Every task runs in a fresh git repository.** A session that stays in one repository sees fewer cold first requests in either harness.+- **Small sample.** 3 runs per task, one machine, one account, one run.++## Reproduce++```sh+cd graff-evals+# Two arms in harnesses.json (local, not committed):+# graff: ["<graff>", "-p", "{prompt}", "--model", "{model}", "--yolo"], usage graff-stderr,+# default_model "xai/grok-4.7"+# grok-cli: ["grok", "-p", "{prompt}", "-m", "{model}", "--always-approve",+# "--output-format", "streaming-messages-json"], answer/usage grok-stream,+# env GROK_HOME=<signed-in grok config dir>, default_model "grok-4.7"+# Run with HOME set to an empty directory; graff needs its xAI sign-in reachable there.+# Keep --output-root outside any repository, with scripts/linear_fixture_mcp.py+# copied to <parent of output-root>/scripts/ for the mcp suite.+python3 run.py --harness graff,grok-cli --suite core,mcp --reps 3 -j 6 --interleave \+ --output-root /path/outside/repo/run \+ --task exact-reply --task fix-fib --task file-ops --task json-transform --task regex-count \+ --task refactor-rename --task write-tests --task git-ops --task recall-noise --task csv-sum \+ --task dead-code --task linear-issues --task linear-warm --task linear-sidecar \+ --task linear-split --task linear-quiet --task linear-quiet-warm --task linear-nohint \+ --task linear-nohint-warm --task linear-reduce --task linear-reduce-warm+```diff --git a/evals/graff-vs-grok-cli/results.json b/evals/graff-vs-grok-cli/results.jsonnew file mode 100644index 00000000..ca7ac47a--- /dev/null+++ b/evals/graff-vs-grok-cli/results.json@@ -0,0 +1,2289 @@+{+ "model": {+ "graff-v302-12": "grok-4.7 (public xAI API)",+ "grok-cli": "grok-4.7, served as grok-4.7-build on the Grok CLI route"+ },+ "tasks": 21,+ "reps": 3,+ "time_limit_s": 240,+ "arms": {+ "graff-v302-12": "graff v0.0.302.12 (release)",+ "grok-cli": "Grok CLI 1.0.41"+ },+ "list_rates_per_1m": {+ "input": 2.0,+ "cached_input": 0.5,+ "output": 6.0+ },+ "summary": {+ "all": {+ "graff-v302-12": {+ "runs": 63,+ "passed": 63,+ "timed_out": 0,+ "runs_without_usage": 0,+ "wall_s_per_task": 15.5,+ "list_usd_per_task": 0.0391,+ "prompt_tokens_per_task": 29696,+ "cached_input_share": 0.567,+ "output_tokens_per_task": 823,+ "model_calls_per_task": 4.3+ },+ "grok-cli": {+ "runs": 63,+ "passed": 54,+ "timed_out": 13,+ "runs_without_usage": 13,+ "wall_s_per_task": 106.9,+ "list_usd_per_task": 0.3306,+ "prompt_tokens_per_task": 466465,+ "cached_input_share": 0.895,+ "output_tokens_per_task": 4007,+ "model_calls_per_task": 10.4,+ "grok_reported_usd_per_task": 0.0981+ }+ },+ "core": {+ "graff-v302-12": {+ "runs": 33,+ "passed": 33,+ "timed_out": 0,+ "runs_without_usage": 0,+ "wall_s_per_task": 6.6,+ "list_usd_per_task": 0.0215,+ "prompt_tokens_per_task": 19262,+ "cached_input_share": 0.645,+ "output_tokens_per_task": 263,+ "model_calls_per_task": 3.1+ },+ "grok-cli": {+ "runs": 33,+ "passed": 33,+ "timed_out": 0,+ "runs_without_usage": 0,+ "wall_s_per_task": 14.8,+ "list_usd_per_task": 0.0578,+ "prompt_tokens_per_task": 67046,+ "cached_input_share": 0.794,+ "output_tokens_per_task": 580,+ "model_calls_per_task": 3.6,+ "grok_reported_usd_per_task": 0.0196+ }+ },+ "mcp": {+ "graff-v302-12": {+ "runs": 30,+ "passed": 30,+ "timed_out": 0,+ "runs_without_usage": 0,+ "wall_s_per_task": 25.3,+ "list_usd_per_task": 0.0584,+ "prompt_tokens_per_task": 41174,+ "cached_input_share": 0.527,+ "output_tokens_per_task": 1439,+ "model_calls_per_task": 5.7+ },+ "grok-cli": {+ "runs": 30,+ "passed": 21,+ "timed_out": 13,+ "runs_without_usage": 13,+ "wall_s_per_task": 208.4,+ "list_usd_per_task": 0.8602,+ "prompt_tokens_per_task": 1241807,+ "cached_input_share": 0.906,+ "output_tokens_per_task": 10659,+ "model_calls_per_task": 23.5,+ "grok_reported_usd_per_task": 0.2503+ }+ }+ },+ "per_task": {+ "csv-sum": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.2,+ "model_calls": 3.7,+ "list_usd": 0.0187+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.5,+ "model_calls": 5,+ "list_usd": 0.0673+ }+ },+ "dead-code": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.0,+ "model_calls": 3,+ "list_usd": 0.021+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 12.3,+ "model_calls": 3,+ "list_usd": 0.047+ }+ },+ "exact-reply": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 2.0,+ "model_calls": 1,+ "list_usd": 0.0081+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.0,+ "model_calls": 1,+ "list_usd": 0.0273+ }+ },+ "file-ops": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.2,+ "model_calls": 2,+ "list_usd": 0.0131+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.1,+ "model_calls": 2,+ "list_usd": 0.0294+ }+ },+ "fix-fib": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 9.6,+ "model_calls": 4,+ "list_usd": 0.0294+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 17.0,+ "model_calls": 4,+ "list_usd": 0.0716+ }+ },+ "git-ops": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.5,+ "model_calls": 3,+ "list_usd": 0.0303+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 16.7,+ "model_calls": 3,+ "list_usd": 0.0528+ }+ },+ "json-transform": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 5.8,+ "model_calls": 3,+ "list_usd": 0.0184+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 16.4,+ "model_calls": 4,+ "list_usd": 0.0621+ }+ },+ "linear-issues": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 19.7,+ "model_calls": 4,+ "list_usd": 0.0494+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 173.0,+ "model_calls": 16.5,+ "list_usd": 0.5809+ }+ },+ "linear-nohint": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 24.5,+ "model_calls": 6,+ "list_usd": 0.0676+ },+ "grok-cli": {+ "passed": 2,+ "runs": 3,+ "wall_s": 214.5,+ "model_calls": 28.5,+ "list_usd": 1.0344+ }+ },+ "linear-nohint-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 20.0,+ "model_calls": 5.7,+ "list_usd": 0.048+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 189.9,+ "model_calls": 27,+ "list_usd": 1.0498+ }+ },+ "linear-quiet": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.3,+ "model_calls": 4.3,+ "list_usd": 0.0424+ },+ "grok-cli": {+ "passed": 2,+ "runs": 3,+ "wall_s": 208.1,+ "model_calls": 24,+ "list_usd": 0.933+ }+ },+ "linear-quiet-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 23.1,+ "model_calls": 4,+ "list_usd": 0.0388+ },+ "grok-cli": {+ "passed": 2,+ "runs": 3,+ "wall_s": 203.9,+ "model_calls": 22.5,+ "list_usd": 0.7481+ }+ },+ "linear-reduce": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 21.3,+ "model_calls": 4.7,+ "list_usd": 0.0527+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 214.5,+ "model_calls": 18,+ "list_usd": 0.5054+ }+ },+ "linear-reduce-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 14.9,+ "model_calls": 4.3,+ "list_usd": 0.0469+ },+ "grok-cli": {+ "passed": 2,+ "runs": 3,+ "wall_s": 210.9,+ "model_calls": 18,+ "list_usd": 0.6274+ }+ },+ "linear-sidecar": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 46.7,+ "model_calls": 9,+ "list_usd": 0.0878+ },+ "grok-cli": {+ "passed": 2,+ "runs": 3,+ "wall_s": 233.3,+ "model_calls": 29,+ "list_usd": 1.0836+ }+ },+ "linear-split": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 42.5,+ "model_calls": 11.3,+ "list_usd": 0.1089+ },+ "grok-cli": {+ "passed": 0,+ "runs": 3,+ "wall_s": 240.5,+ "model_calls": 0,+ "list_usd": 0+ }+ },+ "linear-warm": {+ "suite": "mcp",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.7,+ "model_calls": 4,+ "list_usd": 0.0416+ },+ "grok-cli": {+ "passed": 2,+ "runs": 3,+ "wall_s": 194.9,+ "model_calls": 20.5,+ "list_usd": 0.791+ }+ },+ "recall-noise": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 4.0,+ "model_calls": 2,+ "list_usd": 0.0171+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 7.3,+ "model_calls": 2.3,+ "list_usd": 0.0439+ }+ },+ "refactor-rename": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 8.4,+ "model_calls": 5,+ "list_usd": 0.0316+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 18.8,+ "model_calls": 5,+ "list_usd": 0.0706+ }+ },+ "regex-count": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 10.8,+ "model_calls": 4,+ "list_usd": 0.0268+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 26.4,+ "model_calls": 5.3,+ "list_usd": 0.0765+ }+ },+ "write-tests": {+ "suite": "core",+ "graff-v302-12": {+ "passed": 3,+ "runs": 3,+ "wall_s": 6.6,+ "model_calls": 3,+ "list_usd": 0.0218+ },+ "grok-cli": {+ "passed": 3,+ "runs": 3,+ "wall_s": 16.8,+ "model_calls": 5,+ "list_usd": 0.087+ }+ }+ },+ "runs": [+ {+ "arm": "graff-v302-12",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.87,+ "model_calls": 4,+ "input_tokens": 25430,+ "cached_input_tokens": 19840,+ "output_tokens": 272,+ "list_usd": 0.022732+ },+ {+ "arm": "graff-v302-12",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.62,+ "model_calls": 4,+ "input_tokens": 25344,+ "cached_input_tokens": 19840,+ "output_tokens": 226,+ "list_usd": 0.022284+ },+ {+ "arm": "graff-v302-12",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.19,+ "model_calls": 3,+ "input_tokens": 18617,+ "cached_input_tokens": 18176,+ "output_tokens": 181,+ "list_usd": 0.011056+ },+ {+ "arm": "grok-cli",+ "task": "csv-sum",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.46,+ "model_calls": 5,+ "input_tokens": 92148,+ "cached_input_tokens": 85120,+ "output_tokens": 532,+ "list_usd": 0.059808,+ "grok_reported_usd": 0.020335+ },+ {+ "arm": "grok-cli",+ "task": "csv-sum",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.59,+ "model_calls": 5,+ "input_tokens": 92602,+ "cached_input_tokens": 85120,+ "output_tokens": 926,+ "list_usd": 0.06308,+ "grok_reported_usd": 0.021447+ },+ {+ "arm": "grok-cli",+ "task": "csv-sum",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.4,+ "model_calls": 5,+ "input_tokens": 92665,+ "cached_input_tokens": 74496,+ "output_tokens": 910,+ "list_usd": 0.079046,+ "grok_reported_usd": 0.026876+ },+ {+ "arm": "graff-v302-12",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.08,+ "model_calls": 3,+ "input_tokens": 18681,+ "cached_input_tokens": 18176,+ "output_tokens": 102,+ "list_usd": 0.01071+ },+ {+ "arm": "graff-v302-12",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.22,+ "model_calls": 3,+ "input_tokens": 18621,+ "cached_input_tokens": 7808,+ "output_tokens": 206,+ "list_usd": 0.026766+ },+ {+ "arm": "graff-v302-12",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.77,+ "model_calls": 3,+ "input_tokens": 18613,+ "cached_input_tokens": 8320,+ "output_tokens": 110,+ "list_usd": 0.025406+ },+ {+ "arm": "grok-cli",+ "task": "dead-code",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.77,+ "model_calls": 3,+ "input_tokens": 55529,+ "cached_input_tokens": 48512,+ "output_tokens": 380,+ "list_usd": 0.04057,+ "grok_reported_usd": 0.013794+ },+ {+ "arm": "grok-cli",+ "task": "dead-code",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.26,+ "model_calls": 3,+ "input_tokens": 55263,+ "cached_input_tokens": 48512,+ "output_tokens": 316,+ "list_usd": 0.039654,+ "grok_reported_usd": 0.013482+ },+ {+ "arm": "grok-cli",+ "task": "dead-code",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.01,+ "model_calls": 3,+ "input_tokens": 55538,+ "cached_input_tokens": 35072,+ "output_tokens": 374,+ "list_usd": 0.060712,+ "grok_reported_usd": 0.020642+ },+ {+ "arm": "graff-v302-12",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.43,+ "model_calls": 1,+ "input_tokens": 6055,+ "cached_input_tokens": 1152,+ "output_tokens": 32,+ "list_usd": 0.010574+ },+ {+ "arm": "graff-v302-12",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 2.78,+ "model_calls": 1,+ "input_tokens": 6055,+ "cached_input_tokens": 6016,+ "output_tokens": 32,+ "list_usd": 0.003278+ },+ {+ "arm": "graff-v302-12",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 1.7,+ "model_calls": 1,+ "input_tokens": 6055,+ "cached_input_tokens": 1152,+ "output_tokens": 26,+ "list_usd": 0.010538+ },+ {+ "arm": "grok-cli",+ "task": "exact-reply",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.7,+ "model_calls": 1,+ "input_tokens": 18126,+ "cached_input_tokens": 4608,+ "output_tokens": 30,+ "list_usd": 0.02952,+ "grok_reported_usd": 0.010037+ },+ {+ "arm": "grok-cli",+ "task": "exact-reply",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.72,+ "model_calls": 1,+ "input_tokens": 18125,+ "cached_input_tokens": 1664,+ "output_tokens": 29,+ "list_usd": 0.033928,+ "grok_reported_usd": 0.011536+ },+ {+ "arm": "grok-cli",+ "task": "exact-reply",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.43,+ "model_calls": 1,+ "input_tokens": 18126,+ "cached_input_tokens": 12032,+ "output_tokens": 30,+ "list_usd": 0.018384,+ "grok_reported_usd": 0.006251+ },+ {+ "arm": "graff-v302-12",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.32,+ "model_calls": 2,+ "input_tokens": 12272,+ "cached_input_tokens": 2304,+ "output_tokens": 241,+ "list_usd": 0.022534+ },+ {+ "arm": "graff-v302-12",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.73,+ "model_calls": 2,+ "input_tokens": 12251,+ "cached_input_tokens": 12032,+ "output_tokens": 280,+ "list_usd": 0.008134+ },+ {+ "arm": "graff-v302-12",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.57,+ "model_calls": 2,+ "input_tokens": 12261,+ "cached_input_tokens": 12032,+ "output_tokens": 339,+ "list_usd": 0.008508+ },+ {+ "arm": "grok-cli",+ "task": "file-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.14,+ "model_calls": 2,+ "input_tokens": 36599,+ "cached_input_tokens": 30080,+ "output_tokens": 213,+ "list_usd": 0.029356,+ "grok_reported_usd": 0.009981+ },+ {+ "arm": "grok-cli",+ "task": "file-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.63,+ "model_calls": 2,+ "input_tokens": 36458,+ "cached_input_tokens": 30080,+ "output_tokens": 279,+ "list_usd": 0.02947,+ "grok_reported_usd": 0.01002+ },+ {+ "arm": "grok-cli",+ "task": "file-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.63,+ "model_calls": 2,+ "input_tokens": 36468,+ "cached_input_tokens": 30080,+ "output_tokens": 254,+ "list_usd": 0.02934,+ "grok_reported_usd": 0.009976+ },+ {+ "arm": "graff-v302-12",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.6,+ "model_calls": 4,+ "input_tokens": 25459,+ "cached_input_tokens": 14976,+ "output_tokens": 266,+ "list_usd": 0.03005+ },+ {+ "arm": "graff-v302-12",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.83,+ "model_calls": 4,+ "input_tokens": 26049,+ "cached_input_tokens": 9728,+ "output_tokens": 585,+ "list_usd": 0.041016+ },+ {+ "arm": "graff-v302-12",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.43,+ "model_calls": 4,+ "input_tokens": 25880,+ "cached_input_tokens": 24960,+ "output_tokens": 483,+ "list_usd": 0.017218+ },+ {+ "arm": "grok-cli",+ "task": "fix-fib",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.61,+ "model_calls": 4,+ "input_tokens": 74856,+ "cached_input_tokens": 45056,+ "output_tokens": 778,+ "list_usd": 0.086796,+ "grok_reported_usd": 0.029511+ },+ {+ "arm": "grok-cli",+ "task": "fix-fib",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.6,+ "model_calls": 4,+ "input_tokens": 74808,+ "cached_input_tokens": 54016,+ "output_tokens": 740,+ "list_usd": 0.073032,+ "grok_reported_usd": 0.024831+ },+ {+ "arm": "grok-cli",+ "task": "fix-fib",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 17.75,+ "model_calls": 4,+ "input_tokens": 75374,+ "cached_input_tokens": 67840,+ "output_tokens": 982,+ "list_usd": 0.05488,+ "grok_reported_usd": 0.018659+ },+ {+ "arm": "graff-v302-12",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.83,+ "model_calls": 3,+ "input_tokens": 19044,+ "cached_input_tokens": 7808,+ "output_tokens": 289,+ "list_usd": 0.02811+ },+ {+ "arm": "graff-v302-12",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.47,+ "model_calls": 3,+ "input_tokens": 19053,+ "cached_input_tokens": 8320,+ "output_tokens": 334,+ "list_usd": 0.02763+ },+ {+ "arm": "graff-v302-12",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.2,+ "model_calls": 3,+ "input_tokens": 19363,+ "cached_input_tokens": 3456,+ "output_tokens": 257,+ "list_usd": 0.035084+ },+ {+ "arm": "grok-cli",+ "task": "git-ops",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.17,+ "model_calls": 3,+ "input_tokens": 55655,+ "cached_input_tokens": 42496,+ "output_tokens": 770,+ "list_usd": 0.052186,+ "grok_reported_usd": 0.017743+ },+ {+ "arm": "grok-cli",+ "task": "git-ops",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 13.88,+ "model_calls": 3,+ "input_tokens": 55929,+ "cached_input_tokens": 48640,+ "output_tokens": 587,+ "list_usd": 0.04242,+ "grok_reported_usd": 0.014423+ },+ {+ "arm": "grok-cli",+ "task": "git-ops",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.98,+ "model_calls": 3,+ "input_tokens": 56324,+ "cached_input_tokens": 36608,+ "output_tokens": 1026,+ "list_usd": 0.063892,+ "grok_reported_usd": 0.021723+ },+ {+ "arm": "graff-v302-12",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.23,+ "model_calls": 3,+ "input_tokens": 18583,+ "cached_input_tokens": 8320,+ "output_tokens": 187,+ "list_usd": 0.025808+ },+ {+ "arm": "graff-v302-12",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.33,+ "model_calls": 3,+ "input_tokens": 18513,+ "cached_input_tokens": 18176,+ "output_tokens": 154,+ "list_usd": 0.010686+ },+ {+ "arm": "graff-v302-12",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.97,+ "model_calls": 3,+ "input_tokens": 18575,+ "cached_input_tokens": 13312,+ "output_tokens": 244,+ "list_usd": 0.018646+ },+ {+ "arm": "grok-cli",+ "task": "json-transform",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.08,+ "model_calls": 5,+ "input_tokens": 93868,+ "cached_input_tokens": 86272,+ "output_tokens": 1074,+ "list_usd": 0.064772,+ "grok_reported_usd": 0.022022+ },+ {+ "arm": "grok-cli",+ "task": "json-transform",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.76,+ "model_calls": 4,+ "input_tokens": 73895,+ "cached_input_tokens": 47104,+ "output_tokens": 619,+ "list_usd": 0.080848,+ "grok_reported_usd": 0.027488+ },+ {+ "arm": "grok-cli",+ "task": "json-transform",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.51,+ "model_calls": 3,+ "input_tokens": 55144,+ "cached_input_tokens": 48384,+ "output_tokens": 498,+ "list_usd": 0.0407,+ "grok_reported_usd": 0.013838+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.65,+ "model_calls": 4,+ "input_tokens": 29665,+ "cached_input_tokens": 11392,+ "output_tokens": 1952,+ "list_usd": 0.053954+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.2,+ "model_calls": 4,+ "input_tokens": 27981,+ "cached_input_tokens": 14976,+ "output_tokens": 1087,+ "list_usd": 0.04002+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.29,+ "model_calls": 4,+ "input_tokens": 28490,+ "cached_input_tokens": 7552,+ "output_tokens": 1441,+ "list_usd": 0.054298+ },+ {+ "arm": "grok-cli",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 103.43,+ "model_calls": 12,+ "input_tokens": 482817,+ "cached_input_tokens": 362368,+ "output_tokens": 7106,+ "list_usd": 0.464718,+ "grok_reported_usd": 0.158004+ },+ {+ "arm": "grok-cli",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 175.26,+ "model_calls": 21,+ "input_tokens": 949340,+ "cached_input_tokens": 841344,+ "output_tokens": 10073,+ "list_usd": 0.697102,+ "grok_reported_usd": 0.237015+ },+ {+ "arm": "grok-cli",+ "task": "linear-issues",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": true,+ "wall_s": 240.29,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 21.25,+ "model_calls": 4,+ "input_tokens": 28486,+ "cached_input_tokens": 19200,+ "output_tokens": 1482,+ "list_usd": 0.037064+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.78,+ "model_calls": 4,+ "input_tokens": 27843,+ "cached_input_tokens": 9984,+ "output_tokens": 679,+ "list_usd": 0.044784+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 27.48,+ "model_calls": 10,+ "input_tokens": 78496,+ "cached_input_tokens": 29312,+ "output_tokens": 1323,+ "list_usd": 0.120962+ },+ {+ "arm": "grok-cli",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 213.72,+ "model_calls": 25,+ "input_tokens": 1408433,+ "cached_input_tokens": 1233408,+ "output_tokens": 13385,+ "list_usd": 1.047064,+ "grok_reported_usd": 0.356002+ },+ {+ "arm": "grok-cli",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 2,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.16,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-nohint",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 189.64,+ "model_calls": 32,+ "input_tokens": 1551071,+ "cached_input_tokens": 1428096,+ "output_tokens": 10284,+ "list_usd": 1.021702,+ "grok_reported_usd": 0.347379+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 20.88,+ "model_calls": 7,+ "input_tokens": 53157,+ "cached_input_tokens": 39808,+ "output_tokens": 1368,+ "list_usd": 0.05481+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.53,+ "model_calls": 5,+ "input_tokens": 35971,+ "cached_input_tokens": 23424,+ "output_tokens": 659,+ "list_usd": 0.04076+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 26.64,+ "model_calls": 5,+ "input_tokens": 36533,+ "cached_input_tokens": 20352,+ "output_tokens": 999,+ "list_usd": 0.048532+ },+ {+ "arm": "grok-cli",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 169.92,+ "model_calls": 25,+ "input_tokens": 1300683,+ "cached_input_tokens": 1123456,+ "output_tokens": 10414,+ "list_usd": 0.978666,+ "grok_reported_usd": 0.332746+ },+ {+ "arm": "grok-cli",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 213.3,+ "model_calls": 32,+ "input_tokens": 1820740,+ "cached_input_tokens": 1674752,+ "output_tokens": 10705,+ "list_usd": 1.193582,+ "grok_reported_usd": 0.405818+ },+ {+ "arm": "grok-cli",+ "task": "linear-nohint-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 186.58,+ "model_calls": 24,+ "input_tokens": 1548710,+ "cached_input_tokens": 1456128,+ "output_tokens": 10677,+ "list_usd": 0.97729,+ "grok_reported_usd": 0.332279+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 36.39,+ "model_calls": 5,+ "input_tokens": 39325,+ "cached_input_tokens": 24576,+ "output_tokens": 2686,+ "list_usd": 0.057902+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.7,+ "model_calls": 4,+ "input_tokens": 27753,+ "cached_input_tokens": 13056,+ "output_tokens": 1010,+ "list_usd": 0.041982+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.75,+ "model_calls": 4,+ "input_tokens": 27195,+ "cached_input_tokens": 20480,+ "output_tokens": 610,+ "list_usd": 0.02733+ },+ {+ "arm": "grok-cli",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 175.49,+ "model_calls": 21,+ "input_tokens": 1055864,+ "cached_input_tokens": 945408,+ "output_tokens": 10229,+ "list_usd": 0.75499,+ "grok_reported_usd": 0.256697+ },+ {+ "arm": "grok-cli",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 208.65,+ "model_calls": 27,+ "input_tokens": 1609676,+ "cached_input_tokens": 1456896,+ "output_tokens": 12836,+ "list_usd": 1.111024,+ "grok_reported_usd": 0.377748+ },+ {+ "arm": "grok-cli",+ "task": "linear-quiet",+ "suite": "mcp",+ "rep": 3,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.26,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.07,+ "model_calls": 4,+ "input_tokens": 28290,+ "cached_input_tokens": 18944,+ "output_tokens": 922,+ "list_usd": 0.033696+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.45,+ "model_calls": 4,+ "input_tokens": 30738,+ "cached_input_tokens": 27264,+ "output_tokens": 2076,+ "list_usd": 0.033036+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.89,+ "model_calls": 4,+ "input_tokens": 30195,+ "cached_input_tokens": 14720,+ "output_tokens": 1913,+ "list_usd": 0.049788+ },+ {+ "arm": "grok-cli",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 149.0,+ "model_calls": 14,+ "input_tokens": 586404,+ "cached_input_tokens": 515200,+ "output_tokens": 9160,+ "list_usd": 0.454968,+ "grok_reported_usd": 0.154689+ },+ {+ "arm": "grok-cli",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.41,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-quiet-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 222.33,+ "model_calls": 31,+ "input_tokens": 1712511,+ "cached_input_tokens": 1638528,+ "output_tokens": 12329,+ "list_usd": 1.041204,+ "grok_reported_usd": 0.354009+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.6,+ "model_calls": 4,+ "input_tokens": 27229,+ "cached_input_tokens": 18048,+ "output_tokens": 585,+ "list_usd": 0.030896+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.44,+ "model_calls": 6,+ "input_tokens": 42016,+ "cached_input_tokens": 12288,+ "output_tokens": 1452,+ "list_usd": 0.074312+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.79,+ "model_calls": 4,+ "input_tokens": 28643,+ "cached_input_tokens": 8960,+ "output_tokens": 1502,+ "list_usd": 0.052858+ },+ {+ "arm": "grok-cli",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 162.9,+ "model_calls": 18,+ "input_tokens": 743688,+ "cached_input_tokens": 697472,+ "output_tokens": 10703,+ "list_usd": 0.505386,+ "grok_reported_usd": 0.171831+ },+ {+ "arm": "grok-cli",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": true,+ "wall_s": 240.09,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-reduce",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": true,+ "wall_s": 240.36,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.68,+ "model_calls": 5,+ "input_tokens": 35763,+ "cached_input_tokens": 14848,+ "output_tokens": 927,+ "list_usd": 0.054816+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.93,+ "model_calls": 4,+ "input_tokens": 29420,+ "cached_input_tokens": 13056,+ "output_tokens": 1400,+ "list_usd": 0.047656+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.22,+ "model_calls": 4,+ "input_tokens": 28176,+ "cached_input_tokens": 14720,+ "output_tokens": 658,+ "list_usd": 0.03822+ },+ {+ "arm": "grok-cli",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 151.94,+ "model_calls": 18,+ "input_tokens": 839586,+ "cached_input_tokens": 738944,+ "output_tokens": 9438,+ "list_usd": 0.627384,+ "grok_reported_usd": 0.213311+ },+ {+ "arm": "grok-cli",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.5,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-reduce-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": true,+ "wall_s": 240.19,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 90.94,+ "model_calls": 11,+ "input_tokens": 78174,+ "cached_input_tokens": 48384,+ "output_tokens": 2329,+ "list_usd": 0.097746+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 25.12,+ "model_calls": 9,+ "input_tokens": 63155,+ "cached_input_tokens": 27648,+ "output_tokens": 1462,+ "list_usd": 0.09361+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 23.94,+ "model_calls": 7,+ "input_tokens": 49237,+ "cached_input_tokens": 22656,+ "output_tokens": 1265,+ "list_usd": 0.07208+ },+ {+ "arm": "grok-cli",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 227.95,+ "model_calls": 28,+ "input_tokens": 1173801,+ "cached_input_tokens": 1011840,+ "output_tokens": 12284,+ "list_usd": 0.903546,+ "grok_reported_usd": 0.009746+ },+ {+ "arm": "grok-cli",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 2,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.46,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-sidecar",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 231.62,+ "model_calls": 30,+ "input_tokens": 2012313,+ "cached_input_tokens": 1892224,+ "output_tokens": 12893,+ "list_usd": 1.263648,+ "grok_reported_usd": 0.009541+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 40.8,+ "model_calls": 13,+ "input_tokens": 89608,+ "cached_input_tokens": 50816,+ "output_tokens": 2727,+ "list_usd": 0.119354+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 49.21,+ "model_calls": 8,+ "input_tokens": 57236,+ "cached_input_tokens": 23040,+ "output_tokens": 2549,+ "list_usd": 0.095206+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 37.53,+ "model_calls": 13,+ "input_tokens": 90059,+ "cached_input_tokens": 56320,+ "output_tokens": 2750,+ "list_usd": 0.112138+ },+ {+ "arm": "grok-cli",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 1,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.58,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 2,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.49,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-split",+ "suite": "mcp",+ "rep": 3,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.44,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 24.97,+ "model_calls": 4,+ "input_tokens": 28948,+ "cached_input_tokens": 4608,+ "output_tokens": 1393,+ "list_usd": 0.059342+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 11.25,+ "model_calls": 4,+ "input_tokens": 27939,+ "cached_input_tokens": 19968,+ "output_tokens": 617,+ "list_usd": 0.029628+ },+ {+ "arm": "graff-v302-12",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.88,+ "model_calls": 4,+ "input_tokens": 29492,+ "cached_input_tokens": 20736,+ "output_tokens": 1345,+ "list_usd": 0.03595+ },+ {+ "arm": "grok-cli",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 178.9,+ "model_calls": 19,+ "input_tokens": 1031519,+ "cached_input_tokens": 905984,+ "output_tokens": 9122,+ "list_usd": 0.758794,+ "grok_reported_usd": 0.25799+ },+ {+ "arm": "grok-cli",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 2,+ "pass": false,+ "timed_out": true,+ "wall_s": 240.28,+ "model_calls": null,+ "input_tokens": null,+ "cached_input_tokens": null,+ "output_tokens": null,+ "list_usd": null,+ "grok_reported_usd": null+ },+ {+ "arm": "grok-cli",+ "task": "linear-warm",+ "suite": "mcp",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 165.4,+ "model_calls": 22,+ "input_tokens": 1283560,+ "cached_input_tokens": 1200896,+ "output_tokens": 9559,+ "list_usd": 0.82313,+ "grok_reported_usd": 0.279864+ },+ {+ "arm": "graff-v302-12",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 4.51,+ "model_calls": 2,+ "input_tokens": 12390,+ "cached_input_tokens": 7168,+ "output_tokens": 105,+ "list_usd": 0.014658+ },+ {+ "arm": "graff-v302-12",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.92,+ "model_calls": 2,+ "input_tokens": 12390,+ "cached_input_tokens": 7168,+ "output_tokens": 117,+ "list_usd": 0.01473+ },+ {+ "arm": "graff-v302-12",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 3.66,+ "model_calls": 2,+ "input_tokens": 12399,+ "cached_input_tokens": 2304,+ "output_tokens": 107,+ "list_usd": 0.021984+ },+ {+ "arm": "grok-cli",+ "task": "recall-noise",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.02,+ "model_calls": 3,+ "input_tokens": 54881,+ "cached_input_tokens": 34816,+ "output_tokens": 174,+ "list_usd": 0.058582,+ "grok_reported_usd": 0.019918+ },+ {+ "arm": "grok-cli",+ "task": "recall-noise",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.01,+ "model_calls": 2,+ "input_tokens": 36589,+ "cached_input_tokens": 30080,+ "output_tokens": 131,+ "list_usd": 0.028844,+ "grok_reported_usd": 0.009807+ },+ {+ "arm": "grok-cli",+ "task": "recall-noise",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 6.73,+ "model_calls": 2,+ "input_tokens": 36569,+ "cached_input_tokens": 19712,+ "output_tokens": 123,+ "list_usd": 0.044308,+ "grok_reported_usd": 0.015065+ },+ {+ "arm": "graff-v302-12",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.42,+ "model_calls": 4,+ "input_tokens": 25559,+ "cached_input_tokens": 14976,+ "output_tokens": 294,+ "list_usd": 0.030418+ },+ {+ "arm": "graff-v302-12",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 7.49,+ "model_calls": 4,+ "input_tokens": 25551,+ "cached_input_tokens": 19584,+ "output_tokens": 307,+ "list_usd": 0.023568+ },+ {+ "arm": "graff-v302-12",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.22,+ "model_calls": 7,+ "input_tokens": 45672,+ "cached_input_tokens": 34816,+ "output_tokens": 307,+ "list_usd": 0.040962+ },+ {+ "arm": "grok-cli",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 14.83,+ "model_calls": 5,+ "input_tokens": 92924,+ "cached_input_tokens": 66048,+ "output_tokens": 526,+ "list_usd": 0.089932,+ "grok_reported_usd": 0.030577+ },+ {+ "arm": "grok-cli",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 19.18,+ "model_calls": 5,+ "input_tokens": 93469,+ "cached_input_tokens": 86016,+ "output_tokens": 533,+ "list_usd": 0.061112,+ "grok_reported_usd": 0.020778+ },+ {+ "arm": "grok-cli",+ "task": "refactor-rename",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 22.34,+ "model_calls": 5,+ "input_tokens": 92953,+ "cached_input_tokens": 85760,+ "output_tokens": 596,+ "list_usd": 0.060842,+ "grok_reported_usd": 0.020686+ },+ {+ "arm": "graff-v302-12",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 12.08,+ "model_calls": 4,+ "input_tokens": 24937,+ "cached_input_tokens": 19584,+ "output_tokens": 683,+ "list_usd": 0.024596+ },+ {+ "arm": "graff-v302-12",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 10.44,+ "model_calls": 4,+ "input_tokens": 25043,+ "cached_input_tokens": 13824,+ "output_tokens": 404,+ "list_usd": 0.031774+ },+ {+ "arm": "graff-v302-12",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 9.77,+ "model_calls": 4,+ "input_tokens": 24959,+ "cached_input_tokens": 19456,+ "output_tokens": 539,+ "list_usd": 0.023968+ },+ {+ "arm": "grok-cli",+ "task": "regex-count",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 29.86,+ "model_calls": 6,+ "input_tokens": 113525,+ "cached_input_tokens": 99200,+ "output_tokens": 1468,+ "list_usd": 0.087058,+ "grok_reported_usd": 0.0296+ },+ {+ "arm": "grok-cli",+ "task": "regex-count",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 33.87,+ "model_calls": 6,+ "input_tokens": 115681,+ "cached_input_tokens": 107136,+ "output_tokens": 1777,+ "list_usd": 0.08132,+ "grok_reported_usd": 0.027649+ },+ {+ "arm": "grok-cli",+ "task": "regex-count",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.44,+ "model_calls": 4,+ "input_tokens": 74378,+ "cached_input_tokens": 61184,+ "output_tokens": 698,+ "list_usd": 0.061168,+ "grok_reported_usd": 0.020797+ },+ {+ "arm": "graff-v302-12",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.86,+ "model_calls": 3,+ "input_tokens": 18684,+ "cached_input_tokens": 13312,+ "output_tokens": 305,+ "list_usd": 0.01923+ },+ {+ "arm": "graff-v302-12",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 8.11,+ "model_calls": 3,+ "input_tokens": 18684,+ "cached_input_tokens": 8320,+ "output_tokens": 362,+ "list_usd": 0.02706+ },+ {+ "arm": "graff-v302-12",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 5.85,+ "model_calls": 3,+ "input_tokens": 18610,+ "cached_input_tokens": 13312,+ "output_tokens": 304,+ "list_usd": 0.019076+ },+ {+ "arm": "grok-cli",+ "task": "write-tests",+ "suite": "core",+ "rep": 1,+ "pass": true,+ "timed_out": false,+ "wall_s": 16.68,+ "model_calls": 5,+ "input_tokens": 92336,+ "cached_input_tokens": 85248,+ "output_tokens": 569,+ "list_usd": 0.060214,+ "grok_reported_usd": 0.020473+ },+ {+ "arm": "grok-cli",+ "task": "write-tests",+ "suite": "core",+ "rep": 2,+ "pass": true,+ "timed_out": false,+ "wall_s": 15.69,+ "model_calls": 5,+ "input_tokens": 92880,+ "cached_input_tokens": 49408,+ "output_tokens": 591,+ "list_usd": 0.115194,+ "grok_reported_usd": 0.039166+ },+ {+ "arm": "grok-cli",+ "task": "write-tests",+ "suite": "core",+ "rep": 3,+ "pass": true,+ "timed_out": false,+ "wall_s": 18.1,+ "model_calls": 5,+ "input_tokens": 92832,+ "cached_input_tokens": 69248,+ "output_tokens": 612,+ "list_usd": 0.085464,+ "grok_reported_usd": 0.029058+ }+ ]+}diff --git a/graff-evals/README.md b/graff-evals/README.mdindex 8eabb2a3..078fda44 100644--- a/graff-evals/README.md+++ b/graff-evals/README.md@@ -68,6 +68,7 @@ cd graff-evals./run.py --harness grok --task fix-fib --reps 3# harness-vs-harness on the same model+# Results on grok-4.7: evals/graff-vs-grok-cli/RESULTS.md./run.py --harness graff,grok --model grok-4.6# a different model through graff