Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

Add an mlx eval arm and the Flint INT4 results

Rach Pradhanc82aa7e75131
diff --git a/evals/flint-mlx/RESULTS.md b/evals/flint-mlx/RESULTS.md
new file mode 100644
index 00000000..15eccde5
--- /dev/null
+++ b/evals/flint-mlx/RESULTS.md
@@ -0,0 +1,72 @@
+# Recursive Flint on a local MLX server
+
+Measured 2026-10-22. Recursive Flint (39.8B total parameters, about 2.5B active per token) ran the same 21 tasks as the gateway write-ups, 2 runs of each, through graff's `mlx` provider. The weights are the MLX INT4 checkpoint (`flint-int4`), served on one Apple M4 Pro with 48 GB by Recursive's `rec` 0.1.0, an OpenAI-compatible server at `127.0.0.1:8080`. Every call was local, so the runs cost nothing.
+
+| Arm | What ran |
+| --- | --- |
+| graff | This branch's build as a scripted `graff repl` on the `graff-mlx-repl` arm (`--model mlx/flint`), at the default effort. |
+
+| | Pass | Wall per task | Input read from cache | Output tokens per task | Model calls per task |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| graff + Flint INT4 (local) | 39/42 | 80.4s | 84% | 3,439 | 6.6 |
+
+By suite:
+
+| Suite | Pass | Wall per task | Input read from cache | Output tokens per task |
+| --- | ---: | ---: | ---: | ---: |
+| core (11 coding tasks) | 22/22 | 16.8s | 80% | 619 |
+| mcp (10 MCP tasks) | 17/20 | 150.4s | 86% | 6,541 |
+
+All three misses are timeouts, not wrong answers: both runs of `linear-split` and one run of `linear-quiet` reached the 240 s task limit before writing their report. Flint reasons at length on the MCP tasks (5,000 to 11,000 output tokens per run), so the tasks that time out are the ones where that reasoning plus local decoding runs past the limit. The `linear-quiet` run that finished took 187 s. Every finished run passed its held-out check.
+
+Against the hosted models in [mimo-v2.6](../mimo-v2.6/RESULTS.md), the pass rate matches MiMo v2.6 Pro with thinking off (39/42), and the core suite takes about as long as MiMo with thinking on (16.8s against 15.8s). The MCP tasks take longer because Flint writes more and decodes it locally.
+
+## Prompt cache
+
+`rec` keeps one prompt cache and reuses the longest prefix it shares with the next request. Inside a session, each request reused the whole previous prompt, so only the new tool output and reply were prefilled. graff sends each assistant turn's `reasoning_content` back in the history, which keeps the rendered prompt identical to what the server cached. Across runs, the shared system prompt hits the cache too: `exact-reply` took 11.5 s cold and 1.7 s warm. Because there is one cache, the suite ran one task at a time (`-j 1`); parallel sessions would evict each other's prefix.
+
+## Throughput
+
+Single streamed requests with no tools and a unique prefix (so nothing came from cache), 512 output tokens:
+
+| Prompt | Prefill | Decode |
+| --- | ---: | ---: |
+| ~900 tokens | 660 to 790 tok/s | 117 to 123 tok/s |
+| ~7,000 tokens | 480 to 600 tok/s | 105 to 109 tok/s |
+| ~28,000 tokens | 500 to 545 tok/s | 80 tok/s |
+
+In the suite's long MCP runs, where output dominates, decoding ran at about 63 to 96 tok/s with 20,000 to 50,000 tokens of context per request.
+
+## Per task
+
+Wall time is the mean of 2 runs; a task that failed in a run is marked with its pass count.
+
+| Task | Suite | Wall | Output tokens | Model calls |
+| --- | --- | ---: | ---: | ---: |
+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 21.2s | 513 | 6.0 |
+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 13.2s | 496 | 3.0 |
+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 6.6s | 123 | 1.0 |
+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 14.3s | 551 | 3.0 |
+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 23.5s | 1,035 | 5.5 |
+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 16.2s | 456 | 3.5 |
+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 10.2s | 417 | 3.0 |
+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 7.4s | 104 | 2.0 |
+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 16.0s | 395 | 6.0 |
+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 30.9s | 1,740 | 6.0 |
+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 25.6s | 977 | 7.0 |
+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 86.7s | 5,545 | 7.0 |
+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 172.0s | 11,315 | 11.5 |
+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 125.1s | 6,642 | 6.0 |
+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 213.5s (1/2) | 5,338 | 5.0 |
+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 134.0s | 8,904 | 7.0 |
+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 97.2s | 5,691 | 12.0 |
+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 93.9s | 5,941 | 8.0 |
+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 211.0s | 9,802 | 18.5 |
+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 240.1s (0/2) | timed out | timed out |
+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 130.3s | 6,234 | 18.0 |
+
+A timed-out run records no usage, so its output and calls count as 0 in the averages above.
+
+## Reproduce
+
+Install `rec` ([recursive-oss/rec](https://github.com/recursive-oss/rec)), then `rec pull` and `rec serve --port 8080`. With `MLX_API_KEY=$(rec auth token)` set, run `graff-evals/run.py --suite core,mcp --harness graff-mlx-repl --model flint --reps 2 -j 1` with `--output-root` outside any repository and `scripts/linear_fixture_mcp.py` copied next to it.
diff --git a/evals/flint-mlx/results.json b/evals/flint-mlx/results.json
new file mode 100644
index 00000000..94e0770a
--- /dev/null
+++ b/evals/flint-mlx/results.json
@@ -0,0 +1,800 @@
+{
+ "model": "Recursive Flint INT4 (flint-int4), served locally by rec 0.1.0 as an OpenAI-compatible server",
+ "tasks": 21,
+ "reps": 2,
+ "arms": {
+ "graff": "graff (this branch) as a scripted graff repl on the mlx provider"
+ },
+ "summary": {
+ "all": {
+ "graff": {
+ "runs": 42,
+ "passed": 39,
+ "wall_s_per_task": 80.4,
+ "cached_input_share": 0.844,
+ "output_tokens_per_task": 3439,
+ "model_calls_per_task": 6.6
+ }
+ },
+ "core": {
+ "graff": {
+ "runs": 22,
+ "passed": 22,
+ "wall_s_per_task": 16.8,
+ "cached_input_share": 0.803,
+ "output_tokens_per_task": 619,
+ "model_calls_per_task": 4.2
+ }
+ },
+ "mcp": {
+ "graff": {
+ "runs": 20,
+ "passed": 17,
+ "wall_s_per_task": 150.4,
+ "cached_input_share": 0.855,
+ "output_tokens_per_task": 6541,
+ "model_calls_per_task": 9.3
+ }
+ }
+ },
+ "per_task": {
+ "csv-sum": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 21.2,
+ "model_calls": 6.0,
+ "output_tokens": 513
+ }
+ },
+ "dead-code": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 13.2,
+ "model_calls": 3.0,
+ "output_tokens": 496
+ }
+ },
+ "exact-reply": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 6.6,
+ "model_calls": 1.0,
+ "output_tokens": 123
+ }
+ },
+ "file-ops": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 14.3,
+ "model_calls": 3.0,
+ "output_tokens": 551
+ }
+ },
+ "fix-fib": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 23.5,
+ "model_calls": 5.5,
+ "output_tokens": 1035
+ }
+ },
+ "git-ops": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 16.2,
+ "model_calls": 3.5,
+ "output_tokens": 456
+ }
+ },
+ "json-transform": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 10.2,
+ "model_calls": 3.0,
+ "output_tokens": 417
+ }
+ },
+ "linear-issues": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 86.7,
+ "model_calls": 7.0,
+ "output_tokens": 5545
+ }
+ },
+ "linear-nohint": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 172.0,
+ "model_calls": 11.5,
+ "output_tokens": 11315
+ }
+ },
+ "linear-nohint-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 125.1,
+ "model_calls": 6.0,
+ "output_tokens": 6642
+ }
+ },
+ "linear-quiet": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 1,
+ "runs": 2,
+ "wall_s": 213.5,
+ "model_calls": 5.0,
+ "output_tokens": 5338
+ }
+ },
+ "linear-quiet-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 134.0,
+ "model_calls": 7.0,
+ "output_tokens": 8904
+ }
+ },
+ "linear-reduce": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 97.2,
+ "model_calls": 12.0,
+ "output_tokens": 5691
+ }
+ },
+ "linear-reduce-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 93.9,
+ "model_calls": 8.0,
+ "output_tokens": 5941
+ }
+ },
+ "linear-sidecar": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 211.0,
+ "model_calls": 18.5,
+ "output_tokens": 9802
+ }
+ },
+ "linear-split": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 0,
+ "runs": 2,
+ "wall_s": 240.1,
+ "model_calls": 0.0,
+ "output_tokens": 0
+ }
+ },
+ "linear-warm": {
+ "suite": "mcp",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 130.3,
+ "model_calls": 18.0,
+ "output_tokens": 6234
+ }
+ },
+ "recall-noise": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 7.4,
+ "model_calls": 2.0,
+ "output_tokens": 104
+ }
+ },
+ "refactor-rename": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 16.0,
+ "model_calls": 6.0,
+ "output_tokens": 395
+ }
+ },
+ "regex-count": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 30.9,
+ "model_calls": 6.0,
+ "output_tokens": 1740
+ }
+ },
+ "write-tests": {
+ "suite": "core",
+ "graff": {
+ "passed": 2,
+ "runs": 2,
+ "wall_s": 25.6,
+ "model_calls": 7.0,
+ "output_tokens": 977
+ }
+ }
+ },
+ "runs": [
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 11.53,
+ "model_calls": 1,
+ "input_tokens": 7383,
+ "cached_input_tokens": 0,
+ "output_tokens": 123
+ },
+ {
+ "arm": "graff",
+ "task": "exact-reply",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 1.74,
+ "model_calls": 1,
+ "input_tokens": 7383,
+ "cached_input_tokens": 7383,
+ "output_tokens": 123
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.99,
+ "model_calls": 5,
+ "input_tokens": 40572,
+ "cached_input_tokens": 31977,
+ "output_tokens": 1018
+ },
+ {
+ "arm": "graff",
+ "task": "fix-fib",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 25.05,
+ "model_calls": 6,
+ "input_tokens": 49192,
+ "cached_input_tokens": 39276,
+ "output_tokens": 1052
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 21.58,
+ "model_calls": 3,
+ "input_tokens": 23001,
+ "cached_input_tokens": 15119,
+ "output_tokens": 551
+ },
+ {
+ "arm": "graff",
+ "task": "file-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.98,
+ "model_calls": 3,
+ "input_tokens": 23001,
+ "cached_input_tokens": 22519,
+ "output_tokens": 551
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.2,
+ "model_calls": 3,
+ "input_tokens": 22774,
+ "cached_input_tokens": 14986,
+ "output_tokens": 417
+ },
+ {
+ "arm": "graff",
+ "task": "json-transform",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 5.17,
+ "model_calls": 3,
+ "input_tokens": 22774,
+ "cached_input_tokens": 22422,
+ "output_tokens": 417
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.89,
+ "model_calls": 6,
+ "input_tokens": 48802,
+ "cached_input_tokens": 39557,
+ "output_tokens": 1740
+ },
+ {
+ "arm": "graff",
+ "task": "regex-count",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 30.87,
+ "model_calls": 6,
+ "input_tokens": 48802,
+ "cached_input_tokens": 39557,
+ "output_tokens": 1740
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 16.05,
+ "model_calls": 6,
+ "input_tokens": 45850,
+ "cached_input_tokens": 38014,
+ "output_tokens": 395
+ },
+ {
+ "arm": "graff",
+ "task": "refactor-rename",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 15.99,
+ "model_calls": 6,
+ "input_tokens": 45850,
+ "cached_input_tokens": 38014,
+ "output_tokens": 395
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 28.5,
+ "model_calls": 7,
+ "input_tokens": 54706,
+ "cached_input_tokens": 46467,
+ "output_tokens": 977
+ },
+ {
+ "arm": "graff",
+ "task": "write-tests",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 22.78,
+ "model_calls": 7,
+ "input_tokens": 54706,
+ "cached_input_tokens": 46467,
+ "output_tokens": 977
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.74,
+ "model_calls": 4,
+ "input_tokens": 30916,
+ "cached_input_tokens": 22913,
+ "output_tokens": 678
+ },
+ {
+ "arm": "graff",
+ "task": "git-ops",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 13.71,
+ "model_calls": 3,
+ "input_tokens": 22744,
+ "cached_input_tokens": 15011,
+ "output_tokens": 234
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 12.69,
+ "model_calls": 2,
+ "input_tokens": 15103,
+ "cached_input_tokens": 7423,
+ "output_tokens": 104
+ },
+ {
+ "arm": "graff",
+ "task": "recall-noise",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 2.08,
+ "model_calls": 2,
+ "input_tokens": 15103,
+ "cached_input_tokens": 14846,
+ "output_tokens": 104
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 18.11,
+ "model_calls": 6,
+ "input_tokens": 46154,
+ "cached_input_tokens": 38248,
+ "output_tokens": 513
+ },
+ {
+ "arm": "graff",
+ "task": "csv-sum",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 24.26,
+ "model_calls": 6,
+ "input_tokens": 46154,
+ "cached_input_tokens": 38248,
+ "output_tokens": 513
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 19.54,
+ "model_calls": 3,
+ "input_tokens": 23067,
+ "cached_input_tokens": 15061,
+ "output_tokens": 496
+ },
+ {
+ "arm": "graff",
+ "task": "dead-code",
+ "suite": "core",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 6.91,
+ "model_calls": 3,
+ "input_tokens": 23067,
+ "cached_input_tokens": 22500,
+ "output_tokens": 496
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 85.67,
+ "model_calls": 7,
+ "input_tokens": 83930,
+ "cached_input_tokens": 68400,
+ "output_tokens": 5545
+ },
+ {
+ "arm": "graff",
+ "task": "linear-issues",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 87.73,
+ "model_calls": 7,
+ "input_tokens": 83930,
+ "cached_input_tokens": 68400,
+ "output_tokens": 5545
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 105.88,
+ "model_calls": 15,
+ "input_tokens": 216467,
+ "cached_input_tokens": 195962,
+ "output_tokens": 5380
+ },
+ {
+ "arm": "graff",
+ "task": "linear-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 154.68,
+ "model_calls": 21,
+ "input_tokens": 329725,
+ "cached_input_tokens": 306385,
+ "output_tokens": 7088
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 198.91,
+ "model_calls": 18,
+ "input_tokens": 239522,
+ "cached_input_tokens": 195069,
+ "output_tokens": 9145
+ },
+ {
+ "arm": "graff",
+ "task": "linear-sidecar",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 223.02,
+ "model_calls": 19,
+ "input_tokens": 234962,
+ "cached_input_tokens": 187402,
+ "output_tokens": 10460
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": false,
+ "timed_out": true,
+ "wall_s": 240.24,
+ "model_calls": 0,
+ "input_tokens": 0,
+ "cached_input_tokens": 0,
+ "output_tokens": 0
+ },
+ {
+ "arm": "graff",
+ "task": "linear-split",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": false,
+ "timed_out": true,
+ "wall_s": 240.05,
+ "model_calls": 0,
+ "input_tokens": 0,
+ "cached_input_tokens": 0,
+ "output_tokens": 0
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": false,
+ "timed_out": true,
+ "wall_s": 240.46,
+ "model_calls": 0,
+ "input_tokens": 0,
+ "cached_input_tokens": 0,
+ "output_tokens": 0
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 186.56,
+ "model_calls": 10,
+ "input_tokens": 185550,
+ "cached_input_tokens": 159596,
+ "output_tokens": 10675
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 125.34,
+ "model_calls": 8,
+ "input_tokens": 122586,
+ "cached_input_tokens": 104167,
+ "output_tokens": 8128
+ },
+ {
+ "arm": "graff",
+ "task": "linear-quiet-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 142.6,
+ "model_calls": 6,
+ "input_tokens": 90117,
+ "cached_input_tokens": 70946,
+ "output_tokens": 9680
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 160.13,
+ "model_calls": 11,
+ "input_tokens": 182826,
+ "cached_input_tokens": 160142,
+ "output_tokens": 10346
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 183.87,
+ "model_calls": 12,
+ "input_tokens": 192222,
+ "cached_input_tokens": 170078,
+ "output_tokens": 12284
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 103.22,
+ "model_calls": 7,
+ "input_tokens": 115728,
+ "cached_input_tokens": 94807,
+ "output_tokens": 5589
+ },
+ {
+ "arm": "graff",
+ "task": "linear-nohint-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 146.95,
+ "model_calls": 5,
+ "input_tokens": 80083,
+ "cached_input_tokens": 57615,
+ "output_tokens": 7695
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 96.76,
+ "model_calls": 12,
+ "input_tokens": 144080,
+ "cached_input_tokens": 128395,
+ "output_tokens": 5691
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 97.62,
+ "model_calls": 12,
+ "input_tokens": 144080,
+ "cached_input_tokens": 128395,
+ "output_tokens": 5691
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 1,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 94.09,
+ "model_calls": 8,
+ "input_tokens": 95572,
+ "cached_input_tokens": 80041,
+ "output_tokens": 5941
+ },
+ {
+ "arm": "graff",
+ "task": "linear-reduce-warm",
+ "suite": "mcp",
+ "rep": 2,
+ "pass": true,
+ "timed_out": false,
+ "wall_s": 93.7,
+ "model_calls": 8,
+ "input_tokens": 95572,
+ "cached_input_tokens": 80041,
+ "output_tokens": 5941
+ }
+ ]
+}
\ No newline at end of file
diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.json
index ac3f186a..b1ffc7c5 100644
--- a/graff-evals/harnesses.json
+++ b/graff-evals/harnesses.json
@@ -384,6 +384,7 @@
"note": "Claude Code (`claude -p`, stream-json) with the task's .mcp.json as its only MCP config. Set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN for the endpoint; a chat-completions-only endpoint needs an Anthropic-format proxy in front (evals/graff-vs-claude-code). Pass --model."
},
"graff-deepseek": {"cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"], "stdin": "\u001b[200~{prompt}\u001b[201~\n", "answer": "stdout", "usage": "graff-stderr", "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"}, "model_prefix": "deepseek", "default_model": "deepseek-v4-pro", "note": "scripted graff repl on DeepSeek's own API (DEEPSEEK_API_KEY)"},
+ "graff-mlx-repl": {"cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"], "stdin": "\u001b[200~{prompt}\u001b[201~\n", "answer": "stdout", "usage": "graff-stderr", "env": {"GRAFF_POST_DEADLINE_SECS": "300", "GRAFF_NO_TELEMETRY": "1"}, "model_prefix": "mlx", "default_model": "flint", "note": "scripted graff repl on a local MLX server at 127.0.0.1:8080 (MLX_API_KEY; for Flint, `rec serve --port 8080` and MLX_API_KEY=$(rec auth token)). The server keeps one prompt cache, so run -j 1; the longer post deadline covers a cold local prefill"},
"deepseek-harness": {"cmd": ["python3", "{repo}/graff-evals/dsh_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "env": {"DSH_PERMISSION_MODE": "danger-full-access"}, "default_model": "deepseek-v4-pro", "note": "deepseek-harness (@deepseek-ai/dsh, DSH_BIN when dsh is not on PATH) headless on DeepSeek's own API; the model and the task's MCP servers go in a patch layer"},
"graff-gateway-repl-norlm": {"cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"], "stdin": "\u001b[200~{prompt}\u001b[201~\n", "answer": "stdout", "usage": "graff-stderr", "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1", "GRAFF_RLM": "0"}, "model_prefix": "codegraff", "note": "graff-gateway-repl with rlm off (GRAFF_RLM=0: the structured-only catalog), the counterfactual for code-mode comparisons; pass --model"},
"pi1": {"cmd": ["python3", "{repo}/graff-evals/pi1_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "model_prefix": "codegraff", "env": {"PI1_CODEMODE": "0"}, "note": "Pi 1.0 (@earendil-works/pi-coding-agent 1.x) headless with its default tools; MCP tools are reached through codemode, Pi's default MCP exposure. PI1_MODELS / PI1_AUTH copy a models.json / auth.json into the run's private ~/.pi/agent, PI1_THINKING passes --thinking (match the other arm's effort), PI1_BIN picks Pi 1.x; pass --model provider/model for a provider other than codegraff"},