Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
5.4 KB
Raw Blame

Recursive Flint on a local MLX server

Measured 2026-10-22. Recursive Flint (39.8B total parameters, about 2.5B active per token) ran the same 21 tasks as the gateway write-ups, 2 runs of each, through graff's mlx provider. The weights are the MLX INT4 checkpoint (flint-int4), served on one Apple M4 Pro with 48 GB by Recursive's rec 0.1.0, an OpenAI-compatible server at 127.0.0.1:8080. Every call was local, so the runs cost nothing.

Arm What ran
graff This branch's build as a scripted graff repl on the graff-mlx-repl arm (--model mlx/flint), at the default effort.
Pass Wall per task Input read from cache Output tokens per task Model calls per task
graff + Flint INT4 (local) 39/42 80.4s 84% 3,439 6.6

By suite:

Suite Pass Wall per task Input read from cache Output tokens per task
core (11 coding tasks) 22/22 16.8s 80% 619
mcp (10 MCP tasks) 17/20 150.4s 86% 6,541

All three misses are timeouts, not wrong answers: both runs of linear-split and one run of linear-quiet reached the 240 s task limit before writing their report. Flint reasons at length on the MCP tasks (5,000 to 11,000 output tokens per run), so the tasks that time out are the ones where that reasoning plus local decoding runs past the limit. The linear-quiet run that finished took 187 s. Every finished run passed its held-out check.

Against the hosted models in mimo-v2.6, the pass rate matches MiMo v2.6 Pro with thinking off (39/42), and the core suite takes about as long as MiMo with thinking on (16.8s against 15.8s). The MCP tasks take longer because Flint writes more and decodes it locally.

Prompt cache

rec keeps one prompt cache and reuses the longest prefix it shares with the next request. Inside a session, each request reused the whole previous prompt, so only the new tool output and reply were prefilled. graff sends each assistant turn's reasoning_content back in the history, which keeps the rendered prompt identical to what the server cached. Across runs, the shared system prompt hits the cache too: exact-reply took 11.5 s cold and 1.7 s warm. Because there is one cache, the suite ran one task at a time (-j 1); parallel sessions would evict each other's prefix.

Throughput

Single streamed requests with no tools and a unique prefix (so nothing came from cache), 512 output tokens:

Prompt Prefill Decode
~900 tokens 660 to 790 tok/s 117 to 123 tok/s
~7,000 tokens 480 to 600 tok/s 105 to 109 tok/s
~28,000 tokens 500 to 545 tok/s 80 tok/s

In the suite's long MCP runs, where output dominates, decoding ran at about 63 to 96 tok/s with 20,000 to 50,000 tokens of context per request.

Per task

Wall time is the mean of 2 runs; a task that failed in a run is marked with its pass count.

Task Suite Wall Output tokens Model calls
csv-sum core 21.2s 513 6.0
dead-code core 13.2s 496 3.0
exact-reply core 6.6s 123 1.0
file-ops core 14.3s 551 3.0
fix-fib core 23.5s 1,035 5.5
git-ops core 16.2s 456 3.5
json-transform core 10.2s 417 3.0
recall-noise core 7.4s 104 2.0
refactor-rename core 16.0s 395 6.0
regex-count core 30.9s 1,740 6.0
write-tests core 25.6s 977 7.0
linear-issues mcp 86.7s 5,545 7.0
linear-nohint mcp 172.0s 11,315 11.5
linear-nohint-warm mcp 125.1s 6,642 6.0
linear-quiet mcp 213.5s (1/2) 5,338 5.0
linear-quiet-warm mcp 134.0s 8,904 7.0
linear-reduce mcp 97.2s 5,691 12.0
linear-reduce-warm mcp 93.9s 5,941 8.0
linear-sidecar mcp 211.0s 9,802 18.5
linear-split mcp 240.1s (0/2) timed out timed out
linear-warm mcp 130.3s 6,234 18.0

A timed-out run records no usage, so its output and calls count as 0 in the averages above.

Reproduce

Install rec (recursive-oss/rec), then rec pull and rec serve --port 8080. With MLX_API_KEY=$(rec auth token) set, run graff-evals/run.py --suite core,mcp --harness graff-mlx-repl --model flint --reps 2 -j 1 with --output-root outside any repository and scripts/linear_fixture_mcp.py copied next to it.