Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
5.8 KB
Raw Blame

graff vs deepseek-harness on DeepSeek V4 Pro

Measured 2026-10-02. Both harnesses ran DeepSeek V4 Pro (deepseek-v4-pro) on DeepSeek's own API with the same key, side by side on one machine, interleaved task by task.

v0.0.302.15 changed what graff's default effort sends to DeepSeek; DEFAULT-EFFORT.md measures it against this page's build.

Arm What ran
graff The v0.0.302.14 code (#1445, #1451-#1454) as a scripted graff repl with --model deepseek/deepseek-v4-pro, at graff's default effort.
deepseek-harness @deepseek-ai/dsh 0.2.0-rc.2: dsh --profile headless --json, through graff-evals/dsh_run.py. dsh has no --model flag, so a patch layer pins the model; dsh does not read .mcp.json, so the same layer adds the task's MCP servers as dsh-mcp-client entries. It runs at its own defaults.

Results

The 11 coding tasks and 10 MCP tasks, mean per task over 2 runs of each:

Pass Wall per task Input read from cache Output tokens per task Model calls per task
graff 42/42 22.6s 94% 2,635 4.1
deepseek-harness 42/42 27.7s 87% 2,537 6.3

By suite:

Suite graff deepseek-harness
core (11 coding tasks), wall 6.4s 7.4s
core (11 coding tasks), pass 22/22 22/22
core (11 coding tasks), model calls 3.3 3.5
mcp (10 MCP tasks), wall 40.4s 50.2s
mcp (10 MCP tasks), pass 20/20 20/20
mcp (10 MCP tasks), model calls 5.0 9.2

graff finishes the 21 tasks in 19% less time (22.6s against 27.7s per task). Most of the gap is the MCP tasks (40.4s against 50.2s), where graff makes about half the model calls (5.0 against 9.2): it slims large list results and often fetches, reduces and writes the report in one rlm script. On the coding tasks graff is also faster (6.4s against 7.4s) and reads more of its prompt from cache.

Sub-agents

The subagents suite (five tasks that ask for parallel sub-agents) and its subagents-solo counterfactual (the same work without delegation), 2 runs each:

Suite graff deepseek-harness
subagents, wall 63.7s 70.4s
subagents, pass 9/10 8/10
subagents-solo, wall 27.7s 38.1s
subagents-solo, pass 9/10 9/10

deepseek-harness's sub-agents run in its own process, and its run events report the parent's model calls only, so token and call counts are not compared for these suites.

Per task

Wall time is the mean of 2 runs; a task that failed in a run is marked with its pass count.

Task Suite graff deepseek-harness
csv-sum core 4.6s 6.1s
dead-code core 3.6s 9.3s
exact-reply core 1.5s 2.3s
file-ops core 5.2s 4.8s
fix-fib core 8.1s 9.8s
git-ops core 10.3s 8.5s
json-transform core 4.8s 5.6s
recall-noise core 3.0s 3.1s
refactor-rename core 8.2s 12.9s
regex-count core 6.5s 8.0s
write-tests core 14.3s 10.8s
linear-issues mcp 43.5s 41.1s
linear-nohint mcp 29.2s 29.0s
linear-nohint-warm mcp 22.3s 27.6s
linear-quiet mcp 81.4s 34.6s
linear-quiet-warm mcp 56.1s 98.4s
linear-reduce mcp 37.1s 32.9s
linear-reduce-warm mcp 18.7s 43.7s
linear-sidecar mcp 49.1s 36.6s
linear-split mcp 34.5s 86.6s
linear-warm mcp 31.6s 71.0s
sa-census subagents 79.2s 62.4s
sa-coverage subagents 29.1s (1/2) 37.8s (0/2)
sa-fix-three subagents 45.2s 54.4s
sa-rename-api subagents 77.2s 69.8s
sa-sidecar subagents 87.8s 127.6s
solo-census subagents-solo 20.9s 34.0s
solo-coverage subagents-solo 18.1s (1/2) 21.5s (1/2)
solo-fix-three subagents-solo 28.5s 37.0s
solo-rename-api subagents-solo 34.5s 40.5s
solo-sidecar subagents-solo 36.3s 57.7s

Reproduce

cd graff-evals
export DEEPSEEK_API_KEY=...
python3 run.py --harness graff-deepseek,deepseek-harness --model deepseek-v4-pro --suite core,mcp --reps 2 --interleave
python3 run.py --harness graff-deepseek,deepseek-harness --model deepseek-v4-pro --suite subagents,subagents-solo --reps 2 --interleave