graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Benchmarks
Reproducible head-to-head of graff vs Claude Code vs Codex, behind the numbers in the main README's same-model table.
These are small, honest benchmarks (a few read-only code questions on one machine), not a leaderboard. Run them yourself. Your numbers will vary with the task, the model, and the network.
The 12-PR live harness board (graff / grok / OpenCode / Pi / exo on grok-4.6 SuperGrok) is graff-live-12-report.html plus artifacts/graff-evals-live/RECEIPT.md.
graff against the Codex app server and pi on the same model and ChatGPT account (21 coding and MCP tasks, every run's data) is evals/graff-vs-codex-app-server/RESULTS.md, graff against Claude Code, OpenCode 2 and pi on one model through the Codegraff gateway is evals/graff-vs-claude-code/RESULTS.md (an earlier run without Claude Code: evals/graff-vs-opencode-and-pi/RESULTS.md). The same 21 tasks against the Grok CLI on the same SuperGrok account are evals/graff-vs-grok-cli/RESULTS.md, and what both mean for an engineering budget is evals/enterprise-cost.md.
Prerequisites
graff,claude, andcodexon your PATH, each authenticated.- For
latency.py, graff needs the Codex CLI's sign-in (codex login, read from~/.codex/auth.json) so thatgraff --model codexhits the same ChatGPT endpoint ascodex exec(no gateway).graff login codexnow runs the ChatGPT sign-in for open-source apps (ADR 0229). - macOS for
memory.py(it uses BSD/usr/bin/time -l). - Python 3.
What each script measures
| Script | Measures | Result we got |
|---|---|---|
cost.py |
USD per task (each tool's own usage, gateway prices) | graff/deepseek-v4-pro $0.022 vs Claude Code/Opus-4.8 $0.51 vs Codex/gpt-5.5 $0.42 |
memory.py |
Peak RSS + CPU (/usr/bin/time -l) |
graff ~25 MB floor vs Node ~410 MB vs Rust ~206 MB |
latency.py |
One-shot turn latency, same gpt-5.5 endpoint, concurrent pairs | graff 4.4 s vs codex 8.9 s (one-shot only) |
memory_session.py |
RSS-vs-turn slope plus session/scratch arena capacity for ONE persistent --json session (the #124 leak detector; memory.py is one-shot peak RSS and structurally blind to per-turn growth) |
flat slope after warmup = no per-turn leak |
model_shape.py |
Correctness, quality-diversity, latency, tool/model calls, and routing for configurable Sol-root worker shapes | writes a local correctness-first paired report; no default promotion |
python3 benchmarks/cost.py
python3 benchmarks/memory.py
python3 benchmarks/latency.py 8
python3 benchmarks/memory_session.py 50 # N turns, one persistent session
python3 benchmarks/model_shape.py --graff zig-out/bin/graff --repetitions 1
memory_session.py drives and samples one turn at a time so the child remains
alive at every RSS sample. Set GRAFF_BENCH_MODEL to test a model other than
deepseek-v4-pro; the script enables GRAFF_MEM_DEBUG itself and prints both
arena capacities beside process RSS.
Tasks live in tasks.py (edit to test other work).
Root/worker model shapes
model_shape.py creates disposable synthetic repositories and runs paired arms
with the same gpt-5.6-sol root. Its defaults remain:
all-sol: both direct workers also use Sol;sol-terra: both direct workers usegpt-5.6-terra.
Every run must produce exactly two correctly routed child trajectory nodes. Visible tests provide normal agent feedback, while held-back cases grade the finished workspace afterward. The report selects correctness first and total tool economy second. It also builds a small QD archive whose cells are declared task niches: QD score sums each niche elite's normalized correctness, qualified coverage counts fully correct/valid cells, and tool calls then wall time break fitness ties. Promotion stays manual until at least 20 independent task pairs are present. One repetition is therefore a smoke/effect-size check; seven repetitions across the three core fixtures provide 21 pairs:
python3 benchmarks/model_shape.py \
--graff zig-out/bin/graff \
--repetitions 7 \
--output-root /path/to/results
To compare Sol workers with explicitly consented Kimi K3 workers on the three frontend niches (responsive layout, accessible interaction, and form state):
python3 benchmarks/model_shape.py \
--graff zig-out/bin/graff \
--task-set frontend \
--worker codex:gpt-5.6-sol \
--worker kimi:k3 \
--output-root /path/to/results
The runner disables external telemetry, behavioral upload, fleet sharing, Smolify, and optional companion tools. Only synthetic fixture content reaches the selected provider accounts. Operational traces and the JSON report stay inside the output directory.
Honest caveats (please read)
- Cost is model freedom, not token savings. On the same model, graff and Codex use comparable tokens (graff is leaner on single-tool lookups, heavier on multi-tool tasks because it makes more tool-calling rounds, each re-sending cached context). graff is cheaper because it can run a cheap capable model where Claude Code is Claude-only and Codex is GPT-only.
- Speed is a one-shot win.
latency.pymeasures cold-start-p/execinvocations, where graff's tiny Zig startup beats Codex's heavier per-call launch. In a long interactive session this amortizes and turns are model-bound. - Memory is task-dependent. graff's RSS grows with how much code it reads into
context, so the floor (~25 MB) is the headline, not a fixed number. Growth over
a long session (independent of what is read) is a leak;
memory_session.pymeasures exactly that slope per turn. cost.pyomits graff ongpt-5.5: the codegraff gateway alias is not in graff's local price table, so graff reports$0for it (deepseek/glm price fine).