graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Live A/B: graff-195 on grok-4.6 (2026-09-08)
Sat #195 through both harnesses before scaling to #726. Same SuperGrok
seat. n=3. Results: graff-evals/results/run-20260908-173944.jsonl
(gitignored).
./run.py --suite live --harness graff-dev,grok --task graff-195 --reps 3 -j 1
graff-dev is this tree's zig-out/bin/graff (ReleaseSafe). grok is
grok-build 1.0.21. Model grok-4.6.
Task pass
| harness | pass (reps) | tasks | wall | rss | in | out | calls | list$ |
|---|---|---|---|---|---|---|---|---|
| graff-dev | 3/3 | 1/1 | 606.2s | 11.6M | 237373 | 19257 | 72 | $2.6917 |
| grok | 3/3 | 1/1 | 1430.0s | 128.6M | 393032 | 27419 | 45 | $3.8877 |
Live scoring: task pass is ≥2/3; list$ is passing reps only. Graff
first-token is boot › (0.0–0.33s) — do not score it. Groks's first
token is ~2.6s.
Graff r1–r3 all finished inside timeout_s=600 (timed_out=false).
Grok r1 hit the 600s wall (exit=-9, timed_out=true) with no usage
parse (list$=0); the public+hidden check still passed on the tree it
left. Grok r2/r3 finished clean (382s / 448s).
This is a real exam
Not an auth/harness fail. Both agents edited src/agent_context.zig
and the named #193 test plus the hidden server-meter case went green.
G1–G6 (no model) already passed; this is the model run on that gate.
Do not treat the other eleven published tasks as G1–G6 certified. Scale only because #195 held.