Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
1.4 KB
Raw Blame

Live A/B: graff-195 on grok-4.6 (2026-09-08)

Sat #195 through both harnesses before scaling to #726. Same SuperGrok seat. n=3. Results: graff-evals/results/run-20260908-173944.jsonl (gitignored).

./run.py --suite live --harness graff-dev,grok --task graff-195 --reps 3 -j 1

graff-dev is this tree's zig-out/bin/graff (ReleaseSafe). grok is grok-build 1.0.21. Model grok-4.6.

Task pass

harness pass (reps) tasks wall rss in out calls list$
graff-dev 3/3 1/1 606.2s 11.6M 237373 19257 72 $2.6917
grok 3/3 1/1 1430.0s 128.6M 393032 27419 45 $3.8877

Live scoring: task pass is ≥2/3; list$ is passing reps only. Graff first-token is boot › (0.0–0.33s) — do not score it. Groks's first token is ~2.6s.

Graff r1–r3 all finished inside timeout_s=600 (timed_out=false). Grok r1 hit the 600s wall (exit=-9, timed_out=true) with no usage parse (list$=0); the public+hidden check still passed on the tree it left. Grok r2/r3 finished clean (382s / 448s).

This is a real exam

Not an auth/harness fail. Both agents edited src/agent_context.zig and the named #193 test plus the hidden server-meter case went green. G1–G6 (no model) already passed; this is the model run on that gate.

Do not treat the other eleven published tasks as G1–G6 certified. Scale only because #195 held.