graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
evals: graff vs deepseek-harness on DeepSeek V4 Pro
justrachc251ce3159ec
diff --git a/evals/graff-vs-deepseek-harness/RESULTS.md b/evals/graff-vs-deepseek-harness/RESULTS.mdnew file mode 100644index 00000000..5dd72355--- /dev/null+++ b/evals/graff-vs-deepseek-harness/RESULTS.md@@ -0,0 +1,91 @@+# graff vs deepseek-harness on DeepSeek V4 Pro++Measured 2026-10-02. Both harnesses ran DeepSeek V4 Pro (`deepseek-v4-pro`) on DeepSeek's own API with the same key, side by side on one machine, interleaved task by task.++| Arm | What ran |+| --- | --- |+| graff | The v0.0.302.14 code (#1445, #1451-#1454) as a scripted `graff repl` with `--model deepseek/deepseek-v4-pro`, at graff's default effort. |+| deepseek-harness | `@deepseek-ai/dsh` 0.2.0-rc.2: `dsh --profile headless --json`, through [`graff-evals/dsh_run.py`](../../graff-evals/dsh_run.py). dsh has no `--model` flag, so a patch layer pins the model; dsh does not read `.mcp.json`, so the same layer adds the task's MCP servers as `dsh-mcp-client` entries. It runs at its own defaults. |++## Results++The 11 coding tasks and 10 MCP tasks, mean per task over 2 runs of each:++| | Pass | Wall per task | Input read from cache | Output tokens per task | Model calls per task |+| --- | ---: | ---: | ---: | ---: | ---: |+| graff | 42/42 | 22.6s | 94% | 2,635 | 4.1 |+| deepseek-harness | 42/42 | 27.7s | 87% | 2,537 | 6.3 |++By suite:++| Suite | graff | deepseek-harness |+| --- | ---: | ---: |+| core (11 coding tasks), wall | 6.4s | 7.4s |+| core (11 coding tasks), pass | 22/22 | 22/22 |+| core (11 coding tasks), model calls | 3.3 | 3.5 |+| mcp (10 MCP tasks), wall | 40.4s | 50.2s |+| mcp (10 MCP tasks), pass | 20/20 | 20/20 |+| mcp (10 MCP tasks), model calls | 5.0 | 9.2 |++graff finishes the 21 tasks in 19% less time (22.6s against 27.7s per task). Most of the gap is the MCP tasks (40.4s against 50.2s), where graff makes about half the model calls (5.0 against 9.2): it slims large list results and often fetches, reduces and writes the report in one `rlm` script. On the coding tasks graff is also faster (6.4s against 7.4s) and reads more of its prompt from cache.++## Sub-agents++The `subagents` suite (five tasks that ask for parallel sub-agents) and its `subagents-solo` counterfactual (the same work without delegation), 2 runs each:++| Suite | graff | deepseek-harness |+| --- | ---: | ---: |+| subagents, wall | 63.7s | 70.4s |+| subagents, pass | 9/10 | 8/10 |+| subagents-solo, wall | 27.7s | 38.1s |+| subagents-solo, pass | 9/10 | 9/10 |++deepseek-harness's sub-agents run in its own process, and its run events report the parent's model calls only, so token and call counts are not compared for these suites.++## Per task++Wall time is the mean of 2 runs; a task that failed in a run is marked with its pass count.++| Task | Suite | graff | deepseek-harness |+| --- | --- | ---: | ---: |+| [`csv-sum`](../../graff-evals/tasks/11-csv-sum.json) | core | 4.6s | 6.1s |+| [`dead-code`](../../graff-evals/tasks/12-dead-code.json) | core | 3.6s | 9.3s |+| [`exact-reply`](../../graff-evals/tasks/01-exact-reply.json) | core | 1.5s | 2.3s |+| [`file-ops`](../../graff-evals/tasks/03-file-ops.json) | core | 5.2s | 4.8s |+| [`fix-fib`](../../graff-evals/tasks/02-fix-fib.json) | core | 8.1s | 9.8s |+| [`git-ops`](../../graff-evals/tasks/08-git-ops.json) | core | 10.3s | 8.5s |+| [`json-transform`](../../graff-evals/tasks/04-json-transform.json) | core | 4.8s | 5.6s |+| [`recall-noise`](../../graff-evals/tasks/09-recall-noise.json) | core | 3.0s | 3.1s |+| [`refactor-rename`](../../graff-evals/tasks/06-refactor-rename.json) | core | 8.2s | 12.9s |+| [`regex-count`](../../graff-evals/tasks/05-regex-count.json) | core | 6.5s | 8.0s |+| [`write-tests`](../../graff-evals/tasks/07-write-tests.json) | core | 14.3s | 10.8s |+| [`linear-issues`](../../graff-evals/tasks/24-linear-issues.json) | mcp | 43.5s | 41.1s |+| [`linear-nohint`](../../graff-evals/tasks/30-linear-nohint.json) | mcp | 29.2s | 29.0s |+| [`linear-nohint-warm`](../../graff-evals/tasks/31-linear-nohint-warm.json) | mcp | 22.3s | 27.6s |+| [`linear-quiet`](../../graff-evals/tasks/28-linear-quiet.json) | mcp | 81.4s | 34.6s |+| [`linear-quiet-warm`](../../graff-evals/tasks/29-linear-quiet-warm.json) | mcp | 56.1s | 98.4s |+| [`linear-reduce`](../../graff-evals/tasks/32-linear-reduce.json) | mcp | 37.1s | 32.9s |+| [`linear-reduce-warm`](../../graff-evals/tasks/33-linear-reduce-warm.json) | mcp | 18.7s | 43.7s |+| [`linear-sidecar`](../../graff-evals/tasks/26-linear-sidecar.json) | mcp | 49.1s | 36.6s |+| [`linear-split`](../../graff-evals/tasks/27-linear-split.json) | mcp | 34.5s | 86.6s |+| [`linear-warm`](../../graff-evals/tasks/25-linear-warm.json) | mcp | 31.6s | 71.0s |+| [`sa-census`](../../graff-evals/tasks/63-sa-census.json) | subagents | 79.2s | 62.4s |+| [`sa-coverage`](../../graff-evals/tasks/65-sa-coverage.json) | subagents | 29.1s (1/2) | 37.8s (0/2) |+| [`sa-fix-three`](../../graff-evals/tasks/62-sa-fix-three.json) | subagents | 45.2s | 54.4s |+| [`sa-rename-api`](../../graff-evals/tasks/64-sa-rename-api.json) | subagents | 77.2s | 69.8s |+| [`sa-sidecar`](../../graff-evals/tasks/66-sa-sidecar.json) | subagents | 87.8s | 127.6s |+| [`solo-census`](../../graff-evals/tasks/68-solo-census.json) | subagents-solo | 20.9s | 34.0s |+| [`solo-coverage`](../../graff-evals/tasks/70-solo-coverage.json) | subagents-solo | 18.1s (1/2) | 21.5s (1/2) |+| [`solo-fix-three`](../../graff-evals/tasks/67-solo-fix-three.json) | subagents-solo | 28.5s | 37.0s |+| [`solo-rename-api`](../../graff-evals/tasks/69-solo-rename-api.json) | subagents-solo | 34.5s | 40.5s |+| [`solo-sidecar`](../../graff-evals/tasks/71-solo-sidecar.json) | subagents-solo | 36.3s | 57.7s |++## Reproduce++```sh+cd graff-evals+export DEEPSEEK_API_KEY=...+python3 run.py --harness graff-deepseek,deepseek-harness --model deepseek-v4-pro --suite core,mcp --reps 2 --interleave+python3 run.py --harness graff-deepseek,deepseek-harness --model deepseek-v4-pro --suite subagents,subagents-solo --reps 2 --interleave+```+diff --git a/graff-evals/dsh_run.py b/graff-evals/dsh_run.pynew file mode 100755index 00000000..50ea34f4--- /dev/null+++ b/graff-evals/dsh_run.py@@ -0,0 +1,61 @@+#!/usr/bin/env python3+"""Run deepseek-harness (dsh) headless on one eval task and report like graff does.++usage: dsh_run.py [PROVIDER/]MODEL PROMPT (cwd = the task sandbox)++dsh has no --model flag, so a patch layer pins `agent-default-model`+(provider defaults to deepseek-official). dsh does not read `.mcp.json`; each+server there becomes a `@deepseek-ai/dsh-mcp-client` entry in the same patch.+The final message goes to stdout; a graff-format `[usage]` line goes to+stderr, summed over the run's step_end events. stdin is empty.+"""+import json, os, subprocess, sys, tempfile++DSH = os.environ.get("DSH_BIN", "dsh")+model, prompt = sys.argv[1], sys.argv[2]+provider, _, name = model.rpartition("/")+cwd = os.getcwd()++patch = [{"id": "agent-default-model", "config": {"provider": provider or "deepseek-official", "model": name}}]+if os.path.exists(".mcp.json"):+ for server, spec in json.load(open(".mcp.json")).get("mcpServers", {}).items():+ args = [os.path.join(cwd, a) if os.path.exists(os.path.join(cwd, a)) else a for a in spec.get("args", [])]+ cfg = {"serverName": server, "transport": "stdio", "command": spec["command"], "args": args}+ if spec.get("env"):+ cfg["env"] = spec["env"]+ patch.append({"id": f"mcp-{server}", "name": "@deepseek-ai/dsh-mcp-client", "config": cfg})+fd, path = tempfile.mkstemp(prefix="dsh-patch-", suffix=".yml")+os.write(fd, json.dumps(patch).encode()) # JSON is valid YAML+os.close(fd)+try:+ r = subprocess.run([DSH, "--profile", "headless", "--patch", path, "--json", prompt],+ stdin=subprocess.DEVNULL, capture_output=True, text=True)+finally:+ os.unlink(path)++final, texts = None, []+calls = tin = tout = tread = twrite = 0+for line in r.stdout.splitlines():+ try:+ ev = json.loads(line)+ except ValueError:+ continue+ kind = ev.get("type")+ if kind == "final":+ final = ev.get("text")+ elif kind == "text":+ texts.append(ev.get("text", ""))+ elif kind == "status" and ev.get("phase") == "step_end":+ u = ev.get("usage") or {}+ calls += 1+ tin += u.get("inputTokens") or 0+ tout += u.get("outputTokens") or 0+ tread += u.get("cacheReadTokens") or 0+ twrite += u.get("cacheWriteTokens") or 0+print(final if final is not None else (texts[-1] if texts else ""))+sys.stderr.write(r.stderr)+# pi-ai reports input excluding cache reads/writes; add them back for total prompt tokens.+sys.stderr.write(f"\n[usage] {calls} api call(s) · {tin + tread + twrite} in ({tread} cached, {twrite} cache writes) + {tout} out tokens\n")+with open(".eval-dsh-events.jsonl", "w") as f:+ f.write(r.stdout)+sys.exit(r.returncode)diff --git a/graff-evals/harnesses.json b/graff-evals/harnesses.jsonindex 96da5052..b70122b4 100644--- a/graff-evals/harnesses.json+++ b/graff-evals/harnesses.json@@ -374,5 +374,7 @@"usage": "graff-stderr","env": {"DISABLE_TELEMETRY": "1", "DISABLE_AUTOUPDATER": "1", "DISABLE_ERROR_REPORTING": "1", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"},"note": "Claude Code (`claude -p`, stream-json) with the task's .mcp.json as its only MCP config. Set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN for the endpoint; a chat-completions-only endpoint needs an Anthropic-format proxy in front (evals/graff-vs-claude-code). Pass --model."- }+ },+ "graff-deepseek": {"cmd": ["{repo}/zig-out/bin/graff", "repl", "--model", "{model}", "--yolo"], "stdin": "\u001b[200~{prompt}\u001b[201~\n", "answer": "stdout", "usage": "graff-stderr", "env": {"GRAFF_POST_DEADLINE_SECS": "90", "GRAFF_NO_TELEMETRY": "1"}, "model_prefix": "deepseek", "default_model": "deepseek-v4-pro", "note": "scripted graff repl on DeepSeek's own API (DEEPSEEK_API_KEY)"},+ "deepseek-harness": {"cmd": ["python3", "{repo}/graff-evals/dsh_run.py", "{model}", "{prompt}"], "answer": "stdout", "usage": "graff-stderr", "env": {"DSH_PERMISSION_MODE": "danger-full-access"}, "default_model": "deepseek-v4-pro", "note": "deepseek-harness (@deepseek-ai/dsh, DSH_BIN when dsh is not on PATH) headless on DeepSeek's own API; the model and the task's MCP servers go in a patch layer"}}