graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
{"task": "anko-typed-variable-bindings", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh79betfed7ets4an20cr4j57182y9wt-v1.1", "wall_s": 478.0, "agent_exit": 0, "agent_tail": "pes still error (`undefined type` / `unknown type`); zero values used when no initializer\n- Mismatches and invalid nils return `type error` with name, source (`<nil>` for nil), and Go reflect target type (env/envTyped.go:16)\n- `_` is not constraint-checked; a new `var` does not inherit an old constraint\n\nEnable with `&vm.Options{TypedBindings: true}`.\n[usage] 31 api call(s) \u00b7 1853301 in (1609472 cached, 0 cache writes) + 25081 out tokens \u00b7 $0.0000 \u00b7 31 subscription call(s), flat-rate (not in $)\n", "calls": 31, "tok_in": 1853301, "tok_cached": 1609472, "tok_out": 25081, "cost_usd": 0.0, "list_ordinary": 243829, "list_cached": 1609472, "list_out": 25081, "list_prompt": 1853301, "list_tokens": 1878382, "list_token_usd": 2.88576, "list_tool_usd": 0.0, "list_usd": 1.44288, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.8684, "tok_per_call": 59784, "suite": "swe", "patch_tail": "823289 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 823289}
{"task": "arktype-json-schema-refs-dependencies", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh771gpr8crkjsnt9pj81bafgs8229em-v1.1", "wall_s": 931.5, "agent_exit": 0, "agent_tail": "e to resolve $ref \"#/$defs/NonExistentDef\" from root $defs`\n- Implicit `type: \"object\"` when object keywords are present without `type` (`ark/json-schema/json.ts`)\n- `enum` object/array deep equality in `ark/json-schema/common.ts`\n- anyOf `$ref` branches unwrap fully resolved types before composition\n\n`pnpm --filter @ark/json-schema tnt` \u2014 70 passing.\n[usage] 59 api call(s) \u00b7 4154661 in (3811456 cached, 0 cache writes) + 52534 out tokens \u00b7 $0.0000 \u00b7 59 subscription call(s), flat-rate (not in $)\n", "calls": 59, "tok_in": 4154661, "tok_cached": 3811456, "tok_out": 52534, "cost_usd": 0.0, "list_ordinary": 343205, "list_cached": 3811456, "list_out": 52534, "list_prompt": 4154661, "list_tokens": 4207195, "list_token_usd": 5.814684, "list_tool_usd": 0.0, "list_usd": 2.907342, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.9174, "tok_per_call": 70418, "suite": "swe", "patch_tail": "1342898 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 1342898}
{"task": "expr-try-catch-errors", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh71gkadwafw4ry4r6g37era0182qpms", "wall_s": 1094.0, "agent_exit": 0, "agent_tail": " `catch name is \"sub\"` / `finally { }` (`parser/parser.go:367`)\n- `throw(value)`, `retry` (max 3, then `\"retry\"`), `errtype(err)` (`builtin/builtin.go:1082`, `vm/runtime/error.go`)\n- VM try stack + recover loop (`vm/vm.go:773`) classifies index/conversion/type/nil/retry/custom/none\n\nDocs: `docs/language-definition.md`. Tests: `error_handling_test.go`.\n[usage] 65 api call(s) \u00b7 5355809 in (4879232 cached, 0 cache writes) + 54687 out tokens \u00b7 $0.0000 \u00b7 65 subscription call(s), flat-rate (not in $)\n", "calls": 65, "tok_in": 5355809, "tok_cached": 4879232, "tok_out": 54687, "cost_usd": 0.0, "list_ordinary": 476577, "list_cached": 4879232, "list_out": 54687, "list_prompt": 5355809, "list_tokens": 5410496, "list_token_usd": 7.441784, "list_tool_usd": 0.0, "list_usd": 3.720892, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.911, "tok_per_call": 82397, "suite": "swe", "patch_tail": "1360670 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 1360670}
{"task": "fastapi-deprecation-response-headers", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh75azsnb5eqs3mf4xm0zkzha582rvcd-v1.1", "wall_s": 576.7, "agent_exit": 0, "agent_tail": "ne-aware nearest-wins inheritance (`fastapi/routing.py:1501`, `1931`). OpenAPI gets `x-sunset` / `x-deprecation-date` / `x-successor-url` (`fastapi/openapi/utils.py:262`).\n\n`DeprecationTrackingMiddleware` in `fastapi/middleware/deprecation.py:6` counts per-path `deprecated_hits`/`sunset_hits` for HTTP only, with copy `get_stats()` and `reset_stats()`.\n[usage] 41 api call(s) \u00b7 2256279 in (1871616 cached, 0 cache writes) + 31392 out tokens \u00b7 $0.0000 \u00b7 41 subscription call(s), flat-rate (not in $)\n", "calls": 41, "tok_in": 2256279, "tok_cached": 1871616, "tok_out": 31392, "cost_usd": 0.0, "list_ordinary": 384663, "list_cached": 1871616, "list_out": 31392, "list_prompt": 2256279, "list_tokens": 2287671, "list_token_usd": 3.786972, "list_tool_usd": 0.0, "list_usd": 1.893486, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.8295, "tok_per_call": 55031, "suite": "swe", "patch_tail": "930196 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 930196}
{"task": "httpx-multipart-response-parsing", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh7fr6f1yw1289ye5ttsfvaqen8319dq", "wall_s": 695.7, "agent_exit": 0, "agent_tail": "eader folding/duplicates, framing errors \u2192 `DecodingError`.\n- Wired on `Response` in `httpx/_models.py` (`iter_multipart` 940, `aiter_multipart` 1063). In-memory `_content` is repeatable; a live stream is consumed via `iter_bytes`/`aiter_bytes` and closed (`StreamConsumed` on a second pass).\n- Exported as `httpx.MultipartPart` (`httpx/__init__.py` 65).\n[usage] 23 api call(s) \u00b7 1079379 in (868096 cached, 0 cache writes) + 40431 out tokens \u00b7 $0.0000 \u00b7 23 subscription call(s), flat-rate (not in $)\n", "calls": 23, "tok_in": 1079379, "tok_cached": 868096, "tok_out": 40431, "cost_usd": 0.0, "list_ordinary": 211283, "list_cached": 868096, "list_out": 40431, "list_prompt": 1079379, "list_tokens": 1119810, "list_token_usd": 2.1984, "list_tool_usd": 0.0, "list_usd": 1.0992, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.8043, "tok_per_call": 46930, "suite": "swe", "patch_tail": "766568 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 766568}
{"task": "katex-multicolumn-array-spans", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh7fkjmy67qny9z98mt6bj52m182ygdy-v1.1", "wall_s": 921.1, "agent_exit": 0, "agent_tail": "bmatrix/Bmatrix/vmatrix/Vmatrix/cases/rcases/aligned/smallmatrix.\n- HTML: per-row vertical rules, internal seps in a span suppressed (`hasColspan` at 452). Alignment override via cell wrapper.\n- MathML: `columnspan` and `columnalign` on `<mtd>` (747).\n- Docs updated in `docs/support_table.md` and `docs/supported.md`.\n\nExisting `katex-spec` tests pass.\n[usage] 71 api call(s) \u00b7 5181515 in (4750720 cached, 0 cache writes) + 42736 out tokens \u00b7 $0.0000 \u00b7 71 subscription call(s), flat-rate (not in $)\n", "calls": 71, "tok_in": 5181515, "tok_cached": 4750720, "tok_out": 42736, "cost_usd": 0.0, "list_ordinary": 430795, "list_cached": 4750720, "list_out": 42736, "list_prompt": 5181515, "list_tokens": 5224251, "list_token_usd": 6.986732, "list_tool_usd": 0.0, "list_usd": 3.493366, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.9169, "tok_per_call": 72979, "suite": "swe", "patch_tail": "1065002 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 1065002}
{"task": "meriyah-explicit-resource-declarations", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh7398skqnxqwg9hbmdj7ncmk1822aa0-v1.1", "wall_s": 1161.2, "agent_exit": 0, "agent_tail": "d\n\u203a\n [main] \u21af 1 failed\n\u203a\n [main] \u21af 1 failed\n\u203a\n [main] \u21af 1 failed\n\u203a\n [main] \u21af 2 failed\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n [main] \u21af 1 failed\n\u203a\n\u203a\n\u203a\n [main] \u21af 1 failed\n\u203a\n\u203a\n [main] \u21af 1 failed\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n [main] \u21af 1 failed\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n [main] \u21af 1 failed\n\u203a\n\u203a\n\u203a\n\u203a\n [main] \u21af 1 failed\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n\u203a\n[usage] 88 api call(s) \u00b7 8701525 in (8391552 cached, 0 cache writes) + 49036 out tokens \u00b7 $0.0000 \u00b7 88 subscription call(s), flat-rate (not in $)\n", "calls": 88, "tok_in": 8701525, "tok_cached": 8391552, "tok_out": 49036, "cost_usd": 0.0, "list_ordinary": 309973, "list_cached": 8391552, "list_out": 49036, "list_prompt": 8701525, "list_tokens": 8750561, "list_token_usd": 10.219876, "list_tool_usd": 0.0, "list_usd": 5.109938, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.9644, "tok_per_call": 98881, "suite": "swe", "patch_tail": "1529328 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 1529328}
{"task": "python-statemachine-state-data-scoping", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh719np6e210pf8skawv132rw183pfyy", "wall_s": 1063.4, "agent_exit": 0, "agent_tail": "py:22`, `73`)\n- Runtime API: `get_state_data`, `set_state_data`, `state_data_values`, `get_data_changes` (`statemachine/statemachine.py:471-505`); `state_data` injected into callbacks\n- History restores descendant snapshots (deep/shallow); data survives pickle\n- Per-state SCXML `<data id expr>` parsed as Python literals; diagrams annotate `data:` vars\n[usage] 63 api call(s) \u00b7 5470486 in (4971008 cached, 0 cache writes) + 49526 out tokens \u00b7 $0.0000 \u00b7 63 subscription call(s), flat-rate (not in $)\n", "calls": 63, "tok_in": 5470486, "tok_cached": 4971008, "tok_out": 49526, "cost_usd": 0.0, "list_ordinary": 499478, "list_cached": 4971008, "list_out": 49526, "list_prompt": 5470486, "list_tokens": 5520012, "list_token_usd": 7.563232, "list_tool_usd": 0.0, "list_usd": 3.781616, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.9087, "tok_per_call": 86833, "suite": "swe", "patch_tail": "1295829 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 1295829}
{"task": "scc-bounded-memory-spilling", "harness": "graff", "model": "grok-4.6", "image": "public.ecr.aws/d3j8x8q7/swe-bench-202605:kh71j0gr2axgn6v1f92cam6s6x82rkv6-v1.1", "wall_s": 794.8, "agent_exit": 0, "agent_tail": "\n- `--bounded-memory-dir`\n- `--bounded-memory-max-in-memory-files`\n- `--bounded-memory-stats`\n\nSpill dir is created if missing, excluded from scans, and left in place. csv-stream honors file destinations and sort order; json/json2/csv matched unbounded output in self-checks. Stats: one stderr line `bounded-memory: spills=<N> peak_in_memory_files=<M>`.\n[usage] 45 api call(s) \u00b7 2718332 in (2386432 cached, 0 cache writes) + 42642 out tokens \u00b7 $0.0000 \u00b7 45 subscription call(s), flat-rate (not in $)\n", "calls": 45, "tok_in": 2718332, "tok_cached": 2386432, "tok_out": 42642, "cost_usd": 0.0, "list_ordinary": 331900, "list_cached": 2386432, "list_out": 42642, "list_prompt": 2718332, "list_tokens": 2760974, "list_token_usd": 4.225736, "list_tool_usd": 0.0, "list_usd": 2.112868, "list_price_kind": "xai-list", "list_high_band": false, "cache_hit": 0.8779, "tok_per_call": 60407, "suite": "swe", "patch_tail": "935461 /logs/artifacts/model.patch\n", "verified": false, "pass_": null, "verify_tail": "DeepSWE grader hidden; patch collected, not scored", "patch_bytes": 935461}