Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
2.7 KB
Raw Blame

Terminal-Bench (Harbor) adapter

Runs graff as a Harbor installed agent, so it can be scored on Terminal-Bench and other Harbor datasets beside other agents. Each trial installs the static Linux graff binary in the task container and solves the task as one headless graff -p run with every tool approved. Headless runs wait for their own background work before they end (ADR 0215), so a build or test suite started in the background is collected.

Needs Python 3.12+, uv, and a Harbor environment: Docker, or apple-container on Apple silicon (Apple's container CLI).

uv sync --project graff-evals/harbor

Run

The model is provider/model as graff names it. Harbor passes the provider's key through to graff (ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, XAI_API_KEY, DEEPSEEK_API_KEY, ...). graff has no generic base-URL override, so a custom endpoint needs a graff provider for it.

export ANTHROPIC_API_KEY=...
uv run --project graff-evals/harbor harbor run \
  -d terminal-bench/terminal-bench-2-1 \
  -a graff_harbor.agent:Graff \
  -m anthropic/claude-sonnet-5-5 \
  --ak version=v0.0.302.10 \
  -e docker -n 4 --job-name tb-graff

Agent options (--ak):

option meaning
version Release to install, such as v0.0.302.10. Omit for the latest release. The tarball is checked against the release's SHA256SUMS.
binary A local static Linux graff binary to upload instead, for an unreleased build: zig build graff -Doptimize=ReleaseFast -Dtarget=x86_64-linux (or aarch64-linux).
max_model_calls Stop the run after this many model calls (--max-model-calls).

Each trial's agent/ directory holds graff.stdout (the answer), graff.stderr (progress and the usage footer that fills Harbor's token counts), and graff-state/ (the run's .graff/: session and traces). The .graff/ directory is moved out of the task tree so the verifier sees only the agent's changes. Inspect a job with harbor view jobs/<job-name>.

Tests

uv run --project graff-evals/harbor python -m unittest discover -s graff-evals/harbor/tests
uv run --project graff-evals/harbor python graff-evals/harbor/tests/smoke.py --environment docker

The smoke test runs one real Harbor trial against graff's scripted test model inside the task container (graff's lmstudio port), so it needs no API key. Under apple-container, containers on some hosts cannot resolve names; pass --binary with the matching Linux build so nothing is downloaded inside the container:

uv run --project graff-evals/harbor python graff-evals/harbor/tests/smoke.py \
  --environment apple-container --binary /path/to/graff-aarch64-linux/graff