Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
3.7 KB
Raw Blame

The self-correcting release: the harness measured itself, then fixed what it found

v0.0.237 is one closed loop, run end to end in a day: eval the harness, mine the trajectories, redesign what lost, verify the redesign with the same evals — and ship the machinery that makes the next loop automatic.

Ultracode: smallest rung that fits

The old ultracode was measured against a single agent on five deterministic evals and lost: 80 mean score at 3.7× the calls, three budget deaths, and a bugfix run that spent 30 calls orchestrating without ever editing the file. It was redesigned around an escalation ladder informed by how Codex, OpenCode, and grok-build gate delegation:

  • The codeword now means "escalate to the smallest rung that fits": solo for 1–2 known files, one scout when exploration would flood context, a fleet only for 3+ independent workstreams or after a failed attempt, judges only for audit-class asks.
  • An admission gate and a budget-reservation ledger enforce it: duplicate briefs collapse at spawn, implement phases carry an edit contract (no diff = error + retry), and a run that can't afford its later phases truncates gracefully instead of dying mid-flight.
  • Re-measured on the same five evals: 100 mean at 1.22× baseline calls, zero budget deaths — and 72% fewer uncached input tokens and 78% less output than the old ultracode. A genuinely large audit task still fans out (4 workers) and lands its answer from partial evidence.
  • Every rung decision writes an orch trajectory row keyed by task class, budget band, and model stratum — the hand ladder is the bootstrap; the fleet learns the policy from its own outcomes.

The trajectory failures, closed

From two production post-mortems (a design-fleet session and its failed sequel):

  • Constraints survive now (#381): /never <text> — or the model calling the append-only note_constraint tool when you reject something — lands in .graff/playbook.jsonl; a HARD CONSTRAINTS block derived from the file at brief-assembly time rides every worker brief and the root prompt, immune to compaction and session boundaries. Measured: 0/4 obeyed before, 4/4 after.
  • Reskin fleets get caught (#382): 3+ near-identical variant briefs draw a calibrated similarity warning on both fan-out paths; the new concept-fleet shape demands a design thesis and one signature system per variant.
  • Image tasks route to eyes (#380): a task naming an image re-seats automatic workers onto a vision-capable model (source=vision-ask), and a worker that admits it couldn't view the image gets flagged so inference can never masquerade as observation.
  • The harness learns from its runs (#383): target-met evals distill up to three provenance-stamped insight bullets, curated by deterministic code, injected as an advisory block below user constraints.

The five-flake-a-day CI bug, root-caused

The ubuntu pty "shutdown hang" was never a shutdown hang: a Ctrl-D landing in a canonical-mode tty window is delivered by Linux as NUL, and readline dropped it — the REPL then waited at the prompt, correctly, forever. Type-ahead bytes are now normalized across the mode switch (15/15 container hangs → 25/25 passes), and teardown stamps 12 phases under GRAFF_SHUTDOWN_DEBUG so any future hang names its stalling phase (#364).

Filed for what's next

  • #385: stage the image bytes into vision-routed workers (routing landed; sight is next). #390: reserve the final narration call on tight budgets.
  • #384: weakness mining — automate the trajectory post-mortem that produced this release. #387–#389: usage-footer and gateway-error robustness found by this release's own eval studies.

Full Changelog: https://github.com/justrach/codegraff/compare/v0.0.236...v0.0.237