Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
8.2 KB
Raw Blame

Private, correctness-first prompt-policy tournaments

v0.0.215 turns the experimental local learning loop into a resumable, privacy-bounded tournament and adds native remote MCP support. It generates and evaluates competing prompt variants in parallel, ranks correctness before measured tool economy, exposes only the primary winner to the hidden holdout, and connects Streamable HTTP tools without a Node bridge.

Safer evidence and promotion

  • Mutation arms and primary evaluations run concurrently with shared pair seeds and one verified parent baseline.
  • Ranking minimizes critical failures and regressions before considering correctness, tool calls, cost, and prompt size. Latency is reported but does not influence selection.
  • Expensive progress is checkpointed after each barrier and can be resumed without silently repeating completed work.
  • Promotion remains a separate manual action by default. Automatic promotion still requires both configuration and command-line consent, a holdout, and all statistical gates.
  • Adapter programs and declared inputs are hashed, copied into private scratch, and executed from the verified snapshots. Timeouts clean up descendant processes as well as their parent.
  • Inner evaluations now audit a clean, closed behavioral stream and recompute tool calls plus successful-tool-completion scores from paired events. Correctness still ranks first, then tool economy, then behavioral score. Local reports surface the aggregate evidence; protocol/trace disagreement fails the attempt.

Enforceable eval repair loop

  • eval runs as a solo verifier boundary. Red results drop the current plan and stop the model turn; the next turn resumes in repair mode.
  • Completion requires a green verifier result for the latest workspace state. A later mutating tool makes that result stale.
  • Private, atomic task-class eval notes are re-injected every turn so confirmed facts and hypotheses survive compaction. They are not telemetry, but they are sent to the selected model provider as prompt context.

Explicit, revocable aggregate sharing

  • learn run --submit grades and publishes one signed aggregate receipt in the same operation; learn submit RUN_ID retries an existing immutable run.
  • The model-facing learn_candidate tool can request that bundled operation, but cannot promote a candidate. Local mode requires a one-shot interactive confirmation that --yolo, saved approvals, subagents, and noninteractive runs cannot bypass.
  • Receipt payloads have no fields for prompts, tasks, code, paths, adapter output, errors, or traces. Learning submissions also omit stable installation identity, OS, architecture, and service version.
  • Backend receipts expire after 30 days. graff learn delete-remote RUN_ID removes one earlier aggregate receipt using a run-scoped capability derived locally from immutable evidence. Revocation remains available even when telemetry and fleet sharing are disabled.
  • Local genomes, reports, and immutable evidence are never deleted by the remote deletion command.

Root/worker model shapes

  • --subagent-model <name> (or GRAFF_SUBAGENT_MODEL) keeps the root on --model while routing direct subagents, workflow workers/retries, and judges to the pinned model on the same provider. For example, Sol can orchestrate and synthesize while Terra performs parallel delegated work.
  • --subagent-provider <id> can opt into a worker on another service, but only with --allow-cross-provider-subagents. Without that separate consent the process fails before a model call, because worker prompts, code, and tool results cross a credential, billing, and data boundary. The HTTP session API exposes the same explicit provider/model/consent fields.
  • Every split is visible in operational API traces and local child trajectory nodes, including the actual child provider and model. Without a pin, children continue to inherit the root provider/model.
  • The reusable benchmark now supports configurable provider:model arms, three deterministic frontend niches, and a correctness-first quality-diversity archive. Fully correct, correctly routed cells determine coverage; tools and wall time only break correctness ties.

Streamable HTTP MCP and OAuth

  • Remote MCP servers accept JSON or SSE Streamable HTTP responses, retain Mcp-Session-Id, and negotiate the supported MCP protocol version.
  • graff mcp login NAME performs protected-resource and authorization-server discovery, dynamic client registration, browser PKCE login, persistent token storage, and automatic refresh.
  • Authentication tokens live outside the repository with user-only permissions. Static headers remain available for servers that do not use OAuth.
  • Smolify public-read schemas are bundled locally, removing its hosted discovery round trips from startup. The anonymous transport initializes only after an approved call; recognizable credentials in arguments fail closed before egress.
  • In five enabled runs before the change, the Smolify startup phase took 185–1,094 ms. With the bundled catalog it measured 0 ms, matching the disabled path while preserving a successful live first-tool-call handshake.
  • The default catalog excludes identity, private-project, rating, proposal, and publishing tools. GRAFF_SMOLIFY_ACCESS=full explicitly opts into the full OAuth-aware catalog; GRAFF_NO_SMOLIFY=1 removes the surface entirely.
  • First-party Zig modules now obey the repository's 600-line source ceiling. Cohesive prompt, stream-test, HTTP-header, title-job, model-cache-test, and Codegraff OAuth responsibilities were split into sibling modules. CI checks both tracked and newly created sources so the ceiling cannot silently drift.

Reliability fixes

  • Raw terminal reads and mode changes now mask POSIX job-control signals around the syscall. A transient foreground-process-group race therefore returns an input error instead of suspending the whole graff --yolo job with SIGTTIN. YOLO-launched skill installers also receive a null stdin instead of inheriting the controlling terminal.
  • Operational traces remain isolated in exclusively created .graff/traces/<run_id>.jsonl files. CI now runs concurrent tool turns through WS→SSE fallback and rejects NUL bytes, malformed records, mixed run identity, timestamp rollback, or recreation of the retired shared trace path.

Validation

  • Complete Zig test suite and native build
  • Windows cross-build
  • A real PTY job-control regression that fails the pre-fix binary with STOPPED 21 and keeps the fixed YOLO process running
  • Concurrent tool/fallback trace validation with independently parseable, monotonic, fully correlated JSONL
  • End-to-end local learning and hidden-holdout tournament flows
  • Adapter retry/crash recovery and process-tree cleanup
  • Aggregate receipt signature and deletion-token cross-checks
  • Privacy efficacy suite, including an API-key canary that was absent from all 8/8 captured SDK egress cases
  • A live three-task paired root/worker benchmark kept both arms at 27/27 correctness and exact routing. Sol-root/Terra-workers used 29.3% fewer tools and 11.4% less total wall time, but only 1.6% fewer model calls, used 1.4% more summed context tokens, and was 2.5% slower on one task. Three pairs are intentionally under the 20-pair promotion floor, so the result is provisional and cannot change the default.
  • A separate live frontend smoke run compared all-Sol with Sol-root/Kimi-K3 workers. All-Sol won provisionally: its normalized QD score was 0.815 versus 0.667, it completed all 3/3 requested shapes versus K3's 2/3, and it was faster on every pair (K3 used 66.8% more total wall time). K3 did use 58.2% fewer tools, 35.4% fewer API calls, and 50.0% fewer summed context tokens, while matching 9/9 correctness on the form and responsive niches. The accessibility arm timed out after 10 minutes and one responsive call observed and recovered from Kimi engine overload. With only three pairs, K3 remains an explicit experiment rather than an automatic frontend route.

This branch intentionally requires manual promotion until the evidence is adequately powered.

Full changelog: https://github.com/justrach/codegraff/compare/v0.0.214...v0.0.215