Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python
5.8 KB
Raw Blame

Formal checks in the evolution loop

Formal evidence is an admission condition, not a fitness reward. Passing a model check must never compensate for failed task correctness, and neither formal success nor a smaller prompt establishes a performance improvement.

Prompt evolution and source evolution

examples/dgm_loop.py evolves prompt genomes. Its correctness gate is the external held-out replay judge. Its current passing score ranks tool economy: total / (total + tool_calls). Formal checks validate the pinned harness baseline beneath those prompt candidates; they do not prove prompt behavior. Keep the reward formula unchanged when adding the gate.

For harness-source evolution, each candidate instead needs an immutable source identity, a reviewed mapping between its lifecycle changes and formal actions, model checks, and runtime regressions. Hashing a source tree and an executable records their identities; it does not prove the executable was built from that source. A reproducible build or attested build receipt supplies that separate link.

Evaluation flow

flowchart TD
    A[Pin evaluator, models, source and executable] --> B[Run formal baseline checks]
    B -->|Fail or unavailable| X[Stop formal-enabled evaluation]
    B -->|Pass| C[Generate prompt candidate]
    C --> D[Revalidate pins and bind candidate receipt]
    D --> E[Run held-out replay tests]
    E -->|Any failure| F[Keep below passing score band]
    E -->|All pass| G[Compute existing efficiency score]
    G --> H[Revalidate inputs before score write]
    H --> I[Archive signed evidence digest and score]

The same checked baseline can be reused for multiple prompts while every pinned input remains unchanged. Input drift invalidates admission; it must not silently trigger a new baseline or allow the candidate to rewrite its own checker. Keep evaluator copies and pin files outside the editable task workspace. These integrity checks do not replace OS isolation for adversarial evolution.

A receipt should distinguish source/model identity, finite checked properties, checker result, candidate prompt identity, held-out suite identity and executable identity. Preserve the original replay suite hash and bind the formal receipt into score provenance. Retain formal failure separately from replay failure, unknown measurement and an ordinary low efficiency score.

Python example opt-in

Create a new verifier directory outside the candidate checkout using examples/dgm_formal_gate.py --pin, supplying the harness executable, Java, TLC archive, and explicit mutation and replay model selectors. Set GRAFF_DGM_FORMAL_PIN to the resulting pin file and GRAFF_SCORE_KEY_FILE to the existing signing key before running examples/dgm_loop.py. The helper checks the pinned formal bundle once per unchanged identity. Normal mode remains unchanged when no formal pin is configured.

Mutation and replay roles may intentionally use different model selectors; record both rather than silently conflating them. Selectors and binary hashes are not proof of the provider route actually served during inference. Historical scores remain possible parents, but do not become formally checked ancestry. Only new evaluations admitted through formal mode carry its evidence.

python3 examples/dgm_formal_gate.py --pin "$PIN_DIR" \
  --binary "$HARNESS_BINARY" --java "$JAVA" --jar "$TLA2TOOLS" \
  --main-model "$MAIN_MODEL" --replay-model "$REPLAY_MODEL"
GRAFF_DGM_FORMAL_PIN="$PIN_DIR/pin.json" \
  GRAFF_SCORE_KEY_FILE="$SCORE_KEY_FILE" \
  python3 examples/dgm_loop.py "Your evaluation task" 3

The pin directory must be new and outside the source checkout. Set the model selectors explicitly for the mutation and replay roles. Receipts and bounded checker logs stay private. To audit an archived signed score, set the same signing-key environment variable and run:

python3 examples/dgm_formal_gate.py --verify-row "$SCORE_ROW_JSON" \
  --pin-file "$PIN_DIR/pin.json"

Formal score rows resolve to score-<artifact digest>.json receipts. Validation checks the signature, prompt/report hashes, original held-out hash, preflight receipt, pinned identity and archived checker evidence. It also runs the current pinned model checks; their raw output need not match an earlier successful log.

Native learning and remaining promotion paths

Native graff learn has a separate opt-in formal_check configuration; see local learning for setup and its trust boundary. src/learn_formal.zig checks the pinned baseline before mutation and binds evidence to the parent or selected prompt. The existing tournament and holdout rules still decide candidate eligibility.

Formal-enabled runs use pending-record version 2 and run-record version 4. Resume and both manual and automatic promotion recheck the stored evidence and pinned identity. Legacy records remain readable but cannot satisfy formal-mode promotion. The manifest's executable must also be a pinned input and argument to both learning adapters; this records the adapter contract, not a proof that an arbitrary adapter executes that argument.

Native formal-enabled runs stay local: explicit submission is rejected, and automatic learning omits contribution. The signed aggregate receipt format does not yet bind formal evidence. The separate src/fleet.zig persona archive promotion path remains ungated. Historical scores must not be presented as formally checked merely because a newer evaluation enables this option.

Performance evidence remains separate

Wall time, input/output tokens, cached input, cache writes and cost need paired runtime measurements with preserved failures and per-metric completeness. Unknown usage is not zero, and subscription usage is not a billed-dollar receipt. The formal gate does not change these measurement requirements or establish that a candidate is faster, cheaper, or more capable.