Skip to content

graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs

Python

Gate local learning with pinned formal baseline

Rach Pradhan85484c876737
diff --git a/build.zig b/build.zig
index e90eee0c..42bb594d 100644
--- a/build.zig
+++ b/build.zig
@@ -155,6 +155,9 @@ pub fn build(b: *std.Build) void {
const subagent_resume_test = b.addSystemCommand(&.{ "python3", "scripts/test-subagent-resume.py" });
subagent_resume_test.addArtifactArg(exe);
test_step.dependOn(&subagent_resume_test.step);
+ const learn_formal_test = b.addSystemCommand(&.{ "python3", "tests/learn_formal_e2e.py", "--graff" });
+ learn_formal_test.addArtifactArg(exe);
+ test_step.dependOn(&learn_formal_test.step);
test_step.dependOn(&b.addSystemCommand(&.{ "python3", "graff-evals/test_measurement.py" }).step);
test_step.dependOn(&b.addSystemCommand(&.{ "python3", "scripts/test-eval-cache-affinity.py" }).step);
diff --git a/docs/adr/0194-native-learning-formal-baseline.md b/docs/adr/0194-native-learning-formal-baseline.md
new file mode 100644
index 00000000..2b9932f3
--- /dev/null
+++ b/docs/adr/0194-native-learning-formal-baseline.md
@@ -0,0 +1,36 @@
+# 0194. Native learning checks a pinned formal baseline
+
+Status: accepted 2026-09-23
+
+## Context
+
+The local learning store already pins its adapters and evaluation suites, and
+rechecks their immutable evidence before promotion. A separate, bounded formal
+checker can establish properties of the shared engine's modeled control flow.
+Its success is not evidence that a newly generated prompt is correct. The
+existing signed aggregate receipt also has no field for a formal check.
+
+## Decision
+
+An optional `formal_check` configuration pins the checker executable, its
+script, and its input manifest. The manifest's engine binary must be an exact
+pinned input and argument to both learning adapters. Before mutation, and
+again for a selected genome, the checker must succeed for the respective
+prompt fingerprint. The store saves content-addressed check evidence in a
+versioned pending record and run record. Resume and manual or automatic
+promotion re-run the pinned checker and reject missing, stale, or mismatched
+evidence. Default configurations do not enable this gate.
+
+Formal-enabled trials stay local. `run --submit`, `learn submit`, and direct
+submission reject them until a signed receipt version binds the formal
+evidence. The legacy fleet persona promotion path remains separate and has no
+formal gate.
+
+## Consequences
+
+The checker consumes local time at admission, selection, resume, and
+promotion. Operators must supply its signing key file and trust the pinned
+checker, adapters, and manifest; pinning proves their byte identity, not that
+an adapter actually uses its binary argument. The model-check result concerns
+the pinned shared engine baseline. Behavioral correctness of a candidate
+prompt still depends on the normal paired evaluations and holdout gate.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index b0467a25..5c4a157b 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -219,6 +219,7 @@ record only when you need the evidence or the edge cases.
| [0191](0191-acp-effort-is-session-configuration.md) | ACP thought-level options follow the active model and share `/effort` state; changes received during a turn apply after that turn. |
| [0192](0192-mimo-thinking-is-a-binary-setting.md) | MiMo exposes Off/On, preserves legacy positive settings as On, and uses the documented thinking control on each wire. |
| [0193](0193-jev-selects-effort-only.md) | Optional Jev selects only supported session effort at the next request boundary; arbitrary judgments are not exposed. |
+| [0194](0194-native-learning-formal-baseline.md) | Optional native learning admits and promotes only with a fresh pinned formal baseline check bound to the evaluated binary; remote submission waits for a compatible signed receipt. |
| [0195](0195-embedded-browser-replaces-kuri.md) | Desktop browser uses embedded Chromium, web preview uses the paired extension, and CLI fetch uses built-in HTTP without a sidecar. |
| [0198](0198-project-choice-survives-session-isolation.md) | New chats inherit the chosen project while ACP reports each worker checkout; generated worktrees are rooted at the main checkout. |
| [0199](0199-generated-checkouts-belong-to-project-history.md) | Generated worktree activity rolls up to the owning project in automatic suggestions; explicitly saved checkouts stay selectable. |
diff --git a/docs/local-learning.md b/docs/local-learning.md
index a0e8647e..c2f608a6 100644
--- a/docs/local-learning.md
+++ b/docs/local-learning.md
@@ -316,6 +316,43 @@ as immutable. Unknown JSON fields are rejected.
Cohort fields are exact labels used to prevent unlike evaluations from being
compared as though they were interchangeable.
+### Optional pinned formal baseline
+
+An operator can add `formal_check` to an explicit configuration to require a
+local model-check result before mutation, after selecting a candidate, and on
+checkpoint resume or promotion. It is disabled by default. The checker uses
+the read-only `--check` interface described in
+[the formal harness](../formal/README.md), with a pinned manifest and helper:
+
+```json
+"formal_check": {
+ "checker": {
+ "program": "/absolute/python3",
+ "sha256": "<Python executable SHA-256>",
+ "args": ["/absolute/dgm_formal_gate.py"],
+ "inputs": [{"path": "/absolute/dgm_formal_gate.py", "sha256": "<helper SHA-256>"}]
+ },
+ "pin": {"path": "/absolute/formal-pin.json", "sha256": "<manifest SHA-256>"},
+ "timeout_ms": 900000
+}
+```
+
+The manifest's `binary.path` and SHA-256 must match a pinned input passed as
+an argument to both the mutator and evaluator. The normal adapter runner
+copies those verified bytes into private scratch space. Set
+`GRAFF_SCORE_KEY_FILE` to an absolute path for the helper's pinned signing-key
+identity check; a missing key fails the trial before mutation. The checker and
+manifest are operator trust inputs, and the formal result describes the
+shared engine model at that pinned binary, not the candidate prompt's behavior.
+
+Formal trials write `pending.v2` and `run.v4` records with immutable evidence
+IDs. Resume and promotion reject mismatches and re-run the checker. This adds
+local compute time and does not change the statistical or holdout gates.
+Formal trials cannot use `run --submit` or `learn submit` because the current
+signed receipt does not bind formal evidence; background auto trials remain
+local even when aggregate contribution is enabled. Legacy fleet persona
+promotion is a separate path and is not covered by this gate.
+
### Pinned program snapshots
The engine does not hash a tool and then reopen that original pathname for
diff --git a/src/learn_auto.zig b/src/learn_auto.zig
index fd4e928f..d3a704ad 100644
--- a/src/learn_auto.zig
+++ b/src/learn_auto.zig
@@ -228,7 +228,10 @@ pub fn maybeStart(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.
// Contribution needs a signing key as well as consent. Asking for
// `--submit` without one fails the whole trial in preflight, so a machine
// that cannot sign simply learns locally.
- const contribute = options.contribute and scoring.loadScoreKey(io, arena, environ) != null;
+ // Formal evidence is not represented by the current signed remote receipt.
+ // Keep this trial local even when ordinary aggregate contribution is on.
+ const contribute = config.value.formal_check == null and options.contribute and
+ scoring.loadScoreKey(io, arena, environ) != null;
const resumed = hasPendingTrial(&store);
const exe_path = std.process.executablePathAlloc(io, gpa) catch return .{ .skipped = .unconfigured };
defer gpa.free(exe_path);
diff --git a/src/learn_checkpoint.zig b/src/learn_checkpoint.zig
index 1a769f3d..144197ea 100644
--- a/src/learn_checkpoint.zig
+++ b/src/learn_checkpoint.zig
@@ -10,6 +10,7 @@ const eval = @import("learn_eval.zig");
const store_mod = @import("learn_store.zig");
pub const schema = "codegraff.learn.pending.v1";
+pub const formal_schema = "codegraff.learn.pending.v2";
pub const file_name = "pending.json";
pub const Record = struct {
@@ -25,6 +26,7 @@ pub const Record = struct {
planned_candidates: usize,
repetitions: usize,
auto_requested: bool,
+ formal_admission_evidence_id: ?[]const u8 = null,
primary_baseline: ?eval.PrimaryBaselineRecord = null,
candidates: []const eval.CandidateRecord,
};
@@ -41,12 +43,17 @@ fn validCandidate(candidate: eval.CandidateRecord) bool {
}
pub fn validate(record: Record) !void {
- if (!std.mem.eql(u8, record.schema, schema) or
+ const formal = std.mem.eql(u8, record.schema, formal_schema);
+ if ((!std.mem.eql(u8, record.schema, schema) and !formal) or
!store_mod.validId(record.trial_id) or
!store_mod.validId(record.nonce) or
!store_mod.validId(record.config_id) or
!store_mod.validId(record.parent_genome_id) or
!store_mod.validId(record.parent_transaction_id)) return error.InvalidPendingRun;
+ if (formal) {
+ if (record.formal_admission_evidence_id == null or
+ !store_mod.validId(record.formal_admission_evidence_id.?)) return error.InvalidPendingRun;
+ } else if (record.formal_admission_evidence_id != null) return error.InvalidPendingRun;
if (record.harness_version.len == 0 or record.harness_version.len > 128 or
record.planned_candidates == 0 or record.planned_candidates > 16 or
record.candidates.len != record.planned_candidates or
@@ -84,3 +91,35 @@ pub fn clear(store: *store_mod.Store) !void {
};
try store_mod.syncDirectory(store.io, store.refs);
}
+
+test "formal checkpoint requires admission evidence and legacy rejects it" {
+ const id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ const candidate: eval.CandidateRecord = .{
+ .genome_id = id,
+ .mutation = .{ .seed = id, .request_evidence_id = id, .response_evidence_id = id },
+ .primary = null,
+ .holdout = null,
+ .eligible = false,
+ .reason = "pending",
+ };
+ var record: Record = .{
+ .schema = formal_schema,
+ .trial_id = id,
+ .nonce = id,
+ .created_unix_ms = 0,
+ .harness_version = "test",
+ .config_id = id,
+ .parent_genome_id = id,
+ .parent_generation = 0,
+ .parent_transaction_id = id,
+ .planned_candidates = 1,
+ .repetitions = 1,
+ .auto_requested = false,
+ .candidates = &.{candidate},
+ };
+ try std.testing.expectError(error.InvalidPendingRun, validate(record));
+ record.formal_admission_evidence_id = id;
+ try validate(record);
+ record.schema = schema;
+ try std.testing.expectError(error.InvalidPendingRun, validate(record));
+}
diff --git a/src/learn_cli.zig b/src/learn_cli.zig
index 44e44dbf..695d1482 100644
--- a/src/learn_cli.zig
+++ b/src/learn_cli.zig
@@ -15,6 +15,7 @@ const diagnostics = @import("learn_diagnostics.zig");
const mutation_verify = @import("learn_mutation_verify.zig");
const primary = @import("learn_primary.zig");
const learn_run = @import("learn_run.zig");
+const formal_gate = @import("learn_formal.zig");
const learn_delete = @import("learn_delete.zig");
const submit = @import("learn_submit.zig");
const tournament = @import("learn_tournament.zig");
@@ -239,8 +240,10 @@ pub fn parse(args: []const []const u8) !Parsed {
}
fn verifyRun(
+ gpa: Allocator,
arena: Allocator,
io: Io,
+ environ: *const std.process.Environ.Map,
store: *store_mod.Store,
config: store_mod.LoadedConfig,
run_id: []const u8,
@@ -259,7 +262,8 @@ fn verifyRun(
_ = try store.readGenome(arena, run.parent_genome_id, config.value.limits.genome_bytes);
try learn_init.verifyPins(io, arena, config.value);
- const current_schema = std.mem.eql(u8, run.schema, eval.run_schema);
+ const current_schema = std.mem.eql(u8, run.schema, eval.run_schema) or
+ std.mem.eql(u8, run.schema, eval.formal_run_schema);
var baseline_response: ?eval.PrimaryBaselineResponse = null;
if (current_schema) if (run.primary_baseline) |baseline| {
baseline_response = try primary.verifyBaseline(
@@ -342,6 +346,13 @@ fn verifyRun(
if (primary_winner) |id| if (!std.mem.eql(u8, id, run.primary_winner_genome_id.?)) return error.SelectionMismatch;
if ((finalized.selected_genome_id == null) != (run.selected_genome_id == null)) return error.SelectionMismatch;
if (finalized.selected_genome_id) |id| if (!std.mem.eql(u8, id, run.selected_genome_id.?)) return error.SelectionMismatch;
+ if (config.value.formal_check) |formal| {
+ if (!std.mem.eql(u8, run.schema, eval.formal_run_schema)) return error.FormalRunRequired;
+ try formal_gate.verify(gpa, arena, io, environ, store, formal, run.formal_admission_evidence_id orelse return error.FormalEvidenceMissing, &config.id, run.trial_id, "admission", try store.readGenome(arena, run.parent_genome_id, config.value.limits.genome_bytes));
+ if (finalized.selected_genome_id) |selected| {
+ try formal_gate.verify(gpa, arena, io, environ, store, formal, run.formal_selection_evidence_id orelse return error.FormalEvidenceMissing, &config.id, run.trial_id, "selection", try store.readGenome(arena, selected, config.value.limits.genome_bytes));
+ }
+ } else if (std.mem.eql(u8, run.schema, eval.formal_run_schema)) return error.FormalConfigRequired;
return .{ .run = run, .selected = finalized.selected_genome_id };
}
@@ -364,6 +375,7 @@ fn runCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.proc
if (args.auto and !config.value.auto.enabled) return error.AutoNotEnabled;
if (args.auto and config.value.holdout_suite == null) return error.AutoRequiresHoldout;
+ if (args.submit and config.value.formal_check != null) return error.FormalReceiptUnsupported;
if (args.submit) try submit.preflight(io, arena, environ);
// The adapters see a scrubbed environment, so a `graff login` credential
@@ -383,7 +395,7 @@ fn runCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.proc
if (result.selected_genome_id) |genome_id| {
try out.print("selected {s}\n", .{genome_id});
if (args.auto) {
- const verified = try verifyRun(arena, io, &store, config, &result.run_id);
+ const verified = try verifyRun(gpa, arena, io, environ, &store, config, &result.run_id);
if (verified.selected == null or !std.mem.eql(u8, verified.selected.?, genome_id)) return error.SelectionMismatch;
const current = try store.loadActive(arena, config);
if (!activeMatchesRun(current.ref, verified.run)) return error.ActiveParentChanged;
@@ -395,7 +407,7 @@ fn runCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.proc
} else try out.writeAll("no unique candidate produced primary evidence; active genome unchanged\n");
}
-fn statusCommand(arena: Allocator, io: Io, timeout_ms: u64, out: *Io.Writer, verify_all: bool) !void {
+fn statusCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.process.Environ.Map, timeout_ms: u64, out: *Io.Writer, verify_all: bool) !void {
var store = try store_mod.Store.openAt(io, Io.Dir.cwd());
defer store.deinit();
var lock = try store.acquireLock(timeout_ms);
@@ -405,7 +417,13 @@ fn statusCommand(arena: Allocator, io: Io, timeout_ms: u64, out: *Io.Writer, ver
const pending = try checkpoint.load(arena, &store);
if (verify_all) {
try learn_init.verifyPins(io, arena, config.value);
- if (pending) |item| _ = try learn_run.restorePending(arena, io, &store, config, active, item);
+ if (pending) |item| {
+ if (config.value.formal_check) |formal| {
+ const id = item.formal_admission_evidence_id orelse return error.FormalEvidenceMissing;
+ try formal_gate.verify(gpa, arena, io, environ, &store, formal, id, &config.id, item.trial_id, "admission", active.genome);
+ } else if (std.mem.eql(u8, item.schema, checkpoint.formal_schema)) return error.FormalConfigRequired;
+ _ = try learn_run.restorePending(arena, io, &store, config, active, item);
+ }
}
const Status = struct {
schema: []const u8 = "codegraff.learn.status.v1",
@@ -446,14 +464,14 @@ fn countCompleted(candidates: []const eval.CandidateRecord, holdouts: bool) usiz
return count;
}
-fn promoteCommand(gpa: Allocator, arena: Allocator, io: Io, args: PromoteArgs, out: *Io.Writer) !void {
+fn promoteCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.process.Environ.Map, args: PromoteArgs, out: *Io.Writer) !void {
var store = try store_mod.Store.openAt(io, Io.Dir.cwd());
defer store.deinit();
var lock = try store.acquireLock(args.lock_timeout_ms);
defer lock.deinit();
const config = try store.loadConfig(arena);
const active = try store.loadActive(arena, config);
- const verified = try verifyRun(arena, io, &store, config, args.run_id);
+ const verified = try verifyRun(gpa, arena, io, environ, &store, config, args.run_id);
const selected = verified.selected orelse return error.NoPromotableCandidate;
if (!activeMatchesRun(active.ref, verified.run)) return error.ActiveParentChanged;
try store.activate(gpa, active.ref, &config.id, selected, args.run_id, "promote", config.value.limits.genome_bytes, util.unixMs(io));
@@ -466,7 +484,8 @@ fn submitCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.p
var lock = try store.acquireLock(args.lock_timeout_ms);
defer lock.deinit();
const config = try store.loadConfig(arena);
- const verified = try verifyRun(arena, io, &store, config, args.run_id);
+ if (config.value.formal_check != null) return error.FormalReceiptUnsupported;
+ const verified = try verifyRun(gpa, arena, io, environ, &store, config, args.run_id);
const sent = try submit.submitVerifiedRun(io, gpa, arena, environ, config.value, args.run_id, verified.run);
try out.print("submitted {d} signed aggregate grade(s) for run {s}; prompt text stayed local\n", .{ sent.grades, args.run_id });
}
@@ -520,11 +539,11 @@ pub fn command(io: Io, gpa: Allocator, arena: Allocator, init: std.process.Init,
else
try learn_init.fromPaths(gpa, arena, io, args, out),
.run => |args| try runCommand(gpa, arena, io, init.environ_map, args, out),
- .status => |args| try statusCommand(arena, io, args.lock_timeout_ms, out, false),
+ .status => |args| try statusCommand(gpa, arena, io, init.environ_map, args.lock_timeout_ms, out, false),
.submit => |args| try submitCommand(gpa, arena, io, init.environ_map, args, out),
.delete_remote => |args| try learn_delete.command(io, gpa, arena, init.environ_map, args.run_id, args.lock_timeout_ms, out),
- .verify => |args| try statusCommand(arena, io, args.lock_timeout_ms, out, true),
- .promote => |args| try promoteCommand(gpa, arena, io, args, out),
+ .verify => |args| try statusCommand(gpa, arena, io, init.environ_map, args.lock_timeout_ms, out, true),
+ .promote => |args| try promoteCommand(gpa, arena, io, init.environ_map, args, out),
.rollback => |args| try rollbackCommand(gpa, arena, io, args, out),
.hash => |args| {
const digest = try store_mod.hashFileNoFollow(io, args.path);
diff --git a/src/learn_eval.zig b/src/learn_eval.zig
index 95f0bcf2..390e5a8e 100644
--- a/src/learn_eval.zig
+++ b/src/learn_eval.zig
@@ -25,6 +25,7 @@ pub const primary_evaluation_request_schema = protocol.primary_evaluation_reques
pub const primary_evaluation_response_schema = protocol.primary_evaluation_response_schema;
pub const legacy_run_schema = protocol.legacy_run_schema;
pub const run_schema = protocol.run_schema;
+pub const formal_run_schema = protocol.formal_run_schema;
pub const GenomeRef = protocol.GenomeRef;
pub const MutationRequest = protocol.MutationRequest;
pub const MutationResponse = protocol.MutationResponse;
@@ -532,13 +533,19 @@ pub fn verifyComparison(
pub fn validateRun(run: RunRecord) !void {
const current = std.mem.eql(u8, run.schema, run_schema);
+ const formal = std.mem.eql(u8, run.schema, formal_run_schema);
const legacy = std.mem.eql(u8, run.schema, legacy_run_schema);
- if ((!current and !legacy) or !store_mod.validId(run.trial_id) or !store_mod.validId(run.nonce) or !store_mod.validId(run.config_id) or !store_mod.validId(run.parent_genome_id) or !store_mod.validId(run.parent_transaction_id)) return error.InvalidRun;
+ if ((!current and !formal and !legacy) or !store_mod.validId(run.trial_id) or !store_mod.validId(run.nonce) or !store_mod.validId(run.config_id) or !store_mod.validId(run.parent_genome_id) or !store_mod.validId(run.parent_transaction_id)) return error.InvalidRun;
+ if (formal) {
+ if (run.formal_admission_evidence_id == null or !store_mod.validId(run.formal_admission_evidence_id.?) or
+ (run.selected_genome_id == null) != (run.formal_selection_evidence_id == null)) return error.InvalidRun;
+ if (run.formal_selection_evidence_id) |id| if (!store_mod.validId(id)) return error.InvalidRun;
+ } else if (run.formal_admission_evidence_id != null or run.formal_selection_evidence_id != null) return error.InvalidRun;
if (run.planned_candidates == 0 or run.planned_candidates > 16 or run.candidates.len != run.planned_candidates) return error.InvalidRun;
if (run.repetitions == 0 or run.repetitions > 100 or run.harness_version.len == 0 or run.harness_version.len > 128) return error.InvalidRun;
if (run.primary_winner_genome_id) |id| if (!store_mod.validId(id)) return error.InvalidRun;
if (run.selected_genome_id) |id| if (!store_mod.validId(id)) return error.InvalidRun;
- if (current) {
+ if (current or formal) {
if (run.primary_baseline) |baseline| {
if (!store_mod.validId(baseline.suite_sha256) or !store_mod.validId(baseline.request_evidence_id) or !store_mod.validId(baseline.response_evidence_id)) return error.InvalidRun;
}
diff --git a/src/learn_eval_tests.zig b/src/learn_eval_tests.zig
index 91007026..eb6b46cb 100644
--- a/src/learn_eval_tests.zig
+++ b/src/learn_eval_tests.zig
@@ -26,6 +26,43 @@ const executable_permissions = learn_eval.executable_permissions;
const dir_permissions = learn_eval.dir_permissions;
const builtin = @import("builtin");
+test "formal run schema binds admission and selected evidence" {
+ const id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ const candidate: learn_eval.CandidateRecord = .{
+ .genome_id = id,
+ .mutation = .{ .seed = id, .request_evidence_id = id, .response_evidence_id = id },
+ .primary = null,
+ .holdout = null,
+ .eligible = false,
+ .reason = "pending",
+ };
+ var run: learn_eval.RunRecord = .{
+ .schema = learn_eval.formal_run_schema,
+ .trial_id = id,
+ .nonce = id,
+ .created_unix_ms = 0,
+ .harness_version = "test",
+ .config_id = id,
+ .parent_genome_id = id,
+ .parent_generation = 0,
+ .parent_transaction_id = id,
+ .planned_candidates = 1,
+ .repetitions = 1,
+ .auto_requested = false,
+ .candidates = &.{candidate},
+ .selected_genome_id = null,
+ };
+ try std.testing.expectError(error.InvalidRun, learn_eval.validateRun(run));
+ run.formal_admission_evidence_id = id;
+ try learn_eval.validateRun(run);
+ run.selected_genome_id = id;
+ try std.testing.expectError(error.InvalidRun, learn_eval.validateRun(run));
+ run.formal_selection_evidence_id = id;
+ try learn_eval.validateRun(run);
+ run.schema = learn_eval.run_schema;
+ try std.testing.expectError(error.InvalidRun, learn_eval.validateRun(run));
+}
+
test "paired tail has known exact values" {
try std.testing.expectApproxEqAbs(@as(f64, 0.5), pairedTail(1, 0), 1e-12);
try std.testing.expectApproxEqAbs(@as(f64, 0.0546875), pairedTail(8, 2), 1e-12);
diff --git a/src/learn_eval_types.zig b/src/learn_eval_types.zig
index 7681cbc1..0c406c8d 100644
--- a/src/learn_eval_types.zig
+++ b/src/learn_eval_types.zig
@@ -10,6 +10,7 @@ pub const primary_evaluation_request_schema = "codegraff.learn.primary-evaluatio
pub const primary_evaluation_response_schema = "codegraff.learn.primary-evaluation.response.v1";
pub const legacy_run_schema = "codegraff.learn.run.v2";
pub const run_schema = "codegraff.learn.run.v3";
+pub const formal_run_schema = "codegraff.learn.run.v4";
pub const GenomeRef = struct {
id: []const u8,
@@ -238,6 +239,8 @@ pub const RunRecord = struct {
planned_candidates: usize,
repetitions: usize,
auto_requested: bool,
+ formal_admission_evidence_id: ?[]const u8 = null,
+ formal_selection_evidence_id: ?[]const u8 = null,
primary_baseline: ?PrimaryBaselineRecord = null,
candidates: []const CandidateRecord,
/// Best primary-suite contender. This remains populated when the evidence
diff --git a/src/learn_formal.zig b/src/learn_formal.zig
new file mode 100644
index 00000000..e8eb0c0a
--- /dev/null
+++ b/src/learn_formal.zig
@@ -0,0 +1,333 @@
+//! Optional, pinned model-check baseline for native prompt-policy learning.
+//! The result concerns shared engine source, not the candidate prompt's behavior.
+const std = @import("std");
+const Io = std.Io;
+const Allocator = std.mem.Allocator;
+const store_mod = @import("learn_store.zig");
+const jobs = @import("jobs.zig");
+const util = @import("util.zig");
+
+pub const evidence_schema = "codegraff.learn.formal-evidence.v1";
+const check_schema = "codegraff.dgm.formal-check.v1";
+const stdout_cap = 4096;
+const stderr_cap = 4096;
+
+pub const Evidence = struct {
+ schema: []const u8 = evidence_schema,
+ phase: []const u8,
+ config_id: []const u8,
+ trial_id: []const u8,
+ candidate_prompt_sha256: []const u8,
+ formal_identity_sha256: []const u8,
+ checker_output_sha256: []const u8,
+ binary_sha256: []const u8,
+ formal_pin_sha256: []const u8,
+ checker_sha256: []const u8,
+ helper_sha256: []const u8,
+};
+
+fn validHash(value: []const u8) bool {
+ return store_mod.validId(value);
+}
+
+fn promptHash(prompt: []const u8) [64]u8 {
+ var digest: [32]u8 = undefined;
+ std.crypto.hash.sha2.Sha256.hash(prompt, &digest, .{});
+ return std.fmt.bytesToHex(digest, .lower);
+}
+
+fn checkedString(obj: std.json.ObjectMap, key: []const u8) ![]const u8 {
+ const value = util.strFieldObj(obj, key) orelse return error.InvalidFormalCheckResult;
+ if (!validHash(value)) return error.InvalidFormalCheckResult;
+ return value;
+}
+
+fn parseResult(arena: Allocator, raw: []const u8, candidate_sha: []const u8, binary_sha: []const u8) !Evidence {
+ const parsed = try std.json.parseFromSliceLeaky(std.json.Value, arena, raw, .{});
+ if (parsed != .object or parsed.object.count() != 6 or
+ !std.mem.eql(u8, util.strFieldObj(parsed.object, "schema") orelse "", check_schema))
+ return error.InvalidFormalCheckResult;
+ const ok = parsed.object.get("ok") orelse return error.InvalidFormalCheckResult;
+ if (ok != .bool or !ok.bool) return error.InvalidFormalCheckResult;
+ const got_candidate = try checkedString(parsed.object, "candidate_prompt_sha256");
+ const got_binary = try checkedString(parsed.object, "binary_sha256");
+ if (!std.mem.eql(u8, got_candidate, candidate_sha) or !std.mem.eql(u8, got_binary, binary_sha))
+ return error.FormalCheckIdentityMismatch;
+ return .{
+ .phase = "",
+ .config_id = "",
+ .trial_id = "",
+ .candidate_prompt_sha256 = got_candidate,
+ .formal_identity_sha256 = try checkedString(parsed.object, "formal_identity_sha256"),
+ .checker_output_sha256 = try checkedString(parsed.object, "checker_output_sha256"),
+ .binary_sha256 = got_binary,
+ .formal_pin_sha256 = "",
+ .checker_sha256 = "",
+ .helper_sha256 = "",
+ };
+}
+
+fn binaryPin(arena: Allocator, io: Io, formal: store_mod.FormalCheck) !store_mod.PinnedFile {
+ const bytes = try store_mod.readPinnedFileAlloc(io, arena, formal.pin, store_mod.max_config_bytes);
+ const parsed = try std.json.parseFromSliceLeaky(std.json.Value, arena, bytes, .{});
+ if (parsed != .object) return error.InvalidFormalPin;
+ const binary = parsed.object.get("binary") orelse return error.InvalidFormalPin;
+ if (binary != .object) return error.InvalidFormalPin;
+ const sha = util.strFieldObj(binary.object, "sha256") orelse return error.InvalidFormalPin;
+ const path = util.strFieldObj(binary.object, "path") orelse return error.InvalidFormalPin;
+ if (!validHash(sha) or !std.fs.path.isAbsolute(path)) return error.InvalidFormalPin;
+ return .{ .path = path, .sha256 = sha };
+}
+
+fn matchesAdapter(program: store_mod.Program, binary: store_mod.PinnedFile) bool {
+ var pinned = false;
+ var passed = false;
+ for (program.inputs) |input| {
+ if (std.mem.eql(u8, input.path, binary.path) and std.mem.eql(u8, input.sha256, binary.sha256))
+ pinned = true;
+ }
+ for (program.args) |arg| {
+ if (std.mem.eql(u8, arg, binary.path)) passed = true;
+ }
+ return pinned and passed;
+}
+
+pub fn verifyBinaryBinding(arena: Allocator, io: Io, config: store_mod.Config) !void {
+ const formal = config.formal_check orelse return;
+ const binary = try binaryPin(arena, io, formal);
+ if (!matchesAdapter(config.mutator, binary) or !matchesAdapter(config.evaluator, binary))
+ return error.FormalBinaryMismatch;
+}
+
+fn check(
+ gpa: Allocator,
+ arena: Allocator,
+ io: Io,
+ environ: *const std.process.Environ.Map,
+ formal: store_mod.FormalCheck,
+ prompt: []const u8,
+) !Evidence {
+ const key_path = environ.get("GRAFF_SCORE_KEY_FILE") orelse return error.FormalSigningKeyRequired;
+ if (!std.fs.path.isAbsolute(key_path)) return error.FormalSigningKeyRequired;
+ if (formal.checker.args.len != 1 or formal.checker.inputs.len != 1)
+ return error.InvalidFormalChecker;
+ try store_mod.verifyProgram(io, formal.checker);
+ try store_mod.verifyPinnedFile(io, formal.pin);
+ const expected_binary = (try binaryPin(arena, io, formal)).sha256;
+ const candidate_sha = promptHash(prompt);
+ var child_env = std.process.Environ.Map.init(gpa);
+ defer child_env.deinit();
+ try child_env.put("PATH", "/usr/bin:/bin");
+ try child_env.put("LANG", "C");
+ try child_env.put("LC_ALL", "C");
+ try child_env.put("GRAFF_SCORE_KEY_FILE", key_path);
+ try child_env.put("PYTHONDONTWRITEBYTECODE", "1");
+ const argv = [_][]const u8{
+ formal.checker.program,
+ formal.checker.args[0],
+ "--check",
+ formal.pin.path,
+ "--candidate-sha256",
+ &candidate_sha,
+ };
+ const run = try jobs.runCappedWithOptions(gpa, io, &argv, stdout_cap, stderr_cap, formal.timeout_ms, .{
+ .environ_map = &child_env,
+ .kill_process_tree = true,
+ });
+ defer {
+ gpa.free(run.stdout);
+ gpa.free(run.stderr);
+ }
+ if (run.timed_out or run.cancelled) return error.FormalCheckTimedOut;
+ if (run.stdout_truncated or run.stderr_truncated) return error.FormalCheckOutputTooLarge;
+ if (run.term != .exited or run.term.exited != 0) return error.FormalCheckFailed;
+ // The pin is checked after execution too. The external pin/helper is an
+ // operator trust input; this detects drift, not adversarial transient swap.
+ try store_mod.verifyProgram(io, formal.checker);
+ try store_mod.verifyPinnedFile(io, formal.pin);
+ var result = try parseResult(arena, run.stdout, &candidate_sha, expected_binary);
+ result.formal_pin_sha256 = formal.pin.sha256;
+ result.checker_sha256 = formal.checker.sha256;
+ result.helper_sha256 = formal.checker.inputs[0].sha256;
+ return result;
+}
+
+pub fn record(
+ gpa: Allocator,
+ arena: Allocator,
+ io: Io,
+ environ: *const std.process.Environ.Map,
+ store: *store_mod.Store,
+ formal: store_mod.FormalCheck,
+ config_id: []const u8,
+ trial_id: []const u8,
+ phase: []const u8,
+ prompt: []const u8,
+) ![]const u8 {
+ if (!std.mem.eql(u8, phase, "admission") and !std.mem.eql(u8, phase, "selection"))
+ return error.InvalidFormalPhase;
+ var result = try check(gpa, arena, io, environ, formal, prompt);
+ result.phase = phase;
+ result.config_id = config_id;
+ result.trial_id = trial_id;
+ const bytes = try store_mod.jsonBytes(gpa, result);
+ defer gpa.free(bytes);
+ const id = try store.writeEvidence(gpa, bytes);
+ return try arena.dupe(u8, &id);
+}
+
+pub fn verify(
+ gpa: Allocator,
+ arena: Allocator,
+ io: Io,
+ environ: *const std.process.Environ.Map,
+ store: *store_mod.Store,
+ formal: store_mod.FormalCheck,
+ evidence_id: []const u8,
+ config_id: []const u8,
+ trial_id: []const u8,
+ phase: []const u8,
+ prompt: []const u8,
+) !void {
+ if (formal.checker.args.len != 1 or formal.checker.inputs.len != 1)
+ return error.InvalidFormalChecker;
+ const bytes = try store.readEvidence(arena, evidence_id, store_mod.max_record_bytes);
+ const saved = try std.json.parseFromSliceLeaky(Evidence, arena, bytes, .{});
+ const expected_sha = promptHash(prompt);
+ if (!std.mem.eql(u8, saved.schema, evidence_schema) or
+ !std.mem.eql(u8, saved.phase, phase) or
+ !std.mem.eql(u8, saved.config_id, config_id) or
+ !std.mem.eql(u8, saved.trial_id, trial_id) or
+ !std.mem.eql(u8, saved.candidate_prompt_sha256, &expected_sha) or
+ !std.mem.eql(u8, saved.formal_pin_sha256, formal.pin.sha256) or
+ !std.mem.eql(u8, saved.checker_sha256, formal.checker.sha256) or
+ !std.mem.eql(u8, saved.helper_sha256, formal.checker.inputs[0].sha256) or
+ !validHash(saved.formal_identity_sha256) or !validHash(saved.checker_output_sha256) or
+ !validHash(saved.binary_sha256)) return error.FormalEvidenceMismatch;
+ const fresh = try check(gpa, arena, io, environ, formal, prompt);
+ // TLC output contains timestamps/seeds, so its output hash may differ on a
+ // fresh run. The pinned identity, binary and candidate must match.
+ if (!std.mem.eql(u8, saved.formal_identity_sha256, fresh.formal_identity_sha256) or
+ !std.mem.eql(u8, saved.binary_sha256, fresh.binary_sha256))
+ return error.FormalEvidenceMismatch;
+}
+
+test "formal helper result rejects false success and wrong candidate" {
+ const a = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ const b = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
+ var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);
+ defer arena_state.deinit();
+ const arena = arena_state.allocator();
+ const good = try std.fmt.allocPrint(arena, "{{\"schema\":\"{s}\",\"ok\":true,\"candidate_prompt_sha256\":\"{s}\",\"formal_identity_sha256\":\"{s}\",\"checker_output_sha256\":\"{s}\",\"binary_sha256\":\"{s}\"}}", .{ check_schema, a, a, a, b });
+ _ = try parseResult(arena, good, a, b);
+ try std.testing.expectError(error.FormalCheckIdentityMismatch, parseResult(arena, good, b, b));
+ const false_success = try std.fmt.allocPrint(arena, "{{\"schema\":\"{s}\",\"ok\":false,\"candidate_prompt_sha256\":\"{s}\",\"formal_identity_sha256\":\"{s}\",\"checker_output_sha256\":\"{s}\",\"binary_sha256\":\"{s}\"}}", .{ check_schema, a, a, a, b });
+ try std.testing.expectError(error.InvalidFormalCheckResult, parseResult(arena, false_success, a, b));
+}
+
+test "formal binary must be pinned and passed to both adapters" {
+ const a = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ const b = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
+ const binary: store_mod.PinnedFile = .{ .path = "/opt/graff", .sha256 = a };
+ const correct: store_mod.Program = .{
+ .program = "/opt/adapter",
+ .sha256 = b,
+ .args = &.{"/opt/graff"},
+ .inputs = &.{binary},
+ };
+ try std.testing.expect(matchesAdapter(correct, binary));
+ var wrong = correct;
+ wrong.args = &.{"/opt/other"};
+ try std.testing.expect(!matchesAdapter(wrong, binary));
+ wrong = correct;
+ wrong.inputs = &.{.{ .path = "/opt/graff", .sha256 = b }};
+ try std.testing.expect(!matchesAdapter(wrong, binary));
+}
+
+test "default configuration skips the optional formal checker" {
+ const hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ const config: store_mod.Config = .{
+ .schema = store_mod.config_schema,
+ .agent_name = "candidate",
+ .mutation_instruction = "change one behavior",
+ .mutator = .{ .program = "/missing/mutator", .sha256 = hash },
+ .evaluator = .{ .program = "/missing/evaluator", .sha256 = hash },
+ .evaluation_suite = .{ .path = "/missing/suite", .sha256 = hash },
+ .cohort = .{ .provider = "test", .model = "test", .task_family = "test", .adapter_version = "v1", .verifier_version = "v1" },
+ };
+ try verifyBinaryBinding(std.testing.allocator, std.testing.io, config);
+}
+
+test "pinned helper evidence is rechecked for resume and promotion contexts" {
+ if (@import("builtin").os.tag == .windows) return;
+ const io = std.testing.io;
+ const gpa = std.testing.allocator;
+ var arena_state = std.heap.ArenaAllocator.init(gpa);
+ defer arena_state.deinit();
+ const arena = arena_state.allocator();
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ var root_buf: [std.fs.max_path_bytes]u8 = undefined;
+ const root_len = try tmp.dir.realPath(io, &root_buf);
+ const root = root_buf[0..root_len];
+ const helper_path = try std.fmt.allocPrint(arena, "{s}/helper.sh", .{root});
+ const pin_path = try std.fmt.allocPrint(arena, "{s}/pin.json", .{root});
+ const key_path = try std.fmt.allocPrint(arena, "{s}/key", .{root});
+ const binary_hash = try store_mod.hashFileNoFollow(io, "/bin/sh");
+ const identity = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ const script = try std.fmt.allocPrint(arena, "printf '{{\"schema\":\"{s}\",\"ok\":true,\"candidate_prompt_sha256\":\"%s\",\"formal_identity_sha256\":\"{s}\",\"checker_output_sha256\":\"{s}\",\"binary_sha256\":\"{s}\"}}\\n' \"$4\"\n", .{ check_schema, identity, identity, &binary_hash });
+ {
+ const file = try tmp.dir.createFile(io, "helper.sh", .{ .permissions = .fromMode(0o600) });
+ defer file.close(io);
+ try file.writeStreamingAll(io, script);
+ }
+ const pin_bytes = try std.fmt.allocPrint(arena, "{{\"binary\":{{\"path\":\"/bin/sh\",\"sha256\":\"{s}\"}}}}\n", .{&binary_hash});
+ {
+ const file = try tmp.dir.createFile(io, "pin.json", .{ .permissions = .fromMode(0o600) });
+ defer file.close(io);
+ try file.writeStreamingAll(io, pin_bytes);
+ }
+ {
+ const file = try tmp.dir.createFile(io, "key", .{ .permissions = .fromMode(0o600) });
+ defer file.close(io);
+ try file.writeStreamingAll(io, "test key");
+ }
+ const helper_hash = store_mod.rawSha256(script);
+ const pin_hash = store_mod.rawSha256(pin_bytes);
+ const inputs = [_]store_mod.PinnedFile{.{ .path = helper_path, .sha256 = &helper_hash }};
+ const formal: store_mod.FormalCheck = .{
+ .checker = .{ .program = "/bin/sh", .sha256 = &binary_hash, .args = &.{helper_path}, .inputs = &inputs },
+ .pin = .{ .path = pin_path, .sha256 = &pin_hash },
+ .timeout_ms = 2000,
+ };
+ var env = std.process.Environ.Map.init(gpa);
+ defer env.deinit();
+ try std.testing.expectError(error.FormalSigningKeyRequired, check(gpa, arena, io, &env, formal, "parent prompt"));
+ try env.put("GRAFF_SCORE_KEY_FILE", key_path);
+ var store = try store_mod.Store.initAt(io, tmp.dir);
+ defer store.deinit();
+ const config_id = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";
+ const trial_id = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc";
+ const admission = try record(gpa, arena, io, &env, &store, formal, config_id, trial_id, "admission", "parent prompt");
+ try verify(gpa, arena, io, &env, &store, formal, admission, config_id, trial_id, "admission", "parent prompt");
+ try std.testing.expectError(error.FormalEvidenceMismatch, verify(gpa, arena, io, &env, &store, formal, admission, config_id, trial_id, "admission", "changed parent"));
+ try std.testing.expectError(error.FormalEvidenceMismatch, verify(gpa, arena, io, &env, &store, formal, admission, config_id, trial_id, "selection", "parent prompt"));
+ const selection = try record(gpa, arena, io, &env, &store, formal, config_id, trial_id, "selection", "selected prompt");
+ try verify(gpa, arena, io, &env, &store, formal, selection, config_id, trial_id, "selection", "selected prompt");
+ try std.testing.expectError(error.FormalEvidenceMismatch, verify(gpa, arena, io, &env, &store, formal, selection, config_id, trial_id, "selection", "different selection"));
+ const missing = "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd";
+ try std.testing.expectError(error.FileNotFound, verify(gpa, arena, io, &env, &store, formal, missing, config_id, trial_id, "admission", "parent prompt"));
+
+ const slow_script = "sleep 2\n";
+ {
+ const file = try tmp.dir.createFile(io, "helper.sh", .{ .truncate = true, .permissions = .fromMode(0o600) });
+ defer file.close(io);
+ try file.writeStreamingAll(io, slow_script);
+ }
+ const slow_hash = store_mod.rawSha256(slow_script);
+ const slow_inputs = [_]store_mod.PinnedFile{.{ .path = helper_path, .sha256 = &slow_hash }};
+ var slow_formal = formal;
+ slow_formal.checker.inputs = &slow_inputs;
+ slow_formal.timeout_ms = 50;
+ try std.testing.expectError(error.FormalCheckTimedOut, check(gpa, arena, io, &env, slow_formal, "parent prompt"));
+}
diff --git a/src/learn_init.zig b/src/learn_init.zig
index 24d84e36..4d8748a4 100644
--- a/src/learn_init.zig
+++ b/src/learn_init.zig
@@ -9,6 +9,7 @@ const Allocator = std.mem.Allocator;
const store_mod = @import("learn_store.zig");
const learn_run = @import("learn_run.zig");
const bootstrap = @import("learn_bootstrap.zig");
+const formal_gate = @import("learn_formal.zig");
const credentials = @import("learn_credentials.zig");
const util = @import("util.zig");
@@ -58,6 +59,11 @@ pub fn zeroConfig(
pub fn verifyPins(io: Io, arena: Allocator, config: store_mod.Config) !void {
try store_mod.verifyProgram(io, config.mutator);
try store_mod.verifyProgram(io, config.evaluator);
+ if (config.formal_check) |formal| {
+ try store_mod.verifyProgram(io, formal.checker);
+ try store_mod.verifyPinnedFile(io, formal.pin);
+ try formal_gate.verifyBinaryBinding(arena, io, config);
+ }
const primary_suite = try store_mod.loadSuite(io, arena, config.evaluation_suite);
try store_mod.validateSuitePower(primary_suite.manifest, config.gate);
if (config.holdout_suite) |suite| {
diff --git a/src/learn_progress.zig b/src/learn_progress.zig
new file mode 100644
index 00000000..3ff03909
--- /dev/null
+++ b/src/learn_progress.zig
@@ -0,0 +1,37 @@
+//! Immutable, resumable tournament progress records.
+const build_options = @import("build_options");
+const checkpoint = @import("learn_checkpoint.zig");
+const eval = @import("learn_eval.zig");
+const store = @import("learn_store.zig");
+
+pub fn record(
+ config_id: []const u8,
+ active: store.LoadedActive,
+ nonce: []const u8,
+ trial_id: []const u8,
+ created_unix_ms: i64,
+ candidate_count: usize,
+ repetitions: usize,
+ auto_requested: bool,
+ formal_admission_evidence_id: ?[]const u8,
+ primary_baseline: ?eval.PrimaryBaselineRecord,
+ candidates: []const eval.CandidateRecord,
+) checkpoint.Record {
+ return .{
+ .schema = if (formal_admission_evidence_id != null) checkpoint.formal_schema else checkpoint.schema,
+ .trial_id = trial_id,
+ .nonce = nonce,
+ .created_unix_ms = created_unix_ms,
+ .harness_version = build_options.version,
+ .config_id = config_id,
+ .parent_genome_id = active.ref.genome_id,
+ .parent_generation = active.ref.generation,
+ .parent_transaction_id = active.ref.transaction_id,
+ .planned_candidates = candidate_count,
+ .repetitions = repetitions,
+ .auto_requested = auto_requested,
+ .formal_admission_evidence_id = formal_admission_evidence_id,
+ .primary_baseline = primary_baseline,
+ .candidates = candidates,
+ };
+}
diff --git a/src/learn_run.zig b/src/learn_run.zig
index f942ce5f..2528dead 100644
--- a/src/learn_run.zig
+++ b/src/learn_run.zig
@@ -7,9 +7,11 @@ const build_options = @import("build_options");
const checkpoint = @import("learn_checkpoint.zig");
const eval = @import("learn_eval.zig");
+const formal_gate = @import("learn_formal.zig");
const holdout = @import("learn_holdout.zig");
const mutation_verify = @import("learn_mutation_verify.zig");
const primary = @import("learn_primary.zig");
+const progressRecord = @import("learn_progress.zig").record;
const report = @import("learn_report.zig");
const store_mod = @import("learn_store.zig");
const submit = @import("learn_submit.zig");
@@ -142,6 +144,11 @@ fn mutationIsDuplicate(parent_id: []const u8, candidates: []const eval.Candidate
fn verifyPins(io: Io, arena: Allocator, config: store_mod.Config) !void {
try store_mod.verifyProgram(io, config.mutator);
try store_mod.verifyProgram(io, config.evaluator);
+ if (config.formal_check) |formal| {
+ try store_mod.verifyProgram(io, formal.checker);
+ try store_mod.verifyPinnedFile(io, formal.pin);
+ try formal_gate.verifyBinaryBinding(arena, io, config);
+ }
const primary_suite = try store_mod.loadSuite(io, arena, config.evaluation_suite);
try store_mod.validateSuitePower(primary_suite.manifest, config.gate);
if (config.holdout_suite) |suite| {
@@ -182,35 +189,6 @@ fn copyMutationCandidate(arena: Allocator, outcome: eval.MutationOutcome) !eval.
};
}
-fn progressRecord(
- config_id: []const u8,
- active: store_mod.LoadedActive,
- nonce: []const u8,
- trial_id: []const u8,
- created_unix_ms: i64,
- candidate_count: usize,
- repetitions: usize,
- auto_requested: bool,
- primary_baseline: ?eval.PrimaryBaselineRecord,
- candidates: []const eval.CandidateRecord,
-) checkpoint.Record {
- return .{
- .trial_id = trial_id,
- .nonce = nonce,
- .created_unix_ms = created_unix_ms,
- .harness_version = build_options.version,
- .config_id = config_id,
- .parent_genome_id = active.ref.genome_id,
- .parent_generation = active.ref.generation,
- .parent_transaction_id = active.ref.transaction_id,
- .planned_candidates = candidate_count,
- .repetitions = repetitions,
- .auto_requested = auto_requested,
- .primary_baseline = primary_baseline,
- .candidates = candidates,
- };
-}
-
fn verifyRestoredProgress(
arena: Allocator,
io: Io,
@@ -344,6 +322,10 @@ pub fn execute(
out: *Io.Writer,
) !Result {
if (options.resume_run and options.restart) return error.ConflictingOptions;
+ if (config.value.formal_check != null) {
+ try verifyPins(io, arena, config.value);
+ if (options.submit) return error.FormalReceiptUnsupported;
+ }
if (options.restart) try checkpoint.clear(store);
const pending = try checkpoint.load(arena, store);
@@ -356,6 +338,7 @@ pub fn execute(
var repetitions: usize = undefined;
var primary_baseline: ?eval.PrimaryBaselineRecord = null;
var candidates: []eval.CandidateRecord = undefined;
+ var formal_admission_evidence_id: ?[]const u8 = null;
if (options.resume_run) {
const saved = pending orelse return error.NoPendingRun;
@@ -364,6 +347,13 @@ pub fn execute(
options.auto != saved.auto_requested) return error.ResumeOptionMismatch;
nonce = saved.nonce;
trial_id = saved.trial_id;
+ if ((config.value.formal_check != null) != std.mem.eql(u8, saved.schema, checkpoint.formal_schema))
+ return error.FormalPendingContextMismatch;
+ if (config.value.formal_check) |formal| {
+ const evidence_id = saved.formal_admission_evidence_id orelse return error.FormalPendingContextMismatch;
+ try formal_gate.verify(gpa, arena, io, environ, store, formal, evidence_id, &config.id, saved.trial_id, "admission", active.genome);
+ formal_admission_evidence_id = evidence_id;
+ }
created_unix_ms = saved.created_unix_ms;
candidate_count = saved.planned_candidates;
repetitions = saved.repetitions;
@@ -386,6 +376,9 @@ pub fn execute(
trial_buffer = trialId(&config.id, active.ref.genome_id, active.ref.generation, active.ref.transaction_id, &nonce_buffer);
nonce = &nonce_buffer;
trial_id = &trial_buffer;
+ if (config.value.formal_check) |formal| {
+ formal_admission_evidence_id = try formal_gate.record(gpa, arena, io, environ, store, formal, &config.id, trial_id, "admission", active.genome);
+ }
created_unix_ms = util.unixMs(io);
candidates = try arena.alloc(eval.CandidateRecord, candidate_count);
@@ -420,6 +413,7 @@ pub fn execute(
candidate_count,
repetitions,
options.auto,
+ formal_admission_evidence_id,
null,
candidates,
));
@@ -450,7 +444,7 @@ pub fn execute(
config.value.evaluation_suite,
repetitions,
);
- try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, primary_baseline, candidates));
+ try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, formal_admission_evidence_id, primary_baseline, candidates));
}
var missing_primary: std.ArrayList(usize) = .empty;
@@ -478,7 +472,7 @@ pub fn execute(
if (result) |completed| {
candidates[index].primary = completed.comparison;
candidates[index].reason = candidates[index].primary.?.reason;
- try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, primary_baseline, candidates));
+ try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, formal_admission_evidence_id, primary_baseline, candidates));
} else |err| if (primary_failure == null) {
primary_failure = err;
}
@@ -510,12 +504,19 @@ pub fn execute(
repetitions,
1,
);
- try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, primary_baseline, candidates));
+ try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, formal_admission_evidence_id, primary_baseline, candidates));
}
};
}
const finalized = try tournament.finalize(active.ref.genome_id, candidates, config.value.holdout_suite != null);
const primary_winner = if (finalized.primary_winner_index) |index| candidates[index].genome_id else null;
+ const formal_selection_evidence_id: ?[]const u8 = if (finalized.selected_genome_id) |selected| blk: {
+ if (config.value.formal_check) |formal| {
+ const prompt = try store.readGenome(arena, selected, config.value.limits.genome_bytes);
+ break :blk try formal_gate.record(gpa, arena, io, environ, store, formal, &config.id, trial_id, "selection", prompt);
+ }
+ break :blk null;
+ } else null;
// Re-check every externally controlled executable and suite after use and
// before committing the immutable run record.
@@ -523,7 +524,7 @@ pub fn execute(
try verifyTrialPower(io, arena, config.value, candidate_count);
const record: eval.RunRecord = .{
- .schema = eval.run_schema,
+ .schema = if (config.value.formal_check != null) eval.formal_run_schema else eval.run_schema,
.trial_id = trial_id,
.nonce = nonce,
.created_unix_ms = created_unix_ms,
@@ -535,6 +536,8 @@ pub fn execute(
.planned_candidates = candidate_count,
.repetitions = repetitions,
.auto_requested = options.auto,
+ .formal_admission_evidence_id = formal_admission_evidence_id,
+ .formal_selection_evidence_id = formal_selection_evidence_id,
.primary_baseline = primary_baseline,
.candidates = candidates,
.primary_winner_genome_id = primary_winner,
diff --git a/src/learn_store.zig b/src/learn_store.zig
index 9f8a5759..bc4363cf 100644
--- a/src/learn_store.zig
+++ b/src/learn_store.zig
@@ -21,6 +21,7 @@ pub const max_suite_bytes = model.max_suite_bytes;
pub const max_pairs = model.max_pairs;
pub const PinnedFile = model.PinnedFile;
pub const Program = model.Program;
+pub const FormalCheck = model.FormalCheck;
pub const Suite = model.Suite;
pub const Limits = model.Limits;
pub const Gate = model.Gate;
diff --git a/src/learn_store_tests.zig b/src/learn_store_tests.zig
index 206e4eda..2494bb44 100644
--- a/src/learn_store_tests.zig
+++ b/src/learn_store_tests.zig
@@ -50,6 +50,32 @@ test "learning config parser rejects unknown fields and unsafe auto" {
try std.testing.expectError(error.UnknownField, std.json.parseFromSlice(Config, std.testing.allocator, unknown, .{}));
}
+test "formal checker shape requires one pinned script argument" {
+ const hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ var config: Config = .{
+ .schema = config_schema,
+ .agent_name = "candidate",
+ .mutation_instruction = "change one behavior",
+ .mutator = .{ .program = "/bin/mutate", .sha256 = hash },
+ .evaluator = .{ .program = "/bin/evaluate", .sha256 = hash },
+ .evaluation_suite = .{ .path = "/suite", .sha256 = hash },
+ .cohort = .{ .provider = "test", .model = "test", .task_family = "test", .adapter_version = "v1", .verifier_version = "v1" },
+ .formal_check = .{
+ .checker = .{ .program = "/bin/python", .sha256 = hash, .args = &.{"/pinned/helper.py"}, .inputs = &.{.{ .path = "/pinned/helper.py", .sha256 = hash }} },
+ .pin = .{ .path = "/pinned/manifest.json", .sha256 = hash },
+ },
+ };
+ try validateConfig(config);
+ var formal = config.formal_check.?;
+ formal.checker.args = &.{"/other/helper.py"};
+ config.formal_check = formal;
+ try std.testing.expectError(error.InvalidFormalChecker, validateConfig(config));
+ formal.checker.args = &.{"/pinned/helper.py"};
+ formal.checker.inputs = &.{};
+ config.formal_check = formal;
+ try std.testing.expectError(error.InvalidFormalChecker, validateConfig(config));
+}
+
test "suite statistical units are additive, validated, and criticality-consistent" {
const legacy =
\\{"schema":"codegraff.learn.suite.v1","suite_id":"legacy","cases":[{"id":"one"},{"id":"two"}]}
diff --git a/src/learn_store_types.zig b/src/learn_store_types.zig
index 75b2810e..38323558 100644
--- a/src/learn_store_types.zig
+++ b/src/learn_store_types.zig
@@ -29,6 +29,14 @@ pub const Program = struct {
pass_env: []const []const u8 = &.{},
};
+/// Optional external formal baseline check. This binds a model-check result
+/// to a prompt fingerprint; it does not prove the prompt's behavior.
+pub const FormalCheck = struct {
+ checker: Program,
+ pin: PinnedFile,
+ timeout_ms: u64 = 900_000,
+};
+
pub const Suite = struct {
path: []const u8,
sha256: []const u8,
@@ -89,6 +97,7 @@ pub const Config = struct {
mutation_instruction: []const u8,
mutator: Program,
evaluator: Program,
+ formal_check: ?FormalCheck = null,
evaluation_suite: Suite,
holdout_suite: ?Suite = null,
limits: Limits = .{},
diff --git a/src/learn_store_validate.zig b/src/learn_store_validate.zig
index fc48f08b..176d2a52 100644
--- a/src/learn_store_validate.zig
+++ b/src/learn_store_validate.zig
@@ -61,6 +61,16 @@ pub fn validateConfig(config: Config) !void {
if (!validText(config.mutation_instruction, 4096)) return error.InvalidMutationInstruction;
try validateProgram(config.mutator);
try validateProgram(config.evaluator);
+ if (config.formal_check) |formal| {
+ try validateProgram(formal.checker);
+ if (formal.checker.args.len != 1 or formal.checker.inputs.len != 1 or
+ formal.checker.pass_env.len != 0 or
+ !std.mem.eql(u8, formal.checker.args[0], formal.checker.inputs[0].path))
+ return error.InvalidFormalChecker;
+ if (!std.fs.path.isAbsolute(formal.pin.path) or !validText(formal.pin.path, std.fs.max_path_bytes) or
+ !validId(formal.pin.sha256) or formal.timeout_ms == 0 or formal.timeout_ms > 900_000)
+ return error.InvalidFormalChecker;
+ }
try validateSuitePin(config.evaluation_suite);
if (config.holdout_suite) |suite| try validateSuitePin(suite);
diff --git a/src/learn_submit.zig b/src/learn_submit.zig
index e07f2ccc..db32ca8d 100644
--- a/src/learn_submit.zig
+++ b/src/learn_submit.zig
@@ -127,6 +127,9 @@ pub fn submitVerifiedRun(
run_id: []const u8,
run: eval.RunRecord,
) !SubmitResult {
+ // The signed receipt schema does not yet bind formal admission evidence.
+ if (config.formal_check != null or std.mem.eql(u8, run.schema, eval.formal_run_schema))
+ return error.FormalReceiptUnsupported;
try preflight(io, arena, environ);
var primary_count: usize = 0;
var holdout_count: usize = 0;
@@ -222,3 +225,41 @@ test "learning pass-rate grade uses repeated pair count" {
test {
_ = @import("learn_receipt.zig");
}
+
+test "formal learning runs cannot enter the unsigned receipt path" {
+ const id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+ var config: store.Config = .{
+ .schema = store.config_schema,
+ .agent_name = "candidate",
+ .mutation_instruction = "change one behavior",
+ .mutator = .{ .program = "/missing/mutator", .sha256 = id },
+ .evaluator = .{ .program = "/missing/evaluator", .sha256 = id },
+ .evaluation_suite = .{ .path = "/missing/suite", .sha256 = id },
+ .cohort = .{ .provider = "test", .model = "test", .task_family = "test", .adapter_version = "v1", .verifier_version = "v1" },
+ };
+ var run: eval.RunRecord = .{
+ .schema = eval.formal_run_schema,
+ .trial_id = id,
+ .nonce = id,
+ .created_unix_ms = 0,
+ .harness_version = "test",
+ .config_id = id,
+ .parent_genome_id = id,
+ .parent_generation = 0,
+ .parent_transaction_id = id,
+ .planned_candidates = 1,
+ .repetitions = 1,
+ .auto_requested = false,
+ .candidates = &.{},
+ .selected_genome_id = null,
+ };
+ var env = std.process.Environ.Map.init(std.testing.allocator);
+ defer env.deinit();
+ try std.testing.expectError(error.FormalReceiptUnsupported, submitVerifiedRun(std.testing.io, std.testing.allocator, std.testing.allocator, &env, config, id, run));
+ run.schema = eval.run_schema;
+ config.formal_check = .{
+ .checker = .{ .program = "/missing/checker", .sha256 = id },
+ .pin = .{ .path = "/missing/pin", .sha256 = id },
+ };
+ try std.testing.expectError(error.FormalReceiptUnsupported, submitVerifiedRun(std.testing.io, std.testing.allocator, std.testing.allocator, &env, config, id, run));
+}
diff --git a/src/main.zig b/src/main.zig
index 5532ad57..01b71ff5 100644
--- a/src/main.zig
+++ b/src/main.zig
@@ -582,7 +582,7 @@ test { // ── Unit tests (`zig build test`): pull in tests from imported modu
_ = .{ @import("acp_protocol.zig"), @import("subagent_mimo_tests.zig"), @import("mimo_effort_tests.zig"), @import("gateway_picker_catalog.zig") };
_ = @import("task_outcome.zig"); // goal-outcome telemetry events
_ = @import("learn_delete.zig"); // #303: its tests were dead until listed here
- _ = .{ @import("additional_tests.zig"), @import("req_stats.zig"), @import("exact_reply.zig"), @import("rlm_order_tests.zig"), @import("async_tool_policy.zig"), @import("agent_async_tools.zig"), @import("request_usage_attempts.zig") };
+ _ = .{ @import("additional_tests.zig"), @import("req_stats.zig"), @import("exact_reply.zig"), @import("rlm_order_tests.zig"), @import("async_tool_policy.zig"), @import("agent_async_tools.zig"), @import("request_usage_attempts.zig"), @import("learn_formal.zig") };
_ = @import("goal_pacing_autonomous_test.zig");
_ = @import("goal_state.zig");
_ = @import("goal_persist_tests.zig");
diff --git a/src/test_hooks.zig b/src/test_hooks.zig
index 4f5ca292..a3b17548 100644
--- a/src/test_hooks.zig
+++ b/src/test_hooks.zig
@@ -22,6 +22,7 @@
//! checking the guard still does.
// Learning / DGM promotion pipeline.
+const learn_checkpoint = @import("learn_checkpoint.zig");
const learn_holdout = @import("learn_holdout.zig");
const learn_receipt = @import("learn_receipt.zig");
const learn_report = @import("learn_report.zig");
@@ -235,6 +236,7 @@ const commands_sandbox = @import("commands_sandbox.zig");
const sandbox_tests = @import("sandbox_tests.zig");
test {
+ _ = learn_checkpoint;
_ = learn_holdout;
_ = learn_receipt;
_ = learn_report;
diff --git a/tests/learn_formal_e2e.py b/tests/learn_formal_e2e.py
new file mode 100644
index 00000000..5a16a3e1
--- /dev/null
+++ b/tests/learn_formal_e2e.py
@@ -0,0 +1,193 @@
+#!/usr/bin/env python3
+"""Offline CLI wiring test for the optional native learning formal gate.
+
+The helper is deliberately a pinned fixture, not a TLC proof. This test checks
+that native command boundaries invoke and reject its result as configured.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+import shutil
+import sys
+import tempfile
+
+from learn_e2e import (
+ EVALUATOR_BODY,
+ MUTATOR_BODY,
+ invoke,
+ json_bytes,
+ raw_sha256,
+ run_id_from,
+ status,
+ write_interpreter_shim,
+ write_private,
+)
+
+
+def setup(graff: Path, root: Path) -> tuple[Path, Path, dict[str, str], Path, Path, Path]:
+ tools = root / "tools"
+ tools.mkdir(mode=0o700)
+ binary = Path("/bin/sh").resolve()
+ binary_pin = {"path": str(binary), "sha256": raw_sha256(binary)}
+ mutator_marker = root / "mutator-called"
+ evaluator_fail = root / "evaluator-fail"
+ mutator_body = MUTATOR_BODY.replace(
+ "operation, request_path, response_path = sys.argv[1:4]",
+ f"binary = pathlib.Path(sys.argv.pop(1))\nassert binary.is_file()\n"
+ f"pathlib.Path({str(mutator_marker)!r}).write_text('called')\n"
+ "operation, request_path, response_path = sys.argv[1:4]",
+ )
+ evaluator_body = EVALUATOR_BODY.replace(
+ "operation, request_path, response_path = sys.argv[1:4]",
+ f"binary = pathlib.Path(sys.argv.pop(1))\nassert binary.is_file()\n"
+ f"if pathlib.Path({str(evaluator_fail)!r}).exists(): raise SystemExit(9)\n"
+ "operation, request_path, response_path = sys.argv[1:4]",
+ )
+ python = Path(sys.executable).resolve()
+ mutator, mutator_source = write_interpreter_shim(tools, "formal-mutator", python, mutator_body)
+ evaluator, evaluator_source = write_interpreter_shim(tools, "formal-evaluator", python, evaluator_body)
+ for program in (mutator, evaluator):
+ program["args"].append(str(binary))
+ program["inputs"].append(binary_pin)
+ mutator["pass_env"] = ["FIXTURE_MUTATOR_SOURCE"]
+ evaluator["pass_env"] = ["FIXTURE_EVALUATOR_SOURCE"]
+
+ helper_control = root / "helper-control"
+ helper_count = root / "helper-count"
+ helper = tools / "formal-helper.py"
+ helper_body = f'''import json, pathlib, sys
+control = pathlib.Path({str(helper_control)!r})
+count = pathlib.Path({str(helper_count)!r})
+count.write_text(str(int(count.read_text()) + 1 if count.exists() else 1))
+mode = control.read_text() if control.exists() else "good"
+if mode == "fail": raise SystemExit(7)
+print(json.dumps({{
+ "schema": "codegraff.dgm.formal-check.v1", "ok": mode != "false",
+ "candidate_prompt_sha256": sys.argv[-1],
+ "formal_identity_sha256": "a" * 64,
+ "checker_output_sha256": "b" * 64,
+ "binary_sha256": {binary_pin['sha256']!r},
+}}))
+'''
+ write_private(helper, helper_body.encode())
+ formal_pin = tools / "formal-pin.json"
+ write_private(formal_pin, json_bytes({"binary": binary_pin}))
+ score_key = root / "score-key"
+ write_private(score_key, b"fixture key\n")
+
+ primary = tools / "primary.json"
+ holdout = tools / "holdout.json"
+ for path, suite_id in ((primary, "formal-primary"), (holdout, "formal-holdout")):
+ write_private(path, json_bytes({
+ "schema": "codegraff.learn.suite.v1", "suite_id": suite_id,
+ "cases": [{"id": f"{suite_id}-{index}"} for index in range(6)],
+ }))
+ parent = root / "parent.md"
+ write_private(parent, b"Initial fixture policy.\n")
+ config = root / "config.json"
+ write_private(config, json_bytes({
+ "schema": "codegraff.learn.config.v1",
+ "agent_name": "formal-fixture",
+ "mutation_instruction": "append the fixture policy sentence",
+ "mutator": mutator, "evaluator": evaluator,
+ "formal_check": {
+ "checker": {"program": str(python), "sha256": raw_sha256(python),
+ "args": [str(helper.resolve())],
+ "inputs": [{"path": str(helper.resolve()), "sha256": raw_sha256(helper)}]},
+ "pin": {"path": str(formal_pin.resolve()), "sha256": raw_sha256(formal_pin)},
+ "timeout_ms": 10000,
+ },
+ "evaluation_suite": {"path": str(primary.resolve()), "sha256": raw_sha256(primary)},
+ "holdout_suite": {"path": str(holdout.resolve()), "sha256": raw_sha256(holdout)},
+ "gate": {"alpha_ppm": 50000, "minimum_delta_ppm": 100000,
+ "minimum_pairs": 6, "default_candidates": 1, "default_repetitions": 3},
+ "auto": {"enabled": True},
+ "cohort": {"provider": "fixture", "model": "fixture", "task_family": "local-e2e",
+ "adapter_version": "v1", "verifier_version": "v1"},
+ }))
+ env = {**os.environ, "GRAFF_NO_TELEMETRY": "1", "GRAFF_LEARN_AUTO": "off",
+ "GRAFF_SCORE_KEY_FILE": str(score_key.resolve()),
+ "FIXTURE_MUTATOR_SOURCE": str(mutator_source.resolve()),
+ "FIXTURE_EVALUATOR_SOURCE": str(evaluator_source.resolve())}
+ workspace = root / "manual"
+ workspace.mkdir(mode=0o700)
+ invoke(graff, workspace, env, "init", "--parent", str(parent.resolve()),
+ "--config", str(config.resolve()))
+ return workspace, parent, env, helper_control, helper_count, evaluator_fail
+
+
+def exercise(graff: Path, root: Path) -> None:
+ workspace, parent, env, control, count, eval_fail = setup(graff, root)
+ marker = root / "mutator-called"
+ refused = invoke(graff, workspace, env, "run", "--submit", succeeds=False)
+ assert "FormalReceiptUnsupported" in refused.stderr
+ assert not marker.exists()
+
+ control.write_text("fail")
+ failed = invoke(graff, workspace, env, "run", succeeds=False)
+ assert "FormalCheckFailed" in failed.stderr and not marker.exists()
+ control.write_text("false")
+ false = invoke(graff, workspace, env, "run", succeeds=False)
+ assert "InvalidFormalCheckResult" in false.stderr and not marker.exists()
+ control.unlink()
+
+ eval_fail.touch()
+ failed_eval = invoke(graff, workspace, env, "run", succeeds=False)
+ assert "ProcessFailed" in failed_eval.stderr and marker.exists()
+ pending_path = workspace / ".graff" / "learn" / "refs" / "pending.json"
+ pending = json.loads(pending_path.read_text())
+ assert pending["schema"] == "codegraff.learn.pending.v2"
+ original_evidence = pending["formal_admission_evidence_id"]
+ pending["formal_admission_evidence_id"] = "d" * 64
+ write_private(pending_path, json_bytes(pending))
+ rejected = invoke(graff, workspace, env, "run", "--resume", succeeds=False)
+ assert "FileNotFound" in rejected.stderr
+ pending["formal_admission_evidence_id"] = original_evidence
+ write_private(pending_path, json_bytes(pending))
+ eval_fail.unlink()
+ manual_run = run_id_from(invoke(graff, workspace, env, "run", "--resume").stdout)
+ run_path = workspace / ".graff" / "learn" / "runs" / f"{manual_run}.json"
+ run = json.loads(run_path.read_text())
+ assert run["schema"] == "codegraff.learn.run.v4"
+ assert run["formal_admission_evidence_id"] == original_evidence
+ assert run["formal_selection_evidence_id"]
+ refused = invoke(graff, workspace, env, "submit", manual_run, succeeds=False)
+ assert "FormalReceiptUnsupported" in refused.stderr
+ before_manual = int(count.read_text())
+ invoke(graff, workspace, env, "promote", manual_run)
+ assert int(count.read_text()) >= before_manual + 2
+ assert status(graff, workspace, env)["generation"] == 1
+
+ automatic = root / "automatic"
+ automatic.mkdir(mode=0o700)
+ invoke(graff, automatic, env, "init", "--parent", str(parent.resolve()),
+ "--config", str((root / "config.json").resolve()))
+ before_auto = int(count.read_text())
+ auto_run = run_id_from(invoke(graff, automatic, env, "run", "--auto").stdout)
+ assert auto_run
+ assert int(count.read_text()) >= before_auto + 4
+ assert status(graff, automatic, env)["generation"] == 1
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--graff", type=Path, default=Path("zig-out/bin/graff"))
+ args = parser.parse_args()
+ graff = args.graff.resolve(strict=True)
+ root = Path(tempfile.mkdtemp(prefix="codegraff-learn-formal."))
+ try:
+ exercise(graff, root)
+ except Exception:
+ print(f"formal learning fixture preserved at {root}", file=sys.stderr)
+ raise
+ shutil.rmtree(root)
+ print("formal learning CLI boundaries passed")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())