graff — a fast agentic coding harness in Zig: multi-provider, MCP, workflows, DGM evolution loop, TS/Python SDKs
Python
Gate local learning with pinned formal baseline
Rach Pradhan85484c876737
diff --git a/build.zig b/build.zigindex e90eee0c..42bb594d 100644--- a/build.zig+++ b/build.zig@@ -155,6 +155,9 @@ pub fn build(b: *std.Build) void {const subagent_resume_test = b.addSystemCommand(&.{ "python3", "scripts/test-subagent-resume.py" });subagent_resume_test.addArtifactArg(exe);test_step.dependOn(&subagent_resume_test.step);+ const learn_formal_test = b.addSystemCommand(&.{ "python3", "tests/learn_formal_e2e.py", "--graff" });+ learn_formal_test.addArtifactArg(exe);+ test_step.dependOn(&learn_formal_test.step);test_step.dependOn(&b.addSystemCommand(&.{ "python3", "graff-evals/test_measurement.py" }).step);test_step.dependOn(&b.addSystemCommand(&.{ "python3", "scripts/test-eval-cache-affinity.py" }).step);diff --git a/docs/adr/0194-native-learning-formal-baseline.md b/docs/adr/0194-native-learning-formal-baseline.mdnew file mode 100644index 00000000..2b9932f3--- /dev/null+++ b/docs/adr/0194-native-learning-formal-baseline.md@@ -0,0 +1,36 @@+# 0194. Native learning checks a pinned formal baseline++Status: accepted 2026-09-23++## Context++The local learning store already pins its adapters and evaluation suites, and+rechecks their immutable evidence before promotion. A separate, bounded formal+checker can establish properties of the shared engine's modeled control flow.+Its success is not evidence that a newly generated prompt is correct. The+existing signed aggregate receipt also has no field for a formal check.++## Decision++An optional `formal_check` configuration pins the checker executable, its+script, and its input manifest. The manifest's engine binary must be an exact+pinned input and argument to both learning adapters. Before mutation, and+again for a selected genome, the checker must succeed for the respective+prompt fingerprint. The store saves content-addressed check evidence in a+versioned pending record and run record. Resume and manual or automatic+promotion re-run the pinned checker and reject missing, stale, or mismatched+evidence. Default configurations do not enable this gate.++Formal-enabled trials stay local. `run --submit`, `learn submit`, and direct+submission reject them until a signed receipt version binds the formal+evidence. The legacy fleet persona promotion path remains separate and has no+formal gate.++## Consequences++The checker consumes local time at admission, selection, resume, and+promotion. Operators must supply its signing key file and trust the pinned+checker, adapters, and manifest; pinning proves their byte identity, not that+an adapter actually uses its binary argument. The model-check result concerns+the pinned shared engine baseline. Behavioral correctness of a candidate+prompt still depends on the normal paired evaluations and holdout gate.diff --git a/docs/adr/README.md b/docs/adr/README.mdindex b0467a25..5c4a157b 100644--- a/docs/adr/README.md+++ b/docs/adr/README.md@@ -219,6 +219,7 @@ record only when you need the evidence or the edge cases.| [0191](0191-acp-effort-is-session-configuration.md) | ACP thought-level options follow the active model and share `/effort` state; changes received during a turn apply after that turn. || [0192](0192-mimo-thinking-is-a-binary-setting.md) | MiMo exposes Off/On, preserves legacy positive settings as On, and uses the documented thinking control on each wire. || [0193](0193-jev-selects-effort-only.md) | Optional Jev selects only supported session effort at the next request boundary; arbitrary judgments are not exposed. |+| [0194](0194-native-learning-formal-baseline.md) | Optional native learning admits and promotes only with a fresh pinned formal baseline check bound to the evaluated binary; remote submission waits for a compatible signed receipt. || [0195](0195-embedded-browser-replaces-kuri.md) | Desktop browser uses embedded Chromium, web preview uses the paired extension, and CLI fetch uses built-in HTTP without a sidecar. || [0198](0198-project-choice-survives-session-isolation.md) | New chats inherit the chosen project while ACP reports each worker checkout; generated worktrees are rooted at the main checkout. || [0199](0199-generated-checkouts-belong-to-project-history.md) | Generated worktree activity rolls up to the owning project in automatic suggestions; explicitly saved checkouts stay selectable. |diff --git a/docs/local-learning.md b/docs/local-learning.mdindex a0e8647e..c2f608a6 100644--- a/docs/local-learning.md+++ b/docs/local-learning.md@@ -316,6 +316,43 @@ as immutable. Unknown JSON fields are rejected.Cohort fields are exact labels used to prevent unlike evaluations from beingcompared as though they were interchangeable.+### Optional pinned formal baseline++An operator can add `formal_check` to an explicit configuration to require a+local model-check result before mutation, after selecting a candidate, and on+checkpoint resume or promotion. It is disabled by default. The checker uses+the read-only `--check` interface described in+[the formal harness](../formal/README.md), with a pinned manifest and helper:++```json+"formal_check": {+ "checker": {+ "program": "/absolute/python3",+ "sha256": "<Python executable SHA-256>",+ "args": ["/absolute/dgm_formal_gate.py"],+ "inputs": [{"path": "/absolute/dgm_formal_gate.py", "sha256": "<helper SHA-256>"}]+ },+ "pin": {"path": "/absolute/formal-pin.json", "sha256": "<manifest SHA-256>"},+ "timeout_ms": 900000+}+```++The manifest's `binary.path` and SHA-256 must match a pinned input passed as+an argument to both the mutator and evaluator. The normal adapter runner+copies those verified bytes into private scratch space. Set+`GRAFF_SCORE_KEY_FILE` to an absolute path for the helper's pinned signing-key+identity check; a missing key fails the trial before mutation. The checker and+manifest are operator trust inputs, and the formal result describes the+shared engine model at that pinned binary, not the candidate prompt's behavior.++Formal trials write `pending.v2` and `run.v4` records with immutable evidence+IDs. Resume and promotion reject mismatches and re-run the checker. This adds+local compute time and does not change the statistical or holdout gates.+Formal trials cannot use `run --submit` or `learn submit` because the current+signed receipt does not bind formal evidence; background auto trials remain+local even when aggregate contribution is enabled. Legacy fleet persona+promotion is a separate path and is not covered by this gate.+### Pinned program snapshotsThe engine does not hash a tool and then reopen that original pathname fordiff --git a/src/learn_auto.zig b/src/learn_auto.zigindex fd4e928f..d3a704ad 100644--- a/src/learn_auto.zig+++ b/src/learn_auto.zig@@ -228,7 +228,10 @@ pub fn maybeStart(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.// Contribution needs a signing key as well as consent. Asking for// `--submit` without one fails the whole trial in preflight, so a machine// that cannot sign simply learns locally.- const contribute = options.contribute and scoring.loadScoreKey(io, arena, environ) != null;+ // Formal evidence is not represented by the current signed remote receipt.+ // Keep this trial local even when ordinary aggregate contribution is on.+ const contribute = config.value.formal_check == null and options.contribute and+ scoring.loadScoreKey(io, arena, environ) != null;const resumed = hasPendingTrial(&store);const exe_path = std.process.executablePathAlloc(io, gpa) catch return .{ .skipped = .unconfigured };defer gpa.free(exe_path);diff --git a/src/learn_checkpoint.zig b/src/learn_checkpoint.zigindex 1a769f3d..144197ea 100644--- a/src/learn_checkpoint.zig+++ b/src/learn_checkpoint.zig@@ -10,6 +10,7 @@ const eval = @import("learn_eval.zig");const store_mod = @import("learn_store.zig");pub const schema = "codegraff.learn.pending.v1";+pub const formal_schema = "codegraff.learn.pending.v2";pub const file_name = "pending.json";pub const Record = struct {@@ -25,6 +26,7 @@ pub const Record = struct {planned_candidates: usize,repetitions: usize,auto_requested: bool,+ formal_admission_evidence_id: ?[]const u8 = null,primary_baseline: ?eval.PrimaryBaselineRecord = null,candidates: []const eval.CandidateRecord,};@@ -41,12 +43,17 @@ fn validCandidate(candidate: eval.CandidateRecord) bool {}pub fn validate(record: Record) !void {- if (!std.mem.eql(u8, record.schema, schema) or+ const formal = std.mem.eql(u8, record.schema, formal_schema);+ if ((!std.mem.eql(u8, record.schema, schema) and !formal) or!store_mod.validId(record.trial_id) or!store_mod.validId(record.nonce) or!store_mod.validId(record.config_id) or!store_mod.validId(record.parent_genome_id) or!store_mod.validId(record.parent_transaction_id)) return error.InvalidPendingRun;+ if (formal) {+ if (record.formal_admission_evidence_id == null or+ !store_mod.validId(record.formal_admission_evidence_id.?)) return error.InvalidPendingRun;+ } else if (record.formal_admission_evidence_id != null) return error.InvalidPendingRun;if (record.harness_version.len == 0 or record.harness_version.len > 128 orrecord.planned_candidates == 0 or record.planned_candidates > 16 orrecord.candidates.len != record.planned_candidates or@@ -84,3 +91,35 @@ pub fn clear(store: *store_mod.Store) !void {};try store_mod.syncDirectory(store.io, store.refs);}++test "formal checkpoint requires admission evidence and legacy rejects it" {+ const id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ const candidate: eval.CandidateRecord = .{+ .genome_id = id,+ .mutation = .{ .seed = id, .request_evidence_id = id, .response_evidence_id = id },+ .primary = null,+ .holdout = null,+ .eligible = false,+ .reason = "pending",+ };+ var record: Record = .{+ .schema = formal_schema,+ .trial_id = id,+ .nonce = id,+ .created_unix_ms = 0,+ .harness_version = "test",+ .config_id = id,+ .parent_genome_id = id,+ .parent_generation = 0,+ .parent_transaction_id = id,+ .planned_candidates = 1,+ .repetitions = 1,+ .auto_requested = false,+ .candidates = &.{candidate},+ };+ try std.testing.expectError(error.InvalidPendingRun, validate(record));+ record.formal_admission_evidence_id = id;+ try validate(record);+ record.schema = schema;+ try std.testing.expectError(error.InvalidPendingRun, validate(record));+}diff --git a/src/learn_cli.zig b/src/learn_cli.zigindex 44e44dbf..695d1482 100644--- a/src/learn_cli.zig+++ b/src/learn_cli.zig@@ -15,6 +15,7 @@ const diagnostics = @import("learn_diagnostics.zig");const mutation_verify = @import("learn_mutation_verify.zig");const primary = @import("learn_primary.zig");const learn_run = @import("learn_run.zig");+const formal_gate = @import("learn_formal.zig");const learn_delete = @import("learn_delete.zig");const submit = @import("learn_submit.zig");const tournament = @import("learn_tournament.zig");@@ -239,8 +240,10 @@ pub fn parse(args: []const []const u8) !Parsed {}fn verifyRun(+ gpa: Allocator,arena: Allocator,io: Io,+ environ: *const std.process.Environ.Map,store: *store_mod.Store,config: store_mod.LoadedConfig,run_id: []const u8,@@ -259,7 +262,8 @@ fn verifyRun(_ = try store.readGenome(arena, run.parent_genome_id, config.value.limits.genome_bytes);try learn_init.verifyPins(io, arena, config.value);- const current_schema = std.mem.eql(u8, run.schema, eval.run_schema);+ const current_schema = std.mem.eql(u8, run.schema, eval.run_schema) or+ std.mem.eql(u8, run.schema, eval.formal_run_schema);var baseline_response: ?eval.PrimaryBaselineResponse = null;if (current_schema) if (run.primary_baseline) |baseline| {baseline_response = try primary.verifyBaseline(@@ -342,6 +346,13 @@ fn verifyRun(if (primary_winner) |id| if (!std.mem.eql(u8, id, run.primary_winner_genome_id.?)) return error.SelectionMismatch;if ((finalized.selected_genome_id == null) != (run.selected_genome_id == null)) return error.SelectionMismatch;if (finalized.selected_genome_id) |id| if (!std.mem.eql(u8, id, run.selected_genome_id.?)) return error.SelectionMismatch;+ if (config.value.formal_check) |formal| {+ if (!std.mem.eql(u8, run.schema, eval.formal_run_schema)) return error.FormalRunRequired;+ try formal_gate.verify(gpa, arena, io, environ, store, formal, run.formal_admission_evidence_id orelse return error.FormalEvidenceMissing, &config.id, run.trial_id, "admission", try store.readGenome(arena, run.parent_genome_id, config.value.limits.genome_bytes));+ if (finalized.selected_genome_id) |selected| {+ try formal_gate.verify(gpa, arena, io, environ, store, formal, run.formal_selection_evidence_id orelse return error.FormalEvidenceMissing, &config.id, run.trial_id, "selection", try store.readGenome(arena, selected, config.value.limits.genome_bytes));+ }+ } else if (std.mem.eql(u8, run.schema, eval.formal_run_schema)) return error.FormalConfigRequired;return .{ .run = run, .selected = finalized.selected_genome_id };}@@ -364,6 +375,7 @@ fn runCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.procif (args.auto and !config.value.auto.enabled) return error.AutoNotEnabled;if (args.auto and config.value.holdout_suite == null) return error.AutoRequiresHoldout;+ if (args.submit and config.value.formal_check != null) return error.FormalReceiptUnsupported;if (args.submit) try submit.preflight(io, arena, environ);// The adapters see a scrubbed environment, so a `graff login` credential@@ -383,7 +395,7 @@ fn runCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.procif (result.selected_genome_id) |genome_id| {try out.print("selected {s}\n", .{genome_id});if (args.auto) {- const verified = try verifyRun(arena, io, &store, config, &result.run_id);+ const verified = try verifyRun(gpa, arena, io, environ, &store, config, &result.run_id);if (verified.selected == null or !std.mem.eql(u8, verified.selected.?, genome_id)) return error.SelectionMismatch;const current = try store.loadActive(arena, config);if (!activeMatchesRun(current.ref, verified.run)) return error.ActiveParentChanged;@@ -395,7 +407,7 @@ fn runCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.proc} else try out.writeAll("no unique candidate produced primary evidence; active genome unchanged\n");}-fn statusCommand(arena: Allocator, io: Io, timeout_ms: u64, out: *Io.Writer, verify_all: bool) !void {+fn statusCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.process.Environ.Map, timeout_ms: u64, out: *Io.Writer, verify_all: bool) !void {var store = try store_mod.Store.openAt(io, Io.Dir.cwd());defer store.deinit();var lock = try store.acquireLock(timeout_ms);@@ -405,7 +417,13 @@ fn statusCommand(arena: Allocator, io: Io, timeout_ms: u64, out: *Io.Writer, verconst pending = try checkpoint.load(arena, &store);if (verify_all) {try learn_init.verifyPins(io, arena, config.value);- if (pending) |item| _ = try learn_run.restorePending(arena, io, &store, config, active, item);+ if (pending) |item| {+ if (config.value.formal_check) |formal| {+ const id = item.formal_admission_evidence_id orelse return error.FormalEvidenceMissing;+ try formal_gate.verify(gpa, arena, io, environ, &store, formal, id, &config.id, item.trial_id, "admission", active.genome);+ } else if (std.mem.eql(u8, item.schema, checkpoint.formal_schema)) return error.FormalConfigRequired;+ _ = try learn_run.restorePending(arena, io, &store, config, active, item);+ }}const Status = struct {schema: []const u8 = "codegraff.learn.status.v1",@@ -446,14 +464,14 @@ fn countCompleted(candidates: []const eval.CandidateRecord, holdouts: bool) usizreturn count;}-fn promoteCommand(gpa: Allocator, arena: Allocator, io: Io, args: PromoteArgs, out: *Io.Writer) !void {+fn promoteCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.process.Environ.Map, args: PromoteArgs, out: *Io.Writer) !void {var store = try store_mod.Store.openAt(io, Io.Dir.cwd());defer store.deinit();var lock = try store.acquireLock(args.lock_timeout_ms);defer lock.deinit();const config = try store.loadConfig(arena);const active = try store.loadActive(arena, config);- const verified = try verifyRun(arena, io, &store, config, args.run_id);+ const verified = try verifyRun(gpa, arena, io, environ, &store, config, args.run_id);const selected = verified.selected orelse return error.NoPromotableCandidate;if (!activeMatchesRun(active.ref, verified.run)) return error.ActiveParentChanged;try store.activate(gpa, active.ref, &config.id, selected, args.run_id, "promote", config.value.limits.genome_bytes, util.unixMs(io));@@ -466,7 +484,8 @@ fn submitCommand(gpa: Allocator, arena: Allocator, io: Io, environ: *const std.pvar lock = try store.acquireLock(args.lock_timeout_ms);defer lock.deinit();const config = try store.loadConfig(arena);- const verified = try verifyRun(arena, io, &store, config, args.run_id);+ if (config.value.formal_check != null) return error.FormalReceiptUnsupported;+ const verified = try verifyRun(gpa, arena, io, environ, &store, config, args.run_id);const sent = try submit.submitVerifiedRun(io, gpa, arena, environ, config.value, args.run_id, verified.run);try out.print("submitted {d} signed aggregate grade(s) for run {s}; prompt text stayed local\n", .{ sent.grades, args.run_id });}@@ -520,11 +539,11 @@ pub fn command(io: Io, gpa: Allocator, arena: Allocator, init: std.process.Init,elsetry learn_init.fromPaths(gpa, arena, io, args, out),.run => |args| try runCommand(gpa, arena, io, init.environ_map, args, out),- .status => |args| try statusCommand(arena, io, args.lock_timeout_ms, out, false),+ .status => |args| try statusCommand(gpa, arena, io, init.environ_map, args.lock_timeout_ms, out, false),.submit => |args| try submitCommand(gpa, arena, io, init.environ_map, args, out),.delete_remote => |args| try learn_delete.command(io, gpa, arena, init.environ_map, args.run_id, args.lock_timeout_ms, out),- .verify => |args| try statusCommand(arena, io, args.lock_timeout_ms, out, true),- .promote => |args| try promoteCommand(gpa, arena, io, args, out),+ .verify => |args| try statusCommand(gpa, arena, io, init.environ_map, args.lock_timeout_ms, out, true),+ .promote => |args| try promoteCommand(gpa, arena, io, init.environ_map, args, out),.rollback => |args| try rollbackCommand(gpa, arena, io, args, out),.hash => |args| {const digest = try store_mod.hashFileNoFollow(io, args.path);diff --git a/src/learn_eval.zig b/src/learn_eval.zigindex 95f0bcf2..390e5a8e 100644--- a/src/learn_eval.zig+++ b/src/learn_eval.zig@@ -25,6 +25,7 @@ pub const primary_evaluation_request_schema = protocol.primary_evaluation_requespub const primary_evaluation_response_schema = protocol.primary_evaluation_response_schema;pub const legacy_run_schema = protocol.legacy_run_schema;pub const run_schema = protocol.run_schema;+pub const formal_run_schema = protocol.formal_run_schema;pub const GenomeRef = protocol.GenomeRef;pub const MutationRequest = protocol.MutationRequest;pub const MutationResponse = protocol.MutationResponse;@@ -532,13 +533,19 @@ pub fn verifyComparison(pub fn validateRun(run: RunRecord) !void {const current = std.mem.eql(u8, run.schema, run_schema);+ const formal = std.mem.eql(u8, run.schema, formal_run_schema);const legacy = std.mem.eql(u8, run.schema, legacy_run_schema);- if ((!current and !legacy) or !store_mod.validId(run.trial_id) or !store_mod.validId(run.nonce) or !store_mod.validId(run.config_id) or !store_mod.validId(run.parent_genome_id) or !store_mod.validId(run.parent_transaction_id)) return error.InvalidRun;+ if ((!current and !formal and !legacy) or !store_mod.validId(run.trial_id) or !store_mod.validId(run.nonce) or !store_mod.validId(run.config_id) or !store_mod.validId(run.parent_genome_id) or !store_mod.validId(run.parent_transaction_id)) return error.InvalidRun;+ if (formal) {+ if (run.formal_admission_evidence_id == null or !store_mod.validId(run.formal_admission_evidence_id.?) or+ (run.selected_genome_id == null) != (run.formal_selection_evidence_id == null)) return error.InvalidRun;+ if (run.formal_selection_evidence_id) |id| if (!store_mod.validId(id)) return error.InvalidRun;+ } else if (run.formal_admission_evidence_id != null or run.formal_selection_evidence_id != null) return error.InvalidRun;if (run.planned_candidates == 0 or run.planned_candidates > 16 or run.candidates.len != run.planned_candidates) return error.InvalidRun;if (run.repetitions == 0 or run.repetitions > 100 or run.harness_version.len == 0 or run.harness_version.len > 128) return error.InvalidRun;if (run.primary_winner_genome_id) |id| if (!store_mod.validId(id)) return error.InvalidRun;if (run.selected_genome_id) |id| if (!store_mod.validId(id)) return error.InvalidRun;- if (current) {+ if (current or formal) {if (run.primary_baseline) |baseline| {if (!store_mod.validId(baseline.suite_sha256) or !store_mod.validId(baseline.request_evidence_id) or !store_mod.validId(baseline.response_evidence_id)) return error.InvalidRun;}diff --git a/src/learn_eval_tests.zig b/src/learn_eval_tests.zigindex 91007026..eb6b46cb 100644--- a/src/learn_eval_tests.zig+++ b/src/learn_eval_tests.zig@@ -26,6 +26,43 @@ const executable_permissions = learn_eval.executable_permissions;const dir_permissions = learn_eval.dir_permissions;const builtin = @import("builtin");+test "formal run schema binds admission and selected evidence" {+ const id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ const candidate: learn_eval.CandidateRecord = .{+ .genome_id = id,+ .mutation = .{ .seed = id, .request_evidence_id = id, .response_evidence_id = id },+ .primary = null,+ .holdout = null,+ .eligible = false,+ .reason = "pending",+ };+ var run: learn_eval.RunRecord = .{+ .schema = learn_eval.formal_run_schema,+ .trial_id = id,+ .nonce = id,+ .created_unix_ms = 0,+ .harness_version = "test",+ .config_id = id,+ .parent_genome_id = id,+ .parent_generation = 0,+ .parent_transaction_id = id,+ .planned_candidates = 1,+ .repetitions = 1,+ .auto_requested = false,+ .candidates = &.{candidate},+ .selected_genome_id = null,+ };+ try std.testing.expectError(error.InvalidRun, learn_eval.validateRun(run));+ run.formal_admission_evidence_id = id;+ try learn_eval.validateRun(run);+ run.selected_genome_id = id;+ try std.testing.expectError(error.InvalidRun, learn_eval.validateRun(run));+ run.formal_selection_evidence_id = id;+ try learn_eval.validateRun(run);+ run.schema = learn_eval.run_schema;+ try std.testing.expectError(error.InvalidRun, learn_eval.validateRun(run));+}+test "paired tail has known exact values" {try std.testing.expectApproxEqAbs(@as(f64, 0.5), pairedTail(1, 0), 1e-12);try std.testing.expectApproxEqAbs(@as(f64, 0.0546875), pairedTail(8, 2), 1e-12);diff --git a/src/learn_eval_types.zig b/src/learn_eval_types.zigindex 7681cbc1..0c406c8d 100644--- a/src/learn_eval_types.zig+++ b/src/learn_eval_types.zig@@ -10,6 +10,7 @@ pub const primary_evaluation_request_schema = "codegraff.learn.primary-evaluatiopub const primary_evaluation_response_schema = "codegraff.learn.primary-evaluation.response.v1";pub const legacy_run_schema = "codegraff.learn.run.v2";pub const run_schema = "codegraff.learn.run.v3";+pub const formal_run_schema = "codegraff.learn.run.v4";pub const GenomeRef = struct {id: []const u8,@@ -238,6 +239,8 @@ pub const RunRecord = struct {planned_candidates: usize,repetitions: usize,auto_requested: bool,+ formal_admission_evidence_id: ?[]const u8 = null,+ formal_selection_evidence_id: ?[]const u8 = null,primary_baseline: ?PrimaryBaselineRecord = null,candidates: []const CandidateRecord,/// Best primary-suite contender. This remains populated when the evidencediff --git a/src/learn_formal.zig b/src/learn_formal.zignew file mode 100644index 00000000..e8eb0c0a--- /dev/null+++ b/src/learn_formal.zig@@ -0,0 +1,333 @@+//! Optional, pinned model-check baseline for native prompt-policy learning.+//! The result concerns shared engine source, not the candidate prompt's behavior.+const std = @import("std");+const Io = std.Io;+const Allocator = std.mem.Allocator;+const store_mod = @import("learn_store.zig");+const jobs = @import("jobs.zig");+const util = @import("util.zig");++pub const evidence_schema = "codegraff.learn.formal-evidence.v1";+const check_schema = "codegraff.dgm.formal-check.v1";+const stdout_cap = 4096;+const stderr_cap = 4096;++pub const Evidence = struct {+ schema: []const u8 = evidence_schema,+ phase: []const u8,+ config_id: []const u8,+ trial_id: []const u8,+ candidate_prompt_sha256: []const u8,+ formal_identity_sha256: []const u8,+ checker_output_sha256: []const u8,+ binary_sha256: []const u8,+ formal_pin_sha256: []const u8,+ checker_sha256: []const u8,+ helper_sha256: []const u8,+};++fn validHash(value: []const u8) bool {+ return store_mod.validId(value);+}++fn promptHash(prompt: []const u8) [64]u8 {+ var digest: [32]u8 = undefined;+ std.crypto.hash.sha2.Sha256.hash(prompt, &digest, .{});+ return std.fmt.bytesToHex(digest, .lower);+}++fn checkedString(obj: std.json.ObjectMap, key: []const u8) ![]const u8 {+ const value = util.strFieldObj(obj, key) orelse return error.InvalidFormalCheckResult;+ if (!validHash(value)) return error.InvalidFormalCheckResult;+ return value;+}++fn parseResult(arena: Allocator, raw: []const u8, candidate_sha: []const u8, binary_sha: []const u8) !Evidence {+ const parsed = try std.json.parseFromSliceLeaky(std.json.Value, arena, raw, .{});+ if (parsed != .object or parsed.object.count() != 6 or+ !std.mem.eql(u8, util.strFieldObj(parsed.object, "schema") orelse "", check_schema))+ return error.InvalidFormalCheckResult;+ const ok = parsed.object.get("ok") orelse return error.InvalidFormalCheckResult;+ if (ok != .bool or !ok.bool) return error.InvalidFormalCheckResult;+ const got_candidate = try checkedString(parsed.object, "candidate_prompt_sha256");+ const got_binary = try checkedString(parsed.object, "binary_sha256");+ if (!std.mem.eql(u8, got_candidate, candidate_sha) or !std.mem.eql(u8, got_binary, binary_sha))+ return error.FormalCheckIdentityMismatch;+ return .{+ .phase = "",+ .config_id = "",+ .trial_id = "",+ .candidate_prompt_sha256 = got_candidate,+ .formal_identity_sha256 = try checkedString(parsed.object, "formal_identity_sha256"),+ .checker_output_sha256 = try checkedString(parsed.object, "checker_output_sha256"),+ .binary_sha256 = got_binary,+ .formal_pin_sha256 = "",+ .checker_sha256 = "",+ .helper_sha256 = "",+ };+}++fn binaryPin(arena: Allocator, io: Io, formal: store_mod.FormalCheck) !store_mod.PinnedFile {+ const bytes = try store_mod.readPinnedFileAlloc(io, arena, formal.pin, store_mod.max_config_bytes);+ const parsed = try std.json.parseFromSliceLeaky(std.json.Value, arena, bytes, .{});+ if (parsed != .object) return error.InvalidFormalPin;+ const binary = parsed.object.get("binary") orelse return error.InvalidFormalPin;+ if (binary != .object) return error.InvalidFormalPin;+ const sha = util.strFieldObj(binary.object, "sha256") orelse return error.InvalidFormalPin;+ const path = util.strFieldObj(binary.object, "path") orelse return error.InvalidFormalPin;+ if (!validHash(sha) or !std.fs.path.isAbsolute(path)) return error.InvalidFormalPin;+ return .{ .path = path, .sha256 = sha };+}++fn matchesAdapter(program: store_mod.Program, binary: store_mod.PinnedFile) bool {+ var pinned = false;+ var passed = false;+ for (program.inputs) |input| {+ if (std.mem.eql(u8, input.path, binary.path) and std.mem.eql(u8, input.sha256, binary.sha256))+ pinned = true;+ }+ for (program.args) |arg| {+ if (std.mem.eql(u8, arg, binary.path)) passed = true;+ }+ return pinned and passed;+}++pub fn verifyBinaryBinding(arena: Allocator, io: Io, config: store_mod.Config) !void {+ const formal = config.formal_check orelse return;+ const binary = try binaryPin(arena, io, formal);+ if (!matchesAdapter(config.mutator, binary) or !matchesAdapter(config.evaluator, binary))+ return error.FormalBinaryMismatch;+}++fn check(+ gpa: Allocator,+ arena: Allocator,+ io: Io,+ environ: *const std.process.Environ.Map,+ formal: store_mod.FormalCheck,+ prompt: []const u8,+) !Evidence {+ const key_path = environ.get("GRAFF_SCORE_KEY_FILE") orelse return error.FormalSigningKeyRequired;+ if (!std.fs.path.isAbsolute(key_path)) return error.FormalSigningKeyRequired;+ if (formal.checker.args.len != 1 or formal.checker.inputs.len != 1)+ return error.InvalidFormalChecker;+ try store_mod.verifyProgram(io, formal.checker);+ try store_mod.verifyPinnedFile(io, formal.pin);+ const expected_binary = (try binaryPin(arena, io, formal)).sha256;+ const candidate_sha = promptHash(prompt);+ var child_env = std.process.Environ.Map.init(gpa);+ defer child_env.deinit();+ try child_env.put("PATH", "/usr/bin:/bin");+ try child_env.put("LANG", "C");+ try child_env.put("LC_ALL", "C");+ try child_env.put("GRAFF_SCORE_KEY_FILE", key_path);+ try child_env.put("PYTHONDONTWRITEBYTECODE", "1");+ const argv = [_][]const u8{+ formal.checker.program,+ formal.checker.args[0],+ "--check",+ formal.pin.path,+ "--candidate-sha256",+ &candidate_sha,+ };+ const run = try jobs.runCappedWithOptions(gpa, io, &argv, stdout_cap, stderr_cap, formal.timeout_ms, .{+ .environ_map = &child_env,+ .kill_process_tree = true,+ });+ defer {+ gpa.free(run.stdout);+ gpa.free(run.stderr);+ }+ if (run.timed_out or run.cancelled) return error.FormalCheckTimedOut;+ if (run.stdout_truncated or run.stderr_truncated) return error.FormalCheckOutputTooLarge;+ if (run.term != .exited or run.term.exited != 0) return error.FormalCheckFailed;+ // The pin is checked after execution too. The external pin/helper is an+ // operator trust input; this detects drift, not adversarial transient swap.+ try store_mod.verifyProgram(io, formal.checker);+ try store_mod.verifyPinnedFile(io, formal.pin);+ var result = try parseResult(arena, run.stdout, &candidate_sha, expected_binary);+ result.formal_pin_sha256 = formal.pin.sha256;+ result.checker_sha256 = formal.checker.sha256;+ result.helper_sha256 = formal.checker.inputs[0].sha256;+ return result;+}++pub fn record(+ gpa: Allocator,+ arena: Allocator,+ io: Io,+ environ: *const std.process.Environ.Map,+ store: *store_mod.Store,+ formal: store_mod.FormalCheck,+ config_id: []const u8,+ trial_id: []const u8,+ phase: []const u8,+ prompt: []const u8,+) ![]const u8 {+ if (!std.mem.eql(u8, phase, "admission") and !std.mem.eql(u8, phase, "selection"))+ return error.InvalidFormalPhase;+ var result = try check(gpa, arena, io, environ, formal, prompt);+ result.phase = phase;+ result.config_id = config_id;+ result.trial_id = trial_id;+ const bytes = try store_mod.jsonBytes(gpa, result);+ defer gpa.free(bytes);+ const id = try store.writeEvidence(gpa, bytes);+ return try arena.dupe(u8, &id);+}++pub fn verify(+ gpa: Allocator,+ arena: Allocator,+ io: Io,+ environ: *const std.process.Environ.Map,+ store: *store_mod.Store,+ formal: store_mod.FormalCheck,+ evidence_id: []const u8,+ config_id: []const u8,+ trial_id: []const u8,+ phase: []const u8,+ prompt: []const u8,+) !void {+ if (formal.checker.args.len != 1 or formal.checker.inputs.len != 1)+ return error.InvalidFormalChecker;+ const bytes = try store.readEvidence(arena, evidence_id, store_mod.max_record_bytes);+ const saved = try std.json.parseFromSliceLeaky(Evidence, arena, bytes, .{});+ const expected_sha = promptHash(prompt);+ if (!std.mem.eql(u8, saved.schema, evidence_schema) or+ !std.mem.eql(u8, saved.phase, phase) or+ !std.mem.eql(u8, saved.config_id, config_id) or+ !std.mem.eql(u8, saved.trial_id, trial_id) or+ !std.mem.eql(u8, saved.candidate_prompt_sha256, &expected_sha) or+ !std.mem.eql(u8, saved.formal_pin_sha256, formal.pin.sha256) or+ !std.mem.eql(u8, saved.checker_sha256, formal.checker.sha256) or+ !std.mem.eql(u8, saved.helper_sha256, formal.checker.inputs[0].sha256) or+ !validHash(saved.formal_identity_sha256) or !validHash(saved.checker_output_sha256) or+ !validHash(saved.binary_sha256)) return error.FormalEvidenceMismatch;+ const fresh = try check(gpa, arena, io, environ, formal, prompt);+ // TLC output contains timestamps/seeds, so its output hash may differ on a+ // fresh run. The pinned identity, binary and candidate must match.+ if (!std.mem.eql(u8, saved.formal_identity_sha256, fresh.formal_identity_sha256) or+ !std.mem.eql(u8, saved.binary_sha256, fresh.binary_sha256))+ return error.FormalEvidenceMismatch;+}++test "formal helper result rejects false success and wrong candidate" {+ const a = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ const b = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";+ var arena_state = std.heap.ArenaAllocator.init(std.testing.allocator);+ defer arena_state.deinit();+ const arena = arena_state.allocator();+ const good = try std.fmt.allocPrint(arena, "{{\"schema\":\"{s}\",\"ok\":true,\"candidate_prompt_sha256\":\"{s}\",\"formal_identity_sha256\":\"{s}\",\"checker_output_sha256\":\"{s}\",\"binary_sha256\":\"{s}\"}}", .{ check_schema, a, a, a, b });+ _ = try parseResult(arena, good, a, b);+ try std.testing.expectError(error.FormalCheckIdentityMismatch, parseResult(arena, good, b, b));+ const false_success = try std.fmt.allocPrint(arena, "{{\"schema\":\"{s}\",\"ok\":false,\"candidate_prompt_sha256\":\"{s}\",\"formal_identity_sha256\":\"{s}\",\"checker_output_sha256\":\"{s}\",\"binary_sha256\":\"{s}\"}}", .{ check_schema, a, a, a, b });+ try std.testing.expectError(error.InvalidFormalCheckResult, parseResult(arena, false_success, a, b));+}++test "formal binary must be pinned and passed to both adapters" {+ const a = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ const b = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";+ const binary: store_mod.PinnedFile = .{ .path = "/opt/graff", .sha256 = a };+ const correct: store_mod.Program = .{+ .program = "/opt/adapter",+ .sha256 = b,+ .args = &.{"/opt/graff"},+ .inputs = &.{binary},+ };+ try std.testing.expect(matchesAdapter(correct, binary));+ var wrong = correct;+ wrong.args = &.{"/opt/other"};+ try std.testing.expect(!matchesAdapter(wrong, binary));+ wrong = correct;+ wrong.inputs = &.{.{ .path = "/opt/graff", .sha256 = b }};+ try std.testing.expect(!matchesAdapter(wrong, binary));+}++test "default configuration skips the optional formal checker" {+ const hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ const config: store_mod.Config = .{+ .schema = store_mod.config_schema,+ .agent_name = "candidate",+ .mutation_instruction = "change one behavior",+ .mutator = .{ .program = "/missing/mutator", .sha256 = hash },+ .evaluator = .{ .program = "/missing/evaluator", .sha256 = hash },+ .evaluation_suite = .{ .path = "/missing/suite", .sha256 = hash },+ .cohort = .{ .provider = "test", .model = "test", .task_family = "test", .adapter_version = "v1", .verifier_version = "v1" },+ };+ try verifyBinaryBinding(std.testing.allocator, std.testing.io, config);+}++test "pinned helper evidence is rechecked for resume and promotion contexts" {+ if (@import("builtin").os.tag == .windows) return;+ const io = std.testing.io;+ const gpa = std.testing.allocator;+ var arena_state = std.heap.ArenaAllocator.init(gpa);+ defer arena_state.deinit();+ const arena = arena_state.allocator();+ var tmp = std.testing.tmpDir(.{});+ defer tmp.cleanup();+ var root_buf: [std.fs.max_path_bytes]u8 = undefined;+ const root_len = try tmp.dir.realPath(io, &root_buf);+ const root = root_buf[0..root_len];+ const helper_path = try std.fmt.allocPrint(arena, "{s}/helper.sh", .{root});+ const pin_path = try std.fmt.allocPrint(arena, "{s}/pin.json", .{root});+ const key_path = try std.fmt.allocPrint(arena, "{s}/key", .{root});+ const binary_hash = try store_mod.hashFileNoFollow(io, "/bin/sh");+ const identity = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ const script = try std.fmt.allocPrint(arena, "printf '{{\"schema\":\"{s}\",\"ok\":true,\"candidate_prompt_sha256\":\"%s\",\"formal_identity_sha256\":\"{s}\",\"checker_output_sha256\":\"{s}\",\"binary_sha256\":\"{s}\"}}\\n' \"$4\"\n", .{ check_schema, identity, identity, &binary_hash });+ {+ const file = try tmp.dir.createFile(io, "helper.sh", .{ .permissions = .fromMode(0o600) });+ defer file.close(io);+ try file.writeStreamingAll(io, script);+ }+ const pin_bytes = try std.fmt.allocPrint(arena, "{{\"binary\":{{\"path\":\"/bin/sh\",\"sha256\":\"{s}\"}}}}\n", .{&binary_hash});+ {+ const file = try tmp.dir.createFile(io, "pin.json", .{ .permissions = .fromMode(0o600) });+ defer file.close(io);+ try file.writeStreamingAll(io, pin_bytes);+ }+ {+ const file = try tmp.dir.createFile(io, "key", .{ .permissions = .fromMode(0o600) });+ defer file.close(io);+ try file.writeStreamingAll(io, "test key");+ }+ const helper_hash = store_mod.rawSha256(script);+ const pin_hash = store_mod.rawSha256(pin_bytes);+ const inputs = [_]store_mod.PinnedFile{.{ .path = helper_path, .sha256 = &helper_hash }};+ const formal: store_mod.FormalCheck = .{+ .checker = .{ .program = "/bin/sh", .sha256 = &binary_hash, .args = &.{helper_path}, .inputs = &inputs },+ .pin = .{ .path = pin_path, .sha256 = &pin_hash },+ .timeout_ms = 2000,+ };+ var env = std.process.Environ.Map.init(gpa);+ defer env.deinit();+ try std.testing.expectError(error.FormalSigningKeyRequired, check(gpa, arena, io, &env, formal, "parent prompt"));+ try env.put("GRAFF_SCORE_KEY_FILE", key_path);+ var store = try store_mod.Store.initAt(io, tmp.dir);+ defer store.deinit();+ const config_id = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb";+ const trial_id = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc";+ const admission = try record(gpa, arena, io, &env, &store, formal, config_id, trial_id, "admission", "parent prompt");+ try verify(gpa, arena, io, &env, &store, formal, admission, config_id, trial_id, "admission", "parent prompt");+ try std.testing.expectError(error.FormalEvidenceMismatch, verify(gpa, arena, io, &env, &store, formal, admission, config_id, trial_id, "admission", "changed parent"));+ try std.testing.expectError(error.FormalEvidenceMismatch, verify(gpa, arena, io, &env, &store, formal, admission, config_id, trial_id, "selection", "parent prompt"));+ const selection = try record(gpa, arena, io, &env, &store, formal, config_id, trial_id, "selection", "selected prompt");+ try verify(gpa, arena, io, &env, &store, formal, selection, config_id, trial_id, "selection", "selected prompt");+ try std.testing.expectError(error.FormalEvidenceMismatch, verify(gpa, arena, io, &env, &store, formal, selection, config_id, trial_id, "selection", "different selection"));+ const missing = "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd";+ try std.testing.expectError(error.FileNotFound, verify(gpa, arena, io, &env, &store, formal, missing, config_id, trial_id, "admission", "parent prompt"));++ const slow_script = "sleep 2\n";+ {+ const file = try tmp.dir.createFile(io, "helper.sh", .{ .truncate = true, .permissions = .fromMode(0o600) });+ defer file.close(io);+ try file.writeStreamingAll(io, slow_script);+ }+ const slow_hash = store_mod.rawSha256(slow_script);+ const slow_inputs = [_]store_mod.PinnedFile{.{ .path = helper_path, .sha256 = &slow_hash }};+ var slow_formal = formal;+ slow_formal.checker.inputs = &slow_inputs;+ slow_formal.timeout_ms = 50;+ try std.testing.expectError(error.FormalCheckTimedOut, check(gpa, arena, io, &env, slow_formal, "parent prompt"));+}diff --git a/src/learn_init.zig b/src/learn_init.zigindex 24d84e36..4d8748a4 100644--- a/src/learn_init.zig+++ b/src/learn_init.zig@@ -9,6 +9,7 @@ const Allocator = std.mem.Allocator;const store_mod = @import("learn_store.zig");const learn_run = @import("learn_run.zig");const bootstrap = @import("learn_bootstrap.zig");+const formal_gate = @import("learn_formal.zig");const credentials = @import("learn_credentials.zig");const util = @import("util.zig");@@ -58,6 +59,11 @@ pub fn zeroConfig(pub fn verifyPins(io: Io, arena: Allocator, config: store_mod.Config) !void {try store_mod.verifyProgram(io, config.mutator);try store_mod.verifyProgram(io, config.evaluator);+ if (config.formal_check) |formal| {+ try store_mod.verifyProgram(io, formal.checker);+ try store_mod.verifyPinnedFile(io, formal.pin);+ try formal_gate.verifyBinaryBinding(arena, io, config);+ }const primary_suite = try store_mod.loadSuite(io, arena, config.evaluation_suite);try store_mod.validateSuitePower(primary_suite.manifest, config.gate);if (config.holdout_suite) |suite| {diff --git a/src/learn_progress.zig b/src/learn_progress.zignew file mode 100644index 00000000..3ff03909--- /dev/null+++ b/src/learn_progress.zig@@ -0,0 +1,37 @@+//! Immutable, resumable tournament progress records.+const build_options = @import("build_options");+const checkpoint = @import("learn_checkpoint.zig");+const eval = @import("learn_eval.zig");+const store = @import("learn_store.zig");++pub fn record(+ config_id: []const u8,+ active: store.LoadedActive,+ nonce: []const u8,+ trial_id: []const u8,+ created_unix_ms: i64,+ candidate_count: usize,+ repetitions: usize,+ auto_requested: bool,+ formal_admission_evidence_id: ?[]const u8,+ primary_baseline: ?eval.PrimaryBaselineRecord,+ candidates: []const eval.CandidateRecord,+) checkpoint.Record {+ return .{+ .schema = if (formal_admission_evidence_id != null) checkpoint.formal_schema else checkpoint.schema,+ .trial_id = trial_id,+ .nonce = nonce,+ .created_unix_ms = created_unix_ms,+ .harness_version = build_options.version,+ .config_id = config_id,+ .parent_genome_id = active.ref.genome_id,+ .parent_generation = active.ref.generation,+ .parent_transaction_id = active.ref.transaction_id,+ .planned_candidates = candidate_count,+ .repetitions = repetitions,+ .auto_requested = auto_requested,+ .formal_admission_evidence_id = formal_admission_evidence_id,+ .primary_baseline = primary_baseline,+ .candidates = candidates,+ };+}diff --git a/src/learn_run.zig b/src/learn_run.zigindex f942ce5f..2528dead 100644--- a/src/learn_run.zig+++ b/src/learn_run.zig@@ -7,9 +7,11 @@ const build_options = @import("build_options");const checkpoint = @import("learn_checkpoint.zig");const eval = @import("learn_eval.zig");+const formal_gate = @import("learn_formal.zig");const holdout = @import("learn_holdout.zig");const mutation_verify = @import("learn_mutation_verify.zig");const primary = @import("learn_primary.zig");+const progressRecord = @import("learn_progress.zig").record;const report = @import("learn_report.zig");const store_mod = @import("learn_store.zig");const submit = @import("learn_submit.zig");@@ -142,6 +144,11 @@ fn mutationIsDuplicate(parent_id: []const u8, candidates: []const eval.Candidatefn verifyPins(io: Io, arena: Allocator, config: store_mod.Config) !void {try store_mod.verifyProgram(io, config.mutator);try store_mod.verifyProgram(io, config.evaluator);+ if (config.formal_check) |formal| {+ try store_mod.verifyProgram(io, formal.checker);+ try store_mod.verifyPinnedFile(io, formal.pin);+ try formal_gate.verifyBinaryBinding(arena, io, config);+ }const primary_suite = try store_mod.loadSuite(io, arena, config.evaluation_suite);try store_mod.validateSuitePower(primary_suite.manifest, config.gate);if (config.holdout_suite) |suite| {@@ -182,35 +189,6 @@ fn copyMutationCandidate(arena: Allocator, outcome: eval.MutationOutcome) !eval.};}-fn progressRecord(- config_id: []const u8,- active: store_mod.LoadedActive,- nonce: []const u8,- trial_id: []const u8,- created_unix_ms: i64,- candidate_count: usize,- repetitions: usize,- auto_requested: bool,- primary_baseline: ?eval.PrimaryBaselineRecord,- candidates: []const eval.CandidateRecord,-) checkpoint.Record {- return .{- .trial_id = trial_id,- .nonce = nonce,- .created_unix_ms = created_unix_ms,- .harness_version = build_options.version,- .config_id = config_id,- .parent_genome_id = active.ref.genome_id,- .parent_generation = active.ref.generation,- .parent_transaction_id = active.ref.transaction_id,- .planned_candidates = candidate_count,- .repetitions = repetitions,- .auto_requested = auto_requested,- .primary_baseline = primary_baseline,- .candidates = candidates,- };-}-fn verifyRestoredProgress(arena: Allocator,io: Io,@@ -344,6 +322,10 @@ pub fn execute(out: *Io.Writer,) !Result {if (options.resume_run and options.restart) return error.ConflictingOptions;+ if (config.value.formal_check != null) {+ try verifyPins(io, arena, config.value);+ if (options.submit) return error.FormalReceiptUnsupported;+ }if (options.restart) try checkpoint.clear(store);const pending = try checkpoint.load(arena, store);@@ -356,6 +338,7 @@ pub fn execute(var repetitions: usize = undefined;var primary_baseline: ?eval.PrimaryBaselineRecord = null;var candidates: []eval.CandidateRecord = undefined;+ var formal_admission_evidence_id: ?[]const u8 = null;if (options.resume_run) {const saved = pending orelse return error.NoPendingRun;@@ -364,6 +347,13 @@ pub fn execute(options.auto != saved.auto_requested) return error.ResumeOptionMismatch;nonce = saved.nonce;trial_id = saved.trial_id;+ if ((config.value.formal_check != null) != std.mem.eql(u8, saved.schema, checkpoint.formal_schema))+ return error.FormalPendingContextMismatch;+ if (config.value.formal_check) |formal| {+ const evidence_id = saved.formal_admission_evidence_id orelse return error.FormalPendingContextMismatch;+ try formal_gate.verify(gpa, arena, io, environ, store, formal, evidence_id, &config.id, saved.trial_id, "admission", active.genome);+ formal_admission_evidence_id = evidence_id;+ }created_unix_ms = saved.created_unix_ms;candidate_count = saved.planned_candidates;repetitions = saved.repetitions;@@ -386,6 +376,9 @@ pub fn execute(trial_buffer = trialId(&config.id, active.ref.genome_id, active.ref.generation, active.ref.transaction_id, &nonce_buffer);nonce = &nonce_buffer;trial_id = &trial_buffer;+ if (config.value.formal_check) |formal| {+ formal_admission_evidence_id = try formal_gate.record(gpa, arena, io, environ, store, formal, &config.id, trial_id, "admission", active.genome);+ }created_unix_ms = util.unixMs(io);candidates = try arena.alloc(eval.CandidateRecord, candidate_count);@@ -420,6 +413,7 @@ pub fn execute(candidate_count,repetitions,options.auto,+ formal_admission_evidence_id,null,candidates,));@@ -450,7 +444,7 @@ pub fn execute(config.value.evaluation_suite,repetitions,);- try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, primary_baseline, candidates));+ try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, formal_admission_evidence_id, primary_baseline, candidates));}var missing_primary: std.ArrayList(usize) = .empty;@@ -478,7 +472,7 @@ pub fn execute(if (result) |completed| {candidates[index].primary = completed.comparison;candidates[index].reason = candidates[index].primary.?.reason;- try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, primary_baseline, candidates));+ try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, formal_admission_evidence_id, primary_baseline, candidates));} else |err| if (primary_failure == null) {primary_failure = err;}@@ -510,12 +504,19 @@ pub fn execute(repetitions,1,);- try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, primary_baseline, candidates));+ try checkpoint.write(gpa, store, progressRecord(&config.id, active, nonce, trial_id, created_unix_ms, candidate_count, repetitions, options.auto, formal_admission_evidence_id, primary_baseline, candidates));}};}const finalized = try tournament.finalize(active.ref.genome_id, candidates, config.value.holdout_suite != null);const primary_winner = if (finalized.primary_winner_index) |index| candidates[index].genome_id else null;+ const formal_selection_evidence_id: ?[]const u8 = if (finalized.selected_genome_id) |selected| blk: {+ if (config.value.formal_check) |formal| {+ const prompt = try store.readGenome(arena, selected, config.value.limits.genome_bytes);+ break :blk try formal_gate.record(gpa, arena, io, environ, store, formal, &config.id, trial_id, "selection", prompt);+ }+ break :blk null;+ } else null;// Re-check every externally controlled executable and suite after use and// before committing the immutable run record.@@ -523,7 +524,7 @@ pub fn execute(try verifyTrialPower(io, arena, config.value, candidate_count);const record: eval.RunRecord = .{- .schema = eval.run_schema,+ .schema = if (config.value.formal_check != null) eval.formal_run_schema else eval.run_schema,.trial_id = trial_id,.nonce = nonce,.created_unix_ms = created_unix_ms,@@ -535,6 +536,8 @@ pub fn execute(.planned_candidates = candidate_count,.repetitions = repetitions,.auto_requested = options.auto,+ .formal_admission_evidence_id = formal_admission_evidence_id,+ .formal_selection_evidence_id = formal_selection_evidence_id,.primary_baseline = primary_baseline,.candidates = candidates,.primary_winner_genome_id = primary_winner,diff --git a/src/learn_store.zig b/src/learn_store.zigindex 9f8a5759..bc4363cf 100644--- a/src/learn_store.zig+++ b/src/learn_store.zig@@ -21,6 +21,7 @@ pub const max_suite_bytes = model.max_suite_bytes;pub const max_pairs = model.max_pairs;pub const PinnedFile = model.PinnedFile;pub const Program = model.Program;+pub const FormalCheck = model.FormalCheck;pub const Suite = model.Suite;pub const Limits = model.Limits;pub const Gate = model.Gate;diff --git a/src/learn_store_tests.zig b/src/learn_store_tests.zigindex 206e4eda..2494bb44 100644--- a/src/learn_store_tests.zig+++ b/src/learn_store_tests.zig@@ -50,6 +50,32 @@ test "learning config parser rejects unknown fields and unsafe auto" {try std.testing.expectError(error.UnknownField, std.json.parseFromSlice(Config, std.testing.allocator, unknown, .{}));}+test "formal checker shape requires one pinned script argument" {+ const hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ var config: Config = .{+ .schema = config_schema,+ .agent_name = "candidate",+ .mutation_instruction = "change one behavior",+ .mutator = .{ .program = "/bin/mutate", .sha256 = hash },+ .evaluator = .{ .program = "/bin/evaluate", .sha256 = hash },+ .evaluation_suite = .{ .path = "/suite", .sha256 = hash },+ .cohort = .{ .provider = "test", .model = "test", .task_family = "test", .adapter_version = "v1", .verifier_version = "v1" },+ .formal_check = .{+ .checker = .{ .program = "/bin/python", .sha256 = hash, .args = &.{"/pinned/helper.py"}, .inputs = &.{.{ .path = "/pinned/helper.py", .sha256 = hash }} },+ .pin = .{ .path = "/pinned/manifest.json", .sha256 = hash },+ },+ };+ try validateConfig(config);+ var formal = config.formal_check.?;+ formal.checker.args = &.{"/other/helper.py"};+ config.formal_check = formal;+ try std.testing.expectError(error.InvalidFormalChecker, validateConfig(config));+ formal.checker.args = &.{"/pinned/helper.py"};+ formal.checker.inputs = &.{};+ config.formal_check = formal;+ try std.testing.expectError(error.InvalidFormalChecker, validateConfig(config));+}+test "suite statistical units are additive, validated, and criticality-consistent" {const legacy =\\{"schema":"codegraff.learn.suite.v1","suite_id":"legacy","cases":[{"id":"one"},{"id":"two"}]}diff --git a/src/learn_store_types.zig b/src/learn_store_types.zigindex 75b2810e..38323558 100644--- a/src/learn_store_types.zig+++ b/src/learn_store_types.zig@@ -29,6 +29,14 @@ pub const Program = struct {pass_env: []const []const u8 = &.{},};+/// Optional external formal baseline check. This binds a model-check result+/// to a prompt fingerprint; it does not prove the prompt's behavior.+pub const FormalCheck = struct {+ checker: Program,+ pin: PinnedFile,+ timeout_ms: u64 = 900_000,+};+pub const Suite = struct {path: []const u8,sha256: []const u8,@@ -89,6 +97,7 @@ pub const Config = struct {mutation_instruction: []const u8,mutator: Program,evaluator: Program,+ formal_check: ?FormalCheck = null,evaluation_suite: Suite,holdout_suite: ?Suite = null,limits: Limits = .{},diff --git a/src/learn_store_validate.zig b/src/learn_store_validate.zigindex fc48f08b..176d2a52 100644--- a/src/learn_store_validate.zig+++ b/src/learn_store_validate.zig@@ -61,6 +61,16 @@ pub fn validateConfig(config: Config) !void {if (!validText(config.mutation_instruction, 4096)) return error.InvalidMutationInstruction;try validateProgram(config.mutator);try validateProgram(config.evaluator);+ if (config.formal_check) |formal| {+ try validateProgram(formal.checker);+ if (formal.checker.args.len != 1 or formal.checker.inputs.len != 1 or+ formal.checker.pass_env.len != 0 or+ !std.mem.eql(u8, formal.checker.args[0], formal.checker.inputs[0].path))+ return error.InvalidFormalChecker;+ if (!std.fs.path.isAbsolute(formal.pin.path) or !validText(formal.pin.path, std.fs.max_path_bytes) or+ !validId(formal.pin.sha256) or formal.timeout_ms == 0 or formal.timeout_ms > 900_000)+ return error.InvalidFormalChecker;+ }try validateSuitePin(config.evaluation_suite);if (config.holdout_suite) |suite| try validateSuitePin(suite);diff --git a/src/learn_submit.zig b/src/learn_submit.zigindex e07f2ccc..db32ca8d 100644--- a/src/learn_submit.zig+++ b/src/learn_submit.zig@@ -127,6 +127,9 @@ pub fn submitVerifiedRun(run_id: []const u8,run: eval.RunRecord,) !SubmitResult {+ // The signed receipt schema does not yet bind formal admission evidence.+ if (config.formal_check != null or std.mem.eql(u8, run.schema, eval.formal_run_schema))+ return error.FormalReceiptUnsupported;try preflight(io, arena, environ);var primary_count: usize = 0;var holdout_count: usize = 0;@@ -222,3 +225,41 @@ test "learning pass-rate grade uses repeated pair count" {test {_ = @import("learn_receipt.zig");}++test "formal learning runs cannot enter the unsigned receipt path" {+ const id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";+ var config: store.Config = .{+ .schema = store.config_schema,+ .agent_name = "candidate",+ .mutation_instruction = "change one behavior",+ .mutator = .{ .program = "/missing/mutator", .sha256 = id },+ .evaluator = .{ .program = "/missing/evaluator", .sha256 = id },+ .evaluation_suite = .{ .path = "/missing/suite", .sha256 = id },+ .cohort = .{ .provider = "test", .model = "test", .task_family = "test", .adapter_version = "v1", .verifier_version = "v1" },+ };+ var run: eval.RunRecord = .{+ .schema = eval.formal_run_schema,+ .trial_id = id,+ .nonce = id,+ .created_unix_ms = 0,+ .harness_version = "test",+ .config_id = id,+ .parent_genome_id = id,+ .parent_generation = 0,+ .parent_transaction_id = id,+ .planned_candidates = 1,+ .repetitions = 1,+ .auto_requested = false,+ .candidates = &.{},+ .selected_genome_id = null,+ };+ var env = std.process.Environ.Map.init(std.testing.allocator);+ defer env.deinit();+ try std.testing.expectError(error.FormalReceiptUnsupported, submitVerifiedRun(std.testing.io, std.testing.allocator, std.testing.allocator, &env, config, id, run));+ run.schema = eval.run_schema;+ config.formal_check = .{+ .checker = .{ .program = "/missing/checker", .sha256 = id },+ .pin = .{ .path = "/missing/pin", .sha256 = id },+ };+ try std.testing.expectError(error.FormalReceiptUnsupported, submitVerifiedRun(std.testing.io, std.testing.allocator, std.testing.allocator, &env, config, id, run));+}diff --git a/src/main.zig b/src/main.zigindex 5532ad57..01b71ff5 100644--- a/src/main.zig+++ b/src/main.zig@@ -582,7 +582,7 @@ test { // ── Unit tests (`zig build test`): pull in tests from imported modu_ = .{ @import("acp_protocol.zig"), @import("subagent_mimo_tests.zig"), @import("mimo_effort_tests.zig"), @import("gateway_picker_catalog.zig") };_ = @import("task_outcome.zig"); // goal-outcome telemetry events_ = @import("learn_delete.zig"); // #303: its tests were dead until listed here- _ = .{ @import("additional_tests.zig"), @import("req_stats.zig"), @import("exact_reply.zig"), @import("rlm_order_tests.zig"), @import("async_tool_policy.zig"), @import("agent_async_tools.zig"), @import("request_usage_attempts.zig") };+ _ = .{ @import("additional_tests.zig"), @import("req_stats.zig"), @import("exact_reply.zig"), @import("rlm_order_tests.zig"), @import("async_tool_policy.zig"), @import("agent_async_tools.zig"), @import("request_usage_attempts.zig"), @import("learn_formal.zig") };_ = @import("goal_pacing_autonomous_test.zig");_ = @import("goal_state.zig");_ = @import("goal_persist_tests.zig");diff --git a/src/test_hooks.zig b/src/test_hooks.zigindex 4f5ca292..a3b17548 100644--- a/src/test_hooks.zig+++ b/src/test_hooks.zig@@ -22,6 +22,7 @@//! checking the guard still does.// Learning / DGM promotion pipeline.+const learn_checkpoint = @import("learn_checkpoint.zig");const learn_holdout = @import("learn_holdout.zig");const learn_receipt = @import("learn_receipt.zig");const learn_report = @import("learn_report.zig");@@ -235,6 +236,7 @@ const commands_sandbox = @import("commands_sandbox.zig");const sandbox_tests = @import("sandbox_tests.zig");test {+ _ = learn_checkpoint;_ = learn_holdout;_ = learn_receipt;_ = learn_report;diff --git a/tests/learn_formal_e2e.py b/tests/learn_formal_e2e.pynew file mode 100644index 00000000..5a16a3e1--- /dev/null+++ b/tests/learn_formal_e2e.py@@ -0,0 +1,193 @@+#!/usr/bin/env python3+"""Offline CLI wiring test for the optional native learning formal gate.++The helper is deliberately a pinned fixture, not a TLC proof. This test checks+that native command boundaries invoke and reject its result as configured.+"""++from __future__ import annotations++import argparse+import json+import os+from pathlib import Path+import shutil+import sys+import tempfile++from learn_e2e import (+ EVALUATOR_BODY,+ MUTATOR_BODY,+ invoke,+ json_bytes,+ raw_sha256,+ run_id_from,+ status,+ write_interpreter_shim,+ write_private,+)+++def setup(graff: Path, root: Path) -> tuple[Path, Path, dict[str, str], Path, Path, Path]:+ tools = root / "tools"+ tools.mkdir(mode=0o700)+ binary = Path("/bin/sh").resolve()+ binary_pin = {"path": str(binary), "sha256": raw_sha256(binary)}+ mutator_marker = root / "mutator-called"+ evaluator_fail = root / "evaluator-fail"+ mutator_body = MUTATOR_BODY.replace(+ "operation, request_path, response_path = sys.argv[1:4]",+ f"binary = pathlib.Path(sys.argv.pop(1))\nassert binary.is_file()\n"+ f"pathlib.Path({str(mutator_marker)!r}).write_text('called')\n"+ "operation, request_path, response_path = sys.argv[1:4]",+ )+ evaluator_body = EVALUATOR_BODY.replace(+ "operation, request_path, response_path = sys.argv[1:4]",+ f"binary = pathlib.Path(sys.argv.pop(1))\nassert binary.is_file()\n"+ f"if pathlib.Path({str(evaluator_fail)!r}).exists(): raise SystemExit(9)\n"+ "operation, request_path, response_path = sys.argv[1:4]",+ )+ python = Path(sys.executable).resolve()+ mutator, mutator_source = write_interpreter_shim(tools, "formal-mutator", python, mutator_body)+ evaluator, evaluator_source = write_interpreter_shim(tools, "formal-evaluator", python, evaluator_body)+ for program in (mutator, evaluator):+ program["args"].append(str(binary))+ program["inputs"].append(binary_pin)+ mutator["pass_env"] = ["FIXTURE_MUTATOR_SOURCE"]+ evaluator["pass_env"] = ["FIXTURE_EVALUATOR_SOURCE"]++ helper_control = root / "helper-control"+ helper_count = root / "helper-count"+ helper = tools / "formal-helper.py"+ helper_body = f'''import json, pathlib, sys+control = pathlib.Path({str(helper_control)!r})+count = pathlib.Path({str(helper_count)!r})+count.write_text(str(int(count.read_text()) + 1 if count.exists() else 1))+mode = control.read_text() if control.exists() else "good"+if mode == "fail": raise SystemExit(7)+print(json.dumps({{+ "schema": "codegraff.dgm.formal-check.v1", "ok": mode != "false",+ "candidate_prompt_sha256": sys.argv[-1],+ "formal_identity_sha256": "a" * 64,+ "checker_output_sha256": "b" * 64,+ "binary_sha256": {binary_pin['sha256']!r},+}}))+'''+ write_private(helper, helper_body.encode())+ formal_pin = tools / "formal-pin.json"+ write_private(formal_pin, json_bytes({"binary": binary_pin}))+ score_key = root / "score-key"+ write_private(score_key, b"fixture key\n")++ primary = tools / "primary.json"+ holdout = tools / "holdout.json"+ for path, suite_id in ((primary, "formal-primary"), (holdout, "formal-holdout")):+ write_private(path, json_bytes({+ "schema": "codegraff.learn.suite.v1", "suite_id": suite_id,+ "cases": [{"id": f"{suite_id}-{index}"} for index in range(6)],+ }))+ parent = root / "parent.md"+ write_private(parent, b"Initial fixture policy.\n")+ config = root / "config.json"+ write_private(config, json_bytes({+ "schema": "codegraff.learn.config.v1",+ "agent_name": "formal-fixture",+ "mutation_instruction": "append the fixture policy sentence",+ "mutator": mutator, "evaluator": evaluator,+ "formal_check": {+ "checker": {"program": str(python), "sha256": raw_sha256(python),+ "args": [str(helper.resolve())],+ "inputs": [{"path": str(helper.resolve()), "sha256": raw_sha256(helper)}]},+ "pin": {"path": str(formal_pin.resolve()), "sha256": raw_sha256(formal_pin)},+ "timeout_ms": 10000,+ },+ "evaluation_suite": {"path": str(primary.resolve()), "sha256": raw_sha256(primary)},+ "holdout_suite": {"path": str(holdout.resolve()), "sha256": raw_sha256(holdout)},+ "gate": {"alpha_ppm": 50000, "minimum_delta_ppm": 100000,+ "minimum_pairs": 6, "default_candidates": 1, "default_repetitions": 3},+ "auto": {"enabled": True},+ "cohort": {"provider": "fixture", "model": "fixture", "task_family": "local-e2e",+ "adapter_version": "v1", "verifier_version": "v1"},+ }))+ env = {**os.environ, "GRAFF_NO_TELEMETRY": "1", "GRAFF_LEARN_AUTO": "off",+ "GRAFF_SCORE_KEY_FILE": str(score_key.resolve()),+ "FIXTURE_MUTATOR_SOURCE": str(mutator_source.resolve()),+ "FIXTURE_EVALUATOR_SOURCE": str(evaluator_source.resolve())}+ workspace = root / "manual"+ workspace.mkdir(mode=0o700)+ invoke(graff, workspace, env, "init", "--parent", str(parent.resolve()),+ "--config", str(config.resolve()))+ return workspace, parent, env, helper_control, helper_count, evaluator_fail+++def exercise(graff: Path, root: Path) -> None:+ workspace, parent, env, control, count, eval_fail = setup(graff, root)+ marker = root / "mutator-called"+ refused = invoke(graff, workspace, env, "run", "--submit", succeeds=False)+ assert "FormalReceiptUnsupported" in refused.stderr+ assert not marker.exists()++ control.write_text("fail")+ failed = invoke(graff, workspace, env, "run", succeeds=False)+ assert "FormalCheckFailed" in failed.stderr and not marker.exists()+ control.write_text("false")+ false = invoke(graff, workspace, env, "run", succeeds=False)+ assert "InvalidFormalCheckResult" in false.stderr and not marker.exists()+ control.unlink()++ eval_fail.touch()+ failed_eval = invoke(graff, workspace, env, "run", succeeds=False)+ assert "ProcessFailed" in failed_eval.stderr and marker.exists()+ pending_path = workspace / ".graff" / "learn" / "refs" / "pending.json"+ pending = json.loads(pending_path.read_text())+ assert pending["schema"] == "codegraff.learn.pending.v2"+ original_evidence = pending["formal_admission_evidence_id"]+ pending["formal_admission_evidence_id"] = "d" * 64+ write_private(pending_path, json_bytes(pending))+ rejected = invoke(graff, workspace, env, "run", "--resume", succeeds=False)+ assert "FileNotFound" in rejected.stderr+ pending["formal_admission_evidence_id"] = original_evidence+ write_private(pending_path, json_bytes(pending))+ eval_fail.unlink()+ manual_run = run_id_from(invoke(graff, workspace, env, "run", "--resume").stdout)+ run_path = workspace / ".graff" / "learn" / "runs" / f"{manual_run}.json"+ run = json.loads(run_path.read_text())+ assert run["schema"] == "codegraff.learn.run.v4"+ assert run["formal_admission_evidence_id"] == original_evidence+ assert run["formal_selection_evidence_id"]+ refused = invoke(graff, workspace, env, "submit", manual_run, succeeds=False)+ assert "FormalReceiptUnsupported" in refused.stderr+ before_manual = int(count.read_text())+ invoke(graff, workspace, env, "promote", manual_run)+ assert int(count.read_text()) >= before_manual + 2+ assert status(graff, workspace, env)["generation"] == 1++ automatic = root / "automatic"+ automatic.mkdir(mode=0o700)+ invoke(graff, automatic, env, "init", "--parent", str(parent.resolve()),+ "--config", str((root / "config.json").resolve()))+ before_auto = int(count.read_text())+ auto_run = run_id_from(invoke(graff, automatic, env, "run", "--auto").stdout)+ assert auto_run+ assert int(count.read_text()) >= before_auto + 4+ assert status(graff, automatic, env)["generation"] == 1+++def main() -> int:+ parser = argparse.ArgumentParser()+ parser.add_argument("--graff", type=Path, default=Path("zig-out/bin/graff"))+ args = parser.parse_args()+ graff = args.graff.resolve(strict=True)+ root = Path(tempfile.mkdtemp(prefix="codegraff-learn-formal."))+ try:+ exercise(graff, root)+ except Exception:+ print(f"formal learning fixture preserved at {root}", file=sys.stderr)+ raise+ shutil.rmtree(root)+ print("formal learning CLI boundaries passed")+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())