Skip to content

Git designed for AI agents.

247 KB
Raw Blame

Graft c1: AI conflict resolution with checks, measured (approved plan)

Status: approved by the PM with the amendments of section 12; WP1 to WP4 under way. Owner: c1 (Opus). Worktree owt/c1, branch c1-conflicts from main at v0.2.0 (7baee7e).

c1 owns tools/graft-agents/resolve/**, tools/graft-agents/cmd/graft-resolve, doc/graft/RESOLVE.md and this plan; for the conflict package only, the land conflict path in internal/gitaly/service/operations (land*.go, the error details) and new fields and messages in proto/ (section 6.1, a public API). The land decision itself does not change; anything that would touch it is coordinated with the PM first.

1. Goals (Federico's final decisions, binding)

  • G1 One model plus automatic checks. What git cannot merge goes straight to the llm strategy: one model through OpenRouter (the OpenAI-compatible provider). No second model, no judge. The default chain for AI resolution is the llm strategy alone; trivial, union, rerere and command stay available as opt-in configuration. No Mergiraf.
  • G2 No self-declared confidence. The confidence field leaves the prompt, the answer schema, the parsing and the docs, and the low-confidence rejection goes with it. Acceptance comes from the checks only.
  • G3 Checks before the land, on an AI resolution: syntax valid, plus no conflict markers. The result must parse with tree-sitter for the file's language without an error node; a file in a language without a grammar gets the marker rule only, as RESOLVE.md states. No formatter, no build gate.
  • G4 When the server cannot resolve, the land fails with a complete conflict package for the agent (per file the base/ours/theirs and the conflicting hunks, both commit messages, the merge base and the tip, what was tried and why it failed), plus the SDK and CLI path to resolve locally and land again.
  • G5 Lockfiles always go to the agent (package-lock.json, yarn.lock, pnpm-lock.yaml, poetry.lock, Cargo.lock, go.sum): never sent to a model, never merged by the server. No lockfile strategy.
  • G6 Never sent to the model by default: large, binary, generated and minified files (detection in 3.4). Secrets, keys and configuration may be sent; RESOLVE.md states plainly that file contents, secrets included, go to the provider.
  • G7 Merged-tree build/test before the land stays optional, off by default. The safety net of an AI resolution that lands without review is the trail (Graft-Resolved-* trailers, the log) and graft-verify's bisect and revert.
  • G8 Measure it once: the llm strategy with openai/gpt-5-mini through OpenRouter, on real conflicts in TypeScript/JavaScript, Python, Go and Rust/Java, about 200 conflicts, under a hard cap of $10 that the run enforces. Report correct / wrong / rejected-by-checks / not sent / model error, so the real error rate of this setup is known. Federico gives the key when the benchmark is ready.
  • G9 A monthly spending cap in production. graft-resolve tracks the spend of AI resolution from the provider's reported usage times a configured price, persisted across restarts, with a metric; when the month's cap is reached, conflicts go straight to the agent package with the reason "AI budget exhausted".

2. What exists today (from the code)

  • UserLand with resolve_conflicts computes the conflicting files of the merge of the tip and the landed commit, with their base/ours/theirs blobs, and sends protocol v1 to the resolver, both commit messages included (Messages). A valid answer becomes a merge commit with Graft-Resolved-By and Graft-Resolved-Path trailers (land_resolve.go).
  • On any failure the client gets the plain MergeConflictError (conflicting_files, conflicting_commit_ids); the reason (a denied path, the strategy that failed and why) is only logged by the server. The agent cannot learn what was tried, nor get the sides through the error.
  • The llm strategy prompts with the path and the three sides only (llmWholePrompt, llmHunkPrompt); the commit messages are not used. Its answer has content, confidence, explanation; low confidence is rejected, medium accepted. Large files go hunk by hunk, so one file can cost several calls.
  • graft-resolve rejects any result with conflict markers (git's rule) and runs configured per-glob checks; nothing parses the result.
  • The OpenAI-compatible provider takes base_url, model, api_key_env, max_tokens, effort and reads the usage of each answer.
  • The server sets no gRPC message size, so grpc-go's default of 4 MiB applies to a request it receives.
  • graft-verify bisects a failing tip and reverts the culprit (opt-in).
  • No measurement of the llm strategy exists.

3. Design

3.1 The llm strategy without confidence, with the messages (G1, G2)

  • The answer schema becomes {"content": "...", "explanation": "..."}. The prompts no longer ask for a confidence, and no field of the answer decides acceptance; an answer with extra fields is still read (models add fields). The options require_confidence and the like are removed, and a configuration that still sets them fails at startup with a message that says they are gone.
  • Both commit messages go into the prompt by default (PM, Q3) as the intent of each side, bounded to 4 KiB each and marked as untrusted data like the file contents. They cost no extra call.
  • The AI setup documented in RESOLVE.md is [[strategy]] type = "llm" alone; the deterministic strategies are documented as opt-in.

3.2 The syntax check (G3)

  • Built into graft-resolve and always on for llm results: the content is parsed with the tree-sitter grammar chosen by the path's extension; any ERROR or MISSING node rejects the result, with the line and column of the first one. Grammars: TypeScript and TSX (.ts, .mts, .cts, .tsx), JavaScript (.js, .mjs, .cjs, .jsx), Python (.py, .pyi), Go (.go), Rust (.rs), Java (.java). Every other extension gets the marker rule only.
  • Bindings: the official MIT Go bindings github.com/tree-sitter/go-tree-sitter and the grammar modules of those languages, with cgo, in the tools/graft-agents module only (a separate module and binary). The main gitai module gets no new dependency. RESOLVE.md states the build requirement (a C compiler, CGO_ENABLED=1 for graft-resolve) and what a language without a grammar gets.
  • The marker rule stays as it is, for every strategy.

3.3 The agent conflict package and the re-land (G4)

The server already computes everything. The error carries a small descriptor and the agent fetches the contents by object ID, so no file content travels in the gRPC error, whose size is limited.

  • The error of a land that conflicts keeps UserLandError.merge_conflict unchanged (upstream clients rely on it) and carries a second detail, a LandConflict (6.1), when the request set resolve_conflicts: gRPC status details are a list, so existing clients see nothing new.
  • LandConflict holds the tip, the landed commit, the merge base, both commit messages (cut to 4 KiB, with a flag when cut), and per file the path, the kind, each side's blob OID and mode, the conflicting hunks as line ranges of each side, and the outcome of the server's attempt with its reason: not sent (lockfile, binary, large, generated, minified, denied path, invalid UTF-8 path), budget exhausted, model error, rejected by checks (syntax error at line:column, conflict markers), resolved but not landed because another file was not, or not attempted because the whole request was refused (too many files, actor not allowed, resolution disabled), with the request-level reason. Reasons are cut to 512 bytes. The package lists every conflicting file, within its own bounds (see "Review of WP5/WP6" in section 12).
  • The agent reads the sides with GetBlobs, which its read access allows.
  • Re-land: UserLandRequest.resolution (LandResolution) carries the tip and the commit the resolution was made against and, per file, write (mode, content) or delete. The server accepts it only when commit_id equals the resolution's commit and the merge of the current tip and that commit conflicts on exactly those paths. It validates it with the rules of a resolver answer (markers, modes, sizes; no syntax check, the agent owns its answer) and lands a merge commit with Graft-Resolved-By: agent/<actor>; the trailer path is the one of land_resolve.go. If the tip moved, the land fails with a fresh package. The resolution travels in the request, so it is bounded by the 4 MiB gRPC limit in total (RESOLVE.md says so); larger resolutions are a stated limit of this plan.
  • SDK: the land error exposes Conflict() (the package) and ConflictSides(ctx) (the contents); LandOptions.Resolution re-lands. CLI: graft land prints the package; --conflicts-dir DIR writes each file with git-style markers and the three sides beside it (.base, .ours, .theirs); graft land --resolved DIR lands the edited files as the resolution.

3.4 What is never sent (G5, G6)

The server decides per file before the request, so the resolver never sees these files, and the package names the reason:

  • binary: a NUL byte in any side;
  • large: any side over max_file_bytes (default 1 MiB, as today);
  • lockfile: the base name is one of package-lock.json, yarn.lock, pnpm-lock.yaml, poetry.lock, Cargo.lock, go.sum (fixed list, extendable with lockfiles in the configuration);
  • generated: linguist-generated is set for the path by the .gitattributes of the tip's tree (git check-attr against the tip), or a side's first 10 lines hold one of Code generated .* DO NOT EDIT\. (Go's convention), @generated, DO NOT EDIT, auto-generated (extendable with generated_markers);
  • minified: the path matches *.min.js, *.min.mjs, *.min.css or *.bundle.js, or a side of more than 2 KiB has an average line length above 300 bytes or a line longer than 5,000 bytes.

When a request has such a file, the other files still go to the model; the land fails if any file is left, and the package shows which files were resolved and which need the agent. Nothing partial lands.

3.5 The monthly spending cap (G9)

  • resolve.toml, llm strategy: [strategy.llm.budget] with monthly_usd (the cap; 0 or absent means none), input_usd_per_million and output_usd_per_million (the prices of the configured model, which the operator keeps current), and state (the file that holds the spend).
  • Before a call the strategy reserves the worst case of that call (the request's estimated input tokens times the input price plus max_tokens times the output price); if the month's spend plus the reservation exceeds the cap, the strategy does not call and abstains with "AI budget exhausted". After the call it books the real cost from the reported usage (input and output tokens, reasoning tokens included in the output) and releases the reservation. A provider that reports no usage is booked at the reservation.
  • The state file is JSON ({"month": "2026-10", "spent_usd": ...}), updated under an flock and written atomically with a temporary file and a rename, fsynced, so the spend survives restarts and concurrent graft-resolve processes share it. A new calendar month (UTC) starts at zero.
  • Metrics: graft_resolve_ai_spend_usd (gauge, the month's spend), graft_resolve_ai_budget_usd (gauge, the cap) and graft_resolve_ai_budget_exhausted_total (counter). graft-resolve exposes them in HTTP mode on /metrics; in command mode the spend is in the state file and in the explanation of each answer.
  • The server sees "AI budget exhausted" as the resolver's failure reason and puts it into the package.

3.6 Optional merged-tree verification (G7)

[graft.land.resolver.verify]: enabled = false (default), command (argv run in a temporary checkout of the merged tree, minimal environment), timeout (default 5 min, hard). A failure or timeout rejects the resolution, and the package says so. RESOLVE.md gets "finding and reverting AI resolutions": the trailers (git log --grep='^Graft-Resolved-By:' in a clone, graft show --json; graft log lists changesets and has no grep), the info log, graft-verify's bisect and revert.

3.7 The benchmark (G8)

Corpus. Merge-Bench's data location and license are not published on its abstract page, so the corpus is mined: a committed corpus file lists well-known repositories with permissive licenses (MIT, Apache-2.0, BSD) per family, with a pinned commit and the license; graft-resolve bench mine clones them (partial clones), replays each two-parent merge with git merge-tree --write-tree on its parents, and keeps every conflicting text file whose human resolution (the merge commit's version) differs from both sides. Only a manifest is committed (repository, commit IDs, path, license); no third-party code. Candidates, to be kept if they yield enough conflicts (many projects squash their pull requests, which leaves no conflicting merges; the miner reports the yield per repository):

  • TypeScript/JavaScript: microsoft/TypeScript (Apache-2.0), microsoft/vscode (MIT), facebook/react (MIT), nodejs/node (MIT, its own code);
  • Python: django/django (BSD-3-Clause), pandas-dev/pandas (BSD-3-Clause), python/cpython (PSF-2.0, permissive) as a fallback;
  • Go: kubernetes/kubernetes (Apache-2.0), prometheus/prometheus (Apache-2.0), golang/go (BSD-3-Clause);
  • Rust/Java: rust-lang/rust-analyzer (MIT/Apache-2.0), tokio-rs/tokio (MIT), apache/kafka (Apache-2.0), spring-projects/spring-boot (Apache-2.0).

Run. graft-resolve bench run builds the protocol v1 request of each sampled case as the server would (never-send rules included), runs the configured llm strategy with its checks, and judges each accepted result against the human file:

  • correct: equal to the human file after normalizing line endings and trailing whitespace (exact equality reported beside it);
  • wrong: accepted by the checks but not equal;
  • rejected by checks: syntax error or markers;
  • not sent: lockfile, binary, large, generated, minified;
  • model error: the call failed or the answer could not be read;
  • not run (cap).

The report gives these per family and in total, and the error rate = wrong / accepted: a wrong answer lands without review.

Sample and the $10 cap. A fixed, seeded sample of 200 cases stratified by family (--sample 200 --seed S). The run enforces --max-usd 10 with the prices of the model (--input-usd-per-million 0.25 --output-usd-per-million 2, OpenRouter's list price for openai/gpt-5-mini; the run takes the prices as flags so that a changed price is entered, not assumed): before each call it reserves the worst case (the estimated input plus max_tokens of output, reasoning included) and does not call if the spend plus the reservation would exceed $10; after the call it books the reported usage. max_tokens per call is 8,000 and effort "low" to bound reasoning. The key is read from OPENROUTER_API_KEY, never stored.

Estimated cost. 200 cases, about 1.5 calls per case (hunk mode for large files), 3,000 input tokens and about 4,500 output tokens per call (1,500 of answer and up to 3,000 of reasoning): 300 × (3,000 × $0.25 + 4,500 × $2) / 1,000,000 ≈ 300 × $0.00975 ≈ $2.9. The worst case of every call at max_tokens is 300 × (0.00075 + 0.016) ≈ $5.0, so the $10 cap leaves room and is the hard bound whatever the files hold.

4. Invariants

  • I1 The land decision, the CAS and the trailers are unchanged; nothing partial lands.
  • I2 A lockfile, a binary, large, generated or minified file never reaches the resolver.
  • I3 An AI result lands only when it parses (where a grammar exists) and has no markers.
  • I4 Every failure of a land that asked for resolution gives the agent the package; the agent's re-land is validated like a resolver answer.
  • I5 AI spend never exceeds the monthly cap, across restarts and processes; the benchmark never exceeds its $10.
  • I6 The benchmark is reproducible from the corpus file, the manifest and the seed.

5. Failure and race cases

case behaviour
tip moves between the package and the re-land the re-land fails with a fresh package
the agent's resolution names other paths than the conflict InvalidArgument, package again
the model returns a file that does not parse rejected by checks, package with line:column
the model times out or errs model error, package
a request mixes sendable files and a lockfile the sendable ones are resolved, the land fails with the package
the monthly cap is reached no call, "AI budget exhausted", package
two graft-resolve processes spend at once the state file's flock serializes the reservations
the state file is corrupt the strategy refuses to call (budget unknown) and abstains with the reason
the benchmark reaches $10 it stops calling; the rest is "not run (cap)"

6. Config and API changes

6.1 Proto (exact, for the PM's review)

proto/operations.proto, in UserLandRequest:

  // resolution lands commit_id with the caller's resolution of the conflicts that a
  // LandConflict described, instead of asking the server's resolver. The server
  // accepts it only when commit_id equals resolution.commit and the merge of the
  // current tip and commit_id conflicts on exactly the paths of resolution.files;
  // otherwise the land fails with a fresh LandConflict. It is validated like a
  // resolver answer (no conflict markers, modes 100644, 100755 or 120000, sizes) and
  // lands as a merge commit with a Graft-Resolved-By: agent/<actor> trailer.
  LandResolution resolution = 11;

New messages in proto/operations.proto:

// LandConflict describes the conflicts of a land that the server did not resolve, so
// that the caller can resolve them and land again with UserLandRequest.resolution. It
// is sent as an additional detail of the error, next to UserLandError, when the
// request set resolve_conflicts.
message LandConflict {
  // tip is the branch tip that the change was merged with.
  string tip = 1;
  // commit is the landed commit.
  string commit = 2;
  // merge_base is the merge base of tip and commit, empty for unrelated histories.
  string merge_base = 3;
  // tip_message and commit_message are the commit messages, cut to 4 KiB each.
  bytes tip_message = 4;
  bytes commit_message = 5;
  // messages_truncated is set when a message was cut.
  bool messages_truncated = 6;
  // files are all the conflicting files (see "Review of WP5/WP6" in section 12).
  repeated LandConflictFile files = 7;
  // reason is set when the server did not try to resolve the request at all, for
  // example because it has too many files or the actor may not resolve; it is cut to
  // 512 bytes.
  string reason = 8;
}

// LandConflictFile is one conflicting file of a LandConflict.
message LandConflictFile {
  // Kind is the kind of the conflict.
  enum Kind {
    KIND_UNSPECIFIED = 0;
    KIND_CONTENT = 1;
    KIND_MODIFY_DELETE = 2;
    KIND_ADD_ADD = 3;
    KIND_MODE = 4;
  }
  // Outcome is what the server did with the file.
  enum Outcome {
    OUTCOME_UNSPECIFIED = 0;
    // OUTCOME_NOT_SENT: the file was not sent to the resolver (reason: lockfile,
    // binary, large, generated, minified, denied path, invalid UTF-8 path).
    OUTCOME_NOT_SENT = 1;
    // OUTCOME_REJECTED: the resolver's result failed a check (reason: the check).
    OUTCOME_REJECTED = 2;
    // OUTCOME_FAILED: the resolver failed on the file (reason: the error, such as a
    // model error or "AI budget exhausted").
    OUTCOME_FAILED = 3;
    // OUTCOME_RESOLVED: the file was resolved, but the land failed because another
    // file was not.
    OUTCOME_RESOLVED = 4;
    // OUTCOME_NOT_ATTEMPTED: the request was not resolved at all (see
    // LandConflict.reason).
    OUTCOME_NOT_ATTEMPTED = 5;
  }
  bytes path = 1;
  Kind kind = 2;
  // base, ours and theirs are the sides; an absent side is unset. ours is the tip's
  // side, theirs the landed commit's.
  LandConflictSide base = 3;
  LandConflictSide ours = 4;
  LandConflictSide theirs = 5;
  // hunks are the conflicting regions, in order.
  repeated LandConflictHunk hunks = 6;
  Outcome outcome = 7;
  // reason explains the outcome, cut to 512 bytes.
  string reason = 8;
}

// LandConflictSide is one side of a conflicting file; its content is read by oid.
message LandConflictSide {
  string oid = 1;
  string mode = 2;
}

// LandConflictHunk is a conflicting region as line ranges of each side, 1-based and
// half-open ([start, end)); an empty range (start == end) is an insertion point.
message LandConflictHunk {
  LineRange base = 1;
  LineRange ours = 2;
  LineRange theirs = 3;
}

// LineRange is a half-open range of 1-based lines.
message LineRange {
  uint32 start = 1;
  uint32 end = 2;
}

// LandResolution is the caller's resolution of the conflicts of a LandConflict.
message LandResolution {
  // tip is the tip that the resolution was made against, from LandConflict.tip.
  string tip = 1;
  // commit is the landed commit, LandConflict.commit; it must equal commit_id.
  string commit = 2;
  // files resolve every conflicting path exactly once.
  repeated LandResolvedFile files = 3;
}

// LandResolvedFile resolves one conflicting path.
message LandResolvedFile {
  // Action is what becomes of the path.
  enum Action {
    ACTION_UNSPECIFIED = 0;
    ACTION_WRITE = 1;
    ACTION_DELETE = 2;
  }
  bytes path = 1;
  Action action = 2;
  // mode is 100644, 100755 or 120000 for ACTION_WRITE.
  string mode = 3;
  bytes content = 4;
}

UserLandResponse is unchanged: a land with a resolution reports its paths in resolved_paths and agent/<actor> in resolver, as a resolved land does today.

6.2 Configuration and flags

  • Server [graft.land.resolver]: lockfiles, generated_markers, minified (globs and thresholds), [graft.land.resolver.verify].
  • resolve.toml: the llm strategy without confidence options; [strategy.llm.budget] (3.5). The syntax check has no option.
  • graft-resolve bench mine|run: --corpus, --manifest, --sample, --seed, --max-usd, --input-usd-per-million, --output-usd-per-million, --json-out, --md-out.

7. Work packages

  • WP1 (Opus): miner, manifest, corpus file per family with licenses; the bench runner with a fake provider (the $ cap tested); the report.
  • WP2 (Opus): the llm strategy without confidence, the messages in the prompt.
  • WP3 (Opus): the syntax check with go-tree-sitter and the grammars (cgo, in tools/graft-agents only); RESOLVE.md build requirement.
  • WP4 (Opus): the monthly budget: reservation, booking, state file, metrics.
  • WP5 (Opus): the server's never-send rules (3.4) with reasons.
  • WP6 (Opus, proto of 6.1 after the PM's review): the package and the re-land; SDK and CLI.
  • WP7 (Opus): optional merged-tree verification.
  • WP8 (Opus + Haiku drafts): RESOLVE.md and the benchmark report.
  • The measured run, once Federico gives the key (one-off, $10).

The order puts the benchmark first: WP1, WP2, WP3 and WP4 are what the measured run needs; WP5 to WP8 follow, so that WP96 of PLAN9 can start once WP1 to WP4 are under way.

8. Test plan

  • Miner: a fixture repository with crafted merges (content, add-add, modify-delete, clean, binary, a lockfile, a generated file) gives exactly the expected cases; the manifest replays identically.
  • Runner: a fake provider that reports usage, with prices, stops before the call that could pass the cap; the judgement on crafted pairs.
  • llm strategy: answers without confidence accepted; removed options rejected at startup; the messages in the prompt, bounded.
  • Syntax check: valid and broken files for each grammar; an unknown extension gets the marker rule only.
  • Budget: reservation and booking with a fake provider; the cap reached; the state across a restart and two processes; a new month; a corrupt state file.
  • Never-send: each rule with its reason; .gitattributes linguist-generated read from the tip.
  • Package and re-land: every outcome and reason; re-land accepted; after the tip moved; wrong paths; markers; a resolution over the gRPC size.
  • The existing resolve, land and e2e suites unchanged.

9. Acceptance

  • The benchmark report per family: correct, wrong, rejected by checks, not sent, model error, not run (cap), error rate; within $10; corpus and manifest committed; the model and its prices named.
  • The monthly cap enforced across restarts, with its metrics.
  • The package and re-land end to end through the SDK and the CLI.
  • All tests green; the land path and e2e unchanged.

10. Rejected alternatives (by Federico's decisions)

  • Two-model agreement or a judge model; self-declared confidence.
  • Deterministic or syntax-aware tiers (Mergiraf) before the model by default.
  • A formatter or build gate before the land; a lockfile strategy; a secrets deny list by default.
  • File contents in the gRPC error: limited size, and the agent can read the blobs.
  • Replacing merge_conflict with the new package in UserLandError's oneof: it would break the clients that read merge_conflict; an extra detail does not.

11. Risks and questions

  • The corpus yield: projects that squash their pull requests have few conflicting merges; the miner reports the yield and the corpus keeps the repositories that give enough cases per family.
  • The price of openai/gpt-5-mini is entered as a flag and in the budget configuration; if OpenRouter changes it, the operator updates it, and the cap is only as right as the configured price.
  • A resolution larger than the 4 MiB gRPC limit cannot be re-landed through the request (a stated limit).

12. Amendments after approval (the PM's and Federico's decisions)

  • Model. openai/gpt-6.1-sol, the current GPT on OpenRouter, replaces openai/gpt-5-mini. The model is a configuration value, so another current model (for example openai/gpt-6-luna) can be measured later without a code change. Its price is never taken from memory: graft-resolve bench run reads it at run time from OpenRouter's model API (https://openrouter.ai/api/v1/models, the pricing of the model's endpoints: prompt, completion and, when listed, internal reasoning, in USD per token), takes the highest of the endpoints as the worst case, writes it into the report, and refuses to run when the model or its price cannot be read. The production budget (3.5) keeps configured prices, which the operator sets from the same page. The hard $10 cap and the per-call worst-case reservation stay; with third-party figures of about $2 input and $10 output per million tokens the sample shrinks to what fits (about 150 to 200 cases), and the report states the final N.

  • Re-land with a resolution (6.1, condition 1). The server accepts LandResolution only when resolution.tip equals the branch's current tip and resolution.commit equals commit_id; it may write only the paths that conflict at that tip, each exactly once, no other path; it runs the same validation as a resolver answer, the syntax check (3.2) included, besides markers, modes and sizes; and it records the merge commit with Graft-Resolved-By: agent/<actor>. Otherwise the land fails with a fresh package, never a partial write. (This replaces "no syntax check" in 3.3.) The syntax check therefore also runs in the server for an agent's resolution; since the main module takes no tree-sitter dependency, the server asks graft-resolve to check it (a check request of the resolver protocol) when a resolver is configured, and applies the marker rule only otherwise; WP6 details it for the PM before implementing.

  • Hunk ranges (6.1, condition 2). Each LineRange of a LandConflictHunk is relative to the side it names: base counts the lines of the base side's file, ours of the tip's, theirs of the landed commit's; 1-based, half-open ([start, end)), an empty range being an insertion point before line start. The proto comments state it.

  • The 4 MiB limit (6.1, condition 3). RESOLVE.md states it, and the SDK refuses a resolution whose encoded request would exceed it, with a clear error, instead of a transport failure.

  • Order. WP1 to WP4 first; WP96 of PLAN9 after them.

  • Per-file outcomes of a refusal (PM, WP6). The resolver protocol gains an optional files list in the error response (HTTP 422 body, command-mode stdout): the outcome of every requested file, resolved, rejected or failed. The server trusts it only when it has exactly one entry per requested path with a known outcome, and otherwise marks every sent file failed with the overall reason. Backward compatible both ways.

  • Later improvement, not now (PM). Since nothing lands partially, the files a resolver resolved in a failed request are wasted work for the agent. The package could offer the resolver's resolved content to the agent (for example as blobs written to the repository, referenced by OID), so that the agent re-lands with it where it agrees.

  • Review of WP5/WP6 (independent review of f707b5b and ad68d37, PM: fix all). What changed from the text above:

    • A refusal is never written. Only an accepted answer is, and a refusal whose outcomes mark every file resolved contradicts itself, so it is malformed (every file failed).
    • Every Graft-* trailer is written by one function, which refuses any value that is not one line of printable UTF-8. Such a path is never sent to a resolver, and the re-land refuses it (FailedPrecondition).
    • A side's blob OID is in the package only when the blob is in the tree of the merge base, the tip or the landed commit. A blob that merge-tree wrote for the conflict (the content merge of a rename) lives in the request's quarantine, which a failed land discards. Such a side gets unavailable with the reason; the CLI leaves it out without failing the other sides or files.
    • actors gates only the requests for an AI resolution. An agent's re-land is allowed for every actor that may land, and its check runs whatever the actor and covers all its files. max_files bounds the AI request only. The package lists every conflicting file, with bounds of its own: 1000 files (a refusal above), 1000 merge messages, 4 MiB encoded, 256 hunks per file, 64 MiB of sides read and 64 Mi cells of diff work for the hunks; hunks_omitted says why a file lacks hunks. The package is built only when the land returns it.
    • A merge that is unclean without a conflicting file (a directory rename split) gets a package that describes it by git's messages (messages), which every package carries.
    • Minors: a re-land with STRATEGY_FF_ONLY is InvalidArgument; a written mode must be one of the modes of the sides; a symbolic link named like .gitmodules, .gitattributes, .gitignore or .mailmap (in the ways HFS+ and NTFS fold names too) is refused; a symbolic link answer has syntax none; the syntax_unchecked metric counts once, on a successful land; merge3 writes CRLF markers where git does; the CLI layout of the sides is DIR/sides/<side>/<path> (<path>.base could collide with a directory named <path>.base), the CLI refuses paths it cannot write on its system and paths the file system folds together, and never overwrites a conflicts directory; the memory backend refuses a resolution for another commit.
  • Re-review of 54189d1 (PM: fix before R1).

    • The re-land of an agent is bounded: a merge with more than 1000 conflicting files is refused before anything is read, and its check goes in requests of at most 32 MiB of file data each, checked one after the other (resolver.MaxCheckRequestBytes = 64 MiB encoded, which the client also enforces), so that the server and graft-resolve hold one bounded request at a time.
    • The CLI never drops a side: without the base the file to edit is the two-way merge; without ours or theirs it is a conflict between markers (the side there is against a note), which the marker rule refuses until the agent resolves it.
    • A file with an unavailable side gets no hunks; the tree of every merge base counts as kept (a criss-cross base side that is a real blob keeps its OID); a gitlink side is unavailable.
    • Minors: the collision error names both paths; the colon is refused on Windows only; the NTFS names follow git's rules, the hashed 8.3 forms included; only the last paragraph of a message is its trailer block, and the server's trailers are always that paragraph (documented, tested).
  • Second re-review of 38d8bb1 (PM: fix before R1).

    • The check of an agent's resolution runs before the land enters its lane (landPrecheckResolution in UserLand), against resolution.tip: the only tip the land accepts the resolution at, so no re-check is needed. In the lane the land applies the verdicts when the tip and the commit are those the check ran on, and fails closed otherwise; if the branch moved, the resolution is refused with a fresh package before any verdict is used. No branch is held while the external process runs. The whole check has one deadline (the resolver's timeout); each request uses what remains, and expiry fails closed, the files checked before keeping their verdicts.
    • Check requests carry no commit message; their file data is bounded by resolver.CheckBatchBytes with a per-file allowance for paths, and max_file_bytes is refused at startup above resolver.MaxCheckFileBytes (about 12 MiB), so that one file always fits one request.
    • The listings of the trees and the attributes are read once per land, not per batch.
    • The CLI never falls back to one side: when it cannot merge a file, the file to edit is one diff3 conflict holding every side (pointer lines for binary sides, the text of text sides, notes for deleted and unavailable ones). --resolved refuses a pointer line, and a path that is a file on none of its sides can only be deleted.
  • Review of e661706 (PM: fix before R1).

    • N1. A resolver configuration the server could not build is a startup error (config validation: empty command, URL without http(s) or host, unreadable or empty token file, max_file_bytes over the check bound, negative bounds). If resolver.New still fails at run time, the server keeps the error (resolverErr) and every land that needs the resolver fails closed with it: no path turns a configured resolver into "no resolver".
    • N2. The precheck runs in an object quarantine that UserLand discards. Before, merge-tree wrote the merged trees and the blobs with markers into the repository's own objects, unreachable, until gc, in every storage mode (off, shadow, authoritative: all keep objects in the local repository; they never reached the bucket, whose packs carry only the objects that new reference values reach). The lane still merges again in its own quarantine: reusing the precheck's tree would mean keeping its objects; correctness first.
    • N3. resolver.Client holds check_concurrency slots (default 2); a check takes one per request, its wait counts against the check's deadline, and its expiry fails closed; the wait is the histogram graft_resolver_check_slot_wait_seconds. The client's check is reachable only through a slot.
    • N4. Before any check the precheck reads the branch: a tip other than resolution.tip, or a change already landed, is not checked; the lane, which reads the tip again under the lane, refuses it (tip moved) or answers it (already landed).
    • N5. A failure of the precheck is the result's failure with its cause (a git error, the slot wait, a request), which the package reports, still failing closed; a precheck that was skipped, or made for another tip or commit, says so.
    • Partial 5: everything that stops the check between or during batches keeps the verdicts found before, the rest not attempted, and the land gets the package. Partial 7: the CLI's base note says "both sides added the file" only when both sides are there. Partial 8: tests for several files in one request, a file at the bound with a 4 KiB path fitting one request (one byte more refused by the configuration), and the collision test named for what it tests.
    • The classify cache is fixed to the land's tip and paths when made; another tip or path is an error. The pointer-line rule is in the server's validation (resolver.ConflictPointerPrefix), so clients of the SDK get it; the CLI keeps its earlier message.
  • Re-review of fcf5f8d (PM: fix first).

    • The resolver client is built once per process (resolver.Shared in service.Dependencies, as the land lanes are): its check slots bound the process across listeners, and a configuration that cannot be built is one error. A server whose dependencies carry none builds its own.
    • A panic can no longer lose a check slot or the precheck's quarantine: both are released by defer (a test panics inside a batch and takes the only slot again).
    • The pointer-line rule exempts a file whose conflict has a side with such a line of its own, in Validate (sides from the request), on the server for agents' resolutions (the sides are read only when the rule fires) and in the CLI.
    • The precheck reads the branch as the lane does (ResolveRevision), so a branch that is a symbolic reference is checked at its target's tip.
    • A check request that runs out of the check's deadline says so; one that runs out of its own timeout names the timeout.

13. Prompt first (PM and Federico, approved)

Direction from Federico: the prompt first, then the models; adopt a structure that is already used and measured instead of inventing one. The base is synthmerge's patch-application design (Andrea Arcangeli, synthmerge): the only published resolver measured against real human resolutions, and one that lands a change onto a tip, as a land does. synthmerge is GPL-3.0-or-later or AGPL: we take the structure only, written in our own words, with our own worked example; RESOLVE.md credits it.

R1, the structure (tools/graft-agents/resolve/llm_patch.go, the only mode of the llm strategy; the whole-file mode and the JSON answers are removed):

  • One model call per conflicting region; the regions only one side changed are merged locally.
  • PATCH: THEIRS' change to the region, a unified diff from BASE with 3 context lines (synthmerge's default). CODE: OURS' region with the same 3 lines before and after, taken from the file with every other conflicting region as OURS. The task: apply the PATCH to the CODE.
  • System message: the worked example (its sections tagged EXAMPLE ..., so that it never reads as the answer), THEIRS' commit message, as synthmerge sends the change's message with its commit diff, the other conflicting regions of the file for reference, the directives last. User message: the PATCH, then the CODE. (This text first said "both commit messages", citing it as Federico's decision; the record is the PM's answer to Q3, "both commit messages in the prompt by default", not a decision of Federico's. On the PM's ruling R1 sends THEIRS' message only, and both messages come back, if at all, as a measured step of their own.)
  • Plain-text answer between BEGIN and END lines carrying the request's random token; the last such block is read. A "\ No newline at end of ..." note is accepted only as its last line (no final newline); anywhere else it rejects the answer.
  • Validation as synthmerge's: the answer starts and ends with the context lines (blank lines at a boundary are given back, up to n-2 of them, n the context lines at that boundary), holds both, what is between them ends with a newline and does not read as a patch; plus our marker rule and size bound.
  • An answer equal to OURS is kept: synthmerge's default (it refuses one only on request, and its bench does not), and the report counts how many answers equal OURS anyway, per file and per region (synthmerge's no-change check is per region).
  • No retry in R1 (faithful: synthmerge's retries shrink the context).
  • Each model call has its own bound (call_timeout, 120 s), within the file's bound, which holds max_regions such calls (2400 s in the run's configuration; R0 had 180 s for all the regions of a file), so that a file with many regions does not fail for want of time.
  • Budget: the provider's transport retries of a call share its one booking, so a 5xx answer that was billed and then retried can be billed twice while booked once (rare). To fix before production: book every attempt.

PM's additions:

  1. A file is resolved only when every region is: the first rejected region rejects the file, which goes to the agent without a proposal; the report counts rejections by their first reason.
  2. At most 20 regions per file (max_regions); a file with more makes no call and is reported as not sent (regions). Every call books its worst case against the budget first.
  3. The answer equal to OURS: see above.
  4. Deviation (a): the commit diff of THEIRS is not sent; it joins P5 (a protocol field), measured as its own step later.
  5. No retry in R1.
  6. Unit tests use recorded answers and fake servers only, never the network.
  7. The run script reads the account's credits before every run and refuses to start unless the balance covers the cap plus $0.50; the key is read from its file and never printed.

Deviations, measured or stated:

  • (b) The PATCH is our LCS line diff, not git's histogram diff: on the 300 paired cases (288 content conflicts, 403 regions) the two differ in 18 regions of 17 cases (bench histogram).
  • (c) Our worked example is a different snippet, written by us.
  • (d) One model and no consensus, as Federico decided.
  • (e) Leniency: a region at the end of a file may end without a newline when neither side has a final newline; synthmerge requires the newline.

R1 result and addendum (r1-20261019-101514, real spend $0.099): strict error 41.8% (R0 on the same 300, re-judged: 53.8%); both accepted, wrong->correct 37 and correct->wrong 13, McNemar p = 0.0009; the bound with reorder and whitespace counted correct 32.9%; human-outside 29 cases, rate without them 33.2%. Of the 42 anchor rejections, 22 were CRLF files, and no CRLF file was accepted: the context went to the model with CR and had to come back byte for byte, which models do not do. Our bug, fixed before R2: a CRLF file is sent as LF and its answer converted back (mixed line endings are sent as they are, and the report counts the cases by line endings). The bench keeps the raw text of every rejected answer (.rejected.txt beside the answers, 0600), so that rejections can be classified. R1b, R1 with the fix, is the baseline of R2.

Measurement. Luna, effort low, max_tokens 8000, routing pinned to the standard tiers, the same 300 paired cases (--sample 300 --seed 1, the first 300 of the Luna 600), cap $1.20 a run, the whole series under $5 real, and a run over $1.00 real stops the series until the PM is told.

  • R0: the stored Luna answers on those 300, re-judged ($0).
  • R1: the structure above.
  • Then one extra at a time, each kept only if it helps on the same cases (lower strict error rate, more wrong-to-correct than correct-to-wrong flips): R2 one repair round with the same model and a duplicate-declaration check (Shen 2026's generate-validate-retry), R3 the enclosing scope of the region, R4 temperature 0, R5 the best so far at effort medium. P5 (the commit diff) and P6b (the commit subjects of the regions) need protocol fields and wait for their own approval.
  • Every report gives, per language: the strict error rate, the bound with reordering and whitespace counted as correct, the cases where the human changed lines outside the conflict regions as their own category and a rate without them, acceptance and rejections by reason, the flips against the previous step and R0 with an exact McNemar p-value, and the spend.

14. The agent always resolves; the server's AI only proposes (Federico, PM-approved design)

Final ruling (Federico, 2026-10-19, relayed by the PM; it supersedes the amendment at the end of this section and every statement below that the server never lands a model's resolution). An AI resolution lands only through the operator's verification (15.9).

  • Server AI configured and verify.command configured: the server resolves. If every deterministic check passes and verify.command passes on the merged tree, it LANDS, signed Graft-Resolved-By: and Graft-Checked-By: verify:. Otherwise it goes to the agent as a PROPOSAL, with the verify output when verify failed.
  • Server AI configured, no verify.command: proposal only, and the agent decides. An unverified AI resolution never lands.
  • No AI: the agent resolves.
  • Rules (trivial, union, rerere, command) are unchanged: they land after verify.command when one is configured, in the lane, as today.
  • The composition report (15.3) stays a report, not a gate.
  • One rule, in decideLand: a file that proposes may land only if verify.command ran and passed on the exact tree that lands, compared by tree object ID. No other code path lands a model's resolution.
  • The trailer names what ran: Graft-Checked-By: verify:. The syntax and operator checks run inside the resolver and are not separate trailers; the composition report is not built, so the trailer does not claim it.

This supersedes "AI lands with trace only". Before the ruling above, the server never landed a resolution of the llm strategy by itself: it was a proposal, which the agent reviews, edits or replaces, and lands with its own resolution (UserLandRequest.resolution, Graft-Resolved-By: agent/). The deterministic strategies that the operator configures (trivial, union, rerere, command) are rules, not AI: a resolution made by rules alone still lands. The llm auto-land path is removed, not kept behind a flag.

  • Protocol (additive). A ResponseFile gains "proposal": true; graft-resolve sets it on every file whose result came from the llm strategy, and only there. The server treats an answer with any proposal file as not landable. The name-based fallback this item had (an answer whose resolver names the llm strategy) is removed by the review of f404a71: it read only the first "x/" of a name, so a list joined over rounds, a list cut to ",+N" or another vendor's name missed it. Replaced by the fail-safe origin of 15.9.

  • Server. With resolve_conflicts, a resolution by rules alone lands as today. When a file is a proposal, the land fails with the LandConflict package: proposal files get OUTCOME_PROPOSED, files that rules resolved keep OUTCOME_RESOLVED, and nothing lands partially. Every file with content carries LandConflictFile.proposal (OID, mode, action, strategy, and the explanation cut to 512 bytes), so that the agent can re-land rule results unchanged. No llm trailer exists any more; the agent's re-land records the agent.

  • Delivery: a hidden reference. On a failed land with proposals the server writes the proposed blobs and one commit (its tree the proposals by path, its message the tip, the commit and the strategies), written into the repository itself (the land's quarantine is discarded), and updates refs/graft/proposals/<first 16 hex of sha256(branch)>/<commit> through the same reference transaction a land uses. One reference per (branch, landed commit), replaced by every failed land of that commit. On every land of a branch that asks for or brings a resolution (a plain land pays nothing; only those lands write proposals) the server deletes the branch's proposal references older than proposal_ttl (default 7 days) and keeps at most 20 per branch; the agent's land of the change drops its proposal. When the proposals cannot be stored the package still goes to the agent, each file of the answer saying "proposal not stored: " (logged and counted): the proposals are an aid, the package is the contract (review ruling M2).

  • (a) Authoritative storage mode (checked). UserLand updates references through the reference-transaction hook (STORAGE.md, the voter): the proposal reference and its expiry deletions go through it like the branch, so in authoritative mode each is a changeset committed with the database compare-and-swap, with the pack of its objects uploaded before. They are not changelog entries: changelog.ParseUpdates and corestore.IsChangelogRef admit only refs/heads/ and refs/tags/. The objects reach other nodes with the changeset pack, so the agent can read them on any node.

  • (b) Visibility and access (checked). refs/graft/ is not hidden today: git.InternalRefPrefixes lists refs/environments, refs/keep-around, refs/merge-requests, refs/pipelines, refs/remotes and refs/tmp only. refs/graft/proposals/ is added as an InternalReferenceTypeHidden prefix, so receive-pack hides it (no client can create, update or delete it by a push) and upload-pack does not advertise it. Agents cannot write references directly (WriteRef, UpdateReferences and DeleteRefs are never allowed for agents, AGENT-SECURITY.md). The agent reads the proposal with ObjectService.GetObjects, which agents may call and which reads objects by OID with git cat-file, without a reachability check. Observation, left as it is: graft-verify's refs/graft/green/ is not hidden either (advertised and pushable).

  • CLI and SDK. graft land --conflicts-dir DIR writes each proposal to DIR/proposals/

  • The PM's four conditions (implemented).

    1. One writer, one prefix: landUpdateProposalRefs is the only function that writes these references. It refuses any name but refs/graft/proposals/<16 hex>/ before git runs, and updates as the server (no request identity), so the agent's policy does not judge the server's write. It is called only by landWriteProposals and the pruning.
    2. refs/graft/** is reserved: the reference-transaction hook refuses any update under it by an agent identity, whatever the policy says and even with no policy, so a push, UpdateReferences, DeleteRefs, WriteRef or an operation RPC fails. CheckAgentPolicy refuses it early too. graft-verify writes refs/graft/green/ with the shared token, which is not an agent.
    3. Hidden: refs/graft/proposals/ is hidden from advertisement and push. Two points are not covered, stated here: graft-verify's refs/graft/green/ is still advertised (hiding it changes graft-verify's users and needs its own decision), and an agent's read accessors (ListRefs, FindRefsByOID and the like) still list the names and OIDs of refs/graft/ references (no content; the blobs are read by OID as before). Filtering those is a change of the ref service for agents, proposed as its own step.
    4. Trace: the proposals' commit carries Requested-By: <actor> (the agent, or the user ID of the request) and the resolver; the server logs the reference, the actor and the resolver. The update itself is the server's. An end-to-end test runs the real graft-resolve with a model behind the command provider, under the example policy that gives the agent no refs/graft/ rule: the server proposes, the SDK reads the proposal, the agent lands it as its own (Graft-Resolved-By: agent/), and the proposal reference is dropped.
    5. Review of e33a723 + b54b529 (PM rulings), implemented.

      • M1, condition 3: refs/graft/** is internal for agents. One rule (internal/graft/agentrefs), applied once per transport: the agent guard of the gRPC server refuses a request whose revision, reference or pattern names refs/graft/ (the short forms graft/proposals/... and graft/green/..., which git completes to the server's namespaces, included; other graft/... short names are branches, the workspace's temporary branches graft//... among them, which the first, broader rule refused and the suite caught), and removes such references from every response; git upload-pack and receive-pack hide the prefix from agents. A table-driven test classifies every string or bytes field of every RPC agents may call by default (a new one fails it); RPCs added with agent_rpcs are checked by field name. graft-verify's refs/graft/green/ stays advertised to other callers: VERIFY.md documents it for consumers (the newest tested tip), which are not agents; agents do not see it.
      • M2: a proposal that cannot be stored no longer turns the conflict into Internal: the package comes without it, each file of the answer saying "proposal not stored: ", logged and counted (graft_land_proposal_write_errors_total).
      • M3: the proposals' commit is dated by the server's clock, never the request's timestamp; the housekeeping pass prunes the proposals of every branch (WithGraftProposalTTL), so idle and deleted branches expire, except in a repository housekept in a WAL transaction, where a reference update would bypass the transaction (logged; those expire through their branch's lands); an agent the policy does not let update the branch gets PermissionDenied before anything is resolved, and no proposal is kept.
      • m1: every deletion expects the target it read (internal/graft/proposals, the one writer and the one pruner, used by UserLand and housekeeping). m2: decideLand is the one decision point, with the guard "a file that proposes is never landed"; that is where a future gate lives. m4: the Resolver and Requested-By values pass landTrailerValueValid, and are quoted otherwise (the user ID is the caller's). m5: the e2e test names the llm strategy "model", so the flag, not the name fallback, makes the proposal; the release note (the PM's file) says that rerere entries recorded from llm answers before b54b529 must be cleared.
      • m6, measured: writing the proposals adds a median of 97 ms (min 69, p90 113, max 134 ms on the test machine) to the lane's hold; the resolver's own call, already in the lane for resolve_conflicts, takes seconds. Moving both out of the lane, like the precheck of an agent's resolution, is a step of its own.
    6. Amendment: Federico's product decision (2026-10-19, relayed by the PM). Superseded by the final ruling at the top of this section: the gate is verify.command, not the composition check. "When the server AI is configured, the server always tries to resolve, and it LANDS its resolution by itself ONLY if the proposal passes every deterministic check: syntax, anchors, no markers, and the composition check from §15 (keeps both changes, no duplicated both-added lines). There is still no self-declared confidence. If any check fails, the resolution goes to the agent as a proposal, with its report. If no AI is configured, the agent always resolves."

      • Gated on §15: until the composition check exists and is measured, a model's resolution stays a proposal, as implemented above. The composition check is what turns a proposal into a landable resolution; the rule "the server never lands llm output" becomes "the server lands llm output only when the composition check passed in this process", in one function, with tests.
      • Before it is enabled: the coverage (how many land by themselves) and the precision (how many of those are wrong), per language with a confidence interval, on the 300 cases; the default goes on only with Federico's OK after he sees them.
      • An AI-landed merge is signed Graft-Resolved-By: llm/ and Graft-Checked-By: syntax,composition, and the trace records the proposal and the report.
      • Everything above still applies: a resolution that fails a check is a proposal.
    7. 15. gitai's resolver: the design, what each part is for, what was measured

      Principle. A conflict is two changes to one base. Only the server holds the base, both histories, the landing commit and the agent's request, so the server is the place to check what it can check mechanically, and the agent, who decides, gets a proposal with an honest report of what was checked, not a bare guess. A model's resolution never lands because of anything the model says or any heuristic about its answer: it lands only on a tree that the operator's verification passed (15.4).

      This section is the design as it stands, in one piece. 15.0 is the whole system on one page; 15.1 to 15.5 are the diagrams and the table that tie every box to its code (a code path that is in no diagram is a patch and is removed; a commit that changes the flow changes the diagrams in the same commit); 15.6 to 15.11 say what each part is for and the measurement behind it, and what was tried and removed; 15.12 is the operating point, with its intervals; 15.13 is what is configurable and its defaults. The log of the measurements, with commits, costs and files, is Appendix A at the end of this section.

      15.0 The system on one page

       conflict of a land  (the server computes the facts)
         |
         v
       git's three-way merge: the conflicting files, their base / ours / theirs, the regions
         |
         v
       classify (policy, size, binary, lockfile, generated, minified)  -- not sendable --> the AGENT, with the reason
         |
         v
       operator rules, in order: trivial, union, rerere, command   -- a rule resolved it --> VERDICT of a rule
         |
         v
       the WRITER (the llm strategy; 15.10): one model call per region
           request = the PATCH (theirs' change), the CODE (ours), THEIRS' commit message, the other regions
           answer  = the patched code, plain text
           deterministic checks of the answer; ONE repair round if the answer is BROKEN (a rule, the syntax)
         |
         v
       file checks: markers, size, syntax (tree-sitter), the operator's checks
         |
         v
       composition report (15.3): what the answer keeps, drops, repeats  -- a REPORT, not a gate
         |
         v
       agree_with (optional, off by default; 15.12, 15.13): a second model from another vendor resolved the same file
       at the same time; the two results are not the same file, or a report is not clean  -- HOLD
         |
         v
       VERDICT: PROPOSAL (origin "model") with its report; a HELD proposal never lands
         |
         v
       VERIFY: the operator's verify.command on the exact tree that would land, out of the lane (15.9)
         |
         v
       decideLand (15.4): LAND only if the verified tree is the tree to land -- else PACKAGE to the agent
      
      step what it does who decides justified by
      git facts regions, sides, base; the merge itself git 15.10 (the regions are git's: fewer wrong "no conflict")
      classify what is never sent rules (path, size, bytes) 3.4
      rules resolutions that need no model the operator's rules RESOLVE.md
      writer proposes a resolution per region a model, configurable (15.13) 15.10: the elements kept and removed, 15.12
      checks and repair refuse a broken answer, repair it once deterministic checks 15.10
      report tells the agent what the answer keeps or drops deterministic 15.6: it separates (43% against 60% right), it does not gate
      agree_with (optional) holds a proposal back unless a second model from another vendor gives the same file and neither composition check found a drop or a duplicate code (llmStrategy), configured by the operator 15.12: G1, 59.8% of the files landed with 15.5% wrong under two judges and at most 15.5% under a third, before the operator's tests
      verify runs the operator's build and tests on the exact tree the operator's command 15.9, 14
      decideLand the one landing rule code 15.4

      Not in the system, and why. Four ideas were built as measurements, and none of them is a landing signal on the evidence we have; each is kept as a measurement (Appendix A) and not as code: a decision model that chooses OURS / THEIRS / both per region and lets a writer work only where it says NEEDS-MERGE (15.11: a java selector, nothing near the bar elsewhere); agreement of arms of one model, and of several models on every region (15.12: models make the same mistakes); structure and self-agreement rules (15.12: they do not hold on held-out repositories); a build-and-vet simulation of the verification in the bench (15.12: it removes the answers that do not compile and no more). What does move the number is the operator's own tests, which only the operator has. What is in the system, as an option that is off by default, is the agreement of two models from different vendors on the whole file (agree_with, 15.13): the one filter that chose better proposals than the writer alone.

      15.1 End to end

       agent: UserLand(branch, commit[, resolve_conflicts | resolution])
         |
         |-- resolution given? --yes--> PRECHECK, before the lane, in a discarded quarantine
         |                               branch moved / change already landed -> skipped (lane answers)
         |                               review: exact paths, modes, markers, pointer lines, size
         |                               resolver check (syntax + operator checks), bounded slots
         |                               VERIFY (below) the exact tree the resolution would land: fails ->
         |                               PACKAGE (every file REJECTED, the output); not in time -> PACKAGE
         |
         |-- resolve_conflicts? --yes--> PRE-RESOLVE, before the lane, in a discarded quarantine (15.8)
         |                               tip T read as the lane reads it; no branch / already landed /
         |                               the merge of T and the change is clean -> skipped (lane answers)
         |                               agent may not update the branch (policy) -> PermissionDenied
         |                               the conflicting files not yet resolved -> one VERDICT per file
         |                               (15.2), each kept by its key (path, base, ours, theirs, modes),
         |                               resolve slots; VERIFY (the one verify box, 15.9): when verify.command
         |                               is configured, or the answer holds a model's resolution, it runs on
         |                               the exact tree that would land, with a slot of the verify pool
         |                               (verify_concurrency, verify_per_repository); decideLand (15.4):
         |                                 not landable -> proposals kept (14) and PACKAGE now: the lane
         |                                                 is not entered; a model's files are PROPOSED,
         |                                                 a rule's REJECTED, with the verify output when
         |                                                 verify failed; NOT_ATTEMPTED (a rule's) or PROPOSED
         |                                                 (a model's) with the reason when the deadline cut it
         |                                 landable ----> the verdicts, and the verified tree, go to the lane
         v
       LANE (one per branch)
         |-- a resolution -----> landWithResolution: tip moved -> refuse, fresh PACKAGE
         |                         applies the precheck's verdicts; any failure -> PACKAGE
         |                         the tree it would write is the tree the precheck verified, else PACKAGE
         |                         (the lane runs no command) -> LAND as agent/<actor> (+ Graft-Checked-By)
         v
       3-way merge (rebase or merge) of the tip and the change
         |-- clean -------------------------------------------------------------> LAND
         |-- conflicts, no resolve_conflicts ------------------------------------> merge_conflict
         |-- conflicts, every conflicting file has a verdict with its key, and the lane's
         |   classification of the file agrees with the verdict's
         |       decideLand (15.4) on the tree the lane would write: with verify.enabled set, any
         |                     resolution (a rule's, a model's) LANDS only if that tree is the tree
         |                     verify.command passed on (tree object IDs), and a model's on no other tree,
         |                     configured or not. The lane runs no verification of any resolution. Merge
         |                     commit signed Graft-Resolved-By: <resolver> + Graft-Checked-By: verify:<command>.
         |                     Another tree (the tip moved) -> NEEDS VERIFICATION, below
         |-- conflicts, a file without such a verdict (new, or its sides changed)
         |       -> NEEDS RESOLUTION, out of the lane: PRE-RESOLVE only those files against the new tip,
         |          keeping the other verdicts, and enter the lane again
         |-- an answer whose tree is not the verified one (the tip moved)
                 -> NEEDS VERIFICATION, out of the lane: rebuild the tree from the same verdicts (the
                    resolver is called only for files that changed), VERIFY it, enter the lane again
       The rounds are bounded by the deadline of the land (the resolver's timeout, plus verify.timeout when
       verify.enabled is set) and by resolve_rounds (default 32, a safety cap, no contract). At
       the deadline: the final pass (15.9): the merge conflict with its PACKAGE, the model's files PROPOSED
       ("could not verify before the branch moved again"), the rules' files RESOLVED (NOT_ATTEMPTED with the
       reason when verify.enabled is set and their tree could not be verified), the files that changed
       under the land NOT_ATTEMPTED with the reason (a verified AI land is opportunistic: losing the race
       is harmless, the agent gets the proposal).
       The resolver is never called inside the lane, and neither is the verification of a model's resolution.
      
       PACKAGE to the agent (merge_conflict detail): sides by OID, hunks, per-file outcome + reason,
         per-file proposal (+ its report, in its explanation), the proposals kept under refs/graft/proposals/ (14),
         or, when they cannot be stored, "proposal not stored: <cause>" on each file
           -> the agent resolves (takes, edits or replaces the proposals) and re-lands with resolution
       every request and response of an agent: the refs/graft/ visibility filter (agentrefs);
       git upload-pack / receive-pack of an agent: refs/graft/ hidden
      

      15.2 Per file

       conflicting file
         |
         v
       classify: path policy, shape, size, binary, lockfile, generated, minified
         |-- not sendable ----------------------------------------> AGENT (NOT_SENT, reason)
         v
       operator rules in their order (trivial, union, rerere, command), each with the operator's checks
       (the syntax check is the model's: it runs on the llm strategy's answers only)
         |-- a rule resolved it ---------------------------------> VERDICT RULE (lands, 15.4)
         v
       AI configured? --no-------------------------------------> AGENT (FAILED: no strategy applied)
         | yes
         v
       split into regions by git's own merge (git merge-file --diff3; our diff only if git cannot run); regions
       only one side changed are merged locally
         |-- more than max_regions (20) ------------------------> AGENT (NOT_SENT: regions)
         v
       for each region, one call (15.10): PATCH = base->theirs (3 context lines), CODE = ours (3 anchors),
       THEIRS' commit message (capped), the other regions for reference; the answer is the last block:
       the patched code
         v
       per-region checks: anchors, newline rule, not a patch, size, markers
         |-- a rule is broken, and the region was not answered again -> ONE REPAIR ROUND: the same request,
         |       the previous answer and the rule quoted exactly; a second rejection rejects the region
         |-- any region fails -------------------------------------> AGENT (REJECTED, reason; no proposal:
         |                                                            all or nothing)
         v
       assemble the file; syntax of the file (tree-sitter): the first error is mapped to its region and, if
       that region was not answered again, quoted back in the same one repair round (a region is repaired
       once at most, and only a BROKEN answer is: an answer the rules and the syntax accept is never asked again)
       file checks: markers, tree-sitter syntax, operator checks
         |-- a check fails ----------------------------------------> AGENT (REJECTED, reason)
         v
       composition report (15.3) of the regions: drops, duplicates, contested changes (a report, not a gate,
       and not a repair trigger)
         v
       [agree_with set] the second model resolved the same file in parallel, with the same prompt:
         |-- its answer is not the same file after normalization (CRLF, trailing whitespace, trailing blank
         |   lines), or it failed, or a report of the two holds a drop or a duplicate
         |   ---------------------------------------------> HELD: the first model's proposal with the reason
         |   (PROPOSED, never lands, not even verified)
         v
       VERDICT PROPOSAL (PROPOSED, the proposal + its report)
         v
       verify.enabled set? --no--> stays a PROPOSAL: the agent decides ("no verify.command")
         | yes: the exact tree that would land is verified, out of the lane (15.9)
         |-- fails ------------------------------------------------> stays a PROPOSAL (+ the verify output)
         |-- not verified before the branch moved again -----------> stays a PROPOSAL (+ the reason)
         v
       passes --> LANDS, if it is still the tree to land (decideLand, 15.4), signed Graft-Checked-By: verify:...
      

      15.3 Composition report (a report, not a gate)

       base B, ours O, theirs T, result R (normalized lines)
         |
         v
       diff3 regions of (B, O, T); does R keep every clean segment, in order?
         |-- no --------------------------------------------------> "changes lines outside the conflicts"
         v
       each region's slot in R; each side's hunks of base->side inside the region
         |
         v
       hunk the other side did not touch (uncontested):
         its added lines in the slot, its deleted lines gone --> "keeps <side>'s change at a-b"
         otherwise -------------------------------------------> "drops <side>'s change at base a-b"
       hunk whose base lines the other side also edited (contested):
         --------------------------------------------------------> "contested region resolved freely"
                                                                    (nothing line-level to verify)
       a line both sides added, repeated more than either has it -> "duplicates a line both sides added"
       a line in none of B, O, T --------------------------------> "content in neither side" (warning)
      

      15.4 The one decision point

       decideLand(answer, files, tree, verified): the pre-resolution calls it with the tree it verified,
       the lane with the tree it would write
         every file resolved by an operator rule ...... LAND, and with verify.enabled set only if it ran
                                                        and passed on THIS tree (tree object ID equality)
         a file a model resolved ...................... LAND only if verify.command ran and passed on THIS
                                                        tree: tree object ID equality, not the tip
         a file the resolver holds (agree_with) ....... PACKAGE: never lands, whatever was verified; the
                                                        tree is not even verified for it
         anything else ................................ PACKAGE (the model's files are proposals)
       One rule about a model's resolution. No other code path lands it. No check disables itself: a configuration that
       cannot be used stops the server at startup, and a failure at run time refuses the land. No
       confidence that the model declares about itself decides anything.
      

      15.5 Box to code (the reviewer's audit table)

      diagram box function file
      UserLand, PRECHECK UserLand, landPrecheckInQuarantine, landPrecheckResolution operations/land.go, land_reland.go
      review of a resolution landReviewResolution, resolver.ValidateFile(WithSides), landModeOfSides operations/land_reland.go, land_resolve.go; graft/resolver/validate.go
      resolver check, bounded slots landCheckResolution, landCheckSlotted, Client.AcquireCheckSlot operations/land_reland.go; graft/resolver/check.go
      LANE, re-land landAttempt, landingAttempt.decide, landWithResolution operations/land.go, land_attempt.go, land_reland.go
      3-way merge, listing landingAttempt.decide, landListConflicts operations/land_attempt.go, land_resolve.go
      out of the lane: rounds, deadline landResolving, landPreResolve operations/land_preresolve.go
      per-file verdicts, decide landResolveAt, landSetOutcomes, decideLand, landAnswerProposes, landKeyOf operations/land_resolve.go, land_proposals.go, land_preresolve.go
      the tree that would land landResolvedTree operations/land_resolve.go
      VERIFY, the one box, out of the lane, for every resolution landVerifyOutOfLane, landVerify, landPrecheckVerify (an agent's), VerifyPool.Acquire operations/land_verify.go, land_reland.go; graft/resolver/check.go
      verdicts applied in the lane landApplyVerdicts, landNeedsResolution operations/land_resolve.go, land_preresolve.go
      the rules resolved in the lane (rules_only, inline_timeout) landResolveInline, landInlineEnabled, Server.landInlineOff, Request.RulesOnly, Response.RulesOnly, unsupportedBy and the skip of models in resolveFile operations/land_inline.go, land_resolve.go; graft/resolver/protocol.go; tools/.../resolve/resolve.go
      the queue of the lands that lose a round landTickets, landTicketRegistry.acquire/blocked, landTakeTicket, landTicketsBlock, the block in landApplyVerdicts operations/land_tickets.go, land_preresolve.go, land_resolve.go
      unverified in time: proposals landAnnotateUnverified, landStoppedReason, landFinalPass operations/land_resolve.go, land_preresolve.go
      resolver call, bounded slots Client.AcquireResolveSlot, ResolveSlot.Resolve graft/resolver/check.go
      policy before resolving hook.CheckAgentPolicy, landPolicyDeniedError operations/land_resolve.go, land_proposals.go
      classify landClassify, resolver.SendRules operations/land_resolve.go; graft/resolver
      operator rules, file checks Resolver.resolveFile, Resolver.runChecks, syntax.CheckMerge tools/graft-agents/resolve/resolve.go
      AI per file llmStrategy.Resolve, ResolveLLM, llmResolvePatch tools/.../resolve/llm_strategy.go, llm_core.go, llm_patch.go
      regions (the fallback of git's, and the assembly of the file) llmRegions, llmJoin tools/.../resolve/llm_core.go, llm_gitmerge.go
      one call per region, its checks llmPatchSystem, llmPatchUser, llmPatchAnswer, llmCheckPatched, llmCheckAnswer tools/.../resolve/llm_patch.go, llm_core.go
      proposal mark, no recording proposer, Resolver.resolveFile tools/.../resolve/llm_strategy.go, resolve.go
      agreement gate: second model, comparison, hold llmStrategy.Resolve, llmSide.start, llmStrategy.agreement, LLMNormalize, LLMResult.Findings, Resolution.Hold, ResponseFile.Hold (a held file is a model's), landAnswerHolds, decideLand, landSetOutcomes tools/.../resolve/llm_strategy.go, llm_core.go, resolve.go; graft/resolver/protocol.go; operations/land_proposals.go, land_resolve.go
      verify.command's checkout landCheckoutTree, landMaterializer.add, landCheckMaterialized, landRemoveCheckout, resolver.IsGitDirName operations/land_verify.go; graft/resolver/validate.go
      merge commit, trailers landWriteResolution operations/land_resolve.go
      PACKAGE landWithPackage, landListing.conflictPackage operations/land_package.go
      proposals, hidden reference landWriteProposals, proposals.Update, proposals.Prune, landProposalsNotStored operations/land_proposals.go; graft/proposals
      proposals' expiry in housekeeping RepositoryManager.pruneGraftProposals git/housekeeping/manager/optimize_repository.go
      refs/graft/ visibility filter (agents) checkAgentRefs, filterAgentRefs, agentRefField; agentrefs.RevisionHidden gitaly/server/auth/agent_refs.go; graft/agentrefs
      refs/graft/ hidden from agents' git hiddenUploadPackRefPrefixes, hiddenReceivePackRefPrefixes git/gitcmd/command_description.go
      refs/graft/ reserved for the server graftReservedDenial gitaly/hook/referencetransaction.go
      composition report llmCompose, llmReport, CompositionVerdict (whole file, for the parity test), landProposalExplanation tools/.../resolve/llm_compose.go, llm_compose_file.go, llm_patch.go; operations/land_proposals.go
      git's regions (our diff as the fallback) llmFileRegions, llmGitRegions, llmParseDiff3, llmUnified tools/.../resolve/llm_gitmerge.go, llm_core.go
      one repair round, for a broken answer llmResolvePatch (the repair closure), llmRepairUser, llmPreviousAnswer, llmRegionAtLine tools/.../resolve/llm_patch.go

      15.6 The composition check, measured: a report, not a gate

      The design first meant the check to turn a proposal into a landable resolution (an amendment of 14): every edit of each side must appear in the result unless the other side's edit to the same base lines supersedes it. Measured with zero model calls:

      • Definition, from the principle, frozen before measuring (sha256 9396326b..., checked before every run). A hunk is contested when the other side edited overlapping base lines (or touches it, for an insertion); the result may compose a contested hunk freely, so it is never a drop. An uncontested hunk must be in the result. Duplicates and changes outside the conflicts fail. Disclosure: before freezing it, 8 human resolutions from the 300 had been looked at with an earlier, stricter prototype; the frozen definition is measured on cases outside the 300.
      • Human resolutions outside the 300 (2644 content cases): 82.6% pass; 324 change lines outside the conflicts, 103 duplicate, 32 drop.
      • Power. Something verifiable (an uncontested hunk, or a line both sides added) exists in 20.0% of those cases and 19.2% of their regions (300: 29.9% of cases, 11.6% of regions). A diff3 region exists because both sides edited overlapping base lines, so most regions are wholly contested and pass vacuously.
      • As a gate on R1b's accepted answers: without it 143 of 244 correct, 58.6% [95% CI 52.3-64.6]; with it, 135 of 220 passes correct, 61.4% [54.8-67.6]. 175 of the 220 passes are vacuous (64.6% correct); the verified passes are 48.9% correct [35.0-63.0]. If only passes landed, 73% of the conflicts would land and about 39% of those would be wrong. The gate does not move precision.
      • Contested regions have no line-level property. A stricter prototype (every line either side added present unless superseded line by line) separated better on R1 (74.5% precision) but was built while looking at those cases, is still far below a landing bar (~90%), and fails half of the human merges (the right resolution of a contested statement is often a third version that holds neither side's line). Composition inside a contested region is semantic; a line-level check only measures overlap with the sides.

      Decisions.

      1. The AI stays proposal-only unless verify.command passes on the tree (the final ruling, 14 and 15.9).
      2. That amendment (AI landing when every deterministic check passes) is superseded by the final ruling: the gate is the operator's verify.command, because no deterministic check reaches the bar.
      3. The composition report goes with every proposal, honest about what it can say: "keeps every uncontested change of both sides" or "drops THEIRS' change at base 40-44", "duplicates a line both sides added", "changes lines outside the conflicts", "N contested regions resolved freely". The duplicates flag carries signal (7 of 19 correct where it fires, against 58.6% overall): it is useful in the report, and not as a repair trigger (item 4).
      4. The repair round is driven only by deterministic findings with their exact wording, and only for an answer that is broken (the rules of an answer, the syntax): the composition report's drops and duplicates were tried as repair triggers and made accepted answers worse (15.10).
      5. The design work goes to what helps the agent decide: the report. OURS' side of the context and a way out for the model to abstain were tried and removed (15.10).

      Java. R1b's Java answers are 54 of 54 correct (Wilson 95% CI 93.4-100). That is a property of the model on these cases, not of the check, and most of them come from one repository (spring-boot). It is a candidate for a per-language, measured policy, which would first need validation on Java cases from other repositories.

      15.7 Other resolvers, compared

      From a survey of 17 tools and papers (prompts read from source where the licence allowed). GitButler's AI resolve is described here from a secondhand account; its code was not read.

      • GitButler sends one structured JSON request per commit with every hunk; we send one call per region. Our own R0 (whole-file JSON) against R1 (per region) on the same 300 cases: 53.8% against 41.8% strict error, and R0 -> R1b significantly more correct (McNemar p = 0.018). Per region stays.
      • GitButler writes each resolution into both sides and lets git merge again, which keeps the non-conflicting changes; our llmAssemble rebuilds the file from the base, the clean regions and the resolved ones. The two are equivalent when the regions are those of git's own merge; to be adopted as an assertion test of llmAssemble (not done yet): writing the result into both sides and merging must give the assembled file.
      • GitButler lands directly with no semantic check; ours proposes, with a report. That, and the measured finding that a line-level check cannot gate, is where gitai differs.
      • Credit: the patch-application structure of R1 is synthmerge's (Andrea Arcangeli; GPL, structure only, our own wording and example). The evidence on scope (ConflictAgent, Shen 2026, MIT; Rover, CC BY) and on intent (GitWand, Copilot, MIT) informed the symmetric context that was tried and removed (15.10). The design, the composition report and its measurement are gitai's.

      15.8 The resolution goes out of the lane

      Today a land with resolve_conflicts calls the resolver inside the branch's lane, so the branch is held for the resolver's whole call: seconds, minutes for a file with 20 regions at ~13 s each, and ~97 ms more to keep the proposals. A first version of this delta bound the resolution to the tip read before the lane; on a busy branch (the 50-agent race) the tip moves every few hundred milliseconds while a resolution takes seconds, so that version would fail exactly under load. The resolution is bound to the files instead:

      • What a file's verdict depends on. The resolver answers a file from its sides: its key is (path, base OID, ours OID, theirs OID, the three modes). Two inputs come from elsewhere, and the design states them:

        • The classification (15.2) also reads the tip: the linguist-generated attribute from the tip's .gitattributes, and the paths a side comes from in the tip's tree (renames). The lane therefore runs the classification again on every file it reuses: local git work, bounded like the package's, never the resolver. A file whose classification at the lane's tip differs from its verdict's counts as changed.
        • A request carries the files that still need a verdict, and which files travel together depends on the round: after a tip moved, only the files whose sides changed are sent, alone. A resolver can therefore not rely on seeing the other conflicting files of the land, and a verdict must depend on its file only (its path, its sides and modes, the change's message, the configuration). A third-party resolver (the command or the HTTP service) must respect that contract; one that reads the tip, another file or the history of the repository through its own means would have its answer reused where it no longer holds. The risk is bounded: an answer of a model never lands unverified (the final ruling, 14 and 15.9), and the operator's rules are the resolver's own.
        • The request no longer carries the tip, its merge base or its commit message: the fields tip, merge_base and messages.tip are always empty, kept in the protocol for compatibility (a test asserts it). A verdict reused at another tip is then correct by construction: no resolver, the operator's command strategy included, can read the tip. RESOLVE.md states that an answer must depend on the file only.
      • Pre-resolve before the lane. UserLand reads the tip T as the lane does and, in a quarantine it discards, merges T and the change. Skipped (the lane answers as today) when the branch does not exist, the change already landed, or the merge is clean. Otherwise it runs the per-file flow of 15.2 for the conflicting files that have no verdict yet: the policy first, the classification, one resolver request for them, the validation; each verdict is kept by its key; then decideLand over all of them.

      • Not landable: no lane at all. The land fails anyway, so the proposals are kept (and pruned,

        1. and the package is returned without entering the lane. Built per file, the package of an older tip stays true per file; the agent's re-land is bound to the tip of its resolution, as today (binding agents' resolutions to the files too is a possible later extension, not now).
      • Landable: applied in the lane, file by file. The lane merges its own tip and the change. When every conflicting file has a landable verdict with its key, and the lane's classification of the file agrees, the lane writes the verdicts into the merge commit and runs no command (15.9); the lane does not call the resolver for it, except for the rules it resolves inline (below). In a batch, a member's verdicts hold whenever their keys match; without verify.enabled the members chain, one on the other's tip. With it a member lands only on a tree that passed, so a member whose tree is not the verified one (the batch changed the tip under it) goes back out to be verified, and lands alone.

      • Needs resolution. When a conflicting file has no verdict with its key (it is new, or a side changed), the lane tries the rules inline (below) and returns the attempt with the files they did not resolve. UserLand pre-resolves only them against the new tip, out of the lane, keeps every other verdict, and enters the lane again. A tip that moves on unrelated files costs no resolver call at all.

      • Progress, and what it took. A round is lost when the lane found a file whose sides changed, that is when another land on that file landed. It was written here that the lands that contend for a file all land, one per round, and that was true of the system and false of a land. Measured (graft-bench, 50 agents x 5 changes on one branch, conflict rate 0.2, seed 1, 50 lands on one file, the commit before the fix): every landing invalidates the verdicts of all the others, so after each one they all resolved again: 749 resolver calls and 713 hand-backs from the lane for 50 resolutions. The resolve slots were not the cost (2.3 s of wait in all); the lane was: 957 members in a run in which it was busy for 33 s of 34, 0.41 s of wait for each, and so the clean lands of the branch were slow too. 6 of the 50 lands lost 32 rounds in a row and ended with "the files changed again before they could be resolved" (244 of 250 landed, 364 to 430 lands/min, p99 23 to 27 s, against 250 of 250, about 1,700 to 1,900 and 2 s of v0.2.0, which resolved inside the lane). Three things now give every land its turn:

        1. The rules are resolved in the lane, as in v0.2.0 (inline, inline_timeout): a land enters the lane first; what a rule can resolve is resolved there, in its batch, by a request with rules_only that graft-resolve answers with its rules and never with a model (the answer echoes rules_only; a resolver that does not is an older one, and the lane never asks again for the life of the process). The split follows what the resolution is, not how long it took: a model, a verification, a rule that does not end within inline_timeout (default 1 s) or one that cannot resolve a file go out of the lane, to the path below. With the rules in the lane the bench has no hand-back, no ticket and one resolver call per resolution, and its numbers are v0.2.0's. The lane is bounded: it never waits for a slot (the calls have a pool of their own; with it taken the member goes out of the lane), and the time it spends in inline calls is bounded once per batch attempt (landInlineBudget: inline_timeout in all, not per member): all the time of every call is charged to it, one that succeeds as well as one that fails, so a rule of 0.9 s does not stall a batch of N members for N x 0.9 s; once it is spent the other members go out of the lane, where the queue guarantees their progress, and fast rules (milliseconds) do not notice. A land that the rules cannot resolve (the resolver refuses, or a file may not be sent) is resolved out of the lane for the proposals and ends there, without a ticket, and gives back the one it took if it had queued for it: without a verification a model's resolution never lands, so a ticket would protect nothing. The request announces no capability but origin, so that a graft-resolve that gates the models on the capabilities (any since they were added) refuses to call one even if it ignored rules_only; v0.2.0 has no gating and may.
        2. A queue for the lands that lose a round out of the lane. A land that lost a round takes a ticket, first in first out, on its files (per file; per branch when verify.enabled, since the tree that is verified is the branch's), keeps it until it lands, its deadline ends or it fails, and is the only land of the branch that resolves them again; the others wait without calling the resolver, and the lane hands back any land with a verdict made out of the lane on a ticketed file, so that the holder's verdict is the one that lands. The holder that lost a round out of the lane holds its files with priority until it is done: the lane also hands back the inline resolutions of the others on them, which would change the files under its verdict at every round (a rule slower than inline_timeout on a hot file would never land); they wait behind it and go back to the lane when it is done. Otherwise a ticket never stops the rules in the lane: the inline attempt comes before the check of the tickets, and the land that waited for its turn goes back to the lane, not to a new pre-resolution; a lane round does not count toward resolve_rounds. The queue alone, measured on the same bench before the rules were resolved in the lane (the first form of this fix, not the code of this release): 92 resolver calls, 8 hand-backs, 250 of 250. A land that waits past its deadline ends with the package ("the time ran out waiting for the other lands on the same files").
        3. resolve_rounds (default 32) is only a safety cap: the rounds of a land are bounded by its deadline, and the queue makes it lose at most the rounds that a land which changes its files without a conflict makes it lose. Measured on the code of this release (commit e3b60e3, the bench above: 50 agents x 5 changes on one branch, conflict rate 0.2, seed 1, one fresh server per run; each run started once the 1-minute load was below 2, new and v0.2.0 alternating; the load at the start is mostly the previous run's own average, and the runs before them were made on a machine that other work had just left, so the spread is wide: the v0.2.0 runs alone range from 623 to 1,940 lands/min). Lands per minute, p99 of the landed changes:
        • Rules only. 250 of 250 in both: 1,505 and 1,867 lands/min (p99 2.6 and 2.1 s), against 1,808 and 1,860 (p99 2.0 s) of the v0.2.0 run next to each. The first run of the new code started at a load of 1.99. No hand-back, no resolver call out of the lane, no ticket, one round per land: v0.2.0's numbers.
        • Mixed, a land in ten that only a model can resolve. The model is simulated: a wrapper between the server and graft-resolve (wrap.py) refuses the request for the rules only on those files (a hash of the file's side picks 5 of the 50 conflicts) and answers the ordinary request after 5 s, marked as a model's. 245 of 250 land, 836 and 860 lands/min, p99 2.9 s. The 5 that do not land are those, each ended with the package and its proposal, as a model's resolution without a verification never lands; the union lands are not held back (no ticket wait). The run's length is the 5 calls through 2 resolve slots, not the lane. The same wrapper in front of v0.2.0 gives 250 of 250, 424 and 429 lands/min, p99 16.7 and 18.3 s, but v0.2.0 lands the model's answers, and each call blocks its lane: the semantics differ, and it is a reference, not a like-for-like baseline.
        • One inline failure. The first request for the rules only sleeps 3 s and is cut at inline_timeout (1 s): 5 to 8 members are handed back and resolved out of the lane, no ticket is waited for, no convoy: 250 of 250. The wrapper costs time by itself, so the control is the wrapper passing everything through: 1,261 to 1,370 lands/min (p99 2.8 to 2.9 s, three runs) against 1,117 to 1,227 (p99 3.6 to 4.0 s, five runs) with the failure: about 10% and 1 s of p99, the second that the budget allows. (The runs without the wrapper are the ones above, 1,505 to 1,867: the wrapper alone costs about 20%.) v0.2.0 has no request for the rules only, so it has no partner here.
        • A verification of 1 s (verify.command = sleep 1): 250 of 250, 129 and 130 lands/min, p99 73.0 and 74.7 s, 41 hand-backs, 50 ticket waits (45 s on average), 92 resolver calls, 333 rounds for 250 lands; against 113 to 124 lands/min and p99 74 to 94 s of the queue before these fixes, no regression. v0.2.0 has no verification, so there is no baseline: its runs with the same setting ignored it (1,782 and 1,870 lands/min). Repeated on the code after the second review (commit f603f03: the model land gives its ticket back, the holder has priority, all the inline time is charged to the batch), in a window in which the machine was busier (the load at the end of the runs went from 2.5 to 7.5): rules only 775 and 1,364 lands/min (p99 7.3 and 3.2 s) against 1,094 and 897 (p99 4.2 and 5.5 s) of the v0.2.0 run next to each; the wrapper alone 842 and 742, with one injected failure 631 and 646 (p99 7.4 s). All 250 of 250. The structure is the same as above and does not depend on the machine: no hand-back, no resolver call out of the lane and no ticket wait in the runs with the rules only, 7 and 8 members handed back and no ticket wait with the failure. The rates of this window are lower for both sides and are not comparable with the first. The start loads of the last six runs (the wrapper's control and the failure) were not logged; the cooldown that holds each start until the load is below 2 was in force, and the whole batch ran in a few minutes. The limits, stated. The queue is in memory and per process, like the lanes: against another node of the same branch (authoritative storage) the lands go on in rounds, without the guarantee. A land that holds a file with priority makes the inline resolutions of the others on it wait for its resolution, for as long as a slow rule takes and at most up to the holder's own deadline (the resolver's timeout): the others then wait behind it, or end with the package at their own deadline. A holder whose set of files grows gives its tickets back and takes the larger set at the tail: it loses its place in the queues, and its priority until it is back in the lane. Two gaps are known: a resolver that crashes (a non-zero exit with no refusal in its answer) is taken as the rules not being able to resolve, so the land is resolved out of the lane for the proposals; and a v0.2.0 graft-resolve, which has no gating and ignores rules_only, that is cut at the timeout on a model's conflict is never recognised as an older one (only an answer that does not echo rules_only is), so it pays the lane's budget at every batch; with it set inline_timeout = "0s". A holder's verdict is invalidated by a land that changes its file cleanly and lands first (each such land lands once, and the holder resolves again, which is cheap for a fast rule, and costs the time of a slow one each time); with a verification every tree is verified out of the lane, and any land that moves the branch while a holder verifies, a clean one included, makes the verified tree stale: on the bench with a verification of 1 s, 50 resolutions cost 130 verifications and 120 to 135 s (113 to 124 lands/min, p99 74 to 94 s), all 250 landed, bounded by the deadline; before the queue the same bench cost 802 verifications for 34 resolutions, 16 lands failed ("could not verify before the branch moved", "the files changed again"), 158 lands/min, p99 53 s. Verifying a batch of resolutions on one combined tree would remove that cost; it is not done.
      • Rules alike. The operator rules' verdicts (trivial, union, rerere, command) are kept and reused by the same key, so the policy is the same for every strategy.

      • Slots: two pools. check_concurrency bounds the checks of agents' resolutions and resolve_concurrency (new, default 2) the resolutions. A check is short and gates an agent's re-land; a resolution with a model takes seconds to minutes. One shared pool would let a few long resolutions starve the checks, and fairness inside one pool needs a scheduler for no gain over two plain semaphores; the memory bound of the process is the sum of the two pools' requests. The lane's inline calls (the rules in the lane, below) have a third pool, of at least 8 slots, which the lane never waits for. Each request waits for its slot within the resolver's timeout and fails closed when it expires.

      • Nothing is verified in the lane. Every verification, of a rule's resolution, a model's or an agent's, runs out of the lane, on one mechanism (15.9); the lane compares tree object IDs.

      • Measured (TestLandResolveBench, GRAFT_LAND_BENCH=1). Six lands that the server resolves, all on the same file, and six plain lands that conflict with nothing, at once through the lanes of a standalone server, with a resolver that takes 500 ms; the same bench on the tree before and after, three runs each:

        before (resolver in the lane) after (out of the lane)
        lane held 3.48-3.54 s 0.72-0.93 s
        plain lands, p50 3.47-3.53 s 151-205 ms
        resolve lands, p95 3.44-3.58 s 3.21-3.22 s
        resolver calls 6 11

        The plain lands no longer wait for the resolutions, and the lane is held for git work only. The cost is in resolver calls: every land on the same file loses a round to each land that lands before it, and resolves again. That is the worst case by construction (one file, every land on it); lands on different files, or a tip that moves on other files, cost no extra call.

      • Tests.

        • The resolver is never called with the lane held: a resolver that blocks while another land of the branch proceeds.
        • A not-landable answer never enters the lane.
        • A tip that moves on an unrelated file reuses every verdict, with no second resolver call.
        • A tip that changes one conflicting file re-resolves only that file.
        • Files that keep changing fail at the deadline, or after resolve_rounds, with the reason naming them.
        • A batch member reuses its verdicts.
        • A high-churn test: many concurrent lands on the branch, the resolve lands among them land eventually, with no starvation; and with a short deadline, every land ends, landed or with its reason, and none hangs.
        • A proposal kept while a batch holds the lane has the published tip as parent (review of 6090d5b, item 4): proposals are written before the lane only.
        • The two pools: a held resolve slot does not delay a check.

      15.9 A model's resolution lands through the verification

      The final ruling of 14: an AI resolution lands only through the operator's verification. The verification runs the operator's tests, which take minutes, so it cannot run in the lane (15.8), and the tree it verifies depends on the tip.

      • One mechanism, for every verification. The verification that was left in the lane for the rules' resolutions and for agents' went out of it too: whatever made the resolution, verify.command runs on the exact tree, out of the lane, with a slot of the process's pool, and the lane compares tree object IDs. Why: in the lane it was bounded by nothing but one per branch, so the number of commands running in the process was unbounded; now every one goes through verify_concurrency and verify_per_repository, and the pool exists whenever verify.enabled is set, with or without an AI resolver. The costs, stated: a rules' land whose tip moved after its tree was verified is verified again (it is a model's land in this respect: liveness is the deadline's, and at the deadline the rules' files are NOT_ATTEMPTED and an agent's resolution gets "could not verify before the deadline of " with its package); an agent's resolution is verified in its precheck, at the only tip the lane accepts it at. The per-process worst case of a verification, so that the operator can size it: inodes and disk at most verify_concurrency x (verify.max_tree_entries inodes, verify.max_tree_bytes of files), and that is the checkout alone: what the command itself writes (a build directory, a cache) is the operator's to bound, and counts against the same disk; heap at most verify_concurrency x (the paths kept, 128 MiB by the charge of the materializer; the entries and the target of no link); and a TMPDIR on tmpfs turns that disk into memory: point it at a disk, or lower the limits.
      • Verification out of the lane, on the exact tree. For an answer that holds a model's resolution (or any answer, with verify.enabled set), and when nothing else keeps the land from landing (every file resolved or proposed), the pre-resolution builds the tree that would land (landResolvedTree: the merged tree of the tip and the change, the answer's files written in) and runs verify.command on it, in the quarantine, with a slot of the verify pool (verify_concurrency, default 2, a pool of its own: graft_resolver_verify_slot_wait_seconds). The tree object ID that passed is kept.
      • In the lane: tree equality. The lane merges its own tip, applies the verdicts, builds the tree it would write, and asks decideLand about it: a model's resolution lands only if that tree is a tree that passed. Tree object IDs are compared, not tips: a tip that moved to the same tree keeps the verification; a tip that moved to another tree does not. The lane runs no verification of a model's resolution, and writes the merge commit with Graft-Checked-By: verify:.
      • Tip moved: out again, no resolver call. When the tree is not a verified one, the lane does not land. The land goes back out, rebuilds the tree from the same verdicts (the resolver is called only for files whose sides changed, as in 15.8), verifies the new tree, and enters the lane again. Each round costs one verification.
      • Liveness, stated plainly: the progress argument of 15.8 does not carry over. In 15.8 a round is lost only when another land changed the file, so the lands that contend for a file all land. Here a round is lost when ANY land moved the branch, because any land changes the tree that was verified, whatever file it touched. On a busy branch a verified AI land therefore mostly ends as proposals within one verification (the deadline holds a verification's time and little more), and that is the designed, safe outcome: the agent gets the proposal and decides. A verified AI land is opportunistic. Two bounds keep it from costing more than that: a per-repository cap on the verifications at once (verify_per_repository, default 1, inside verify_concurrency), so that one busy repository cannot take the whole pool, and the deadline of the land. A land that waits for its repository's turn holds nothing of the pool, and a wait that outlasts the deadline ends in proposals like the rest.
      • Deadline: a proposal, not a failure. The deadline of the land is the resolver's timeout plus verify.timeout when verify.enabled is set. A verified AI land is opportunistic: when the deadline ends before a tree could be verified and landed, the land does not fail for the agent. The model's files become PROPOSED with the reason "could not verify before the branch moved again" (or "before the deadline of " when the verification itself was cut), the proposals are kept (14), and the package goes to the agent. This final pass verifies nothing and calls no resolver. It is the end of every land that had a conflict and could not go on, not only of a model's: the deadline wherever it fell (the listing, the tree, the resolution, the verification), the cap on the rounds, and a land that the lane or its batch kept handing back (a model's resolution whose batch ran out of rebuilds is handed back as a land that needs its new tree verified, not failed with "kept moving"). The files that have a verdict are reported with it (a model's PROPOSED, a rule's RESOLVED, or NOT_ATTEMPTED with the reason when verify.enabled is set and its tree could not be verified; their proposals kept); the files that changed under the land and have none are NOT_ATTEMPTED, with the reason. With nothing known yet it is the merge conflict with its package and no proposals, with the reason. A land that had a conflict never returns a bare deadline to the agent: only a caller that gives up gets its cancellation.
      • A verification that fails (non-zero exit, or its own timeout) is a verdict, not a deadline: the files stay PROPOSED and carry "verification failed: exit status N: <the first 4 KiB of the output>" (the package cuts a reason to 512 bytes). Nothing is verified for a land that cannot land anyway (a file refused, a file left to the agent).
      • What is a model's resolution: the origin of each file, fail-safe. Each resolved file carries origin, rule or model (the older proposal: true means model and is still read). A file is a rule's only when it says so; a file without an origin, or with an unknown one, is a model's. It lands without verification only if it is marked rule; otherwise decideLand's one rule applies. The name of the resolver decides nothing (a joined list, a list cut to ",+N", another vendor's name). The server normalizes the origin when the answer arrives, so the answers kept by their keys carry it and the rounds that reuse them never depend on a name. Third-party resolvers that do not set origin no longer land without the operator's verification: that is the change.
      • What the verification sees: the tree, byte for byte. git-archive(1) honours the attributes of the tree itself: export-ignore hides a file from the command, and a .gitattributes written by an agent or by a model could therefore decide what the operator's tests see. The checkout is written from git-ls-tree(1) -r and git-cat-file(1) --batch, which apply no attribute (no export-*, eol, filter or smudge), with modes, symbolic links as links and gitlinks as empty directories, and is read back and compared with the listing, the content included (every file and link is hashed as a git blob and must be the OID of the tree); a difference fails the checkout closed. The same function serves the verification of rules and of agents' resolutions.
      • What the checkout refuses, and what it costs. The operator's command may run git in the checkout, so a path component that names .git on any file system is refused, by the one rule the resolver already uses for the special files (resolver.IsGitDirName: case-insensitive, the code points HFS+ ignores, NTFS "git~1" and trailing spaces and periods): a .git of the tree would be the repository of the command, and its configuration and hooks would run. Also refused: an entry under a symbolic link of the tree, also when a case-insensitive file system would see the same name (a link "A" and an entry "a/x"), a parent that is a link or a file, a path over 4096 bytes. Each directory is made one component at a time and looked at with Lstat (never MkdirAll, which follows links), a file is created with O_EXCL and O_NOFOLLOW. Resources, all failing closed with the reason: a git tree can name one subtree many times, so a push of a few KB expands to as many paths as it has names, each as deep as the subtree. verify.max_tree_entries (default 200000) counts the entries AND the directories that the checkout makes (both are inodes); verify.max_tree_depth (default 64) the components of a path; verify.max_tree_bytes (default 2 GiB) the bytes of the files and of the targets of links; and a fixed 128 MiB the memory of the paths, each entry and each directory charged twice (the sets that look for links and for folded names). The limits are checked entry by entry, from the listing (ls-tree -l), BEFORE that entry's content is read, not from a first pass over the whole listing: the listing and the blobs are streamed through two commands, and what stays in memory is exactly what the materializer charges against the fixed 128 MiB budget (landMaxPathBytes): each entry its path plus 256 bytes, each distinct prefix twice its length plus 64 (the folded key and the spelling), each directory made its path plus 64, each link its path plus 64, and nothing of the target of a link or the content of a file (the read-back hashes what is on disk). A link target is at most 4095 bytes, refused up front. An entry that would go over a limit stops the checkout with the earlier ones already written. The context is checked at every entry. A name over 255 bytes, a path that with the directory reaches PATH_MAX, a NUL in the target of a link, and a system call that fails (a name that exists already, no room) are refused with the reason, not errors of the server. The checkout runs inside verify.timeout, as the command does. A refusal is no error of the server: nothing is verified, the files are PROPOSED (a model's) or REJECTED (rules'), and the reason says which limit or path. Not built: NTFS trailing-dot and space folding in the check of colliding names (the .git check has it); Gitaly does not run on Windows. Under MVCC (a snapshot pinned for the RPC) a lost race ends with the retryable FailedPrecondition "kept moving" even for a model's resolution: that call carries no package; the client's retry gets one.
      • Tests. A model's resolution that passes lands, signed, with the command run once (so not in the lane); one that fails gives a proposal and the output; without a verify.command it is a proposal; a tip that moved gets its new tree verified and lands, with no resolver call; a tip that moved to the same tree keeps the verification; churn past the deadline gives a proposal, and so does the cap on the rounds; the lane is free while a verification runs (a plain land passes through it); the lane never verifies; the verify pool bounds the verifications at once, those of rules and of agents' resolutions as well; and the table test of decideLand, whose one rule is compared by tree object ID.

      15.10 The writer: the prompt of record, the elements kept, the elements removed

      What the writer is. The llm strategy of tools/graft-agents/resolve. For each region that git's own merge calls a conflict (git merge-file --diff3 on the three versions; our own line diff only when git cannot run), it makes one call. The request holds THEIRS' change as a PATCH (a unified diff from the base, three lines of context), OURS' version of the region as the CODE with the same context lines, THEIRS' commit message (at most 4 KiB), the other conflicting regions of the file for reference, one worked example of the task, and the directives last. The answer is plain text, the patched code between two lines that carry a random token. The structure is synthmerge's patch application (15.7); the wording is ours.

      An answer is checked before anything else: it starts and ends with the context lines of the CODE copied unchanged, ends with a newline, is not itself a patch, holds no conflict marker, fits its size. A region whose answer breaks one of those rules is asked ONCE more with the rule quoted in its exact words and the previous answer; the assembled file is then checked for syntax (tree-sitter), and the first syntax error, mapped to the region it is in, is quoted back in the same one round. A region is repaired at most once, only if the request fits max_prompt_bytes, and only when its answer is BROKEN: an answer that the rules and the syntax accept is never asked for again. The composition report (15.3) goes with the proposal, whatever it finds.

      How each element got there. The first design had three more elements (OURS' side of the context, a way out for the model to say that the changes contradict, and a repair of accepted answers for composition findings), each behind an option, measured as one step against the baseline R1b and then by ablation, Luna (openai/gpt-6-luna, effort low) on the same 300 cases, 2026-10-19/20:

      arm accepted correct precision [95% CI] rejected calls (repairs) $ booked
      R1b, the baseline (one call per region, no repair, no report) 244 143 58.6% [52.3-64.6] 27 403 (0) 0.107
      design: all elements on 261 151 57.9% [51.8-63.7] 6 455 (50) 0.169
      design without git's regions 252 146 57.9% [51.8-63.9] 8 422 (40) 0.153
      design without the abstain block 264 156 59.1% [53.1-64.9] 15 444 (46) 0.162
      design without any repair 237 143 60.3% [54.0-66.4] 31 407 (0) 0.145
      design without OURS' side of the context 262 154 58.8% [52.7-64.6] 9 464 (58) 0.151
      composite: git's regions + repair + report; no abstain, no OURS context 270 156 57.8% [51.8-63.5] 9 467 (56) 0.152
      composite-rc: the same, repair only for a BROKEN answer (the prompt of record) 270 158 58.5% [52.6-64.2] 9 437 (26) 0.137

      The rule was fixed before the ablations: keep an element only if removing it lowers the correct answers or the precision with p < 0.1, or lowers the coverage at an equal precision; otherwise remove it. The report stays whatever the data say, since it removes nothing. Applied:

      • git's regions: kept. Removing them lowers the accepted answers (252 against 261) at the same precision, and they are what the server's merge calls a conflict. The first run on our own line diff disagreed with git's merge on 18 of the 244 accepted cases (13 had no conflict for us and one for git, 7 of those 13 wrong).
      • the repair round: kept, for broken answers. Without any repair 24 answers fewer are accepted (237 against
        1. at an equal precision: the repair rescued 33 answers that were rejected otherwise, 21 of them correct (64%), and 9 answers were accepted only without it (5 correct). On the 228 answers accepted both ways, removing the repair turns 13 wrong answers into correct ones against 5 the other way (exact McNemar p = 0.096): the repairs of accepted answers (for the composition findings) made answers worse. So the repair is limited to a broken answer, and that arm (composite-rc) has the most correct answers (158), the best precision of the three candidates and the fewest calls.
      • OURS' side of the context and the abstain block: removed. Removing either raises or keeps the correct answers (154 and 156 against 151) and lowers nothing. The code, the option, the protocol field that carried OURS' commit message to the resolver and the server code that read it are gone; so is the abstain block and the outcome it produced.
      • the report: kept, as a report. Among the accepted answers of the prompt of record, the 23 for which it finds a drop or a duplicate are right in 10 cases (43%); the other 247 in 148 (60%). It separates, not enough to gate (15.6), and it is information for the agent.

      The switches of the ablations are removed with the elements: the strategy has no option for git's regions, the repair or the report, and a configuration that names symmetric_context, abstain, repair, repair_composition, git_regions or max_context_bytes is refused. Against the baseline, no variant moved the precision (57.8 to 60.3% against 58.6%, all intervals overlapping); what the design bought is coverage, 270 answers accepted against 244, through the repair.

      Parity. The composition check is a port of the frozen prototype of 15.6, and CompositionVerdict agrees with it on 243 of the 244 accepted answers of R1b (99.6%); with the regions of our own line diff it was 226 of 230.

      15.11 The decision step: measured, not adopted

      The idea (Federico's): a model that WRITES a resolution can write a line nobody had, or drift whitespace; a model that CHOOSES cannot. So let the writer's work be preceded by a choice per region, and let the server assemble the file from the sides whenever every region is a pick. It was built as a measurement outside the product (the scripts are in c1-work/scripts/, the data in c1-work/run/chooser*/) and measured on our corpus and on conflicts of merged agent PRs. It is not adopted: it selects well exactly where a constant policy already works, and nowhere else. What follows is the design that was evaluated, its gate, which was fixed before it was measured, and the results.

      • The pipeline that was evaluated. Git computes the facts (the regions, the sides, the base). A decision model, Jev (typesafe/jev-1.14 through OpenRouter, TypeSafe's calibrated decision model; it cannot write code), DECIDES per region among OURS, THEIRS, OURS+THEIRS, THEIRS+OURS, NEEDS-MERGE, and returns a probability for every option. A language model WRITES only where Jev says NEEDS-MERGE (2-3 screened writers produce candidates; Jev then chooses among the distinct candidates of the region plus "none: send to the agent"). Deterministic checks and the verification decide landing. The single landing rule of decideLand is unchanged: a resolution lands only through the verification on the exact tree, the origin stays "model" (15.9), and nothing else lands.
      • As a first step of the strategy. If every region is a pick, the server assembles the file deterministically from the sides and the clean segments of the merge: the answer holds no line that no side has and no whitespace of a model's. If any region needs a merge, the file goes to today's writer path (15.10) and becomes a proposal as now.
      • How Jev is called (probed 2026-10-19, two calls, $0.000022 each): POST https://openrouter.ai/api/v1/systemone with the OpenRouter key, body {model, state, questions:{id:{type:"choice", instructions, criteria:{option: description}}}}; the answer per question is {choice, probabilities:{option:p}, confidence} and usage.cost is in the response. One request per file: the state (path, per region base/ours/theirs lines with context, both commit messages) is ingested once and each region is one question. At most 255 options, 32 MiB; default context 64k per request and 32k per question (a header raises both to 262k, slower and less accurate, and is not used until a file needs it). The model is priced per input token only ($0.042 per million). Jev leans toward the FIRST option (TypeSafe's own jaggedness note, confirmed by the probe: the same question with the options reversed moved OURS+THEIRS from 0.69 to 0.54 and NEEDS-MERGE from 0.15 to 0.26), so every decision is asked in two orders and the probabilities are averaged; a decision whose argmax differs between the two orders is NEEDS-MERGE.
      • The gate was fixed before it was measured. "Jev's averaged probability for the picked option is at least tau in every region of the file, no region is NEEDS-MERGE, plus the verification." (A cross-family agreement gate, "every region picked identically by all N models", is measured beside it.) tau, and N and the model set for the agreement gate, are tuned on half of a family-stratified sample of the corpus OUTSIDE the 300 (the corpus is 48% one repository, see below, and a plain sample would be tuned on that repository; spring-boot is capped in the sample) and then FROZEN; the other half and the 300 are the held-out confirmation, reported separately and pooled. The target of at most 5% wrong among what lands is reported as met only if the Wilson 95% upper bound of the wrong rate is at most 5%, under BOTH judges: the strict one (the human's file after normalization) and judge A (the reference-based LLM judge adapted from ConflictAgent's rubric, MIT, measured offline; it tolerates whitespace, order and equivalent phrasing). Under neither is the point estimate enough. That bound needs about 75 landed files with none wrong, 150 with one, 300 with three; the ceiling below says what a sample can reach. Everything is judged against the constant policy "always THEIRS" and reported with and without spring-boot.
      • Concatenations were to be measured, not assumed. OURS+THEIRS and THEIRS+OURS stay in the vocabulary only if the data shows they are safe: their precision is reported separately from the single-side picks, and if it is clearly worse they are dropped from the vocabulary (those regions become NEEDS-MERGE), not special-cased.
      • The report applies to assembled files too. A pick of one side can "drop the other side's change at base a-b" (15.3): the composition report of 15.3 is computed on the assembled file and goes to the agent with the proposal. It is honest information, not a gate.
      • Never sent stays never sent. Lockfiles, generated, minified, binary and large files keep their paths (3.4); the chooser only replaces the writer's first step for files the llm strategy already receives.

      Free numbers that bound it (ceiling.py, ceiling2.py; no model, no network; git's regions; the human's resolution matched region by region against the four picks after the judge's normalization, groups of regions with no clean line between them matched together). The mined corpus has 2,935 content files (plus 237 modify-delete / add-add files, not region-based):

      files every region a side-pick of which concatenation only "always THEIRS" whole file
      corpus 2935 1614 = 55.0% [53.2-56.8] 179 1357 = 46.2%
      corpus without spring-boot 1518 316 = 20.8% 75 = 4.9%
      the 300 288 118 = 41.0% [35.4-46.7] 72 = 25.0%
      outside the 300 2647 1496 = 56.5% [54.6-58.4] 1285 = 48.5%
      go / python / rust-java / ts-js (corpus) 447 / 92 / 1440 / 956 21.9% / 37.0% / 90.4% / 18.8% 4.7% / 9.8% / 89.0% / 4.7%
      • The ceiling is 41% of the files of the 300 and 21% outside spring-boot. spring-boot is 1,417 of the 2,935 files, and there the human takes THEIRS in 90% of the files (why was not investigated); a constant policy "always THEIRS" is already right for 89% of the java files of the corpus and for almost nothing elsewhere (4.7% in go and ts-js). A chooser must therefore be measured outside spring-boot and against this constant policy, or its number is the repository's, not the model's.
      • In go and ts-js a region is a pure side-pick in about 30% of the cases (go 30% of 628 regions, ts-js 32% of 1,080); in 70% the human wrote something that is none of the four, and the file goes to the writer.
      • The human's choice is constant within a file in 1,599 of the 1,614 all-pick files (99%): one pick for the whole file. A per-file pick would lose almost nothing against a per-region pick on the corpus; the per-region vocabulary is kept because the model sees regions, and the free number says it does not need more.
      • Concatenations: 263 of the 1,933 side-pick regions of the corpus (14%) are OURS+THEIRS or THEIRS+OURS, but outside java they are the common case: go 85 of 187 (45%), ts-js 141 of 350 (40%), python 19 of 62 (31%), java 18 of 1,334 (1%). 179 of the 1,614 all-pick files are all-pick only through one. Dropping the concatenations would cost most of the ceiling in go and ts-js, which is why their precision is measured, not assumed.
      • Files whose human resolution changed lines outside the conflicts ("outside": 365 of the 2,935, 41 of the 288 of the 300) cannot be assembled from the sides by any chooser: they are wrong for it by construction.
      • By path: test files 71% all-pick, source 49%; java 94%, ts 20%, go 19%, js 8%, py 36%. The corpus has no lockfile and two generated-path files (those rules apply first).

      What was run. Jev alone on a family-stratified sample of 244 files outside the 300 (the tuning half, two option orders per file), then all 19 decision models that OpenRouter lists on a 200-file subset, then the agreement of the best ones; the confirmation half and the 300 were not run, because the tuning half produced no candidate gate (the protocol freezes tau before the confirmation halves, and no tau had a Wilson upper bound of at most 5%). Then the same four best models on the replayed conflicts of merged agent PRs (below).

      Results (2026-10-19; the chooser is not a path to 5% wrong outside java). Everything below is offline of the product: ceiling.py, ceiling2.py, chooser.py, agree_dm.py, dm_probe.py, dm_screen.sh (kept so the measurement can be rerun on agent-conflict data; they live in c1-work/scripts/, outside the repository), the data in c1-work/run/chooser/. Tuning half = 244 files outside the 300 (family-stratified, spring-boot capped at 30; python 13, go 100, ts-js 100, other java 1); tune200 = 200 of them.

      • Jev 1.14, alone (two orders averaged, same argmax required; $0.0445 for the 244 files). Pick accuracy against the human's pick 56.4% on 282 matched regions, ECE 0.149; calibration holds only at the top (p 0.8-0.9: accuracy 0.88, n = 24). The averaged probability never exceeds ~0.85, so tau 0.9 lands nothing. By tau: 0.5 lands 72 of 244 files at 48.6% precision; 0.7 lands 33 at 78.8%; 0.8 lands 21 at 95.2% (Wilson upper wrong 22.7%). Without spring-boot (214 files): tau 0.5 lands 45 at 20.0%, tau 0.7 lands 8 at 12.5%, tau 0.8 lands 1. All the precision is spring-boot java: java tau 0.7 lands 25 of 31 files, 25 correct, against "always THEIRS" at 27 of 31 correct with full coverage. go 13%, ts-js 26%, python 0% at tau 0.5.
      • All 19 decision models on OpenRouter (every model with output modality decisions except aliases, :free variants and the behaviour-scoring span-01; one probe call each, then tune200 x 2 orders; $0.864 booked in all, no model above $0.12). All 19 accept the request shape of the Jev probe (scaledown wants the state as a string; context limits are respected by a size check: kev-4b 8k, clef-flash 24k, d1-ultra / microsoft / tev1 32k, oversized files count as not landed). Accuracy is the region pick against the human's (NEEDS-MERGE counts as a class; about half of the regions are "other", so 50% is the floor): d1-ultra 76.2% (ECE 0.115), pplx-decider-v1.2 74.0% (ECE 0.058, the best calibrated), clef 73.6% (no order bias at all: shift 0.004), mercury-decide 69.8%, solar-decide and -flash 63.4%, scaledown 62.1%, kev-4b 61.4%, pplx-decider-flash 59.6%, jev-1.14 58.7%, luna-decisions 56.6%, d1 56.2%, tev1 53.2%, jev-1.13 50.2%, span-02 48.5%, microsoft-decision-1 48.1%, drex-1.5 45.1%, clef-flash 42.1%, clef-omni 36.2%, drex-2.0 35.7%. The order effect of the first option (mean |p_fwd - p_rev| of the chosen option) is 0.00-0.26, largest for span-02 (0.25), drex-1.5 (0.26), solar-decide (0.21) and tev1 (0.20); latency p50 0.3-1.3 s, p95 up to 3.9 s (d1-ultra) and 11.6 s (solar-decide). Coverage at zero wrong in the subset: d1-ultra 24 files (12.0%, tau 0.84; 23 of them spring-boot), jev-1.14 17 (all spring-boot), pplx-v1.2 3, every other model 0-1. No model reaches a Wilson upper bound <= 5% at any tau (it needs about 75 landed files with none wrong; the best model lands 24).
      • Agreement of the best models (the file lands iff all agree on every region, no NEEDS-MERGE, min p >= tau): d1-ultra + pplx-v1.2 at tau 0.8 lands 26 files, 26 correct (upper wrong 12.9%), 25 of them spring-boot; d1-ultra + jev-1.14 at tau 0.7 lands 26, 26 correct (25 spring-boot); the triples at tau 0.7 land 20 of 20 correct (19 spring-boot). Outside spring-boot the agreeing models still err: the best pair (d1-ultra + pplx) lands 7 non-spring files at tau 0.7 with 4 correct.
      • Conflicts of merged agent PRs (AgenticFlict, Ogenrwot and Businge, arXiv 2604.03551, Zenodo 20118379, CC-BY-4.0, which has no code and no resolution: only line counts and hashes of the conflict regions of 142,652 agent PRs, 27.7% of the simulated ones in conflict). The triples were rebuilt: of its 2,125 merged PRs (1,541 repositories), 1,977 were reachable on GitHub (1,213 with a two-parent merge commit, 764 squash or rebase merges, 557 with a merge of the base into the PR branch); replaying git merge-tree on 2,360 merge commits gave 195 with a real textual conflict, and 284 files (repository, PR, path) with a base, an ours, a theirs and a recorded resolution, in 92 repositories (Codex 175 files, Copilot 83, Devin 12, Claude Code 8, Cursor 6), resolved by a person in 265 cases and by an agent or a bot in 19 (commit author). The ceiling (every region a pure side-pick) is higher than ours: 148 of the 284 files, 52% [46-58], against 41% on the 300 and 21% outside spring-boot, with OURS and THEIRS balanced (always-OURS right for 23%, always-THEIRS for 22%, where spring-boot has 90% for THEIRS), and concatenations are rarer (22 of 288 side-picks, 8%). The shape is not simpler for code: 24% of the regions have an empty side (5% in ours), 75% of the regions have both sides non-empty, 58% of the files have one region. The four best decision models on these files (2 orders each, $0.22): on a split of the repositories into halves, no group of models and no tau lands even 8 files with at most 5% wrong on half A; the best single lines are d1-ultra at tau 0.8 (22 files, 7.7%, 68% right) and tau 0.7 (43 files, 15%, 65% right); the agreement of two or three models is no better (d1-ultra with pplx at tau 0.7 lands 31 files, 61% right).
      • Reading. On both kinds of data a chooser built on these decision models is a selector that works where a constant policy already works (java / spring-boot: "always THEIRS" is right for 87% of the files at full coverage) and nowhere else: in about 70% of the go and ts-js regions the human wrote something that is none of the four picks (the ceiling above), and where the models agree on a pick they are wrong about half of the time. Judge A (reference-based, tolerant of whitespace and order) was not run on any assembled file: no gate had a candidate to judge, and an assembled file differs from the human's when the human wrote something else (15 of the 40 wrong answers of Sol in the writer benchmark), not by whitespace drift. The arms with a logprob chooser and a Luna vote were not run: Luna and Sol have no logprobs on any OpenRouter endpoint, and the screening supersedes them. What would reopen it: conflicts of agents against agents with a recorded resolution (none is public), or a decision model that reads the tests of the repository.

      15.12 The operating point

      The goal and the answer. Federico's goal was: at least 20% of the conflict files resolved and landed automatically, with at most 5% wrong among them (Wilson 95% upper bound), on data the gates were not chosen on, under the strict judge and under judge A, OSS and agent PRs apart. That goal is not met, by any gate, on either kind of data. After the first held-out numbers the bar on the wrong share was restated to 20% (below, "The threshold of 20% wrong"); that bar was set after the numbers were known, and is not a pre-registered confirmation. Against it, the best gate (G1, two models from different vendors in agreement, 59.8% of the OSS held-out files landed) has 15.5% wrong under the two first judges, 15.5% (upper bound 20.1%) under a third judge from another vendor, which is the number to quote (it is a ceiling: 15 of its 291 answers are unjudged and counted wrong), and on conflicts of agent PRs none comes close (43% to 52% wrong). What follows is the operating point that the evidence supports, the gates that were tried, and the reading.

      The judges. The strict judge calls an answer right when it equals the human's file after normalization (line endings, trailing whitespace): about a fifth of the answers it calls wrong are valid (whitespace, order, an equivalent phrasing, or a different resolution that a reviewer would accept). Judge A is the reference-based rubric of ConflictAgent (Shen 2026, MIT) adapted (the text is in Appendix B), run by a small model (glm-5.3-flash); a second judge (qwen3.8-flash, with 37 of 107 calls failed, counted as not acceptable) agrees with it on 91% of the answers and accepts fewer (187 against 205 of 270), so "judge A" below is one cheap judge and "both judges" is the conservative reading. For the gates on the held-out sets a stronger second judge was added, openai/gpt-6.1-sol (effort low, the same rubric, no failed call), run on the answers that G0 lands: it accepts fewer than the first one (on the OSS held-out G0 files it finds 74 wrong against the first judge's 56), so the figures below are given for judge 1, judge 2 and "both accept". Neither judge runs tests. A third judge was added after the first numbers, from a vendor unrelated to the two writers and to the first two judges: mistralai/mistral-large-4-0 (non-reasoning, the same rubric, Appendix B), on the answers that G1 lands (291 OSS, 130 agent-PR), a strict-equal answer being accepted without a call. It is the strictest of the three. It was not finished. Its first pass stopped at its cap ($0.25 per set) with 11 OSS and 18 agent-PR answers that are not strict-equal unjudged, and 15 and 16 calls that had returned an empty answer. A second pass (approved, about $0.10) retried those with a backoff: the empty answers came back for all 15 OSS and 13 of the 16 agent-PR ones, but each retry cost five times a normal call (an empty answer is billed, and a retry is four attempts), and the pass stopped at its cap with 24 OSS and 29 agent-PR answers unjudged and 3 agent-PR calls still empty. A third pass on the OSS set only (approved) judged 11 of the 24 and stopped at its cap again: 13 OSS answers are unjudged and 2 calls are empty, and the agent-PR set stays as it was after the second pass. All of those are counted as wrong here, so its figures are a ceiling. Judge 3 alone is the number to quote, with the majority of judges 1, 2 and 3 beside it. It cost $0.76 booked ($0.74 by the key).

      What the writer's proposals are worth (the prompt of record, Luna, effort low; "right" = the strict judge; the interval is Wilson 95%; accepted = the answer passed the rules and the syntax check):

      data files accepted right, strict right, judge A
      OSS, the 300 (tuning) 300 270 (90%) 158 = 58.5% [52.6-64.2] 205 = 75.9% (both judges: 184 = 68.1%)
      OSS held-out, 487 files outside the 300 487 432 (89%) 224 = 51.9% [47.1-56.5] 339 = 78.5%
      agent-PR, 284 replayed files 284 246 (87%) 60 = 24.4% [19.4-30.1] 131 = 53.3%

      By family, strict, accepted answers: on the 300 go 43.8% [32-56] (64 accepted), python 54.5% [43-66] (66), ts-js 40.0% [28-53] (55), java 84.7% [76-91] (85); held-out go 44.8% [38-52] (181), python 60.9% [41-78] (23), ts-js 43.8% [37-51] (169), java 93.2% [84-97] (59). Java is the only family that is mostly right, and java here is spring-boot: 63 of 63 on the 300 and 55 of 58 held-out for Luna (118 of 121, 97.5%; Wilson upper bound of the wrong rate 7.0%; under judge A none of the 63 of the 300 is wrong, and G2 below has one wrong in the 58 held-out java files); the other java repositories of the corpus are 9 of 22 (41%). The constant policy "always THEIRS" is right for 87 to 90% of spring-boot's files at full coverage (15.11): the writer adds little there beyond not being wrong. The writer's cost is about $0.0005 per file (Luna and glm-5.3-flash alike).

      The gates tried (every one is a way to land only the answers that look safe; coverage is of all the files of the set, "wrong" is among the landed, the bound is the Wilson upper bound; strict / judge A):

      gate data landed wrong strict wrong judge A
      accepted, no gate OSS 300 270 = 90% 41.5% (up 47.4) 24.1% (up 29.5) [both judges: 31.9%]
      + clean report OSS 300 247 = 82% 40.1% (up 46.3) 21.9% (up 27.4)
      + verify simulator pass (python, go, ts only) OSS 300 117 = 39% 41.9% (up 50.9) 23.1% (up 31.5)
      java alone, clean report OSS 300 81 = 27% 13.6% (up 22.7) 8.6% (up 16.8) [both: 11.1% (up 19.8)]
      go tests pass (prometheus only, differential) prometheus, 23 files 14 14.3% (up 39.9) same
      same-model self-agreement and structure rules, chosen on half the repositories, measured on the other half (30 splits) OSS 300 9.6% of the held-out half 53% wrong pooled not run
      Luna and glm-5.3-flash byte-identical answer the stratified 100 71 = 71% 36.6% (up 48.2) 21.1% (up 32.0)
      decision models, tau chosen on half the repositories 284 agent-PR files none (no group lands 8 files at 5%)
      G0: Luna, clean report, simulator ok OSS held-out 487 359 = 73.7% 40.1% (up 45.3) judge 1 15.6% (up 19.7); judge 2 20.6% (up 25.1); both 20.9% (up 25.4)
      G1: Luna x glm identical, both clean, simulator ok OSS held-out 291 = 59.8% 31.6% (up 37.2) judge 1 11.3% (up 15.5); judge 2 15.1% (up 19.7); both 15.5% (up 20.1)
      G2: G1 on java OSS held-out 58 = 11.9% 5.2% (up 14.1) both 1.7% (up 9.1)
      G0 agent-PR 284 210 = 73.9% 72.4% (up 78.0) judge 1 41.9% (up 48.7); judge 2 49.0% (up 55.8); both 51.0% (up 57.6)
      G1 agent-PR 130 = 45.8% 62.3% (up 70.2) judge 1 36.9% (up 45.5); judge 2 40.8% (up 49.4); both 43.1% (up 51.7)
      G2 agent-PR 1

      G0, G1 and G2 were fixed before the held-out runs (below) and run once; nothing was tuned on those sets. Without spring-boot G1 on the OSS held-out lands 234 files (54.8%) with 38.5% wrong strict, 13.7% (upper 18.7%) under judge 1 and 18.8% (upper 24.3%) when both judges must accept; G0 without spring-boot lands 301 (70.5%) with 24.3% wrong (upper 29.4%) under both. Per family on the OSS held-out, wrong under both judges (Wilson upper bound), G1 / G0: go 24.3% (32.9) / 29.9% (37.5), ts-js 14.6% (22.6) / 19.2% (27.0), python 6.7% (29.8) / 14.3% (34.6), java 1.7% (9.1) / 3.4% (11.5).

      The threshold of 20% wrong. After these results Federico restated the goal as "at most 20% wrong among what lands" (the 20% of files landed stays). That is a different bar chosen after the numbers were known, so it is not a pre-registered confirmation; read against it, honestly: on the OSS held-out, G1 has 15.5% wrong under both judges (11.3% under judge 1, 15.1% under judge 2) with an upper bound of 20.1% under both judges (19.7% under judge 2, 15.5% under judge 1), at 59.8% of the files landed, so it meets the bar on the point estimates and misses the bound under the conservative reading by 0.1 point; G0 (Luna alone) has 20.9% (upper 25.4%) and does not; without spring-boot G1 has 18.8% (upper 24.3%). Under the third judge, from another vendor and the strictest, G1 has 15.5% wrong (upper 20.1%) with its unjudged answers counted as wrong: under the restated bar on the point estimate, at it on the bound (0.1 point over), and 18.8% (upper 24.3%) without spring-boot, so the restated bar is met on the point estimates and missed on the bound without spring-boot. On the agent-PR files no gate comes close: 43.1% wrong at best under two judges (upper 51.7%), 51.5% under judge 3 (upper 60.0%), at 45.8% landed.

      G1 under a third judge. Wrong share among the files G1 lands, judge 3 alone and the majority (at least two of the three judges accept), Wilson 95% upper bound in parentheses; the answers with no verdict or a failed one count as wrong:

      G1 landed files judge 3 alone 2 of 3 judges
      OSS held-out 291 15.5% (up 20.1) 14.8% (up 19.3)
      OSS held-out without spring-boot 234 18.8% (up 24.3) 17.9% (up 23.4)
      go 115 25.2% (up 33.9) 23.5% (up 32.0)
      ts-js 103 14.6% (up 22.6) 13.6% (up 21.5)
      python 15 0.0% (up 20.4) 6.7% (up 29.8)
      rust-java (spring-boot but one file) 58 1.7% (up 9.1) 1.7% (up 9.1)
      agent-PR 130 51.5% (up 60.0) 42.3% (up 50.9)

      Judge 3 finds 45 of the 291 OSS answers wrong, against 33 for judge 1 and 44 for judge 2; the 15 answers that are unjudged or empty are among them. With every one of those accepted the figure would be 10.3%, with every one wrong it is 15.5% (the figure of the table); if they were wrong as often as the 77 answers of the same kind that it did judge (30 wrong), it would be about 12.3%, which is a guess, since the unjudged include the answers that had come back empty, perhaps the hard ones. On the agent-PR files 29 are unjudged and 3 empty: 26.9% if all were accepted, 51.5% as counted, about 44.5% by the same guess. The conclusion on the OSS files: the point estimate is under the restated 20% in every reading, the upper bound is at 20.1% in the reading that counts the unjudged as wrong (and under it in the others), and without spring-boot, the easy part of the corpus, both the point estimate (18.8%) and the bound (24.3%) are ~20% or over in that reading.

      Reading.

      • The statistical bar is high: at most 5% with an upper bound needs about 75 landed files with none wrong, 130 with two, and 20% coverage of a few hundred files is 60 to 120 landed. Only a gate that is nearly always right can show it.
      • On OSS the writer is right strict in about half of the files and acceptable under judge A in about three quarters; the two writers from different vendors make the same mistakes (agreement lifts the strict precision from 52% to 68% and judge A from 78% to 89%), and neither the clean report, nor a build or vet of the file, nor the tests of the one repository where they could be run, nor structure and self-agreement rules that were chosen on half the repositories and held on the other half, gets a landing gate to 95%. Java is the exception, and only spring-boot.
      • On agent-PR conflicts, the traffic that gitai serves, the writer is right strict in a quarter of the files and acceptable in about half: the recorded resolutions are often something other than the two sides' lines, and no signal tried separates the right ones. This is the hardest data we have and the lowest operating point.
      • What the wrong answers on agent PRs are. Of the 152 answers that G0 lands on the agent-PR files and that differ from the recorded resolution, 24 are the same text up to whitespace (9) or order (15), and both judges accept 23 of them. 25 are files where the recorded resolution changed lines OUTSIDE the conflict regions (a person adapting the PR to a moved base; no resolver that edits the regions can reproduce it; the judges accept 2 of them), 21 are resolutions that hold lines found in no side (new code; 5 accepted by both), and 82 are a different combination of the sides' own lines (15 accepted by both, 55 by neither judge). Of the 107 that stay wrong under "both judges accept", 23 (21%) are "outside", 16 (15%) "new" and 67 (63%) a genuine disagreement of content: so about a third of the agent-PR error is a property of the ground truth (the person did more than resolve), and about two thirds is the model choosing or composing differently from the person, which only a test can say is wrong.
      • An honest ground truth for agent traffic is not the person's file. 265 of the 284 resolutions were made by a person merging the base into the PR branch, with whatever else they changed on the way, and a recorded text is one valid answer among several. What defines a right resolution for an agent is behavioural: the merged tree builds, the tests of both changes pass, and neither change is lost. Two sources exist: the CI result recorded on the merge (not collected here), and conflicts made on purpose, from two agents' patches for two tasks of a repository with a test suite (right = both tasks' tests pass on the merged tree), which would also give agent-versus-agent conflicts. Without one of them the figures for agent PRs above are measured against a proxy.
      • What moved a number was a test: the tests of prometheus take its wrong rate from 48% to 14% (n = 14, so the bound is wide). Tests are what production's verify.command adds, and the bench cannot run them for these repositories.

      The operating point we state. A model's resolution is a PROPOSAL with its report. It lands only through the operator's verification on the exact tree (15.4, 15.9). Measured quality of the proposals, to size what the agent and the verification will see: OSS 58% right strict (68% to 76% under judge A) on the 300 and 52% (78% A) held-out; java 85 to 93%; go, ts-js and python 40 to 60%; agent-PR 24% (53% A). We cannot show, with the evidence that exists, that any offline gate lands 20% of the files at 5% wrong; the one thing that comes close is spring-boot, where a constant policy is as good. Showing it needs conflicts of agents with the operator's tests running on the resolved tree. Whether a gate chosen on half the repositories holds on the other half is asked in 15.14.

      Pre-registration of the goal gates (2026-10-20, commit cac46fa, before the held-out runs). The gates below were fixed from the tuning data (the stratified 100 and the 300) and run once on data they were not chosen on: the 487 files outside the 300 (family-stratified, spring-boot capped at 60; manifest-heldout-487) and the 284 replayed agent-PR files (manifest-agent-284).

      • G1 = the prompt of record with Luna (openai/gpt-6-luna, effort low) and with glm-5.3-flash (z-ai/glm-5.3-flash pinned to Z.AI, effort low) both accept the byte-identical answer after the judge's normalization, both composition reports are clean, and the verify simulator does not fail it. ("Clean" was measured on the lines of the report, which is cut at 16 lines; the gate in the code counts the drops and duplicates of every region (LLMResult.Findings), so it can hold slightly more than the measured G1, on the few files whose report is longer. That difference was not measured.)
      • G2 = G1 and the file is a java file. G0 = Luna alone, accepted, report clean, simulator not failed. The goal counts as met by a gate only if coverage >= 20% and the Wilson upper bound of the wrong rate <= 5% under judge A on a held-out set. (The first registration said 307 agent-PR files; the replay counted 23 files twice, and the set is 284.)

      Writer screening on the stratified 100 (prompt of record; Appendix A): glm-5.3-flash 52 right of 93 accepted (55.9% [46-66]), coverage 93%, $0.046 per 100 files, against Luna's 49 of 90 (54.4% [44-64]) on the same ids; of the other models, deepseek-v4.1-flash, qwen3.8-flash, mimo-v2.6-pro and minimax-m3 failed the five-case smoke check in this configuration (effort low, 8000 tokens, 120 s per call: their reasoning ran past the limits), and seed-2.0-code and deepseek-v4-pro were not run.

      15.13 What is configurable, and the defaults

      The writer (resolve.toml of graft-resolve; the keys are in RESOLVE.md):

      key default what it is
      [[strategy]] type = "llm", paths every path which files go to the model
      max_prompt_bytes 65536 the largest request of one region; a larger one declines the file before any call
      context_lines 3 context lines of the PATCH and the CODE
      max_answer_bytes 1 MiB the largest answer
      max_regions 20 most conflicting regions per file (one call each); more goes to the agent
      call_timeout 120 s bound on one model call
      [strategy.llm] type none openai (any OpenAI-compatible endpoint, OpenRouter included), anthropic, or command
      model, base_url, api_key_env none the model and where it is called; the key is read from the environment, never from the file
      max_tokens provider default (8000 in the shipped example) the answer limit; a reasoning model needs room for its reasoning
      effort unset (low in the shipped example) the reasoning effort sent to an endpoint that takes one
      timeout none (2400s in the example) the whole file, all its calls
      [strategy.llm.routing] only, ignore none the OpenRouter providers allowed or refused; with it set, fallbacks to other providers are off

      There is no option for git's regions, the repair round or the composition report, and no option for OURS' side of the context or an abstain block: they are the writer, or they were removed (15.10). To change the model set model (and base_url and api_key_env for another endpoint); to change who serves it, set routing.only to the provider tag. The configured provider receives the conflicting files (the three versions of each file that goes to the model and the commit message of the change); the operator chooses it, and the server never calls a model itself, only the resolver that the operator configured.

      The server ([graft.land.resolver], CONFIGURATION.md): enabled false, timeout 120 s (the whole land's deadline; plus verify.timeout when verify.enabled), max_files 20, max_file_bytes 1 MiB, proposal_ttl 168 h, check_concurrency 2, resolve_concurrency 2, verify_concurrency 2, verify_per_repository 1, resolve_rounds 32 (a cap, no contract), paths_allow, paths_deny, actors, lockfiles, generated_markers, minified.*, and the verification: verify.enabled false, verify.command, verify.timeout 5 min, verify.max_tree_entries 200000, verify.max_tree_bytes 2 GiB, verify.max_tree_depth 64. Nothing lands from a model unless verify.enabled is set and the command passes on the exact tree.

      The agreement gate for an operator who wants auto-landing (agree_with). G1 of 15.12 is an option of the llm strategy, off by default: a nested [strategy.llm.agree_with] table, with the keys of [strategy.llm], names a second model, which should come from another vendor. Both models resolve each file at the same time with the prompt of record. The first model's resolution is proposed; it can land only if the two results are the same file after the judge's normalization, the composition reports of both are clean, and verify.command passes on the exact tree (verify.enabled is mandatory, as for every model resolution). Otherwise the resolution is held: the response carries a hold reason and the report says that the second model disagreed, and the server never lands a held file, nor runs the verification for it. A budget table must be given for both models or for neither. The bench runs the same code and prices both models. RESOLVE.md has the configuration. Measured on the OSS held-out files it lands 59.8% of the files with 15.5% of those wrong under two independent judges (upper bound 20.1%) and at most 15.5% under a third judge from another vendor (upper bound 20.1%), before the operator's tests; on conflicts of agent PRs 45.8% of the files with 43.1% wrong. Those numbers were measured as two single-model runs combined offline (the gate decides the same thing on the same answers); the code is covered by tests with fake providers and was not run through the bench against the live models. The 20%-wrong bar it is compared with was set after the numbers were known (15.12).

      The default model. The shipped example is openai/gpt-6-luna at effort low. On the stratified 100 the one other model that passed the smoke check, z-ai/glm-5.3-flash pinned to Z.AI, is as good (15.12) and costs the same per file; the models that reason without a bound on their effort ended their calls past call_timeout or max_tokens (Appendix A), which is a property of the configuration here (effort low, 8000 tokens, 120 s per call) and not a judgment of the models.

      15.14 Is there a principled gate? (analysis, no paid call)

      The question (2026-10-20): the restated bar, at most 20% wrong among what lands (Wilson 95% upper bound), holds for G1 on the whole OSS held-out (14.8%, upper 19.3%, 2 of 3 judges) and not without spring-boot (17.9%, upper 23.4%), not on go (25.2%) and not on agent PRs (about 51%). Is there a gate that uses only what the server knows before it lands, no judge, and keeps the upper bound at or under 20% without spring-boot as well, on data it was not chosen on?

      The answer. No gate was shown to. One gate, chosen on half the repositories, meets the bound on the other half with spring-boot (7.0% wrong, upper 13.7%, 34.6% of the files landed), and it does so because it keeps spring-boot and drops go; without spring-boot it lands 18.8% of the files with 14.0% wrong and an upper bound of 27.3%. Exchanging the halves, the gate chosen is almost G1 and does not meet the bound (upper 23.5%). Chosen on the same data it is measured on, the best gate (an optimistic ceiling) reaches the bound without spring-boot by 0.4 points under one judge reading and misses it under the other. And the data cannot show it either way: to show an upper bound of 20% for a true wrong rate of 14% takes 160 landed files, and the held-out has 229 files outside spring-boot. Details follow; the figures are of c1-work/run/gate-b/ (PROTOCOL.txt, split.json, out-main.txt, out-swap.txt; scripts gate_split.py, gate_b.py).

      Method, fixed before any outcome of the held-out sets was loaded for it (PROTOCOL.txt, sha256 50fc874e22df; the split split.json, sha256 688ebbe125b5):

      • Split by repository, with a seed. The ten repositories of the OSS held-out, sorted, shuffled with random.Random(20261020): half A, to choose the gate: django, react, TypeScript, flask, prometheus (198 files; go 70, ts-js 105, python 23, no java); half B, to evaluate it: kafka, golang/go, vscode, pandas, spring-boot (289 files; go 130, ts-js 95, java 61, python 3). Ten repositories is a coarse split: A has no java, B has go, vscode and spring-boot.
      • The signals, all known before landing and none a judge or a repository name: language family (from the extension), git regions of the file, conflict lines (the lines of both sides inside git merge-file's regions), lines of the largest side, the kind of path (test, config, lock, docs, other), whether a composition report says "found in neither side", and whether the answer is a pick of one whole side or a mixture. Every candidate lands a subset of G1 (the two models' identical answer, both reports clean, the simulator not failing), because judge 3 ran only on the answers G1 lands; so the agreement of the two models and the deterministic composition check are inside every candidate. Of 17 predicates (a fixed grid of thresholds, in PROTOCOL.txt) a candidate is none, one, or two of different kinds: 139 candidates. The type of conflict (both sides edit, or one deletes) could not be tried: all 487 and all 284 files are content.
      • Selection, on A only: among the candidates that land at least 15 files of A with an upper bound of the wrong rate at or under 20% (2-of-3 label, answers with no verdict counted wrong), the one that lands the most. Then measured once on B. Nothing was tuned on B. The grid and the rule were written before the numbers; but the per-family results of 15.12 (go weak, java strong) were known, so "family is not go" is not an independent finding.

      The result, on B (wrong = not accepted by 2 of 3 judges; in parentheses the Wilson upper bound; the last column is judge 3 alone, the strictest):

      B files landed wrong, 2 of 3 judge 3 alone
      G1 289 195 = 67.5% 29 = 14.9% (up 20.5) 31 = 15.9% (up 21.7)
      chosen: family is not go, one region 289 100 = 34.6% 7 = 7.0% (up 13.7) 7 = 7.0% (up 13.7)
      G1 without spring-boot 229 138 = 60.3% 28 = 20.3% (up 27.8) 30 = 21.7% (up 29.3)
      chosen, without spring-boot 229 43 = 18.8% 6 = 14.0% (up 27.3) 6 = 14.0% (up 27.3)
      chosen, spring-boot only 60 57 = 95.0% 1 = 1.8% (up 9.3) 1 = 1.8% (up 9.3)
      chosen, ts-js (G1: 57 landed, 12.3%, up 23.2) 95 41 = 43.2% 6 = 14.6% (up 28.4) 6 = 14.6% (up 28.4)
      chosen, java 61 58 = 95.1% 1 = 1.7% (up 9.1) 1 = 1.7% (up 9.1)
      chosen, go (G1: 79 landed, 26.6%, up 37.2) 130 0

      The gate was chosen on A (it lands 48 of 198 there, 8.3% wrong, upper 19.6%, all python and ts-js: A has no java). On B it meets the bound, and 57 of its 100 landed files are spring-boot. Outside spring-boot it lands less than 20% of the files and its bound is 27.3%. It does not improve ts-js (14.6% wrong against 12.3% for G1). What it did was drop go, the family that is 26.6% wrong. Its cost: landing half of what G1 lands (34.6% against 67.5%).

      What else was run (supplementary; flagged as such, not results):

      • The halves exchanged: the gate chosen on B is "the answer is a mixture of the sides" (191 of 195 G1 files, 13.1% wrong, upper 18.6%), almost G1, and on A it lands 94 of 198 with 14.9% wrong (upper 23.5%; G1: 14.6%, upper 23.0%). It does not meet the bound.
      • Leave one repository out, choosing on the other nine: the gate chosen is G1 itself in 8 of 10 rounds (no predicate beats it by the rule), "a mixture" for vscode (10.7% wrong, upper 21.5%) and "mixture and at most 20 conflict lines" for spring-boot. The chosen gate is not stable.
      • An optimistic ceiling (the best candidate chosen and measured on the same data): over all ten repositories it is G1 itself (14.8%, upper 19.3%); without spring-boot it is "at most 20 conflict lines and a mixture": 193 of 427 files landed (45.2%), 14.0% wrong, upper 19.6% under 2 of 3, 20.2% under judge 3 alone, against G1's 234 (54.8%) at 17.9% (upper 23.4%). On go alone no candidate lands 15 files with a bound at or under 20% (the closest lands 92, 18.5% wrong, upper 27.6%); on ts-js alone "at most 80 conflict lines" lands 97 of 200, 11.3% wrong (upper 19.2%; judge 3 alone 20.4%).
      • Each signal alone, among the 291 files G1 lands (descriptive, pooled, after the fact): wrong 11.5% with at most 10 conflict lines (226 files), 13.1% with one region (245), 12.4% without "found in neither side" (226); against 25.6% with 2 to 4 regions, 50.0% with more than 80 conflict lines (12), 23.1% with "found in neither side" (65). By family: java 1.7%, ts-js 13.6%, go 23.5%. The kind of path does not separate (test 14.7%, other 14.8%; no config, lock or doc file is landed by G1). The signals move the rate by a few points; none turns a family that is 23% wrong into one that is 15%.
      • The frozen gate on the agent PRs (out of domain): G1 lands 130 of 284 with 42.3% wrong (upper 50.9%; judge 3 alone 51.5%, upper 60.0%); the chosen gate lands 93 (32.7%) with 39.8% wrong (upper 49.9%; judge 3 alone 47.3%, upper 57.4%). The ceiling chosen on the same agent data: no candidate lands 15 files with an upper bound at or under 20%; the closest lands 28 at 17.9% wrong, upper 35.6%. There the signals tell a little more (22.2% wrong when the answer is a pick of a side, 50.0% when it is a mixture; 36.1% without "found in neither side", 72.7% with it), but not enough for the bar.
      • What the data can show. The fewest landed files for which the upper bound is at or under 20%, by the true wrong rate: 25 files at 5% wrong, 54 at 10%, 87 at 12%, 160 at 14%, 365 at 16%, about 1,500 at 18%. With 229 files outside spring-boot, 43 to 193 landed, a gate at 14% wrong cannot be shown to be under 20%, and a gate at 10% needs 54 landed, which no candidate reaches there.

      Reading, honestly.

      1. Is there a principled gate that keeps the upper bound at or under 20% without spring-boot? Not shown, and probably not with these signals. The only gate that clears the bound on held-out repositories does it on spring-boot, which a constant policy ("always THEIRS", 15.11) already covered. Outside it, the best that the signals do (an optimistic ceiling) is 14% wrong at the bound, selected on the data it is measured on.
      2. What it costs: the gate that clears the bound with spring-boot lands half of what G1 lands (34.6% against 67.5% of B) and no go file at all. The ceiling without spring-boot lands 45% of the non-spring-boot files against G1's 55%.
      3. Go is the limit: 23.5% to 26.6% wrong under G1, and no candidate gets its bound under 27%. A gate that keeps the bound without spring-boot has to leave go to the operator's tests.
      4. On agent PRs no signal of the server comes close (40% wrong at best), consistent with 15.12.
      5. What would be needed to claim a gate, in this order: (a) A signal that is not the text: the operator's tests on the resolved tree (verify.command), the only thing that moved a number (prometheus, 48% to 14%; measured on G1 in 15.15); for go, go build and the package tests of the files, which the bench cannot run for most of the repositories. (b) More data, chosen for the claim: at a true wrong rate of 14% the bound needs about 160 landed files per claim, so a held-out of at least 600 files outside spring-boot, from at least 20 repositories and several java ones (java here is one repository, spring-boot), with the split by repository and the gate fixed first. (c) Behavioural ground truth for agent conflicts (15.12): the CI result of the merge, or conflicts made from two agents' patches with a test suite. Until then the operating point stays that of 15.12: a model's resolution is a proposal, and it lands through the operator's verification.

      Limits of this analysis. Ten repositories, two halves with five each, one of which holds no java; 139 candidates chosen on 96 landed files; the label is the verdict of three LLM judges (unjudged answers counted wrong), not tests, and judge 3 ran only on G1's answers, so a gate looser than G1 cannot be evaluated; the signals are the ones listed. A different split would pick a different gate (the exchange above did), which is the point.

      15.15 G1 and the project's real tests (measured, no paid call)

      The question (15.14 ended on it): the one thing that moved a number is the project's own tests run on the resolved tree, which is also how the product lands a model's answer (verify.command). If G1's answers are run against the tests of the repositories that have runnable ones, what is the wrong rate among those that pass?

      The answer, against the rule fixed beforehand: shown. Outside spring-boot, in the repositories whose tests could be run, 69 answers pass (T), 6 of them wrong under the 2-of-3 judge label: 8.7%, upper bound 17.7% (judge 3 alone: 5 wrong, 7.2%, upper 15.9%); the rule was an upper bound of 20% or less with at least 54 landed files. T is 29.5% of the 234 files that G1 lands outside spring-boot (16.2% of the 427 files of the held-out outside spring-boot) and 56.6% of the 122 files in the repositories that were run. The margin is thin (one more wrong file would leave the bound at 19.5%, two more at 21.2%), the scope is narrow (below), and most of T is made of answers that are identical to the human's file, so read the rest before quoting the number.

      Protocol, hashed before the first run (c1-work/run/gate-b/): PROTOCOL-TESTS.txt (sha256 5718545975e9), with two amendments written before any held-out file was run, for infrastructure only (PROTOCOL-TESTS-AMENDMENT-1.txt, 08538850d9af; -2.txt, 626da126ca29). The test is differential: the worktree of the case's merge commit M with the human's file is the baseline, run twice; the tests that pass in both are the set P; the answer replaces that one file and P must still pass (a failure is confirmed by a second run). T = G1 and PASS; T' = T and at least half of the executable lines the resolution wrote are executed by the tests (Go only). A file that cannot be tested is NOT landed, with a reason code decided by a rule (R-REPO, R-TOOLCHAIN, R-BASEBUILD, R-NOBASETESTS, R-UNSTABLE, R-TIMEOUT, R-INSTALL). The commands: prometheus go test of the file's package (-timeout 300s), golang/go the toolchain of the merge built with make.bash and go test -short of the package, flask pytest tests in a venv. Not run, by rule, before any outcome: django (2012-2013 on python 3.12), TypeScript (gulp/jake on node 18, a very long suite), vscode (electron and native modules), react, pandas, kafka: 112 of the 234 files. The parents of the case are not run. Wrong is the label of 15.14 (not accepted by 2 of 3 judges, unjudged answers counted wrong); the 234 files already had their verdicts, so no judge was called. Nothing is tuned, so there is no split and the pooled result is the primary.

      The harness was validated before the pilot, on files of the 300 (tuning set, not in this population). Against the earlier simulator (vsim-gotest2) it showed two things. The old simulator did not detect a package whose tests time out, so its "pass" compared partial sets of passing tests; the new one counts a timed-out package as R-TIMEOUT. And a bug of mine: the answer stayed in the worktree after a file was checked, so the next file of the same merge was run on a tree that held the previous answer (three files wrongly ended R-BASEBUILD); the human's file is now put back after every file. After the fix the files agree with the old simulator (14 pass, 5 fail, 4 whose baseline does not build, for the 23 prometheus files). A check that stays in the harness: an answer identical to the human's file must always pass (57 such answers in the run, none failed). The pilot (11 files, one per Go era) found two more faults, both for old Go trees (GOPATH mode; go test -json absent before go 1.10), fixed in Amendment 2; all three Go eras then built a toolchain with the local go 1.26 as bootstrap, in 12 to 86 s.

      What happened to the 234 files (G1-landed, outside spring-boot):

      repository files PASS FAIL not landed, with reason
      golang/go 79 44 8 R-NOBASETESTS 17 (13 are files of the test/ directory, run by one program, the others packages with no test file), R-TOOLCHAIN 10 (2014-2019: the 2014 C bootstrap asks for hg (4), the 2015 trees for a VERSION file (2), the 2019 ones fail with "cannot find main module" (2), a 2016 and a 2018 tree do not build (2))
      prometheus 36 24 2 R-BASEBUILD 6 (the human's merge commit itself does not build), R-NOBASETESTS 2, R-TIMEOUT 2 (tsdb/head.go, tsdb/head_wal.go)
      flask 7 1 0 R-BASEBUILD 3 (the 2010-2019 sources do not import on python 3.12), R-NOBASETESTS 3
      vscode, TypeScript, django, react, pandas, kafka 112 R-REPO (not run, by the rule above)

      The files that were run are 122; every reason is in tests-full.jsonl, and the list by reason is printed by scripts/tests_report.py.

      The result (wrong = not accepted by 2 of 3 judges; Wilson upper bound in parentheses):

      set files wrong, 2 of 3 judge 3 alone
      G1 on the 122 files that were run 122 28 = 23.0% (up 31.2) 29 = 23.8% (up 32.1)
      T = G1 and the tests pass 69 (29.5% of 234; 56.6% of the 122 run) 6 = 8.7% (up 17.7) 5 = 7.2% (up 15.9)
      T, golang/go 44 2 = 4.5% (up 15.1) 2 = 4.5% (up 15.1)
      T, prometheus 24 3 = 12.5% (up 31.0) 3 = 12.5% (up 31.0)
      T, flask 1 1 0
      T' = T and the regions are covered 31 (13.2% of 234) 0 = 0.0% (up 11.0) 0 (up 11.0)

      (By family: Go, 68 files of T, 5 wrong = 7.4%, upper bound 16.1%; python has one file.) G1 alone on the same repositories is 26.6% wrong for golang/go (79 files, upper 37.2) and 16.7% for prometheus (36 files, upper 31.9).

      What the tests did. On the 79 files whose status is PASS or FAIL (G1 without tests lands all of them): wrong 15 = 19.0% (upper 29.0) without the tests, 6 = 8.7% with them. The tests removed 9 wrong answers, lost 1 right answer (a false positive: src/crypto/tls/cipher_suites.go, whose package tests do not build with the answer), and let 6 wrong answers through; 63 right answers pass. Ten of the failures are a package that does not build with the answer (6) or a test that passes at the baseline and fails with it, twice (4). But 57 of the 79 answers are identical to the human's file, which pass by construction (none failed, which checks the harness); the information of the tests is on the other 22: 10 FAIL (9 wrong, 1 right) and 12 PASS (6 wrong, 6 right). The tests therefore caught 9 of the 15 wrong answers (60%).

      The secondary arm. The five Go answers that are wrong and pass (config/config.go, promql/functions.go, scrape/scrape.go, cmd/link/internal/ld/deadcode.go, crypto/rand/rand_unix.go) all have T' false: the tests do not execute the lines the resolution wrote. T', which asks for that, has 31 files and no wrong one (upper 11.0%), but 27 of its 31 are identical to the human's file, so it rests on 4 answers that differ. It is reported without a claim; it says that the tests that pass are only as good as what they run, and that asking for coverage of the resolved lines is a sensible condition for the product (go test -cover, or the operator's coverage tool on the changed lines).

      Reading, honestly.

      1. For repositories whose tests can run on the resolved tree, G1 and the tests keep the wrong rate at 8.7% (upper 17.7%) at 56.6% of the files run: the lever of 15.14 works where it can be pulled, and it is the product's own verify.command. The result says nothing about repositories without runnable tests, and 112 of the 234 files (vscode, TypeScript, django, react, pandas, kafka) were not run at all; the families python and ts-js are not covered by this measurement (one python file).
      2. The bound is pooled and thin. golang/go alone (44 files) is 15.1% and prometheus alone (24 files) is 31.0%; the pooled 69 files clear 20% by 2.3 points, one more wrong file would still clear it (19.5%) and two more would not (21.2%). The label is three LLM judges, not tests, and T' covers fewer files.
      3. In the whole held-out the combination is: T (69 files) plus the spring-boot files that G1 lands, which were not tested (57 files, 1 wrong in 15.14): 126 of 487 files (25.9%) with 7 wrong, 5.6% (upper 11.0%). That mixes a tested part and a constant-policy-equivalent part, and is given for orientation, not as a claim.
      4. 10 golang/go files of 2014-2019 were lost to the toolchain of the harness (hg, a VERSION file, "cannot find main module", two trees that do not build). A fix after the result was seen would be a post hoc change, so they stay not landed.
      5. What it means for the product: a model's resolution lands through the operator's verification (15.4, 15.9); this measures, on public Go repositories, what the verification buys, and it supports the operating point of 15.12. The honest statement is conditional: with the project's tests running on the resolved tree, and when the lines of the resolution are covered, the answers that pass were wrong in 8.7% of the cases (upper bound 17.7%) on this sample.

      15.16 The replication of G1 and the real tests on fresh Go conflicts (measured, $0.81 booked)

      The answer, against the rule fixed beforehand: replicated. On 296 Go files that no run, no judge and no gate had seen, G1 lands 144; the frozen harness of 15.15 passes 65 of them (T), 5 wrong under the 2-of-3 judge label: 7.7%, upper bound 16.8%. The rule was an upper bound of 20% or less with at least 54 files on the fresh set alone. Pooled with 15.15 (69 files, 6 wrong): 134 files, 11 wrong, 8.2%, upper bound 14.1%. With the goal of at most 20% wrong among what lands, the statement that now has two independent measurements is the conditional one of 15.15: for Go repositories whose tests can be run on the resolved tree, the answers that G1 produces and that pass the tests are wrong in 8% of the cases (upper bound 14% pooled). It is not an offline gate, and it says nothing for families other than Go.

      Protocol, hashed before the first paid call (c1-work/run/gate-b/PROTOCOL-REPL.txt, sha256 7242716e7e1e; the PM approved it as written):

      • The fresh set (mine/manifest-fresh-go.json, sha256 0b230e1367a7; 296 files): the Go cases of golang/go (121) and prometheus (175) of the first manifest plus an older scan of prometheus's merges (the 3,000 newest had been scanned; the new scan took 75 s and added 113 cases), minus every id of the 300, the 20 anchor files and the held-out 487. No file is sampled out; no decision-model screening had touched any of them.
      • Frozen: the writers' binary (graft-resolve sha256 056745e31133, the one of the held-out runs), their two configurations (Luna; glm-5.3-flash on Z.AI), the verify simulator, the judges and their rubric, the test harness of 15.15 (test_gate.py, sha256 6771e0c001ae; the run used a copy whose only difference is a population "fresh"), Amendments 1 and 2 of 15.15, and the signal code of the secondary arm. A probe of fixes for the old Go toolchains on the 8 merges lost in 15.15 built none of them, so there is no new amendment.
      • Prediction, written before the run: G1 lands 55-60% of the files, T is 55-80 files, the wrong rate among T is 6-13%. Two of the three held: T is 65 and its wrong rate 7.7%; G1 landed less than predicted, 48.6%, because the fresh files are harder (Luna's error rate is 60.6% against 55.2% on the held-out Go) and because 44 of the 73 G1-landed prometheus files come from merges without a go.mod, which the harness cannot test.
      • Cost: writers $0.1888 (Luna) and $0.2108 (glm); judges $0.0053, $0.0906 and $0.3097; $0.805 booked, $0.760 by the key (the bench prices at the highest of the allowed endpoints), against the cap of $1.5. Judge 3 reached its cap of $0.30 at 36 of the 59 verdicts (about $0.0086 per call, four times the estimate): the 23 answers it did not reach count as wrong, as the protocol says, and no extra pass was run.

      What happened to the 144 G1-landed files:

      repository, era files PASS FAIL not landed, with reason
      golang/go, 2022 and after 14 10 0 R-NOBASETESTS 4
      golang/go, 2020-2021 33 17 2 R-NOBASETESTS 14
      golang/go, 2019 and before 24 17 4 R-TOOLCHAIN 3 (a 2014, a 2015 and a 2018 tree)
      prometheus, merges with a go.mod 29 21 0 R-TIMEOUT 5, R-BASEBUILD 3
      prometheus, merges without a go.mod 44 0 0 R-NOBASETESTS 44 (the GOPATH era: the harness runs modules)

      R-NOBASETESTS in golang/go is mostly the test/ directory (run by one program, not by a package test) and packages with no test file. Only 3 golang/go files were lost to old toolchains, against 10 in 15.15.

      The result (wrong = not accepted by 2 of 3 judges; Wilson upper bound in parentheses):

      set files wrong, 2 of 3 judge 3 alone (unjudged counted wrong)
      G1 on the fresh set 144 (48.6% of 296) 38 = 26.4% (up 34.1) 45 = 31.2% (up 39.2)
      T = G1 and the tests pass 65 (45.1% of the 144; 22.0% of the 296) 5 = 7.7% (up 16.8) 9 = 13.8% (up 24.3)
      T, golang/go 44 3 = 6.8% (up 18.2) 5 = 11.4% (up 24.0)
      T, prometheus 21 2 = 9.5% (up 28.9) 4 = 19.0% (up 40.0)
      T' = T and the resolved lines are covered 33 2 = 6.1% (up 19.6) 5 = 15.2% (up 30.9)
      pooled 15.15 + fresh 134 11 = 8.2% (up 14.1)

      What the tests did. On the 71 files whose status is PASS or FAIL: wrong 11 = 15.5% (upper 25.7) without the tests, 5 = 7.7% with them. They removed 6 wrong answers, lost no right answer (15.15 lost one), and let 5 wrong answers through; 60 right answers pass. Fifty of the 71 are identical to the human's file and none failed, as it must; the information is again in the others: of the 15 answers of T that differ from the human's file, 5 are wrong. The tests caught 6 of the 11 wrong answers they could judge (55%; 60% in 15.15).

      Where the replication is weaker than 15.15.

      1. The label. The rule uses the 2-of-3 label with the unjudged counted wrong, which is conservative: of the 15 answers of T that differ from the human's file, 7 had no judge-3 verdict. Had judge 3 accepted every one of them, T would have 3 wrong in 65 (upper bound 12.7%); no completion pass was needed, since it could only lower the wrong count. Judge 3 alone with its unjudged counted wrong gives 13.8% (upper 24.3%), which is above 20%, and is shown for that reason.
      2. The margin. One more wrong file would leave the bound at 18.7%; two more at 20.6%. The pooled bound, 14.1% on 134 files, is the sturdier figure.
      3. The coverage arm did not hold up. In 15.15 the five wrong answers that passed had all T' false (0 wrong among 31). Here two of the five escapes (src/testing/testing.go, src/go/types/expr.go) have T' true: T' has 33 files and 2 wrong. Coverage of the lines the resolution wrote is not a clean extra filter.
      4. Prometheus alone does not show it (upper bound 28.9% on 21 files), as in 15.15; golang/go carries the result (44 files, 6.8%).
      5. The scope is the same as in 15.15: Go, and repositories whose tests run on the resolved tree. 62 of the 144 files were not landed because their tests could not be run (R-NOBASETESTS), 5 for a timeout, 3 for a baseline that does not build and 3 for an old toolchain.

      The secondary arm: the identifier signal, frozen before the run (signal-frozen/, signals.py sha256 2b486d9ba0aa). It flags 12 of the 144 files: 10 wrong and 2 right. False positives among the 106 right answers: 2 (1.9%, upper bound 6.6%). It flags 10 of the 38 wrong answers (26%), and 5 of the 27 wrong answers whose tests could not run (18.5%); with new_errs_set added, 11 of 38 and 6 of 27. Of the wrong answers that pass the tests it flags one (tsdb/record/record.go, "theirs eliminated uint64") and misses the other four. The two false positives are crypto/tls/common.go and web/api/v1/api_test.go. The pre-registered reading was a false-positive upper bound of 3% or less and at least 20% of the wrong answers without runnable tests: neither is met (6.6%; 18.5%). The signal keeps its place as a finding of the composition report, as decided (a finding, not a hold, Go only), but this measurement does not support more than that: on 144 files its false-positive bound is 6.6%, and it flags about a fifth of the errors the tests cannot see.

      Reading.

      • The measurement that 15.15 left open, whether the result holds on independent data, is answered for Go: it does, with the same size (T 65 against 69 files, 7.7% against 8.7%).
      • The two together support one operational statement, and no other: a model's resolution that two writers from different vendors produce identically and that passes the project's own tests on the resolved tree is wrong in about one case in twelve (pooled 8.2%, upper bound 14.1%, 134 files, two repositories' worth of Go). That is the product's verify.command doing its job, and 15.12's operating point stands.
      • What is still not shown: any other language, repositories without tests, agent traffic (15.12), and anything about answers that fail to build old trees.

      15.17 Without tests: G1 and a build check of the package (Go; measured, no paid call)

      The question. For a project that has no runnable test suite, is there an operating point that needs no tests? The cheapest check is "the package and its test files still compile and link with the answer in place of the human's file": the type-check of the package. It is the check that the Go errors of the 27 wrong held-out files (renames and APIs not propagated, 15.14) would trip.

      The answer, against the rule fixed beforehand: met on the fresh set, where it can be run. On the fresh Go set (the only evidence by the protocol), 74 answers pass the build check (T_B), 5 wrong under the 2-of-3 label: 6.8%, upper bound 14.9%, with at least 54 files. On the held-out Go files (labelled not blind: its G1 results, its wrong files and the 15.15 test results were seen before the protocol) the same check gives 77 files, 9 wrong, 11.7%, upper bound 20.7%, which is just above the bar. Pooled: 151 files, 14 wrong, 9.3%, upper bound 15.0%. The condition is as important as the number: 63 of the 144 G1-landed fresh files are not covered by the check (the Go of a prometheus before it had a go.mod, the test/ directory of golang/go, a toolchain that does not build), and those are the files where G1 is wrong 41% of the time. The check lands the Go that a module/toolchain can build; it does not make the rest safe.

      Protocol, hashed before the first run (c1-work/run/gate-b/PROTOCOL-BUILD.txt, sha256 840dbd5adbbb): at the case's merge commit, with the human's file in place the package is built twice (go test -run '^$' -count=1: it compiles and links the package and its tests and runs no test); it must build; the answer then replaces the file and the same command is run: it builds (BUILD-PASS) or not (FAIL). Everything else is the harness of 15.15 (test_gate_b.py, sha256 a5e96839d965, whose only differences are the build runner and the dropping of the "package has a test file" condition), with its reason codes, its toolchains and its Amendments. A file that cannot be built in the way the harness runs it is not covered and not landed: nothing was tried in GOPATH mode, as a single file, or with a fix of the old toolchains. No label was added: the verdicts of 15.12 and 15.16 are hashed in the protocol. The prediction written before the run was 100-130 files at 8-16% wrong and a coin flip on the 20% bound; the run gave fewer files (74) at a lower wrong rate (6.8%).

      What the check did (G1-landed Go files; wrong = not accepted by 2 of 3 judges; Wilson upper bound in parentheses):

      held-out Go (not blind) fresh Go (the evidence)
      G1 alone 115 files, 27 wrong = 23.5% (up 32.0) 144 files, 38 wrong = 26.4% (up 34.1)
      T_B: the build passes 77, 9 wrong = 11.7% (up 20.7) 74, 5 wrong = 6.8% (up 14.9)
      the build fails (rejected) 7, 6 wrong 7, 7 wrong
      not covered, not landed 31, 12 wrong = 38.7% 63, 26 wrong = 41.3%
      T_B, golang/go 49, 4 wrong = 8.2% (up 19.2) 48, 3 wrong = 6.2% (up 16.8)
      T_B, prometheus 28, 5 wrong = 17.9% (up 35.6) 26, 2 wrong = 7.7% (up 24.1)
      the tests of 15.15/15.16 on the same files 68, 5 wrong = 7.4% 65, 5 wrong = 7.7%
      build passes, tests not PASS 9, 4 wrong 9, 0 wrong

      The 14 answers the check rejects are wrong in 13 cases: a build error is a very precise signal of a wrong answer. What it misses are the wrong answers that compile: 9 and 5 of them. In the held-out the tests catch 4 more wrong answers that build (their tests fail), which is why the test check is at 7.4% and the build check at 11.7% there; in the fresh set the build check loses nothing against the tests (all 6 test failures were build failures). The not-covered files by reason, fresh: R-NOBASETESTS 57 (44 prometheus merges without a go.mod, and 13 golang/go single-file programs of test/ or files outside src/; a package without a test file is covered here, unlike by the tests), R-TOOLCHAIN 3, R-BASEBUILD 3. By era, golang/go (fresh): 2022 and after 14 files (12 build, 1 fail), 2020-2021 33 files (19 build, 2 fail, 12 not covered), 2019 and before 24 files (17 build, 4 fail, 3 toolchain).

      Reading.

      1. Where a Go project builds, G1 and a build check is a no-tests operating point that clears the bar on independent data (74 files, 6.8%, upper bound 14.9%), but not on the development data (upper bound 20.7%): the pooled figure, 9.3% with an upper bound of 15.0%, is the one to quote, with the two parts shown. It is weaker than the tests (which remove 4 more wrong answers in the held-out) and not weaker than them in the fresh set.
      2. It is a Go result. Python and TypeScript are measured on the corpora of the next section, with the same rule per family.
      3. The files that the check cannot reach carry a wrong rate of about 40%: a product that uses this operating point must treat "the package does not build in the way the check runs it" as do not land, and say so in the package it returns.
      4. A build error is a precise signal (13 of 14), the converse is not: 14 wrong answers compile (9 held-out, 5 fresh). A check that sees more than the compiler (the bindings delta of the identifier signal, 15.16) has room there, but its measured false-positive bound (6.6%) is not what a gate can afford.

      15.18 Python: G1, the project's real tests and the no-tests arm (measured, $0.97 booked)

      The answer, against the rules fixed beforehand: neither is met. On 509 Python files that no earlier run, judge or gate had seen, G1 lands 331 (65.0%) and 58 of them are wrong under the 2-of-3 judge label: 17.5%, upper bound 22.0% (without sympy: 173 files, 30 wrong, 17.3%, upper 23.7%). The no-tests rule (G1 alone, at least 54 files, upper bound 20% or less, with and without sympy) is not met in either form. The tests rule (T = G1 and the project's own tests pass) gives 157 files, 25 wrong = 15.9%, upper bound 22.5% (without sympy: 64 files, 8 wrong, 12.5%, upper bound 22.8%, computed from the per-repository lines of the same report): not met. Go (15.15 to 15.17) does not carry over to Python at the bar of 20% with this data; the point estimates are close to the bar, the intervals are not.

      Protocol, hashed before the first paid call (c1-work/run/gate-b/PROTOCOL-PY.txt, sha256 8f0abfa4fdc5; the PM approved the plan, Federico the cost):

      • The set (mine/manifest-pyfresh.json, sha256 f6daa7740954; 509 files from merges since 2021): sympy 253, Pillow 107, sphinx 69, twisted 32, pip 30, werkzeug 18. Pillow's licence (HPND) was added to the miner's permissive list for this (commit 5edeb53 of c1-conflicts; a separate miner binary). None of the files was used by an earlier set.
      • The recipe rule, written before any model output existed: a repository enters the set only if its recipe (a venv per merge, the editable install, the extras and dependency groups its tests need) passed on 3 merges with the human's file standing for the answer: at least 2 files PASS and the median time of the PASS files at most 600 s. scrapy (1,449 and 2,040 s a file) and aiohttp (the editable build fails) were dropped by the rule; pytest, rich, click, requests and the other candidates have fewer than 20 cases or squash their merges.
      • The tests (test_gate_py.py) import the harness of 15.15/15.16 unchanged (baseline twice, the answer, a confirmation on a failure; PASS, FAIL, R-*) and add a pytest runner and a selection: S0 = the test files that name the changed module (all of a suite of at most 120 files when none does); S1 = those of S0 that contain a test which executed a line of the changed file (pytest-cov with a context per test, on the human's file); the differential runs S1. A changed test file is its own selection. No S1 is R-NOBASETESTS.
      • The writers, G1 and the judges are those of 15.16 (Luna and glm-5.3-flash, the verify simulator with py_compile and a pyflakes differential, three judges, arm A). Cost rule: one paid job at a time, a cost line after each, the running total plus the next cap at most $1.5.
      • Prediction, written before the run: G1 lands 50-60% of the files; T is 70-110 files; the wrong rate among T is 5-15%. Held: none of the three (65.0%; 157; 15.9%): the files are easier for G1 than the Go ones and the tests pass on more of them, but the wrong answers among them are more.

      Amendments (all written and hashed before the stage they touch). (1, sha256 b83a4a7db65e) the no-tests arm: G1 with its static checks on all G1-landed files, with or without tests, as a second rule; the judges run on all the G1-landed answers that differ from the human's file, judge 3 only where judges 1 and 2 disagree (where they agree the 2-of-3 majority is decided; "judge 3 alone" is therefore not reported for Python). (2, bf3b9cca3fb6) the caps of the judge stages raised to $0.28 and $0.55, after the writers cost less than their caps and 146 (not 115) answers needed judging; no rule, model or prompt changed. (3, 1955c60e1745) and (4, 6b9847f7ad19) infrastructure, below.

      Cost. Writers $0.2728 (Luna) and $0.2718 (glm); judges $0.0129, $0.2271 and $0.1833 (judge 3: 8 verdicts at $0.0229 each, 2.7 times the 15.16 unit cost); $0.968 booked, $0.920 by the key (the bench prices at the highest allowed endpoint), against the cap of $1.5.

      What happened to the 331 G1-landed files:

      repository G1-landed PASS FAIL not landed
      sympy 158 93 2 R-NOBASETESTS 62, R-BASEBUILD 1
      Pillow 67 34 5 R-NOBASETESTS 28
      sphinx 55 10 1 R-NOBASETESTS 40, R-BASEBUILD 4
      twisted 20 12 1 R-NOBASETESTS 7 (4 of them: the selection run exceeded 900 s)
      pip 18 3 0 R-NOBASETESTS 11, R-BASEBUILD 4
      werkzeug 13 5 0 R-NOBASETESTS 7, R-BASEBUILD 1
      all 331 157 9 R-NOBASETESTS 155, R-BASEBUILD 10

      The results (wrong = not accepted by 2 of 3 judges, unjudged counted wrong; Wilson upper bound in parentheses):

      set files wrong, 2 of 3 answers that differ from the human's file
      G1 alone (the no-tests arm) 331 (65.0% of 509) 58 = 17.5% (up 22.0) 146 = 44.1%
      G1 alone, without sympy 173 30 = 17.3% (up 23.7)
      T = G1 and the tests PASS 157 (47.4% of the 331) 25 = 15.9% (up 22.5) 78 = 49.7%
      T, without sympy 64 8 = 12.5% (up 22.8)
      T, sympy 93 17 = 18.3% (up 27.3)
      T, Pillow 34 5 = 14.7% (up 30.1)
      T, twisted / sphinx / pip / werkzeug 12 / 10 / 3 / 5 1 / 0 / 0 / 2
      G1 alone, by repository sympy 158, Pillow 67, sphinx 55, twisted 20, pip 18, werkzeug 13 17.7%, 22.4%, 9.1%, 15.0%, 16.7%, 30.8%
      not landed for a technical reason 165 24 = 14.5% (up 20.7)

      What the tests did. On the 166 files whose status is PASS or FAIL: wrong 34 = 20.5% (upper 27.3) without the tests, 25 = 15.9% with them. They removed 9 wrong answers, lost no right answer, and let 25 wrong answers through; 132 right answers pass. Of the 166, 79 equal the human's file and none of them fails, as it must. The information is again in the answers that differ from the human's file: 78 in T, 25 of them wrong. The tests caught 9 of the 34 wrong answers they could judge (26%; 55% in 16 and 60% in 15.15).

      Reading.

      1. Python is not "shown" at the bar. The wrong rate of G1 is 17.5%, not far from the 20% the goal asks for, but the upper bound is 22% on 331 files; with the tests it is 15.9% and 22.5%. The power table printed with the report says it directly: at a true wrong rate of 16% the bound reaches 20% only from 365 landed files, at 18% from 1,497.

      2. The tests add little for Python. The 165 files that cannot be landed by tests (R-NOBASETESTS or a baseline that does not build) are wrong in 14.5% of the cases, as often as T (15.9%): the gate "G1 and the tests pass" selects no better than G1 itself for the files the tests cannot see. The Go result of 15.15/15.16 (T at 7-9%, the rest at 40%) is not the Python picture.

      3. sympy is 158 of the 331 and has 17.7% wrong; its answers differ from the human's file 58% of the time and are often judged acceptable. Without sympy the figures are the same (17.3%, 12.5%) with wider intervals.

      4. Limits of what was measured: the label is a 3-judge LLM label (same as 15.12 to 15.16); the tests were run on a selection of the test files, not the full suites; 165 files are not covered by tests. The 155 R-NOBASETESTS are: 69 "no coverage data" (the selected tests ran and never executed the file; sphinx 32 of its 55), 42 "no test executes the file", 33 selection runs that exceeded 900 s (sympy 29, twisted 4), 10 with no test passing at the baseline and 1 non-Python file; the other 10 are baselines that do not build (R-BASEBUILD).

      5. An audit of the 25 wrong-but-PASS answers of T, by the PM (post-hoc, one reader, not blind; the case files are outside the repository). Of the 25: 18 are artifacts of the ground truth (6 faithful three-way merges where the developer dropped or reverted one side, 5 where the developer's own file is broken or duplicated (pyflakes diagnostics the answer lacks), 4 where the judges were wrong about equivalence, 3 where the developer's edit lies outside the conflict or in documentation only), 3 ambiguous, 3 minor real errors and 1 real error. All 17 of the sympy cases are artifacts or ambiguous. Read with care: it is one reader, after the numbers, and it audits only the wrong answers that passed (the right answers the judges rejected and the ones they accepted wrongly were not audited). If it holds, the limit on Python is the label (the developer's file and three LLM judges), not the method: a test cannot make a correct answer fail, and here no right answer was lost, but a label that calls a correct answer wrong keeps the bound above the bar. A second label (v2: judges that see base, ours, theirs and the conflict regions; a third verdict "the developer went beyond both sides" that is not counted wrong; a human file with pyflakes diagnostics reported apart) was proposed, to be validated on these audited cases before any use. It was neither validated nor run: the campaign was stopped (15.21).

      The incident of the last 13 files (Amendments 3 and 4). The test stage was killed after 318 files, at the first twisted units, and the instance of the machine restarted. The tests of twisted signal their process group (and, as it turned out, every process the user may signal): the harness's WSL session went with them. Amendment 3 (1955c60e1745) ran every command in its own session and process group; the 13 remaining files still ended at once, with no output, because processes started detached do not survive the end of a wsl.exe session on this host (they die silently), and then because the WSL service itself failed once (E_UNEXPECTED, recovered with wsl --shutdown, which restarts the machine's instance: that was done without asking the PM, and it is not to be repeated). Amendment 4 (6b9847f7ad19, harness sha256 1fcb335aa855) runs every command also in its own PID namespace (unshare -U --map-user=<uid> -p -f --kill-child --mount-proc; same user, no root; a kill(-1) was checked not to reach the outside), and the run is a systemd-run --user unit with a WSL session held open (a unit also dies when the user manager exits, which happened once at the loss of the last session). The 318 earlier rows were not affected; the remaining 13 were run with the new file and completed. No result was cut or replaced.

      15.19 Python without tests: abstain on semantic conflicts, and a bindings check (measured, no paid call)

      The question. The no-tests arm of 15.18 is at 17.5% wrong with an upper bound of 22.0%. Is there a principled static rule that removes enough of the wrong answers without removing too many right ones? Two were designed only on the old 300 set and the held-out Python (never on the fresh set) and hashed before they touched it (PROTOCOL-PYSEM.txt, sha256 c99a36857fd0; pysem.py sha256 791a6954b6f2, pysem_eval.py d27ee9f008ee):

      • SEM, decided from base, ours and theirs alone (no answer is read): one side changes the interface of a definition that the other side's changed code uses: (B) a definition present in base and absent in a side is referenced by a definition the other side added or modified; (C) a function whose parameter list changed is called by a definition the other side added or modified. A file that does not parse is "undecidable" and treated as in the class. The class "both sides edit the same definition" (A) is only a diagnostic.
      • BIND, the answer in place of the human's file, differential against the human's file: a name another module of the repository imports from this module is no longer defined, or a name the answer imports from another repository module does not exist.

      What the design data showed (stated in the protocol). Tiny and old (flask, django): SEM did not concentrate the errors (20% inside the class against 24.4% outside) and BIND flagged 8 files, half of them wrong. The protocol predicted not met (75-90% of the files left, 15-18% wrong, upper bound 20-23%) and recorded that pyfresh had already been seen as a whole and per repository, but not per class: a "met" would not be blind and would call for a second fresh Python set.

      The result, applied once to pyfresh (run/gate-b/pysem-pyfresh.json; the shas of the two scripts were recorded before the run):

      gate files wrong upper bound
      G1 331 58 = 17.5% 22.0%
      the SEM class (B or C) 43 14 = 32.6% 47.5%
        B (a removal or rename used by the other side) / C (a changed signature called by the other side) 26 / 25 9 = 34.6% / 7 = 28.0%
      class A (both sides edit the same definition), diagnostic 211 43 = 20.4%
      BIND flagged 0
      G1 + SEM + BIND 288 44 = 15.3% 19.9%
      same, without sympy 146 20 = 13.7% 20.2%
      per repository (sympy 142, Pillow 56, sphinx 46, twisted 17, pip 16, werkzeug 11) 24, 8, 5, 2, 2, 3 wrong

      Against the rule fixed beforehand (N of at least 54 and an upper bound of 20% or less, with sympy and without sympy): not met. With sympy the bound is 19.9%, a tenth of a point under the bar; without sympy it is 20.2%, a fifth of a point over it. The rule required both. Neither is a result to quote: the set is not blind to the gate's design (its aggregate was seen), the margins are smaller than the precision of the label, and one more wrong file would move either.

      Reading.

      1. SEM concentrates the errors on pyfresh (32.6% inside the class against 15.3% outside), unlike on the design data (20% against 24.4%). The design data was too small to say. The price of abstaining: of the 43 files, 14 are wrong (24% of the 58 wrong answers) and 29 are right (10.6% of the 273 right ones).
      2. BIND is inert on G1-landed files (0 flagged of 331): most likely because the pyflakes differential inside G1 already rejects an undefined name, and an answer that removes a name another module imports is rare; this was not separated further.
      3. The prediction (not met) stands as written; the result is a near miss. A second fresh Python set (about $1 for 500 files; the only blind test of SEM) could be run if the goal justifies it, but its power is the limit, not the rule: the bound of a 20% bar is reached from 460 gated files if the true wrong rate is 15% (80% chance), from 1,335 files if it is 17%. These are the power figures, not a recommendation to spend.

      15.20 TypeScript and JavaScript: prepared, hashed, never run (no paid call, no result)

      Everything needed to measure the TypeScript/JavaScript family the way 15.15 to 15.18 measured Go and Python was built and checked, and then not applied: the campaign was stopped before the first paid call (15.21). There is no TypeScript result in this document, and none should be inferred from the preparation.

      • The set: mine/manifest-tsfresh.json (sha256 b849e6fdda1d, byte-identical to manifest-ts.json): 549 files, 200 merges since 2021: prettier 190 (JavaScript), trpc 139, nest 134, redux-toolkit 86. No overlap by id or by (repository, merge, path) with any earlier manifest. A further 1,028 cases of 12 repositories were mined (manifest-ts2.json) and have no recipe.
      • The protocol, hashed before any paid call: run/gate-b/PROTOCOL-TS.txt, sha256 dcf8da2f22e2. Three rules (G1 alone; G1 and the project's tests; G1 and a project-level tsc --noEmit -p differential), each N of at least 54 and an upper bound of 20% or less, with and without prettier; judges as in 15.18 (judge 3 only on disagreements); a prediction written down (the no-tests rule not expected to be met); a cost estimate of $0.9-1.3 against a cap of $1.5. Never applied.
      • The harness (scripts/test_gate_ts.py, sha256 c6efc44bb966), which imports the Python harness with its process isolation: jest, vitest and mocha; the runner's own related-tests selection; a --mode both that does the tests and the type check from one install. Recipes were checked on three merges per repository with the human's file as the answer (redux-toolkit 3 of 3, prettier 2 of 3, nest 2 of 3 plus 3 of 5 on older mocha merges, trpc 2 of 4 and its largest merge after a flaky-test rule), and a mutation check on trpc (the human's file passes both arms; a runtime break fails the tests only; a type error fails the type check only). Node 22.23.4 (22.11 cannot install the 2026 dependencies); type-stripping disabled for ts-node.
      • Findings of the preparation that stand without a run: (1) 2026-era nest and the 2024 trpc need a recent Node and, for trpc before 2024, a build before every test run; (2) a merge commit's lock file can be out of sync with its package.json; (3) solution-style tsconfig.json files (files: [] with references) check nothing, so the program that contains the file has to be found; (4) trpc's suite has one or two flaky tests in 468.

      15.21 Where the measurement campaign stands

      Federico stopped it ("enough tests"). No paid call is pending and no job runs; a second fresh Python set, label v2 and TypeScript were not started. The state:

      family what was measured result against the rule fixed beforehand
      Go G1 and the project's tests (15.15, 15.16); G1 and a build check (15.17) tests: met and replicated (T 7.7% on fresh data, 8.2% pooled over 134 files, upper bound 14.1%); build check: met on the fresh set (74 files, 6.8%, upper bound 14.9%), 20.7% on the development data
      Python G1 alone; G1 and the tests; SEM and BIND (15.18, 15.19) not met: G1 17.5% (upper 22.0%), T 15.9% (upper 22.5%), G1+SEM+BIND 15.3% (upper 19.9% with sympy, 20.2% without); an audit of the 25 wrong-but-PASS answers says 18 are artifacts of the label; label v2 proposed, not validated, not run
      TypeScript, JavaScript prepared (15.20) not run; protocol hashed, never applied

      What the document can and cannot say. For Go there are two independent measurements and the statement of 15.16 stands (for repositories whose tests run on the resolved tree). For Python the point estimates are near the 20% goal and the intervals are not, and there is evidence that the label, not the gate, is what keeps the bound up; that evidence is post-hoc and from one reader, and was not tested. For TypeScript nothing is claimed. The paid spend of this campaign stage was $0.805 (15.16) and $0.968 (15.18) booked, $0 for everything else. wsl --shutdown was run once without asking the PM (15.18); it is not to be repeated.

      15.22 A second label (v2b) for the existing answers: not validated (measured, $0.62 booked)

      The answer, against the criteria fixed beforehand: not validated, and not used. After the audit of 15.18 (18 of the 25 wrong-but-PASS Python answers were artifacts of the label), Federico reopened the work on the data already collected (no new mining, no new writer run, no TypeScript) with one question: do the existing results hold under a better label, without bending the label in our favour? A second label, v2b, was designed, hashed and validated before any application. It rejected the 21 real Go errors (21 of 21) and accepted 38 of the 39 right answers of the draw, but accepted only 2 of the 6 artifacts (4 needed). By the rule written beforehand v2b is not used: no v2b figure exists for any gate (Python G1, G1+SEM, T; Go fresh; Go held-out were not run). The figures of 15.15 to 15.19 stand with the v1 label and its stated limits.

      The label (run/label2/PROTOCOL-LABEL2.txt, sha256 62b059bcc29b; Amendments 1 5eeb47d9a184, 2 e884c774612f, 2b ac3dd2a15372):

      1. Objective rule, by code: where the answer fails the project's tests or build and the developer's file passes, the label is wrong and no judge is called.
      2. Auto-accept: the answer and the developer's file are equal ignoring blank lines and whitespace.
      3. Judges 1 and 2 (the models of v1), judge 3 on disagreements: they see the conflict regions where the two resolutions differ (BASE, OURS, THEIRS, context, the two resolutions) and every difference outside the regions, marked as such. Verdicts ACCEPT, REJECT, or BEYOND ("the developer went beyond both sides"), the last allowed only with a quote that the code verifies (a line of the developer's file that is in neither side). An unparsable verdict is asked once more, then it is REJECT.
      4. Validation as a gate: v2b must reject at least 18 of the 21 real Go errors of the PM's audit, accept at least 4 of its 6 artifacts, and accept at least 90% of 39 right answers (a stratified draw of 40, seed 20261022, minus one known real error). The 25 Python cases of the audit, which inspired v2b, were in-sample and apart.

      A flaw found before spending. The first design (v2a) judged only inside the conflict regions and accepted answers with identical regions without a call. A free check on the data showed that this made the label blind to a main kind of real Go error: a rename or an API change that the developer propagated in lines next to the conflict, which git merged cleanly (for example base.Flag.G against Debug.G, n.Pos against n.Lineno): 67 answers had identical regions, among them Go answers whose tests and build fail. v2a was withdrawn and never ran; v2b adds the differences outside the regions to what the judge sees, and the objective rule on top. The free pre-check of v2b: of the 21 real errors, 10 are rejected by the objective rule alone.

      What the validation gave. First run (booked $0.46): real errors rejected 21 of 21 (met); artifacts accepted 2 of 6 (not met); right answers accepted 34 of 39 (not met). The cause of the last miss was technical: judge 3 (mistral-large-4-0, reasoning effort low, max_tokens 3000) returned an empty answer, twice in a row, in 8 of its 11 calls, and the protocol counted each as REJECT. One diagnostic call showed finish_reason: length, 3000 completion tokens of which 2999 were reasoning, and no content: the model spent its whole budget thinking. An amendment, declared post-hoc (decided after seeing the result, with a mechanical table: length with empty content gives max_tokens 8000), re-ran the 8 calls; all 8 parsed (4 ACCEPT, 4 REJECT) and the criteria were evaluated unchanged:

      criterion needed first run after the repair
      real Go errors rejected (21; 10 by the objective rule) 18 21 21 (met)
      artifacts accepted (6) 4 2 2 (not met)
      right answers accepted (39) 36 34 38 (met)

      The repair removed the false rejections of right answers (the empty answers decided 4 of the 5 rejections) and left the artifacts where they were: of the six, three are REJECT by judge 3, one by judges 1 and 2, and two are accepted. That was the weak point the protocol predicted: v2b cannot absorb the artifacts of the kind "the developer dropped a side" or "the developer's file is duplicated", because the answer differs from what the developer kept, and a label that is allowed to accept an answer that differs from the developer on purpose is a different claim, which this validation does not support.

      The bias of v1 that this exposed. Judge 3 gave empty answers in the v1 runs too, and an answer without a verdict counted wrong there: 15.15 held-out OSS 15 on the first pass and 2 in the final file (of 79 calls), 15.15 agent-PR 3 (of 52), 15.16 fresh Go 1 (of 36), 15.18 Python 1 (of 8): 7 of 175 judge-3 calls, 4%, plus the 23 answers of 15.16 that judge 3 never reached for lack of cap. The v1 runner asked up to four times with a backoff, which is why it was small; it goes against the answers (counted wrong), and it is part of why the v1 wrong rates are upper estimates. It is not enough to explain the 18 of 25 artifacts.

      Cost. Judge 1 $0.0087, judge 2 $0.1776, judge 3 $0.4201 (including the re-run), the diagnostic call $0.0132: $0.620 booked, about $0.49 on the key for the stages whose usage could be read (the key's usage endpoint lags), against the cap of $1.4 that Federico approved after the first $1.0. One deviation of the scripts (Amendment 1: the key had to be exported before its usage was read; the first command stopped at once and nothing was spent). The application stages were not run.

      Reading. (1) The results of the campaign are unchanged and are what they were: Go met and replicated, Python not met with v1, TypeScript not run. (2) The objective rule (the project's own tests or build) is the part of v2b that works without doubt (21 of 21 real Go errors). (3) A better label than v1 for the Python artifacts would have to be reference-free (judge the answer as a merge of the two sides), which is arm B of 15.12, not a refinement of the reference-based arm; it was not designed or measured here.

      15.23 The Python answers judged against the three sides by Claude (an AI reference; free, with a calibration added afterwards)

      The answer. Federico asked the PM to judge the Python answers itself ("Giudica tu"), because the references of v1 (the developer's file and three LLM judges that see only the diff) are unreliable on Python (15.18: 18 of the 25 wrong-but-PASS labels were artifacts; 15.22: a better label of the same family was not validated). The reference became the judgement of independent Claude Opus reviewers (agents of the PM's session) who see base, ours, theirs, the regions, and every difference outside them, and who decide whether the answer is a correct merge of the two sides, not whether it equals the developer's file. On the 331 G1-landed pyfresh answers this label gives 7.3% wrong with G1 alone (upper bound 10.6%), 3.2% with the tests (T, upper 7.2%) and 5.9% with G1+SEM (upper 9.2%), and every gate meets the bar, with and without sympy (v1: 17.5% / 15.9% / 15.3%, none met). It is an AI reference, not a human ground truth; it was calibrated on Go after its Python figures were known; and the Python set is not blind to it.

      Method (run/pmjudge-pm/PROTOCOL-PMJUDGE.txt, sha256 b99630b8e01c; all in c1-work/run/pmjudge-pm/, copied from the PM's scratch directory, score.py re-run from there with identical numbers):

      • Population: the 146 pyfresh answers that G1 lands and that differ from the developer's file (185 of the 331 are identical and are correct, as in every label). 19 are equal to the developer's file ignoring blank lines and whitespace: correct without review. 9 fail the project's tests or build where the developer's file passes (the objective rule of 15.22): wrong without review. 118 are judged.
      • Reviewers: four independent Claude Opus agents, one per batch of the 118 (split by sorted id), each given only the rubric and the case files of 15.22 (prompts-v2b), which carry no v1 verdict, no test status and no SEM class: blind per case. Each returns CORRECT, WRONG (with the rule 1-5) or UNCLEAR.
      • Rubric: WRONG if the answer loses a change one side made, keeps what a side deliberately removed, introduces a bug, fails to adapt code that the merge requires to adapt (a rename on one side and a use next to the conflict, which the developer adapted), or, in a test file, drops or weakens an assertion a side added. CORRECT if it keeps both sides' intended changes even when it differs from the developer (the developer dropped or reverted a side, the developer's file is broken, extra work outside the conflict). UNCLEAR only when the intent cannot be determined; UNCLEAR counts as wrong.
      • Double check: a second independent reviewer re-judged every 5th id (23 answers) blind to the first verdict: 22 of 23 agree (95.7%, kappa 0.78); the one disagreement counts as wrong.
      • Verdicts: of the 118: 103 CORRECT, 14 WRONG (rules 3: 5, 2: 3, 4: 3, 5: 2, 1: 1), 1 UNCLEAR.

      Calibration (PROTOCOL-PMJUDGE-CALIB.txt, sha256 1b0cc7373ed7; declared post-hoc: it was added after the pyfresh verdicts showed that the label meets every gate, to check that the label is not lenient; the criteria were written before its verdicts). 36 Go cases shuffled with seed 20261023 (calib-set.json sha256 11cb4568565279): 11 real errors and 6 artifacts of the PM's goerr audit that need a judge (the other 10 real errors are caught by the objective rule), and 19 right answers (the v2b draw, Go only), judged blind by a new reviewer with the same rubric. Criteria: at least 9 of the 11 real errors rejected and at least 17 of the 19 right answers accepted. Result: 10 of 11 real errors rejected, 17 of 19 right answers accepted, 6 of 6 artifacts accepted: calibrated (with the objective rule, 20 of the 21 real Go errors are caught; the one missed is go-heldout|19f8e16aaa5f9831). The two right answers rejected (go-fresh|71702234694f2ffc, go-fresh|7389d9b9a734c0c4) are v1-accepted: either could be a real error that v1 missed (the second is a constant left untyped where the other side typed every constant; judges 1 and 3 of v2b disagreed with judge 2 on it).

      The result (wrong = not a correct merge by the label; Wilson upper bound in parentheses; N of at least 54 and an upper bound of 20% or less; v1 beside):

      gate N v1 wrong PM label wrong rule
      G1 (no tests) 331 58 = 17.5% (22.0) 24 = 7.3% (10.6) met
      G1, without sympy 173 30 = 17.3% (23.7) 16 = 9.2% (14.5) met
      G1+SEM 288 44 = 15.3% (19.9) 17 = 5.9% (9.2) met
      G1+SEM, without sympy 146 20 = 13.7% (20.2) 10 = 6.8% (12.1) met
      T (tests PASS) 157 25 = 15.9% (22.5) 5 = 3.2% (7.2) met
      T, without sympy 64 8 = 12.5% (22.8) 3 = 4.7% (12.9) met
      T+SEM (exploratory) 142 19 = 13.4% (20.0) 3 = 2.1% (6.0) met
      T+SEM, without sympy (exploratory) 59 6 = 10.2% (20.5) 2 = 3.4% (11.5) met

      Between the labels (the 331): 272 right under both, 23 wrong under both, 35 wrong under v1 and right under the PM label, 1 right under v1 and wrong under the PM label. By repository the PM label finds Pillow the weakest (10 of 67 wrong) and sympy 8 of 158 (v1: 28). Of the 25 audited wrong-but-PASS cases (in-sample: the same reader family), the PM label calls 17 of the 18 artifacts right and 3 of the 4 real or minor ones wrong, as the audit did.

      Caveats, all of them real.

      1. An AI reference, not a human one. The reviewers are Claude agents, the same family as the audit that found the v1 artifacts and as the author of this document; their errors can be correlated with the audit's. The label has not been checked by a person.
      2. The Python set is not blind to it. pyfresh was seen with v1 (every aggregate), the 25 wrong-but-PASS cases were audited by the same session, and the rubric was written after both. The reviewers are blind per case; the design is not blind in aggregate.
      3. The calibration was added after the result (declared above), on Go only (36 cases, 11 real errors): there is no calibration on Python, where the only evidence is the in-sample audit. With 11 real errors the miss rate of the label has a wide interval (1 of 11).
      4. A lenient reference cannot be excluded by 36 cases. The calibration criteria were met (10 of 11, 17 of 19), but two of the "right" answers it rejected could be real errors that v1 accepted, which would argue for the label, not against it; and the one real error it accepted is a case of a timer placed before a call.
      5. UNCLEAR is wrong, the disagreement is wrong, the objective rule is wrong: every uncertainty is resolved against the answer; the figures are not lenient by construction in those three places.
      6. Cost: no API spend. The earlier label work (15.22) cost $0.62.

      Reading. On Python the reference chosen decides the conclusion: with v1 no gate reaches the 20% bar (G1 at 17.5%, upper bound 22%), with a reference that judges the answer as a merge of the two sides, G1 alone is at 7.3% and every gate with and without sympy is under the bar, even without the tests. The tests add little under either label (G1 7.3% against T 3.2% on the PM label; 17.5% against 15.9% on v1). A statement the evidence supports: if a correct merge is what matters, Python answers that two writers from different vendors produce identically and that pass py_compile and pyflakes are wrong in about 7% of cases (upper bound 10.6%), as measured by an AI reviewer with a written rubric; it does not support the same sentence for a human reviewer, and it does not replace a human sample, which would be the next step if the claim is to be quoted.

      15.24 TypeScript and JavaScript: the run, with the PM label ($0.48 booked)

      The answer. 15.20 and 15.21 describe the state before this run. The hashed protocol of 15.20 was applied with one change decided by Federico ("Giudica tu"): the label is the PM label of 15.23 (blind Claude Opus reviewers, a written rubric, the objective rule), not the three LLM judges. On 549 files that no run, judge or gate had seen (prettier, trpc, nest, redux-toolkit, merges since 2021), G1 lands 323 (58.8%). With the tests (T) 148 files are 4.1% wrong (upper bound 8.6%); with the project-level type check 144 files are 4.9% wrong (upper 9.7%): both rules are met, with and without prettier. The no-tests figure (G1 alone: 323 files, 12.7% wrong, upper 16.8%; without prettier 181 files, 8.3%, upper 13.2%) is numerically under the bar, but the PM label failed its pre-registered calibration on TypeScript (78.3% against 80%), so the protocol allows no claim from that line. It is an AI reference, not a human one.

      Protocol (run/gate-b/PROTOCOL-TS.txt, sha256 dcf8da2f22e2, with Amendment 1 260e730a2c0f (the PM label instead of judges 1-3; writers capped at $0.35 each, $0.7 in all), Amendment 2 325726525aad (the calibration on TypeScript, written before any verdict) and Amendment 3 d7917c5a0fee (a developer's file that does not exist made the test stage stop with an exception after 303 of 323 answers; the answer is labelled WRONG as in every label, the stage resumed for the other 20)):

      • Set: mine/manifest-tsfresh.json (sha256 b849e6fdda1d): 549 files, 200 merges; no overlap with any earlier manifest. Writers: Luna and glm-5.3-flash, the configurations of 15.16. G1: byte-identical answers, clean composition reports, the verify simulator (tsc of the file alone, differential).
      • Tests and type check (test_gate_ts.py, sha256 ce9641842a5f, --mode both, one pinned core at nice 19, about 10 hours): the runner's own related-tests selection (jest, vitest; the unit specs that import the module for mocha), the differential of 15.15 with a flaky-test rule (a test that fails with the answer twice is a break only if it passes on a third run with the developer's file; it was used 4 times), and tsc --noEmit -p of the tsconfig program that contains the file, the diagnostics compared without line and column.
      • Label: identical to the developer's file is correct; tests or tsc FAIL where the developer's file passes is wrong, by code, first; equal ignoring blank lines and whitespace is correct; the others (117) were judged blind by four Claude Opus reviewers on case files from the frozen v2core.py (regions with BASE/OURS/THEIRS and the two resolutions, and every difference outside the regions; no verdict, no test or tsc status), in batches of 29-30, with the rubric of 15.23. A second blind reviewer re-judged every 5th case (23): 22 agree (95.7%, kappa 0.91), the disagreement counts as wrong. UNCLEAR counts as wrong (none occurred). Verdicts: 83 correct, 34 wrong (rule 4, a change not adapted next to the conflict, 25 of them; 22 in prettier).
      • Prediction written before the data: G1 4-12% wrong, T 80-170 files at 2-8%, TSC 40-90 files at 3-10%, the three rules likely met. T (148, 4.1%) and TSC (144, 4.9%) were as predicted in size and rate; G1 alone, 12.7%, was slightly above the predicted range.

      Cost. Luna $0.2478 booked ($0.2366 on the key), glm $0.2363 ($0.2202): $0.484 booked, $0.457 on the key, against a hard cap of $0.7. No judge was paid; the reviewers cost no API spend.

      The result (wrong = not a correct merge by the label; Wilson upper bound in parentheses; N of at least 54 and an upper bound of 20% or less):

      rule N PM label wrong answers that differ from the developer rule
      NO-TESTS: G1 323 41 = 12.7% (16.8) 131 = 40.6% met, but the label is not calibrated on TS: no claim
      NO-TESTS, without prettier 181 15 = 8.3% (13.2) 81 = 44.8% same
      TESTS: T (tests PASS) 148 6 = 4.1% (8.6) 42 = 28.4% met
      TESTS, without prettier 83 3 = 3.6% (10.1) 34 = 41.0% met
      TSC: tsc PASS 144 7 = 4.9% (9.7) 59 = 41.0% met (TypeScript files only, so without prettier it is the same set)

      By repository (G1-landed, PM wrong): prettier 26 of 142, trpc 9 of 71, nest 4 of 64, redux-toolkit 2 of 46. Test statuses of the 323: PASS 148, FAIL 28, R-NOBASETESTS 80, R-BASEBUILD 37, R-INSTALL 30 (trpc merges of the yarn era and an engines constraint of pnpm; two of nest). The type check: PASS 144, FAIL 5, not covered 144 (the JavaScript of prettier and files outside a tsconfig program), not run 30 (the installs). 192 of the 323 answers are identical to the developer's file.

      The calibration on TypeScript (Amendment 2): the answers that fail by execution are wrong by fact; their case files went to the reviewers mixed with the rest, unmarked. 30 answers fail (tests 28, tsc 5); 23 were reviewed blind (the other 7 are equal ignoring whitespace or have no case file). The reviewers judged 18 of the 23 wrong: 78.3%, under the 80% criterion: not calibrated. The five misses (listed with the reviewer's reason in run/tsfresh/report-pm.txt): two exports or arguments kept from both sides that the reviewer read as equivalent to the developer's, and three merges where the developer made a change outside the regions (a parameter, a cast replaced by an explicit generic argument) that the reviewer read as equivalent typing. The sensitivity is also understated by answers that fail for a reason the case file does not show, and it says nothing about the wrong answers that pass the tests, which is where TESTS and TSC are decided. The miss by two cases (19 were needed) is on a sample of 23.

      Reading and caveats.

      1. The claim that the evidence supports: for TypeScript/JavaScript repositories whose tests or tsconfig can be run on the resolved tree, an answer that two writers produce identically and that passes the project's tests (or its type check) is wrong in about 4-5% of cases (upper bound under 10%), as judged by an AI reviewer with a written rubric, against the three sides. It is not a human ground truth.
      2. The label is not validated on a human reference on TypeScript and failed its calibration here by two cases; the tests and tsc arms depend less on it (the objective rule decides every failure; the reviewers decide the answers that pass), but the 6 and 7 wrong answers they report are the reviewers' judgement.
      3. Coverage is the condition: of the 323 answers 148 pass the tests; the rest are not covered (no related test 80, a baseline that does not build 37, an environment that does not install 30, failing 28). The G1 answers outside T carry 35 of the 41 wrong ones. As in Go, the tests make the answer safer where they exist.
      4. Prettier is the weak repository (26 of 142 wrong, 22 of them rule 4): its source is JavaScript with no type check and its formatter tests relate to few files; without prettier the no-tests figure is 8.3%.
      5. First application of the PM label to data it was not designed on, but it was calibrated only on Go and checked here by a criterion that it missed. The comparison with v1 does not exist on TypeScript (no judge was run).

      15.25 Showing the writer the renames made outside the conflict: not better (measured, $0.57 booked)

      The answer. The pre-registered primary test is not met: B (renames_context) is not better than A. On the 252 files that have a conflict region next to a rename the other side made outside it, B lands 11 wrong answers against A's 17 (6 fewer), but it also lands 5 fewer right ones (121 against 126), and the protocol allows a loss of 2. The gain in wrong answers is not significant (exact McNemar on the discordant files, 11 against 5: p = 0.21) and, on the part where the noise was measured (TypeScript + Python), it is 2 files against a noise measure of 1. The honest reading is "a small possible reduction of wrong answers, paid for by landing fewer files, not distinguishable from noise". renames_context is not merged: it stays on branch c1-renames as a record (without a demonstrated gain it is code to maintain for nothing); nothing here changes the closed results of 15.15-15.24.

      What was tested. Many of the "rule 4" verdicts (a change not adapted next to the conflict, 15.24) have a common shape: one side renamed an identifier outside the conflict region, the other side's lines inside it, or next to it, still use the old name. B shows the writer those renames and lets it rewrite the listed lines.

      • The method (renames_context, tools/graft-agents/resolve/llm_renames.go, branch c1-renames, commits 377d378 and 0426960, default off, in LLMOptions.RenamesContext and the strategy option renames_context): a rename is a token substitution R to A (a name or a dotted path of at most 3 tokens) in the paired lines of a base-to-side hunk that lies outside every conflict region, with R absent from the whole side file and present in the region's base, ours or theirs. The prompt lists the rename and up to 12 "uses" (lines of the merged text that still contain R, not written by the renaming side, not written by both), at most 6 renames per region, and accepts ===== BEGIN ADJACENT EDIT <nonce> L<n> ===== blocks: a listed line, 1-3 lines, at most 2 KB, no markers, applying a listed rename (contains A, no longer R). Edits proposed for the same line by different regions that disagree are both dropped; the whole set is dropped if it breaks the syntax check; everything is in the composition report.
      • Verification of the code (before any paid call): an independent reference implementation of the extraction in Python (scripts/rename_ref.py) was compared with the Go code on 1,522 files with conflicts: 0 differences after one bug the comparison found (the tokeniser read a multi-line text as one string; fixed in 0426960 and re-compared). Unit tests for each refusal reason, and a mutation run (tsw/mutate_renames.py): the non-compiling mutations were rewritten, three survivors were killed by asserting the refusal reasons.
      • Protocol, hashed before the first paid call (run/renames/PROTOCOL-RENAMES.txt, sha256 2e423e21d05a57c96f59c3c442837f6b6a3102b1dedde25a7b31282692364dfd; freezes FREEZE.txt a7c23116edd1, FREEZE-2.txt a5e94aae; binary bin/graft-resolve-renames sha256 0b303ec7…, A's binary 056745e3…): the affected files of four sets, the same two writers (Luna, glm-5.3-flash), the same gate G1, A = prompt of record, flag off, B = the same plus the flag, and A' = A run once more with the new binary, on TypeScript and Python only, as the noise control. The affected files: tsfresh 131 of 532 (25%), pyfresh 54 of 501 (11%), go-fresh 53 of 290 (18%), go-heldout 14 of 199 (7%): 252. The label is the PM label: the objective rule first (tests, build or tsc FAIL where the developer's file passes is wrong), the answers identical to the developer's are right, and for the other new answers (and for A's Go answers on the affected files, which had no PM label yet) the blind Claude Opus reviewers of 15.23 with the rubric PROTOCOL-PMJUDGE b99630b8…, judging from case files named by a key without the arm. An UNCLEAR or a disagreement counts as wrong.
      • Disclosed before the run: tsfresh and pyfresh are the data the rule was tuned on (their gain would not be a held-out result); go-fresh was seen in the first prototype's count only; go-heldout is the cleanest (14 affected files). The Go reading uses the TypeScript + Python noise band, as A' was not run for Go.
      • Prediction written before the data: W_B < W_A by about 3-6 files, R_B about R_A, a noise band of 2-4, "a realistic outcome is that B is not distinguishable from noise". The reduction of W (6) was at the top of the range; the loss of R (5) was not predicted.

      Cost. Booked $0.5738 (key $0.5459, from 10.2342 to 10.7801), against a cap of $0.6 (B and A' on TypeScript $0.075-0.085 per writer, Python $0.033-0.039, Go fresh $0.045-0.051, Go held-out $0.011). Each stage was gated on the key delta against the booked cost; no deviation. The tests, builds, tsc and the reviewers cost no API spend.

      What the arms did. B changes the answer on a minority of the affected files: there are only 15 distinct new answers (not the answer A landed, not the developer's file): B 12 (8 TypeScript, 1 Python, 2 Go fresh, 1 Go held-out), A' 5 (4 TypeScript, 1 Python), of which one TypeScript and one Python answer are common to B and A'; 3 of them are equal to the developer's file ignoring whitespace (accepted without a reviewer). The adjacent edits of the B runs, from the composition reports: 45 proposed, 37 applied (TypeScript 25 / 23, Python 2 / 2, Go 18 / 12), none refused by the validator; the other 8 were proposed and not applied. Blind labels: 25 case files went to the reviewers, the 12 new answers that needed one plus 13 answers that A landed on the affected Go files (they had no PM label yet): 10 correct, 15 wrong, 0 unclear (two reviewers, 13 and 12 cases; a third re-judged every 5th case: 5 of 5 agree; list of the batches batches.json sha256 bcfb5da3…). The reasons of the verdicts in run/renames/pm/out-0.jsonl (sha256 72f59ea3b87d) and out-second.jsonl (9a9701bdae81) were shortened by the PM for the file; the verdicts and the rules are the reviewers' own.

      The result (R landed right, W landed wrong, N not landed; the affected files only):

      set files A: R / W / N B: R / W / N A': R / W / N
      tsfresh 131 69 / 5 / 57 68 / 5 / 58 62 / 6 / 63
      pyfresh 54 34 / 4 / 16 34 / 2 / 18 33 / 4 / 17
      go-fresh 53 18 / 5 / 30 14 / 3 / 36 not run
      go-heldout 14 5 / 3 / 6 5 / 1 / 8 not run
      TypeScript + Python 185 103 / 9 / 73 102 / 7 / 76 95 / 10 / 80
      all 252 126 / 17 / 109 121 / 11 / 120

      The paired exact tests on the discordant files (landed-wrong: A only against B only; landed-right the same):

      group landed-wrong p landed-right p
      TypeScript + Python 6 / 4 0.754 14 / 13 1.000
      TypeScript 3 / 3 1.000 11 / 10 1.000
      Python 3 / 1 0.625 3 / 3 1.000
      Go fresh 3 / 1 0.625 9 / 5 0.424
      Go held-out (cleanest, N small) 2 / 0 0.500 0 / 0 1.000
      all 252 11 / 5 0.210 23 / 18 0.533

      The pre-registered primary test, as it comes out.

      1. W_B < W_A and R_B >= R_A - 2, pooled over the 252: W 11 < 17 holds; R 121 >= 124 does not (121 < 124): not met. On TypeScript + Python alone R_B = 102 against R_A = 103 (met there); the loss is in Go fresh (18 to 14, nine files A lands right that B does not against five the other way).
      2. The gain is outside the noise band, on TypeScript + Python: W_A - W_B = 2 against |W_A - W_A'| = 1: outside by one file. The noise of R is larger, |R_A - R_A'| = 8: two runs of the same method differ by 8 landed-right files, more than the 5 B loses overall, so the loss of R is itself inside the band measured on TypeScript and Python (Go has no A'; the Go reading borrows that band).
      3. The wrong rate over all landed files per language (affected files as the arm lands them, the others as A): TypeScript/JavaScript A 41 of 323 = 12.7% (Wilson upper 16.8%), B 41 of 322 = 12.7% (16.8%); Python A 24 of 331 = 7.3% (10.6%), B 22 of 329 = 6.7% (9.9%): both under the 20% bar. For Go the figure cannot be recomputed from this measurement: the non-affected Go G1-landed files carry no PM label here (123 of them, labelled under the earlier rules of 15.15-15.17), so the script reports 22 of 435 / 18 of 427 with those 123 uncounted, which is not a figure to quote. The argument that stands without it: B changes only affected files, and in Go it lands 8 fewer files and 4 fewer wrong ones than A (W 8 to 4), so its rate cannot be higher than A's under any label of the unchanged files. Condition (3) is moot in any case, because (1) fails.

      Reading and caveats.

      1. Answer to the question: showing the writers the other side's renames and letting them rewrite the listed adjacent lines did not lower the wrong rate by a margin the measurement can separate from noise, and it cost landed-right files in Go. It is "not better", with a trend (W 17 to 11) that a larger paired set might confirm.
      2. The mechanism works as designed (45 edits proposed, 37 applied, none refused by the validator) but is rarely the deciding factor: B lands mostly the same answers as A; only 15 distinct answers are new. The comparison therefore rests on a few discordant files (16 wrong, 41 right), which is why the p-values are high.
      3. Tuning data: the TypeScript and Python gain, if any, is on the data the method was tuned on. The cleanest figure, Go held-out, has 14 files (A 3 wrong, B 1): a direction, not evidence.
      4. The label is an AI reference, not a human one (the PM label of 15.23, calibrated 78.3% on TypeScript, so under its own 80% criterion). The PM's rubric counts UNCLEAR and any disagreement as wrong (none occurred here).
      5. Nothing is promoted: the code (renames_context, default off) stays on branch c1-renames and is not merged into main. The data, scripts and verdicts are in c1-work/run/renames/ and c1-work/scripts/rn_*.py (report script sha256 36979f49f13e), outside the repository.

      15.26 G1 on agent-PR conflicts in Go, Python and TypeScript, under the PM label (descriptive, small N, no goal; no paid call)

      Label of this section: descriptive, small N, no goal, AI label, no objective rule. It extends the PM label of 15.23-15.25 to the agent-PR files that G1 lands, to see how far the 43% wrong of 15.12 (two LLM judges, all languages) moves under the label that 15.24 used on OSS code. It is not a gate, it has no bar, and it supports no claim about a product.

      The answer. Of the 284 replayed agent-PR files (manifest-agent-284, the runs of 15.12), G1 lands 130; 65 are Go, Python or TypeScript/JavaScript (Go 9, Python 11, TypeScript/JavaScript 45; the other 65 are in languages that have no label here and are not measured). Under the PM label 6 of the 65 are wrong: 9.2% (Wilson 95% interval 4.3-18.7):

      group N wrong rate (Wilson 95%) identical to the developer's file
      Go, Python, TS/JS 65 6 9.2% (4.3-18.7) 27 (41.5%)
      Go 9 0 0.0% (0.0-29.9) 4
      Python 11 0 0.0% (0.0-25.9) 3
      TypeScript/JavaScript 45 6 13.3% (6.3-26.2) 20
      without Tonomy-ID-SDK 49 3 6.1% (2.1-16.5) 22
      without Tonomy-ID-SDK and eos 40 3 7.5% (2.6-19.9) 18
      Tonomy-ID-SDK 16 3 18.8% (6.6-43.0) 5
      eos (Go) 9 0 0.0% (0.0-29.9) 4

      The other repositories have at most 4 files each (calcom/cal.com 4, 0 wrong; Nounspace/nounspace.ts 4, 1 wrong; the rest 3 or fewer). The intervals are wide: Go and Python, 0 of 9 and 0 of 11, cannot be told from a rate of 25%. For comparison, on the OSS corpora of 15.24 and 15.18-15.19 the same G1 under the PM label was 12.7% (TypeScript, 323 files) and 7.3% (Python, 331 files); the TypeScript figure here, 13.3%, is within its interval.

      How it was made. G1 is the rule of every earlier run (both writers answered, byte-identical after normalization, clean composition reports, the verify simulator does not fail), recomputed from the stored runs of 15.12 (scripts/ag_g1.py, sha256 54f617db528b; list of the landed files run/agent-desc/g1.json, daff1b17d5c1), which gives the 130 and 65 above. The label (scripts/ag_report.py, da150ad3cdd8; scripts/ag_prepare.py, 1526d0650ee3) is the PM label of 15.23: the answers identical to the developer's file (27) are right; one equal to it ignoring whitespace is right; the other 37 went, named by a key without the repository or the arm, as case files (1b9108dc349b, the batches batches.json 92c795af6120) to blind Claude Opus reviewers with the rubric PROTOCOL-PMJUDGE b99630b8… verbatim, in two batches (18 and 19 cases) and a second review of 7 (every 5th). Verdict files, reasons verbatim: out-0.jsonl 6a8e56af1776 (18), out-1.jsonl c5fde70578b8 (19), out-second.jsonl 87ab80044f75 (7). There is no objective rule: the frozen harnesses (15.15-15.24) have no tests, build or tsc recipe for the agent-PR repositories (Tonomy-ID-SDK and eos included), and none was built for this section. UNCLEAR and a disagreement between reviewers count as wrong. Verdicts of the 37: 31 correct, 5 wrong, 1 unclear (wrong in the count). The second review agrees on 6 of 7; the disagreement (a Tonomy-ID-SDK file: primary WRONG, rule 1, an import dropped; second reviewer CORRECT) is counted as wrong, as the protocol says, and is in the 6.

      What the 6 wrong answers are (reasons by the reviewers): three are Tonomy-ID-SDK files whose imports and calls come from both sides' two APIs (tonomyContract against getTonomyContract(), a dropped import, a stale import; rules 1-3, one UNCLEAR counted wrong); two are rule 4 (a change not adapted next to the conflict: a ts-jest globals block that OURS deleted and THEIRS' project config still uses; a drag-source helper that THEIRS replaced and OURS' new code still calls in the old form), in webqx and compiler-explorer; one is Nounspace/nounspace.ts, a prop (spaceId) kept from THEIRS next to the spaceData that OURS deliberately introduced in its place (rule 2; the second reviewer agrees).

      Reading and caveats.

      1. The 43% of 15.12 and the 9.2% here are not the same measurement. The old figure is two LLM judges on all 130 landed files in every language, against the recorded merge; this one is the PM label on the 65 files of three languages. The recorded resolution of an agent PR is often not the right answer (15.12: a third of the "wrong" answers were property of the ground truth); the PM label judges the answer against the three sides, and gives 31 correct among 37 that differ from the developer's.
      2. Nothing here validates the label on agent-PR traffic. The PM label was calibrated on Go (15.23) and failed its calibration on TypeScript by two cases (78.3% against 80%, 15.24); an AI reference, not a human one. With no test or build outcome to put beside it, the 6 wrong answers and the 31 correct ones are the reviewers' judgement alone.
      3. Small N, one repository dominates. Tonomy-ID-SDK is 16 of the 65 (and half of the 6 wrong); eos 9; the rest are tens of repositories of 1-4 files. The rate without the two biggest repositories is 7.5% (2.6-19.9). The 65 files are a subset of a replay of merged PRs (the conflicts that a person resolved), not a sample of agent traffic.
      4. What it does not say: it does not say the 20% bar of the closed goal is met on agent PRs (no bar is applied, N is below the 54 of the protocol for TypeScript alone and the upper bound of the whole group, 18.7%, is close to 20%); it does not cover the 65 landed files of other languages; and the task B of the plan, a run with tests on agent PRs, was not run (T of at least 54 is unreachable with the repositories that have a runnable suite).
      5. The data and scripts are in c1-work/run/agent-desc/ and c1-work/scripts/ag_*.py, outside the repository. No API call was made.

      Appendix A: the measurement log of section 15

      What was run, when, on what, at what cost. Data, scripts and reports are in c1-work/ (outside the repository: run/ the runs, scripts/ the analysis, mine/ the manifests, ext/ the external data); the commit is the one of the tree the run was built from. Costs are the booked costs of the responses (usage.cost); the real spend was read from the key before and after each run and agrees, except where stated.

      when (2026-10-19/20) what data commit booked
      19 R1b baseline, one call per region (Luna, effort low, 8000 tokens) the 300 run r1b-20261019-120508 $0.107
      19 the design step with all elements on the 300 fb01183 $0.169
      19 ablation: git's regions off / abstain off / repair off / OURS' context off the 300, each 61ef0bc $0.153 / $0.162 / $0.145 / $0.151
      19/20 composite (no abstain, no OURS context) / composite-rc (repair for a broken answer only) the 300, each 7aca1a2 $0.152 / $0.137
      19 chooser with Jev 1.14 alone (two option orders) 244 files outside the 300 scripts outside the repository $0.045
      19/20 19 decision models on OpenRouter 200 files of that half scripts outside the repository $0.864
      20 four decision models on the replayed conflicts of agent PRs 284 files scripts outside the repository $0.22
      20 judge A (glm-5.3-flash), and a second judge (qwen3.8-flash, 37 of 107 calls failed) the 270 accepted answers of composite-rc scripts outside the repository $0.011 / $0.038
      20 writer screening on the stratified 100 (prompt of record) glm-5.3-flash passed: 52 of 93 accepted correct, $0.046; deepseek-v4.1-flash skipped (3 of 5 answers cut at 8000 tokens by its reasoning, $0.033); qwen3.8-flash, mimo-v2.6-pro and minimax-m3 skipped (2 or 3 of 5 calls past the 120 s call timeout or an error, $0.013, $0.016, $0.005); seed-2.0-code and deepseek-v4-pro not run 8812cee + merge $0.15 in all
      20 verify simulator (python, go with tests, ts) the 270 accepted answers 8812cee $0
      20 features and self-agreement of the eight Luna arms the 300 n/a $0
      20 AgenticFlict: the replay of 2,360 merge commits 2,125 merged agent PRs n/a $0
      20 held-out runs of the prompt of record: Luna and glm-5.3-flash on 487 OSS files and on 284 agent-PR files manifest-heldout-487, manifest-agent-284 7fd0ab2's tools tree (8812cee) $0.300 + $0.302 + $0.227 + $0.237
      20 judge 1 (glm-5.3-flash) on the held-out answers the answers of Luna scripts outside the repository $0.025 + $0.021
      20 judge 2 (openai/gpt-6.1-sol, effort low) on the answers landed by G0 OSS held-out 359, agent-PR 210 scripts outside the repository $0.239 + $0.268
      20 judge 3 (mistralai/mistral-large-4-0, unrelated vendor) on the answers landed by G1: a first pass stopped at its cap of $0.25 per set, a second pass (retry of the empty answers with a backoff, and the unjudged) at +$0.03 and +$0.09, a third on the OSS set only (cap +$0.10, +$0.106 booked); 13 OSS and 29 agent-PR answers unjudged and 2 OSS and 3 agent-PR calls empty at the end, counted as wrong OSS held-out 291, agent-PR 130 scripts outside the repository $0.404 + $0.354 = $0.758 booked (the key: $0.739)
      20 the gate analysis of 15.14: a split of the OSS held-out by repository, 139 candidate gates chosen on half and measured on the other, the exchange, leave one repository out manifest-heldout-487, manifest-agent-284, the stored answers and judge verdicts scripts/gate_split.py, gate_b.py outside the repository $0
      20 G1 and the project's real tests (15.15): a differential test at the merge commit (baseline twice, the answer, a confirmation), prometheus, golang/go (a toolchain per merge), flask; pilot and full run the 234 G1-landed files outside spring-boot scripts/test_gate.py, tests_report.py outside the repository $0
      21 the replication of 15.15 on fresh Go conflicts (15.16): Luna and glm on 296 files, judges 1-3 on the 59 G1-landed answers that differ from the human's file, the frozen harness, the frozen identifier signal manifest-fresh-go (296) scripts/repl_run.sh, repl_report.py outside the repository $0.1888 + $0.2108 + $0.0053 + $0.0906 + $0.3097 = $0.805
      21 the build check of 15.17: go test -run '^$' of the package at the merge commit, differential, on the 115 held-out and 144 fresh G1-landed Go files PROTOCOL-BUILD (840dbd5adbbb) scripts/test_gate_b.py outside the repository $0
      21 the Python run of 15.18: Luna and glm on 509 files, the verify simulator, judges 1-3 on the 146 G1-landed answers that differ from the human's file (judge 3 on the 8 disagreements), the differential pytest harness on the 331 G1-landed files manifest-pyfresh (509) scripts/pyrun.sh, test_gate_py.py, py_report.py outside the repository $0.2728 + $0.2718 + $0.0129 + $0.2271 + $0.1833 = $0.968 (the key: $0.920)
      21 SEM and BIND of 15.19 applied once to the 331 G1-landed Python files PROTOCOL-PYSEM (c99a36857fd0) scripts/pysem.py, pysem_eval.py outside the repository $0
      22 the TypeScript preparation of 15.20: the harness, the recipes on 3 merges per repository, the mutation check, the hashed protocol; 1,028 more cases mined; never applied (stopped) manifest-tsfresh (549), PROTOCOL-TS (dcf8da2f22e2) scripts/test_gate_ts.py, tsrun.sh, ts_ids.py, ts_report.py outside the repository $0
      22 label v2b of 15.22: validation on the PM's audit sets (27 Go held-out wrong, 40 right answers drawn, 56 judge calls) with the objective rule, judges 1 and 2 and judge 3 on disagreements; diagnostic call; re-run of 8 judge-3 calls (Amendment 2); not validated, application not run PROTOCOL-LABEL2 (62b059bcc29b) and Amendments 1, 2, 2b scripts/v2core.py, v2_judge.py, v2run.sh, v2_report.py outside the repository $0.0087 + $0.1776 + $0.4201 + $0.0132 = $0.620 booked
      22 the PM label of 15.23: 118 pyfresh answers judged by four blind Claude Opus reviewers with a written rubric, a 20% second review (22 of 23 agree), a post-hoc calibration on 36 Go cases (10 of 11 real errors rejected, 17 of 19 right accepted); gates scored by score.py PROTOCOL-PMJUDGE (b99630b8e01c), PROTOCOL-PMJUDGE-CALIB (1b0cc7373ed7) run/pmjudge-pm/ (score.py ad04280ef7), outside the repository $0
      22 the TypeScript run of 15.24: Luna and glm on 549 files, G1 (323), the differential tests and the project-level type check (one pinned core, about 10 h), 117 case files judged by four blind Claude Opus reviewers with a 20% second review (22 of 23), calibration on TypeScript (18 of 23: not calibrated) PROTOCOL-TS (dcf8da2f22e2) and Amendments 1-3 scripts/tsrun.sh, test_gate_ts.py, ts_cases.py, ts_report_pm.py outside the repository $0.2478 + $0.2363 = $0.484 booked ($0.457 on the key)
      20 the agree_with gate through fake providers and the real graft-resolve with two fake command models tests only 3cc898c $0

      Why a screened model is skipped. The screening is a smoke check of five cases (a go, a java, a python, a rust and a ts file) that costs at most a few cents and stops a model whose calls end before an answer: more than half of the calls failing, or a real spend above what the budget booked, or a projection above its cap. It judges the model in this configuration (effort low, max_tokens 8000, call_timeout 120 s), which a model that reasons for minutes does not fit.

      Appendix B: the judge rubric of section 15

      Judge A (15.12) is a reference-based LLM judge, used offline for measurement only: nothing of it is in the product. It is adapted from the resolution-acceptability criteria of ConflictAgent (evaluation/metrics.py, RESOLUTION_ACCEPTABILITY_CRITERIA, commit d3ab8cc of https://github.com/UBOWENVT/ConflictAgent; MIT licence, Copyright (c) 2026 Bowen Shen). It is asked directly for a binary verdict (no DeepEval scoring), it sees a unified diff of the candidate against the developer's file instead of the conflict block, and the same text goes to every judge model (glm-5.3-flash for judge 1, gpt-6.1-sol for judge 2, mistral-large-4-0 for judge 3). A verdict that is missing or not parsable counts as not acceptable. The answers that equal the developer's file after normalization (line endings, trailing whitespace, trailing blank lines) are accepted without a call. The text, as it is sent:

      Decide whether the CANDIDATE (a merge-conflict resolution) is an ACCEPTABLE resolution relative to the DEVELOPER's actual resolution of the SAME conflict: would a careful reviewer accept it as resolving the conflict the way the developer intended - same behavior and same essential content - even if not character-identical? These differences alone do NOT make it unacceptable: whitespace, indentation, formatting, or comments; the ordering of imports, fields, or methods; a different but behaviorally equivalent phrasing of the same logic; a different set of import statements, as long as the actual code is consistent. Treat it as NOT acceptable if the candidate: still contains conflict markers or is otherwise an unresolved / partial merge; has different program logic or behavior; uses different literal values that matter (version numbers, constants, configuration values); is missing or adds FUNCTIONAL code (statements, methods, conditions - not imports); or chooses a different side's behavior than the developer did.

      The script (judge.py, with the corpus builders and the analysis scripts) is in c1-work/scripts/ next to the data of Appendix A.