Git designed for AI agents.
graft: the agreement gate (G1) is part of the llm strategy: agree_with
Federico Elia3cc898c46f2b
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.mdindex 0f24aea..9d852f6 100644--- a/doc/graft/PLAN10.md+++ b/doc/graft/PLAN10.md@@ -920,7 +920,11 @@ end of this section.composition report (15.3): what the answer keeps, drops, repeats -- a REPORT, not a gate|v- VERDICT: PROPOSAL (origin "model") with its report+ agree_with (optional, off by default; 15.12, 15.13): a second model from another vendor resolved the same file+ at the same time; the two results are not the same file, or a report is not clean -- HOLD+ |+ v+ VERDICT: PROPOSAL (origin "model") with its report; a HELD proposal never lands|vVERIFY: the operator's verify.command on the exact tree that would land, out of the lane (15.9)@@ -937,17 +941,20 @@ end of this section.| writer | proposes a resolution per region | a model, configurable (15.13) | 15.10: the elements kept and removed, 15.12 || checks and repair | refuse a broken answer, repair it once | deterministic checks | 15.10 || report | tells the agent what the answer keeps or drops | deterministic | 15.6: it separates (43% against 60% right), it does not gate |+| agree_with (optional) | holds a proposal back unless a second model from another vendor gives the same file and both reports are clean | code (`llmStrategy`), configured by the operator | 15.12: G1, 59.8% of the files landed with 15.5% wrong under two judges, before the operator's tests || verify | runs the operator's build and tests on the exact tree | the operator's command | 15.9, 14 || decideLand | the one landing rule | code | 15.4 |**Not in the system, and why.** Four ideas were built as measurements, and none of them is a landingsignal on the evidence we have; each is kept as a measurement (Appendix A) and not as code:a decision model that chooses OURS / THEIRS / both per region and lets a writer work only where it-says NEEDS-MERGE (15.11: a java selector, nothing near the bar elsewhere); agreement of several-models or arms on the same answer (15.12: models make the same mistakes); structure and+says NEEDS-MERGE (15.11: a java selector, nothing near the bar elsewhere); agreement of arms of one+model, and of several models on every region (15.12: models make the same mistakes); structure andself-agreement rules (15.12: they do not hold on held-out repositories); a build-and-vet simulationof the verification in the bench (15.12: it removes the answers that do not compile and no more).-What does move the number is the operator's own tests, which only the operator has.+What does move the number is the operator's own tests, which only the operator has. What is in the+system, as an option that is off by default, is the agreement of two models from different vendors on the+whole file (`agree_with`, 15.13): the one filter that chose better proposals than the writer alone.### 15.1 End to end@@ -1027,8 +1034,8 @@ What does move the number is the operator's own tests, which only the operator hclassify: path policy, shape, size, binary, lockfile, generated, minified|-- not sendable ----------------------------------------> AGENT (NOT_SENT, reason)v- operator rules in their order (trivial, union, rerere, command), each with the syntax check and- the operator's checks+ operator rules in their order (trivial, union, rerere, command), each with the operator's checks+ (the syntax check is the model's: it runs on the llm strategy's answers only)|-- a rule resolved it ---------------------------------> VERDICT RULE (lands, 15.4)vAI configured? --no-------------------------------------> AGENT (FAILED: no strategy applied)@@ -1057,6 +1064,12 @@ What does move the number is the operator's own tests, which only the operator hcomposition report (15.3) of the regions: drops, duplicates, contested changes (a report, not a gate,and not a repair trigger)v+ [agree_with set] the second model resolved the same file in parallel, with the same prompt:+ |-- its answer is not the same file after normalization (CRLF, trailing whitespace, trailing blank+ | lines), or it failed, or a report of the two holds a drop or a duplicate+ | ---------------------------------------------> HELD: the first model's proposal with the reason+ | (PROPOSED, never lands, not even verified)+ vVERDICT PROPOSAL (PROPOSED, the proposal + its report)vverify.enabled set? --no--> stays a PROPOSAL: the agent decides ("no verify.command")@@ -1098,6 +1111,8 @@ What does move the number is the operator's own tests, which only the operator hand passed on THIS tree (tree object ID equality)a file a model resolved ...................... LAND only if verify.command ran and passed on THIStree: tree object ID equality, not the tip+ a file the resolver holds (agree_with) ....... PACKAGE: never lands, whatever was verified; the+ tree is not even verified for itanything else ................................ PACKAGE (the model's files are proposals)One rule about a model's resolution. No other code path lands it. No check disables itself: a configuration thatcannot be used stops the server at startup, and a failure at run time refuses the land. No@@ -1124,9 +1139,10 @@ What does move the number is the operator's own tests, which only the operator h| classify | `landClassify`, `resolver.SendRules` | operations/land_resolve.go; graft/resolver || operator rules, file checks | `Resolver.resolveFile`, `Resolver.runChecks`, `syntax.CheckMerge` | tools/graft-agents/resolve/resolve.go || AI per file | `llmStrategy.Resolve`, `ResolveLLM`, `llmResolvePatch` | tools/.../resolve/llm_strategy.go, llm_core.go, llm_patch.go |-| regions, assemble | `llmRegions`, `llmAssemble` | tools/.../resolve/llm_core.go |+| regions (the fallback of git's, and the assembly of the file) | `llmRegions`, `llmJoin` | tools/.../resolve/llm_core.go, llm_gitmerge.go || one call per region, its checks | `llmPatchSystem`, `llmPatchUser`, `llmPatchAnswer`, `llmCheckPatched`, `llmCheckAnswer` | tools/.../resolve/llm_patch.go, llm_core.go || proposal mark, no recording | `proposer`, `Resolver.resolveFile` | tools/.../resolve/llm_strategy.go, resolve.go |+| agreement gate: second model, comparison, hold | `llmStrategy.Resolve`, `llmSide.start`, `llmStrategy.agreement`, `LLMNormalize`, `LLMReportClean`, `Resolution.Hold`, `ResponseFile.Hold` (a held file is a model's), `landAnswerHolds`, `decideLand`, `landSetOutcomes` | tools/.../resolve/llm_strategy.go, llm_core.go, resolve.go; graft/resolver/protocol.go; operations/land_proposals.go, land_resolve.go || verify.command's checkout | `landCheckoutTree`, `landMaterializer.add`, `landCheckMaterialized`, `landRemoveCheckout`, `resolver.IsGitDirName` | operations/land_verify.go; graft/resolver/validate.go || merge commit, trailers | `landWriteResolution` | operations/land_resolve.go || PACKAGE | `landWithPackage`, `landListing.conflictPackage` | operations/land_package.go |@@ -1177,7 +1193,7 @@ the same base lines supersedes it. Measured with zero model calls:uncontested change of both sides" or "drops THEIRS' change at base 40-44", "duplicates a lineboth sides added", "changes lines outside the conflicts", "N contested regions resolved freely".The duplicates flag carries signal (7 of 19 correct where it fires, against 58.6% overall): it is- useful in the report and as a repair trigger.+ useful in the report, and not as a repair trigger (item 4).4. The repair round is driven only by deterministic findings with their exact wording, and only for ananswer that is broken (the rules of an answer, the syntax): the composition report's drops andduplicates were tried as repair triggers and made accepted answers worse (15.10).@@ -1809,13 +1825,20 @@ verification: `verify.enabled` false, `verify.command`, `verify.timeout` 5 min,`verify.max_tree_bytes` 2 GiB, `verify.max_tree_depth` 64. Nothing lands from a model unless `verify.enabled` is setand the command passes on the exact tree.-**The recommended gate for an operator who wants auto-landing.** G1 of 15.12, as a recipe and not as server code:-two models from different vendors give the byte-identical resolution, both composition reports are clean, and-`verify.command` passes on the exact tree (`verify.enabled` is mandatory with it, and the wrapper is a `command`-strategy with `origin = "model"`; RESOLVE.md has the recipe). Measured on the OSS held-out files it lands 59.8% of the-files with 15.5% of those wrong under two independent judges (upper bound 20.1%), before the operator's tests; on conflicts-of agent PRs 45.8% of the files with 43.1% wrong. The 20%-wrong bar it is compared with was set after the numbers were-known (15.12).+**The agreement gate for an operator who wants auto-landing (`agree_with`).** G1 of 15.12 is an option of the llm+strategy, off by default: a nested `[strategy.llm.agree_with]` table, with the keys of `[strategy.llm]`, names a+second model, which should come from another vendor. Both models resolve each file at the same time with the prompt of+record. The first model's resolution is proposed; it can land only if the two results are the same file after the+judge's normalization, the composition reports of both are clean, and `verify.command` passes on the exact tree+(`verify.enabled` is mandatory, as for every model resolution). Otherwise the resolution is held: the response carries+a `hold` reason and the report says that the second model disagreed, and the server never lands a held file, nor+runs the verification for it. A budget table must be given for both models or for neither. The bench runs the same+code and prices both models. RESOLVE.md has the configuration. Measured on the OSS held-out files it lands 59.8% of+the files with 15.5% of those wrong under two independent judges (upper bound 20.1%), before the operator's tests; on+conflicts of agent PRs 45.8% of the files with 43.1% wrong. Those numbers were measured as two single-model runs+combined offline (the gate decides the same thing on the same answers); the code is covered by tests with fake+providers and was not run through the bench against the live models. The 20%-wrong bar it is compared with was set+after the numbers were known (15.12).**The default model.** The shipped example is `openai/gpt-6-luna` at effort low. On the stratified 100 the one othermodel that passed the smoke check, `z-ai/glm-5.3-flash` pinned to Z.AI, is as good (15.12) and costs the same perdiff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.mdindex 561d443..96d7bb0 100644--- a/doc/graft/RESOLVE.md+++ b/doc/graft/RESOLVE.md@@ -143,8 +143,10 @@ A client asks for resolution per call:[What is never sent](#what-is-never-sent)), sends the others to theresolver in one request (protocol below), and validates the answer (see[Validation](#validation)).-4. Only when every file is resolved by rules alone, it builds the merged tree from the- merge-tree result, writes the resolved+4. Only when every conflicting file is resolved and the decision lands it (a rule's+ resolution does; a model's resolution only after `verify.command` passed on the exact+ tree that lands, see [Proposals](#proposals-and-when-a-models-resolution-lands)), it+ builds the merged tree from the merge-tree result, writes the resolvedfiles as new blobs or deletes them, and creates a merge commit with parents`[tip, commit]`. The message is `merge_message` or`Merge <short commit> into <branch>`. Trailers:@@ -161,7 +163,8 @@ A client asks for resolution per call:against the new tip.Otherwise (resolution disabled or refused, a file that is never sent, an-invalid answer, a timeout, an unresolvable file, or a file a model resolved)+invalid answer, a timeout, an unresolvable file, or a file a model resolved+that `verify.command` did not pass, or that no `verify.command` checks)the land fails with the normal `merge_conflict` error, the reason is logged,and the error carries the conflict package as a second detail (see[The conflict package and the re-land](#the-conflict-package-and-the-re-land)).@@ -691,26 +694,58 @@ origin = "model" # its files are proposals, and land only if verify.command pa`origin` is `"rule"` (the default) or `"model"`. A model command's answers arenot recorded by `rerere`.-**The recommended gate for auto-landing a model's resolution.** The `llm` strategy alone produces a proposal that lands-only if `verify.command` passes on the exact tree. An operator who wants model resolutions to land without an agent in-the loop gets the best measured result (PLAN10 15.12) from this gate, which the server does not implement and an-operator builds with the `command` strategy above:+**The agreement gate: `agree_with`.** The `llm` strategy alone produces a proposal that lands only if `verify.command`+passes on the exact tree. An operator who wants model resolutions to land without an agent in the loop gets the best+measured result (PLAN10 15.12) by adding a second model from another vendor to the strategy, in the `agree_with`+table (the same keys as `[strategy.llm]`, except `agree_with` itself):-1. **Two models from different vendors resolve the file** (two `graft-resolve` configurations, each with one `llm`- strategy and its own provider), and the resolution is used only if their results are byte-identical after- normalizing line endings and trailing whitespace;+```toml+[strategy.llm.agree_with]+type = "openai"+base_url = "https://openrouter.ai/api/v1"+api_key_env = "OPENROUTER_API_KEY"+model = "z-ai/glm-5.3-flash"+effort = "low"+max_tokens = 8000+timeout = "2400s"++[strategy.llm.agree_with.routing]+only = ["z-ai/fp8"]+```++Both models resolve each file at the same time with the same prompt of record. The first model's resolution is the+one that is proposed. It lands only if all three hold:++1. **the two results are the same file** after normalizing line endings, trailing whitespace and trailing blank lines;2. **the composition report of both is clean** (no line saying that the answer drops or duplicates a change of a side);-3. **`verify.command` passes** on the exact tree: `verify.enabled = true` is mandatory with this gate, and the strategy- must be `origin = "model"` as above, so that the server itself refuses to land it otherwise.+3. **`verify.command` passes** on the exact tree, as for every model resolution: `verify.enabled = true` is mandatory+ (the gate does not replace it, it chooses the proposals worth verifying).++When 1 or 2 fails, or the second model gives no usable answer, the first model's resolution is **held**: the response+marks the file with a `hold` reason (`ResolvedFile.Hold`, "the second model disagreed" and why, also in the report),+the server never lands a held file, whatever `verify.command` says (it does not even run it for that land), and the+agent gets the conflict package with the proposal as usual. When the first model fails, the file fails as it does+without the gate. The gate is off by default (no `agree_with` table, no second call). Pick two models from different+vendors: two models of one family make the same mistakes. A `[strategy.llm.budget]` must have an+`[strategy.llm.agree_with.budget]` beside it (a cap that covers one of two models is not a cap), and the spend of both+is counted for the file. The bench (`graft-resolve bench run`) runs the same code, prices the two models from+OpenRouter and splits `--max-usd` between them; its report counts the held answers apart. Upgrade `graft-resolve` and the+server together: an older server does not know `hold` and would verify a held proposal like any other.-The wrapper (the program of the `command` strategy) writes `{out}` only when 1 and 2 hold and fails otherwise; the chain-then goes on, and the agent gets the conflict package with the proposals as usual. The product ships no such wrapper.What it was worth, measured on 487 held-out files of 13 open-source repositories (Luna and glm-5.3-flash, with theverify simulation of the bench in place of the operator's tests): 59.8% of the files landed, and 15.5% of those differedfrom what two independent reviewing models accept (11.3% for one of them; Wilson upper bound 20.1% for both), against 43%-on conflicts of merged agent PRs. These figures are before the operator's own tests, which are the one signal that-caught more errors where we could run them; the gate does not replace `verify.command`, it only chooses which proposals-are worth verifying. Cost: two model calls per region.+on conflicts of merged agent PRs. These figures were measured by combining two separate single-model runs, which is what+the gate decides; the gate in the code is covered by tests with fake providers, not re-measured through itself (PLAN10+15.12). They are before the operator's own tests, which are the one signal that caught more errors where we could run+them. Cost: two model calls per region.++**Upgrade `graft-resolve` together with the server.** A resolution is a rule's only if its file says so (`origin`+`"rule"`, protocol v1); a file without an origin is a model's, which lands only through `verify.command`. A+`graft-resolve` of an earlier release answers without an origin, so with a new server every file it resolves,+`trivial` and `union` included, is a proposal, and lands only if `verify.command` passes on the exact tree (without+one, it never lands). The other way round, an older server does not know `hold` (the `agree_with` gate) and would+verify a held proposal like any other. Upgrade both in the same maintenance window.**The `rerere` store and the upgrade.** Entries are versioned and record theirorigin. The stores of v0.1.0 and v0.2.0 recorded the `llm` answers as well, with@@ -976,8 +1011,9 @@ above).The provider's API keys are withheld from the model: they are read from theenvironment or a file by the provider, never put in the prompt. Everything-else is sent to the provider, so the three versions of a file and both commit-messages are shared with it, and any secret that they contain goes with them. Do not send+else is sent to the provider, so the three versions of a file and the commit+message of the change (the tip's message is not sent) are shared with it, and any secret that they contain goes with them. With+`agree_with` two providers receive them. Do not sendfiles you would not send to that provider. The server never calls the model.Turn AI on only if you accept that the resolution is produced by a model.Every AI resolution carries the `Graft-Resolved-By` trailer, so it can bediff --git a/internal/gitaly/service/operations/land_proposals.go b/internal/gitaly/service/operations/land_proposals.goindex 5d9a7e5..17348c9 100644--- a/internal/gitaly/service/operations/land_proposals.go+++ b/internal/gitaly/service/operations/land_proposals.go@@ -54,6 +54,23 @@ const landVerifyDisabledReason = "verification is disabled (verify.enabled is fa// landNoVerifyReason follows landProposedReason when no verify.command is configured.const landNoVerifyReason = "no verify.command is configured, so nothing verifies a model's resolution"+// landHeldReason is the reason of a file that the resolver holds back: the second model of the llm+// strategy's agree_with gate did not agree with the first, so the resolution is a proposal that+// does not land by itself, even on a tree that verify.command passed on. The resolver's own words+// follow it.+const landHeldReason = "held back by the resolver, so it does not land by itself even if verify.command passes (see proposal): "++// landAnswerHolds reports whether a resolver's answer holds back a file (ResponseFile.Hold): such an+// answer never lands, whatever verify.command says.+func landAnswerHolds(answer *resolver.Response) bool {+ for _, f := range answer.Files {+ if f.Hold != "" {+ return true+ }+ }+ return false+}+// landNormalizeOrigins gives each file of an answer its origin, as the server reads it: OriginRule// only for a file that says it is a rule's and is not marked as a proposal, OriginModel for any// other. It is done when the answer arrives, so that the answers kept by their keys and the rounds@@ -231,6 +248,10 @@ func decideLand(answer *resolver.Response, files []*landFileState, tree, verifieif answer == nil {return false}+ // A file the resolver holds back never lands: the second model did not agree with the first.+ if landAnswerHolds(answer) {+ return false+ }proposes := landAnswerProposes(answer)if (proposes || verifyRequired) && (verified == "" || verified != tree) {return falsediff --git a/internal/gitaly/service/operations/land_proposals_test.go b/internal/gitaly/service/operations/land_proposals_test.goold mode 100755new mode 100644index 4dc4aa8..42de354--- a/internal/gitaly/service/operations/land_proposals_test.go+++ b/internal/gitaly/service/operations/land_proposals_test.go@@ -373,6 +373,18 @@ func TestDecideLand(t *testing.T) {require.False(t, decideLand(rules, resolved(), "t1", "t2", true), "rules, verified on another tree (the tip moved)")require.False(t, decideLand(rules, resolved(), "", "", true), "rules, no tree at all: empty is not equal")+ // A file the resolver holds back (the second model did not agree) never lands, whatever was+ // verified, whatever its origin says.+ held := &resolver.Response{Resolver: "graft-resolve/llm", Files: []resolver.ResponseFile{+ {Path: "a", Origin: resolver.OriginModel, Proposal: true, Hold: "the second model disagreed"}, {Path: "b", Origin: resolver.OriginRule},+ }}+ heldOutcomes := []*landFileState{{outcome: gitalypb.LandConflictFile_OUTCOME_PROPOSED}, {outcome: gitalypb.LandConflictFile_OUTCOME_RESOLVED}}+ require.False(t, decideLand(held, heldOutcomes, "tree-1", "tree-1", false), "held, verified on this very tree")+ require.False(t, decideLand(held, heldOutcomes, "tree-1", "tree-1", true), "held, verified on this very tree, verification required")+ require.True(t, held.Files[0].FromModel())+ ruleHeld := resolver.ResponseFile{Path: "a", Origin: resolver.OriginRule, Hold: "x"}+ require.True(t, ruleHeld.FromModel(), "a held file is a model's, whatever its origin says")+// THE rule (PLAN10 15.9): a file that proposes may land only if verify.command passed on the exact// tree that lands, compared by object ID.flagged := &resolver.Response{Resolver: "graft-resolve/trivial", Files: []resolver.ResponseFile{diff --git a/internal/gitaly/service/operations/land_resolve.go b/internal/gitaly/service/operations/land_resolve.goindex 80ea71e..8d91a66 100644--- a/internal/gitaly/service/operations/land_resolve.go+++ b/internal/gitaly/service/operations/land_resolve.go@@ -226,6 +226,9 @@ func (s *Server) landResolveAt(var tree, verified git.ObjectIDif combined != nil && (landAnswerProposes(combined) || verifyRequired) {switch {+ case landAnswerHolds(combined):+ // A file is held back: the land cannot land, so the tree is not verified (that would hold+ // the repository's verify slot for nothing). The files say why in their reasons.case pre.unverifiable != "":landAnnotateUnverified(files, combined, pre.unverifiable)case !verifyRequired && len(s.resolveCfg.Verify.Command) > 0:@@ -310,6 +313,8 @@ func landSetOutcomes(answer *resolver.Response, files []*landFileState) {f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_FAILED, "the resolver gave no answer for the file"case err != nil:f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_REJECTED, err.Error()+ case rf.Hold != "":+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_PROPOSED, landCutString(landHeldReason+rf.Hold, landPackageReasonBytes)case rf.FromModel():f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_PROPOSED, landProposedReason}diff --git a/internal/gitaly/service/operations/land_verified_test.go b/internal/gitaly/service/operations/land_verified_test.goindex 62f33ca..6a76073 100644--- a/internal/gitaly/service/operations/land_verified_test.go+++ b/internal/gitaly/service/operations/land_verified_test.go@@ -799,3 +799,35 @@ func TestUserLand_finalPass_aLostRaceStillEndsWithThePackage(t *testing.T) {require.Equal(t, "the time ran out", a.GetReason())require.Equal(t, gitalypb.LandConflictFile_OUTCOME_RESOLVED, packageFile(t, pkg, "b").GetOutcome())}++// TestUserLand_aHeldModelResolutionNeverLands: a file the resolver holds back (the second model of+// agree_with did not agree) is a proposal that does not land even when verify.command would pass on+// the tree that would land; the command does not even run. The same answer without the hold lands.+func TestUserLand_aHeldModelResolutionNeverLands(t *testing.T) {+ t.Parallel()++ held := modelAnswer()+ held.Files[0].Hold = "the second model (glm) disagreed"+ fx, rdir, vdir := verifiedFixture(t, held, "grep -qx merged-a a && grep -qx merged-b b && grep -qx d d", nil)++ _, err := fx.client.UserLand(fx.ctx, fx.request("Land agent work"))+ requireMergeConflict(t, err)+ pkg := landConflictDetail(err)+ require.NotNil(t, pkg)+ var reasons []string+ for _, f := range pkg.GetFiles() {+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), f.GetPath())+ reasons = append(reasons, f.GetReason())+ }+ require.Contains(t, strings.Join(reasons, "\n"), landHeldReason+"the second model (glm) disagreed")+ require.Equal(t, 1, resolverCalls(t, rdir))+ require.Equal(t, 0, verifyRuns(t, vdir), "a land that cannot land is not verified")+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)+ require.NotEmpty(t, proposalRefTarget(t, fx), "the held resolution is kept as a proposal")++ // The hold is the whole difference: the same answer without it lands through the verification.+ fx2, _, vdir2 := verifiedFixture(t, modelAnswer(), "grep -qx merged-a a && grep -qx merged-b b && grep -qx d d", nil)+ _, err = fx2.client.UserLand(fx2.ctx, fx2.request("Land agent work"))+ require.NoError(t, err)+ require.Equal(t, 1, verifyRuns(t, vdir2))+}diff --git a/internal/graft/resolver/protocol.go b/internal/graft/resolver/protocol.goindex 3a4a52b..666d39c 100644--- a/internal/graft/resolver/protocol.go+++ b/internal/graft/resolver/protocol.go@@ -229,6 +229,11 @@ type ResponseFile struct {Content []byte `json:"content,omitempty"`Origin string `json:"origin,omitempty"`Proposal bool `json:"proposal,omitempty"`+ // Hold is the reason the resolver holds a model's resolution back (the second model of the llm+ // strategy's agree_with did not agree): the file is a proposal for the agent and never lands by+ // itself, whatever verify.command says (decideLand). A held file is a model's, whatever its+ // origin says.+ Hold string `json:"hold,omitempty"`// Report is the composition report of a model's resolution (PLAN10 15.3): in exact words what it// keeps, drops and repeats of the two sides' changes. A report, never a gate: nothing in it// decides whether the resolution lands. It goes to the agent with the proposal.@@ -236,7 +241,7 @@ type ResponseFile struct {}// FromModel reports whether the file is a model's resolution: anything that is not marked as a-// rule's, and anything marked as a proposal.+// rule's, anything marked as a proposal, and anything held back.func (f ResponseFile) FromModel() bool {- return f.Proposal || f.Origin != OriginRule+ return f.Proposal || f.Hold != "" || f.Origin != OriginRule}diff --git a/internal/graft/resolver/testdata/protocol-fields.json b/internal/graft/resolver/testdata/protocol-fields.jsonindex 59125a2..938e478 100644--- a/internal/graft/resolver/testdata/protocol-fields.json+++ b/internal/graft/resolver/testdata/protocol-fields.json@@ -7,7 +7,7 @@"side": ["content", "mode", "oid"],"limits": ["max_file_bytes"],"response": ["explanation", "files", "resolver", "version"],- "response_file": ["action", "content", "mode", "origin", "path", "proposal", "report"],+ "response_file": ["action", "content", "hold", "mode", "origin", "path", "proposal", "report"],"error_response": ["error", "files"],"file_outcome": ["outcome", "path", "reason"],"check_response": ["files", "operation", "resolver", "version"],diff --git a/tools/graft-agents/cmd/graft-resolve/bench.go b/tools/graft-agents/cmd/graft-resolve/bench.goindex 962f371..fb82d76 100644--- a/tools/graft-agents/cmd/graft-resolve/bench.go+++ b/tools/graft-agents/cmd/graft-resolve/bench.go@@ -171,60 +171,63 @@ func benchRun(ctx context.Context, args []string, stdout, stderr io.Writer) int// A chain with a model runs under a hard cap: every call reserves its worst case first// (resolve.Budget), and the calls that would pass the cap are not made.- var budget *resolve.Budget- var pricing *bench.Pricing+ //+ // A chain with an agree_with model has two models with two prices (both priced from OpenRouter),+ // and the cap is split in two: each model has its own budget of half of it.+ var budget, agreeBudget *resolve.Budget+ var pricing, agreePricing *bench.Pricingif models := res.LLMModels(); len(models) > 0 {if *maxUSD <= 0 {fmt.Fprintln(stderr, "graft-resolve bench run: the chain calls a model, so --max-usd is required")return 2}- var prices resolve.Prices+ agreeModels := res.LLMAgreeModels()+ if len(agreeModels) > 0 && *pricingSource != "openrouter" {+ fmt.Fprintln(stderr, "graft-resolve bench run: a chain with an agree_with model has two prices: use --pricing openrouter")+ return 2+ }+ cap := *maxUSD+ if len(agreeModels) > 0 {+ cap /= 2+ }+ var prices, agreePrices resolve.Pricesswitch *pricingSource {case "openrouter":- distinct := map[string]bool{}- for _, m := range models {- distinct[m] = true- }- if len(distinct) != 1 {- fmt.Fprintf(stderr, "graft-resolve bench run: --pricing openrouter prices one model, the chain has %v\n", models)- return 2+ var code int+ if prices, pricing, code = benchPrice(ctx, stderr, *openRouterAPI, models, res.LLMRoutings(), res.LLMMaxPromptTokens()); code != 0 {+ return code}- var model string- for m := range distinct {- model = m- }- // The strategies of the one model must route it the same way, or no one price fits.- var routing *llm.Routing- first := true- for _, r := range res.LLMRoutings() {- if !first && !reflect.DeepEqual(r, routing) {- fmt.Fprintln(stderr, "graft-resolve bench run: --pricing openrouter needs one routing, the llm strategies route differently")- return 2+ if len(agreeModels) > 0 {+ if agreePrices, agreePricing, code = benchPrice(ctx, stderr, *openRouterAPI, agreeModels, res.LLMAgreeRoutings(), res.LLMMaxPromptTokens()); code != 0 {+ return code}- routing, first = r, false- }- p, err := bench.OpenRouterPricing(ctx, &http.Client{Timeout: 30 * time.Second}, *openRouterAPI, model, res.LLMMaxPromptTokens(), routing)- if err != nil {- fmt.Fprintf(stderr, "graft-resolve bench run: %v\n", err)- return 1}- pricing, prices = &p, p.Prices()case "fixed":prices = resolve.Prices{InputPerMillion: *inputPrice, OutputPerMillion: *outputPrice}default:fmt.Fprintf(stderr, "graft-resolve bench run: unknown --pricing %q, want openrouter or fixed\n", *pricingSource)return 2}- if budget, err = resolve.NewBudget(*maxUSD, prices, ""); err != nil {+ if budget, err = resolve.NewBudget(cap, prices, ""); err != nil {fmt.Fprintf(stderr, "graft-resolve bench run: %v\n", err)return 1}- if _, err := res.UseBudget(budget); err != nil {+ if len(agreeModels) > 0 {+ if agreeBudget, err = resolve.NewBudget(cap, agreePrices, ""); err != nil {+ fmt.Fprintf(stderr, "graft-resolve bench run: %v\n", err)+ return 1+ }+ }+ if _, err := res.UseBudgets(budget, agreeBudget); err != nil {fmt.Fprintf(stderr, "graft-resolve bench run: %v\n", err)return 1}fmt.Fprintf(stderr, "cap $%.2f; $%g per million input tokens, $%g per million output tokens, $%g per request\n",- *maxUSD, prices.InputPerMillion, prices.OutputPerMillion, prices.PerRequest)+ cap, prices.InputPerMillion, prices.OutputPerMillion, prices.PerRequest)+ if agreeBudget != nil {+ fmt.Fprintf(stderr, "cap $%.2f for the agree_with model; $%g per million input tokens, $%g per million output tokens, $%g per request\n",+ cap, agreePrices.InputPerMillion, agreePrices.OutputPerMillion, agreePrices.PerRequest)+ }}cases := bench.Sample(manifest.Cases, *sample, *seed)@@ -246,7 +249,11 @@ func benchRun(ctx context.Context, args []string, stdout, stderr io.Writer) int// same way, and a run that goes on only measures the refusals.if budget != nil {runner.Stop = func() string {- if n := budget.PaymentRefusals(); n > 0 {+ n := budget.PaymentRefusals()+ if agreeBudget != nil {+ n += agreeBudget.PaymentRefusals()+ }+ if n > 0 {return "the provider refused a call with HTTP 402 (no credit)"}return ""@@ -261,6 +268,15 @@ func benchRun(ctx context.Context, args []string, stdout, stderr io.Writer) intreturn 1}report.SpendUSD, report.MaxUSD, report.Pricing = snap.SpentUSD, *maxUSD, pricing+ if agreeBudget != nil {+ asnap, err := agreeBudget.Snapshot()+ if err != nil {+ fmt.Fprintf(stderr, "graft-resolve bench run: %v\n", err)+ return 1+ }+ report.SpendUSD += asnap.SpentUSD+ report.AgreePricing = agreePricing+ }}if err := writeReport(report, *jsonOut, *mdOut, stdout); err != nil {@@ -274,6 +290,39 @@ func benchRun(ctx context.Context, args []string, stdout, stderr io.Writer) intreturn 0}+// benchPrice reads from OpenRouter the price of the one model that the strategies in models use,+// routed the way routings say (the strategies of the one model must route it the same way, or no+// one price fits). It returns the exit code to use when it fails, else 0.+func benchPrice(ctx context.Context, stderr io.Writer, api string, models map[string]string, routings map[string]*llm.Routing, maxPromptTokens int) (resolve.Prices, *bench.Pricing, int) {+ distinct := map[string]bool{}+ for _, m := range models {+ distinct[m] = true+ }+ if len(distinct) != 1 {+ fmt.Fprintf(stderr, "graft-resolve bench run: --pricing openrouter prices one model, the chain has %v\n", models)+ return resolve.Prices{}, nil, 2+ }+ var model string+ for m := range distinct {+ model = m+ }+ var routing *llm.Routing+ first := true+ for _, r := range routings {+ if !first && !reflect.DeepEqual(r, routing) {+ fmt.Fprintln(stderr, "graft-resolve bench run: --pricing openrouter needs one routing, the llm strategies route differently")+ return resolve.Prices{}, nil, 2+ }+ routing, first = r, false+ }+ p, err := bench.OpenRouterPricing(ctx, &http.Client{Timeout: 30 * time.Second}, api, model, maxPromptTokens, routing)+ if err != nil {+ fmt.Fprintf(stderr, "graft-resolve bench run: %v\n", err)+ return resolve.Prices{}, nil, 1+ }+ return p.Prices(), &p, 0+}+// benchRejudge judges the stored answers of a run again, with no model call://// graft-resolve bench rejudge --report-in FILE --manifest FILE --clones DIR --answers-dir DIRdiff --git a/tools/graft-agents/protocol/protocol.go b/tools/graft-agents/protocol/protocol.goindex 07fb568..dd5be0c 100644--- a/tools/graft-agents/protocol/protocol.go+++ b/tools/graft-agents/protocol/protocol.go@@ -158,6 +158,12 @@ type ResolvedFile struct {Origin string `json:"origin,omitempty"`// Proposal is the older mark of a model's resolution, still sent with OriginModel.Proposal bool `json:"proposal,omitempty"`+ // Hold is the reason the resolver holds a model's resolution back: the file is a proposal for+ // the agent and does not land by itself, even when verify.command passes on the tree that+ // would land. It is set when the llm strategy's agree_with gate is on and the second model did+ // not agree with the first (resolve.Resolution.Hold). A resolver that does not know the field+ // is an older one: upgrade graft-resolve together with the server.+ Hold string `json:"hold,omitempty"`// Report is the composition report of a model's resolution (PLAN10 15.3): in exact words what it// keeps, drops and repeats of the two sides' changes. A report and not a gate: nothing in it// decides whether the resolution lands. It goes to the agent with the proposal.diff --git a/tools/graft-agents/resolve/bench/bench_test.go b/tools/graft-agents/resolve/bench/bench_test.goindex e81ee29..a8f9ac4 100644--- a/tools/graft-agents/resolve/bench/bench_test.go+++ b/tools/graft-agents/resolve/bench/bench_test.go@@ -167,3 +167,26 @@ func TestBlankLinesOnly(t *testing.T) {}}}++// TestReportCountsTheAnswersTheAgreeGateHolds: an accepted answer with a Hold is counted apart; what+// would land through verify is the rest, and its wrong rate is the one the gate is judged by.+func TestReportCountsTheAnswersTheAgreeGateHolds(t *testing.T) {+ rep := NewReport("agree", []Result{+ {ID: "1", Family: "go", Outcome: OutcomeCorrect},+ {ID: "2", Family: "go", Outcome: OutcomeWrong},+ {ID: "3", Family: "go", Outcome: OutcomeWrong, Hold: "the second model disagreed"},+ {ID: "4", Family: "go", Outcome: OutcomeCorrect, Hold: "the second model disagreed"},+ {ID: "5", Family: "go", Outcome: OutcomeRejected, Hold: "never held: not accepted"},+ })+ if got := rep.Total; got.Held != 2 || got.HeldWrong != 1 || got.Accepted() != 4 {+ t.Fatalf("total = %+v", got)+ }+ var b strings.Builder+ if err := rep.WriteMarkdown(&b); err != nil {+ t.Fatal(err)+ }+ want := "agree_with gate: 4 accepted answers, 2 held back as proposals (the second model did not agree; 1 of them wrong), 2 would land through verify (1 wrong, 50.0%)."+ if !strings.Contains(b.String(), want) {+ t.Fatalf("report lacks the gate line:\n%s", b.String())+ }+}diff --git a/tools/graft-agents/resolve/bench/judge.go b/tools/graft-agents/resolve/bench/judge.goindex bd40c48..a3d5077 100644--- a/tools/graft-agents/resolve/bench/judge.go+++ b/tools/graft-agents/resolve/bench/judge.go@@ -5,21 +5,13 @@ import ("path""regexp""strings"++ "github.com/feder-cr/git-ai/tools/graft-agents/resolve")// normalize returns content with line endings as LF, trailing whitespace of every line removed// and trailing blank lines dropped: the equivalence by which a result is judged correct.-func normalize(content []byte) []byte {- text := strings.ReplaceAll(string(content), "\r\n", "\n")- lines := strings.Split(text, "\n")- for i, line := range lines {- lines[i] = strings.TrimRight(line, " \t\r")- }- for len(lines) > 0 && lines[len(lines)-1] == "" {- lines = lines[:len(lines)-1]- }- return []byte(strings.Join(lines, "\n"))-}+func normalize(content []byte) []byte { return resolve.LLMNormalize(content) }// judge reports whether a result equals the human resolution after normalization, and whether it// is byte-identical. A nil content is a deletion.diff --git a/tools/graft-agents/resolve/bench/report.go b/tools/graft-agents/resolve/bench/report.goindex c591810..083fdb5 100644--- a/tools/graft-agents/resolve/bench/report.go+++ b/tools/graft-agents/resolve/bench/report.go@@ -32,6 +32,11 @@ type Counts struct {SyntaxUnchecked int `json:"syntax_unchecked"`// EqualsOurs counts the accepted results equal to OURS (part of Correct and Wrong).EqualsOurs int `json:"equals_ours"`+ // Held counts the accepted results the agree_with gate holds back as proposals (the second+ // model did not agree), and HeldWrong those of them that are wrong; both are part of Correct+ // and Wrong. What would land through verify is the rest.+ Held int `json:"held"`+ HeldWrong int `json:"held_wrong"`// Regions counts the conflicting regions of the accepted llm answers, and RegionsEqualOurs// those answered with OURS' region unchanged.Regions int `json:"regions"`@@ -73,6 +78,12 @@ func (c *Counts) add(r Result) {c.InputTokens += r.InputTokensc.OutputTokens += r.OutputTokensc.ReasoningTokens += r.ReasoningTokens+ if r.Hold != "" && (r.Outcome == OutcomeCorrect || r.Outcome == OutcomeWrong) {+ c.Held+++ if r.Outcome == OutcomeWrong {+ c.HeldWrong+++ }+ }switch r.Outcome {case OutcomeCorrect:c.Correct++@@ -117,6 +128,9 @@ type Report struct {MaxUSD float64 `json:"max_usd,omitempty"`// Pricing is the price the spend was computed with, as read at run time.Pricing *Pricing `json:"pricing,omitempty"`+ // AgreePricing is the price of the agree_with model, when the chain has one; the spend is the+ // sum of the two models' (each under half of the cap).+ AgreePricing *Pricing `json:"agree_pricing,omitempty"`// StopReason is why the run stopped before its last cases (OutcomeStopped), if it did.StopReason string `json:"stop_reason,omitempty"`Results []Result `json:"results"`@@ -152,6 +166,14 @@ func NewReport(label string, results []Result) Report {return rep}+// percent is 100 * n / d, or 0 when d is 0.+func percent(n, d int) float64 {+ if d == 0 {+ return 0+ }+ return 100 * float64(n) / float64(d)+}+// WriteMarkdown writes the report as Markdown tables.func (rep Report) WriteMarkdown(w io.Writer) error {var b strings.Builder@@ -167,13 +189,20 @@ func (rep Report) WriteMarkdown(w io.Writer) error {fmt.Fprintf(&b, "; AI spend $%.4f of the $%.2f cap", rep.SpendUSD, rep.MaxUSD)}b.WriteString(".\n\n")- if p := rep.Pricing; p != nil {+ for i, p := range []*Pricing{rep.Pricing, rep.AgreePricing} {+ if p == nil {+ continue+ }over := fmt.Sprintf("highest of %d endpoints", p.Endpoints)if len(p.Priced) > 0 {over = fmt.Sprintf("highest of the %d of %d endpoints the routing allows: %s", len(p.Priced), p.Endpoints, strings.Join(p.Priced, ", "))}- fmt.Fprintf(&b, "Model %s, priced at run time (%s, %s, %s): $%g per million input tokens, $%g per million output tokens (completion $%g + internal reasoning $%g), $%g per request.\n\n",- p.Model, p.Source, p.FetchedAt.Format(time.RFC3339), over,+ role := "Model"+ if i == 1 {+ role = "Second model (agree_with)"+ }+ fmt.Fprintf(&b, "%s %s, priced at run time (%s, %s, %s): $%g per million input tokens, $%g per million output tokens (completion $%g + internal reasoning $%g), $%g per request.\n\n",+ role, p.Model, p.Source, p.FetchedAt.Format(time.RFC3339), over,PerMillion(p.Prompt), PerMillion(p.Completion, p.Reasoning), PerMillion(p.Completion), PerMillion(p.Reasoning), p.Request)}b.WriteString("| family | cases | correct (exact) | wrong | rejected by checks | not sent | model error | not run (cap) | stopped | abstained | error rate |\n")@@ -248,6 +277,11 @@ func (rep Report) WriteMarkdown(w io.Writer) error {rep.Total.EqualsOurs, rep.Total.Accepted(), rep.equalsOurs(OutcomeCorrect), rep.equalsOurs(OutcomeWrong))fmt.Fprintf(&b, "\nRegions of the accepted model answers answered with OURS' region unchanged: %d of %d.\n",rep.Total.RegionsEqualOurs, rep.Total.Regions)+ if t := rep.Total; t.Held > 0 || rep.AgreePricing != nil {+ landed := t.Accepted() - t.Held+ fmt.Fprintf(&b, "\nagree_with gate: %d accepted answers, %d held back as proposals (the second model did not agree; %d of them wrong), %d would land through verify (%d wrong, %.1f%%).\n",+ t.Accepted(), t.Held, t.HeldWrong, landed, t.Wrong-t.HeldWrong, percent(t.Wrong-t.HeldWrong, landed))+ }if t := rep.Total; t.Calls > 0 {fmt.Fprintf(&b, "\nModel calls: %d, of which %d repair rounds driven by deterministic findings.\n", t.Calls, t.Repairs)}diff --git a/tools/graft-agents/resolve/bench/run.go b/tools/graft-agents/resolve/bench/run.goindex 15d7c83..2d9dec7 100644--- a/tools/graft-agents/resolve/bench/run.go+++ b/tools/graft-agents/resolve/bench/run.go@@ -73,6 +73,10 @@ type Result struct {Providers map[string]int `json:"providers,omitempty"`Repairs int `json:"repairs,omitempty"`Report []string `json:"report,omitempty"`+ // Hold is why the resolution of an accepted answer is held back as a proposal (agree_with: the+ // second model did not agree). The server never lands a held file; the report counts the+ // held answers apart from the ones that would land through verify.+ Hold string `json:"hold,omitempty"`// LineEndings are those of the sides of a content conflict (resolve.LLMLineEndings): the llm// strategy sends a CRLF file as LF, and a mixed one as it is.LineEndings string `json:"line_endings,omitempty"`@@ -283,6 +287,7 @@ func (r Runner) runCase(ctx context.Context, c Case, maxBytes int64) (Result, erres.Strategy = strings.TrimPrefix(resp.Resolver, "graft-resolve/")res.Answer = f.Actionres.Report = f.Report+ res.Hold = f.Holdif r.AnswersDir != "" && f.Action != protocol.ActionDelete {if err := os.WriteFile(answerPath(r.AnswersDir, c.ID), f.Content, 0o600); err != nil {return res, errdiff --git a/tools/graft-agents/resolve/budget_strategy.go b/tools/graft-agents/resolve/budget_strategy.goindex 7b378f9..e97a538 100644--- a/tools/graft-agents/resolve/budget_strategy.go+++ b/tools/graft-agents/resolve/budget_strategy.go@@ -91,9 +91,17 @@ func llmBudgetConfig(spec StrategySpec, tbl map[string]any) (*Budget, error) {// UseBudget makes every llm strategy of r go through b, in place of a configured budget, and// returns how many there are. It must be called before the first Resolve. Each llm strategy-// needs max_tokens.+// needs max_tokens. A strategy with an agree_with model has two prices: UseBudgets.func (r *Resolver) UseBudget(b *Budget) (int, error) {- if b == nil {+ return r.UseBudgets(b, nil)+}++// UseBudgets makes the first model of every llm strategy of r go through first and the second+// model of the strategies that have an agree_with go through second, in place of the configured+// budgets, and returns how many strategies there are. A strategy with an agree_with needs second:+// a cap that covers one of two models would not be a cap.+func (r *Resolver) UseBudgets(first, second *Budget) (int, error) {+ if first == nil {return 0, errors.New("resolve: nil budget")}n := 0@@ -105,8 +113,18 @@ func (r *Resolver) UseBudget(b *Budget) (int, error) {if s.opt.MaxTokens <= 0 {return 0, fmt.Errorf("strategy %q: a budget needs max_tokens in [strategy.llm]", bd.Name)}- s.budget = b- s.provider = budgetProvider{p: s.base, b: b}+ if s.agree != nil && s.agree.opt.MaxTokens <= 0 {+ return 0, fmt.Errorf("strategy %q: a budget needs max_tokens in [strategy.llm.agree_with]", bd.Name)+ }+ if s.agree != nil && second == nil {+ return 0, fmt.Errorf("strategy %q: it has an agree_with model, which has its own price: give a budget for each model", bd.Name)+ }+ s.budget = first+ s.provider = budgetProvider{p: s.base, b: first}+ if s.agree != nil {+ s.agree.budget = second+ s.agree.provider = budgetProvider{p: s.agree.base, b: second}+ }n++}return n, nil@@ -123,6 +141,18 @@ func (r *Resolver) LLMModels() map[string]string {return out}+// LLMAgreeModels returns the configured agree_with model of each llm strategy of r that has one,+// by strategy name.+func (r *Resolver) LLMAgreeModels() map[string]string {+ out := map[string]string{}+ for _, bd := range r.bindings {+ if s, ok := bd.Strategy.(*llmStrategy); ok && s.agree != nil {+ out[bd.Name] = s.agree.model+ }+ }+ return out+}+// LLMRoutings returns the OpenRouter routing of each llm strategy of r, by strategy name; nil for// a strategy without one.func (r *Resolver) LLMRoutings() map[string]*llm.Routing {@@ -135,6 +165,17 @@ func (r *Resolver) LLMRoutings() map[string]*llm.Routing {return out}+// LLMAgreeRoutings is LLMRoutings for the agree_with model of the strategies that have one.+func (r *Resolver) LLMAgreeRoutings() map[string]*llm.Routing {+ out := map[string]*llm.Routing{}+ for _, bd := range r.bindings {+ if s, ok := bd.Strategy.(*llmStrategy); ok && s.agree != nil {+ out[bd.Name] = s.agree.routing+ }+ }+ return out+}+// LLMMaxRegions returns the largest max_regions of r's llm strategies, or 0 when r has none.func (r *Resolver) LLMMaxRegions() int {n := 0@@ -160,13 +201,21 @@ func (r *Resolver) LLMMaxPromptTokens() int {return n}-// Budgets returns the budgets of r's strategies, by strategy name.+// Budgets returns the budgets of r's strategies, by strategy name; the budget of the agree_with+// model of a strategy is under its name followed by "/agree_with".func (r *Resolver) Budgets() map[string]*Budget {out := map[string]*Budget{}for _, bd := range r.bindings {- if s, ok := bd.Strategy.(*llmStrategy); ok && s.budget != nil {+ s, ok := bd.Strategy.(*llmStrategy)+ if !ok {+ continue+ }+ if s.budget != nil {out[bd.Name] = s.budget}+ if s.agree != nil && s.agree.budget != nil {+ out[bd.Name+agreeSuffix] = s.agree.budget+ }}return out}diff --git a/tools/graft-agents/resolve/llm_agree_test.go b/tools/graft-agents/resolve/llm_agree_test.gonew file mode 100644index 0000000..a32be0a--- /dev/null+++ b/tools/graft-agents/resolve/llm_agree_test.go@@ -0,0 +1,269 @@+//go:build cgo++package resolve++import (+ "context"+ "errors"+ "regexp"+ "strings"+ "testing"++ "github.com/feder-cr/git-ai/tools/graft-agents/llm"+ "github.com/feder-cr/git-ai/tools/graft-agents/protocol"+)++// The tests of the agree_with gate (PLAN10 15.13): two models resolve each file with the prompt of+// record, and the resolution is held back as a proposal unless they agree.++// agreeB is the provider handed out for the agree_with table of the tests.+var agreeB = &wp19Fake{answer: func(context.Context, llm.Request) (*llm.Response, error) {+ return wp19Answer("unused\n"), nil+}}++func init() {+ llm.Register("wp19-agree-b", func(llm.Config) (llm.Provider, error) { return agreeB, nil })+}++func agreeSetB(fn func(ctx context.Context, req llm.Request) (*llm.Response, error)) *wp19Fake {+ agreeB = &wp19Fake{answer: fn}+ return agreeB+}++const agreeConfig = wp19LLMConfig + `+[strategy.llm.agree_with]+type = "wp19-agree-b"+model = "model-b"+max_tokens = 256++[strategy.llm.agree_with.routing]+only = ["z-ai/fp8"]+`++func agreeAnswer(text string) func(context.Context, llm.Request) (*llm.Response, error) {+ return func(context.Context, llm.Request) (*llm.Response, error) { return wp19Answer(text), nil }+}++func agreeFail(msg string) func(context.Context, llm.Request) (*llm.Response, error) {+ return func(context.Context, llm.Request) (*llm.Response, error) { return nil, errors.New(msg) }+}++// resolveOne resolves a.txt with the agree config and returns the file the response holds.+func resolveOne(t *testing.T) (protocol.ResolvedFile, error) {+ t.Helper()+ r := wp19Build(t, agreeConfig)+ resp, err := r.Resolve(context.Background(), wp19Request(wp19Conflict("a.txt")))+ if err != nil {+ return protocol.ResolvedFile{}, err+ }+ if len(resp.Files) != 1 {+ t.Fatalf("files = %+v", resp.Files)+ }+ return resp.Files[0], nil+}++func TestAgreeWithTwoModelsThatAgreeAreNotHeld(t *testing.T) {+ a := wp19SetAnswer(agreeAnswer("merged\n"))+ b := agreeSetB(agreeAnswer("merged\n"))+ got, err := resolveOne(t)+ if err != nil {+ t.Fatal(err)+ }+ if got.Hold != "" || string(got.Content) != "merged\n" || got.Origin != protocol.OriginModel || !got.Proposal {+ t.Fatalf("file = %+v", got)+ }+ if a.calls() != 1 || b.calls() != 1 {+ t.Fatalf("calls = %d, %d", a.calls(), b.calls())+ }+ // Both models are asked with the same prompt of record.+ nonce := regexp.MustCompile("[0-9a-f]{24}")+ strip := func(r llm.Request) string { return nonce.ReplaceAllString(r.System+"\n"+r.Prompt, "NONCE") }+ if strip(a.lastReq()) != strip(b.lastReq()) {+ t.Fatal("the two models were not given the same prompt")+ }+}++func TestAgreeWithNormalizesLikeTheJudge(t *testing.T) {+ wp19SetAnswer(agreeAnswer("merged\n"))+ agreeSetB(agreeAnswer("merged \n\n"))+ got, err := resolveOne(t)+ if err != nil {+ t.Fatal(err)+ }+ if got.Hold != "" {+ t.Fatalf("trailing whitespace and blank lines hold the file: %q", got.Hold)+ }+}++func TestAgreeWithDisagreementHoldsTheFirstModelsProposal(t *testing.T) {+ wp19SetAnswer(agreeAnswer("merged\n"))+ b := agreeSetB(agreeAnswer("something else\n"))+ got, err := resolveOne(t)+ if err != nil {+ t.Fatal(err)+ }+ if !strings.Contains(got.Hold, "model-b") || !strings.Contains(got.Hold, "differently") {+ t.Fatalf("hold = %q", got.Hold)+ }+ if string(got.Content) != "merged\n" || got.Origin != protocol.OriginModel || !got.Proposal {+ t.Fatalf("the first model's proposal is not what is held: %+v", got)+ }+ if last := got.Report[len(got.Report)-1]; !strings.Contains(last, "the second model disagreed") {+ t.Fatalf("report = %v", got.Report)+ }+ if b.calls() != 1 {+ t.Fatalf("calls of the second model = %d", b.calls())+ }+}++func TestAgreeWithASecondModelThatFailsHolds(t *testing.T) {+ wp19SetAnswer(agreeAnswer("merged\n"))+ agreeSetB(agreeFail("provider down"))+ got, err := resolveOne(t)+ if err != nil {+ t.Fatal(err)+ }+ if !strings.Contains(got.Hold, "gave no usable answer") || !strings.Contains(got.Hold, "provider down") {+ t.Fatalf("hold = %q", got.Hold)+ }+ if string(got.Content) != "merged\n" {+ t.Fatalf("content = %q", got.Content)+ }+}++func TestAgreeWithAFirstModelThatFailsFailsTheFile(t *testing.T) {+ wp19SetAnswer(agreeFail("provider down"))+ b := agreeSetB(agreeAnswer("merged\n"))+ _, err := resolveOne(t)+ var unresolved *UnresolvedError+ if !errors.As(err, &unresolved) {+ t.Fatalf("err = %v", err)+ }+ // The second model was asked at the same time and its call is waited for: the cost is real.+ if b.calls() != 1 {+ t.Fatalf("calls of the second model = %d", b.calls())+ }+}++func TestAgreeWithAnUncleanReportHoldsEvenWhenTheModelsAgree(t *testing.T) {+ r := wp19Build(t, agreeConfig)+ s := r.bindings[0].Strategy.(*llmStrategy)+ clean := LLMResult{Content: []byte("x\n"), Report: []string{"keeps every uncontested change of both sides (2)"}}+ dirty := LLMResult{Content: []byte("x\n"), Report: []string{"region 1: drops theirs' change: the line \"y\" it adds is missing"}}+ repeats := LLMResult{Content: []byte("x\n"), Report: []string{"region 2: duplicates a line both sides added: \"z\""}}+ run := func(res LLMResult) *agreeRun { return &agreeRun{res: res} }++ if why := s.agreement(clean, run(clean)); why != "" {+ t.Fatalf("clean reports and the same file are held: %q", why)+ }+ for name, c := range map[string]struct {+ first LLMResult+ second LLMResult+ }{+ "first drops": {dirty, clean},+ "second drops": {clean, dirty},+ "first duplicates": {repeats, clean},+ "both drop": {dirty, dirty},+ } {+ if why := s.agreement(c.first, run(c.second)); why == "" || !strings.Contains(why, "finding") {+ t.Fatalf("%s: not held: %q", name, why)+ }+ }+ if why := s.agreement(clean, &agreeRun{err: ErrLLMNotApplicable}); !strings.Contains(why, "could not take") {+ t.Fatalf("not applicable: %q", why)+ }+}++func TestAgreeWithCountsTheCallsOfBothModels(t *testing.T) {+ wp19SetAnswer(func(context.Context, llm.Request) (*llm.Response, error) {+ return &llm.Response{Text: "merged\n", Model: "a", Usage: llm.Usage{InputTokens: 10, OutputTokens: 1}, Provider: "pa"}, nil+ })+ agreeSetB(func(context.Context, llm.Request) (*llm.Response, error) {+ return &llm.Response{Text: "merged\n", Model: "b", Usage: llm.Usage{InputTokens: 20, OutputTokens: 2}, Provider: "pb"}, nil+ })+ stats := &LLMRegionStats{}+ r := wp19Build(t, agreeConfig)+ if _, err := r.Resolve(WithLLMRegionStats(context.Background(), stats), wp19Request(wp19Conflict("a.txt"))); err != nil {+ t.Fatal(err)+ }+ if u := stats.Usage(); u.InputTokens != 30 || u.OutputTokens != 3 {+ t.Fatalf("usage = %+v", u)+ }+ if calls, _ := stats.Calls(); calls != 2 {+ t.Fatalf("calls = %d", calls)+ }+ if p := stats.Providers(); p["pa"] != 1 || p["pb"] != 1 {+ t.Fatalf("providers = %v", p)+ }+}++func TestAgreeWithConfig(t *testing.T) {+ for name, c := range map[string]struct {+ config string+ want string+ }{+ "agree_with of agree_with": {+ agreeConfig + "\n[strategy.llm.agree_with.agree_with]\ntype = \"wp19-agree-b\"\n",+ "unknown keys",+ },+ "an unknown key": {+ strings.Replace(agreeConfig, "max_tokens = 256\n", "max_tokens = 256\nmodle = \"x\"\n", 1),+ "llm.agree_with: unknown keys",+ },+ "a budget for the first model only": {+ strings.Replace(wp19LLMConfig, "[strategy.llm.options]", "[strategy.llm.budget]\nmonthly_usd = 5\ninput_usd_per_million = 1\noutput_usd_per_million = 1\nstate = \"/tmp/x-agree-budget.json\"\n\n[strategy.llm.options]", 1) + "\n[strategy.llm.agree_with]\ntype = \"wp19-agree-b\"\nmodel = \"model-b\"\nmax_tokens = 256\n",+ "go together",+ },+ "no provider type": {+ wp19LLMConfig + "\n[strategy.llm.agree_with]\nmodel = \"model-b\"\n",+ "type is required",+ },+ } {+ _, err := wp19BuildErr(c.config)+ if err == nil || !strings.Contains(err.Error(), c.want) {+ t.Errorf("%s: err = %v, want %q", name, err, c.want)+ }+ }+ // Without the table the gate is off: one model, no hold.+ wp19SetAnswer(agreeAnswer("merged\n"))+ r := wp19Build(t, wp19LLMConfig)+ if r.bindings[0].Strategy.(*llmStrategy).agree != nil {+ t.Fatal("the gate is on without agree_with")+ }+}++func TestLLMReportClean(t *testing.T) {+ for report, want := range map[string]bool{+ "": true,+ "keeps every uncontested change of both sides (3)": true,+ "2 contested changes resolved freely: nothing line-level to verify": true,+ "1 of 2 regions were answered again after findings of the deterministic checks": true,+ "region 1: drops ours' change 3-4: the line \"a\" it adds is missing": false,+ "region 2: duplicates a line both sides added: \"b\"": false,+ } {+ var lines []string+ if report != "" {+ lines = []string{report}+ }+ if got := LLMReportClean(lines); got != want {+ t.Errorf("LLMReportClean(%q) = %v", report, got)+ }+ }+ if !LLMReportClean([]string{"a", "b", "region 3: the answer has lines found in neither side"}) {+ t.Error("a line found in neither side is a free resolution, not a drop")+ }+}++func TestLLMNormalize(t *testing.T) {+ for in, want := range map[string]string{+ "a\r\nb \r\n\r\n": "a\nb",+ "a\n\n\n": "a",+ " a\t\n": " a",+ "": "",+ "\n": "",+ } {+ if got := string(LLMNormalize([]byte(in))); got != want {+ t.Errorf("LLMNormalize(%q) = %q, want %q", in, got, want)+ }+ }+}diff --git a/tools/graft-agents/resolve/llm_core.go b/tools/graft-agents/resolve/llm_core.goindex c20eecb..53f4d74 100644--- a/tools/graft-agents/resolve/llm_core.go+++ b/tools/graft-agents/resolve/llm_core.go@@ -245,6 +245,67 @@ func (s *LLMRegionStats) Get() (regions, equalOurs int, ok bool) {return s.regions, s.equalOurs, s.recorded}+// Absorb adds the usage, the calls, the providers and the rejected answers of o to s: the cost of+// the second model of the agree_with gate belongs to the file it helped to resolve. The rejected+// answers carry prefix. The counts of the last file and its report stay s's own.+func (s *LLMRegionStats) Absorb(o *LLMRegionStats, prefix string) {+ if o == nil {+ return+ }+ o.mu.Lock()+ usage, calls, repairs := o.usage, o.calls, o.repairs+ providers := make(map[string]int, len(o.providers))+ for k, v := range o.providers {+ providers[k] = v+ }+ rejected := append([]string(nil), o.rejected...)+ o.mu.Unlock()++ s.mu.Lock()+ defer s.mu.Unlock()+ s.usage.InputTokens += usage.InputTokens+ s.usage.OutputTokens += usage.OutputTokens+ s.usage.ReasoningTokens += usage.ReasoningTokens+ s.calls += calls+ s.repairs += repairs+ if len(providers) > 0 && s.providers == nil {+ s.providers = map[string]int{}+ }+ for k, v := range providers {+ s.providers[k] += v+ }+ for _, r := range rejected {+ s.rejected = append(s.rejected, prefix+r)+ }+}++// LLMReportClean reports whether a composition report holds no finding: no line says that the answer+// drops a change of a side or repeats a line both added (llmReport words them first). It is the+// "clean report" of the agree_with gate (PLAN10 15.12): a report is not a gate for a single model,+// and is one of the two conditions under which two models' agreement is trusted.+func LLMReportClean(report []string) bool {+ for _, line := range report {+ if strings.Contains(line, "drops") || strings.Contains(line, "duplicates") {+ return false+ }+ }+ return true+}++// LLMNormalize is a file as the agree_with gate and the judges compare it: line endings as LF, no+// whitespace at the end of a line, no blank line at the end of the file. The bench normalizes+// with it too, so that "the same answer" means the same thing everywhere.+func LLMNormalize(b []byte) []byte {+ lines := strings.Split(strings.ReplaceAll(string(b), "\r\n", "\n"), "\n")+ for i, l := range lines {+ lines[i] = strings.TrimRight(l, " \t\r")+ }+ for len(lines) > 0 && lines[len(lines)-1] == "" {+ lines = lines[:len(lines)-1]+ }+ return []byte(strings.Join(lines, "\n"))+}+type llmRegionStatsKey struct{}// WithLLMRegionStats returns ctx carrying s, for the llm strategy to record into.diff --git a/tools/graft-agents/resolve/llm_strategy.go b/tools/graft-agents/resolve/llm_strategy.goindex 95c94c1..a75c1fd 100644--- a/tools/graft-agents/resolve/llm_strategy.go+++ b/tools/graft-agents/resolve/llm_strategy.go@@ -1,10 +1,13 @@package resolveimport (+ "bytes""context""errors""fmt"+ "sort""time"+ "unicode/utf8""github.com/feder-cr/git-ai/tools/graft-agents/llm""github.com/feder-cr/git-ai/tools/graft-agents/protocol"@@ -39,6 +42,18 @@ import (// [strategy.llm.budget] # optional: monthly spending cap (budget_strategy.go)// ...//+// [strategy.llm.agree_with] # optional, off by default: the agreement gate (PLAN10 15.13)+// type = "openai" # a second model, with the keys of [strategy.llm] (not agree_with)+// model = "..."+//+// With agree_with, both models resolve each file at the same time with the same prompt, and the+// first model's resolution is proposed as always (it lands only through verify.command). It is+// HELD, and never lands, unless the two results are the same file after normalization+// (LLMNormalize) and both composition reports are clean (LLMReportClean): the response carries+// Hold, the report says that the second model disagreed, and the server neither verifies nor+// lands the file. A model that fails makes the file unresolved when it is the first, and holds it+// when it is the second. A budget on one model needs a budget on the other.+//// The strategy declines (the chain continues, no model call is made) for files// it cannot merge: a side that is absent, a symlink, a mode conflict, binary// content, a diff too large to split (the LCS table of the changed middle of@@ -60,6 +75,14 @@ func init() {// llmStrategy is immutable after construction. Resolve may run concurrently// (HTTP mode), so the provider must be safe for concurrent use.type llmStrategy struct {+ llmSide+ // agree is the second model of the agree_with gate, or nil.+ agree *llmSide+}++// llmSide is one model of the strategy: its provider, behind the budget if there is one, and the+// options of its calls.+type llmSide struct {// provider is base, or base behind the budget.provider llm.Providerbase llm.Provider@@ -71,6 +94,9 @@ type llmStrategy struct {timeout time.Duration}+// agreeSuffix names the second model of a strategy where the strategy name keys a map (budgets).+const agreeSuffix = "/agree_with"+func newLLMStrategy(spec StrategySpec) (Strategy, error) {// Every llm answer gets the syntax check, which needs the tree-sitter grammars: a build// without them would reject every answer for those languages, so it does not start.@@ -104,7 +130,36 @@ func newLLMStrategy(spec StrategySpec) (Strategy, error) {}}- cfg, err := llmProviderConfig(spec)+ tbl, err := llmTable(spec)+ if err != nil {+ return nil, err+ }+ side, err := newLLMSide(spec, tbl, "llm", opt)+ if err != nil {+ return nil, err+ }+ s := &llmStrategy{llmSide: *side}++ if raw, ok := tbl["agree_with"]; ok {+ atbl, ok := raw.(map[string]any)+ if !ok {+ return nil, fmt.Errorf("strategy %q: llm.agree_with must be a table", spec.Name)+ }+ if s.agree, err = newLLMSide(spec, atbl, "llm.agree_with", opt); err != nil {+ return nil, err+ }+ // A spending cap that covers one of the two models would not be a cap.+ if (s.budget == nil) != (s.agree.budget == nil) {+ return nil, fmt.Errorf("strategy %q: llm.budget and llm.agree_with.budget go together: the cap must cover both models", spec.Name)+ }+ }+ return s, nil+}++// newLLMSide builds one model of the strategy from its table (where names it in errors). opt holds+// the options of the strategy; the side's own max_tokens and timeout replace the ones in it.+func newLLMSide(spec StrategySpec, tbl map[string]any, where string, opt LLMOptions) (*llmSide, error) {+ cfg, err := llmProviderConfigOf(spec, tbl, where)if err != nil {return nil, err}@@ -117,29 +172,29 @@ func newLLMStrategy(spec StrategySpec) (Strategy, error) {timeout, err := cfg.TimeoutOrDefault()if err != nil {- return nil, fmt.Errorf("strategy %q: %w", spec.Name, err)+ return nil, fmt.Errorf("strategy %q: %s: %w", spec.Name, where, err)}p, err := llm.New(cfg)if errors.Is(err, llm.ErrDisabled) {- return nil, fmt.Errorf("strategy %q: the llm table needs a provider type", spec.Name)+ return nil, fmt.Errorf("strategy %q: the %s table needs a provider type", spec.Name, where)}if err != nil {- return nil, fmt.Errorf("strategy %q: %w", spec.Name, err)+ return nil, fmt.Errorf("strategy %q: %s: %w", spec.Name, where, err)}- s := &llmStrategy{provider: p, base: p, model: cfg.Model, routing: cfg.Routing, opt: opt, timeout: timeout}+ side := &llmSide{provider: p, base: p, model: cfg.Model, routing: cfg.Routing, opt: opt, timeout: timeout}- budget, err := llmBudgetConfig(spec, spec.Options["llm"].(map[string]any))+ budget, err := llmBudgetConfig(spec, tbl)if err != nil {return nil, err}if budget != nil {if opt.MaxTokens <= 0 {- return nil, fmt.Errorf("strategy %q: llm.budget needs max_tokens in [strategy.llm]: it bounds the worst case of a call", spec.Name)+ return nil, fmt.Errorf("strategy %q: %s.budget needs max_tokens in [strategy.%s]: it bounds the worst case of a call", spec.Name, where, where)}- s.budget = budget- s.provider = budgetProvider{p: p, b: budget}+ side.budget = budget+ side.provider = budgetProvider{p: p, b: budget}}- return s, nil+ return side, nil}// Resolve implements Strategy.@@ -161,20 +216,104 @@ func (s *llmStrategy) Resolve(ctx context.Context, f File, req protocol.Request)lf.Base = llmBytes(f.Base.Content)}- ctx, cancel := context.WithTimeout(ctx, s.timeout)- defer cancel()+ stats := llmRegionStatsFrom(ctx)+ var second *agreeRun+ if s.agree != nil {+ // Both models resolve the file with the same prompt of record, at the same time.+ second = s.agree.start(ctx, lf, stats != nil)+ }- res, err := ResolveLLM(ctx, s.provider, lf, s.opt)+ first, cancel := context.WithTimeout(ctx, s.timeout)+ defer cancel()+ res, err := ResolveLLM(first, s.provider, lf, s.opt)+ if second != nil {+ // The second model is waited for whatever the first one did: its calls are paid for, and+ // its usage belongs to the file.+ second.wait()+ if stats != nil {+ stats.Absorb(second.stats, "agree_with: ")+ }+ }switch {case errors.Is(err, ErrLLMNotApplicable):return Resolution{}, false, nilcase err != nil:return Resolution{}, false, err}- if stats := llmRegionStatsFrom(ctx); stats != nil {+ if stats != nil {stats.Record(res.Hunks, res.RegionsEqualOurs)}- return Resolution{Action: protocol.ActionWrite, Mode: f.Ours.Mode, Content: res.Content, Report: res.Report}, true, nil+ out := Resolution{Action: protocol.ActionWrite, Mode: f.Ours.Mode, Content: res.Content, Report: res.Report}+ if second != nil {+ if why := s.agreement(res, second); why != "" {+ out.Hold = why+ out.Report = append(append([]string(nil), res.Report...), "the second model disagreed, so the resolution is held as a proposal: "+why)+ }+ }+ return out, true, nil+}++// agreeRun is the second model resolving a file, started by agree.start.+type agreeRun struct {+ done chan struct{}+ res LLMResult+ err error+ stats *LLMRegionStats+}++// start resolves lf with the side's model in the background. When counting, the run keeps its own+// stats, which the caller absorbs into its own once the run is waited for.+func (s *llmSide) start(ctx context.Context, lf LLMFile, counting bool) *agreeRun {+ r := &agreeRun{done: make(chan struct{})}+ if counting {+ r.stats = &LLMRegionStats{}+ ctx = WithLLMRegionStats(ctx, r.stats)+ }+ go func() {+ defer close(r.done)+ defer func() {+ if p := recover(); p != nil {+ r.err = fmt.Errorf("resolve: the second model panicked: %v", p)+ }+ }()+ ctx, cancel := context.WithTimeout(ctx, s.timeout)+ defer cancel()+ r.res, r.err = ResolveLLM(ctx, s.provider, lf, s.opt)+ }()+ return r+}++// wait blocks until the run is over.+func (r *agreeRun) wait() { <-r.done }++// agreement says why the first model's resolution is not the one both models give, or "" when the+// two agree: the same file after the judge's normalization (LLMNormalize), and neither report+// holds a finding (LLMReportClean). The first model's answer is the one that is proposed.+func (s *llmStrategy) agreement(first LLMResult, second *agreeRun) string {+ switch {+ case second.err != nil && errors.Is(second.err, ErrLLMNotApplicable):+ return fmt.Sprintf("the second model (%s) could not take the file", s.agree.model)+ case second.err != nil:+ return fmt.Sprintf("the second model (%s) gave no usable answer: %s", s.agree.model, llmCutText(second.err.Error(), 200))+ case !LLMReportClean(first.Report):+ return fmt.Sprintf("the composition report of the first model (%s) holds a finding", s.model)+ case !LLMReportClean(second.res.Report):+ return fmt.Sprintf("the composition report of the second model (%s) holds a finding", s.agree.model)+ case !bytes.Equal(LLMNormalize(first.Content), LLMNormalize(second.res.Content)):+ return fmt.Sprintf("the second model (%s) resolved the file differently from the first (%s)", s.agree.model, s.model)+ }+ return ""+}++// llmCutText cuts s to n bytes at a rune boundary.+func llmCutText(s string, n int) string {+ if len(s) <= n {+ return s+ }+ for n > 0 && !utf8.RuneStart(s[n]) {+ n--+ }+ return s[:n] + "..."}// syntaxChecked marks the llm answers for the syntax check (resolveFile).@@ -189,27 +328,37 @@ func llmBytes(b []byte) []byte {return b}-// llmProviderConfig decodes the [strategy.llm] table.-func llmProviderConfig(spec StrategySpec) (llm.Config, error) {+// llmTable returns the [strategy.llm] table.+func llmTable(spec StrategySpec) (map[string]any, error) {raw, ok := spec.Options["llm"]if !ok {- return llm.Config{}, fmt.Errorf("strategy %q: missing [strategy.llm] table", spec.Name)+ return nil, fmt.Errorf("strategy %q: missing [strategy.llm] table", spec.Name)}tbl, ok := raw.(map[string]any)if !ok {- return llm.Config{}, fmt.Errorf("strategy %q: llm must be a table", spec.Name)+ return nil, fmt.Errorf("strategy %q: llm must be a table", spec.Name)}+ return tbl, nil+}+// llmProviderConfigOf decodes a provider table: [strategy.llm], or its agree_with table, which has+// the same keys but no agree_with of its own (where is "llm" or "llm.agree_with").+func llmProviderConfigOf(spec StrategySpec, tbl map[string]any, where string) (llm.Config, error) {var unknown []stringfor k := range tbl {switch k {case "type", "model", "base_url", "api_key_env", "command", "timeout", "max_tokens", "effort", "options", "routing", "budget":+ case "agree_with":+ if where != "llm" {+ unknown = append(unknown, k)+ }default:unknown = append(unknown, k)}}if len(unknown) > 0 {- return llm.Config{}, fmt.Errorf("strategy %q: llm: unknown keys %v", spec.Name, unknown)+ sort.Strings(unknown)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: unknown keys %v", spec.Name, where, unknown)}var cfg llm.Config@@ -223,40 +372,40 @@ func llmProviderConfig(spec StrategySpec) (llm.Config, error) {}for _, s := range strs {if *s.dst, err = stringField(tbl, s.key); err != nil {- return llm.Config{}, fmt.Errorf("strategy %q: llm: %w", spec.Name, err)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: %w", spec.Name, where, err)}}if cfg.Command, err = stringListField(tbl, "command"); err != nil {- return llm.Config{}, fmt.Errorf("strategy %q: llm: %w", spec.Name, err)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: %w", spec.Name, where, err)}if v, ok := tbl["max_tokens"]; ok {n, ok := v.(int64)if !ok || n < 0 || n > 1<<20 {- return llm.Config{}, fmt.Errorf("strategy %q: llm: max_tokens must be an integer between 0 and %d", spec.Name, 1<<20)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: max_tokens must be an integer between 0 and %d", spec.Name, where, 1<<20)}cfg.MaxTokens = int(n)}if v, ok := tbl["options"]; ok {m, ok := v.(map[string]any)if !ok {- return llm.Config{}, fmt.Errorf("strategy %q: llm: options must be a table", spec.Name)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: options must be a table", spec.Name, where)}cfg.Options = make(map[string]string, len(m))for k, val := range m {s, ok := val.(string)if !ok {- return llm.Config{}, fmt.Errorf("strategy %q: llm: option %q must be a string", spec.Name, k)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: option %q must be a string", spec.Name, where, k)}cfg.Options[k] = s}}if v, ok := tbl["routing"]; ok {if cfg.Routing, err = routingField(v); err != nil {- return llm.Config{}, fmt.Errorf("strategy %q: llm: routing: %w", spec.Name, err)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: routing: %w", spec.Name, where, err)}}if cfg.Type == "" {- return llm.Config{}, fmt.Errorf("strategy %q: llm: type is required", spec.Name)+ return llm.Config{}, fmt.Errorf("strategy %q: %s: type is required", spec.Name, where)}return cfg, nil}diff --git a/tools/graft-agents/resolve/resolve.go b/tools/graft-agents/resolve/resolve.goindex 84644a9..6f271ea 100644--- a/tools/graft-agents/resolve/resolve.go+++ b/tools/graft-agents/resolve/resolve.go@@ -57,6 +57,10 @@ type Resolution struct {Content []byte// Report is the composition report that goes with a model's resolution (ResolvedFile.Report).Report []string+ // Hold is why the resolution is held back as a proposal even if it would pass the server's+ // verification (ResolvedFile.Hold): the second model of the llm strategy's agree_with did not+ // agree with the first. Empty for a resolution that is not held.+ Hold string}// Strategy resolves conflicts. Implementations must be safe for concurrent@@ -371,9 +375,11 @@ func (r *Resolver) Resolve(ctx context.Context, req protocol.Request) (protocol.// The month's AI spend of the strategies that answered, so that command mode reports it too.budgets := r.Budgets()for _, name := range used {- if b := budgets[name]; b != nil {- if snap, err := b.Snapshot(); err == nil {- resp.Explanation += "; " + name + ": " + snap.String()+ for _, key := range []string{name, name + agreeSuffix} {+ if b := budgets[key]; b != nil {+ if snap, err := b.Snapshot(); err == nil {+ resp.Explanation += "; " + key + ": " + snap.String()+ }}}}@@ -493,6 +499,7 @@ func (r *Resolver) resolveFile(ctx context.Context, req protocol.Request, f FileContent: contentOf(res),Origin: origin,Proposal: proposal,+ Hold: res.Hold,Report: res.Report,}, b.Name, nil, nil}