Git designed for AI agents.
docs: PLAN10 15.12 with a third judge from another vendor; the rubric (Appendix B)
Federico Elia2fdc52165dda
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.mdindex 53cadea..2f13307 100644--- a/doc/graft/PLAN10.md+++ b/doc/graft/PLAN10.md@@ -941,7 +941,7 @@ end of this section.| writer | proposes a resolution per region | a model, configurable (15.13) | 15.10: the elements kept and removed, 15.12 || checks and repair | refuse a broken answer, repair it once | deterministic checks | 15.10 || report | tells the agent what the answer keeps or drops | deterministic | 15.6: it separates (43% against 60% right), it does not gate |-| agree_with (optional) | holds a proposal back unless a second model from another vendor gives the same file and both reports are clean | code (`llmStrategy`), configured by the operator | 15.12: G1, 59.8% of the files landed with 15.5% wrong under two judges, before the operator's tests |+| agree_with (optional) | holds a proposal back unless a second model from another vendor gives the same file and both reports are clean | code (`llmStrategy`), configured by the operator | 15.12: G1, 59.8% of the files landed with 15.5% wrong under two judges and 17.5% under a third, before the operator's tests || verify | runs the operator's build and tests on the exact tree | the operator's command | 15.9, 14 || decideLand | the one landing rule | code | 15.4 |@@ -1670,10 +1670,15 @@ spring-boot capped at 30; python 13, go 100, ts-js 100, other java 1); `tune200`### 15.12 The operating point-**The goal and the answer.** Federico's goal: at least 20% of the conflict files resolved and landed automatically, with at-most 5% wrong among them (Wilson 95% upper bound), on data the gates were not chosen on, under the strict judge and under-judge A, OSS and agent PRs apart. **It is not met, by any gate, on either kind of data.** What follows is the operating-point that the evidence supports, the gates that were tried, and the reading.+**The goal and the answer.** Federico's goal was: at least 20% of the conflict files resolved and landed automatically, with+at most 5% wrong among them (Wilson 95% upper bound), on data the gates were not chosen on, under the strict judge and+under judge A, OSS and agent PRs apart. **That goal is not met, by any gate, on either kind of data.** After the first+held-out numbers the bar on the wrong share was restated to 20% (below, "The threshold of 20% wrong"); that bar was set+after the numbers were known, and is not a pre-registered confirmation. Against it, the best gate (G1, two models from+different vendors in agreement, 59.8% of the OSS held-out files landed) has 15.5% wrong under the two first judges, 17.5%+(upper bound 22.3%) under a third judge from another vendor, which is the number to quote, and on conflicts of agent PRs+none comes close (43% to 52% wrong). What follows is the operating point that the evidence supports, the gates that were+tried, and the reading.*The judges.* The strict judge calls an answer right when it equals the human's file after normalization (line endings,trailing whitespace): about a fifth of the answers it calls wrong are valid (whitespace, order, an equivalent@@ -1684,7 +1689,12 @@ accepts fewer (187 against 205 of 270), so "judge A" below is one cheap judge anconservative reading. For the gates on the held-out sets a stronger second judge was added, `openai/gpt-6.1-sol` (effortlow, the same rubric, no failed call), run on the answers that G0 lands: it accepts fewer than the first one (on the OSSheld-out G0 files it finds 74 wrong against the first judge's 56), so the figures below are given for judge 1, judge 2-and "both accept", the last being the one to quote. Neither judge runs tests.+and "both accept". Neither judge runs tests. A third judge was added after the first numbers, from a vendor unrelated to the+two writers and to the first two judges: `mistralai/mistral-large-4-0` (non-reasoning, the same rubric, Appendix B), on the+answers that G1 lands (291 OSS, 130 agent-PR), a strict-equal answer being accepted without a call. It is the strictest+of the three. Its cap ($0.25 per set) stopped it before the end: 11 OSS and 18 agent-PR answers that are not strict-equal+have no verdict, and 15 and 16 of its calls failed; all of them are counted as wrong here, so its figures read high. **Judge+3 alone is the number to quote**, with the majority of judges 1, 2 and 3 beside it.**What the writer's proposals are worth** (the prompt of record, Luna, effort low; "right" = the strict judge; theinterval is Wilson 95%; accepted = the answer passed the rules and the syntax check):@@ -1735,8 +1745,30 @@ ts-js 14.6% (22.6) / 19.2% (27.0), python 6.7% (29.8) / 14.3% (34.6), java 1.7%confirmation; read against it, honestly: on the OSS held-out, G1 has 15.5% wrong under both judges (11.3% under judge 1, 15.1%under judge 2) with an upper bound of 20.1% under both judges (19.7% under judge 2, 15.5% under judge 1), at 59.8% ofthe files landed, so it meets the bar on the point estimates and misses the bound under the conservative reading by 0.1-point; G0 (Luna alone) has 20.9% (upper 25.4%) and does not; without spring-boot G1 has 18.8% (upper 24.3%). On the agent-PR-files no gate comes close: 43.1% wrong at best (upper 51.7%), at 45.8% landed.+point; G0 (Luna alone) has 20.9% (upper 25.4%) and does not; without spring-boot G1 has 18.8% (upper 24.3%). Under the+third judge, from another vendor and the strictest, G1 has 17.5% wrong (upper 22.3%): under the restated bar on the+point estimate, over it on the bound, and 21.4% (upper 27.1%) without spring-boot, so the restated bar is met by neither+reading of the bound and by the point estimates of the whole set only. On the agent-PR files no gate comes close: 43.1%+wrong at best under two judges (upper 51.7%), 51.5% under judge 3 (upper 60.0%), at 45.8% landed.++**G1 under a third judge.** Wrong share among the files G1 lands, judge 3 alone and the majority (at least two of the+three judges accept), Wilson 95% upper bound in parentheses; the answers with no verdict or a failed one count as wrong:++| G1 landed | files | judge 3 alone | 2 of 3 judges |+|---|---:|---|---|+| OSS held-out | 291 | **17.5% (up 22.3)** | 15.1% (up 19.7) |+| OSS held-out without spring-boot | 234 | 21.4% (up 27.1) | 18.4% (up 23.8) |+| go | 115 | 27.0% (up 35.7) | 24.3% (up 32.9) |+| ts-js | 103 | 17.5% (up 25.9) | 13.6% (up 21.5) |+| python | 15 | 6.7% (up 29.8) | 6.7% (up 29.8) |+| rust-java (spring-boot but one file) | 58 | 1.7% (up 9.1) | 1.7% (up 9.1) |+| agent-PR | 130 | **51.5% (up 60.0)** | 42.3% (up 50.9) |++Judge 3 finds 51 of the 291 OSS answers wrong, against 33 for judge 1 and 44 for judge 2; the answers that its cap left+unjudged (11) and the failed calls (15) are among them, so with every one of those accepted the figure would be 8.6%, and+it is 17.5% with every one counted wrong. The conclusion does not depend on them: the point estimate is under the restated+20% on the OSS held-out and the upper bound is over it, and without spring-boot, which is the easy part of the corpus,+both are over.**Reading.*** The statistical bar is high: at most 5% with an upper bound needs about 75 landed files with none wrong, 130 with two,@@ -1834,7 +1866,8 @@ judge's normalization, the composition reports of both are clean, and `verify.coa `hold` reason and the report says that the second model disagreed, and the server never lands a held file, norruns the verification for it. A budget table must be given for both models or for neither. The bench runs the samecode and prices both models. RESOLVE.md has the configuration. Measured on the OSS held-out files it lands 59.8% of-the files with 15.5% of those wrong under two independent judges (upper bound 20.1%), before the operator's tests; on+the files with 15.5% of those wrong under two independent judges (upper bound 20.1%) and 17.5% under a third judge from+another vendor (upper bound 22.3%), before the operator's tests; onconflicts of agent PRs 45.8% of the files with 43.1% wrong. Those numbers were measured as two single-model runscombined offline (the gate decides the same thing on the same answers); the code is covered by tests with fakeproviders and was not run through the bench against the live models. The 20%-wrong bar it is compared with was set@@ -1870,6 +1903,8 @@ read from the key before and after each run and agrees, except where stated.| 20 | held-out runs of the prompt of record: Luna and glm-5.3-flash on 487 OSS files and on 284 agent-PR files | manifest-heldout-487, manifest-agent-284 | 7fd0ab2's tools tree (8812cee) | $0.300 + $0.302 + $0.227 + $0.237 || 20 | judge 1 (glm-5.3-flash) on the held-out answers | the answers of Luna | scripts outside the repository | $0.025 + $0.021 || 20 | judge 2 (openai/gpt-6.1-sol, effort low) on the answers landed by G0 | OSS held-out 359, agent-PR 210 | scripts outside the repository | $0.239 + $0.268 |+| 20 | judge 3 (mistralai/mistral-large-4-0, unrelated vendor) on the answers landed by G1; each set stopped at its cap of $0.25 (11 OSS and 18 agent-PR answers unjudged, 15 and 16 failed calls) | OSS held-out 291, agent-PR 130 | scripts outside the repository | $0.255 + $0.253 (the key: $0.497) |+| 20 | the agree_with gate through fake providers and the real graft-resolve with two fake command models | tests only | 3cc898c | $0 |**Why a screened model is skipped.** The screening is a smoke check of five cases (a go, a java, a python, arust and a ts file) that costs at most a few cents and stops a model whose calls end before an answer: more thandiff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.mdindex 96d7bb0..f2adea7 100644--- a/doc/graft/RESOLVE.md+++ b/doc/graft/RESOLVE.md@@ -734,8 +734,8 @@ server together: an older server does not know `hold` and would verify a held prWhat it was worth, measured on 487 held-out files of 13 open-source repositories (Luna and glm-5.3-flash, with theverify simulation of the bench in place of the operator's tests): 59.8% of the files landed, and 15.5% of those differed-from what two independent reviewing models accept (11.3% for one of them; Wilson upper bound 20.1% for both), against 43%-on conflicts of merged agent PRs. These figures were measured by combining two separate single-model runs, which is what+from what two independent reviewing models accept (11.3% for one of them; Wilson upper bound 20.1% for both), and 17.5% from+what a third one, from another vendor, accepts (upper bound 22.3%), against 43% to 52% on conflicts of merged agent PRs. These figures were measured by combining two separate single-model runs, which is whatthe gate decides; the gate in the code is covered by tests with fake providers, not re-measured through itself (PLAN1015.12). They are before the operator's own tests, which are the one signal that caught more errors where we could runthem. Cost: two model calls per region.@@ -887,7 +887,7 @@ differently there is one model call, whose request holds:closest to the answer.Two things more come with the strategy, with no option: both were measured against the alternatives-(PLAN10 15.11) and the alternatives were removed.+(PLAN10 15.10) and the alternatives were removed.- The regions of the file are those of git's own merge (`git merge-file --diff3`on the three versions), so that "no conflict" means what it means to git and