Skip to content

Git designed for AI agents.

docs: version skew and capabilities, the normalization, retries, README, judge 3 on the OSS set

Federico Eliafeb7263ff836
diff --git a/README.md b/README.md
index 6205862..e86d145 100644
--- a/README.md
+++ b/README.md
@@ -47,7 +47,7 @@ ideas, with normal Git repositories underneath.
| **Always-online clients** that fetch exactly what they need | `ObjectService` serves single objects and sparse trees; `graft clone` and `graft sparse` materialize only the paths you ask for. |
| **Source control as an API**: history runs on the server | Blame, path history, ancestry and code search run server-side and are memoized as derived data, computed once per commit. |
| **Landing without a merge queue** | `UserLand` rebases a change onto the current tip and lands it with a compare-and-swap; concurrent lands on one branch are batched into a single reference update. |
-| **Automatic conflict resolution** | A conflicting land is resolved by a resolver you choose (any LLM provider, a deterministic tool, or none) through a versioned, vendor-neutral protocol. A model's resolution is only a proposal: it lands by itself only if the `verify.command` you configure passes on the exact tree that would land, and otherwise the agent gets the conflict package with the proposal. The deterministic rules you configure are not subject to that condition. |
+| **Automatic conflict resolution** | A conflicting land is resolved by a resolver you choose (any LLM provider, a deterministic tool, or none) through a versioned, vendor-neutral protocol. A model's resolution is only a proposal: it lands by itself only if the `verify.command` you configure passes on the exact tree that would land, and otherwise the agent gets the conflict package with the proposal. The deterministic rules you configure need no such condition of their own, and are verified on the exact tree as well whenever you configure a `verify.command`. |
| **Changeset log** | Every accepted reference update is recorded with its operation and actor in a dense, ordered log that clients list or watch live. |
| **Storage core: a database and object storage, stateless servers** | Each change is a changeset in a write-ahead log in SQLite or PostgreSQL, with a compare-and-swap per reference; packs live in object storage (file, S3, GCS, Azure). Server nodes are caches that any node can rebuild, so you add nodes to scale. |
| **Test after landing, revert the culprit** | `graft-verify` tests the tip, bisects a failure and reverts the change that broke it. |
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index 464eecb..796d92f 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -240,7 +240,7 @@ are in [STORAGE.md](STORAGE.md).
| `database` | string | required when on | `sqlite:///absolute/path.db`, or `postgres://user@host:port/db?sslmode=...` (`postgresql://` too). All nodes use the same database. Credentials are not written here: Postgres reads the `PG*` environment variables and `.pgpass`. The schema is created at startup. |
| `bucket` | string | required when on | gocloud URL of the object storage: `file:///absolute/dir`, `s3://bucket?region=...`, `gs://bucket` or `azblob://container`. All nodes use the same bucket. Credentials come from the cloud SDK's environment (`AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`, `AZURE_*`). |
| `node_id` | string | the host name | Name of this node in the database (changesets record it, and `graft storage state` shows it). Must not contain spaces or slashes. Give every node its own. |
-| `sync_timeout` | string (Go duration) | `"10s"` | Budget of one sync round before a request in authoritative mode. A round that cannot reach the head in time keeps what it did: the local repository stays at the changeset it reached, the request fails with `Unavailable`, and the next request's round goes on from there. A round that cannot finish at all (for example because the database is unreachable or a local writer holds a reference lock) fails the request with `Unavailable` too. Concurrent requests share a round; a request that arrives during a round that read the head before it then runs one of its own, so it waits at most two rounds (and stops waiting with `Unavailable` after twice `sync_timeout` plus a second per round, should a round not end within its budget). Clients retry `Unavailable`: the SDK does it for merge and fast-forward lands and for re-lands with a resolution (`LandOptions.RetryUnavailable`, see [LAND-BATCHING](LAND-BATCHING.md#limits)); rebase lands are not retried by default, because the server cannot yet tell that a rebased change already landed. A client without the SDK must follow the same rules. Must be positive. |
+| `sync_timeout` | string (Go duration) | `"10s"` | Budget of one sync round before a request in authoritative mode. A round that cannot reach the head in time keeps what it did: the local repository stays at the changeset it reached, the request fails with `Unavailable`, and the next request's round goes on from there. A round that cannot finish at all (for example because the database is unreachable or a local writer holds a reference lock) fails the request with `Unavailable` too. Concurrent requests share a round; a request that arrives during a round that read the head before it then runs one of its own, so it waits at most two rounds (and stops waiting with `Unavailable` after twice `sync_timeout` plus a second per round, should a round not end within its budget). Clients retry `Unavailable`: the SDK does it for every land (`LandOptions.RetryUnavailable`, see [LAND-BATCHING](LAND-BATCHING.md#limits)): the retry of a land whose response was lost answers `already_landed` from the server's record of the first land, rebase lands included (`SkipRebaseLands` turns the retry of rebase lands off for servers without that record). A client without the SDK must follow the same rules. Must be positive. |
| `shadow_timeout` | string (Go duration) | `"5s"` | Budget of the shadow-mode work of one reference transaction, for its Preparing phase (pack build and upload, before git locks the references) and its Prepared phase together. When it runs out, the work is abandoned, logged and counted, and the git write proceeds. Must be positive. |
| `gc_grace` | string (Go duration) | `"1h"` | How long garbage collection waits before it deletes a pack it observed unreferenced, and how old a checkpoint must be before the packs it covers are deleted. Must be greater than `sync_timeout + gc_margin`. |
| `gc_margin` | string (Go duration) | `"5m"` | Safety margin of the writers' deadline: a write uploads its pack and commits its changeset within `gc_grace - gc_margin`, else it fails. It also covers the difference between a node's clock and the database's. Must be positive. |
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.md
index 15075bf..b7e8354 100644
--- a/doc/graft/PLAN10.md
+++ b/doc/graft/PLAN10.md
@@ -941,7 +941,7 @@ end of this section.
| writer | proposes a resolution per region | a model, configurable (15.13) | 15.10: the elements kept and removed, 15.12 |
| checks and repair | refuse a broken answer, repair it once | deterministic checks | 15.10 |
| report | tells the agent what the answer keeps or drops | deterministic | 15.6: it separates (43% against 60% right), it does not gate |
-| agree_with (optional) | holds a proposal back unless a second model from another vendor gives the same file and both reports are clean | code (`llmStrategy`), configured by the operator | 15.12: G1, 59.8% of the files landed with 15.5% wrong under two judges and at most 16.8% under a third, before the operator's tests |
+| agree_with (optional) | holds a proposal back unless a second model from another vendor gives the same file and both reports are clean | code (`llmStrategy`), configured by the operator | 15.12: G1, 59.8% of the files landed with 15.5% wrong under two judges and at most 15.5% under a third, before the operator's tests |
| verify | runs the operator's build and tests on the exact tree | the operator's command | 15.9, 14 |
| decideLand | the one landing rule | code | 15.4 |
@@ -1675,8 +1675,8 @@ at most 5% wrong among them (Wilson 95% upper bound), on data the gates were not
under judge A, OSS and agent PRs apart. **That goal is not met, by any gate, on either kind of data.** After the first
held-out numbers the bar on the wrong share was restated to 20% (below, "The threshold of 20% wrong"); that bar was set
after the numbers were known, and is not a pre-registered confirmation. Against it, the best gate (G1, two models from
-different vendors in agreement, 59.8% of the OSS held-out files landed) has 15.5% wrong under the two first judges, 16.8%
-(upper bound 21.6%) under a third judge from another vendor, which is the number to quote (it is a ceiling: 24 of its 291
+different vendors in agreement, 59.8% of the OSS held-out files landed) has 15.5% wrong under the two first judges, 15.5%
+(upper bound 20.1%) under a third judge from another vendor, which is the number to quote (it is a ceiling: 15 of its 291
answers are unjudged and counted wrong), and on conflicts of agent PRs none comes close (43% to 52% wrong). What follows is the operating point that the evidence supports, the gates that were
tried, and the reading.
@@ -1695,10 +1695,11 @@ answers that G1 lands (291 OSS, 130 agent-PR), a strict-equal answer being accep
of the three. It was not finished. Its first pass stopped at its cap ($0.25 per set) with 11 OSS and 18 agent-PR answers that
are not strict-equal unjudged, and 15 and 16 calls that had returned an empty answer. A second pass (approved, about $0.10)
retried those with a backoff: the empty answers came back for all 15 OSS and 13 of the 16 agent-PR ones, but each retry cost
-five times a normal call (an empty answer is billed, and a retry is four attempts), and the pass spent $0.15 and stopped at
-its cap with 24 OSS and 29 agent-PR answers unjudged and 3 agent-PR calls still empty. All of those are counted as wrong
-here, so its figures are a ceiling. **Judge 3 alone is the number to quote**, with the majority of judges 1, 2 and 3 beside
-it. It cost $0.65 in all.
+five times a normal call (an empty answer is billed, and a retry is four attempts), and the pass stopped at its cap with 24 OSS
+and 29 agent-PR answers unjudged and 3 agent-PR calls still empty. A third pass on the OSS set only (approved) judged 11 of the
+24 and stopped at its cap again: 13 OSS answers are unjudged and 2 calls are empty, and the agent-PR set stays as it was after
+the second pass. All of those are counted as wrong here, so its figures are a ceiling. **Judge 3 alone is the number to
+quote**, with the majority of judges 1, 2 and 3 beside it. It cost $0.74 in all.
**What the writer's proposals are worth** (the prompt of record, Luna, effort low; "right" = the strict judge; the
interval is Wilson 95%; accepted = the answer passed the rules and the syntax check):
@@ -1750,9 +1751,9 @@ confirmation; read against it, honestly: on the OSS held-out, G1 has 15.5% wrong
under judge 2) with an upper bound of 20.1% under both judges (19.7% under judge 2, 15.5% under judge 1), at 59.8% of
the files landed, so it meets the bar on the point estimates and misses the bound under the conservative reading by 0.1
point; G0 (Luna alone) has 20.9% (upper 25.4%) and does not; without spring-boot G1 has 18.8% (upper 24.3%). Under the
-third judge, from another vendor and the strictest, G1 has 16.8% wrong (upper 21.6%) with its unjudged answers counted as wrong:
-under the restated bar on the point estimate, over it on the bound, and 20.5% (upper 26.1%) without spring-boot, so the
-restated bar is met by neither reading of the bound and by the point estimates of the whole set only. On the agent-PR files no gate comes close: 43.1%
+third judge, from another vendor and the strictest, G1 has 15.5% wrong (upper 20.1%) with its unjudged answers counted as wrong:
+under the restated bar on the point estimate, at it on the bound (0.1 point over), and 18.8% (upper 24.3%) without
+spring-boot, so the restated bar is met on the point estimates and missed on the bound without spring-boot. On the agent-PR files no gate comes close: 43.1%
wrong at best under two judges (upper 51.7%), 51.5% under judge 3 (upper 60.0%), at 45.8% landed.
**G1 under a third judge.** Wrong share among the files G1 lands, judge 3 alone and the majority (at least two of the
@@ -1760,21 +1761,22 @@ three judges accept), Wilson 95% upper bound in parentheses; the answers with no
| G1 landed | files | judge 3 alone | 2 of 3 judges |
|---|---:|---|---|
-| OSS held-out | 291 | **16.8% (up 21.6)** | 15.1% (up 19.7) |
-| OSS held-out without spring-boot | 234 | 20.5% (up 26.1) | 18.4% (up 23.8) |
-| go | 115 | 27.0% (up 35.7) | 24.3% (up 32.9) |
-| ts-js | 103 | 16.5% (up 24.9) | 13.6% (up 21.5) |
+| OSS held-out | 291 | **15.5% (up 20.1)** | 14.8% (up 19.3) |
+| OSS held-out without spring-boot | 234 | 18.8% (up 24.3) | 17.9% (up 23.4) |
+| go | 115 | 25.2% (up 33.9) | 23.5% (up 32.0) |
+| ts-js | 103 | 14.6% (up 22.6) | 13.6% (up 21.5) |
| python | 15 | 0.0% (up 20.4) | 6.7% (up 29.8) |
| rust-java (spring-boot but one file) | 58 | 1.7% (up 9.1) | 1.7% (up 9.1) |
| agent-PR | 130 | **51.5% (up 60.0)** | 42.3% (up 50.9) |
-Judge 3 finds 49 of the 291 OSS answers wrong, against 33 for judge 1 and 44 for judge 2; the 24 answers that are unjudged
-are among them. With every one of those accepted the figure would be 8.6%, with every one wrong it is 16.8% (the figure of the
-table); if they were wrong as often as the 68 answers of the same kind that it did judge (25 wrong), it would be about 11.6%,
-which is a guess, since 13 of the 24 are answers that had come back empty, perhaps the hard ones. On the agent-PR files 29 are unjudged and 3
-empty: 26.9% if all were accepted, 51.5% as counted, about 44.5% by the same guess. The conclusion on the OSS files does not
-depend on them: the point estimate is under the restated 20%; the upper bound is over it only when the unjudged are counted
-wrong, and without spring-boot, the easy part of the corpus, both are over in that reading.
+Judge 3 finds 45 of the 291 OSS answers wrong, against 33 for judge 1 and 44 for judge 2; the 15 answers that are unjudged
+or empty are among them. With every one of those accepted the figure would be 10.3%, with every one wrong it is 15.5% (the
+figure of the table); if they were wrong as often as the 77 answers of the same kind that it did judge (30 wrong), it would be
+about 12.3%, which is a guess, since the unjudged include the answers that had come back empty, perhaps the hard ones. On the
+agent-PR files 29 are unjudged and 3 empty: 26.9% if all were accepted, 51.5% as counted, about 44.5% by the same guess. The
+conclusion on the OSS files: the point estimate is under the restated 20% in every reading, the upper bound is at 20.1% in the
+reading that counts the unjudged as wrong (and under it in the others), and without spring-boot, the easy part of the corpus,
+both the point estimate (18.8%) and the bound (24.3%) are ~20% or over in that reading.
**Reading.**
@@ -1873,8 +1875,8 @@ judge's normalization, the composition reports of both are clean, and `verify.co
a `hold` reason and the report says that the second model disagreed, and the server never lands a held file, nor
runs the verification for it. A budget table must be given for both models or for neither. The bench runs the same
code and prices both models. RESOLVE.md has the configuration. Measured on the OSS held-out files it lands 59.8% of
-the files with 15.5% of those wrong under two independent judges (upper bound 20.1%) and at most 16.8% under a third judge from
-another vendor (upper bound 21.6%), before the operator's tests; on
+the files with 15.5% of those wrong under two independent judges (upper bound 20.1%) and at most 15.5% under a third judge from
+another vendor (upper bound 20.1%), before the operator's tests; on
conflicts of agent PRs 45.8% of the files with 43.1% wrong. Those numbers were measured as two single-model runs
combined offline (the gate decides the same thing on the same answers); the code is covered by tests with fake
providers and was not run through the bench against the live models. The 20%-wrong bar it is compared with was set
@@ -1910,7 +1912,7 @@ read from the key before and after each run and agrees, except where stated.
| 20 | held-out runs of the prompt of record: Luna and glm-5.3-flash on 487 OSS files and on 284 agent-PR files | manifest-heldout-487, manifest-agent-284 | 7fd0ab2's tools tree (8812cee) | $0.300 + $0.302 + $0.227 + $0.237 |
| 20 | judge 1 (glm-5.3-flash) on the held-out answers | the answers of Luna | scripts outside the repository | $0.025 + $0.021 |
| 20 | judge 2 (openai/gpt-6.1-sol, effort low) on the answers landed by G0 | OSS held-out 359, agent-PR 210 | scripts outside the repository | $0.239 + $0.268 |
-| 20 | judge 3 (mistralai/mistral-large-4-0, unrelated vendor) on the answers landed by G1: a first pass stopped at its cap of $0.25 per set, a second pass (retry of the empty answers with a backoff, and the unjudged) at +$0.03 and +$0.09; 24 OSS and 29 agent-PR answers unjudged and 3 agent-PR calls empty at the end, counted as wrong | OSS held-out 291, agent-PR 130 | scripts outside the repository | $0.298 + $0.354 (the key: $0.645) |
+| 20 | judge 3 (mistralai/mistral-large-4-0, unrelated vendor) on the answers landed by G1: a first pass stopped at its cap of $0.25 per set, a second pass (retry of the empty answers with a backoff, and the unjudged) at +$0.03 and +$0.09, a third on the OSS set only at +$0.10; 13 OSS and 29 agent-PR answers unjudged and 2 OSS and 3 agent-PR calls empty at the end, counted as wrong | OSS held-out 291, agent-PR 130 | scripts outside the repository | $0.404 + $0.354 (the key: $0.74) |
| 20 | the agree_with gate through fake providers and the real graft-resolve with two fake command models | tests only | 3cc898c | $0 |
**Why a screened model is skipped.** The screening is a smoke check of five cases (a go, a java, a python, a
diff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.md
index c923abd..ff0cd2c 100644
--- a/doc/graft/RESOLVE.md
+++ b/doc/graft/RESOLVE.md
@@ -242,9 +242,14 @@ Request (server to resolver):
"base": {"oid": "<oid>", "mode": "100644", "content": "<base64>"},
"ours": {"oid": "<oid>", "mode": "100644", "content": "<base64>"},
"theirs": {"oid": "<oid>", "mode": "100644", "content": "<base64>"}}],
- "limits": {"max_file_bytes": 1048576}}
+ "limits": {"max_file_bytes": 1048576},
+ "capabilities": ["origin", "proposal", "hold", "verify"]}
```
+- `capabilities` says what the server does with the answers (an additive field of protocol v1): `origin` it reads the
+ origin of a file, `proposal` a file that is not a rule's is a proposal that lands only on a verified tree, `hold` it
+ never lands a file with a `hold`, `verify` it verifies the exact tree that lands. A request without it comes from an
+ older server, and `graft-resolve` gives it no model's resolution (see "Version skew" below).
- `ours` is the current branch tip side. `theirs` is the change being landed.
- `tip`, `merge_base` and `messages.tip` are always empty since this version;
they stay in the protocol for compatibility. The server reuses a file's
@@ -716,7 +721,12 @@ only = ["z-ai/fp8"]
Both models resolve each file at the same time with the same prompt of record. The first model's resolution is the
one that is proposed. It lands only if all three hold:
-1. **the two results are the same file** after normalizing line endings, trailing whitespace and trailing blank lines;
+1. **the two results are the same file** after a normalization that erases exactly three things: a carriage return before a
+ line feed (CRLF and LF are the same), the spaces, tabs and carriage returns at the end of a line, and the blank lines at
+ the end of the file (so a final newline or its absence is not a difference). Nothing else is erased: indentation, blank
+ lines elsewhere, the order of lines and every other character must be equal. It is the normalization that the
+ measurements and the judges used, and it decides only whether the models agree: the bytes that land, and that the
+ verification runs on, are the first model's, exactly as it wrote them;
2. **the composition report of both is clean** (no line saying that the answer drops or duplicates a change of a side);
3. **`verify.command` passes** on the exact tree, as for every model resolution: `verify.enabled = true` is mandatory
(the gate does not replace it, it chooses the proposals worth verifying).
@@ -734,18 +744,28 @@ server together: an older server does not know `hold` and would verify a held pr
What it was worth, measured on 487 held-out files of 13 open-source repositories (Luna and glm-5.3-flash, with the
verify simulation of the bench in place of the operator's tests): 59.8% of the files landed, and 15.5% of those differed
-from what two independent reviewing models accept (11.3% for one of them; Wilson upper bound 20.1% for both), and at most 16.8% from
-what a third one, from another vendor, accepts (upper bound 21.6%; it is a ceiling, 24 of its 291 verdicts are missing and counted as wrong), against 43% to 52% on conflicts of merged agent PRs. These figures were measured by combining two separate single-model runs, which is what
+from what two independent reviewing models accept (11.3% for one of them; Wilson upper bound 20.1% for both), and at most 15.5% from
+what a third one, from another vendor, accepts (upper bound 20.1%; it is a ceiling, 15 of its 291 verdicts are missing and counted as wrong), against 43% to 52% on conflicts of merged agent PRs. These figures were measured by combining two separate single-model runs, which is what
the gate decides; the gate in the code is covered by tests with fake providers, not re-measured through itself (PLAN10
15.12). They are before the operator's own tests, which are the one signal that caught more errors where we could run
them. Cost: two model calls per region.
-**Upgrade `graft-resolve` together with the server.** A resolution is a rule's only if its file says so (`origin`
-`"rule"`, protocol v1); a file without an origin is a model's, which lands only through `verify.command`. A
-`graft-resolve` of an earlier release answers without an origin, so with a new server every file it resolves,
-`trivial` and `union` included, is a proposal, and lands only if `verify.command` passes on the exact tree (without
-one, it never lands). The other way round, an older server does not know `hold` (the `agree_with` gate) and would
-verify a held proposal like any other. Upgrade both in the same maintenance window.
+**Version skew between the server and `graft-resolve`.** A resolution is a rule's only if its file says so (`origin`
+`"rule"`, protocol v1); a file without an origin is a model's, which lands only through `verify.command`.
+
+* *An older `graft-resolve` with a new server.* It answers without an origin, so every file it resolves, `trivial` and
+ `union` included, is a proposal, and lands only if `verify.command` passes on the exact tree (without one, it never
+ lands). Safe, and the rules stop landing by themselves until `graft-resolve` is upgraded.
+* *A new `graft-resolve` with an older server.* An older server ignores the origin, the proposal mark and the hold, and
+ lands every answer it gets without verification, a held one included. The request therefore says what the server does
+ (`capabilities`, an additive field of protocol v1: `origin`, `proposal`, `hold`, `verify`; a server of this release
+ sends all four), and `graft-resolve` gives a model's resolution (the `llm` strategy, a `command` with `origin =
+ "model"`) only to a server that announces `proposal`, and with `agree_with` only to one that announces `hold` too. For
+ any other server the model is not called and the file stays unresolved, with the reason "the server cannot hold or
+ verify a model's resolution; upgrade the server". The rules are unaffected. A client of the protocol of your own that
+ sends no capabilities gets no model's resolution either.
+
+Upgrade both in the same maintenance window anyway.
**The `rerere` store and the upgrade.** Entries are versioned and record their
origin. The stores of v0.1.0 and v0.2.0 recorded the `llm` answers as well, with