Skip to content

Git designed for AI agents.

graft: a model's resolution lands only through the operator's verification (PLAN10 14, 15.9)

Federico Eliaf404a713fda0
diff --git a/cmd/graft/conflicts.go b/cmd/graft/conflicts.go
index c44c8e0..84121d2 100644
--- a/cmd/graft/conflicts.go
+++ b/cmd/graft/conflicts.go
@@ -30,7 +30,8 @@ import (
// DIR/sides/ours/<path>
// DIR/sides/theirs/<path>
// DIR/proposals/<path> the resolver's proposal for the path, when it made one (a model's
-// resolution, which the server never lands, or the rules' of a land
+// resolution, which did not land because no verification passed on
+// it, or the rules' of a land
// that failed for another file); copy it over DIR/files/<path> to
// land it as yours
// DIR/conflict.json the conflict package
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index 5369d98..2328196 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -193,11 +193,12 @@ cause.
| `command` | array of strings | empty | Program and arguments. It reads one request on stdin and writes the answer on stdout. |
| `url` | string | empty | `http://` or `https://` endpoint that receives the request as a JSON POST. Exactly one of `command` and `url` must be set when `enabled` is true. |
| `token_file` | string | empty | For `url`: file with the bearer token, read at startup. |
-| `timeout` | string | `"120s"` | Bound on one resolver call (Go duration), and on the whole land that the server resolves for `resolve_conflicts`: its rounds (the resolutions, the waits for a resolve slot, and the passes through the branch's lane) end at this deadline. A round that the lane hands back after it fails the land with the merge conflict, "files kept changing", naming the files. |
+| `timeout` | string | `"120s"` | Bound on one resolver call (Go duration), and on the whole land that the server resolves for `resolve_conflicts`: its rounds (the resolutions, the verifications of a model's resolution, the waits for slots, and the passes through the branch's lane) end at this deadline, which is `timeout` plus `verify.timeout` when `verify.enabled`. A round that the lane hands back after it fails the land with the merge conflict, "files kept changing", naming the files. |
| `max_files` | int | `20` | Conflicted files per land. A land with more is not sent to the resolver. |
| `max_file_bytes` | int | `1048576` (1 MiB) | Bound on every side of every file sent and on every resolved file accepted. At most 12566528 (about 12 MiB): one file, its three sides and its resolution, must fit one check request. |
| `proposal_ttl` | string | `"168h"` | How long the proposals of a failed land are kept under `refs/graft/proposals/` (Go duration, by the server's clock); at most 20 are kept per branch. Lands with a resolution and the housekeeping pass prune them. |
| `check_concurrency` | int | `2` | Checks of agents' resolutions that run at once in the Gitaly process, across all its listeners (one resolver client is shared by them); each holds a request of up to 64 MiB. A check waits for a free slot within its deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_check_slot_wait_seconds`. |
+| `verify_concurrency` | int | `2` | Verifications of a model's resolution that run at once in the Gitaly process, out of the lane, in a pool of their own, as `verify.command` takes minutes. A verification waits for a free slot within the land's deadline. The wait is in `graft_resolver_verify_slot_wait_seconds`. |
| `resolve_concurrency` | int | `2` | Resolutions that run at once in the Gitaly process, in a pool apart from the checks', so that long resolutions never delay a check. A resolution waits for a free slot within the land's deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_resolve_slot_wait_seconds`. |
| `resolve_rounds` | int | `32` | Safety cap on the rounds of a land whose conflicting files keep changing while it is resolved out of the lane. The rounds are bounded by `timeout`; the cap is set well above realistic contention and is no contract. |
| `paths_allow` | array of strings | empty (every path) | Globs of paths that may be sent to the resolver, with the syntax of policy refs. |
@@ -208,9 +209,9 @@ cause.
| `minified.globs` | array of strings | empty | Base-name globs of minified files, besides the built-in `*.min.js`, `*.min.mjs`, `*.min.css` and `*.bundle.js`. Such files are never sent. |
| `minified.average_line_bytes` | int | `300` | A side larger than 2 KiB whose average line is longer is minified. |
| `minified.longest_line_bytes` | int | `5000` | A side larger than 2 KiB with a longer line is minified. |
-| `verify.enabled` | bool | `false` | Run `verify.command` on the merged tree of every resolution (a resolver's or an agent's) before it lands; it applies even when `enabled` is false. |
+| `verify.enabled` | bool | `false` | Run `verify.command` on the merged tree of every resolution (a resolver's or an agent's) before it lands; it applies even when `enabled` is false. It is also what lets a model's resolution land: only a tree it passed on lands, and without it a model's resolution is only ever a proposal. |
| `verify.command` | array of strings | empty | Program and arguments, run in a temporary checkout of the merged tree with `PATH`, `HOME` and `TMPDIR` only. A non-zero exit rejects the resolution. |
-| `verify.timeout` | string | `"5m"` | Bound on one run (Go duration); a run that does not finish rejects the resolution. |
+| `verify.timeout` | string | `"5m"` | Bound on one run (Go duration); a run that does not finish rejects the resolution. When `verify.enabled`, it is added to `timeout` for the deadline of a resolved land. |
A file that is never sent goes to the agent with its reason in the conflict
package: large (a side over `max_file_bytes`), binary (a NUL byte), lockfile,
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.md
index 4036aa3..68d637b 100644
--- a/doc/graft/PLAN10.md
+++ b/doc/graft/PLAN10.md
@@ -730,9 +730,30 @@ whole series under $5 real, and a run over $1.00 real stops the series until the
## 14. The agent always resolves; the server's AI only proposes (Federico, PM-approved design)
-This supersedes "AI lands with trace only". The server never lands a resolution of the llm strategy
-by itself: it is a proposal, which the agent reviews, edits or replaces, and lands with its own
-resolution (UserLandRequest.resolution, Graft-Resolved-By: agent/<actor>). The deterministic
+**Final ruling (Federico, 2026-10-19, relayed by the PM; it supersedes the amendment at the end of
+this section and every statement below that the server never lands a model's resolution).** An AI
+resolution lands only through the operator's verification (15.9).
+
+* Server AI configured and verify.command configured: the server resolves. If every deterministic
+ check passes and verify.command passes on the merged tree, it LANDS, signed Graft-Resolved-By:
+ <the resolver> and Graft-Checked-By: verify:<command>. Otherwise it goes to the agent as a
+ PROPOSAL, with the verify output when verify failed.
+* Server AI configured, no verify.command: proposal only, and the agent decides. An unverified AI
+ resolution never lands.
+* No AI: the agent resolves.
+* Rules (trivial, union, rerere, command) are unchanged: they land after verify.command when one is
+ configured, in the lane, as today.
+* The composition report (15.3) stays a report, not a gate.
+* One rule, in decideLand: a file that proposes may land only if verify.command ran and passed on
+ the exact tree that lands, compared by tree object ID. No other code path lands a model's
+ resolution.
+* The trailer names what ran: Graft-Checked-By: verify:<command>. The syntax and operator checks
+ run inside the resolver and are not separate trailers; the composition report is not built, so
+ the trailer does not claim it.
+
+This supersedes "AI lands with trace only". Before the ruling above, the server never landed a
+resolution of the llm strategy by itself: it was a proposal, which the agent reviews, edits or
+replaces, and lands with its own resolution (UserLandRequest.resolution, Graft-Resolved-By: agent/<actor>). The deterministic
strategies that the operator configures (trivial, union, rerere, command) are rules, not AI: a
resolution made by rules alone still lands. The llm auto-land path is removed, not kept behind a
flag.
@@ -835,7 +856,8 @@ flag.
resolve_conflicts, takes seconds. Moving both out of the lane, like the precheck of an agent's
resolution, is a step of its own.
-* **Amendment: Federico's product decision (2026-10-19, relayed by the PM).** "When the server AI
+* **Amendment: Federico's product decision (2026-10-19, relayed by the PM). Superseded by the final
+ ruling at the top of this section: the gate is verify.command, not the composition check.** "When the server AI
is configured, the server always tries to resolve, and it LANDS its resolution by itself ONLY if
the proposal passes every deterministic check: syntax, anchors, no markers, and the composition
check from §15 (keeps both changes, no duplicated both-added lines). There is still no
@@ -880,10 +902,12 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
| agent may not update the branch (policy) -> PermissionDenied
| the conflicting files not yet resolved -> one VERDICT per file
| (15.2), each kept by its key (path, base, ours, theirs, modes),
- | resolve slots; decideLand (15.4):
+ | resolve slots; a model's answer: verify.command runs on the exact
+ | tree that would land, verify slots (15.9); decideLand (15.4):
| not landable -> proposals kept (14) and PACKAGE now: the lane
- | is not entered
- | landable ----> the verdicts go to the lane
+ | is not entered; a model's files are PROPOSED,
+ | with the verify output when verify failed
+ | landable ----> the verdicts, and the verified tree, go to the lane
v
LANE (one per branch)
|-- a resolution -----> landWithResolution: tip moved -> refuse, fresh PACKAGE
@@ -894,16 +918,28 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
3-way merge (rebase or merge) of the tip and the change
|-- clean -------------------------------------------------------------> LAND
|-- conflicts, no resolve_conflicts ------------------------------------> merge_conflict
- |-- conflicts, every conflicting file has a landable verdict with its key, and the lane's
+ |-- conflicts, every conflicting file has a verdict with its key, and the lane's
| classification of the file agrees with the verdict's
- | -> operator verify.command; passes -> merge commit signed by the rules -> LAND
- | fails -> PACKAGE (every file REJECTED, reason)
+ | rules only -> operator verify.command (in the lane, for now); passes -> merge commit
+ | signed by the rules -> LAND; fails -> PACKAGE (every file REJECTED, reason)
+ | a model's file -> decideLand (15.4) on the tree the lane would write: it LANDS only if
+ | that tree is the tree verify.command passed on (tree object IDs). The lane
+ | runs no verification of a model's resolution. Merge commit signed
+ | Graft-Resolved-By: <resolver> + Graft-Checked-By: verify:<command>.
+ | Another tree (the tip moved) -> NEEDS VERIFICATION, below
|-- conflicts, a file without such a verdict (new, or its sides changed)
- -> NEEDS RESOLUTION, out of the lane: PRE-RESOLVE only those files against the new tip,
- keeping the other verdicts, and enter the lane again; at most resolve_rounds rounds
- (default 3) within the resolver's timeout, then merge_conflict, "files kept
- changing: a, b"
- The resolver is never called inside the lane.
+ | -> NEEDS RESOLUTION, out of the lane: PRE-RESOLVE only those files against the new tip,
+ | keeping the other verdicts, and enter the lane again
+ |-- a model's answer whose tree is not the verified one (the tip moved)
+ -> NEEDS VERIFICATION, out of the lane: rebuild the tree from the same verdicts (the
+ resolver is called only for files that changed), VERIFY it, enter the lane again
+ The rounds are bounded by the deadline of the land (the resolver's timeout, plus verify.timeout when
+ a verify.command is configured) and by resolve_rounds (default 32, a safety cap, no contract). At
+ the deadline: files that kept changing -> merge_conflict, "files kept changing: a, b"; a model's
+ resolution that was not verified in time -> PACKAGE, the model's files PROPOSED, "could not verify
+ before the branch moved again" (a verified AI land is opportunistic: losing the race is harmless,
+ the agent gets the proposal).
+ The resolver is never called inside the lane, and neither is the verification of a model's resolution.
PACKAGE to the agent (merge_conflict detail): sides by OID, hunks, per-file outcome + reason,
per-file proposal (+ its report, planned), the proposals kept under refs/graft/proposals/ (14),
@@ -941,11 +977,16 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
assemble the file; file checks: markers, tree-sitter syntax, operator checks
|-- a check fails ----------------------------------------> AGENT (REJECTED, reason)
v
- composition report (15.3) (planned)
+ composition report (15.3) (planned; a report, not a gate)
v
VERDICT PROPOSAL (PROPOSED, the proposal + its report)
- [AI landing when every deterministic check passes: designed by Federico's amendment (14),
- NOT ENABLED: the measurement in 15.6 shows no deterministic check that reaches the bar]
+ v
+ verify.command configured? --no--> stays a PROPOSAL: the agent decides ("no verify.command")
+ | yes: the exact tree that would land is verified, out of the lane (15.9)
+ |-- fails ------------------------------------------------> stays a PROPOSAL (+ the verify output)
+ |-- not verified before the branch moved again -----------> stays a PROPOSAL (+ the reason)
+ v
+ passes --> LANDS, if it is still the tree to land (decideLand, 15.4), signed Graft-Checked-By: verify:...
```
### 15.3 Composition report (planned; a report, not a gate)
@@ -973,10 +1014,13 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
### 15.4 The one decision point
```
- decideLand, before the lane (the pre-resolution calls it; landAnswerProposes is its guard)
- every file resolved by an operator rule ...... LAND (then verify.command)
- anything else, including any model answer .... PACKAGE (the model's files are proposals)
- No other code path lands a model's resolution. No check disables itself: a configuration that
+ decideLand(answer, files, tree, verified): the pre-resolution calls it with the tree it verified,
+ the lane with the tree it would write
+ every file resolved by an operator rule ...... LAND (then verify.command, in the lane)
+ a file a model resolved ...................... LAND only if verify.command ran and passed on THIS
+ tree: tree object ID equality, not the tip
+ anything else ................................ PACKAGE (the model's files are proposals)
+ One rule about a model's resolution. No other code path lands it. No check disables itself: a configuration that
cannot be used stops the server at startup, and a failure at run time refuses the land. No
confidence that the model declares about itself decides anything.
```
@@ -991,8 +1035,11 @@ not exist yet; (not enabled) marks what is designed and measured but switched of
| LANE, re-land | `landAttempt`, `landingAttempt.decide`, `landWithResolution` | operations/land.go, land_attempt.go, land_reland.go |
| 3-way merge, listing | `landingAttempt.decide`, `landListConflicts` | operations/land_attempt.go, land_resolve.go |
| out of the lane: rounds, deadline | `landResolving`, `landPreResolve` | operations/land_preresolve.go |
-| per-file verdicts, decide | `landResolveAt`, `decideLand`, `landAnswerProposes`, `landKeyOf` | operations/land_resolve.go, land_proposals.go, land_preresolve.go |
+| per-file verdicts, decide | `landResolveAt`, `landSetOutcomes`, `decideLand`, `landAnswerProposes`, `landKeyOf` | operations/land_resolve.go, land_proposals.go, land_preresolve.go |
+| the tree that would land | `landResolvedTree` | operations/land_resolve.go |
+| verify a model's resolution, out of the lane | `landVerifyOutOfLane`, `landVerify`, `Client.AcquireVerifySlot` | operations/land_verify.go; graft/resolver/check.go |
| verdicts applied in the lane | `landApplyVerdicts`, `landNeedsResolution` | operations/land_resolve.go, land_preresolve.go |
+| unverified in time: proposals | `landUnverified`, `landNotVerifiedReason` | operations/land_preresolve.go |
| resolver call, bounded slots | `Client.AcquireResolveSlot`, `ResolveSlot.Resolve` | graft/resolver/check.go |
| policy before resolving | `hook.CheckAgentPolicy`, `landPolicyDeniedError` | operations/land_resolve.go, land_proposals.go |
| classify | `landClassify`, `resolver.SendRules` | operations/land_resolve.go; graft/resolver |
@@ -1041,10 +1088,10 @@ the same base lines supersedes it. Measured with zero model calls:
measures overlap with the sides.
**Decisions.**
-1. The AI stays proposal-only.
-2. Federico's amendment (AI landing when every deterministic check passes) stays recorded and is
- not enabled: no deterministic check reaches the bar. Pending the Sol-100 result, which Federico
- uses to decide.
+1. The AI stays proposal-only unless verify.command passes on the tree (the final ruling, 14 and 15.9).
+2. Federico's amendment (AI landing when every deterministic check passes) is superseded by the
+ final ruling: the gate is the operator's verify.command, because no deterministic check reaches
+ the bar.
3. The composition report goes with every proposal, honest about what it can say: "keeps every
uncontested change of both sides" or "drops THEIRS' change at base 40-44", "duplicates a line
both sides added", "changes lines outside the conflicts", "N contested regions resolved freely".
@@ -1149,8 +1196,9 @@ The resolution is bound to the files instead (the PM's correction):
over two plain semaphores; the memory bound of the process is the sum of the two pools' requests.
Each request waits for its slot within the resolver's timeout and fails closed when it expires.
* **Still in the lane, stated:** verify.command, the operator's tests on the merged tree, runs in
- the lane for the rules' resolutions and for agents' resolutions alike. It is the next candidate
- to move out, as its own step.
+ the lane for the rules' resolutions and for agents' resolutions alike. For a model's resolution
+ it already runs out of the lane (15.9). Moving the other two onto the same mechanism is the next
+ step, measured on its own, so that in the end there is one verification mechanism.
* **Measured (TestLandResolveBench, GRAFT_LAND_BENCH=1).** Six lands that the server resolves,
all on the same file, and six plain lands that conflict with nothing, at once through the lanes
of a standalone server, with a resolver that takes 500 ms; the same bench on the tree before
@@ -1182,3 +1230,48 @@ The resolution is bound to the files instead (the PM's correction):
* A proposal kept while a batch holds the lane has the published tip as parent (review of
6090d5b, item 4): proposals are written before the lane only.
* The two pools: a held resolve slot does not delay a check.
+
+### 15.9 Design delta: a model's resolution lands through the verification (PM ruling)
+
+The final ruling of 14: an AI resolution lands only through the operator's verification. The
+verification runs the operator's tests, which take minutes, so it cannot run in the lane (15.8),
+and the tree it verifies depends on the tip.
+
+* **Verification out of the lane, on the exact tree.** For an answer that holds a model's
+ resolution, and when nothing else keeps the land from landing (every file resolved or proposed),
+ the pre-resolution builds the tree that would land (`landResolvedTree`: the merged tree of the
+ tip and the change, the answer's files written in) and runs verify.command on it, in the
+ quarantine, with a slot of the verify pool (`verify_concurrency`, default 2, a pool of its own:
+ `graft_resolver_verify_slot_wait_seconds`). The tree object ID that passed is kept.
+* **In the lane: tree equality.** The lane merges its own tip, applies the verdicts, builds the
+ tree it would write, and asks decideLand about it: a model's resolution lands only if that tree is
+ a tree that passed. Tree object IDs are compared, not tips: a tip that moved to the same tree
+ keeps the verification; a tip that moved to another tree does not. The lane runs no verification
+ of a model's resolution, and writes the merge commit with Graft-Checked-By: verify:<command>.
+* **Tip moved: out again, no resolver call.** When the tree is not a verified one, the lane does not
+ land. The land goes back out, rebuilds the tree from the same verdicts (the resolver is called only
+ for files whose sides changed, as in 15.8), verifies the new tree, and enters the lane again. Each
+ round costs one verification; the progress argument is the one of 15.8 (a round is lost only when
+ another land landed on the branch), and the rounds are bounded by the deadline of the land.
+* **Deadline: a proposal, not a failure.** The deadline of the land is the resolver's timeout plus
+ verify.timeout when a verify.command is configured. A verified AI land is opportunistic: when the
+ deadline ends before a tree could be verified and landed, the land does not fail for the agent. The
+ model's files become PROPOSED with the reason "could not verify before the branch moved again" (or
+ "before the deadline of <t>" when the verification itself was cut), the proposals are kept (14), and
+ the package goes to the agent. This final pass verifies nothing and calls no resolver. A caller that
+ gives up is not a deadline: it gets its cancellation.
+* **A verification that fails** (non-zero exit, or its own timeout) is a verdict, not a deadline: the
+ files stay PROPOSED and carry "verification failed: exit status N: <the first 4 KiB of the
+ output>" (the package cuts a reason to 512 bytes). Nothing is verified for a land that cannot land
+ anyway (a file refused, a file left to the agent).
+* **Rules are unchanged.** A resolution made by rules alone lands after verify.command, which runs in
+ the lane, as before; so does an agent's resolution. Moving them onto the same out-of-lane mechanism
+ is the next step, measured on its own, so that in the end there is one verification mechanism.
+* **Tests.** A model's resolution that passes lands, signed, with the command run once (so not in
+ the lane); one that fails gives a proposal and the output; without a verify.command it is a
+ proposal; a tip that moved gets its new tree verified and lands, with no resolver call; a tip that
+ moved to the same tree keeps the verification; churn past the deadline gives a proposal, and so does
+ the cap on the rounds; the lane is free while a verification runs (a plain land passes through it);
+ the lane never verifies a model's resolution; the verify pool bounds the verifications at once;
+ rules keep their verification in the lane; and the table test of decideLand, whose one rule is
+ compared by tree object ID.
diff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.md
index 027af61..da29a8c 100644
--- a/doc/graft/RESOLVE.md
+++ b/doc/graft/RESOLVE.md
@@ -46,6 +46,9 @@ actors = ["builder", "planner"]
never delay the short checks. `resolve_rounds` (default 32) is a safety cap
on the rounds of a land whose files keep changing (see "Out of the lane"
below); the rounds are bounded by `timeout`, and the cap is no contract.
+- `verify_concurrency` (default 2): the verifications of a model's resolution
+ that run at once in the Gitaly process, in a pool of their own (see "Optional
+ verification of a resolution").
- A configuration the server could not use is a startup error (see
[CONFIGURATION](CONFIGURATION.md)). A configured resolver is never treated
as no resolver: if it cannot be built later, every land that needs it fails
@@ -162,13 +165,18 @@ and the error carries the conflict package as a second detail (see
[The conflict package and the re-land](#the-conflict-package-and-the-re-land)).
Nothing lands partially: the branch is never left in a partial state.
-### Proposals: a model's resolution never lands
-
-The server never lands what a model resolved: the agent always resolves
-(PLAN10 14). graft-resolve marks every file its `llm` strategy resolved as a
-proposal (`"proposal": true`), and the server treats an answer as proposing
-when a file is marked, or when the resolver names the `llm` strategy without
-marking any. Such a land fails with the package: the model's files are
+### Proposals, and when a model's resolution lands
+
+A model's resolution lands only through the operator's verification
+(PLAN10 14 and 15.9): when `verify.command` is configured and passes, out of
+the lane, on the exact tree that would land (see "Optional verification of a
+resolution"). Without a `verify.command` the server never lands what a model
+resolved: the agent decides. graft-resolve marks every file its `llm` strategy
+resolved as a proposal (`"proposal": true`), and the server treats an answer as
+proposing when a file is marked, or when the resolver names the `llm` strategy
+without marking any. A land of such an answer that does not land (no
+`verify.command`, a verification that failed, or none finished in time) fails
+with the package: the model's files are
`PROPOSED`, the files the rules resolved keep `RESOLVED`, and every file the
answer resolved carries its `proposal` (action, blob OID, mode, resolver, and
the explanation cut to 512 bytes). An answer that does not land for another
@@ -381,7 +389,33 @@ temporary directory and runs `command` there, with a minimal environment
(`PATH`, and `HOME` and `TMPDIR` in a directory of their own). A non-zero exit or a
timeout rejects the resolution: the land fails, every file of the package is
`REJECTED`, and the reason is `verification failed:` with the exit status and
-the first 4 KiB of the output.
+the first 4 KiB of the output. That is the verification of a resolution made by
+rules and of an agent's, which runs in the branch's lane, on the merged tree of
+the lane's tip.
+
+**For a model's resolution the verification is the gate that lets it land**
+(PLAN10 15.9), and it runs out of the lane:
+
+- The server builds the tree that would land (the merged tree of the tip and
+ the change, the model's files written in) and runs `command` on it with a slot
+ of the verify pool (`verify_concurrency`, default 2;
+ `graft_resolver_verify_slot_wait_seconds`). The lane never runs it.
+- In the lane, the model's resolution lands only if the tree it would write is
+ a tree that passed (compared by tree object ID, not by tip). It is signed
+ `Graft-Resolved-By: <resolver>` and `Graft-Checked-By: verify:<command>`.
+ The trailer names what ran: the syntax and operator checks run inside the
+ resolver and are not named, and the composition report is not a check.
+- If the branch moved to another tree, the land goes back out: it rebuilds the
+ tree from the same answers (the resolver is called again only for files whose
+ sides changed), verifies the new tree, and tries the lane again.
+- If the verification fails, the model's files stay `PROPOSED`, with the reason
+ `verification failed:` and the output. If the deadline ends before a tree could
+ be verified and landed, the land does not fail for the agent: the files stay
+ `PROPOSED` with the reason `could not verify before the branch moved again`
+ (or `before the deadline of <t>`), and the package carries the proposals. The
+ deadline is `timeout` plus `verify.timeout`.
+- Nothing is verified for a land that cannot land anyway (a file refused, a
+ file left to the agent).
**A verify command such as a build or a test suite executes code from the
merged tree: code that agents and the AI wrote, run as the gitaly user on the
@@ -494,7 +528,7 @@ sides read and 64 Mi cells of diff work for its hunks, and 4 MiB encoded
| `REJECTED` | the result failed a rule or a check, e.g. a syntax error at line:column |
| `FAILED` | the resolver failed on it: a model error, `AI budget exhausted`, no strategy applied |
| `RESOLVED` | resolved by rules, but the land failed because another file was not; see `proposal` |
-| `PROPOSED` | a model resolved it: proposed, never landed; see `proposal` |
+| `PROPOSED` | a model resolved it and it did not land (nothing verified it, the verification failed, or none finished in time): proposed; see `proposal` and the reason |
| `NOT_ATTEMPTED` | the request was not attempted; see `reason` |
The agent reads the sides by OID with `GetBlobs`, resolves every file, and
@@ -952,6 +986,7 @@ in blank lines only, and the accepted results that got no syntax verdict.
| check answer | 64 KiB + 8 KiB per file | server |
| checks of agents' resolutions at once | 2 | `check_concurrency` |
| resolutions at once | 2 | `resolve_concurrency` |
+| verifications of a model's resolution at once | 2 | `verify_concurrency` |
| rounds of a land whose files keep changing | 32 (a safety cap; `timeout` bounds the land) | `resolve_rounds` |
| verification run | 5 min, output kept 4 KiB | `verify.timeout` |
| request body read by `graft-resolve` | 128 MiB (`protocol.DefaultMaxRequestBytes`) | `--max-request-bytes` |
diff --git a/internal/gitaly/config/config_graft.go b/internal/gitaly/config/config_graft.go
index 0afb605..fce28dc 100644
--- a/internal/gitaly/config/config_graft.go
+++ b/internal/gitaly/config/config_graft.go
@@ -327,6 +327,11 @@ type GraftResolver struct {
// pool of their own so that long resolutions never delay the short checks. Zero means
// DefaultGraftResolverResolveConcurrency.
ResolveConcurrency int `json:"resolve_concurrency" toml:"resolve_concurrency,omitempty"`
+ // VerifyConcurrency bounds the verifications that run at once on this server out of the lane
+ // (PLAN10 15.9): the operator's verify.command on the exact tree of a model's resolution, which
+ // lands only if it passes. It is a pool of its own, as the command takes minutes. Zero means
+ // DefaultGraftResolverVerifyConcurrency.
+ VerifyConcurrency int `json:"verify_concurrency" toml:"verify_concurrency,omitempty"`
// ResolveRounds is a safety cap on the rounds of a land whose conflicting files keep changing
// while it is resolved (PLAN10 15.8). The rounds are bounded by the resolver's timeout; the cap
// is set well above realistic contention and is no contract. Zero means
@@ -518,8 +523,8 @@ func (r GraftResolver) validate() error {
if _, err := r.ProposalTTLOrDefault(); err != nil {
return err
}
- if r.MaxFiles < 0 || r.MaxFileBytes < 0 || r.CheckConcurrency < 0 || r.ResolveConcurrency < 0 || r.ResolveRounds < 0 {
- return fmt.Errorf("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency and resolve_rounds must not be negative")
+ if r.MaxFiles < 0 || r.MaxFileBytes < 0 || r.CheckConcurrency < 0 || r.ResolveConcurrency < 0 || r.VerifyConcurrency < 0 || r.ResolveRounds < 0 {
+ return fmt.Errorf("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency, verify_concurrency and resolve_rounds must not be negative")
}
// The check of an agent's resolution sends a file's three sides and its resolution in one
// request, which must fit its bound.
@@ -587,6 +592,17 @@ func (r GraftResolver) CheckConcurrencyOrDefault() int {
return DefaultGraftResolverCheckConcurrency
}
+// DefaultGraftResolverVerifyConcurrency is the default of GraftResolver.VerifyConcurrency.
+const DefaultGraftResolverVerifyConcurrency = 2
+
+// VerifyConcurrencyOrDefault returns VerifyConcurrency or its default.
+func (r GraftResolver) VerifyConcurrencyOrDefault() int {
+ if r.VerifyConcurrency > 0 {
+ return r.VerifyConcurrency
+ }
+ return DefaultGraftResolverVerifyConcurrency
+}
+
// DefaultGraftResolverResolveConcurrency is the default of GraftResolver.ResolveConcurrency.
const DefaultGraftResolverResolveConcurrency = 2
diff --git a/internal/gitaly/config/config_graft_resolver_test.go b/internal/gitaly/config/config_graft_resolver_test.go
index 4d248c6..f9fc16a 100644
--- a/internal/gitaly/config/config_graft_resolver_test.go
+++ b/internal/gitaly/config/config_graft_resolver_test.go
@@ -113,6 +113,9 @@ func TestGraftResolverValidateBuildable(t *testing.T) {
"negative resolve_concurrency": {
GraftResolver{Command: []string{"graft-resolve"}, ResolveConcurrency: -1}, "must not be negative",
},
+ "negative verify_concurrency": {
+ GraftResolver{Command: []string{"graft-resolve"}, VerifyConcurrency: -1}, "must not be negative",
+ },
"negative resolve_rounds": {
GraftResolver{Command: []string{"graft-resolve"}, ResolveRounds: -1}, "must not be negative",
},
@@ -135,6 +138,8 @@ func TestGraftResolverValidateBuildable(t *testing.T) {
require.Equal(t, 5, GraftResolver{CheckConcurrency: 5}.CheckConcurrencyOrDefault())
require.Equal(t, DefaultGraftResolverResolveConcurrency, GraftResolver{}.ResolveConcurrencyOrDefault())
require.Equal(t, 4, GraftResolver{ResolveConcurrency: 4}.ResolveConcurrencyOrDefault())
+ require.Equal(t, DefaultGraftResolverVerifyConcurrency, GraftResolver{}.VerifyConcurrencyOrDefault())
+ require.Equal(t, 3, GraftResolver{VerifyConcurrency: 3}.VerifyConcurrencyOrDefault())
require.Equal(t, DefaultGraftResolverResolveRounds, GraftResolver{}.ResolveRoundsOrDefault())
require.Equal(t, 7, GraftResolver{ResolveRounds: 7}.ResolveRoundsOrDefault())
}
diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.go
index bef5e67..18085ea 100644
--- a/internal/gitaly/service/operations/land_preresolve.go
+++ b/internal/gitaly/service/operations/land_preresolve.go
@@ -53,17 +53,46 @@ type landVerdicts map[landFileKey]landVerdict
// landPreResolution is what the pre-resolution of a land hands to its lane.
type landPreResolution struct {
verdicts landVerdicts
+ // verified are the trees that verify.command passed on, out of the lane (15.9): a tree, by its
+ // object ID, so that a tip that moved to the same tree keeps the verification.
+ verified map[git.ObjectID]bool
+ // unverifiable, when set, is why the land has run out of time to verify a model's resolution:
+ // the pre-resolution then verifies nothing and calls no resolver, and the files of the model
+ // become proposals with this reason.
+ unverifiable string
}
-// landNeedsResolution is the answer of a lane that met conflicting files without a verdict: new
-// files, or files whose sides changed since they were resolved. UserLand resolves them out of the
-// lane and lands again.
+func newLandPreResolution() *landPreResolution {
+ return &landPreResolution{verdicts: landVerdicts{}, verified: map[git.ObjectID]bool{}}
+}
+
+// markVerified records that verify.command passed on tree.
+func (p *landPreResolution) markVerified(tree git.ObjectID) {
+ if p.verified == nil {
+ p.verified = map[git.ObjectID]bool{}
+ }
+ p.verified[tree] = true
+}
+
+// isVerified reports whether verify.command passed on tree, out of the lane.
+func (p *landPreResolution) isVerified(tree git.ObjectID) bool {
+ return p != nil && tree != "" && p.verified[tree]
+}
+
+// landNeedsResolution is the answer of a lane that cannot land the land as it stands: it met
+// conflicting files without a verdict (new files, or files whose sides changed since they were
+// resolved), or the tree it would land is not one that verify.command passed on (unverified: the
+// tip moved). UserLand resolves and verifies again out of the lane, and lands again.
type landNeedsResolution struct {
tip, commit git.ObjectID
paths []string
+ unverified git.ObjectID
}
func (e *landNeedsResolution) Error() string {
+ if e.unverified != "" {
+ return fmt.Sprintf("the tree %s of the land at tip %s was not verified", e.unverified, e.tip)
+ }
return fmt.Sprintf("conflicting files to resolve at tip %s: %s", e.tip, strings.Join(e.paths, ", "))
}
@@ -112,10 +141,16 @@ func (s *Server) landPreResolve(ctx context.Context, in *gitalypb.UserLandReques
return nil
}
- l, _, err := s.landResolveAt(ctx, in, repo, tip, commitID, committer, pre.verdicts)
+ l, _, err := s.landResolveAt(ctx, in, repo, tip, commitID, committer, pre)
+ var unfinished *landVerifyUnfinishedError
+ var needs *landNeedsResolution
switch {
case err == nil, errors.Is(err, errLandNoConflicts):
return nil
+ case errors.As(err, &unfinished), errors.As(err, &needs):
+ // Not the land's failure: UserLand decides what a verification that ran out of time, or a
+ // land that still needs files resolved, comes to.
+ return err
}
// Only a conflict that the resolution left unresolved is a merge conflict, and only the policy's
// refusal is a refusal. Any other error (a git command that failed, a caller that gave up) means
@@ -141,35 +176,56 @@ func (s *Server) landPreResolve(ctx context.Context, in *gitalypb.UserLandReques
return landWithPackage(ctx, landMergeConflictError(tip, commitID, paths, err), err)
}
-// landResolving lands a request with resolve_conflicts (PLAN10 15.8): it pre-resolves out of the
-// lane and lands; when the lane meets files without a verdict, it resolves only those, out of the
-// lane, and lands again. The rounds are bounded by the resolver's timeout for the whole land, and
-// by resolve_rounds, a safety cap. A round is lost only when another land changed one of the
-// files, so the lands that contend for a file all land, one per round; a land that keeps losing
-// to new arrivals fails at its deadline with the merge conflict, naming the files that kept
-// changing.
+// landResolving lands a request with resolve_conflicts (PLAN10 15.8, 15.9): it pre-resolves out of
+// the lane and lands; when the lane meets files without a verdict, or a tree that verify.command
+// did not pass on, it resolves and verifies only what changed, out of the lane, and lands again.
+// The rounds are bounded by the deadline of the land, the resolver's timeout plus the verification's
+// when one is configured, and by resolve_rounds, a safety cap. A round is lost only when another
+// land changed the files or the tip, so the lands that contend for a file all land, one per round.
+// A land that runs out of its deadline fails with the merge conflict naming the files that kept
+// changing; but a model's resolution that could not be verified in time does not fail the agent: its
+// files become proposals, with the reason, in the package (landUnverified).
func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequest, committer git.Signature) (*gitalypb.UserLandResponse, error) {
timeout, err := s.resolveCfg.TimeoutOrDefault()
if err != nil {
return nil, fmt.Errorf("resolver timeout: %w", err)
}
+ if s.resolveCfg.Verify.Enabled {
+ verifyTimeout, err := s.resolveCfg.Verify.TimeoutOrDefault()
+ if err != nil {
+ return nil, fmt.Errorf("verify timeout: %w", err)
+ }
+ timeout += verifyTimeout
+ }
resolveCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
- pre := &landPreResolution{verdicts: landVerdicts{}}
+ pre := newLandPreResolution()
var need *landNeedsResolution
- // deadline reports whether the land ran out of the resolver's timeout, and not its caller.
+ // deadline reports whether the land ran out of its own timeout, and not its caller.
deadline := func() bool {
return ctx.Err() == nil && errors.Is(resolveCtx.Err(), context.DeadlineExceeded)
}
until := fmt.Sprintf("until the deadline of %s", timeout)
+ // giveUp ends a land that cannot go on: a model's resolution that was not verified in time goes
+ // to the agent as proposals; anything else is the merge conflict of files that kept changing.
+ giveUp := func(when, unverifiedReason string) (*gitalypb.UserLandResponse, error) {
+ if unverifiedReason != "" {
+ return s.landUnverified(ctx, in, committer, pre, unverifiedReason, need, when)
+ }
+ return nil, need.keptChanging(when)
+ }
rounds := s.resolveCfg.ResolveRoundsOrDefault()
for round := 1; round <= rounds; round++ {
if err := s.landPreResolve(resolveCtx, in, committer, pre); err != nil {
- // A land that lost a round to its files changing, and then ran out of its deadline
- // while it resolved them again, failed because they kept changing.
- if need != nil && deadline() {
- return nil, need.keptChanging(until)
+ var unfinished *landVerifyUnfinishedError
+ switch {
+ case errors.As(err, &unfinished) && deadline():
+ return s.landUnverified(ctx, in, committer, pre, fmt.Sprintf("could not verify before the deadline of %s", timeout), need, until)
+ case need != nil && deadline():
+ // A land that lost a round to its files or its tip changing, and then ran out of its
+ // deadline while it resolved or verified again, failed because they kept changing.
+ return giveUp(until, landNotVerifiedReason(need))
}
return nil, err
}
@@ -178,8 +234,54 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequest
return resp, err
}
if deadline() {
- return nil, need.keptChanging(until)
+ return giveUp(until, landNotVerifiedReason(need))
}
}
- return nil, need.keptChanging(fmt.Sprintf("after %d rounds", rounds))
+ return giveUp(fmt.Sprintf("after %d rounds", rounds), landNotVerifiedReason(need))
+}
+
+// landNotVerifiedReason is why a model's files become proposals when the lane handed the land back
+// for a tree that was not verified: the branch moved on after the verification. It is empty when the
+// lane asked for files to be resolved.
+func landNotVerifiedReason(need *landNeedsResolution) string {
+ if need != nil && need.unverified != "" {
+ return "could not verify before the branch moved again"
+ }
+ return ""
+}
+
+// landUnverified ends a land whose model resolution could not be verified in time. A verified AI
+// land is opportunistic, and losing the race is harmless: the agent gets the proposals. It
+// pre-resolves once more, with the request's context, in a new quarantine, verifying nothing and
+// calling no resolver (the verdicts are kept): the files of the model are PROPOSED with the reason,
+// the proposals are kept, and the land fails with the package. need, when set, is the state the
+// lane handed back, for the merge conflict if the land cannot be reported as proposals after all.
+func (s *Server) landUnverified(
+ ctx context.Context,
+ in *gitalypb.UserLandRequest,
+ committer git.Signature,
+ pre *landPreResolution,
+ reason string,
+ need *landNeedsResolution,
+ when string,
+) (*gitalypb.UserLandResponse, error) {
+ pre.unverifiable = reason
+ err := s.landPreResolve(ctx, in, committer, pre)
+ var needs *landNeedsResolution
+ switch {
+ case errors.As(err, &needs):
+ if need == nil {
+ need = needs
+ }
+ return nil, need.keptChanging(when)
+ case err != nil:
+ return nil, err
+ }
+ // The land is no longer a conflict to resolve (the change landed, or the merge is clean): the
+ // lane answers it.
+ resp, err := s.landOnce(ctx, in, committer, nil, pre)
+ if errors.As(err, &needs) {
+ return nil, needs.keptChanging(when)
+ }
+ return resp, err
}
diff --git a/internal/gitaly/service/operations/land_preresolve_test.go b/internal/gitaly/service/operations/land_preresolve_test.go
index d873f31..fa0590b 100755
--- a/internal/gitaly/service/operations/land_preresolve_test.go
+++ b/internal/gitaly/service/operations/land_preresolve_test.go
@@ -190,6 +190,10 @@ if [ "$N" = 1 ]; then
i=0
while [ ! -f "$V/release" ] && [ "$i" -lt 600 ]; do sleep 0.1; i=$((i+1)); done
fi
+if [ "$N" != 1 ] && [ -f "$V/fail" ]; then
+ echo "tests failed"
+ exit 3
+fi
exit 0`, vdir))
graft := batchingGraft()
graft.Land.Resolver = withVerify(enabledResolver(argv), verify, "")
@@ -258,9 +262,12 @@ func TestUserLand_resolvesOutOfTheLane(t *testing.T) {
t.Parallel()
h := newHeldLane(t)
+ // A model's answer for b, whose verification fails: b cannot land (15.9). The verification
+ // runs out of the lane too, while the lane is held.
proposal := answerOf("b")
proposal.Resolver = "graft-resolve/llm"
writeAnswerN(t, h.dir, 2, proposal)
+ require.NoError(t, os.WriteFile(filepath.Join(h.vdir, "fail"), nil, 0o600))
// The land of b returns while the lane is still held: it never queued behind a.
req := h.fx.request("Land b")
@@ -269,7 +276,8 @@ func TestUserLand_resolvesOutOfTheLane(t *testing.T) {
requireMergeConflict(t, err)
require.Equal(t, [][]byte{[]byte("b")}, landErrorDetail(t, err).GetMergeConflict().GetConflictingFiles())
requireRunning(t, h.first, "the land of a")
- require.Equal(t, 1, h.verifications(t))
+ require.Equal(t, 2, h.verifications(t), "a's, held in the lane, and b's, out of it")
+ require.Contains(t, packageFile(t, landConflictDetail(err), "b").GetReason(), "verification failed: exit status 3: tests failed")
// The proposal was kept although the lane was busy with a batch: it was written before the
// lane, on the published tip, never on a tip that exists only in a batch's quarantine
@@ -532,12 +540,11 @@ func TestLandApplyVerdicts_neverCallsTheResolver(t *testing.T) {
s := fx.server(t, rcfg)
in := fx.request("Land agent work")
- verdicts := landVerdicts{}
- _, _, err := s.landResolveAt(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), verdicts)
+ pre := newLandPreResolution()
+ _, _, err := s.landResolveAt(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), pre)
require.NoError(t, err)
require.Equal(t, 1, resolverCalls(t, dir))
- require.Len(t, verdicts, 2)
- pre := &landPreResolution{verdicts: verdicts}
+ require.Len(t, pre.verdicts, 2)
// At the tip they were made at, the verdicts are written.
res, err := s.landApplyVerdicts(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), pre)
diff --git a/internal/gitaly/service/operations/land_proposals.go b/internal/gitaly/service/operations/land_proposals.go
index 8b1e22e..13b9dbd 100644
--- a/internal/gitaly/service/operations/land_proposals.go
+++ b/internal/gitaly/service/operations/land_proposals.go
@@ -24,8 +24,9 @@ import (
"github.com/feder-cr/git-ai/proto/go/gitalypb"
)
-// A resolution that a model made is never landed by the server: it is a proposal, which the agent
-// reviews, edits or replaces, and lands as its own (PLAN10 14). The server keeps the proposals of a
+// A resolution that a model made lands only through the operator's verification (PLAN10 14, 15.9):
+// when verify.command ran and passed on the exact tree that lands. Otherwise it is a proposal, which
+// the agent reviews, edits or replaces, and lands as its own. The server keeps the proposals of a
// failed land under a hidden reference, which the agent reads by object ID (GetObjects).
// landProposalPruneErrors counts the prunings of proposal references that failed after a land.
@@ -41,8 +42,13 @@ var landProposalWriteErrors = promauto.NewCounter(prometheus.CounterOpts{
Help: "Resolver answers of a failed land that could not be kept as proposals.",
})
-// landProposedReason is the reason of a file that a model resolved.
-const landProposedReason = "a model resolved the file: the server never lands a model's resolution, it proposes it (see proposal); land it, edited or not, as your own resolution"
+// landProposedReason is the reason of a file that a model resolved and that did not land: a model's
+// resolution lands only after verify.command passed on the merged tree, and the reason that it did
+// not follows it (landNoVerifyReason, the output of the verification, or why it could not run).
+const landProposedReason = "a model resolved the file: it is proposed (see proposal), and lands by itself only after verify.command passed on the merged tree; land it, edited or not, as your own resolution"
+
+// landNoVerifyReason follows landProposedReason when no verify.command is configured.
+const landNoVerifyReason = "no verify.command is configured, so nothing verifies a model's resolution"
// landAnswerFlagsProposals reports whether a file of the answer is marked as a proposal.
func landAnswerFlagsProposals(answer *resolver.Response) bool {
@@ -198,18 +204,29 @@ func landTrailerValueOrQuoted(v string) string {
}
// decideLand is the one decision point of a land that asked the server to resolve its conflicts
-// (PLAN10 15.4): it lands only when the resolver answered, the answer holds no model's resolution,
-// and every conflicting file is resolved. A file that proposes is never landed. This is where a
-// landing gate for model resolutions would live; today there is none.
-func decideLand(answer *resolver.Response, files []*landFileState) bool {
+// (PLAN10 15.4): it lands only when the resolver answered and every conflicting file is resolved,
+// by a rule (RESOLVED) or by a model (PROPOSED). The one rule about a model: a file that proposes
+// may land only if verify.command ran and passed on the exact tree that lands. tree is the tree
+// the land would write and verified the tree the verification passed on, compared by object ID,
+// not by the tip they were made at; a verified tree is never the empty ID. No other code path
+// lands a model's resolution.
+func decideLand(answer *resolver.Response, files []*landFileState, tree, verified git.ObjectID) bool {
if answer == nil {
return false
}
- if landAnswerProposes(answer) {
+ proposes := landAnswerProposes(answer)
+ if proposes && (verified == "" || verified != tree) {
return false
}
for _, f := range files {
- if f.outcome != gitalypb.LandConflictFile_OUTCOME_RESOLVED {
+ switch f.outcome {
+ case gitalypb.LandConflictFile_OUTCOME_RESOLVED:
+ case gitalypb.LandConflictFile_OUTCOME_PROPOSED:
+ // Only an answer that proposes has a PROPOSED file, and it passed the rule above.
+ if !proposes {
+ return false
+ }
+ default:
return false
}
}
diff --git a/internal/gitaly/service/operations/land_proposals_test.go b/internal/gitaly/service/operations/land_proposals_test.go
index 5301a24..cde0b1d 100755
--- a/internal/gitaly/service/operations/land_proposals_test.go
+++ b/internal/gitaly/service/operations/land_proposals_test.go
@@ -54,7 +54,7 @@ func proposalRefTarget(t *testing.T, fx resolveFixture) string {
func TestUserLand_proposals(t *testing.T) {
t.Parallel()
- t.Run("a model's answer is proposed, never landed", func(t *testing.T) {
+ t.Run("a model's answer without a verify.command is proposed, not landed", func(t *testing.T) {
t.Parallel()
dir, argv := resolverScript(t, "")
@@ -79,7 +79,7 @@ func TestUserLand_proposals(t *testing.T) {
for _, p := range []string{"a", "b"} {
f := packageFile(t, pkg, p)
require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
- require.Contains(t, f.GetReason(), "never lands a model's resolution", p)
+ require.Contains(t, f.GetReason(), "no verify.command is configured", p)
prop := f.GetProposal()
require.NotNil(t, prop, p)
require.Equal(t, resolver.ActionWrite, prop.GetAction())
@@ -303,17 +303,41 @@ func TestDecideLand(t *testing.T) {
return []*landFileState{{outcome: gitalypb.LandConflictFile_OUTCOME_RESOLVED}, {outcome: gitalypb.LandConflictFile_OUTCOME_RESOLVED}}
}
rules := &resolver.Response{Resolver: "graft-resolve/trivial,union", Files: []resolver.ResponseFile{{Path: "a"}, {Path: "b"}}}
- require.True(t, decideLand(rules, resolved()), "rules resolved every file")
- require.False(t, decideLand(nil, resolved()), "no answer")
+ require.True(t, decideLand(rules, resolved(), "", ""), "rules resolved every file")
+ require.True(t, decideLand(rules, resolved(), "t1", "t2"), "rules land whatever tree was verified: their verification stays in the lane")
+ require.False(t, decideLand(nil, resolved(), "t1", "t1"), "no answer")
left := resolved()
left[1].outcome = gitalypb.LandConflictFile_OUTCOME_NOT_SENT
- require.False(t, decideLand(rules, left), "a file left to the agent")
-
+ require.False(t, decideLand(rules, left, "", ""), "a file left to the agent")
+ rejected := resolved()
+ rejected[0].outcome = gitalypb.LandConflictFile_OUTCOME_REJECTED
+ require.False(t, decideLand(rules, rejected, "", ""), "a file the answer is refused for")
+ proposedByRules := resolved()
+ proposedByRules[0].outcome = gitalypb.LandConflictFile_OUTCOME_PROPOSED
+ require.False(t, decideLand(rules, proposedByRules, "", ""), "a PROPOSED file in an answer that does not propose")
+
+ // THE rule (PLAN10 15.9): a file that proposes may land only if verify.command passed on the exact
+ // tree that lands, compared by object ID.
flagged := &resolver.Response{Resolver: "graft-resolve/trivial", Files: []resolver.ResponseFile{{Path: "a"}, {Path: "b", Proposal: true}}}
- require.False(t, decideLand(flagged, resolved()), "a file that proposes, even with every outcome RESOLVED")
named := &resolver.Response{Resolver: "graft-resolve/trivial,llm", Files: []resolver.ResponseFile{{Path: "a"}, {Path: "b"}}}
- require.False(t, decideLand(named, resolved()), "a resolver that names the llm strategy")
+ proposing := func() []*landFileState {
+ return []*landFileState{{outcome: gitalypb.LandConflictFile_OUTCOME_RESOLVED}, {outcome: gitalypb.LandConflictFile_OUTCOME_PROPOSED}}
+ }
+ for desc, answer := range map[string]*resolver.Response{"a flagged file": flagged, "the llm strategy by name": named} {
+ require.True(t, decideLand(answer, proposing(), "tree-1", "tree-1"), "%s: verified on this very tree", desc)
+ require.False(t, decideLand(answer, proposing(), "tree-1", ""), "%s: nothing verified", desc)
+ require.False(t, decideLand(answer, proposing(), "tree-1", "tree-2"), "%s: verified on another tree (the tip moved)", desc)
+ require.False(t, decideLand(answer, proposing(), "", ""), "%s: no tree at all: empty is not equal", desc)
+ require.False(t, decideLand(answer, resolved(), "tree-1", ""), "%s: even with every outcome RESOLVED", desc)
+
+ leftOver := proposing()
+ leftOver[0].outcome = gitalypb.LandConflictFile_OUTCOME_FAILED
+ require.False(t, decideLand(answer, leftOver, "tree-1", "tree-1"), "%s: verified, but a file is left to the agent", desc)
+ refused := proposing()
+ refused[1].outcome = gitalypb.LandConflictFile_OUTCOME_REJECTED
+ require.False(t, decideLand(answer, refused, "tree-1", "tree-1"), "%s: verified, but a file is refused", desc)
+ }
}
// TestResolveLandConflicts_policyFirst: an agent the policy does not let update the branch gets
diff --git a/internal/gitaly/service/operations/land_reland.go b/internal/gitaly/service/operations/land_reland.go
index 080ecb7..b739bf7 100644
--- a/internal/gitaly/service/operations/land_reland.go
+++ b/internal/gitaly/service/operations/land_reland.go
@@ -306,7 +306,7 @@ func (s *Server) landWithResolution(
for _, f := range l.files {
resp.Files = append(resp.Files, review.resolved[f.conflict.path])
}
- res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, review.actor, resp, checkedBy)
+ res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, review.actor, resp, checkedBy, "", false)
res.syntaxUnchecked = syntaxUnchecked
return res, l.verifyRejected(err)
}
diff --git a/internal/gitaly/service/operations/land_resolve.go b/internal/gitaly/service/operations/land_resolve.go
index 77b09aa..72d0f6c 100644
--- a/internal/gitaly/service/operations/land_resolve.go
+++ b/internal/gitaly/service/operations/land_resolve.go
@@ -77,24 +77,27 @@ type landFileConflict struct {
}
// landResolveAt resolves, against tip, the conflicting files of the land that have no verdict in
-// verdicts yet (PLAN10 15.8). It runs out of the lane: the policy first (an agent that may not
+// pre yet (PLAN10 15.8). It runs out of the lane: the policy first (an agent that may not
// update the branch gets nothing resolved), the listing, the classification of every file, one
-// resolver request for the files to resolve, taken with a resolve slot, and the validation; then
-// decideLand over every file, the verdicts reused included. A request carries only what a file's
-// verdict may depend on: the sides, the path and modes, the change's message, the configuration;
-// not the tip, its merge base or its message, so that a verdict can be reused at another tip
-// (15.8). When the land is landable, the new verdicts are added to verdicts, and the listing and
-// the answer to write are returned. Otherwise the answer is kept as proposals (14) and the land
-// fails with a *landUnresolvedError, whose package says what happened to each file; a request the
-// server does not attempt at all also gets a package, with the reason. Any other error means the
-// conflicts could not be listed, and the caller reports the merge conflict alone.
+// resolver request for the files to resolve, taken with a resolve slot, and the validation; then,
+// for an answer that holds a model's resolution, the verification of the exact tree that would
+// land (15.9), with a verify slot; then decideLand over every file, the verdicts reused included.
+// A request carries only what a file's verdict may depend on: the sides, the path and modes, the
+// change's message, the configuration; not the tip, its merge base or its message, so that a
+// verdict can be reused at another tip (15.8). When the land is landable, the new verdicts are
+// added to pre, the verified tree is recorded there, and the listing and the answer to write are
+// returned. Otherwise the answer is kept as proposals (14) and the land fails with a
+// *landUnresolvedError, whose package says what happened to each file, and why a model's
+// resolution did not land; a request the server does not attempt at all also gets a package, with
+// the reason. A verification that the context ended is a *landVerifyUnfinishedError. Any other
+// error means the conflicts could not be listed, and the caller reports the merge conflict alone.
func (s *Server) landResolveAt(
ctx context.Context,
in *gitalypb.UserLandRequest,
repo *localrepo.Repo,
tip, commitID git.ObjectID,
committer git.Signature,
- verdicts landVerdicts,
+ pre *landPreResolution,
) (landListing, *resolver.Response, error) {
// An agent that may not update the branch gets nothing resolved, and no proposal is kept for
// it: the policy decides on the actor and the reference alone, so it is checked first.
@@ -136,14 +139,17 @@ func (s *Server) landResolveAt(
}
// A file the classification sends and that has a verdict with its key is not resolved again.
+ // mine are the files with an answer, reused or fresh, whose outcome the answer decides.
reused := map[string]resolver.ResponseFile{}
var reusedNames []string
var sent []resolver.File
+ var mine []*landFileState
for _, f := range files {
if !f.sent() {
continue
}
- if v, ok := verdicts[landKeyOf(f.conflict)]; ok {
+ mine = append(mine, f)
+ if v, ok := pre.verdicts[landKeyOf(f.conflict)]; ok {
reused[f.conflict.path] = v.file
reusedNames = append(reusedNames, v.resolver)
f.outcome = gitalypb.LandConflictFile_OUTCOME_RESOLVED
@@ -151,6 +157,14 @@ func (s *Server) landResolveAt(
}
sent = append(sent, landRequestFile(f))
}
+ // Past its deadline a land only reports what it has: the resolver is not called.
+ if pre.unverifiable != "" && len(sent) > 0 {
+ paths := make([]string, 0, len(sent))
+ for _, f := range sent {
+ paths = append(paths, f.Path)
+ }
+ return l, nil, &landNeedsResolution{tip: tip, commit: commitID, paths: paths}
+ }
// answer is the resolver's accepted answer, the only thing ever written. A refusal is never
// written, whatever its outcomes say: answer stays nil, and the land fails with the package.
@@ -184,32 +198,55 @@ func (s *Server) landResolveAt(
landFailSent(files, gitalypb.LandConflictFile_OUTCOME_FAILED, fmt.Sprintf("the resolver's answer names an invalid resolver %q", resp.Resolver))
default:
answer = resp
- proposes := landAnswerProposes(resp)
- // A resolver that names the llm strategy but marks no file is taken to have proposed
- // every file: the mark may be missing, never the other way round.
- flagged := landAnswerFlagsProposals(resp)
- written := make(map[string]resolver.ResponseFile, len(resp.Files))
- for _, rf := range resp.Files {
- written[rf.Path] = rf
- }
- for _, f := range files {
- if !f.sent() {
- continue
- }
- rf := written[f.conflict.path]
- f.outcome = gitalypb.LandConflictFile_OUTCOME_RESOLVED
- switch err := landModeOfSides(f.conflict, rf); {
- case err != nil:
- f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_REJECTED, err.Error()
- case rf.Proposal || (proposes && !flagged):
- f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_PROPOSED, landProposedReason
+ }
+ }
+
+ combined := landCombinedAnswer(answer, reused, reusedNames, len(sent) > 0)
+ if combined != nil {
+ landSetOutcomes(combined, mine)
+ }
+ // The verdicts of an answer that nothing refuses are kept by their keys before it is verified or
+ // decided: the lane and the next round reuse them, whatever becomes of this verification.
+ if answer != nil && landAllAnswered(files) {
+ for _, f := range files {
+ for _, rf := range answer.Files {
+ if rf.Path == f.conflict.path {
+ pre.verdicts[landKeyOf(f.conflict)] = landVerdict{file: rf, resolver: answer.Resolver}
}
}
}
}
- combined := landCombinedAnswer(answer, reused, reusedNames, len(sent) > 0)
- landed := decideLand(combined, files)
+ // A model's resolution lands only through the operator's verification (15.9): verify.command
+ // runs here, out of the lane, on the exact tree that would land, once nothing else stops the
+ // land. Whatever keeps a model's file from landing is said in the reason of its file.
+ var tree, verified git.ObjectID
+ if combined != nil && landAnswerProposes(combined) {
+ switch {
+ case pre.unverifiable != "":
+ landAnnotateProposed(files, pre.unverifiable)
+ case !s.resolveCfg.Verify.Enabled:
+ landAnnotateProposed(files, landNoVerifyReason)
+ case landAllAnswered(files):
+ var err error
+ if tree, err = s.landResolvedTree(ctx, repo, l.mergedTree, combined); err != nil {
+ return l, nil, err
+ }
+ var verifyErr *landVerifyError
+ switch err := s.landVerifyOutOfLane(ctx, repo, tree); {
+ case err == nil:
+ verified = tree
+ pre.markVerified(tree)
+ s.logger.WithFields(log.Fields{"branch": string(in.GetBranch()), "tip": tip.String(), "commit": commitID.String(), "tree": tree.String()}).
+ InfoContext(ctx, "graft: a model's resolution passed the verification out of the lane")
+ case errors.As(err, &verifyErr):
+ landAnnotateProposed(files, verifyErr.Error())
+ default:
+ return l, nil, err
+ }
+ }
+ }
+ landed := decideLand(combined, files, tree, verified)
// An answer that does not land, because a model made part of it or another file is left, is
// kept for the agent: the files that rules resolved can be landed as they are. The proposals
// are an aid and the package is the contract: when they cannot be kept, the package still goes
@@ -229,16 +266,56 @@ func (s *Server) landResolveAt(
return l, nil, l.unresolved("", landUnresolvedCause(files), resolveErr)
}
- if answer != nil {
- for _, f := range files {
- for _, rf := range answer.Files {
- if rf.Path == f.conflict.path {
- verdicts[landKeyOf(f.conflict)] = landVerdict{file: rf, resolver: answer.Resolver}
- }
- }
+ return l, combined, nil
+}
+
+// landSetOutcomes sets the outcome of each of files from the answer that covers them: RESOLVED, or
+// REJECTED when the resolution writes a mode no side has, or PROPOSED when a model made the file.
+// A resolver that names the llm strategy but marks no file is taken to have proposed every file:
+// the mark may be missing, never the other way round. The lane sets them the same way from the
+// verdicts it applies, so that both ask decideLand the same question.
+func landSetOutcomes(answer *resolver.Response, files []*landFileState) {
+ proposes := landAnswerProposes(answer)
+ flagged := landAnswerFlagsProposals(answer)
+ written := make(map[string]resolver.ResponseFile, len(answer.Files))
+ for _, rf := range answer.Files {
+ written[rf.Path] = rf
+ }
+ for _, f := range files {
+ rf := written[f.conflict.path]
+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_RESOLVED, ""
+ switch err := landModeOfSides(f.conflict, rf); {
+ case err != nil:
+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_REJECTED, err.Error()
+ case rf.Proposal || (proposes && !flagged):
+ f.outcome, f.reason = gitalypb.LandConflictFile_OUTCOME_PROPOSED, landProposedReason
+ }
+ }
+}
+
+// landAllAnswered reports whether every file is resolved or proposed: nothing else keeps the land
+// from landing, so that a model's resolution is worth verifying.
+func landAllAnswered(files []*landFileState) bool {
+ for _, f := range files {
+ if f.outcome != gitalypb.LandConflictFile_OUTCOME_RESOLVED && f.outcome != gitalypb.LandConflictFile_OUTCOME_PROPOSED {
+ return false
+ }
+ }
+ return true
+}
+
+// landAnnotateProposed adds why a model's files did not land to the reason of each.
+func landAnnotateProposed(files []*landFileState, why string) {
+ for _, f := range files {
+ if f.outcome != gitalypb.LandConflictFile_OUTCOME_PROPOSED {
+ continue
+ }
+ if f.reason == "" {
+ f.reason = why
+ } else {
+ f.reason += "; " + why
}
}
- return l, combined, nil
}
// landResolveSlotted sends the resolve request with a resolve slot. The slot is given back when the
@@ -363,7 +440,36 @@ func (s *Server) landApplyVerdicts(
names = append(names, v.resolver)
}
answer := landCombinedAnswer(nil, reused, names, false)
- res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "")
+ landSetOutcomes(answer, l.files)
+
+ // The one rule about a model's resolution, asked of the tree this tip would land: it lands only
+ // if verify.command passed on this very tree (tree OIDs, not tips). When the tip moved, the tree
+ // is another one, and the land goes back out of the lane to be verified again. The lane never
+ // runs the verification of a model's resolution.
+ var tree, verified git.ObjectID
+ if landAnswerProposes(answer) {
+ if tree, err = s.landResolvedTree(ctx, repo, l.mergedTree, answer); err != nil {
+ return landResolution{}, err
+ }
+ if pre.isVerified(tree) {
+ verified = tree
+ }
+ }
+ if !decideLand(answer, l.files, tree, verified) {
+ need := &landNeedsResolution{tip: tip, commit: commitID, unverified: tree}
+ if tree == "" {
+ // Nothing to verify, yet not landable: a file the answer rejects. Resolving again says why.
+ for _, f := range l.files {
+ need.paths = append(need.paths, f.conflict.path)
+ }
+ }
+ return landResolution{}, need
+ }
+ checkedBy := ""
+ if verified != "" {
+ checkedBy = s.landVerifyTrailer()
+ }
+ res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, checkedBy, tree, verified != "")
return res, l.verifyRejected(err)
}
@@ -555,8 +661,12 @@ func landUnresolvedCause(files []*landFileState) string {
}
// landWriteResolution writes a resolution (the resolver's answer, or the agent's) into the merged
-// tree and commits it with parents [tip, commitID]; checkedBy names the resolver that checked an
-// agent's resolution, if one did. The commit message carries the trailers.
+// tree and commits it with parents [tip, commitID]; checkedBy names what checked it, if anything
+// did. The commit message carries the trailers. built is the resolved tree when the caller already
+// built it (landResolvedTree). verified says that verify.command passed on that very tree, out of
+// the lane (15.9), so that it is not run again; otherwise the operator's verification runs here,
+// in the lane, on the merged tree, before it lands: that is a resolution made by rules or by an
+// agent, which keeps its verification in the lane for now.
func (s *Server) landWriteResolution(
ctx context.Context,
in *gitalypb.UserLandRequest,
@@ -566,11 +676,9 @@ func (s *Server) landWriteResolution(
actor string,
resp *resolver.Response,
checkedBy string,
+ built git.ObjectID,
+ verified bool,
) (landResolution, error) {
- tree, err := repo.ReadTree(ctx, git.Revision(mergedTree.String()), localrepo.WithRecursive())
- if err != nil {
- return landResolution{}, fmt.Errorf("reading merged tree: %w", err)
- }
paths := make([]string, 0, len(resp.Files))
for _, f := range resp.Files {
paths = append(paths, f.Path)
@@ -581,35 +689,23 @@ func (s *Server) landWriteResolution(
if err != nil {
return landResolution{}, err
}
- for _, f := range resp.Files {
- if f.Action == resolver.ActionDelete {
- if err := tree.Delete(f.Path); err != nil && !errors.Is(err, localrepo.ErrEntryNotFound) {
- return landResolution{}, fmt.Errorf("deleting %q: %w", f.Path, err)
- }
- continue
- }
-
- blob, err := repo.WriteBlob(ctx, bytes.NewReader(f.Content), localrepo.WriteBlobConfig{Path: f.Path})
- if err != nil {
- return landResolution{}, fmt.Errorf("writing resolved %q: %w", f.Path, err)
- }
- entry := localrepo.TreeEntry{OID: blob, Mode: f.Mode, Path: path.Base(f.Path), Type: localrepo.Blob}
- if err := tree.Add(f.Path, entry, localrepo.WithOverwriteFile()); err != nil {
- return landResolution{}, fmt.Errorf("adding resolved %q: %w", f.Path, err)
+ treeID := built
+ if treeID == "" {
+ var err error
+ if treeID, err = s.landResolvedTree(ctx, repo, mergedTree, resp); err != nil {
+ return landResolution{}, err
}
}
- if err := tree.Write(ctx, repo); err != nil {
- return landResolution{}, fmt.Errorf("writing merged tree: %w", err)
- }
- // The optional verification runs on the merged tree, the resolution included, before it lands.
- if s.resolveCfg.Verify.Enabled {
- if err := s.landVerify(ctx, repo, tree.OID); err != nil {
+ // The optional verification runs on the merged tree, the resolution included, before it lands,
+ // unless it already passed on this tree out of the lane.
+ if s.resolveCfg.Verify.Enabled && !verified {
+ if err := s.landVerify(ctx, repo, treeID); err != nil {
return landResolution{}, err
}
}
newTip, err := repo.WriteCommit(ctx, localrepo.WriteCommitConfig{
- TreeID: tree.OID,
+ TreeID: treeID,
Message: message,
Parents: []git.ObjectID{tip, commitID},
AuthorName: committer.Name,
@@ -647,6 +743,36 @@ func (s *Server) landWriteResolution(
return landResolution{commit: newTip, resolvedPaths: resolvedPaths, resolver: resp.Resolver}, nil
}
+// landResolvedTree writes a resolution into the merged tree and returns the tree: the exact tree
+// that a land of this resolution would commit. Its object ID is what the verification is tied to.
+func (s *Server) landResolvedTree(ctx context.Context, repo *localrepo.Repo, mergedTree git.ObjectID, resp *resolver.Response) (git.ObjectID, error) {
+ tree, err := repo.ReadTree(ctx, git.Revision(mergedTree.String()), localrepo.WithRecursive())
+ if err != nil {
+ return "", fmt.Errorf("reading merged tree: %w", err)
+ }
+ for _, f := range resp.Files {
+ if f.Action == resolver.ActionDelete {
+ if err := tree.Delete(f.Path); err != nil && !errors.Is(err, localrepo.ErrEntryNotFound) {
+ return "", fmt.Errorf("deleting %q: %w", f.Path, err)
+ }
+ continue
+ }
+
+ blob, err := repo.WriteBlob(ctx, bytes.NewReader(f.Content), localrepo.WriteBlobConfig{Path: f.Path})
+ if err != nil {
+ return "", fmt.Errorf("writing resolved %q: %w", f.Path, err)
+ }
+ entry := localrepo.TreeEntry{OID: blob, Mode: f.Mode, Path: path.Base(f.Path), Type: localrepo.Blob}
+ if err := tree.Add(f.Path, entry, localrepo.WithOverwriteFile()); err != nil {
+ return "", fmt.Errorf("adding resolved %q: %w", f.Path, err)
+ }
+ }
+ if err := tree.Write(ctx, repo); err != nil {
+ return "", fmt.Errorf("writing merged tree: %w", err)
+ }
+ return tree.OID, nil
+}
+
// landClassify decides, for every file, whether it may be sent to the resolver, and reads the
// sides of those that may (and of those whose content decided against it, for the hunks of the
// package). The order: the path (a commit trailer must carry it, the protocol must carry it, the
diff --git a/internal/gitaly/service/operations/land_resolve_helper_test.go b/internal/gitaly/service/operations/land_resolve_helper_test.go
index 4f6dc9a..5c318c8 100644
--- a/internal/gitaly/service/operations/land_resolve_helper_test.go
+++ b/internal/gitaly/service/operations/land_resolve_helper_test.go
@@ -13,10 +13,19 @@ import (
// the land is landable: the pre-resolution (landResolveAt) and the lane's write in one call, as the
// tests of the resolution itself need it. UserLand runs the two apart, the first out of the lane.
func (s *Server) resolveLandConflicts(ctx context.Context, in *gitalypb.UserLandRequest, repo *localrepo.Repo, tip, commitID git.ObjectID, committer git.Signature) (landResolution, error) {
- l, answer, err := s.landResolveAt(ctx, in, repo, tip, commitID, committer, landVerdicts{})
+ pre := newLandPreResolution()
+ l, answer, err := s.landResolveAt(ctx, in, repo, tip, commitID, committer, pre)
if err != nil {
return landResolution{}, err
}
- res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "")
+ // As the lane does for a model's resolution: the tree is the one verify.command passed on, out
+ // of the lane, or the answer would not have been landable.
+ var tree git.ObjectID
+ if landAnswerProposes(answer) {
+ if tree, err = s.landResolvedTree(ctx, repo, l.mergedTree, answer); err != nil {
+ return landResolution{}, err
+ }
+ }
+ res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "", tree, pre.isVerified(tree))
return res, l.verifyRejected(err)
}
diff --git a/internal/gitaly/service/operations/land_review_test.go b/internal/gitaly/service/operations/land_review_test.go
index 2cff1d5..4f9e576 100644
--- a/internal/gitaly/service/operations/land_review_test.go
+++ b/internal/gitaly/service/operations/land_review_test.go
@@ -30,7 +30,7 @@ func TestResolveSlot_aPanickingResolverCallGivesItsSlotBack(t *testing.T) {
s.landResolveHook = func() { panic("a resolution failed") }
require.PanicsWithValue(t, "a resolution failed", func() {
- _, _, _ = s.landResolveAt(fx.ctx, fx.request("Land"), fx.repo, fx.tip, fx.commit, fx.committer(), landVerdicts{})
+ _, _, _ = s.landResolveAt(fx.ctx, fx.request("Land"), fx.repo, fx.tip, fx.commit, fx.committer(), newLandPreResolution())
})
require.Equal(t, 0, resolverCalls(t, dir), "the hook ran before the call")
diff --git a/internal/gitaly/service/operations/land_verified_test.go b/internal/gitaly/service/operations/land_verified_test.go
new file mode 100644
index 0000000..f2a2c2a
--- /dev/null
+++ b/internal/gitaly/service/operations/land_verified_test.go
@@ -0,0 +1,524 @@
+package operations
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/grpc/codes"
+ "google.golang.org/grpc/status"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/graft/proposals"
+ "github.com/feder-cr/git-ai/internal/graft/resolver"
+ "github.com/feder-cr/git-ai/internal/helper/text"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+// The tests of a model's resolution that lands through the operator's verification (PLAN10 15.9): it
+// lands only if verify.command passed, out of the lane, on the exact tree that lands.
+
+// countingVerify writes a verification command into a new temporary directory and returns the
+// directory and the argv of the command. Every run appends a line to dir/vlines before body runs,
+// and N is the number of runs so far. body may block, move the branch or fail.
+func countingVerify(t *testing.T, body string) (string, []string) {
+ t.Helper()
+
+ dir := t.TempDir()
+ script := fmt.Sprintf(`#!/bin/sh
+DIR='%s'
+echo x >> "$DIR/vlines"
+N=$(wc -l < "$DIR/vlines" | tr -d ' ')
+%s
+`, dir, body)
+ path := filepath.Join(dir, "verify.sh")
+ require.NoError(t, os.WriteFile(path, []byte(script), 0o755))
+ return dir, []string{path}
+}
+
+// verifyRuns returns the number of times the verification of dir ran.
+func verifyRuns(t *testing.T, dir string) int {
+ t.Helper()
+
+ raw, err := os.ReadFile(filepath.Join(dir, "vlines"))
+ if errors.Is(err, fs.ErrNotExist) {
+ return 0
+ }
+ require.NoError(t, err)
+ return len(raw) / 2
+}
+
+// moveBranchBody makes the n-th run of a countingVerify command move the branch to the commit in
+// dir/move-n, when the test wrote one (writeVerifyMoves), before it goes on.
+var moveBranchBody = fmt.Sprintf(`if [ -f "$DIR/move-$N" ]; then
+ "$(cat "$DIR/git")" -C "$(cat "$DIR/repo")" update-ref refs/heads/%s "$(cat "$DIR/move-$N")"
+fi`, git.DefaultBranch)
+
+// writeVerifyMoves lets the n-th run of the verification of dir move the branch of fx to moves[n-1].
+func writeVerifyMoves(t *testing.T, dir string, fx resolveFixture, moves ...git.ObjectID) {
+ t.Helper()
+
+ writeMoves(t, dir, fx, moves...)
+}
+
+// verifiedFixture is a land whose resolver answers with a model's resolution, and whose
+// verify.command is the countingVerify command with the given body.
+func verifiedFixture(t *testing.T, answer resolver.Response, verifyBody string, tune func(*config.GraftResolver)) (resolveFixture, string, string) {
+ t.Helper()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, answer)
+ vdir, vargv := countingVerify(t, verifyBody)
+ rcfg := withVerify(enabledResolver(rargv), vargv, "")
+ if tune != nil {
+ tune(&rcfg)
+ }
+ return newResolveFixture(t, rcfg), rdir, vdir
+}
+
+// landCommit returns the commit with the given ID.
+func landedCommit(t *testing.T, fx resolveFixture, resp *gitalypb.UserLandResponse) (*gitalypb.GitCommit, git.ObjectID) {
+ t.Helper()
+
+ id := git.ObjectID(resp.GetLandedCommitId())
+ commit, err := fx.repo.ReadCommit(fx.ctx, id.Revision())
+ require.NoError(t, err)
+ return commit, id
+}
+
+func TestUserLand_verifiedModelResolution(t *testing.T) {
+ t.Parallel()
+
+ t.Run("a model's resolution that passes the verification lands, signed", func(t *testing.T) {
+ t.Parallel()
+
+ fx, rdir, vdir := verifiedFixture(t, modelAnswer(), "grep -qx merged-a a && grep -qx merged-b b && grep -qx d d", nil)
+
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land agent work"))
+ require.NoError(t, err)
+ commit, landed := landedCommit(t, fx, resp)
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, landed)
+ require.Equal(t, []string{fx.tip.String(), fx.commit.String()}, commit.GetParentIds())
+ require.Equal(t, "graft-resolve/llm", resp.GetResolver())
+ require.Equal(t, "merged-a\n", fx.blobAt(t, landed, "a"))
+ require.Equal(t, "merged-b\n", fx.blobAt(t, landed, "b"))
+ body := string(commit.GetBody())
+ require.Contains(t, body, "Graft-Resolved-By: graft-resolve/llm\n")
+ require.Contains(t, body, "Graft-Checked-By: verify:verify.sh\n")
+ require.Equal(t, 1, resolverCalls(t, rdir))
+ // Once: out of the lane. The lane lands the verified tree without verifying it again.
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ require.Empty(t, proposalRefTarget(t, fx), "nothing was proposed")
+ })
+
+ t.Run("a model's file next to a rule's file lands when the tree passes", func(t *testing.T) {
+ t.Parallel()
+
+ answer := modelAnswer()
+ answer.Resolver = "graft-resolve/trivial,llm"
+ answer.Files[1].Proposal = false
+ fx, _, vdir := verifiedFixture(t, answer, "exit 0", nil)
+
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.NoError(t, err)
+ _, landed := landedCommit(t, fx, resp)
+ require.Equal(t, "merged-a\n", fx.blobAt(t, landed, "a"))
+ require.Equal(t, "merged-b\n", fx.blobAt(t, landed, "b"))
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ })
+
+ t.Run("a verification that fails gives a proposal and the output", func(t *testing.T) {
+ t.Parallel()
+
+ fx, rdir, vdir := verifiedFixture(t, modelAnswer(), `echo "2 tests failed: TestFoo"; exit 3`, nil)
+
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg)
+ require.Equal(t, proposals.Ref(git.DefaultBranch, fx.commit).String(), pkg.GetProposalRef())
+ require.NotEmpty(t, proposalRefTarget(t, fx))
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, pkg, p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "verification failed: exit status 3: 2 tests failed: TestFoo", p)
+ require.NotNil(t, f.GetProposal(), p)
+ require.Equal(t, "merged-"+p+"\n", string(gittest.Exec(t, fx.cfg, "-C", fx.repoPath, "cat-file", "blob", f.GetProposal().GetOid())))
+ }
+ require.Equal(t, 1, resolverCalls(t, rdir))
+ require.Equal(t, 1, verifyRuns(t, vdir), "once, out of the lane")
+ })
+
+ t.Run("a verification that times out is a failure with its reason", func(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ vdir, vargv := countingVerify(t, "sleep 30")
+ fx := newResolveFixture(t, withVerify(enabledResolver(rargv), vargv, "300ms"))
+
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ f := packageFile(t, landConflictDetail(err), "a")
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome())
+ require.Contains(t, f.GetReason(), "the command did not finish within 300ms")
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ })
+
+ t.Run("without a verify.command a model's resolution is a proposal", func(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ fx := newResolveFixture(t, enabledResolver(rargv))
+
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, landConflictDetail(err), p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "no verify.command is configured", p)
+ require.NotNil(t, f.GetProposal(), p)
+ }
+ })
+
+ t.Run("nothing is verified when another file keeps the land from landing", func(t *testing.T) {
+ t.Parallel()
+
+ // b is written with a mode no side has: refused. Verifying the tree would cost the operator's
+ // tests for a land that cannot land.
+ answer := modelAnswer()
+ answer.Files[1].Mode = "100755"
+ fx, _, vdir := verifiedFixture(t, answer, "exit 0", nil)
+
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ pkg := landConflictDetail(err)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, packageFile(t, pkg, "a").GetOutcome())
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_REJECTED, packageFile(t, pkg, "b").GetOutcome())
+ require.Equal(t, 0, verifyRuns(t, vdir))
+ })
+
+ t.Run("a tip that moved gets the new tree verified, then lands", func(t *testing.T) {
+ t.Parallel()
+
+ // While the first verification runs, a writer adds e to the branch: the tree to land is no
+ // longer the verified one, so the land goes back out, rebuilds the tree with the same verdicts
+ // (the resolver is not called again), verifies it, and lands that.
+ fx, rdir, vdir := verifiedFixture(t, modelAnswer(), moveBranchBody, nil)
+ writer := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.tip),
+ gittest.WithTreeEntries(entries("a", "ours\n", "b", "ours\n", "c", "c\n", "e", "e\n")...))
+ writeVerifyMoves(t, vdir, fx, writer)
+
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.NoError(t, err)
+ commit, landed := landedCommit(t, fx, resp)
+ require.Equal(t, []string{writer.String(), fx.commit.String()}, commit.GetParentIds())
+ require.Equal(t, "e\n", fx.blobAt(t, landed, "e"))
+ require.Equal(t, "merged-a\n", fx.blobAt(t, landed, "a"))
+ require.Equal(t, 1, resolverCalls(t, rdir), "no resolver call: the files did not change")
+ require.Equal(t, 2, verifyRuns(t, vdir), "the tree of the new tip is verified, out of the lane")
+ require.Contains(t, string(commit.GetBody()), "Graft-Checked-By: verify:verify.sh\n")
+ })
+
+ t.Run("a tip that moved to the same tree keeps the verification", func(t *testing.T) {
+ t.Parallel()
+
+ // The tree is what is verified, not the tip: a writer that moves the branch to another commit
+ // with the very tree the land started from leaves the verified tree the tree to land.
+ fx, _, vdir := verifiedFixture(t, modelAnswer(), moveBranchBody, nil)
+ same := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.tip), gittest.WithMessage("same tree, another commit"),
+ gittest.WithTreeEntries(entries("a", "ours\n", "b", "ours\n", "c", "c\n")...))
+ writeVerifyMoves(t, vdir, fx, same)
+
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.NoError(t, err)
+ commit, _ := landedCommit(t, fx, resp)
+ require.Equal(t, []string{same.String(), fx.commit.String()}, commit.GetParentIds())
+ require.Equal(t, 1, verifyRuns(t, vdir), "the same tree: verified once")
+ })
+
+ t.Run("a land the lane hands back for the last round gives a proposal, not a failure", func(t *testing.T) {
+ t.Parallel()
+
+ // resolve_rounds is a cap: past it, a tree the branch moved away from is not verified in time
+ // any more, whatever the clock says. The model's files are proposals, with the reason that the
+ // branch moved again, and the resolver is not called again.
+ fx, rdir, vdir := verifiedFixture(t, modelAnswer(), moveBranchBody, func(r *config.GraftResolver) {
+ r.ResolveRounds = 1
+ })
+ writer := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.tip),
+ gittest.WithTreeEntries(entries("a", "ours\n", "b", "ours\n", "c", "c\n", "e", "e\n")...))
+ writeVerifyMoves(t, vdir, fx, writer)
+
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ requireMergeConflict(t, err)
+ require.NotContains(t, err.Error(), "files kept changing")
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg)
+ require.Equal(t, writer.String(), pkg.GetTip(), "the package is about the tip the branch moved to")
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, pkg, p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "could not verify before the branch moved again", p)
+ require.NotNil(t, f.GetProposal(), p)
+ }
+ require.Equal(t, 1, resolverCalls(t, rdir))
+ require.Equal(t, 1, verifyRuns(t, vdir), "the give-up verifies nothing")
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, writer)
+ })
+
+ t.Run("churn past the deadline gives a proposal, not a failure", func(t *testing.T) {
+ t.Parallel()
+
+ // The branch moves under every verification, faster than the land can verify. At the deadline
+ // the model's files become proposals, with the reason, and the agent gets the package.
+ fx, rdir, vdir := verifiedFixture(t, modelAnswer(), "sleep 0.4\n"+moveBranchBody, func(r *config.GraftResolver) {
+ r.Timeout = "1s"
+ r.Verify.Timeout = "2s"
+ r.ResolveRounds = 1000
+ })
+ var moves []git.ObjectID
+ parent := fx.tip
+ for i := 0; i < 40; i++ {
+ parent = gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(parent),
+ gittest.WithTreeEntries(entries("a", "ours\n", "b", "ours\n", "c", "c\n", "e", fmt.Sprintf("e-%d\n", i))...))
+ moves = append(moves, parent)
+ }
+ writeVerifyMoves(t, vdir, fx, moves...)
+
+ // The bound is a check that the land does not hang, not a measure: the deadline of the land is
+ // 3 s, and a loaded machine takes a multiple of that for the pass that follows it.
+ start := time.Now()
+ _, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ took := time.Since(start)
+ t.Logf("the land ended after %s (deadline 3s)", took)
+ require.Less(t, took, 3*time.Minute)
+ requireMergeConflict(t, err)
+ require.NotContains(t, err.Error(), "files kept changing")
+ pkg := landConflictDetail(err)
+ require.NotNil(t, pkg)
+ require.NotEmpty(t, pkg.GetProposalRef())
+ for _, p := range []string{"a", "b"} {
+ f := packageFile(t, pkg, p)
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_PROPOSED, f.GetOutcome(), p)
+ require.Contains(t, f.GetReason(), "could not verify before", p)
+ require.NotNil(t, f.GetProposal(), p)
+ require.Equal(t, "merged-"+p+"\n", string(gittest.Exec(t, fx.cfg, "-C", fx.repoPath, "cat-file", "blob", f.GetProposal().GetOid())))
+ }
+ require.Equal(t, 1, resolverCalls(t, rdir), "the verdicts are reused: the resolver is called once")
+ // One round or several, by the speed of the machine: a loaded one spends the whole deadline on
+ // the first. What does not depend on speed is that the verification ran, that the writers did
+ // not run out (the deadline ended the land), and that the resolver was called once. The round
+ // that goes back out after a tip moved is the test above, deterministic.
+ require.GreaterOrEqual(t, verifyRuns(t, vdir), 1)
+ require.Less(t, verifyRuns(t, vdir), len(moves), "the deadline ended it, not the writers")
+ // The branch was only moved by the writers: the model's resolution did not land.
+ tip := gittest.ResolveRevision(t, fx.cfg, fx.repoPath, git.DefaultBranch)
+ parents := strings.Fields(text.ChompBytes(gittest.Exec(t, fx.cfg, "-C", fx.repoPath, "rev-list", "--parents", "-n1", tip.String())))
+ require.NotContains(t, parents, fx.commit.String())
+ })
+}
+
+func TestUserLand_verifiedModelResolution_laneIsFree(t *testing.T) {
+ t.Parallel()
+
+ // The first verification blocks until the test releases it. While it does, the branch's lane is
+ // free: a plain land of another change passes through it. When the verification is released, the
+ // tree of the model's land is not the tree that was verified (the plain land moved the branch),
+ // and the land verifies the new tree and lands on top of the plain land.
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ vdir, vargv := countingVerify(t, `if [ "$N" = 1 ]; then
+ i=0
+ while [ ! -f "$DIR/release" ] && [ "$i" -lt 1200 ]; do sleep 0.1; i=$((i+1)); done
+fi`)
+ graft := batchingGraft()
+ graft.Land.Resolver = withVerify(enabledResolver(rargv), vargv, "")
+ fx := newResolveFixtureIn(t, graft,
+ entries("a", "base\n", "b", "base\n", "c", "c\n"),
+ entries("a", "ours\n", "b", "ours\n", "c", "c\n"),
+ entries("a", "theirs\n", "b", "theirs\n", "c", "c\n", "d", "d\n"),
+ )
+ plain := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.base),
+ gittest.WithTreeEntries(entries("a", "base\n", "b", "base\n", "c", "c\n", "z", "z\n")...))
+
+ model := landAsync(fx, fx.commit, "Land the model's")
+ require.Eventually(t, func() bool { return verifyRuns(t, vdir) == 1 }, 30*time.Second, 10*time.Millisecond, "the verification never started")
+
+ // The verification holds until the test releases it (or two minutes pass): a plain land that ends
+ // before that went through a free lane, whatever the speed of the machine.
+ plainReq := fx.request("Land the plain one")
+ plainReq.CommitId = plain.String()
+ plainReq.ResolveConflicts = false
+ plainDone := make(chan landResult, 1)
+ go func() {
+ resp, err := fx.client.UserLand(fx.ctx, plainReq)
+ plainDone <- landResult{resp: resp, err: err}
+ }()
+ var plainResult landResult
+ select {
+ case plainResult = <-plainDone:
+ case <-time.After(60 * time.Second):
+ t.Cleanup(func() { _ = os.WriteFile(filepath.Join(vdir, "release"), nil, 0o600) })
+ require.FailNow(t, "the plain land did not end while the verification ran: the lane is held")
+ }
+ require.NoError(t, plainResult.err, "the lane is free while the verification runs")
+ plainResp := plainResult.resp
+ _, notReleased := os.Stat(filepath.Join(vdir, "release"))
+ require.ErrorIs(t, notReleased, fs.ErrNotExist, "the plain land ended before the release")
+ select {
+ case r := <-model:
+ require.FailNow(t, "the model's land ended before the verification was released", "%v %v", r.resp, r.err)
+ default:
+ }
+ require.Equal(t, 1, verifyRuns(t, vdir))
+
+ require.NoError(t, os.WriteFile(filepath.Join(vdir, "release"), nil, 0o600))
+ resp := requireLanded(t, model)
+ commit, landed := landedCommit(t, fx, resp)
+ require.Equal(t, []string{plainResp.GetLandedCommitId(), fx.commit.String()}, commit.GetParentIds())
+ require.Equal(t, "z\n", fx.blobAt(t, landed, "z"))
+ require.Equal(t, "merged-a\n", fx.blobAt(t, landed, "a"))
+ require.Equal(t, 2, verifyRuns(t, vdir))
+ require.Equal(t, 1, resolverCalls(t, rdir))
+}
+
+// TestUserLand_verifiedModelResolution_poolBoundsVerifications: with verify_concurrency 1, two lands
+// at once never run the operator's command at once.
+func TestUserLand_verifiedModelResolution_poolBoundsVerifications(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ vdir, vargv := countingVerify(t, `if ! mkdir "$DIR/lock" 2>/dev/null; then
+ echo x >> "$DIR/overlap"
+ sleep 0.3
+ exit 0
+fi
+sleep 0.3
+rmdir "$DIR/lock"`)
+ rcfg := withVerify(enabledResolver(rargv), vargv, "")
+ rcfg.VerifyConcurrency = 1
+ graft := batchingGraft()
+ graft.Land.Resolver = rcfg
+ fx := newResolveFixtureIn(t, graft,
+ entries("a", "base\n", "b", "base\n"),
+ entries("a", "ours\n", "b", "ours\n"),
+ entries("a", "theirs-0\n", "b", "theirs-0\n", "d0", "0\n"),
+ )
+ second := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.base),
+ gittest.WithTreeEntries(entries("a", "theirs-1\n", "b", "theirs-1\n", "d1", "1\n")...))
+
+ first := landAsync(fx, fx.commit, "Land 0")
+ other := landAsync(fx, second, "Land 1")
+ requireLanded(t, first)
+ requireLanded(t, other)
+
+ require.GreaterOrEqual(t, verifyRuns(t, vdir), 2)
+ _, err := os.Stat(filepath.Join(vdir, "overlap"))
+ require.ErrorIs(t, err, fs.ErrNotExist, "two verifications ran at once")
+}
+
+// TestLandApplyVerdicts_neverVerifiesAModelsResolution: the lane writes a model's resolution only
+// for a tree that verify.command passed on out of it, and never runs the command itself.
+func TestLandApplyVerdicts_neverVerifiesAModelsResolution(t *testing.T) {
+ t.Parallel()
+
+ rdir, rargv := resolverScript(t, "")
+ writeAnswer(t, rdir, modelAnswer())
+ vdir, vargv := countingVerify(t, "exit 0")
+ rcfg := withVerify(enabledResolver(rargv), vargv, "")
+ fx := newResolveFixture(t, rcfg)
+ s := fx.server(t, rcfg)
+ in := fx.request("Land")
+
+ // Out of the lane: the answer is resolved and its tree verified, once.
+ pre := newLandPreResolution()
+ _, _, err := s.landResolveAt(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), pre)
+ require.NoError(t, err)
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ require.Len(t, pre.verified, 1)
+
+ // In the lane, with the verified tree: it lands, and does not verify.
+ res, err := s.landApplyVerdicts(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), pre)
+ require.NoError(t, err)
+ require.Equal(t, "merged-a\n", fx.blobAt(t, res.commit, "a"))
+ require.Equal(t, 1, verifyRuns(t, vdir), "the lane did not run the command")
+ message := string(gittest.Exec(t, fx.cfg, "-C", fx.repoPath, "log", "-1", "--format=%B", res.commit.String()))
+ require.Contains(t, message, "Graft-Checked-By: verify:verify.sh\n")
+
+ // In the lane, with no tree verified (the tip moved to a tree nobody verified): it writes nothing
+ // and hands the land back to be verified; it does not verify either.
+ unverified := &landPreResolution{verdicts: pre.verdicts, verified: map[git.ObjectID]bool{}}
+ _, err = s.landApplyVerdicts(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), unverified)
+ var need *landNeedsResolution
+ require.ErrorAs(t, err, &need)
+ require.NotEmpty(t, need.unverified, "the tree to verify")
+ require.Equal(t, 1, verifyRuns(t, vdir), "the lane did not run the command")
+
+ // Another verified tree is not this one: tree OIDs are compared.
+ other := &landPreResolution{verdicts: pre.verdicts, verified: map[git.ObjectID]bool{"0000000000000000000000000000000000000000": true}}
+ _, err = s.landApplyVerdicts(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), other)
+ require.ErrorAs(t, err, &need)
+ require.Equal(t, 1, verifyRuns(t, vdir))
+}
+
+// TestUserLand_verifiedRulesKeepTheirVerificationInTheLane: a resolution made by rules alone keeps its
+// verify.command in the lane, as before (15.9): the next step moves it.
+func TestUserLand_verifiedRulesKeepTheirVerificationInTheLane(t *testing.T) {
+ t.Parallel()
+
+ fx, _, vdir := verifiedFixture(t, validAnswer(), "grep -qx merged-a a", nil)
+
+ resp, err := fx.client.UserLand(fx.ctx, fx.request("Land"))
+ require.NoError(t, err)
+ commit, _ := landedCommit(t, fx, resp)
+ require.Equal(t, 1, verifyRuns(t, vdir))
+ require.Contains(t, string(commit.GetBody()), "Graft-Resolved-By: graft-resolve/fake\n")
+ require.NotContains(t, string(commit.GetBody()), "Graft-Checked-By", "rules are unchanged")
+
+ failing, _, failingDir := verifiedFixture(t, validAnswer(), "exit 1", nil)
+ _, err = failing.client.UserLand(failing.ctx, failing.request("Land"))
+ requireMergeConflict(t, err)
+ for _, f := range landConflictDetail(err).GetFiles() {
+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_REJECTED, f.GetOutcome())
+ }
+ require.Equal(t, 1, verifyRuns(t, failingDir))
+}
+
+// TestUserLand_verifiedModelResolution_policyAndCancellation: a land whose caller gives up while it
+// verifies fails with the caller's cancellation, and not as proposals.
+func TestUserLand_verifiedModelResolution_callerGivesUp(t *testing.T) {
+ t.Parallel()
+
+ fx, _, vdir := verifiedFixture(t, modelAnswer(), "sleep 30", func(r *config.GraftResolver) { r.Verify.Timeout = "1m" })
+ ctx, cancel := context.WithCancel(fx.ctx)
+ req := fx.request("Land")
+ done := make(chan error, 1)
+ go func() {
+ _, err := fx.client.UserLand(ctx, req)
+ done <- err
+ }()
+ require.Eventually(t, func() bool { return verifyRuns(t, vdir) == 1 }, 30*time.Second, 10*time.Millisecond)
+ cancel()
+ select {
+ case err := <-done:
+ require.Error(t, err)
+ require.Equal(t, codes.Canceled, status.Code(err), "%v", err)
+ case <-time.After(30 * time.Second):
+ require.FailNow(t, "the land did not end when its caller gave up")
+ }
+ requireBranchAt(t, fx.cfg, fx.repoPath, git.DefaultBranch, fx.tip)
+}
diff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.go
index 00c4845..0f1f9ac 100644
--- a/internal/gitaly/service/operations/land_verify.go
+++ b/internal/gitaly/service/operations/land_verify.go
@@ -91,10 +91,11 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.
}
runErr := cmd.Wait()
switch {
- case errors.Is(runCtx.Err(), context.DeadlineExceeded):
- return &landVerifyError{reason: fmt.Sprintf("the command did not finish within %s", timeout)}
case ctx.Err() != nil:
+ // The caller's deadline or cancellation, not a verdict of the command.
return ctx.Err()
+ case errors.Is(runCtx.Err(), context.DeadlineExceeded):
+ return &landVerifyError{reason: fmt.Sprintf("the command did not finish within %s", timeout)}
case runErr != nil:
reason := strings.TrimSpace(output.String())
if code, ok := command.ExitStatus(runErr); ok {
@@ -105,6 +106,55 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.
return nil
}
+// landVerifyUnfinishedError is a verification that could not be completed, because its context ended
+// (the deadline of the land, or the caller) while it waited for a slot or ran: not a verdict.
+type landVerifyUnfinishedError struct {
+ err error
+}
+
+func (e *landVerifyUnfinishedError) Error() string {
+ return "verification not finished: " + e.err.Error()
+}
+
+func (e *landVerifyUnfinishedError) Unwrap() error { return e.err }
+
+// landVerifyOutOfLane runs the verification of a model's resolution out of the lane (PLAN10 15.9),
+// with a slot of the verify pool. It returns a *landVerifyError when the command fails or times
+// out, a *landVerifyUnfinishedError when ctx ended first, and another error when the verification
+// could not be run.
+func (s *Server) landVerifyOutOfLane(ctx context.Context, repo *localrepo.Repo, tree git.ObjectID) error {
+ slot, err := s.conflictResolver.AcquireVerifySlot(ctx)
+ if err != nil {
+ if ctx.Err() != nil {
+ return &landVerifyUnfinishedError{err: err}
+ }
+ return err
+ }
+ defer slot.Release()
+
+ err = s.landVerify(ctx, repo, tree)
+ var verifyErr *landVerifyError
+ switch {
+ case err == nil, errors.As(err, &verifyErr):
+ return err
+ case ctx.Err() != nil:
+ return &landVerifyUnfinishedError{err: err}
+ }
+ return err
+}
+
+// landVerifyTrailer is what Graft-Checked-By says of a verification that passed: verify:<command>,
+// the base name of the command, or verify when that is not a valid trailer value.
+func (s *Server) landVerifyTrailer() string {
+ name := "verify"
+ if cmd := s.resolveCfg.Verify.Command; len(cmd) > 0 {
+ if v := "verify:" + filepath.Base(cmd[0]); landTrailerValueValid(v) {
+ name = v
+ }
+ }
+ return name
+}
+
// landExtractTar writes the files of the archive of a tree under dir. It refuses an entry that
// would leave dir, or that lies under a symbolic link it wrote: a link is written as a link and
// never followed.
diff --git a/internal/graft/resolver/check.go b/internal/graft/resolver/check.go
index 1359c1a..3744c44 100644
--- a/internal/graft/resolver/check.go
+++ b/internal/graft/resolver/check.go
@@ -128,6 +128,47 @@ func (s *ResolveSlot) Resolve(ctx context.Context, req Request) (*Response, erro
return s.c.resolve(ctx, req)
}
+// verifySlotWait is the time the verifications waited for a verify slot.
+var verifySlotWait = promauto.NewHistogram(prometheus.HistogramOpts{
+ Name: "graft_resolver_verify_slot_wait_seconds",
+ Help: "Time a verification of a model's resolution waited for one of the resolver's verify slots.",
+ Buckets: prometheus.ExponentialBuckets(0.01, 4, 8),
+})
+
+// VerifySlot is the right to run one verification at a time: the client runs at most
+// verify_concurrency of them at once (AcquireVerifySlot), in a pool apart from the checks' and the
+// resolutions'. Release it when the verification is done.
+type VerifySlot struct {
+ c *Client
+ released bool
+}
+
+// AcquireVerifySlot waits for a verify slot. The wait counts against the deadline of ctx: when ctx
+// ends first, it fails, and the land cannot be verified in time.
+func (c *Client) AcquireVerifySlot(ctx context.Context) (*VerifySlot, error) {
+ if c == nil {
+ return nil, ErrDisabled
+ }
+ start := time.Now()
+ select {
+ case c.verifySlots <- struct{}{}:
+ verifySlotWait.Observe(time.Since(start).Seconds())
+ return &VerifySlot{c: c}, nil
+ case <-ctx.Done():
+ verifySlotWait.Observe(time.Since(start).Seconds())
+ return nil, fmt.Errorf("waiting for a verify slot of the resolver: %w", ctx.Err())
+ }
+}
+
+// Release gives the slot back; a second call does nothing.
+func (s *VerifySlot) Release() {
+ if s == nil || s.released {
+ return
+ }
+ s.released = true
+ <-s.c.verifySlots
+}
+
// Check sends one check request with the slot (Client.check).
func (s *CheckSlot) Check(ctx context.Context, req Request) (*CheckResponse, error) {
if s == nil || s.released {
diff --git a/internal/graft/resolver/client.go b/internal/graft/resolver/client.go
index 70ef97b..59c44fd 100644
--- a/internal/graft/resolver/client.go
+++ b/internal/graft/resolver/client.go
@@ -102,6 +102,9 @@ type Client struct {
// the short checks (PLAN10 15.8).
checkSlots chan struct{}
resolveSlots chan struct{}
+ // verifySlots bounds the verifications of a model's resolution that run at once (VerifySlot),
+ // a third pool, as the operator's command takes minutes (PLAN10 15.9).
+ verifySlots chan struct{}
// sendRules decide the files that are never sent (SendRules).
sendRules *SendRules
}
@@ -135,8 +138,8 @@ func New(cfg config.GraftResolver) (*Client, error) {
return nil, err
}
- if cfg.MaxFiles < 0 || cfg.MaxFileBytes < 0 || cfg.CheckConcurrency < 0 || cfg.ResolveConcurrency < 0 || cfg.ResolveRounds < 0 {
- return nil, errors.New("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency and resolve_rounds must not be negative")
+ if cfg.MaxFiles < 0 || cfg.MaxFileBytes < 0 || cfg.CheckConcurrency < 0 || cfg.ResolveConcurrency < 0 || cfg.VerifyConcurrency < 0 || cfg.ResolveRounds < 0 {
+ return nil, errors.New("graft land resolver: max_files, max_file_bytes, check_concurrency, resolve_concurrency, verify_concurrency and resolve_rounds must not be negative")
}
// The check of an agent's resolution sends a file's three sides and its resolution in one
// request, which must fit its bound.
@@ -159,6 +162,7 @@ func New(cfg config.GraftResolver) (*Client, error) {
sendRules: rules,
checkSlots: make(chan struct{}, cfg.CheckConcurrencyOrDefault()),
resolveSlots: make(chan struct{}, cfg.ResolveConcurrencyOrDefault()),
+ verifySlots: make(chan struct{}, cfg.VerifyConcurrencyOrDefault()),
}
if cfg.URL != "" {
diff --git a/internal/graft/resolver/client_test.go b/internal/graft/resolver/client_test.go
index 6997d2f..b3570d5 100644
--- a/internal/graft/resolver/client_test.go
+++ b/internal/graft/resolver/client_test.go
@@ -768,6 +768,55 @@ func TestAcquireResolveSlot_boundsTheResolutionsAtOnce(t *testing.T) {
require.ErrorIs(t, err, ErrDisabled)
}
+// TestAcquireVerifySlot_boundsTheVerificationsAtOnce: at most verify_concurrency slots are held at
+// once; a wait counts against the caller's deadline; a released slot frees its place once; the pool
+// is apart from the checks' and the resolutions'.
+func TestAcquireVerifySlot_boundsTheVerificationsAtOnce(t *testing.T) {
+ cfg := commandConfig([]string{"/bin/true"})
+ cfg.VerifyConcurrency = 1
+ cfg.CheckConcurrency = 1
+ cfg.ResolveConcurrency = 1
+ client := newClient(t, cfg)
+ ctx := testhelper.Context(t)
+
+ first, err := client.AcquireVerifySlot(ctx)
+ require.NoError(t, err)
+
+ waitCtx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
+ defer cancel()
+ _, err = client.AcquireVerifySlot(waitCtx)
+ require.ErrorIs(t, err, context.DeadlineExceeded)
+ require.ErrorContains(t, err, "waiting for a verify slot of the resolver")
+
+ // A held verify slot delays neither a check nor a resolution.
+ quick, cancelQuick := context.WithTimeout(ctx, time.Second)
+ defer cancelQuick()
+ checkSlot, err := client.AcquireCheckSlot(quick)
+ require.NoError(t, err)
+ resolveSlot, err := client.AcquireResolveSlot(quick)
+ require.NoError(t, err)
+ checkSlot.Release()
+ resolveSlot.Release()
+
+ first.Release()
+ first.Release()
+ second, err := client.AcquireVerifySlot(ctx)
+ require.NoError(t, err)
+ waitCtx2, cancel2 := context.WithTimeout(ctx, 50*time.Millisecond)
+ defer cancel2()
+ _, err = client.AcquireVerifySlot(waitCtx2)
+ require.ErrorIs(t, err, context.DeadlineExceeded, "a second release must not free a second place")
+ second.Release()
+
+ var none *VerifySlot
+ none.Release()
+ var disabled *Client
+ _, err = disabled.AcquireVerifySlot(ctx)
+ require.ErrorIs(t, err, ErrDisabled)
+
+ require.Equal(t, config.DefaultGraftResolverVerifyConcurrency, cap(newClient(t, commandConfig([]string{"/bin/true"})).verifySlots))
+}
+
// TestResolveSlots_apartFromTheCheckSlots: the resolutions and the checks have pools of their own
// (PLAN10 15.8): every resolve slot held does not delay a check, and every check slot held does not
// delay a resolution.
@@ -806,6 +855,7 @@ func TestAcquireResolveSlot_defaultConcurrency(t *testing.T) {
func TestNew_refusesNegativePools(t *testing.T) {
for name, set := range map[string]func(*config.GraftResolver){
"resolve_concurrency": func(c *config.GraftResolver) { c.ResolveConcurrency = -1 },
+ "verify_concurrency": func(c *config.GraftResolver) { c.VerifyConcurrency = -1 },
"resolve_rounds": func(c *config.GraftResolver) { c.ResolveRounds = -1 },
} {
cfg := commandConfig([]string{"/bin/true"})
diff --git a/internal/graft/resolver/protocol.go b/internal/graft/resolver/protocol.go
index a87ab21..1fd5dc2 100644
--- a/internal/graft/resolver/protocol.go
+++ b/internal/graft/resolver/protocol.go
@@ -206,8 +206,8 @@ type Response struct {
// ResponseFile is the resolution of one conflicting path. Content is base64
// on the wire and is only meaningful for ActionWrite. Proposal marks a
-// resolution made by a model: the server never lands it, it proposes it to the
-// agent (PLAN10 14).
+// resolution made by a model: the server lands it only if verify.command passed
+// on the tree that lands, and otherwise proposes it to the agent (PLAN10 14, 15.9).
type ResponseFile struct {
Path string `json:"path"`
Action string `json:"action"`
diff --git a/proto/go/gitalypb/operations.pb.go b/proto/go/gitalypb/operations.pb.go
index a598ac2..3dff2b1 100644
--- a/proto/go/gitalypb/operations.pb.go
+++ b/proto/go/gitalypb/operations.pb.go
@@ -278,8 +278,9 @@ const (
// OUTCOME_NOT_ATTEMPTED: the request was not resolved at all (see
// LandConflict.reason).
LandConflictFile_OUTCOME_NOT_ATTEMPTED LandConflictFile_Outcome = 5
- // OUTCOME_PROPOSED: a model resolved the file; the server never lands a
- // model's resolution, it proposes it (see proposal).
+ // OUTCOME_PROPOSED: a model resolved the file; the server lands a
+ // model's resolution only after verify.command passed on the merged tree,
+ // otherwise it proposes it (see proposal).
LandConflictFile_OUTCOME_PROPOSED LandConflictFile_Outcome = 6
)
diff --git a/proto/operations.proto b/proto/operations.proto
index ed5d3b0..541e3c1 100644
--- a/proto/operations.proto
+++ b/proto/operations.proto
@@ -1349,8 +1349,9 @@ message LandConflictFile {
// OUTCOME_NOT_ATTEMPTED: the request was not resolved at all (see
// LandConflict.reason).
OUTCOME_NOT_ATTEMPTED = 5;
- // OUTCOME_PROPOSED: a model resolved the file; the server never lands a
- // model's resolution, it proposes it (see proposal).
+ // OUTCOME_PROPOSED: a model resolved the file; the server lands a
+ // model's resolution only after verify.command passed on the merged tree,
+ // otherwise it proposes it (see proposal).
OUTCOME_PROPOSED = 6;
}
// path is the path of the conflict in the merged tree.
diff --git a/sdk/conflict.go b/sdk/conflict.go
index 4c0af30..512ca1f 100644
--- a/sdk/conflict.go
+++ b/sdk/conflict.go
@@ -28,8 +28,9 @@ func LandConflict(err error) (*gitalypb.LandConflict, bool) {
type ConflictSides struct {
Base, Ours, Theirs []byte
Unreadable map[string]string
- // Proposal is the resolver's resolution of the file, when it made one: a model's, which the
- // server never lands (OUTCOME_PROPOSED), or the rules' in a land that failed for another file.
+ // Proposal is the resolver's resolution of the file, when it made one: a model's, which did not
+ // land because no verification passed on it (OUTCOME_PROPOSED), or the rules' in a land that
+ // failed for another file.
// Land it, edited or not, as your own (ResolutionFromProposals).
Proposal *Proposal
}
diff --git a/tools/graft-agents/protocol/protocol.go b/tools/graft-agents/protocol/protocol.go
index 4709fe9..169bd95 100644
--- a/tools/graft-agents/protocol/protocol.go
+++ b/tools/graft-agents/protocol/protocol.go
@@ -151,8 +151,9 @@ type ResolvedFile struct {
Action string `json:"action"`
Mode string `json:"mode,omitempty"`
Content []byte `json:"content"`
- // Proposal marks a resolution made by a model (the llm strategy): the server never lands it,
- // it proposes it to the agent (PLAN10 14).
+ // Proposal marks a resolution made by a model (the llm strategy): the server lands it only if
+ // verify.command passed on the tree that lands, and otherwise proposes it to the agent
+ // (PLAN10 14, 15.9).
Proposal bool `json:"proposal,omitempty"`
}
diff --git a/tools/graft-agents/resolve/llm_strategy.go b/tools/graft-agents/resolve/llm_strategy.go
index 711714a..15c6ba7 100644
--- a/tools/graft-agents/resolve/llm_strategy.go
+++ b/tools/graft-agents/resolve/llm_strategy.go
@@ -181,7 +181,8 @@ func (s *llmStrategy) Resolve(ctx context.Context, f File, req protocol.Request)
func (s *llmStrategy) syntaxChecked() {}
// proposer marks a strategy whose results are proposals: those of a model, which the server
-// proposes to the agent and never lands (PLAN10 14).
+// proposes to the agent, and lands only when verify.command passed on the tree that lands
+// (PLAN10 14, 15.9).
type proposer interface{ proposes() }
// proposes marks the llm answers as proposals (resolveFile).
diff --git a/tools/graft-agents/resolve/resolve.go b/tools/graft-agents/resolve/resolve.go
index b2d8dbd..e6dbcdf 100644
--- a/tools/graft-agents/resolve/resolve.go
+++ b/tools/graft-agents/resolve/resolve.go
@@ -450,7 +450,8 @@ func (r *Resolver) resolveFile(ctx context.Context, req protocol.Request, f File
attempts = append(attempts, Attempt{Strategy: b.Name, Kind: AttemptCheckFailed, Err: err})
continue
}
- // A model resolution is a proposal: the server never lands it (PLAN10 14). It is not
+ // A model resolution is a proposal: the server lands it only after verify.command passed on
+ // the tree that lands (PLAN10 14, 15.9). It is not
// recorded either: a strategy that learns accepted answers (Recorder) would otherwise
// replay a model's resolution later as a rule's.
_, proposal := b.Strategy.(proposer)