Git designed for AI agents.
graft: review of acb1048: an empty link target is a refusal, a stale tree is never verified, the guard compares the tree
Federico Elia7ffd2189535b
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.mdindex 9098650..fb33c6f 100644--- a/doc/graft/CONFIGURATION.md+++ b/doc/graft/CONFIGURATION.md@@ -198,7 +198,7 @@ cause.| `max_file_bytes` | int | `1048576` (1 MiB) | Bound on every side of every file sent and on every resolved file accepted. At most 12566528 (about 12 MiB): one file, its three sides and its resolution, must fit one check request. || `proposal_ttl` | string | `"168h"` | How long the proposals of a failed land are kept under `refs/graft/proposals/` (Go duration, by the server's clock); at most 20 are kept per branch. Lands with a resolution and the housekeeping pass prune them. || `check_concurrency` | int | `2` | Checks of agents' resolutions that run at once in the Gitaly process, across all its listeners (one resolver client is shared by them); each holds a request of up to 64 MiB. A check waits for a free slot within its deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_check_slot_wait_seconds`. |-| `verify_per_repository` | int | `1` | Verifications that run at once for one repository (of any resolution: a rule's, an agent's, a model's), inside `verify_concurrency`: every land of a busy branch builds a tree of its own to verify, and one repository must not take the whole pool. A land waits for its repository's turn first, holding nothing of the pool, within the land's deadline. |+| `verify_per_repository` | int | `1` | Verifications that run at once for one repository (of any resolution: a rule's, an agent's, a model's), inside `verify_concurrency`: every land of a busy branch builds a tree of its own to verify, and one repository must not take the whole pool. A land waits for its repository's turn first, holding nothing of the pool, within the land's deadline. **The cost, stated:** a resolution lands only on the tree that was verified, so a land whose branch moved after its tree was verified is verified again, and on a busy branch the lands take their turns one repository at a time; at the deadline a land ends with the package (a rule's files `NOT_ATTEMPTED` with the reason, a model's `PROPOSED`) and the agent lands again. Lands of rules are batched only when `verify.enabled` is not set. || `verify_concurrency` | int | `2` | Verifications that run at once in the Gitaly process (of any resolution: a rule's, an agent's, a model's), all out of the lane, in a pool of their own that exists whenever `verify.enabled` is set, with or without an AI resolver, as `verify.command` takes minutes. A verification waits for a free slot within the land's deadline. The wait is in `graft_resolver_verify_slot_wait_seconds`. || `resolve_concurrency` | int | `2` | Resolutions that run at once in the Gitaly process, in a pool apart from the checks', so that long resolutions never delay a check. A resolution waits for a free slot within the land's deadline (`timeout`), and fails closed when it expires. The wait is in `graft_resolver_resolve_slot_wait_seconds`. || `resolve_rounds` | int | `32` | Safety cap on the rounds of a land whose conflicting files keep changing while it is resolved out of the lane. The rounds are bounded by `timeout`; the cap is set well above realistic contention and is no contract. |diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.mdindex 073e00e..c4d4b69 100644--- a/doc/graft/PLAN10.md+++ b/doc/graft/PLAN10.md@@ -928,7 +928,7 @@ not exist yet; (not enabled) marks what is designed and measured but switched of|-- conflicts, no resolve_conflicts ------------------------------------> merge_conflict|-- conflicts, every conflicting file has a verdict with its key, and the lane's| classification of the file agrees with the verdict's- | decideLand (15.4) on the tree the lane would write: with a verify.command configured, any+ | decideLand (15.4) on the tree the lane would write: with verify.enabled set, any| resolution (a rule's, a model's) LANDS only if that tree is the tree| verify.command passed on (tree object IDs), and a model's on no other tree,| configured or not. The lane runs no verification of any resolution. Merge@@ -941,9 +941,10 @@ not exist yet; (not enabled) marks what is designed and measured but switched of-> NEEDS VERIFICATION, out of the lane: rebuild the tree from the same verdicts (theresolver is called only for files that changed), VERIFY it, enter the lane againThe rounds are bounded by the deadline of the land (the resolver's timeout, plus verify.timeout when- a verify.command is configured) and by resolve_rounds (default 32, a safety cap, no contract). At+ verify.enabled is set) and by resolve_rounds (default 32, a safety cap, no contract). Atthe deadline: the final pass (15.9): the merge conflict with its PACKAGE, the model's files PROPOSED- ("could not verify before the branch moved again"), the rules' files RESOLVED, the files that changed+ ("could not verify before the branch moved again"), the rules' files RESOLVED (NOT_ATTEMPTED with the+ reason when verify.enabled is set and their tree could not be verified), the files that changedunder the land NOT_ATTEMPTED with the reason (a verified AI land is opportunistic: losing the raceis harmless, the agent gets the proposal).The resolver is never called inside the lane, and neither is the verification of a model's resolution.@@ -998,7 +999,7 @@ not exist yet; (not enabled) marks what is designed and measured but switched ofvVERDICT PROPOSAL (PROPOSED, the proposal + its report)v- verify.command configured? --no--> stays a PROPOSAL: the agent decides ("no verify.command")+ verify.enabled set? --no--> stays a PROPOSAL: the agent decides ("no verify.command")| yes: the exact tree that would land is verified, out of the lane (15.9)|-- fails ------------------------------------------------> stays a PROPOSAL (+ the verify output)|-- not verified before the branch moved again -----------> stays a PROPOSAL (+ the reason)@@ -1033,7 +1034,7 @@ not exist yet; (not enabled) marks what is designed and measured but switched of```decideLand(answer, files, tree, verified): the pre-resolution calls it with the tree it verified,the lane with the tree it would write- every file resolved by an operator rule ...... LAND, and with a verify.command configured only if it ran+ every file resolved by an operator rule ...... LAND, and with verify.enabled set only if it ranand passed on THIS tree (tree object ID equality)a file a model resolved ...................... LAND only if verify.command ran and passed on THIStree: tree object ID equality, not the tip@@ -1057,7 +1058,7 @@ not exist yet; (not enabled) marks what is designed and measured but switched of| the tree that would land | `landResolvedTree` | operations/land_resolve.go || VERIFY, the one box, out of the lane, for every resolution | `landVerifyOutOfLane`, `landVerify`, `landPrecheckVerify` (an agent's), `VerifyPool.Acquire` | operations/land_verify.go, land_reland.go; graft/resolver/check.go || verdicts applied in the lane | `landApplyVerdicts`, `landNeedsResolution` | operations/land_resolve.go, land_preresolve.go |-| unverified in time: proposals | `landUnverified`, `landNotVerifiedReason` | operations/land_preresolve.go |+| unverified in time: proposals | `landAnnotateUnverified`, `landStoppedReason`, `landFinalPass` | operations/land_resolve.go, land_preresolve.go || resolver call, bounded slots | `Client.AcquireResolveSlot`, `ResolveSlot.Resolve` | graft/resolver/check.go || policy before resolving | `hook.CheckAgentPolicy`, `landPolicyDeniedError` | operations/land_resolve.go, land_proposals.go || classify | `landClassify`, `resolver.SendRules` | operations/land_resolve.go; graft/resolver |@@ -1191,9 +1192,11 @@ The resolution is bound to the files instead (the PM's correction):today (binding agents' resolutions to the files too is a possible later extension, not now).* **Landable: applied in the lane, file by file.** The lane merges its own tip and the change. Whenevery conflicting file has a landable verdict with its key, and the lane's classification of the- file agrees, the lane writes the verdicts into the merge commit (and runs verify.command); the- resolver is not called. In a batch, a member's verdicts hold whenever their keys match, so- members chain.+ file agrees, the lane writes the verdicts into the merge commit and runs no command (15.9); the+ resolver is not called. In a batch, a member's verdicts hold whenever their keys match; without+ verify.enabled the members chain, one on the other's tip. With it a member lands only on a tree+ that passed, so a member whose tree is not the verified one (the batch changed the tip under it) goes+ back out to be verified, and lands alone.* **Needs resolution.** When a conflicting file has no verdict with its key (it is new, or a sidechanged), the lane returns the attempt with those files, without resolving. UserLandpre-resolves only them against the new tip, out of the lane, keeps every other verdict, and@@ -1270,12 +1273,13 @@ and the tree it verifies depends on the tip.an agent's resolution gets "could not verify before the deadline of <t>" with its package); anagent's resolution is verified in its precheck, at the only tip the lane accepts it at. Theper-process worst case of a verification, so that the operator can size it: inodes and disk at- most verify_concurrency x (verify.max_tree_entries inodes, verify.max_tree_bytes of files), heap- at most verify_concurrency x (the paths kept, 128 MiB by the charge of the materializer; the- entries and the target of no link), and a TMPDIR on tmpfs turns that disk into memory: point it- at a disk, or lower the limits.+ most verify_concurrency x (verify.max_tree_entries inodes, verify.max_tree_bytes of files), and+ that is the checkout alone: what the command itself writes (a build directory, a cache) is the+ operator's to bound, and counts against the same disk; heap at most verify_concurrency x (the+ paths kept, 128 MiB by the charge of the materializer; the entries and the target of no link);+ and a TMPDIR on tmpfs turns that disk into memory: point it at a disk, or lower the limits.* **Verification out of the lane, on the exact tree.** For an answer that holds a model's- resolution (or any answer, with a verify.command), and when nothing else keeps the land from landing (every file resolved or proposed),+ resolution (or any answer, with verify.enabled set), and when nothing else keeps the land from landing (every file resolved or proposed),the pre-resolution builds the tree that would land (`landResolvedTree`: the merged tree of thetip and the change, the answer's files written in) and runs verify.command on it, in thequarantine, with a slot of the verify pool (`verify_concurrency`, default 2, a pool of its own:@@ -1300,7 +1304,7 @@ and the tree it verifies depends on the tip.the whole pool, and the deadline of the land. A land that waits for its repository's turn holdsnothing of the pool, and a wait that outlasts the deadline ends in proposals like the rest.* **Deadline: a proposal, not a failure.** The deadline of the land is the resolver's timeout plus- verify.timeout when a verify.command is configured. A verified AI land is opportunistic: when the+ verify.timeout when verify.enabled is set. A verified AI land is opportunistic: when thedeadline ends before a tree could be verified and landed, the land does not fail for the agent. Themodel's files become PROPOSED with the reason "could not verify before the branch moved again" (or"before the deadline of <t>" when the verification itself was cut), the proposals are kept (14), and@@ -1309,8 +1313,8 @@ and the tree it verifies depends on the tip.fell (the listing, the tree, the resolution, the verification), the cap on the rounds, and a land thatthe lane or its batch kept handing back (a model's resolution whose batch ran out of rebuilds ishanded back as a land that needs its new tree verified, not failed with "kept moving"). The files- that have a verdict are reported with it (a model's PROPOSED, a rule's RESOLVED, their proposals- kept); the files that changed under the land and have none are NOT_ATTEMPTED, with the reason. With+ that have a verdict are reported with it (a model's PROPOSED, a rule's RESOLVED, or NOT_ATTEMPTED with+ the reason when verify.enabled is set and its tree could not be verified; their proposals kept); the files that changed under the land and have none are NOT_ATTEMPTED, with the reason. Withnothing known yet it is the merge conflict with its package and no proposals, with the reason. Aland that had a conflict never returns a bare deadline to the agent: only a caller that gives upgets its cancellation.diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.goindex 1e900e7..1f0d969 100644--- a/internal/gitaly/service/operations/land_preresolve.go+++ b/internal/gitaly/service/operations/land_preresolve.go@@ -190,9 +190,14 @@ func (s *Server) landPreResolve(ctx context.Context, in *gitalypb.UserLandRequesl, _, err := s.landResolveAt(ctx, in, repo, tip, commitID, committer, pre)var unfinished *landVerifyUnfinishedErrorvar needs *landNeedsResolution+ var moved *landTipMovedErrorswitch {case err == nil, errors.Is(err, errLandNoConflicts):return nil+ case errors.As(err, &moved):+ // The branch moved while the verification waited for its turn: the verdicts are kept, and the+ // lane hands the land back with the tree of the new tip to be verified (nothing stale was).+ return nilcase errors.As(err, &unfinished), errors.As(err, &needs):// Not the land's failure: UserLand decides what a verification that ran out of time, or a// land that still needs files resolved, comes to.@@ -313,17 +318,18 @@ func landStoppedReason(need *landNeedsResolution, timeout time.Duration) string}// landFinalPasses bounds the passes of landFinalPass: each is cheap, as it calls no resolver and runs-// no verification of a model's resolution, so that losing the branch to another land four times in a-// row is a branch that moves by the millisecond. (A land of rules only may run verify.command once-// in the lane, in the pass that lands; that is the lane's verification, which is not repeated.)+// no verification, so that losing the branch to another land four times in a row is a branch that+// moves by the millisecond. Nothing is verified in the lane either (15.9): a land that reaches it+// without a verified tree is handed back, and the pass ends it with the package.const landFinalPasses = 4// landFinalPass ends a land that could not go on (landResolving): the agent gets the package, not// a deadline. A verified AI land is opportunistic, and losing the race is harmless. It pre-resolves// once more with the request's context, in a new quarantine, verifying nothing and calling no// resolver: the verdicts that are kept give the proposals (a model's files PROPOSED with the reason,-// the rules' files RESOLVED), the files without a verdict are NOT_ATTEMPTED with the reason, and the-// land fails with the package. That is every ending of a land that had a conflict: the package is+// the rules' files RESOLVED, or NOT_ATTEMPTED with the reason when a verification is configured and+// their tree could not be verified), the files without a verdict are NOT_ATTEMPTED with the reason, and+// the land fails with the package. That is every ending of a land that had a conflict: the package is// what landPreResolve returns, and nothing else comes out of it but the refusal of the policy or an// error of the server. When there is nothing to report as a conflict after all (the change landed,// the merge is clean, or every file has a verdict a rule made), the lane answers, and lands.diff --git a/internal/gitaly/service/operations/land_reland.go b/internal/gitaly/service/operations/land_reland.goindex 1be7c58..89407e4 100644--- a/internal/gitaly/service/operations/land_reland.go+++ b/internal/gitaly/service/operations/land_reland.go@@ -6,6 +6,7 @@ import ("fmt""slices""strings"+ "time""github.com/prometheus/client_golang/prometheus""github.com/prometheus/client_golang/prometheus/promauto"@@ -168,10 +169,13 @@ type landPrecheck struct {//// The check runs in the quarantine repo, which the caller discards: the merge it computes leaves no// object behind. A resolution for a tip the branch has left, or for a change that already landed,-// is not checked (skipped): the lane refuses or answers it. It returns nil when no resolver is-// configured. A failure to check is the result's failure, with its cause, which the lane reports.+// is not checked (skipped): the lane refuses or answers it. It returns nil when there is neither a+// resolver nor a verification (verify.enabled) to run, and when the configured resolver cannot be+// built (the lane refuses the resolution with that reason before anything is verified). A failure to+// check is the result's failure, with its cause, which the lane reports. The whole precheck has one+// deadline: the resolver's timeout, plus the verification's when verify.enabled is set.func (s *Server) landPrecheckResolution(ctx context.Context, in *gitalypb.UserLandRequest, repo *localrepo.Repo) (*landPrecheck, error) {- if s.conflictResolver == nil && !s.resolveCfg.Verify.Enabled {+ if s.conflictResolver == nil && (s.resolverErr != nil || !s.resolveCfg.Verify.Enabled) {return nil, nil}objectHash, err := repo.ObjectHash(ctx)@@ -218,19 +222,31 @@ func (s *Server) landPrecheckResolution(ctx context.Context, in *gitalypb.UserLareturn &landPrecheck{tip: tip, commit: commitID, result: landCheckResult{failure: fmt.Sprintf("listing the conflicts: %v", err), failureErr: err}}, nil}- var result landCheckResult- if s.conflictResolver != nil {- timeout, err := s.resolveCfg.TimeoutOrDefault()+ timeout, err := s.resolveCfg.TimeoutOrDefault()+ if err != nil {+ return nil, structerr.NewInternal("resolver timeout: %w", err)+ }+ // One budget for the whole precheck: the check takes the resolver's timeout at most, and the+ // verification what the budget has left, so that the deadline is the one the docs say.+ budget := timeout+ if s.resolveCfg.Verify.Enabled {+ verifyTimeout, err := s.resolveCfg.Verify.TimeoutOrDefault()if err != nil {- return nil, structerr.NewInternal("resolver timeout: %w", err)+ return nil, structerr.NewInternal("verify timeout: %w", err)}- checkCtx, cancel := context.WithTimeout(ctx, timeout)- defer cancel()+ budget += verifyTimeout+ }+ budgetCtx, cancel := context.WithTimeout(ctx, budget)+ defer cancel()+ var result landCheckResult+ if s.conflictResolver != nil {+ checkCtx, cancelCheck := context.WithTimeout(budgetCtx, timeout)+ defer cancelCheck()result = s.landCheckResolution(checkCtx, in, repo, review.listing, review.actor, review.resolved)}pc := &landPrecheck{tip: tip, commit: commitID, result: result}if s.resolveCfg.Verify.Enabled && result.failure == "" && !landCheckRejects(result) {- if err := s.landPrecheckVerify(ctx, in, repo, review, pc); err != nil {+ if err := s.landPrecheckVerify(ctx, budgetCtx, budget, in, repo, review, pc); err != nil {return nil, err}}@@ -259,40 +275,48 @@ func landAgentResponse(review landReview) *resolver.Response {}// landPrecheckVerify verifies the tree of the agent's resolution out of the lane (15.9), with a slot-// of the verify pool, before the land enters its lane, and records the outcome in pc. The deadline is-// the resolver's timeout plus the verification's; a verification that the deadline cut is recorded-// as unfinished, and one that the caller cut is the caller's cancellation. The resolution was made-// against the tip the branch is at, which the lane insists on, so the tree it would write there is-// the tree verified here.-func (s *Server) landPrecheckVerify(ctx context.Context, in *gitalypb.UserLandRequest, repo *localrepo.Repo, review landReview, pc *landPrecheck) error {- timeout, err := s.resolveCfg.TimeoutOrDefault()- if err != nil {- return structerr.NewInternal("resolver timeout: %w", err)- }- verifyTimeout, err := s.resolveCfg.Verify.TimeoutOrDefault()- if err != nil {- return structerr.NewInternal("verify timeout: %w", err)- }- tree, err := s.landResolvedTree(ctx, repo, review.listing.mergedTree, landAgentResponse(review))+// of the verify pool, before the land enters its lane, and records the outcome in pc. caller is the+// request's context and budgetCtx the one of the precheck's budget (the resolver's timeout plus the+// verification's); a verification that the budget cut is recorded as unfinished, and one that the+// caller cut is the caller's cancellation. The resolution was made against the tip the branch is at,+// which the lane insists on, so the tree it would write there is the tree verified here; the branch+// is read again once the slot is taken, and a tree of a tip the branch has left is not verified.+func (s *Server) landPrecheckVerify(caller, budgetCtx context.Context, budget time.Duration, in *gitalypb.UserLandRequest, repo *localrepo.Repo, review landReview, pc *landPrecheck) error {+ tree, err := s.landResolvedTree(budgetCtx, repo, review.listing.mergedTree, landAgentResponse(review))if err != nil {+ if caller.Err() != nil {+ return landCallerGaveUp(caller, err)+ }+ if budgetCtx.Err() != nil {+ pc.verifyErr = fmt.Errorf("could not verify before the deadline of %s", budget)+ return nil+ }return structerr.NewInternal("building the tree to verify: %w", err)}- vctx, cancel := context.WithTimeout(ctx, timeout+verifyTimeout)- defer cancel()+ current := func(ctx context.Context) (bool, error) {+ now, err := repo.ResolveRevision(ctx, git.NewReferenceNameFromBranchName(string(in.GetBranch())).Revision())+ if errors.Is(err, git.ErrReferenceNotFound) {+ return false, nil+ }+ return now == pc.tip, err+ }var verifyErr *landVerifyError- switch err := s.landVerifyOutOfLane(vctx, in.GetRepository(), repo, tree); {+ var moved *landTipMovedError+ switch err := s.landVerifyOutOfLane(budgetCtx, in.GetRepository(), repo, tree, current); {case err == nil:pc.verified = treecase errors.As(err, &verifyErr):pc.verifyErr = err+ case errors.As(err, &moved):+ pc.verifyErr = errdefault:var unfinished *landVerifyUnfinishedError- if errors.As(err, &unfinished) && ctx.Err() == nil {- pc.verifyErr = fmt.Errorf("could not verify before the deadline of %s", timeout+verifyTimeout)+ if errors.As(err, &unfinished) && caller.Err() == nil {+ pc.verifyErr = fmt.Errorf("could not verify before the deadline of %s", budget)return nil}- if ctx.Err() != nil {- return landCallerGaveUp(ctx, err)+ if caller.Err() != nil {+ return landCallerGaveUp(caller, err)}return err}@@ -383,8 +407,7 @@ func (s *Server) landWithResolution(resp := landAgentResponse(review)// The verification ran out of the lane (15.9): the lane compares the tree it would write with// the one that passed, and writes nothing that did not.- var built git.ObjectID- verified := false+ var built, verified git.ObjectIDif s.resolveCfg.Verify.Enabled {switch reason := landPrecheckUnusable(precheck, tip, commitID); {case reason != "":@@ -400,10 +423,13 @@ func (s *Server) landWithResolution(if err != nil {return landResolution{}, err}- if precheck.verified == "" || precheck.verified != tree {+ switch {+ case precheck.verified == "":+ return landResolution{}, l.notAttempted("the resolution could not be verified: the precheck did not verify it", nil)+ case precheck.verified != tree:return landResolution{}, l.notAttempted("the resolution could not be verified: the tree to land is not the tree that was verified", nil)}- built, verified = tree, true+ built, verified = tree, precheck.verified}res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, review.actor, resp, checkedBy, built, verified)res.syntaxUnchecked = syntaxUncheckeddiff --git a/internal/gitaly/service/operations/land_resolve.go b/internal/gitaly/service/operations/land_resolve.goindex 9bc7c49..cc0b563 100644--- a/internal/gitaly/service/operations/land_resolve.go+++ b/internal/gitaly/service/operations/land_resolve.go@@ -247,7 +247,14 @@ func (s *Server) landResolveAt(break}var verifyErr *landVerifyError- switch err := s.landVerifyOutOfLane(ctx, in.GetRepository(), repo, tree); {+ current := func(ctx context.Context) (bool, error) {+ now, err := repo.ResolveRevision(ctx, git.NewReferenceNameFromBranchName(string(in.GetBranch())).Revision())+ if errors.Is(err, git.ErrReferenceNotFound) {+ return false, nil+ }+ return now == tip, err+ }+ switch err := s.landVerifyOutOfLane(ctx, in.GetRepository(), repo, tree, current); {case err == nil:verified = treepre.markVerified(tree)@@ -548,7 +555,7 @@ func (s *Server) landApplyVerdicts(if verified != "" {checkedBy = s.landVerifyTrailer()}- res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, checkedBy, tree, verified != "")+ res, err := s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, checkedBy, tree, verified)return res, err}@@ -743,8 +750,9 @@ func landUnresolvedCause(files []*landFileState) string {// tree and commits it with parents [tip, commitID]; checkedBy names what checked it, if anything// did. The commit message carries the trailers. built is the resolved tree when the caller already// built it (landResolvedTree). verified says that verify.command passed on that very tree, out of-// the lane (15.9). The lane runs no verification: when one is configured and the tree is not the-// one it passed on, the resolution is refused here, closed, whatever the caller did.+// the lane (15.9), and is the tree itself, so that a caller cannot vouch with a bool. The lane runs no+// verification: when one is configured and the tree is not the one it passed on, the resolution is+// refused here, closed, whatever the caller did.func (s *Server) landWriteResolution(ctx context.Context,in *gitalypb.UserLandRequest,@@ -755,7 +763,7 @@ func (s *Server) landWriteResolution(resp *resolver.Response,checkedBy string,built git.ObjectID,- verified bool,+ verified git.ObjectID,) (landResolution, error) {paths := make([]string, 0, len(resp.Files))for _, f := range resp.Files {@@ -776,7 +784,7 @@ func (s *Server) landWriteResolution(}// The verification ran out of the lane, on this very tree, before it got here (15.9): no// resolution lands on a tree it did not pass on, and no code path verifies in the lane.- if s.resolveCfg.Verify.Enabled && !verified {+ if s.resolveCfg.Verify.Enabled && (verified == "" || verified != treeID) {return landResolution{}, errors.New("a resolution reached the lane without the verification of its tree")}diff --git a/internal/gitaly/service/operations/land_resolve_helper_test.go b/internal/gitaly/service/operations/land_resolve_helper_test.goindex 5c71f01..30958b2 100644--- a/internal/gitaly/service/operations/land_resolve_helper_test.go+++ b/internal/gitaly/service/operations/land_resolve_helper_test.go@@ -26,5 +26,9 @@ func (s *Server) resolveLandConflicts(ctx context.Context, in *gitalypb.UserLandreturn landResolution{}, err}}- return s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "", tree, pre.isVerified(tree))+ var verified git.ObjectID+ if pre.isVerified(tree) {+ verified = tree+ }+ return s.landWriteResolution(ctx, in, repo, tip, commitID, l.mergedTree, committer, identity.Actor(ctx, ""), answer, "", tree, verified)}diff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.goindex 98e62b0..66e23d8 100644--- a/internal/gitaly/service/operations/land_verify.go+++ b/internal/gitaly/service/operations/land_verify.go@@ -157,6 +157,14 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.return nil}+// landTipMovedError says that the branch moved while a verification waited for its turn: the tree+// was not verified, because it is not the tree the land would write any more.+type landTipMovedError struct{}++func (*landTipMovedError) Error() string {+ return "the branch moved while the verification waited for its turn"+}+// landVerifyUnfinishedError is a verification that could not be completed, because its context ended// (the deadline of the land, or the caller) while it waited for a slot or ran: not a verdict.type landVerifyUnfinishedError struct {@@ -178,7 +186,7 @@ func landVerifyRepoKey(repository *gitalypb.Repository) string {// with a slot of the verify pool. It returns a *landVerifyError when the command fails or times// out, a *landVerifyUnfinishedError when ctx ended first, and another error when the verification// could not be run.-func (s *Server) landVerifyOutOfLane(ctx context.Context, repository *gitalypb.Repository, repo *localrepo.Repo, tree git.ObjectID) error {+func (s *Server) landVerifyOutOfLane(ctx context.Context, repository *gitalypb.Repository, repo *localrepo.Repo, tree git.ObjectID, current func(context.Context) (bool, error)) error {slot, err := s.verifyPool.Acquire(ctx, landVerifyRepoKey(repository))if err != nil {if ctx.Err() != nil {@@ -187,6 +195,19 @@ func (s *Server) landVerifyOutOfLane(ctx context.Context, repository *gitalypb.Rreturn err}defer slot.Release()+ // The wait for a slot can take as long as the verification before it: the tree was built from a+ // tip that the branch may have left meanwhile. Nothing is verified that is not current.+ if current != nil {+ ok, err := current(ctx)+ switch {+ case err != nil && ctx.Err() != nil:+ return &landVerifyUnfinishedError{err: err}+ case err != nil:+ return err+ case !ok:+ return &landTipMovedError{}+ }+ }err = s.landVerify(ctx, repo, tree)var verifyErr *landVerifyError@@ -546,7 +567,7 @@ func (m *landMaterializer) osError(err error) error {var errno syscall.Errnoif errors.As(err, &errno) {switch errno {- case syscall.EEXIST, syscall.ENAMETOOLONG, syscall.EILSEQ, syscall.ELOOP, syscall.ENOTDIR, syscall.EISDIR, syscall.ENOTEMPTY, syscall.EINVAL:+ case syscall.EEXIST, syscall.ENAMETOOLONG, syscall.EILSEQ, syscall.ELOOP, syscall.ENOTDIR, syscall.EISDIR, syscall.ENOTEMPTY, syscall.EINVAL, syscall.EMLINK:return landRefuse("the tree cannot be written on this file system: %s", m.describe(err, errno))}}@@ -615,6 +636,9 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {if e.sized && !e.isGitlink() && m.limits.bytes > 0 && m.bytes+e.listed > m.limits.bytes {return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)}+ if e.isSymlink() && e.sized && e.listed == 0 {+ return landRefuse("the symbolic link %q has an empty target, which no file system takes", e.path)+ }if e.isSymlink() && e.sized && e.listed > landMaxLinkTarget {return landRefuse("the symbolic link %q has a target of %d bytes, more than the %d that Linux takes", e.path, e.listed, landMaxLinkTarget)}@@ -706,6 +730,9 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {enter()target := filepath.Join(m.dir, filepath.FromSlash(e.path))if e.isSymlink() {+ if size == 0 {+ return landRefuse("the symbolic link %q has an empty target, which no file system takes", e.path)+ }if size > landMaxLinkTarget {return landRefuse("the symbolic link %q has a target of %d bytes, more than the %d that Linux takes", e.path, size, landMaxLinkTarget)}diff --git a/internal/gitaly/service/operations/land_verify_fixes_test.go b/internal/gitaly/service/operations/land_verify_fixes_test.gonew file mode 100644index 0000000..c416de9--- /dev/null+++ b/internal/gitaly/service/operations/land_verify_fixes_test.go@@ -0,0 +1,210 @@+package operations++import (+ "errors"+ "fmt"+ "io/fs"+ "sync"+ "syscall"+ "testing"+ "time"++ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/git/localrepo"+ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+)++// TestLandMaterialize_anEmptyLinkTargetIsRefused: git takes a symbolic link whose blob is empty, and+// no file system does (symlink(2) says ENOENT). It is the tree's doing, so it is a refusal with the+// reason, up front, and never a fault of the server: a pusher cannot make every verification of its+// change an internal error.+func TestLandMaterialize_anEmptyLinkTargetIsRefused(t *testing.T) {+ t.Parallel()++ var refusal *landCheckoutRefusal+ require.ErrorAs(t, materializeFake(t, t.TempDir(), fakeTreeEntry{0o120000, "l", ""}), &refusal)+ require.Contains(t, refusal.reason, "empty target")++ // From the listing, before the blob is read.+ m := newLandMaterializer(testhelper.Context(t), t.TempDir(), landCheckoutLimits{})+ e := landTreeEntry{mode: 0o120000, oid: git.ObjectID("0123456789012345678901234567890123456789"), path: "l", sized: true, listed: 0}+ require.ErrorAs(t, m.add(&e, nil), &refusal)+ require.Contains(t, refusal.reason, "empty target")++ // Through the verification of a real tree: the verdict says so and the command does not run.+ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ tree := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{+ {Mode: "100644", Path: "a", Content: "x\n"},+ {Mode: "120000", Path: "empty-link", Content: ""},+ })+ vdir, vargv := countingVerify(t, "exit 0")+ rcfg := withVerify(config.GraftResolver{}, vargv, "")+ s := newResolveFixture(t, rcfg).server(t, rcfg)+ err := s.landVerify(ctx, localrepo.NewTestRepo(t, cfg, repoProto), tree)+ var verifyErr *landVerifyError+ require.ErrorAs(t, err, &verifyErr)+ require.Contains(t, verifyErr.reason, "empty target")+ require.Equal(t, 0, verifyRuns(t, vdir))+}++// TestLandMaterializer_sortsTheErrnos: what the tree causes is a refusal, what the server's file system+// causes is a fault.+func TestLandMaterializer_sortsTheErrnos(t *testing.T) {+ t.Parallel()++ m := newLandMaterializer(testhelper.Context(t), "/srv/checkout", landCheckoutLimits{})+ for _, errno := range []syscall.Errno{syscall.EEXIST, syscall.ENAMETOOLONG, syscall.EILSEQ, syscall.ELOOP, syscall.ENOTDIR, syscall.EISDIR, syscall.ENOTEMPTY, syscall.EINVAL, syscall.EMLINK} {+ err := m.osError(&fs.PathError{Op: "mkdir", Path: "/srv/checkout/a/b", Err: errno})+ var refusal *landCheckoutRefusal+ require.ErrorAs(t, err, &refusal, errno.Error())+ require.NotContains(t, refusal.reason, "/srv/checkout", "relative paths only")+ require.Contains(t, refusal.reason, "mkdir a/b")+ }+ for _, errno := range []syscall.Errno{syscall.ENOSPC, syscall.EMFILE, syscall.ENFILE, syscall.EIO, syscall.EACCES, syscall.EPERM, syscall.EROFS, syscall.EDQUOT, syscall.ENOMEM, syscall.ENOENT} {+ var fault *landCheckoutFault+ require.ErrorAs(t, m.osError(&fs.PathError{Op: "mkdir", Path: "/srv/checkout/a", Err: errno}), &fault, errno.Error())+ }+}++// TestUserLand_aTreeIsNotVerifiedForATipTheBranchLeft: the wait for a verify slot can take as long as the+// verification before it. A tree built from a tip that the branch left meanwhile is not verified: the command+// runs once, on the tree of the tip the land lands on.+func TestUserLand_aTreeIsNotVerifiedForATipTheBranchLeft(t *testing.T) {+ t.Parallel()++ rdir, rargv := resolverScript(t, "")+ writeAnswer(t, rdir, validAnswer())+ vdir, vargv := countingVerify(t, `test -f e`)+ rcfg := withVerify(enabledResolver(rargv), vargv, "")+ fx, s := heldPool(t, rcfg)++ slot, err := s.verifyPool.Acquire(fx.ctx, landVerifyRepoKey(fx.repoProto))+ require.NoError(t, err)+ done := landAsync(fx, fx.commit, "Land")+ // The land has resolved and waits for the slot of its repository.+ require.Eventually(t, func() bool { return resolverCalls(t, rdir) == 1 }, 30*time.Second, 10*time.Millisecond)+ time.Sleep(300 * time.Millisecond)+ writer := gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.tip), gittest.WithBranch("main"),+ gittest.WithTreeEntries(entries("a", "ours\n", "b", "ours\n", "c", "c\n", "e", "e\n")...))+ slot.Release()++ resp := requireLanded(t, done)+ commit, landed := landedCommit(t, fx, resp)+ require.Equal(t, []string{writer.String(), fx.commit.String()}, commit.GetParentIds())+ require.Equal(t, "e\n", fx.blobAt(t, landed, "e"))+ require.Equal(t, 1, verifyRuns(t, vdir), "the tree of the tip the branch left was never verified: the command ran once, on the new tree")+ require.Equal(t, 1, resolverCalls(t, rdir), "the verdicts were kept")+}++// TestUserLand_contendedRulesLandsWithVerification: N lands of rules on one branch, with a verification+// configured and a deadline short enough to bite. Each ends landed or with the package and its reason; none+// hangs or fails otherwise; at least one lands.+func TestUserLand_contendedRulesLandsWithVerification(t *testing.T) {+ t.Parallel()++ const n = 5+ rdir, rargv := resolverScript(t, "")+ writeAnswer(t, rdir, answerOf("a"))+ vdir, vargv := countingVerify(t, "sleep 0.3")+ rcfg := withVerify(enabledResolver(rargv), vargv, "2s")+ rcfg.Timeout = "4s"+ fx := newResolveFixtureWith(t, rcfg,+ entries("a", "base\n", "f0", "base\n"),+ entries("a", "ours\n", "f0", "base\n"),+ entries("a", "theirs-0\n", "f0", "base\n"),+ )+ commits := []git.ObjectID{fx.commit}+ for i := 1; i < n; i++ {+ commits = append(commits, gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.base),+ gittest.WithTreeEntries(entries("a", fmt.Sprintf("theirs-%d\n", i), "f0", "base\n", fmt.Sprintf("n%d", i), "x\n")...)))+ }++ results := make([]landResult, n)+ var wg sync.WaitGroup+ start := time.Now()+ for i, c := range commits {+ wg.Add(1)+ go func() {+ defer wg.Done()+ req := fx.request(fmt.Sprintf("Land %d", i))+ req.CommitId = c.String()+ resp, err := fx.client.UserLand(fx.ctx, req)+ results[i] = landResult{resp: resp, err: err}+ }()+ }+ wg.Wait()+ require.Less(t, time.Since(start), 90*time.Second, "none hangs")++ landed := 0+ for i, r := range results {+ if r.err == nil {+ landed+++ continue+ }+ requireMergeConflict(t, r.err)+ pkg := landConflictDetail(r.err)+ require.NotNil(t, pkg, "land %d ends with the package", i)+ require.NotEmpty(t, packageFile(t, pkg, "a").GetReason(), "land %d says why", i)+ }+ require.GreaterOrEqual(t, landed, 1, "at least one lands")+ require.GreaterOrEqual(t, verifyRuns(t, vdir), landed)+ t.Logf("%d of %d landed, %d verifications", landed, n, verifyRuns(t, vdir))+}++// TestLandWriteResolution_wantsTheVerifiedTree: the lane's last guard compares the tree itself, not a bool.+func TestLandWriteResolution_wantsTheVerifiedTree(t *testing.T) {+ t.Parallel()++ rdir, rargv := resolverScript(t, "")+ writeAnswer(t, rdir, validAnswer())+ _, vargv := countingVerify(t, "exit 0")+ rcfg := withVerify(enabledResolver(rargv), vargv, "")+ fx := newResolveFixture(t, rcfg)+ s := fx.server(t, rcfg)+ s.locator = config.NewLocator(fx.cfg)+ in := fx.request("Land")++ pre := newLandPreResolution()+ l, answer, err := s.landResolveAt(fx.ctx, in, fx.repo, fx.tip, fx.commit, fx.committer(), pre)+ require.NoError(t, err)+ tree, err := s.landResolvedTree(fx.ctx, fx.repo, l.mergedTree, answer)+ require.NoError(t, err)+ require.True(t, pre.isVerified(tree))++ other := git.ObjectID("0000000000000000000000000000000000000000")+ for name, verified := range map[string]git.ObjectID{"nothing": "", "another tree": other} {+ _, err := s.landWriteResolution(fx.ctx, in, fx.repo, fx.tip, fx.commit, l.mergedTree, fx.committer(), "actor", answer, "", tree, verified)+ require.ErrorContains(t, err, "without the verification of its tree", name)+ }+ res, err := s.landWriteResolution(fx.ctx, in, fx.repo, fx.tip, fx.commit, l.mergedTree, fx.committer(), "actor", answer, "", tree, tree)+ require.NoError(t, err)+ require.NotEmpty(t, res.commit)+}++// TestLandPrecheck_noVerificationWhenTheResolverCannotBeBuilt: the lane refuses the resolution of a+// configured resolver that cannot be built, before anything else; the precheck does not spend a+// verification on it.+func TestLandPrecheck_noVerificationWhenTheResolverCannotBeBuilt(t *testing.T) {+ t.Parallel()++ vdir, vargv := countingVerify(t, "exit 0")+ rcfg := withVerify(config.GraftResolver{}, vargv, "")+ fx := newResolveFixture(t, rcfg)+ s := fx.server(t, rcfg)+ s.locator = config.NewLocator(fx.cfg)+ s.resolverErr = errors.New("the token file is gone")+ in := fx.request("Land")+ in.Resolution = resolutionOf(fx, "agent-a\n", "agent-b\n")++ pc, err := s.landPrecheckResolution(fx.ctx, in, fx.repo)+ require.NoError(t, err)+ require.Nil(t, pc)+ require.Equal(t, 0, verifyRuns(t, vdir))+}