Git designed for AI agents.
graft: review minors 2 (stale verdicts, final pass, checkout bounds and path safety, cancellation)
Federico Eliaa6f959c59f69
diff --git a/doc/graft/AGENT-SECURITY.md b/doc/graft/AGENT-SECURITY.mdindex d9760df..222fab5 100644--- a/doc/graft/AGENT-SECURITY.md+++ b/doc/graft/AGENT-SECURITY.md@@ -234,6 +234,19 @@ start from), and a separate unprivileged user. The post-merge test command of`graft-verify` ([VERIFY](VERIFY.md)) runs the same kind of code on the tip andhas the same advice. Without a `verify.command` nothing of a model's lands.+Three things that the container does not close:++- `--network=none` closes the network, not the output. Up to 4 KiB of what the command prints goes back+ to the agent in the reason of the conflict package, so a command that prints a secret hands it+ out. Print nothing that is not for the agent, and keep secrets out of the checkout's environment+ and file system.+- Symbolic links of the tree are written as links and can point anywhere on the host. Inside a+ container with the checkout as the only mount they point at nothing; outside one, a command that+ follows them reads what the gitaly user can read. The checkout never writes through a link.+- The tree may hold files that other tools read: `graft-resolve`'s own check commands (`[[check]]`)+ also run on a model's output, with the same advice. The checkout itself refuses a `.git`+ anywhere in the tree, and a tree over `verify.max_tree_entries` or `verify.max_tree_bytes`.+What counts as a model's answer is fail-safe on both sides. The server treats anyfile that is not marked `origin: "rule"` as a model's. `graft-resolve` marks as arule only the strategies that say so (trivial, union, rerere, and a `command`diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.mdindex 97272da..ed8cfc2 100644--- a/doc/graft/CONFIGURATION.md+++ b/doc/graft/CONFIGURATION.md@@ -212,7 +212,9 @@ cause.| `minified.longest_line_bytes` | int | `5000` | A side larger than 2 KiB with a longer line is minified. || `verify.enabled` | bool | `false` | Run `verify.command` on the merged tree of every resolution (a resolver's or an agent's) before it lands; it applies even when `enabled` is false. It is also what lets a model's resolution land: only a tree it passed on lands, and without it a model's resolution is only ever a proposal. || `verify.command` | array of strings | empty | Program and arguments, run in a temporary checkout of the merged tree with `PATH`, `HOME` and `TMPDIR` only. A non-zero exit rejects the resolution. |-| `verify.timeout` | string | `"5m"` | Bound on one run (Go duration); a run that does not finish rejects the resolution. When `verify.enabled`, it is added to `timeout` for the deadline of a resolved land. |+| `verify.timeout` | string | `"5m"` | Bound on one verification, the checkout of the tree and the command together (Go duration); one that does not finish rejects the resolution. When `verify.enabled`, it is added to `timeout` for the deadline of a resolved land. |+| `verify.max_tree_entries` | int | `200000` | Files, links and submodules in the tree that is checked out for the command; a bigger tree is not verified, and the package says why. |+| `verify.max_tree_bytes` | int | `2147483648` | Bytes of the files of that tree (checked from the listing, before any content is read). |A file that is never sent goes to the agent with its reason in the conflictpackage: large (a side over `max_file_bytes`), binary (a NUL byte), lockfile,diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.mdindex dccc8bc..5e60a58 100644--- a/doc/graft/PLAN10.md+++ b/doc/graft/PLAN10.md@@ -937,10 +937,10 @@ not exist yet; (not enabled) marks what is designed and measured but switched ofresolver is called only for files that changed), VERIFY it, enter the lane againThe rounds are bounded by the deadline of the land (the resolver's timeout, plus verify.timeout whena verify.command is configured) and by resolve_rounds (default 32, a safety cap, no contract). At- the deadline: files that kept changing -> merge_conflict, "files kept changing: a, b"; a model's- resolution that was not verified in time -> PACKAGE, the model's files PROPOSED, "could not verify- before the branch moved again" (a verified AI land is opportunistic: losing the race is harmless,- the agent gets the proposal).+ the deadline: the final pass (15.9): the merge conflict with its PACKAGE, the model's files PROPOSED+ ("could not verify before the branch moved again"), the rules' files RESOLVED, the files that changed+ under the land NOT_ATTEMPTED with the reason (a verified AI land is opportunistic: losing the race+ is harmless, the agent gets the proposal).The resolver is never called inside the lane, and neither is the verification of a model's resolution.PACKAGE to the agent (merge_conflict detail): sides by OID, hunks, per-file outcome + reason,@@ -1187,8 +1187,10 @@ The resolution is bound to the files instead (the PM's correction):32) is a safety cap well above realistic contention, documented as a cap and not a contract(decided after the 12-concurrent graft-bench test failed with a cap of 3). The limit, stated:under continuous new arrivals on the same file, one land can keep losing until its deadline,- and then fails with the merge conflict and the reason "files kept changing while the land was- resolved, until the deadline of <timeout>: a"; past the cap, "after <n> rounds".+ and then ends with the merge conflict and its package (15.9, the final pass): the files that+ changed under it are NOT_ATTEMPTED with the reason "the files changed again before they could be+ resolved" (the old "files kept changing" message is gone, as it carried no package); past the cap+ the same.* **Rules alike.** The operator rules' verdicts (trivial, union, rerere, command) are kept andreused by the same key, so the policy is the same for every strategy.* **Slots: two pools.** check_concurrency bounds the checks of agents' resolutions and@@ -1299,8 +1301,23 @@ and the tree it verifies depends on the tip.agent or by a model could therefore decide what the operator's tests see. The checkout is writtenfrom git-ls-tree(1) -r and git-cat-file(1) --batch, which apply no attribute (no export-*, eol,filter or smudge), with modes, symbolic links as links and gitlinks as empty directories, and is- read back and compared with the listing; a difference fails the checkout closed. The same function+ read back and compared with the listing, the content included (every file and link is hashed as a+ git blob and must be the OID of the tree); a difference fails the checkout closed. The same functionserves the verification of rules and of agents' resolutions.+* **What the checkout refuses, and what it costs.** The operator's command may run git in the checkout,+ so a path component that names .git on any file system is refused, by the one rule the resolver+ already uses for the special files (resolver.IsGitDirName: case-insensitive, the code points HFS++ ignores, NTFS "git~1" and trailing spaces and periods): a .git of the tree would be the repository+ of the command, and its configuration and hooks would run. Also refused: an entry under a symbolic+ link of the tree, also when a case-insensitive file system would see the same name (a link "A" and+ an entry "a/x"), a parent that is a link or a file, a path over 4096 bytes. Each directory is made+ one component at a time and looked at with Lstat (never MkdirAll, which follows links), a file is+ created with O_EXCL and O_NOFOLLOW. Resources: verify.max_tree_entries (default 200000) and+ verify.max_tree_bytes (default 2 GiB), checked from the listing (ls-tree -l) before the content is+ read; the listing and the blobs are streamed, entry by entry, through two commands, so the memory is+ that of the entries. The checkout runs inside verify.timeout, as the command does. A refusal is no+ error of the server: nothing is verified, the files are PROPOSED (a model's) or REJECTED (rules'),+ and the reason says which limit or path.* **Tests.** A model's resolution that passes lands, signed, with the command run once (so not inthe lane); one that fails gives a proposal and the output; without a verify.command it is aproposal; a tip that moved gets its new tree verified and lands, with no resolver call; a tip thatdiff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.mdindex ec8e781..4cdaddf 100644--- a/doc/graft/RESOLVE.md+++ b/doc/graft/RESOLVE.md@@ -411,8 +411,16 @@ contents of `git cat-file --batch`, with their modes, the symbolic links aslinks and the submodules as empty directories. No attribute is applied: not`export-ignore`, which would hide a file from the command (`git archive`honours it), nor `export-subst`, `eol`, a filter or a smudge. The directory is-read back and compared with the listing, and a difference fails the checkout-closed. It then runs `command` there, with a minimal environment+read back and compared with the listing, the content of every file included (it+is hashed as a git blob and must be the object ID of the tree), and a difference+fails the checkout closed. The listing and the blobs are streamed, not held in+memory, and the checkout refuses, with the reason in the package and without+running the command: a tree over `verify.max_tree_entries` entries or+`verify.max_tree_bytes` bytes, a path component that names `.git` on any file+system (the command may run git there, and a `.git` of the tree would be its+repository), and an entry under a symbolic link, also when a case-insensitive+file system would see the same name. The checkout and the command run inside+`verify.timeout` together. It then runs `command` there, with a minimal environment(`PATH`, and `HOME` and `TMPDIR` in a directory of their own). A non-zero exit or atimeout rejects the resolution: the land fails, every file of the package is`REJECTED`, and the reason is `verification failed:` with the exit status and@@ -518,8 +526,9 @@ that contend for a file all land, one per round. The rounds are bounded by theresolver's `timeout` for the whole land, and by `resolve_rounds`, a safety capwell above realistic contention. The limit, stated: under continuous newarrivals on the same file, one land can keep losing until its deadline; it then-fails with the merge conflict and the reason `files kept changing while the land-was resolved, until the deadline of <timeout>: <paths>`.+ends with the merge conflict and its package: the files that changed under it are+`NOT_ATTEMPTED` with the reason `the files changed again before they could be+resolved`, the others keep their verdicts (a model's `PROPOSED`).`verify.command` still runs in the lane, on the merged tree of the lane's tip.diff --git a/internal/git/stdin_writers_test.go b/internal/git/stdin_writers_test.goindex b74a2d0..5fccefb 100755--- a/internal/git/stdin_writers_test.go+++ b/internal/git/stdin_writers_test.go@@ -52,6 +52,7 @@ var stdinWriters = map[string]stdinWriter{"internal/graft/corestore/syncer/syncer.go": {safety: stdinNULDelimited},"internal/git/gitattributes/check_attr.go": {safety: stdinNULDelimited},"internal/gitaly/service/operations/land_package.go": {safety: stdinNULDelimited},+ "internal/gitaly/service/operations/land_verify.go": {safety: stdinServerInput},"internal/git/catfile/object_reader.go": {safety: stdinNULDelimited},"internal/gitaly/service/diff/find_changed_paths.go": {safety: stdinServerInput},"internal/git/localrepo/merge_tree_stream.go": {safety: stdinServerInput},diff --git a/internal/gitaly/config/config_graft.go b/internal/gitaly/config/config_graft.goindex 0932b0c..849b290 100644--- a/internal/gitaly/config/config_graft.go+++ b/internal/gitaly/config/config_graft.go@@ -380,11 +380,41 @@ type GraftResolverVerify struct {// Timeout bounds one run, as a Go duration. Empty means// DefaultGraftResolverVerifyTimeout.Timeout string `json:"timeout" toml:"timeout,omitempty"`+ // MaxTreeEntries bounds the files, links and submodules of the tree that is checked out for+ // the command; a bigger tree is not verified, and the reason says so. Zero means+ // DefaultGraftResolverVerifyMaxTreeEntries.+ MaxTreeEntries int `json:"max_tree_entries" toml:"max_tree_entries,omitempty"`+ // MaxTreeBytes bounds the bytes of the files of that tree. Zero means+ // DefaultGraftResolverVerifyMaxTreeBytes.+ MaxTreeBytes int64 `json:"max_tree_bytes" toml:"max_tree_bytes,omitempty"`}// DefaultGraftResolverVerifyTimeout is the default GraftResolverVerify.Timeout.const DefaultGraftResolverVerifyTimeout = "5m"+const (+ // DefaultGraftResolverVerifyMaxTreeEntries is the default GraftResolverVerify.MaxTreeEntries.+ DefaultGraftResolverVerifyMaxTreeEntries = 200000+ // DefaultGraftResolverVerifyMaxTreeBytes is the default GraftResolverVerify.MaxTreeBytes: 2 GiB.+ DefaultGraftResolverVerifyMaxTreeBytes int64 = 2 << 30+)++// MaxTreeEntriesOrDefault returns MaxTreeEntries or its default.+func (v GraftResolverVerify) MaxTreeEntriesOrDefault() int {+ if v.MaxTreeEntries > 0 {+ return v.MaxTreeEntries+ }+ return DefaultGraftResolverVerifyMaxTreeEntries+}++// MaxTreeBytesOrDefault returns MaxTreeBytes or its default.+func (v GraftResolverVerify) MaxTreeBytesOrDefault() int64 {+ if v.MaxTreeBytes > 0 {+ return v.MaxTreeBytes+ }+ return DefaultGraftResolverVerifyMaxTreeBytes+}+// TimeoutOrDefault parses Timeout, applying the default.func (v GraftResolverVerify) TimeoutOrDefault() (time.Duration, error) {s := v.Timeout@@ -399,6 +429,9 @@ func (v GraftResolverVerify) TimeoutOrDefault() (time.Duration, error) {}func (v GraftResolverVerify) validate() error {+ if v.MaxTreeEntries < 0 || v.MaxTreeBytes < 0 {+ return fmt.Errorf("graft land resolver verify: max_tree_entries and max_tree_bytes must not be negative (0 means the default)")+ }if !v.Enabled {return nil}diff --git a/internal/gitaly/config/config_graft_resolver_test.go b/internal/gitaly/config/config_graft_resolver_test.goindex 7d99d87..de94f38 100644--- a/internal/gitaly/config/config_graft_resolver_test.go+++ b/internal/gitaly/config/config_graft_resolver_test.go@@ -53,12 +53,18 @@ func TestGraftResolverValidateVerify(t *testing.T) {d, err := GraftResolverVerify{}.TimeoutOrDefault()require.NoError(t, err)require.Equal(t, "5m0s", d.String())+ require.Equal(t, 200000, GraftResolverVerify{}.MaxTreeEntriesOrDefault())+ require.Equal(t, int64(2<<30), GraftResolverVerify{}.MaxTreeBytesOrDefault())+ require.Equal(t, 7, GraftResolverVerify{MaxTreeEntries: 7}.MaxTreeEntriesOrDefault())+ require.Equal(t, int64(9), GraftResolverVerify{MaxTreeBytes: 9}.MaxTreeBytesOrDefault())for name, v := range map[string]GraftResolverVerify{- "no command": {Enabled: true},- "empty program": {Enabled: true, Command: []string{""}},- "bad timeout": {Enabled: true, Command: []string{"x"}, Timeout: "soon"},- "zero timeout": {Enabled: true, Command: []string{"x"}, Timeout: "0s"},+ "no command": {Enabled: true},+ "empty program": {Enabled: true, Command: []string{""}},+ "bad timeout": {Enabled: true, Command: []string{"x"}, Timeout: "soon"},+ "zero timeout": {Enabled: true, Command: []string{"x"}, Timeout: "0s"},+ "negative entries": {MaxTreeEntries: -1},+ "negative bytes": {MaxTreeBytes: -1},} {require.Error(t, GraftResolver{Verify: v}.validate(), name)}diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.goindex 0c93326..4603071 100644--- a/internal/gitaly/service/operations/land_preresolve.go+++ b/internal/gitaly/service/operations/land_preresolve.go@@ -63,20 +63,33 @@ type landPreResolution struct {// the pre-resolution then verifies nothing and calls no resolver, and the files of the model// become proposals with this reason.unverifiable string+ // current are the keys of the conflicting files at the latest tip the land looked at: of the+ // verdicts kept, only these are the land's now. A verdict of a file that has since changed (its+ // key is another) is stale, and decides nothing about the land.+ current map[landFileKey]bool}func newLandPreResolution() *landPreResolution {return &landPreResolution{verdicts: landVerdicts{}, verified: map[git.ObjectID]bool{}}}-// hasModel reports whether a verdict of the land is a model's: the land is a model's resolution-// that only a verified tree lands.+// setCurrent records which files the land has now: the verdicts of other files, or of the same files+// as they were, are stale.+func (p *landPreResolution) setCurrent(files []*landFileState) {+ p.current = make(map[landFileKey]bool, len(files))+ for _, f := range files {+ p.current[landKeyOf(f.conflict)] = true+ }+}++// hasModel reports whether a verdict for a current file of the land is a model's: the land is a+// model's resolution that only a verified tree lands. A stale verdict does not count.func (p *landPreResolution) hasModel() bool {if p == nil {return false}- for _, v := range p.verdicts {- if v.file.FromModel() {+ for k, v := range p.verdicts {+ if p.current[k] && v.file.FromModel() {return true}}@@ -120,17 +133,6 @@ func (e *landNeedsResolution) Error() string {return fmt.Sprintf("conflicting files to resolve at tip %s: %s", e.tip, strings.Join(e.paths, ", "))}-// keptChanging is the merge conflict of a land whose files kept changing until it gave up, when-// says when.-func (e *landNeedsResolution) keptChanging(when string) error {- paths := make([][]byte, 0, len(e.paths))- for _, p := range e.paths {- paths = append(paths, []byte(p))- }- return landMergeConflictError(e.tip, e.commit, paths,- fmt.Errorf("files kept changing while the land was resolved, %s: %s", when, strings.Join(e.paths, ", ")))-}-// landPreResolve resolves, out of the lane and in a quarantine it discards, the conflicting files// of the land at the branch's current tip that have no verdict in pre yet. It returns nil when the// lane can take the land: the verdicts are in pre, or there is nothing to resolve (no branch, the@@ -233,7 +235,6 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequestdeadline := func() bool {return ctx.Err() == nil && errors.Is(resolveCtx.Err(), context.DeadlineExceeded)}- until := fmt.Sprintf("until the deadline of %s", timeout)rounds := s.resolveCfg.ResolveRoundsOrDefault()for round := 1; round <= rounds; round++ {if err := s.landPreResolve(resolveCtx, in, committer, pre); err != nil {@@ -245,7 +246,12 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequestif errors.As(err, &unfinished) {reason = fmt.Sprintf("could not verify before the deadline of %s", timeout)}- return s.landFinalPass(ctx, in, committer, pre, reason, need, until)+ return s.landFinalPass(ctx, in, committer, pre, reason)+ }+ var unfinished *landVerifyUnfinishedError+ if errors.As(err, &unfinished) {+ // Not the deadline of the land: the caller gave up (or its own deadline passed).+ return nil, landCallerGaveUp(ctx, err)}return nil, err}@@ -254,10 +260,22 @@ func (s *Server) landResolving(ctx context.Context, in *gitalypb.UserLandRequestreturn resp, err}if deadline() {- return s.landFinalPass(ctx, in, committer, pre, landStoppedReason(need, timeout), need, until)+ return s.landFinalPass(ctx, in, committer, pre, landStoppedReason(need, timeout))}}- return s.landFinalPass(ctx, in, committer, pre, landStoppedReason(need, timeout), need, fmt.Sprintf("after %d rounds", rounds))+ return s.landFinalPass(ctx, in, committer, pre, landStoppedReason(need, timeout))+}++// landCallerGaveUp is the status of a land whose caller gave up, or whose own deadline passed, while+// it waited for a slot or verified: the caller's cancellation, not an error of the server.+func landCallerGaveUp(ctx context.Context, err error) error {+ switch {+ case errors.Is(ctx.Err(), context.DeadlineExceeded):+ return structerr.NewDeadlineExceeded("verifying the resolution: %w", err)+ case ctx.Err() != nil:+ return structerr.NewCanceled("verifying the resolution: %w", err)+ }+ return err}// landStoppedReason is why the land could not go on, as the files it leaves say: the tree it was@@ -273,38 +291,47 @@ func landStoppedReason(need *landNeedsResolution, timeout time.Duration) stringreturn fmt.Sprintf("the time ran out before the conflicts were resolved (deadline of %s)", timeout)}+// landFinalPasses bounds the passes of landFinalPass: each is cheap, as it calls no resolver and runs+// no command, so that losing the branch to another land four times in a row is a branch that moves+// by the millisecond.+const landFinalPasses = 4+// landFinalPass ends a land that could not go on (landResolving): the agent gets the package, not// a deadline. A verified AI land is opportunistic, and losing the race is harmless. It pre-resolves// once more with the request's context, in a new quarantine, verifying nothing and calling no// resolver: the verdicts that are kept give the proposals (a model's files PROPOSED with the reason,// the rules' files RESOLVED), the files without a verdict are NOT_ATTEMPTED with the reason, and the-// land fails with the package. When there is nothing to report as a conflict after all (the change-// landed, the merge is clean, or every file has a verdict a rule made), the lane answers; need and-// when name the files for the merge conflict if even that hands the land back.+// land fails with the package. That is every ending of a land that had a conflict: the package is+// what landPreResolve returns, and nothing else comes out of it but the refusal of the policy or an+// error of the server. When there is nothing to report as a conflict after all (the change landed,+// the merge is clean, or every file has a verdict a rule made), the lane answers, and lands.+//+// Between the pre-resolution and the lane the branch can move to a tip with other conflicts, and+// the lane hands the land back: the pass is made again, at most landFinalPasses times. A land that+// loses all of them ends as any land that lost the race, with the retryable "kept moving" of+// landKeptMovingError; the package of a conflict is not what it is about.func (s *Server) landFinalPass(ctx context.Context,in *gitalypb.UserLandRequest,committer git.Signature,pre *landPreResolution,reason string,- need *landNeedsResolution,- when string,) (*gitalypb.UserLandResponse, error) {pre.unverifiable = reason- err := s.landPreResolve(ctx, in, committer, pre)- var needs *landNeedsResolution- switch {- case errors.As(err, &needs):- if need == nil {- need = needs+ for pass := 1; ; pass++ {+ if err := s.landPreResolve(ctx, in, committer, pre); err != nil {+ return nil, err+ }+ if s.landFinalPassHook != nil {+ s.landFinalPassHook(pass)+ }+ resp, err := s.landOnce(ctx, in, committer, nil, pre)+ var needs *landNeedsResolution+ if !errors.As(err, &needs) {+ return resp, err+ }+ if pass >= landFinalPasses {+ return nil, structerr.NewFailedPrecondition("branch %q kept moving while the land was being ended, giving up after %d passes", string(in.GetBranch()), pass)}- return nil, need.keptChanging(when)- case err != nil:- return nil, err- }- resp, err := s.landOnce(ctx, in, committer, nil, pre)- if errors.As(err, &needs) {- return nil, needs.keptChanging(when)}- return resp, err}diff --git a/internal/gitaly/service/operations/land_resolve.go b/internal/gitaly/service/operations/land_resolve.goindex 6b4c229..383f445 100644--- a/internal/gitaly/service/operations/land_resolve.go+++ b/internal/gitaly/service/operations/land_resolve.go@@ -137,6 +137,7 @@ func (s *Server) landResolveAt(if err := s.landClassify(ctx, repo, files, base, tip, commitID, newLandClassifyCache(files, tip)); err != nil {return l, nil, err}+ pre.setCurrent(files)// A file the classification sends and that has a verdict with its key is not resolved again.// mine are the files with an answer, reused or fresh, whose outcome the answer decides.@@ -422,6 +423,7 @@ func (s *Server) landApplyVerdicts(if err := s.landClassify(ctx, repo, l.files, l.base, tip, commitID, newLandClassifyCache(l.files, tip)); err != nil {return landResolution{}, err}+ pre.setCurrent(l.files)for _, f := range l.files {if _, ok := pre.verdicts[landKeyOf(f.conflict)]; !ok || !f.sent() {missing = append(missing, f.conflict.path)diff --git a/internal/gitaly/service/operations/land_review_test.go b/internal/gitaly/service/operations/land_review_test.goindex 1c9d798..b1fe906 100644--- a/internal/gitaly/service/operations/land_review_test.go+++ b/internal/gitaly/service/operations/land_review_test.go@@ -155,9 +155,16 @@ func TestLandKeptMoving(t *testing.T) {in := &gitalypb.UserLandRequest{Branch: []byte("main"), CommitId: strings.Repeat("c", 40)}conflict := landConflict{reference: "refs/heads/main", oldOID: git.ObjectID(strings.Repeat("1", 40)), newOID: git.ObjectID(strings.Repeat("2", 40))}- model := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginModel}}}}- rules := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginRule}}}}- unmarked := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a"}}}}+ current := map[landFileKey]bool{{path: "a"}: true}+ model := &landPreResolution{current: current, verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginModel}}}}+ rules := &landPreResolution{current: current, verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginRule}}}}+ unmarked := &landPreResolution{current: current, verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a"}}}}+ // A model's verdict for a file as it was (another key), while the land's file is a rule's: stale.+ stale := &landPreResolution{current: current, verdicts: landVerdicts{+ {path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginRule}},+ {path: "a", oursOID: "old"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginModel}},+ {path: "gone", theirsOID: "old"}: {file: resolver.ResponseFile{Path: "gone", Origin: resolver.OriginModel}},+ }}for name, pre := range map[string]*landPreResolution{"a model's verdict": model, "a verdict that says nothing": unmarked} {var need *landNeedsResolution@@ -166,7 +173,7 @@ func TestLandKeptMoving(t *testing.T) {require.Equal(t, strings.Repeat("c", 40), need.commit.String(), name)require.Equal(t, "could not verify before the branch moved again", landStoppedReason(need, time.Minute), name)}- for name, pre := range map[string]*landPreResolution{"rules only": rules, "no pre-resolution": nil, "no verdict": {}} {+ for name, pre := range map[string]*landPreResolution{"rules only": rules, "stale model verdicts": stale, "no pre-resolution": nil, "no verdict": {}, "no current files": {verdicts: model.verdicts}} {err := landKeptMoving(in, pre, conflict, 20)var need *landNeedsResolutionrequire.False(t, errors.As(err, &need), name)@@ -204,6 +211,7 @@ func TestLandLostRace(t *testing.T) {conflict := landConflict{reference: "refs/heads/main", oldOID: git.ObjectID(strings.Repeat("1", 40)), newOID: git.ObjectID(strings.Repeat("2", 40))}model := &landPreResolution{verdicts: landVerdicts{{path: "a"}: {file: resolver.ResponseFile{Path: "a", Origin: resolver.OriginModel}}},+ current: map[landFileKey]bool{{path: "a"}: true},}ctx := context.Background()diff --git a/internal/gitaly/service/operations/land_verified_test.go b/internal/gitaly/service/operations/land_verified_test.goindex e0b3786..e8a7c7c 100644--- a/internal/gitaly/service/operations/land_verified_test.go+++ b/internal/gitaly/service/operations/land_verified_test.go@@ -664,3 +664,102 @@ func TestUserLand_verifiedModelResolution_aTreeThatPassedIsNotVerifiedAgain(t *trequire.Equal(t, 1, verifyRuns(t, vdir), "the command ran once for the tree, not once per round")require.Equal(t, 1, resolverCalls(t, rdir))}++// TestUserLand_verifiedModelResolution_aCallerThatGivesUpGetsItsCancellation: a caller that cancels while+// the land verifies, or whose own deadline passes, gets Canceled or DeadlineExceeded from the server, and+// not the bare "verification not finished" of the internal error, nor a package: the land is not the+// deadline of the land (landFinalPass), which belongs to the land and not to the caller.+func TestUserLand_verifiedModelResolution_aCallerThatGivesUpGetsItsCancellation(t *testing.T) {+ t.Parallel()++ for name, tc := range map[string]struct {+ code codes.Code+ ctx func(parent context.Context) (context.Context, context.CancelFunc, func())+ }{+ "cancelled": {codes.Canceled, func(parent context.Context) (context.Context, context.CancelFunc, func()) {+ ctx, cancel := context.WithCancel(parent)+ return ctx, cancel, cancel+ }},+ "its own deadline": {codes.DeadlineExceeded, func(parent context.Context) (context.Context, context.CancelFunc, func()) {+ ctx, cancel := context.WithTimeout(parent, 2*time.Second)+ return ctx, cancel, func() {}+ }},+ } {+ t.Run(name, func(t *testing.T) {+ t.Parallel()++ rdir, rargv := resolverScript(t, "")+ writeAnswer(t, rdir, modelAnswer())+ vdir, vargv := countingVerify(t, `echo $$ > "$DIR/pid"+sleep 60`)+ rcfg := withVerify(enabledResolver(rargv), vargv, "1m")+ rcfg.Timeout = "1m"+ fx := newResolveFixture(t, rcfg)+ s := fx.server(t, rcfg)+ s.locator = config.NewLocator(fx.cfg)++ ctx, cancelAll, giveUp := tc.ctx(fx.ctx)+ defer cancelAll()+ done := make(chan error, 1)+ go func() {+ _, err := s.landResolving(ctx, fx.request("Land"), fx.committer())+ done <- err+ }()+ require.Eventually(t, func() bool {+ _, err := os.Stat(filepath.Join(vdir, "pid"))+ return err == nil+ }, 30*time.Second, 10*time.Millisecond, "the verification never started")+ giveUp()++ select {+ case err := <-done:+ require.Error(t, err)+ require.Equal(t, tc.code, status.Code(err), "%v", err)+ require.Nil(t, landConflictDetail(err), "no package for a caller that is gone")+ case <-time.After(60 * time.Second):+ require.FailNow(t, "the land did not end")+ }+ })+ }+}++// TestUserLand_finalPass_aLostRaceStillEndsWithThePackage: the lane hands the land back after the pass+// pre-resolved it (the branch moved to a tip where a file has no verdict): the next pass ends with the+// package, the file that changed NOT_ATTEMPTED with the reason and the other one with its verdict. It+// is not a merge conflict without a package.+func TestUserLand_finalPass_aLostRaceStillEndsWithThePackage(t *testing.T) {+ t.Parallel()++ rdir, rargv := resolverScript(t, "")+ writeAnswer(t, rdir, validAnswer())+ rcfg := enabledResolver(rargv)+ var s *Server+ operationServerHooks.Store(t.Name(), func(served *Server) { s = served })+ fx := newResolveFixture(t, rcfg)+ require.NotNil(t, s, "the service's own Server")++ // The rules' verdicts of a and b, at the tip of the fixture.+ pre := newLandPreResolution()+ require.NoError(t, s.landPreResolve(fx.ctx, fx.request("Land"), fx.committer(), pre))+ require.Len(t, pre.verdicts, 2)++ var passes []int+ s.landFinalPassHook = func(pass int) {+ passes = append(passes, pass)+ if pass != 1 {+ return+ }+ // Another land changes a on the branch while this one ends.+ gittest.WriteCommit(t, fx.cfg, fx.repoPath, gittest.WithParents(fx.tip), gittest.WithBranch("main"),+ gittest.WithTreeEntries(entries("a", "ours changed\n", "b", "ours\n", "c", "c\n")...))+ }+ _, err := s.landFinalPass(fx.ctx, fx.request("Land"), fx.committer(), pre, "the time ran out")+ requireMergeConflict(t, err)+ pkg := landConflictDetail(err)+ require.NotNil(t, pkg, "the package, not a bare merge conflict")+ require.Equal(t, []int{1}, passes, "the second pass pre-resolves and ends with the package")+ a := packageFile(t, pkg, "a")+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_NOT_ATTEMPTED, a.GetOutcome())+ require.Equal(t, "the time ran out", a.GetReason())+ require.Equal(t, gitalypb.LandConflictFile_OUTCOME_RESOLVED, packageFile(t, pkg, "b").GetOutcome())+}diff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.goindex a239602..c0df4de 100644--- a/internal/gitaly/service/operations/land_verify.go+++ b/internal/gitaly/service/operations/land_verify.go@@ -2,8 +2,10 @@ package operationsimport ("bufio"- "bytes""context"+ "crypto/sha1" //nolint:gosec // the hash git names its objects with+ "crypto/sha256"+ "encoding/hex""errors""fmt""io"@@ -13,11 +15,13 @@ import ("path/filepath""strconv""strings"+ "syscall""github.com/feder-cr/git-ai/internal/command""github.com/feder-cr/git-ai/internal/git""github.com/feder-cr/git-ai/internal/git/gitcmd""github.com/feder-cr/git-ai/internal/git/localrepo"+ "github.com/feder-cr/git-ai/internal/graft/resolver""github.com/feder-cr/git-ai/proto/go/gitalypb")@@ -64,12 +68,23 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.}}- if err := landCheckoutTree(ctx, repo, tree, checkout); err != nil {- return fmt.Errorf("checking out the merged tree: %w", err)- }-+ // The timeout bounds the checkout and the command together: a tree that takes the whole of it to+ // write is a verification that did not finish.runCtx, cancel := context.WithTimeout(ctx, timeout)defer cancel()+ limits := landCheckoutLimits{entries: v.MaxTreeEntriesOrDefault(), bytes: v.MaxTreeBytesOrDefault()}+ if err := landCheckoutTree(runCtx, repo, tree, checkout, limits); err != nil {+ var refusal *landCheckoutRefusal+ switch {+ case ctx.Err() != nil:+ return ctx.Err()+ case errors.Is(runCtx.Err(), context.DeadlineExceeded):+ return &landVerifyError{reason: fmt.Sprintf("the checkout of the tree and the command did not finish within %s", timeout)}+ case errors.As(err, &refusal):+ return &landVerifyError{reason: "nothing was verified, the merged tree is not checked out: " + refusal.reason}+ }+ return fmt.Errorf("checking out the merged tree: %w", err)+ }var output landBoundedBufferoutput.limit = landVerifyOutputBytescmd, err := command.New(runCtx, s.logger, v.Command,@@ -154,13 +169,16 @@ func (s *Server) landVerifyTrailer() string {// landTreeEntry is one entry of a tree as git-ls-tree(1) -r lists it: a file, a symbolic link or a// gitlink (a submodule), by path. size and target are filled when it is written: the length of the-// file, the target of the link.+// file, the target of the link. sized and listed are the size that the listing gave (ls-tree -l), when+// it did.type landTreeEntry struct {mode uint32oid git.ObjectIDpath stringsize int64target string+ sized bool+ listed int64}const (@@ -170,70 +188,121 @@ const (landTreeModeGitlink = 0o160000// landMaxLinkTarget bounds the target of a symbolic link, which is read into memory.landMaxLinkTarget = 64 << 10+ // landMaxTreePath bounds a path of the tree, and with it the record that is read of the listing.+ landMaxTreePath = 4096)func (e landTreeEntry) isGitlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeGitlink }func (e landTreeEntry) isSymlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeSymlink }func (e landTreeEntry) isRegular() bool { return e.mode&landTreeModeTypeMask == landTreeModeRegular }+// landCheckoutLimits bound a checkout; a zero field is no limit (the server always sets them, from+// verify.max_tree_entries and verify.max_tree_bytes).+type landCheckoutLimits struct {+ entries int+ bytes int64+}++// landCheckoutRefusal is a tree that the checkout refuses to write, closed: it holds more than the+// limits allow, or a path that is not safe to write under a directory. It is no failure of the+// server: the command does not run, and the reason goes to the agent in the conflict package.+type landCheckoutRefusal struct {+ reason string+}++func (e *landCheckoutRefusal) Error() string { return e.reason }++func landRefuse(format string, args ...any) error {+ return &landCheckoutRefusal{reason: fmt.Sprintf(format, args...)}+}+// landCheckoutTree writes the tree under dir byte for byte: the entries of git-ls-tree(1) -r and the// contents of git-cat-file(1) --batch, which apply no attribute (no export-ignore, export-subst,// eol, filter or smudge), with the modes (a file is executable or not), the symbolic links as links,// and the gitlinks as empty directories. It then reads the directory back and compares it with the-// listing: a difference fails the checkout, closed, since the operator's command would otherwise+// listing, the content of every file included (the blob hash of what was written must be the OID of+// the tree): a difference fails the checkout, closed, since the operator's command would otherwise// verify a tree that is not the one that lands.-func landCheckoutTree(ctx context.Context, repo *localrepo.Repo, tree git.ObjectID, dir string) error {- return landCheckoutTreeThen(ctx, repo, tree, dir, nil)+//+// The listing and the blobs are streamed, one entry at a time, so the memory it takes is that of the+// entries (which limits bounds) and not of the tree. A tree beyond the limits, or with a path that is+// not safe to write, is a *landCheckoutRefusal.+func landCheckoutTree(ctx context.Context, repo *localrepo.Repo, tree git.ObjectID, dir string, limits landCheckoutLimits) error {+ return landCheckoutTreeThen(ctx, repo, tree, dir, limits, nil)}// landCheckoutTreeThen is landCheckoutTree with a function that runs between the writing and the// check; only tests give one, to damage the checkout and see that the check notices.-func landCheckoutTreeThen(ctx context.Context, repo *localrepo.Repo, tree git.ObjectID, dir string, afterWrite func()) error {- var listing bytes.Buffer- var stderr landBoundedBuffer- stderr.limit = landVerifyOutputBytes- if err := repo.ExecAndWait(ctx,- gitcmd.Command{Name: "ls-tree", Flags: []gitcmd.Option{gitcmd.Flag{Name: "-r"}, gitcmd.Flag{Name: "-z"}, gitcmd.Flag{Name: "--full-tree"}}, Args: []string{tree.String()}},- gitcmd.WithStdout(&listing),- gitcmd.WithStderr(&stderr),- ); err != nil {- return fmt.Errorf("listing the tree: %w, stderr: %q", err, stderr.String())- }- entries, err := landParseTreeListing(listing.Bytes())- if err != nil {- return err- }+func landCheckoutTreeThen(ctx context.Context, repo *localrepo.Repo, tree git.ObjectID, dir string, limits landCheckoutLimits, afterWrite func()) error {+ ctx, cancel := context.WithCancel(ctx)+ defer cancel()- // The contents come in the order of the listing, one blob after the other: the files and the links.- var input bytes.Buffer- for _, e := range entries {- if !e.isGitlink() {- input.WriteString(e.oid.String() + "\n")- }+ var lsStderr, catStderr landBoundedBuffer+ lsStderr.limit, catStderr.limit = landVerifyOutputBytes, landVerifyOutputBytes+ ls, err := repo.Exec(ctx,+ gitcmd.Command{Name: "ls-tree", Flags: []gitcmd.Option{gitcmd.Flag{Name: "-r"}, gitcmd.Flag{Name: "-z"}, gitcmd.Flag{Name: "-l"}, gitcmd.Flag{Name: "--full-tree"}}, Args: []string{tree.String()}},+ gitcmd.WithSetupStdout(),+ gitcmd.WithStderr(&lsStderr),+ )+ if err != nil {+ return fmt.Errorf("listing the tree: %w", err)}- stderr = landBoundedBuffer{limit: landVerifyOutputBytes}- cmd, err := repo.Exec(ctx,+ // The blobs are asked for through a pipe, as the listing is read: the contents come in the order+ // of the listing, one blob after the other, files and links, gitlinks left out.+ pipeR, pipeW := io.Pipe()+ cat, err := repo.Exec(ctx,gitcmd.Command{Name: "cat-file", Flags: []gitcmd.Option{gitcmd.Flag{Name: "--batch"}}},- gitcmd.WithStdin(&input),+ gitcmd.WithStdin(pipeR),gitcmd.WithSetupStdout(),- gitcmd.WithStderr(&stderr),+ gitcmd.WithStderr(&catStderr),)if err != nil {+ cancel()+ _ = ls.Wait()return fmt.Errorf("reading the blobs of the tree: %w", err)}- batch := bufio.NewReaderSize(cmd, 1<<20)- writeErr := landMaterialize(entries, batch, dir)- if writeErr == nil {++ listed := make(chan landTreeEntry, 256)+ fed := make(chan error, 1)+ go func() {+ err := landFeed(ctx, bufio.NewReaderSize(ls, 1<<20), pipeW, listed)+ close(listed)+ _ = pipeW.CloseWithError(err) // nil: the end of the input, and cat-file ends+ fed <- err+ }()++ var entries []landTreeEntry+ m := newLandMaterializer(dir, limits)+ batch := bufio.NewReaderSize(cat, 1<<20)+ var writeErr error+ for e := range listed {+ entries = append(entries, e)+ if writeErr = m.add(&entries[len(entries)-1], batch); writeErr != nil {+ break+ }+ }+ if writeErr != nil {+ // Let the feeder and the commands go: none of them is needed any more.+ cancel()+ _ = pipeR.Close()+ }+ feedErr := <-fed+ if writeErr == nil && feedErr == nil {// Nothing else may follow the last blob.if _, err := batch.ReadByte(); !errors.Is(err, io.EOF) {writeErr = fmt.Errorf("git-cat-file(1) answered more than the blobs asked for: %v", err)}}- if waitErr := cmd.Wait(); waitErr != nil && writeErr == nil {- return fmt.Errorf("reading the blobs of the tree: %w, stderr: %q", waitErr, stderr.String())- }- if writeErr != nil {+ lsWait, catWait := ls.Wait(), cat.Wait()+ switch {+ case writeErr != nil:return writeErr+ case feedErr != nil:+ return feedErr+ case lsWait != nil:+ return fmt.Errorf("listing the tree: %w, stderr: %q", lsWait, lsStderr.String())+ case catWait != nil:+ return fmt.Errorf("reading the blobs of the tree: %w, stderr: %q", catWait, catStderr.String())}if afterWrite != nil {afterWrite()@@ -241,34 +310,119 @@ func landCheckoutTreeThen(ctx context.Context, repo *localrepo.Repo, tree git.Obreturn landCheckMaterialized(entries, dir)}+// landFeed reads the entries of the listing (ls-tree -r -z -l), asks cat-file for the blob of each+// that is not a gitlink, one line per object ID, and hands the entries on, in order.+func landFeed(ctx context.Context, listing *bufio.Reader, oids io.Writer, out chan<- landTreeEntry) error {+ for {+ record, err := landReadRecord(listing)+ if errors.Is(err, io.EOF) {+ return nil+ }+ if err != nil {+ return err+ }+ e, err := landParseTreeRecord(record)+ if err != nil {+ return err+ }+ if !e.isGitlink() {+ if _, err := io.WriteString(oids, e.oid.String()+"\n"); err != nil {+ return fmt.Errorf("asking for the blobs of the tree: %w", err)+ }+ }+ select {+ case out <- e:+ case <-ctx.Done():+ return ctx.Err()+ }+ }+}++// landReadRecord reads one NUL-ended record, of at most landMaxTreePath plus the bytes of its+// mode, type, object ID and size. io.EOF means the listing ended between two records.+func landReadRecord(r *bufio.Reader) (string, error) {+ var record []byte+ for {+ chunk, err := r.ReadSlice(0)+ record = append(record, chunk...)+ if len(record) > landMaxTreePath+256 {+ return "", landRefuse("the tree has a path longer than %d bytes", landMaxTreePath)+ }+ switch {+ case err == nil:+ return string(record[:len(record)-1]), nil+ case errors.Is(err, bufio.ErrBufferFull):+ continue+ case errors.Is(err, io.EOF) && len(record) == 0:+ return "", io.EOF+ case errors.Is(err, io.EOF):+ return "", fmt.Errorf("the listing of the tree ends in the middle of an entry: %w", io.ErrUnexpectedEOF)+ }+ return "", fmt.Errorf("reading the listing of the tree: %w", err)+ }+}+// landParseTreeListing parses the output of git-ls-tree(1) -r -z: "<mode> <type> <object>\t<path>"-// entries ended by NUL. A tree holds files, links and gitlinks once listed recursively.+// entries ended by NUL (with -l, "<mode> <type> <object> <size>\t<path>"). A tree holds files, links+// and gitlinks once listed recursively.func landParseTreeListing(raw []byte) ([]landTreeEntry, error) {var entries []landTreeEntryfor _, record := range strings.Split(string(raw), "\x00") {if record == "" {continue}- meta, name, ok := strings.Cut(record, "\t")- fields := strings.Fields(meta)- if !ok || len(fields) != 3 {- return nil, fmt.Errorf("unexpected entry in the listing of the tree: %q", record)- }- mode, err := strconv.ParseUint(fields[0], 8, 32)+ e, err := landParseTreeRecord(record)if err != nil {- return nil, fmt.Errorf("unexpected mode in the listing of the tree: %q", record)- }- e := landTreeEntry{mode: uint32(mode), oid: git.ObjectID(fields[2]), path: name}- switch {- case e.isRegular() && fields[1] == "blob", e.isSymlink() && fields[1] == "blob", e.isGitlink() && fields[1] == "commit":- default:- return nil, fmt.Errorf("entry %q has the mode %s and the type %s, which a checkout does not write", name, fields[0], fields[1])+ return nil, err}entries = append(entries, e)}return entries, nil}+// landParseTreeRecord parses one entry of the listing.+func landParseTreeRecord(record string) (landTreeEntry, error) {+ meta, name, ok := strings.Cut(record, "\t")+ fields := strings.Fields(meta)+ if !ok || (len(fields) != 3 && len(fields) != 4) {+ return landTreeEntry{}, fmt.Errorf("unexpected entry in the listing of the tree: %q", record)+ }+ mode, err := strconv.ParseUint(fields[0], 8, 32)+ if err != nil {+ return landTreeEntry{}, fmt.Errorf("unexpected mode in the listing of the tree: %q", record)+ }+ if !landObjectIDValid(fields[2]) {+ return landTreeEntry{}, fmt.Errorf("unexpected object ID in the listing of the tree: %q", record)+ }+ e := landTreeEntry{mode: uint32(mode), oid: git.ObjectID(fields[2]), path: name}+ if len(fields) == 4 && fields[3] != "-" {+ size, err := strconv.ParseInt(fields[3], 10, 64)+ if err != nil || size < 0 {+ return landTreeEntry{}, fmt.Errorf("unexpected size in the listing of the tree: %q", record)+ }+ e.sized, e.listed = true, size+ }+ switch {+ case e.isRegular() && fields[1] == "blob", e.isSymlink() && fields[1] == "blob", e.isGitlink() && fields[1] == "commit":+ default:+ return landTreeEntry{}, fmt.Errorf("entry %q has the mode %s and the type %s, which a checkout does not write", name, fields[0], fields[1])+ }+ return e, nil+}++// landObjectIDValid reports whether s is an object ID: 40 (SHA-1) or 64 (SHA-256) lower-case hex digits.+func landObjectIDValid(s string) bool {+ if len(s) != 40 && len(s) != 64 {+ return false+ }+ for i := 0; i < len(s); i++ {+ if c := s[i]; (c < '0' || c > '9') && (c < 'a' || c > 'f') {+ return false+ }+ }+ return true+}+// landTreePathValid reports whether the path of a tree entry may be written under a directory: a// clean relative path that does not leave it.func landTreePathValid(name string) bool {@@ -276,75 +430,176 @@ func landTreePathValid(name string) bool {!strings.ContainsRune(name, 0)}+// landFolded is what a case-insensitive file system sees at a path: its first spelling, and whether+// that is a symbolic link.+type landFolded struct {+ spelling string+ link bool+}++// landMaterializer writes the entries of a tree under dir, one at a time.+type landMaterializer struct {+ dir string+ limits landCheckoutLimits+ links map[string]bool // the paths of the symbolic links written+ dirs map[string]bool // the directories made+ folded map[string]landFolded // the paths as a case-insensitive file system sees them+ count int // entries written+ bytes int64 // bytes of files and links written+}++func newLandMaterializer(dir string, limits landCheckoutLimits) *landMaterializer {+ return &landMaterializer{dir: dir, limits: limits, links: map[string]bool{}, dirs: map[string]bool{}, folded: map[string]landFolded{}}+}+// landMaterialize writes the entries under dir, reading the contents of the files and links from batch// in their order (the answer of git-cat-file(1) --batch to the blobs of the listing, gitlinks left-// out). It refuses a path that would leave dir or lie under a symbolic link it wrote: a link is-// written as a link and never followed. It sets size and target of the entries.+// out). See landMaterializer.add for what it refuses. It sets size and target of the entries.func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) error {- links := map[string]bool{}+ m := newLandMaterializer(dir, landCheckoutLimits{})for i := range entries {- e := &entries[i]- if !landTreePathValid(e.path) {- return fmt.Errorf("unsafe path %q in the tree", e.path)+ if err := m.add(&entries[i], batch); err != nil {+ return err}- for p := path.Dir(e.path); p != "."; p = path.Dir(p) {- if links[p] {- return fmt.Errorf("entry %q lies under the symbolic link %q", e.path, p)- }+ }+ return nil+}++// add writes one entry. It refuses, with a *landCheckoutRefusal:+// - a path that is not clean, or leaves dir;+// - a path longer than landMaxTreePath, more entries or more bytes than the limits allow;+// - a component that names .git on some file system (resolver.IsGitDirName): the operator's+// command may run git in the checkout, and a .git of the tree would be its repository, whose+// configuration and hooks run;+// - an entry under a symbolic link that was written, also when a case-insensitive file system+// would see the same name (a link "A" and an entry "a/x");+// - a parent that is not a directory.+//+// Each directory is made one component at a time and looked at with Lstat, never with MkdirAll,+// which would follow a link; a file is created with O_EXCL and O_NOFOLLOW; a link is written as a+// link and never followed.+func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {+ if !landTreePathValid(e.path) {+ return landRefuse("unsafe path %q in the tree", e.path)+ }+ if len(e.path) > landMaxTreePath {+ return landRefuse("the tree has a path longer than %d bytes", landMaxTreePath)+ }+ if m.count++; m.limits.entries > 0 && m.count > m.limits.entries {+ return landRefuse("the tree has more than %d entries (verify.max_tree_entries)", m.limits.entries)+ }+ if e.sized && !e.isGitlink() && m.limits.bytes > 0 && m.bytes+e.listed > m.limits.bytes {+ return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)+ }++ parts := strings.Split(e.path, "/")+ for _, c := range parts {+ if resolver.IsGitDirName(c) {+ return landRefuse("the path %q has a component that names .git, which a checkout does not write", e.path)}- target := filepath.Join(dir, filepath.FromSlash(e.path))- if e.isGitlink() {- if err := os.MkdirAll(target, 0o755); err != nil {- return err- }- continue+ }+ for p := path.Dir(e.path); p != "."; p = path.Dir(p) {+ if m.links[p] {+ return landRefuse("entry %q lies under the symbolic link %q", e.path, p)}+ }+ for i := range parts {+ prefix := strings.Join(parts[:i+1], "/")+ link := i == len(parts)-1 && e.isSymlink()+ f := resolver.FoldGitName(prefix)+ prev, seen := m.folded[f]+ switch {+ case !seen:+ m.folded[f] = landFolded{spelling: prefix, link: link}+ case prev.spelling != prefix && (prev.link || link):+ return landRefuse("%q and %q are one name on a case-insensitive file system, and one is a symbolic link", e.path, prev.spelling)+ }+ }- size, err := landReadBatchHeader(batch, e.oid)+ if e.isGitlink() {+ return m.mkdirAll(parts)+ }+ if m.dirs[e.path] {+ return landRefuse("the entry %q is also a directory of the tree", e.path)+ }+ size, err := landReadBatchHeader(batch, e.oid)+ if err != nil {+ return fmt.Errorf("%q: %w", e.path, err)+ }+ if e.sized && e.listed != size {+ return fmt.Errorf("%q: the listing says %d bytes and git-cat-file(1) %d", e.path, e.listed, size)+ }+ if m.bytes += size; m.limits.bytes > 0 && m.bytes > m.limits.bytes {+ return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)+ }+ if err := m.mkdirAll(parts[:len(parts)-1]); err != nil {+ return err+ }+ target := filepath.Join(m.dir, filepath.FromSlash(e.path))+ if e.isSymlink() {+ if size > landMaxLinkTarget {+ return landRefuse("the symbolic link %q has a target of %d bytes", e.path, size)+ }+ content := make([]byte, size)+ if _, err := io.ReadFull(batch, content); err != nil {+ return fmt.Errorf("%q: %w", e.path, err)+ }+ if err := os.Symlink(string(content), target); err != nil {+ return err+ }+ e.target, e.size = string(content), size+ m.links[e.path] = true+ } else {+ perm := os.FileMode(0o644)+ if e.mode&0o111 != 0 {+ perm = 0o755+ }+ f, err := os.OpenFile(target, os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW, perm)if err != nil {+ return err+ }+ if _, err := io.CopyN(f, batch, size); err != nil {+ _ = f.Close()return fmt.Errorf("%q: %w", e.path, err)}- if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {+ if err := f.Close(); err != nil {return err}- if e.isSymlink() {- if size > landMaxLinkTarget {- return fmt.Errorf("the symbolic link %q has a target of %d bytes", e.path, size)- }- content := make([]byte, size)- if _, err := io.ReadFull(batch, content); err != nil {- return fmt.Errorf("%q: %w", e.path, err)- }- if err := os.Symlink(string(content), target); err != nil {- return err- }- e.target, e.size = string(content), size- links[e.path] = true- } else {- perm := os.FileMode(0o644)- if e.mode&0o111 != 0 {- perm = 0o755- }- f, err := os.OpenFile(target, os.O_WRONLY|os.O_CREATE|os.O_EXCL, perm)- if err != nil {- return err- }- if _, err := io.CopyN(f, batch, size); err != nil {- _ = f.Close()- return fmt.Errorf("%q: %w", e.path, err)- }- if err := f.Close(); err != nil {- return err- }- // The mode is the tree's, whatever the umask of the process.- if err := os.Chmod(target, perm); err != nil {+ // The mode is the tree's, whatever the umask of the process.+ if err := os.Chmod(target, perm); err != nil {+ return err+ }+ e.size = size+ }+ if b, err := batch.ReadByte(); err != nil || b != '\n' {+ return fmt.Errorf("%q: the blob is not followed by a line feed", e.path)+ }+ return nil+}++// mkdirAll makes the directory of the given components under dir, one at a time. A component that+// exists must be a directory by Lstat, which does not follow a link: a link, or a file, in the way+// is a refusal.+func (m *landMaterializer) mkdirAll(parts []string) error {+ current := m.dir+ for i, c := range parts {+ current = filepath.Join(current, c)+ key := strings.Join(parts[:i+1], "/")+ if m.dirs[key] {+ continue+ }+ info, err := os.Lstat(current)+ switch {+ case errors.Is(err, fs.ErrNotExist):+ if err := os.Mkdir(current, 0o755); err != nil {return err}- e.size = size- }- if b, err := batch.ReadByte(); err != nil || b != '\n' {- return fmt.Errorf("%q: the blob is not followed by a line feed", e.path)+ case err != nil:+ return err+ case !info.IsDir():+ return landRefuse("%q is in the way of a directory of the tree, and is not a directory", key)}+ m.dirs[key] = true}return nil}@@ -367,8 +622,23 @@ func landReadBatchHeader(batch *bufio.Reader, want git.ObjectID) (int64, error)return size, nil}+// landBlobOID is the object ID that git gives the blob that r holds, size bytes long: the hash of+// "blob <size>\0" and the content, with the hash of the repository (SHA-256 for an ID of 64 digits).+func landBlobOID(r io.Reader, size int64, like git.ObjectID) (git.ObjectID, error) {+ h := sha1.New() //nolint:gosec // the hash git names its objects with+ if len(like) == 64 {+ h = sha256.New()+ }+ fmt.Fprintf(h, "blob %d\x00", size)+ if _, err := io.Copy(h, r); err != nil {+ return "", err+ }+ return git.ObjectID(hex.EncodeToString(h.Sum(nil))), nil+}+// landCheckMaterialized reads dir back and compares it with the entries that were written: every entry-// is there with its type, its executable bit, the length of its file or the target of its link; a+// is there with its type, its executable bit, the length of its file or the target of its link; the+// content of every file and every link hashes, as a git blob, to the object ID of the tree; a// gitlink is an empty directory; and nothing else is there but the directories that hold the// entries. A difference is an error.func landCheckMaterialized(entries []landTreeEntry, dir string) error {@@ -409,10 +679,22 @@ func landCheckMaterialized(entries []landTreeEntry, dir string) error {if err != nil || info.Mode()&os.ModeSymlink == 0 || got != e.target {return fmt.Errorf("the symbolic link %q does not point to %q", name, e.target)}+ if oid, err := landBlobOID(strings.NewReader(got), int64(len(got)), e.oid); err != nil || oid != e.oid {+ return fmt.Errorf("the symbolic link %q is not the blob %s of the tree", name, e.oid)+ }case isEntry && e.isRegular():if !info.Mode().IsRegular() || info.Size() != e.size || (info.Mode().Perm()&0o111 != 0) != (e.mode&0o111 != 0) {return fmt.Errorf("the file %q is not the file of the tree (size %d, executable %v)", name, e.size, e.mode&0o111 != 0)}+ f, err := os.Open(full)+ if err != nil {+ return err+ }+ oid, err := landBlobOID(f, e.size, e.oid)+ _ = f.Close()+ if err != nil || oid != e.oid {+ return fmt.Errorf("the content of the file %q is not the blob %s of the tree", name, e.oid)+ }case isEntry:return fmt.Errorf("unexpected entry %q", name)case d.IsDir() && parents[name]:diff --git a/internal/gitaly/service/operations/land_verify_test.go b/internal/gitaly/service/operations/land_verify_test.goindex 9fb7d09..769899d 100644--- a/internal/gitaly/service/operations/land_verify_test.go+++ b/internal/gitaly/service/operations/land_verify_test.go@@ -167,7 +167,7 @@ func TestLandCheckoutTree(t *testing.T) {require.Equal(t, "$Format:%H$\n", archived["subst.txt"])dir := t.TempDir()- require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir))+ require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{}))for path, want := range map[string]string{".gitattributes": "export-ignore.log export-ignore\nsubst.txt export-subst\ncrlf.txt text eol=crlf\n*.bin -text\n",@@ -213,7 +213,7 @@ func TestLandCheckoutTree_theCheckIsPartOfIt(t *testing.T) {fx := newTreeFixture(t)ctx := testhelper.Context(t)- require.NoError(t, landCheckoutTreeThen(ctx, fx.repo, fx.tree, t.TempDir(), func() {}), "without damage it passes")+ require.NoError(t, landCheckoutTreeThen(ctx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{}, func() {}), "without damage it passes")for name, damage := range map[string]func(dir string){"a file written in": func(dir string) { _ = os.WriteFile(filepath.Join(dir, "planted"), []byte("x"), 0o644) },"a file of the tree gone": func(dir string) {@@ -224,7 +224,7 @@ func TestLandCheckoutTree_theCheckIsPartOfIt(t *testing.T) {},} {dir := t.TempDir()- err := landCheckoutTreeThen(ctx, fx.repo, fx.tree, dir, func() { damage(dir) })+ err := landCheckoutTreeThen(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{}, func() { damage(dir) })require.ErrorContains(t, err, "the checkout differs from the tree", name)}}@@ -242,18 +242,27 @@ func batchOf(contents map[string]string, entries []landTreeEntry) *bufio.Readerreturn bufio.NewReader(&b)}-func fakeEntries() ([]landTreeEntry, map[string]string) {- oid := func(c string) git.ObjectID { return git.ObjectID(strings.Repeat(c, 40)) }+// blobOID is the object ID git gives a blob with this content.+func blobOID(t testing.TB, content string) git.ObjectID {+ t.Helper()+ oid, err := landBlobOID(strings.NewReader(content), int64(len(content)), git.ObjectID(strings.Repeat("0", 40)))+ require.NoError(t, err)+ return oid+}++func fakeEntries(t testing.TB) ([]landTreeEntry, map[string]string) {+ t.Helper()+ contents := map[string]string{+ "a.txt": "content\n",+ "bin/run": "#!/bin/sh\n",+ "link": "a.txt",+ }return []landTreeEntry{- {mode: 0o100644, oid: oid("1"), path: "a.txt"},- {mode: 0o100755, oid: oid("2"), path: "bin/run"},- {mode: 0o120000, oid: oid("3"), path: "link"},- {mode: 0o160000, oid: oid("4"), path: "mod"},- }, map[string]string{- "a.txt": "content\n",- "bin/run": "#!/bin/sh\n",- "link": "a.txt",- }+ {mode: 0o100644, oid: blobOID(t, contents["a.txt"]), path: "a.txt"},+ {mode: 0o100755, oid: blobOID(t, contents["bin/run"]), path: "bin/run"},+ {mode: 0o120000, oid: blobOID(t, contents["link"]), path: "link"},+ {mode: 0o160000, oid: git.ObjectID(strings.Repeat("4", 40)), path: "mod"},+ }, contents}// TestLandMaterialize_refusesWhatLeavesTheDirectory: a path that is not clean, that is absolute or that@@ -279,7 +288,7 @@ func TestLandMaterialize_refusesWhatLeavesTheDirectory(t *testing.T) {func TestLandMaterialize_refusesAnswersItDidNotAskFor(t *testing.T) {t.Parallel()- entries, contents := fakeEntries()+ entries, contents := fakeEntries(t)good := batchOf(contents, entries)require.NoError(t, landMaterialize(entries, good, t.TempDir()))@@ -291,7 +300,7 @@ func TestLandMaterialize_refusesAnswersItDidNotAskFor(t *testing.T) {"a short blob": entries[0].oid.String() + " blob 30\nabc\n","no line feed behind": entries[0].oid.String() + " blob 3\nabcX",} {- fresh, _ := fakeEntries()+ fresh, _ := fakeEntries(t)require.Error(t, landMaterialize(fresh, bufio.NewReader(strings.NewReader(answer)), t.TempDir()), name)}}@@ -303,7 +312,7 @@ func TestLandCheckMaterialized_failsClosed(t *testing.T) {materialized := func(t *testing.T) ([]landTreeEntry, string) {t.Helper()- entries, contents := fakeEntries()+ entries, contents := fakeEntries(t)dir := t.TempDir()require.NoError(t, landMaterialize(entries, batchOf(contents, entries), dir))require.NoError(t, landCheckMaterialized(entries, dir), "an exact checkout passes")@@ -323,6 +332,9 @@ func TestLandCheckMaterialized_failsClosed(t *testing.T) {"a missing file": func(t *testing.T, dir string) {require.NoError(t, os.Remove(filepath.Join(dir, "a.txt")))},+ "a changed content of the same length": func(t *testing.T, dir string) {+ require.NoError(t, os.WriteFile(filepath.Join(dir, "a.txt"), []byte("CONTENT\n"), 0o644))+ },"a changed length": func(t *testing.T, dir string) {require.NoError(t, os.WriteFile(filepath.Join(dir, "a.txt"), []byte("longer than before\n"), 0o644))},@@ -413,3 +425,242 @@ func TestUserLand_verifySeesTheWholeTree(t *testing.T) {})}}++// fakeTreeEntry is an entry of a tree that landMaterialize writes, with the content that git-cat-file(1)+// would answer for it.+type fakeTreeEntry struct {+ mode uint32+ path string+ content string+}++func materializeFake(t *testing.T, dir string, in ...fakeTreeEntry) error {+ t.Helper()++ entries := make([]landTreeEntry, len(in))+ contents := map[string]string{}+ for i, f := range in {+ entries[i] = landTreeEntry{mode: f.mode, oid: blobOID(t, f.content), path: f.path}+ contents[f.path] = f.content+ }+ return landMaterialize(entries, batchOf(contents, entries), dir)+}++// TestLandMaterialize_pathSafety: what the checkout refuses to write, with the reason, and that nothing+// is written outside the directory whatever the order of the entries (PLAN10 15.9): the paths that name+// .git on some file system (the operator's command may run git in the checkout, and a .git of the tree is+// its repository), an entry under a symbolic link, also when a case-insensitive file system sees the same+// name, a link or a file where a directory is needed, and a link that is already there.+func TestLandMaterialize_pathSafety(t *testing.T) {+ t.Parallel()++ reg, link, gitlink := uint32(0o100644), uint32(0o120000), uint32(0o160000)+ // The links point out of the checkout, at the directory that holds it.+ for name, tc := range map[string]struct {+ entries func(outer string) []fakeTreeEntry+ planted bool // a symbolic link "p" to outer is in the checkout before the first entry+ }{+ "a .git directory": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, ".git/config", "[core]\n"}} }},+ "a .GIT directory": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, ".GIT/hooks/pre-commit", "x"}} }},+ "a nested .Git": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, "a/b/.Git/config", "x"}} }},+ "a .git file": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, ".git", "gitdir: /elsewhere\n"}} }},+ "a .git gitlink": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{gitlink, ".git", ""}} }},+ "an NTFS .git with a space": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, ".git /config", "x"}} }},+ "an NTFS .git with a period": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, ".git./config", "x"}} }},+ "the NTFS short name": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, "GIT~1/config", "x"}} }},+ "an HFS .git": {entries: func(string) []fakeTreeEntry { return []fakeTreeEntry{{reg, ".g\u200cit/config", "x"}} }},+ "an entry under a link": {entries: func(outer string) []fakeTreeEntry {+ return []fakeTreeEntry{{link, "l", outer}, {reg, "l/x", "planted"}}+ }},+ "an entry under a link, case folded": {entries: func(outer string) []fakeTreeEntry {+ return []fakeTreeEntry{{link, "L", outer}, {reg, "l/x", "planted"}}+ }},+ "a link over an entry's directory, case folded, reversed": {entries: func(outer string) []fakeTreeEntry {+ return []fakeTreeEntry{{reg, "l/x", "planted"}, {link, "L", outer}}+ }},+ "a link over an entry's directory, reversed": {entries: func(outer string) []fakeTreeEntry {+ return []fakeTreeEntry{{reg, "l/x", "planted"}, {link, "l", outer}}+ }},+ "a file where a directory is needed": {entries: func(string) []fakeTreeEntry {+ return []fakeTreeEntry{{reg, "a", "file"}, {reg, "a/x", "planted"}}+ }},+ "a link that is already there": {planted: true, entries: func(string) []fakeTreeEntry {+ return []fakeTreeEntry{{reg, "p/x", "planted"}}+ }},+ "a link that is already there, as a parent of a gitlink": {planted: true, entries: func(string) []fakeTreeEntry {+ return []fakeTreeEntry{{gitlink, "p/mod", ""}}+ }},+ } {+ t.Run(name, func(t *testing.T) {+ t.Parallel()++ outer := t.TempDir()+ require.NoError(t, os.WriteFile(filepath.Join(outer, "sentinel"), []byte("keep"), 0o644))+ dir := filepath.Join(outer, "checkout")+ require.NoError(t, os.Mkdir(dir, 0o755))+ if tc.planted {+ require.NoError(t, os.Symlink(outer, filepath.Join(dir, "p")))+ }++ err := materializeFake(t, dir, tc.entries(outer)...)+ var refusal *landCheckoutRefusal+ require.ErrorAs(t, err, &refusal, "a refusal, with its reason")+ require.NotEmpty(t, refusal.reason)++ // Nothing is written outside the checkout: only the checkout and the sentinel are in outer.+ names := []string{}+ children, err := os.ReadDir(outer)+ require.NoError(t, err)+ for _, c := range children {+ names = append(names, c.Name())+ }+ require.ElementsMatch(t, []string{"checkout", "sentinel"}, names)+ got, err := os.ReadFile(filepath.Join(outer, "sentinel"))+ require.NoError(t, err)+ require.Equal(t, "keep", string(got))+ })+ }++ t.Run("what looks like these and is not is written", func(t *testing.T) {+ t.Parallel()++ dir := t.TempDir()+ require.NoError(t, materializeFake(t, dir,+ fakeTreeEntry{reg, ".gitignore", "x\n"},+ fakeTreeEntry{reg, ".github/workflows/ci.yml", "x\n"},+ fakeTreeEntry{reg, "gitx/a", "x\n"},+ fakeTreeEntry{reg, "git~2/a", "x\n"},+ fakeTreeEntry{reg, "README", "upper\n"},+ fakeTreeEntry{reg, "readme", "lower\n"}, // two files that differ in case only: no link in it+ fakeTreeEntry{link, "ok", "README"},+ ))+ })+}++// TestIsGitDirName: the one rule of the resolver package, used by the checkout and by nobody else's copy.+func TestIsGitDirName(t *testing.T) {+ t.Parallel()++ for _, n := range []string{".git", ".GIT", ".Git", ".git ", ".git.", ".git . ", "git~1", "GIT~1", "git~1 ", ".g\u200cit", ".gi\ufeffT", ".git:stream"} {+ require.True(t, resolver.IsGitDirName(n), "%q", n)+ }+ for _, n := range []string{"", "git", ".gitignore", ".github", "gitx", "git~2", ".git~1", "x.git", "a.git ", ".gi t", "git~10"} {+ require.False(t, resolver.IsGitDirName(n), "%q", n)+ }+}++// TestLandCheckoutTree_limits: a tree with more entries or more bytes than the limits is refused, closed,+// with the reason, and exactly at the limit it is written; the limits are checked before the content is+// read, from the listing.+func TestLandCheckoutTree_limits(t *testing.T) {+ t.Parallel()++ fx := newTreeFixture(t)+ ctx := testhelper.Context(t)++ listing := gittest.Exec(t, fx.cfg, "-C", fx.repoPath, "ls-tree", "-r", "-z", "-l", "--full-tree", fx.tree.String())+ entries, err := landParseTreeListing(listing)+ require.NoError(t, err)+ var total int64+ for _, e := range entries {+ require.True(t, e.isGitlink() || e.sized, "the listing gives the size of %q", e.path)+ if !e.isGitlink() {+ total += e.listed+ }+ }++ require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{entries: len(entries), bytes: total}), "at the limits")++ var refusal *landCheckoutRefusal+ dir := t.TempDir()+ require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{entries: len(entries) - 1}), &refusal)+ require.Contains(t, refusal.reason, "more than")+ require.Contains(t, refusal.reason, "verify.max_tree_entries")++ dir = t.TempDir()+ require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{bytes: total - 1}), &refusal)+ require.Contains(t, refusal.reason, "verify.max_tree_bytes")+ // The bytes were refused from the listing, before the file that goes over was read: less than the tree is there.+ var written int64+ require.NoError(t, filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {+ if err == nil && d.Type().IsRegular() {+ info, _ := d.Info()+ written += info.Size()+ }+ return nil+ }))+ require.Less(t, written, total)+}++// TestLandCheckoutTree_streamsALargeTree: a tree whose listing and blobs are far over the buffers between+// the commands (the pipe of the object IDs, the batch reader, the entries in flight) is written exactly:+// nothing waits for the whole listing, and nothing is lost or reordered.+func TestLandCheckoutTree_streamsALargeTree(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ var in []gittest.TreeEntry+ want := map[string]string{}+ for i := 0; i < 1700; i++ {+ name := fmt.Sprintf("d%02d/f%04d.txt", i%50, i)+ content := strings.Repeat(fmt.Sprintf("%d\n", i), 400)+ in = append(in, gittest.TreeEntry{Mode: "100644", Path: name, Content: content})+ want[name] = content+ }+ tree := gittest.WriteTree(t, cfg, repoPath, nestTreeEntries(t, cfg, repoPath, in))++ dir := t.TempDir()+ require.NoError(t, landCheckoutTree(ctx, localrepo.NewTestRepo(t, cfg, repoProto), tree, dir, landCheckoutLimits{entries: 5000, bytes: 1 << 30}))+ for name, content := range want {+ got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(name)))+ require.NoError(t, err, name)+ require.Equal(t, content, string(got), name)+ }+}++// TestLandVerify_aTreeThatCannotBeCheckedOutIsNotVerified: a tree that holds a .git, or more than the+// limits, makes the verification fail with the reason, and the operator's command does not run.+func TestLandVerify_aTreeThatCannotBeCheckedOutIsNotVerified(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ repo := localrepo.NewTestRepo(t, cfg, repoProto)+ hostile := gittest.WriteTree(t, cfg, repoPath, nestTreeEntries(t, cfg, repoPath, []gittest.TreeEntry{+ {Mode: "100644", Path: "a", Content: "x\n"},+ {Mode: "100644", Path: ".git/config", Content: "[core]\n\tfsmonitor = touch /tmp/pwned\n"},+ }))+ plain := gittest.WriteTree(t, cfg, repoPath, nestTreeEntries(t, cfg, repoPath, []gittest.TreeEntry{+ {Mode: "100644", Path: "a", Content: "x\n"},+ {Mode: "100644", Path: "b", Content: "y\n"},+ }))++ for name, tc := range map[string]struct {+ tree git.ObjectID+ tune func(*config.GraftResolverVerify)+ reason string+ }{+ "a .git": {hostile, func(*config.GraftResolverVerify) {}, "names .git"},+ "too many entries": {plain, func(v *config.GraftResolverVerify) { v.MaxTreeEntries = 1 }, "verify.max_tree_entries"},+ "too many bytes": {plain, func(v *config.GraftResolverVerify) { v.MaxTreeBytes = 3 }, "verify.max_tree_bytes"},+ "no time to check it": {plain, func(v *config.GraftResolverVerify) { v.Timeout = "1ns" }, "did not finish within 1ns"},+ } {+ t.Run(name, func(t *testing.T) {+ t.Parallel()++ vdir, vargv := countingVerify(t, "exit 0")+ rcfg := withVerify(config.GraftResolver{}, vargv, "")+ tc.tune(&rcfg.Verify)+ s := newResolveFixture(t, rcfg).server(t, rcfg)++ err := s.landVerify(ctx, repo, tc.tree)+ var verifyErr *landVerifyError+ require.ErrorAs(t, err, &verifyErr, "a rejection with its reason, not an error of the server")+ require.Contains(t, verifyErr.reason, tc.reason)+ require.Equal(t, 0, verifyRuns(t, vdir), "the command did not run")+ })+ }+}diff --git a/internal/gitaly/service/operations/server.go b/internal/gitaly/service/operations/server.goindex 502fbcc..0b39c71 100644--- a/internal/gitaly/service/operations/server.go+++ b/internal/gitaly/service/operations/server.go@@ -44,6 +44,9 @@ type Server struct {// landResolveHook, when set, runs in every resolver call while it holds its resolve slot; only// tests set it.landResolveHook func()+ // landFinalPassHook, when set, runs in every pass of the final pass of a land between its+ // pre-resolution and its lane, with the number of the pass; only tests set it.+ landFinalPassHook func(pass int)// conflictResolver answers the resolve requests of UserLand. Nil when resolution is disabled,// or when the configured resolver cannot be built: then resolverErr says why, and every land// that needs the resolver fails closed with it (resolveLandConflicts, landWithResolution).diff --git a/internal/gitaly/service/operations/testhelper_test.go b/internal/gitaly/service/operations/testhelper_test.goindex 9902239..deac259 100644--- a/internal/gitaly/service/operations/testhelper_test.go+++ b/internal/gitaly/service/operations/testhelper_test.go@@ -2,6 +2,7 @@ package operationsimport ("context"+ "sync""testing"gitalyauth "github.com/feder-cr/git-ai/auth"@@ -92,7 +93,11 @@ func runOperationServiceServerWithLanes(tb testing.TB, cfg config.Cfg, lanes *laif lanes != nil {deps.GraftLandLanes = lanes}- gitalypb.RegisterOperationServiceServer(srv, NewServer(deps))+ operationServer := NewServer(deps)+ if hook, ok := operationServerHooks.Load(tb.Name()); ok {+ hook.(func(*Server))(operationServer)+ }+ gitalypb.RegisterOperationServiceServer(srv, operationServer)gitalypb.RegisterHookServiceServer(srv, hook.NewServer(deps))gitalypb.RegisterRepositoryServiceServer(srv, repository.NewServer(deps))gitalypb.RegisterRefServiceServer(srv, ref.NewServer(deps))@@ -102,6 +107,12 @@ func runOperationServiceServerWithLanes(tb testing.TB, cfg config.Cfg, lanes *la}, options...)}+// operationServerHooks hands the operation Server that a test's service runs to the test, by the+// name of the test: a func(*Server) stored under t.Name() before the service starts is called with the+// Server before it serves. The Server is the real one, with every dependency, which a Server built by+// hand in a test is not.+var operationServerHooks sync.Map+func newOperationClient(tb testing.TB, serverSocketPath string) (gitalypb.OperationServiceClient, *grpc.ClientConn) {conn, err := client.New(testhelper.Context(tb), serverSocketPath)if err != nil {diff --git a/internal/graft/resolver/validate.go b/internal/graft/resolver/validate.goindex eeef5e8..4e56f4e 100644--- a/internal/graft/resolver/validate.go+++ b/internal/graft/resolver/validate.go@@ -175,22 +175,54 @@ var gitSpecialFiles = []struct {// each followed by spaces and periods only. An alternate data stream (":stream") is cut// first, which refuses more than git does.func gitSpecialFile(name string) string {- folded := strings.ToLower(strings.Map(func(r rune) rune {+ folded := FoldGitName(name)+ ntfs := ntfsName(name)+ for _, s := range gitSpecialFiles {+ if folded == s.name || ntfsSpecialName(ntfs, s.name[1:], s.shortHash) {+ return s.name+ }+ }+ return ""+}++// FoldGitName is the name as a case-insensitive file system that ignores the HFS+ ignorable code+// points sees it: lower case, without those code points. Two names that fold to the same string+// may be one name on such a file system.+func FoldGitName(name string) string {+ return strings.ToLower(strings.Map(func(r rune) rune {if hfsIgnorable(r) {return -1}return r}, name))+}++// ntfsName is the lower-case name without an alternate data stream (":stream"), as the NTFS+// rules read it.+func ntfsName(name string) string {ntfs := strings.ToLower(name)if i := strings.IndexByte(ntfs, ':'); i >= 0 {ntfs = ntfs[:i]}- for _, s := range gitSpecialFiles {- if folded == s.name || ntfsSpecialName(ntfs, s.name[1:], s.shortHash) {- return s.name+ return ntfs+}++// IsGitDirName reports whether the path component name stands for ".git" on some file system, by+// the rules of git's verify_path: ".git" case-insensitively, ignoring the code points HFS++// ignores, and on NTFS ".git" or its 8.3 short name "git~1" followed by spaces and periods only.+// A tree is never to hold such a component, as git would treat the directory as the repository's+// own: its config and hooks would run.+func IsGitDirName(name string) bool {+ if FoldGitName(name) == ".git" {+ return true+ }+ ntfs := ntfsName(name)+ for _, prefix := range []string{".git", "git~1"} {+ if strings.HasPrefix(ntfs, prefix) && onlySpacesAndPeriods(ntfs[len(prefix):]) {+ return true}}- return ""+ return false}// ntfsSpecialName reports whether the lower-case name stands on NTFS for "." + dotName, whose