Git designed for AI agents.
graft: the checkout is bounded by what it makes, not by what the tree lists (review of a6f959c)
Federico Elia61dc0e3c8e44
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.mdindex ed8cfc2..cd89129 100644--- a/doc/graft/CONFIGURATION.md+++ b/doc/graft/CONFIGURATION.md@@ -213,8 +213,9 @@ cause.| `verify.enabled` | bool | `false` | Run `verify.command` on the merged tree of every resolution (a resolver's or an agent's) before it lands; it applies even when `enabled` is false. It is also what lets a model's resolution land: only a tree it passed on lands, and without it a model's resolution is only ever a proposal. || `verify.command` | array of strings | empty | Program and arguments, run in a temporary checkout of the merged tree with `PATH`, `HOME` and `TMPDIR` only. A non-zero exit rejects the resolution. || `verify.timeout` | string | `"5m"` | Bound on one verification, the checkout of the tree and the command together (Go duration); one that does not finish rejects the resolution. When `verify.enabled`, it is added to `timeout` for the deadline of a resolved land. |-| `verify.max_tree_entries` | int | `200000` | Files, links and submodules in the tree that is checked out for the command; a bigger tree is not verified, and the package says why. |-| `verify.max_tree_bytes` | int | `2147483648` | Bytes of the files of that tree (checked from the listing, before any content is read). |+| `verify.max_tree_entries` | int | `200000` | Files, links and submodules in the tree that is checked out for the command, plus the directories made for them; a bigger tree is not verified, and the package says why. |+| `verify.max_tree_bytes` | int | `2147483648` | Bytes of the files of that tree and of the targets of its links, checked per entry from the listing, before that entry's content is read. |+| `verify.max_tree_depth` | int | `64` | Components of a path of that tree. A tree can name one deep subtree many times; the expansion is what this and the entry count bound. |A file that is never sent goes to the agent with its reason in the conflictpackage: large (a side over `max_file_bytes`), binary (a NUL byte), lockfile,diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.mdindex 5e60a58..707244f 100644--- a/doc/graft/PLAN10.md+++ b/doc/graft/PLAN10.md@@ -1312,12 +1312,25 @@ and the tree it verifies depends on the tip.link of the tree, also when a case-insensitive file system would see the same name (a link "A" andan entry "a/x"), a parent that is a link or a file, a path over 4096 bytes. Each directory is madeone component at a time and looked at with Lstat (never MkdirAll, which follows links), a file is- created with O_EXCL and O_NOFOLLOW. Resources: verify.max_tree_entries (default 200000) and- verify.max_tree_bytes (default 2 GiB), checked from the listing (ls-tree -l) before the content is- read; the listing and the blobs are streamed, entry by entry, through two commands, so the memory is- that of the entries. The checkout runs inside verify.timeout, as the command does. A refusal is no- error of the server: nothing is verified, the files are PROPOSED (a model's) or REJECTED (rules'),- and the reason says which limit or path.+ created with O_EXCL and O_NOFOLLOW. Resources, all failing closed with the reason: a git tree can+ name one subtree many times, so a push of a few KB expands to as many paths as it has names, each as+ deep as the subtree. verify.max_tree_entries (default 200000) counts the entries AND the directories+ that the checkout makes (both are inodes); verify.max_tree_depth (default 64) the components of a+ path; verify.max_tree_bytes (default 2 GiB) the bytes of the files and of the targets of links; and+ a fixed 128 MiB the memory of the paths, each entry and each directory charged twice (the sets that+ look for links and for folded names). The limits are checked entry by entry, from the listing+ (ls-tree -l), BEFORE that entry's content is read, not from a first pass over the whole listing: the+ listing and the blobs are streamed through two commands, so the memory is that of the paths kept (the+ fixed budget), and an entry that would go over stops the checkout with the earlier ones already+ written. The context is checked at every entry. A name over 255 bytes, a path that with the+ directory reaches PATH_MAX, a NUL in the target of a link, and a system call that fails (a name+ that exists already, no room) are refused with the reason, not errors of the server. The checkout+ runs inside verify.timeout, as the command does. A refusal is no error of the server: nothing is+ verified, the files are PROPOSED (a model's) or REJECTED (rules'), and the reason says which limit+ or path. Not built: NTFS trailing-dot and space folding in the check of colliding names (the+ .git check has it); Gitaly does not run on Windows. Under MVCC (a snapshot pinned for the RPC) a+ lost race ends with the retryable FailedPrecondition "kept moving" even for a model's resolution:+ that call carries no package; the client's retry gets one.* **Tests.** A model's resolution that passes lands, signed, with the command run once (so not inthe lane); one that fails gives a proposal and the output; without a verify.command it is aproposal; a tip that moved gets its new tree verified and lands, with no resolver call; a tip thatdiff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.mdindex 4cdaddf..d7ed9cc 100644--- a/doc/graft/RESOLVE.md+++ b/doc/graft/RESOLVE.md@@ -415,8 +415,11 @@ read back and compared with the listing, the content of every file included (itis hashed as a git blob and must be the object ID of the tree), and a differencefails the checkout closed. The listing and the blobs are streamed, not held inmemory, and the checkout refuses, with the reason in the package and without-running the command: a tree over `verify.max_tree_entries` entries or-`verify.max_tree_bytes` bytes, a path component that names `.git` on any file+running the command: a tree over `verify.max_tree_entries` entries and+directories made, `verify.max_tree_depth` components in a path, or+`verify.max_tree_bytes` bytes of files and link targets (checked per entry,+before that entry's content is read, so the earlier entries are already+written), or paths that take more than 128 MiB of memory, a path component that names `.git` on any filesystem (the command may run git there, and a `.git` of the tree would be itsrepository), and an entry under a symbolic link, also when a case-insensitivefile system would see the same name. The checkout and the command run insidediff --git a/internal/gitaly/config/config_graft.go b/internal/gitaly/config/config_graft.goindex 849b290..046c9ca 100644--- a/internal/gitaly/config/config_graft.go+++ b/internal/gitaly/config/config_graft.go@@ -387,6 +387,9 @@ type GraftResolverVerify struct {// MaxTreeBytes bounds the bytes of the files of that tree. Zero means// DefaultGraftResolverVerifyMaxTreeBytes.MaxTreeBytes int64 `json:"max_tree_bytes" toml:"max_tree_bytes,omitempty"`+ // MaxTreeDepth bounds the components of a path of that tree. Zero means+ // DefaultGraftResolverVerifyMaxTreeDepth.+ MaxTreeDepth int `json:"max_tree_depth" toml:"max_tree_depth,omitempty"`}// DefaultGraftResolverVerifyTimeout is the default GraftResolverVerify.Timeout.@@ -397,8 +400,18 @@ const (DefaultGraftResolverVerifyMaxTreeEntries = 200000// DefaultGraftResolverVerifyMaxTreeBytes is the default GraftResolverVerify.MaxTreeBytes: 2 GiB.DefaultGraftResolverVerifyMaxTreeBytes int64 = 2 << 30+ // DefaultGraftResolverVerifyMaxTreeDepth is the default GraftResolverVerify.MaxTreeDepth.+ DefaultGraftResolverVerifyMaxTreeDepth = 64)+// MaxTreeDepthOrDefault returns MaxTreeDepth or its default.+func (v GraftResolverVerify) MaxTreeDepthOrDefault() int {+ if v.MaxTreeDepth > 0 {+ return v.MaxTreeDepth+ }+ return DefaultGraftResolverVerifyMaxTreeDepth+}+// MaxTreeEntriesOrDefault returns MaxTreeEntries or its default.func (v GraftResolverVerify) MaxTreeEntriesOrDefault() int {if v.MaxTreeEntries > 0 {@@ -429,8 +442,8 @@ func (v GraftResolverVerify) TimeoutOrDefault() (time.Duration, error) {}func (v GraftResolverVerify) validate() error {- if v.MaxTreeEntries < 0 || v.MaxTreeBytes < 0 {- return fmt.Errorf("graft land resolver verify: max_tree_entries and max_tree_bytes must not be negative (0 means the default)")+ if v.MaxTreeEntries < 0 || v.MaxTreeBytes < 0 || v.MaxTreeDepth < 0 {+ return fmt.Errorf("graft land resolver verify: max_tree_entries, max_tree_bytes and max_tree_depth must not be negative (0 means the default)")}if !v.Enabled {return nildiff --git a/internal/gitaly/config/config_graft_resolver_test.go b/internal/gitaly/config/config_graft_resolver_test.goindex de94f38..69a399f 100644--- a/internal/gitaly/config/config_graft_resolver_test.go+++ b/internal/gitaly/config/config_graft_resolver_test.go@@ -55,6 +55,8 @@ func TestGraftResolverValidateVerify(t *testing.T) {require.Equal(t, "5m0s", d.String())require.Equal(t, 200000, GraftResolverVerify{}.MaxTreeEntriesOrDefault())require.Equal(t, int64(2<<30), GraftResolverVerify{}.MaxTreeBytesOrDefault())+ require.Equal(t, 64, GraftResolverVerify{}.MaxTreeDepthOrDefault())+ require.Equal(t, 3, GraftResolverVerify{MaxTreeDepth: 3}.MaxTreeDepthOrDefault())require.Equal(t, 7, GraftResolverVerify{MaxTreeEntries: 7}.MaxTreeEntriesOrDefault())require.Equal(t, int64(9), GraftResolverVerify{MaxTreeBytes: 9}.MaxTreeBytesOrDefault())@@ -65,6 +67,7 @@ func TestGraftResolverValidateVerify(t *testing.T) {"zero timeout": {Enabled: true, Command: []string{"x"}, Timeout: "0s"},"negative entries": {MaxTreeEntries: -1},"negative bytes": {MaxTreeBytes: -1},+ "negative depth": {MaxTreeDepth: -1},} {require.Error(t, GraftResolver{Verify: v}.validate(), name)}diff --git a/internal/gitaly/service/operations/land.go b/internal/gitaly/service/operations/land.goindex d810cea..f795d0b 100644--- a/internal/gitaly/service/operations/land.go+++ b/internal/gitaly/service/operations/land.go@@ -402,11 +402,6 @@ func landMergeConflictError(tip, commitID git.ObjectID, conflictingFiles [][]byt})}-// landKeptMoving is the error of a landing that gave up because the branch kept moving. A model's-// resolution is handed back instead, as a land that needs its new tree verified (it is the same-// cause: the branch moved on): UserLand goes out again, and at its deadline the agent gets the-// proposals, not an error about a branch it has nothing to do with. The others get-// landKeptMovingError.// landSnapshotPinned reports whether the RPC reads a snapshot that is pinned for all of it (MVCC).func landSnapshotPinned(ctx context.Context) bool {return mvcc.CacheFromContext(ctx) != nil@@ -428,6 +423,11 @@ func landLostRace(in *gitalypb.UserLandRequest, pre *landPreResolution, conflictreturn nil}+// landKeptMoving is the error of a landing that gave up because the branch kept moving. A model's+// resolution is handed back instead, as a land that needs its new tree verified (it is the same+// cause: the branch moved on): UserLand goes out again, and at its deadline the agent gets the+// proposals, not an error about a branch it has nothing to do with. The others get+// landKeptMovingError.func landKeptMoving(in *gitalypb.UserLandRequest, pre *landPreResolution, conflict landConflict, attempts int) error {if pre.hasModel() {return &landNeedsResolution{tip: conflict.newOID, commit: git.ObjectID(in.GetCommitId()), moved: true}diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.goindex 4603071..f3ce4a7 100644--- a/internal/gitaly/service/operations/land_preresolve.go+++ b/internal/gitaly/service/operations/land_preresolve.go@@ -292,8 +292,9 @@ func landStoppedReason(need *landNeedsResolution, timeout time.Duration) string}// landFinalPasses bounds the passes of landFinalPass: each is cheap, as it calls no resolver and runs-// no command, so that losing the branch to another land four times in a row is a branch that moves-// by the millisecond.+// no verification of a model's resolution, so that losing the branch to another land four times in a+// row is a branch that moves by the millisecond. (A land of rules only may run verify.command once+// in the lane, in the pass that lands; that is the lane's verification, which is not repeated.)const landFinalPasses = 4// landFinalPass ends a land that could not go on (landResolving): the agent gets the package, notdiff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.goindex c0df4de..b4a93b2 100644--- a/internal/gitaly/service/operations/land_verify.go+++ b/internal/gitaly/service/operations/land_verify.go@@ -2,6 +2,7 @@ package operationsimport ("bufio"+ "bytes""context""crypto/sha1" //nolint:gosec // the hash git names its objects with"crypto/sha256"@@ -72,7 +73,12 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.// write is a verification that did not finish.runCtx, cancel := context.WithTimeout(ctx, timeout)defer cancel()- limits := landCheckoutLimits{entries: v.MaxTreeEntriesOrDefault(), bytes: v.MaxTreeBytesOrDefault()}+ limits := landCheckoutLimits{+ entries: v.MaxTreeEntriesOrDefault(),+ bytes: v.MaxTreeBytesOrDefault(),+ depth: v.MaxTreeDepthOrDefault(),+ pathBytes: landMaxPathBytes,+ }if err := landCheckoutTree(runCtx, repo, tree, checkout, limits); err != nil {var refusal *landCheckoutRefusalswitch {@@ -190,17 +196,33 @@ const (landMaxLinkTarget = 64 << 10// landMaxTreePath bounds a path of the tree, and with it the record that is read of the listing.landMaxTreePath = 4096+ // landNameMax and landPathMax are the limits of a name and of a whole path on the file systems+ // of Linux (NAME_MAX, PATH_MAX): beyond them the checkout refuses, rather than fail on the system call.+ landNameMax = 255+ landPathMax = 4096)func (e landTreeEntry) isGitlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeGitlink }func (e landTreeEntry) isSymlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeSymlink }func (e landTreeEntry) isRegular() bool { return e.mode&landTreeModeTypeMask == landTreeModeRegular }+// landMaxPathBytes bounds the bytes of the distinct paths a checkout keeps in memory: of every entry+// and of every directory made, each charged twice (the sets that look for links and for names that a+// case-insensitive file system folds together). A few thousand bytes of path times many entries is+// where a small push could otherwise take the memory of the server.+const landMaxPathBytes = 128 << 20+// landCheckoutLimits bound a checkout; a zero field is no limit (the server always sets them, from-// verify.max_tree_entries and verify.max_tree_bytes).+// verify.max_tree_entries, verify.max_tree_bytes, verify.max_tree_depth and landMaxPathBytes).+//+// entries counts the entries of the tree AND the directories the checkout makes for them, as both+// are inodes; depth is the number of components of a path; bytes the bytes of the files and links+// (a link's target included); pathBytes the memory of the paths.type landCheckoutLimits struct {- entries int- bytes int64+ entries int+ bytes int64+ depth int+ pathBytes int64}// landCheckoutRefusal is a tree that the checkout refuses to write, closed: it holds more than the@@ -272,7 +294,7 @@ func landCheckoutTreeThen(ctx context.Context, repo *localrepo.Repo, tree git.Ob}()var entries []landTreeEntry- m := newLandMaterializer(dir, limits)+ m := newLandMaterializer(ctx, dir, limits)batch := bufio.NewReaderSize(cat, 1<<20)var writeErr errorfor e := range listed {@@ -439,24 +461,44 @@ type landFolded struct {// landMaterializer writes the entries of a tree under dir, one at a time.type landMaterializer struct {+ ctx context.Contextdir stringlimits landCheckoutLimitslinks map[string]bool // the paths of the symbolic links writtendirs map[string]bool // the directories madefolded map[string]landFolded // the paths as a case-insensitive file system sees them- count int // entries written+ count int // entries written, and directories madebytes int64 // bytes of files and links written+ paths int64 // bytes of the paths kept in memory+}++func newLandMaterializer(ctx context.Context, dir string, limits landCheckoutLimits) *landMaterializer {+ return &landMaterializer{ctx: ctx, dir: dir, limits: limits, links: map[string]bool{}, dirs: map[string]bool{}, folded: map[string]landFolded{}}}-func newLandMaterializer(dir string, limits landCheckoutLimits) *landMaterializer {- return &landMaterializer{dir: dir, limits: limits, links: map[string]bool{}, dirs: map[string]bool{}, folded: map[string]landFolded{}}+// landOSError is the refusal for what the file system would not do: a name too long for it, a name+// that exists already (two names a case-insensitive file system makes one), no room. The tree is+// not one this server can write, which is the agent's to know, not an error of the server.+func landOSError(err error) error {+ if err == nil {+ return nil+ }+ return landRefuse("the tree cannot be written on this file system: %v", err)+}++// landOSWriter makes the errors of writing a file refusals, and not the errors of reading git.+type landOSWriter struct{ w io.Writer }++func (o landOSWriter) Write(p []byte) (int, error) {+ n, err := o.w.Write(p)+ return n, landOSError(err)}// landMaterialize writes the entries under dir, reading the contents of the files and links from batch// in their order (the answer of git-cat-file(1) --batch to the blobs of the listing, gitlinks left// out). See landMaterializer.add for what it refuses. It sets size and target of the entries.func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) error {- m := newLandMaterializer(dir, landCheckoutLimits{})+ m := newLandMaterializer(context.Background(), dir, landCheckoutLimits{})for i := range entries {if err := m.add(&entries[i], batch); err != nil {return err@@ -479,21 +521,38 @@ func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) e// which would follow a link; a file is created with O_EXCL and O_NOFOLLOW; a link is written as a// link and never followed.func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {+ if err := m.ctx.Err(); err != nil {+ return err+ }if !landTreePathValid(e.path) {return landRefuse("unsafe path %q in the tree", e.path)}- if len(e.path) > landMaxTreePath {- return landRefuse("the tree has a path longer than %d bytes", landMaxTreePath)+ // The file system's own limits, refused up front: a name of more than 255 bytes, a path that+ // with the directory of the checkout reaches PATH_MAX.+ if len(e.path) > landMaxTreePath || len(m.dir)+1+len(e.path) >= landPathMax {+ return landRefuse("the tree has a path too long to write (limit %d bytes, with the directory of the checkout)", min(landMaxTreePath, landPathMax-len(m.dir)-2))}- if m.count++; m.limits.entries > 0 && m.count > m.limits.entries {- return landRefuse("the tree has more than %d entries (verify.max_tree_entries)", m.limits.entries)+ if err := m.charge(1); err != nil {+ return err}if e.sized && !e.isGitlink() && m.limits.bytes > 0 && m.bytes+e.listed > m.limits.bytes {return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)}parts := strings.Split(e.path, "/")+ if m.limits.depth > 0 && len(parts) > m.limits.depth {+ return landRefuse("the tree has a path of %d components, more than %d (verify.max_tree_depth)", len(parts), m.limits.depth)+ }+ if !e.isGitlink() {+ // An entry is kept in memory by its path, twice (the entries and the folded names).+ if err := m.chargePath(2 * int64(len(e.path))); err != nil {+ return err+ }+ }for _, c := range parts {+ if len(c) > landNameMax {+ return landRefuse("the path %q has a name of more than %d bytes", e.path, landNameMax)+ }if resolver.IsGitDirName(c) {return landRefuse("the path %q has a component that names .git, which a checkout does not write", e.path)}@@ -544,8 +603,11 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {if _, err := io.ReadFull(batch, content); err != nil {return fmt.Errorf("%q: %w", e.path, err)}+ if bytes.IndexByte(content, 0) >= 0 {+ return landRefuse("the symbolic link %q has a NUL in its target", e.path)+ }if err := os.Symlink(string(content), target); err != nil {- return err+ return landOSError(err)}e.target, e.size = string(content), sizem.links[e.path] = true@@ -556,18 +618,22 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {}f, err := os.OpenFile(target, os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW, perm)if err != nil {- return err+ return landOSError(err)}- if _, err := io.CopyN(f, batch, size); err != nil {+ if _, err := io.CopyN(landOSWriter{f}, batch, size); err != nil {_ = f.Close()+ var refusal *landCheckoutRefusal+ if errors.As(err, &refusal) {+ return err+ }return fmt.Errorf("%q: %w", e.path, err)}if err := f.Close(); err != nil {- return err+ return landOSError(err)}// The mode is the tree's, whatever the umask of the process.if err := os.Chmod(target, perm); err != nil {- return err+ return landOSError(err)}e.size = size}@@ -577,6 +643,22 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {return nil}+// charge counts n more entries or directories against verify.max_tree_entries.+func (m *landMaterializer) charge(n int) error {+ if m.count += n; m.limits.entries > 0 && m.count > m.limits.entries {+ return landRefuse("the tree has more than %d entries and directories (verify.max_tree_entries)", m.limits.entries)+ }+ return nil+}++// chargePath counts n bytes of paths kept in memory against landMaxPathBytes.+func (m *landMaterializer) chargePath(n int64) error {+ if m.paths += n; m.limits.pathBytes > 0 && m.paths > m.limits.pathBytes {+ return landRefuse("the paths of the tree take more than %d bytes of memory", m.limits.pathBytes)+ }+ return nil+}+// mkdirAll makes the directory of the given components under dir, one at a time. A component that// exists must be a directory by Lstat, which does not follow a link: a link, or a file, in the way// is a refusal.@@ -591,11 +673,18 @@ func (m *landMaterializer) mkdirAll(parts []string) error {info, err := os.Lstat(current)switch {case errors.Is(err, fs.ErrNotExist):- if err := os.Mkdir(current, 0o755); err != nil {+ // A directory is an inode and a path kept in memory, charged like the entries.+ if err := m.charge(1); err != nil {return err}+ if err := m.chargePath(2 * int64(len(key))); err != nil {+ return err+ }+ if err := os.Mkdir(current, 0o755); err != nil {+ return landOSError(err)+ }case err != nil:- return err+ return landOSError(err)case !info.IsDir():return landRefuse("%q is in the way of a directory of the tree, and is not a directory", key)}diff --git a/internal/gitaly/service/operations/land_verify_test.go b/internal/gitaly/service/operations/land_verify_test.goindex 769899d..49faf12 100644--- a/internal/gitaly/service/operations/land_verify_test.go+++ b/internal/gitaly/service/operations/land_verify_test.go@@ -4,12 +4,14 @@ import ("archive/tar""bufio""bytes"+ "context""fmt""io""os""path/filepath""strings""testing"+ "time""github.com/stretchr/testify/require"@@ -569,11 +571,22 @@ func TestLandCheckoutTree_limits(t *testing.T) {}}- require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{entries: len(entries), bytes: total}), "at the limits")+ // The entries count, and so does every directory the checkout makes for them (a gitlink is both).+ written := t.TempDir()+ require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, written, landCheckoutLimits{}))+ dirs := 0+ require.NoError(t, filepath.WalkDir(written, func(p string, d os.DirEntry, err error) error {+ if err == nil && d.IsDir() && p != written {+ dirs+++ }+ return nil+ }))+ count := len(entries) + dirs+ require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{entries: count, bytes: total}), "at the limits")var refusal *landCheckoutRefusaldir := t.TempDir()- require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{entries: len(entries) - 1}), &refusal)+ require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{entries: count - 1}), &refusal)require.Contains(t, refusal.reason, "more than")require.Contains(t, refusal.reason, "verify.max_tree_entries")@@ -581,15 +594,15 @@ func TestLandCheckoutTree_limits(t *testing.T) {require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{bytes: total - 1}), &refusal)require.Contains(t, refusal.reason, "verify.max_tree_bytes")// The bytes were refused from the listing, before the file that goes over was read: less than the tree is there.- var written int64+ var bytesWritten int64require.NoError(t, filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {if err == nil && d.Type().IsRegular() {info, _ := d.Info()- written += info.Size()+ bytesWritten += info.Size()}return nil}))- require.Less(t, written, total)+ require.Less(t, bytesWritten, total)}// TestLandCheckoutTree_streamsALargeTree: a tree whose listing and blobs are far over the buffers between@@ -664,3 +677,136 @@ func TestLandVerify_aTreeThatCannotBeCheckedOutIsNotVerified(t *testing.T) {})}}++// shared builds the tree of the attack: a subtree of the given depth, and a root that holds it once per+// name. A few hundred bytes of objects, that git-ls-tree(1) -r expands to one path per name, each as+// deep as the subtree.+func sharedSubtree(t *testing.T, cfg config.Cfg, repoPath string, depth int, names []string) git.ObjectID {+ t.Helper()++ subtree := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{{Mode: "100644", Path: "leaf", Content: "leaf\n"}})+ for i := 1; i < depth; i++ {+ subtree = gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{{Mode: "040000", Path: "d", OID: subtree}})+ }+ var root []gittest.TreeEntry+ for _, name := range names {+ root = append(root, gittest.TreeEntry{Mode: "040000", Path: name, OID: subtree})+ }+ return gittest.WriteTree(t, cfg, repoPath, root)+}++// inodes counts what a checkout made under dir.+func inodes(t *testing.T, dir string) int {+ t.Helper()++ n := 0+ require.NoError(t, filepath.WalkDir(dir, func(string, os.DirEntry, error) error { n++; return nil }))+ return n - 1+}++// TestLandCheckoutTree_aSmallPushCannotBecomeALargeCheckout: a tree of a few hundred bytes that git expands+// (many names for one deep subtree, or many directories) is refused, closed and early, by the depth, the+// entries and directories together, and the memory of the paths; what was made under the directory is+// bounded by the limit, not by the expansion.+func TestLandCheckoutTree_aSmallPushCannotBecomeALargeCheckout(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ repo := localrepo.NewTestRepo(t, cfg, repoProto)+ names := make([]string, 300)+ for i := range names {+ names[i] = fmt.Sprintf("n%03d", i)+ }+ deep := sharedSubtree(t, cfg, repoPath, 200, names[:40])+ medium := sharedSubtree(t, cfg, repoPath, 30, names)++ var wide []gittest.TreeEntry+ for i := 0; i < 600; i++ {+ wide = append(wide, gittest.TreeEntry{Mode: "100644", Path: fmt.Sprintf("w%03d/f", i), Content: "x\n"})+ }+ wideTree := gittest.WriteTree(t, cfg, repoPath, nestTreeEntries(t, cfg, repoPath, wide))++ defaults := landCheckoutLimits{entries: config.DefaultGraftResolverVerifyMaxTreeEntries, bytes: config.DefaultGraftResolverVerifyMaxTreeBytes, depth: config.DefaultGraftResolverVerifyMaxTreeDepth, pathBytes: landMaxPathBytes}+ for name, tc := range map[string]struct {+ tree git.ObjectID+ limits landCheckoutLimits+ reason string+ maxMaker int+ }{+ "the depth, with the defaults": {deep, defaults, "verify.max_tree_depth", 70},+ "directories count as entries": {medium, landCheckoutLimits{entries: 500, depth: 64}, "verify.max_tree_entries", 500},+ "the shallow and wide variant": {wideTree, landCheckoutLimits{entries: 1000, depth: 64}, "verify.max_tree_entries", 1000},+ "the memory of the paths": {medium, landCheckoutLimits{depth: 64, pathBytes: 20000}, "bytes of memory", 400},+ } {+ t.Run(name, func(t *testing.T) {+ t.Parallel()++ dir := t.TempDir()+ start := time.Now()+ err := landCheckoutTree(ctx, repo, tc.tree, dir, tc.limits)+ var refusal *landCheckoutRefusal+ require.ErrorAs(t, err, &refusal)+ require.Contains(t, refusal.reason, tc.reason)+ require.Less(t, inodes(t, dir), tc.maxMaker+1, "what the checkout made is bounded by the limit")+ require.Less(t, time.Since(start), 30*time.Second)+ })+ }++ t.Run("the same trees within the limits are written", func(t *testing.T) {+ t.Parallel()++ require.NoError(t, landCheckoutTree(ctx, repo, medium, t.TempDir(), landCheckoutLimits{entries: 100000, depth: 64, pathBytes: landMaxPathBytes}))+ })+}++// TestLandMaterialize_refusesWhatTheFileSystemWouldNot: the limits of the file system are refused up front, or+// turned into a refusal with the reason when the system call fails: a land with such a tree still gets its+// package, and not an error of the server.+func TestLandMaterialize_refusesWhatTheFileSystemWouldNot(t *testing.T) {+ t.Parallel()++ reg, link := uint32(0o100644), uint32(0o120000)+ longName := strings.Repeat("n", 256)+ var longPath []string+ for i := 0; i < 20; i++ {+ longPath = append(longPath, strings.Repeat("p", 250))+ }+ for name, tc := range map[string]struct {+ dir func(t *testing.T) string+ entries []fakeTreeEntry+ reason string+ }{+ "a name of more than 255 bytes": {entries: []fakeTreeEntry{{reg, longName, "x"}}, reason: "more than 255 bytes"},+ "a path that reaches PATH_MAX": {entries: []fakeTreeEntry{{reg, strings.Join(longPath, "/"), "x"}}, reason: "too long"},+ "a NUL in the target of a link": {entries: []fakeTreeEntry{{link, "l", "a\x00b"}}, reason: "NUL"},+ "a system call that fails": {+ dir: func(t *testing.T) string { return filepath.Join(t.TempDir(), "missing") },+ entries: []fakeTreeEntry{{reg, "a/b", "x"}},+ reason: "cannot be written on this file system",+ },+ } {+ t.Run(name, func(t *testing.T) {+ t.Parallel()++ dir := t.TempDir()+ if tc.dir != nil {+ dir = tc.dir(t)+ }+ var refusal *landCheckoutRefusal+ require.ErrorAs(t, materializeFake(t, dir, tc.entries...), &refusal)+ require.Contains(t, refusal.reason, tc.reason)+ })+ }++ t.Run("a context that ended stops the next entry", func(t *testing.T) {+ t.Parallel()++ ctx, cancel := context.WithCancel(context.Background())+ cancel()+ m := newLandMaterializer(ctx, t.TempDir(), landCheckoutLimits{})+ entries, contents := fakeEntries(t)+ require.ErrorIs(t, m.add(&entries[0], batchOf(contents, entries)), context.Canceled)+ })+}