Skip to content

Git designed for AI agents.

graft: the checkout is bounded by what it makes, not by what the tree lists (review of a6f959c)

Federico Elia61dc0e3c8e44
diff --git a/doc/graft/CONFIGURATION.md b/doc/graft/CONFIGURATION.md
index ed8cfc2..cd89129 100644
--- a/doc/graft/CONFIGURATION.md
+++ b/doc/graft/CONFIGURATION.md
@@ -213,8 +213,9 @@ cause.
| `verify.enabled` | bool | `false` | Run `verify.command` on the merged tree of every resolution (a resolver's or an agent's) before it lands; it applies even when `enabled` is false. It is also what lets a model's resolution land: only a tree it passed on lands, and without it a model's resolution is only ever a proposal. |
| `verify.command` | array of strings | empty | Program and arguments, run in a temporary checkout of the merged tree with `PATH`, `HOME` and `TMPDIR` only. A non-zero exit rejects the resolution. |
| `verify.timeout` | string | `"5m"` | Bound on one verification, the checkout of the tree and the command together (Go duration); one that does not finish rejects the resolution. When `verify.enabled`, it is added to `timeout` for the deadline of a resolved land. |
-| `verify.max_tree_entries` | int | `200000` | Files, links and submodules in the tree that is checked out for the command; a bigger tree is not verified, and the package says why. |
-| `verify.max_tree_bytes` | int | `2147483648` | Bytes of the files of that tree (checked from the listing, before any content is read). |
+| `verify.max_tree_entries` | int | `200000` | Files, links and submodules in the tree that is checked out for the command, plus the directories made for them; a bigger tree is not verified, and the package says why. |
+| `verify.max_tree_bytes` | int | `2147483648` | Bytes of the files of that tree and of the targets of its links, checked per entry from the listing, before that entry's content is read. |
+| `verify.max_tree_depth` | int | `64` | Components of a path of that tree. A tree can name one deep subtree many times; the expansion is what this and the entry count bound. |
A file that is never sent goes to the agent with its reason in the conflict
package: large (a side over `max_file_bytes`), binary (a NUL byte), lockfile,
diff --git a/doc/graft/PLAN10.md b/doc/graft/PLAN10.md
index 5e60a58..707244f 100644
--- a/doc/graft/PLAN10.md
+++ b/doc/graft/PLAN10.md
@@ -1312,12 +1312,25 @@ and the tree it verifies depends on the tip.
link of the tree, also when a case-insensitive file system would see the same name (a link "A" and
an entry "a/x"), a parent that is a link or a file, a path over 4096 bytes. Each directory is made
one component at a time and looked at with Lstat (never MkdirAll, which follows links), a file is
- created with O_EXCL and O_NOFOLLOW. Resources: verify.max_tree_entries (default 200000) and
- verify.max_tree_bytes (default 2 GiB), checked from the listing (ls-tree -l) before the content is
- read; the listing and the blobs are streamed, entry by entry, through two commands, so the memory is
- that of the entries. The checkout runs inside verify.timeout, as the command does. A refusal is no
- error of the server: nothing is verified, the files are PROPOSED (a model's) or REJECTED (rules'),
- and the reason says which limit or path.
+ created with O_EXCL and O_NOFOLLOW. Resources, all failing closed with the reason: a git tree can
+ name one subtree many times, so a push of a few KB expands to as many paths as it has names, each as
+ deep as the subtree. verify.max_tree_entries (default 200000) counts the entries AND the directories
+ that the checkout makes (both are inodes); verify.max_tree_depth (default 64) the components of a
+ path; verify.max_tree_bytes (default 2 GiB) the bytes of the files and of the targets of links; and
+ a fixed 128 MiB the memory of the paths, each entry and each directory charged twice (the sets that
+ look for links and for folded names). The limits are checked entry by entry, from the listing
+ (ls-tree -l), BEFORE that entry's content is read, not from a first pass over the whole listing: the
+ listing and the blobs are streamed through two commands, so the memory is that of the paths kept (the
+ fixed budget), and an entry that would go over stops the checkout with the earlier ones already
+ written. The context is checked at every entry. A name over 255 bytes, a path that with the
+ directory reaches PATH_MAX, a NUL in the target of a link, and a system call that fails (a name
+ that exists already, no room) are refused with the reason, not errors of the server. The checkout
+ runs inside verify.timeout, as the command does. A refusal is no error of the server: nothing is
+ verified, the files are PROPOSED (a model's) or REJECTED (rules'), and the reason says which limit
+ or path. Not built: NTFS trailing-dot and space folding in the check of colliding names (the
+ .git check has it); Gitaly does not run on Windows. Under MVCC (a snapshot pinned for the RPC) a
+ lost race ends with the retryable FailedPrecondition "kept moving" even for a model's resolution:
+ that call carries no package; the client's retry gets one.
* **Tests.** A model's resolution that passes lands, signed, with the command run once (so not in
the lane); one that fails gives a proposal and the output; without a verify.command it is a
proposal; a tip that moved gets its new tree verified and lands, with no resolver call; a tip that
diff --git a/doc/graft/RESOLVE.md b/doc/graft/RESOLVE.md
index 4cdaddf..d7ed9cc 100644
--- a/doc/graft/RESOLVE.md
+++ b/doc/graft/RESOLVE.md
@@ -415,8 +415,11 @@ read back and compared with the listing, the content of every file included (it
is hashed as a git blob and must be the object ID of the tree), and a difference
fails the checkout closed. The listing and the blobs are streamed, not held in
memory, and the checkout refuses, with the reason in the package and without
-running the command: a tree over `verify.max_tree_entries` entries or
-`verify.max_tree_bytes` bytes, a path component that names `.git` on any file
+running the command: a tree over `verify.max_tree_entries` entries and
+directories made, `verify.max_tree_depth` components in a path, or
+`verify.max_tree_bytes` bytes of files and link targets (checked per entry,
+before that entry's content is read, so the earlier entries are already
+written), or paths that take more than 128 MiB of memory, a path component that names `.git` on any file
system (the command may run git there, and a `.git` of the tree would be its
repository), and an entry under a symbolic link, also when a case-insensitive
file system would see the same name. The checkout and the command run inside
diff --git a/internal/gitaly/config/config_graft.go b/internal/gitaly/config/config_graft.go
index 849b290..046c9ca 100644
--- a/internal/gitaly/config/config_graft.go
+++ b/internal/gitaly/config/config_graft.go
@@ -387,6 +387,9 @@ type GraftResolverVerify struct {
// MaxTreeBytes bounds the bytes of the files of that tree. Zero means
// DefaultGraftResolverVerifyMaxTreeBytes.
MaxTreeBytes int64 `json:"max_tree_bytes" toml:"max_tree_bytes,omitempty"`
+ // MaxTreeDepth bounds the components of a path of that tree. Zero means
+ // DefaultGraftResolverVerifyMaxTreeDepth.
+ MaxTreeDepth int `json:"max_tree_depth" toml:"max_tree_depth,omitempty"`
}
// DefaultGraftResolverVerifyTimeout is the default GraftResolverVerify.Timeout.
@@ -397,8 +400,18 @@ const (
DefaultGraftResolverVerifyMaxTreeEntries = 200000
// DefaultGraftResolverVerifyMaxTreeBytes is the default GraftResolverVerify.MaxTreeBytes: 2 GiB.
DefaultGraftResolverVerifyMaxTreeBytes int64 = 2 << 30
+ // DefaultGraftResolverVerifyMaxTreeDepth is the default GraftResolverVerify.MaxTreeDepth.
+ DefaultGraftResolverVerifyMaxTreeDepth = 64
)
+// MaxTreeDepthOrDefault returns MaxTreeDepth or its default.
+func (v GraftResolverVerify) MaxTreeDepthOrDefault() int {
+ if v.MaxTreeDepth > 0 {
+ return v.MaxTreeDepth
+ }
+ return DefaultGraftResolverVerifyMaxTreeDepth
+}
+
// MaxTreeEntriesOrDefault returns MaxTreeEntries or its default.
func (v GraftResolverVerify) MaxTreeEntriesOrDefault() int {
if v.MaxTreeEntries > 0 {
@@ -429,8 +442,8 @@ func (v GraftResolverVerify) TimeoutOrDefault() (time.Duration, error) {
}
func (v GraftResolverVerify) validate() error {
- if v.MaxTreeEntries < 0 || v.MaxTreeBytes < 0 {
- return fmt.Errorf("graft land resolver verify: max_tree_entries and max_tree_bytes must not be negative (0 means the default)")
+ if v.MaxTreeEntries < 0 || v.MaxTreeBytes < 0 || v.MaxTreeDepth < 0 {
+ return fmt.Errorf("graft land resolver verify: max_tree_entries, max_tree_bytes and max_tree_depth must not be negative (0 means the default)")
}
if !v.Enabled {
return nil
diff --git a/internal/gitaly/config/config_graft_resolver_test.go b/internal/gitaly/config/config_graft_resolver_test.go
index de94f38..69a399f 100644
--- a/internal/gitaly/config/config_graft_resolver_test.go
+++ b/internal/gitaly/config/config_graft_resolver_test.go
@@ -55,6 +55,8 @@ func TestGraftResolverValidateVerify(t *testing.T) {
require.Equal(t, "5m0s", d.String())
require.Equal(t, 200000, GraftResolverVerify{}.MaxTreeEntriesOrDefault())
require.Equal(t, int64(2<<30), GraftResolverVerify{}.MaxTreeBytesOrDefault())
+ require.Equal(t, 64, GraftResolverVerify{}.MaxTreeDepthOrDefault())
+ require.Equal(t, 3, GraftResolverVerify{MaxTreeDepth: 3}.MaxTreeDepthOrDefault())
require.Equal(t, 7, GraftResolverVerify{MaxTreeEntries: 7}.MaxTreeEntriesOrDefault())
require.Equal(t, int64(9), GraftResolverVerify{MaxTreeBytes: 9}.MaxTreeBytesOrDefault())
@@ -65,6 +67,7 @@ func TestGraftResolverValidateVerify(t *testing.T) {
"zero timeout": {Enabled: true, Command: []string{"x"}, Timeout: "0s"},
"negative entries": {MaxTreeEntries: -1},
"negative bytes": {MaxTreeBytes: -1},
+ "negative depth": {MaxTreeDepth: -1},
} {
require.Error(t, GraftResolver{Verify: v}.validate(), name)
}
diff --git a/internal/gitaly/service/operations/land.go b/internal/gitaly/service/operations/land.go
index d810cea..f795d0b 100644
--- a/internal/gitaly/service/operations/land.go
+++ b/internal/gitaly/service/operations/land.go
@@ -402,11 +402,6 @@ func landMergeConflictError(tip, commitID git.ObjectID, conflictingFiles [][]byt
})
}
-// landKeptMoving is the error of a landing that gave up because the branch kept moving. A model's
-// resolution is handed back instead, as a land that needs its new tree verified (it is the same
-// cause: the branch moved on): UserLand goes out again, and at its deadline the agent gets the
-// proposals, not an error about a branch it has nothing to do with. The others get
-// landKeptMovingError.
// landSnapshotPinned reports whether the RPC reads a snapshot that is pinned for all of it (MVCC).
func landSnapshotPinned(ctx context.Context) bool {
return mvcc.CacheFromContext(ctx) != nil
@@ -428,6 +423,11 @@ func landLostRace(in *gitalypb.UserLandRequest, pre *landPreResolution, conflict
return nil
}
+// landKeptMoving is the error of a landing that gave up because the branch kept moving. A model's
+// resolution is handed back instead, as a land that needs its new tree verified (it is the same
+// cause: the branch moved on): UserLand goes out again, and at its deadline the agent gets the
+// proposals, not an error about a branch it has nothing to do with. The others get
+// landKeptMovingError.
func landKeptMoving(in *gitalypb.UserLandRequest, pre *landPreResolution, conflict landConflict, attempts int) error {
if pre.hasModel() {
return &landNeedsResolution{tip: conflict.newOID, commit: git.ObjectID(in.GetCommitId()), moved: true}
diff --git a/internal/gitaly/service/operations/land_preresolve.go b/internal/gitaly/service/operations/land_preresolve.go
index 4603071..f3ce4a7 100644
--- a/internal/gitaly/service/operations/land_preresolve.go
+++ b/internal/gitaly/service/operations/land_preresolve.go
@@ -292,8 +292,9 @@ func landStoppedReason(need *landNeedsResolution, timeout time.Duration) string
}
// landFinalPasses bounds the passes of landFinalPass: each is cheap, as it calls no resolver and runs
-// no command, so that losing the branch to another land four times in a row is a branch that moves
-// by the millisecond.
+// no verification of a model's resolution, so that losing the branch to another land four times in a
+// row is a branch that moves by the millisecond. (A land of rules only may run verify.command once
+// in the lane, in the pass that lands; that is the lane's verification, which is not repeated.)
const landFinalPasses = 4
// landFinalPass ends a land that could not go on (landResolving): the agent gets the package, not
diff --git a/internal/gitaly/service/operations/land_verify.go b/internal/gitaly/service/operations/land_verify.go
index c0df4de..b4a93b2 100644
--- a/internal/gitaly/service/operations/land_verify.go
+++ b/internal/gitaly/service/operations/land_verify.go
@@ -2,6 +2,7 @@ package operations
import (
"bufio"
+ "bytes"
"context"
"crypto/sha1" //nolint:gosec // the hash git names its objects with
"crypto/sha256"
@@ -72,7 +73,12 @@ func (s *Server) landVerify(ctx context.Context, repo *localrepo.Repo, tree git.
// write is a verification that did not finish.
runCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
- limits := landCheckoutLimits{entries: v.MaxTreeEntriesOrDefault(), bytes: v.MaxTreeBytesOrDefault()}
+ limits := landCheckoutLimits{
+ entries: v.MaxTreeEntriesOrDefault(),
+ bytes: v.MaxTreeBytesOrDefault(),
+ depth: v.MaxTreeDepthOrDefault(),
+ pathBytes: landMaxPathBytes,
+ }
if err := landCheckoutTree(runCtx, repo, tree, checkout, limits); err != nil {
var refusal *landCheckoutRefusal
switch {
@@ -190,17 +196,33 @@ const (
landMaxLinkTarget = 64 << 10
// landMaxTreePath bounds a path of the tree, and with it the record that is read of the listing.
landMaxTreePath = 4096
+ // landNameMax and landPathMax are the limits of a name and of a whole path on the file systems
+ // of Linux (NAME_MAX, PATH_MAX): beyond them the checkout refuses, rather than fail on the system call.
+ landNameMax = 255
+ landPathMax = 4096
)
func (e landTreeEntry) isGitlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeGitlink }
func (e landTreeEntry) isSymlink() bool { return e.mode&landTreeModeTypeMask == landTreeModeSymlink }
func (e landTreeEntry) isRegular() bool { return e.mode&landTreeModeTypeMask == landTreeModeRegular }
+// landMaxPathBytes bounds the bytes of the distinct paths a checkout keeps in memory: of every entry
+// and of every directory made, each charged twice (the sets that look for links and for names that a
+// case-insensitive file system folds together). A few thousand bytes of path times many entries is
+// where a small push could otherwise take the memory of the server.
+const landMaxPathBytes = 128 << 20
+
// landCheckoutLimits bound a checkout; a zero field is no limit (the server always sets them, from
-// verify.max_tree_entries and verify.max_tree_bytes).
+// verify.max_tree_entries, verify.max_tree_bytes, verify.max_tree_depth and landMaxPathBytes).
+//
+// entries counts the entries of the tree AND the directories the checkout makes for them, as both
+// are inodes; depth is the number of components of a path; bytes the bytes of the files and links
+// (a link's target included); pathBytes the memory of the paths.
type landCheckoutLimits struct {
- entries int
- bytes int64
+ entries int
+ bytes int64
+ depth int
+ pathBytes int64
}
// landCheckoutRefusal is a tree that the checkout refuses to write, closed: it holds more than the
@@ -272,7 +294,7 @@ func landCheckoutTreeThen(ctx context.Context, repo *localrepo.Repo, tree git.Ob
}()
var entries []landTreeEntry
- m := newLandMaterializer(dir, limits)
+ m := newLandMaterializer(ctx, dir, limits)
batch := bufio.NewReaderSize(cat, 1<<20)
var writeErr error
for e := range listed {
@@ -439,24 +461,44 @@ type landFolded struct {
// landMaterializer writes the entries of a tree under dir, one at a time.
type landMaterializer struct {
+ ctx context.Context
dir string
limits landCheckoutLimits
links map[string]bool // the paths of the symbolic links written
dirs map[string]bool // the directories made
folded map[string]landFolded // the paths as a case-insensitive file system sees them
- count int // entries written
+ count int // entries written, and directories made
bytes int64 // bytes of files and links written
+ paths int64 // bytes of the paths kept in memory
+}
+
+func newLandMaterializer(ctx context.Context, dir string, limits landCheckoutLimits) *landMaterializer {
+ return &landMaterializer{ctx: ctx, dir: dir, limits: limits, links: map[string]bool{}, dirs: map[string]bool{}, folded: map[string]landFolded{}}
}
-func newLandMaterializer(dir string, limits landCheckoutLimits) *landMaterializer {
- return &landMaterializer{dir: dir, limits: limits, links: map[string]bool{}, dirs: map[string]bool{}, folded: map[string]landFolded{}}
+// landOSError is the refusal for what the file system would not do: a name too long for it, a name
+// that exists already (two names a case-insensitive file system makes one), no room. The tree is
+// not one this server can write, which is the agent's to know, not an error of the server.
+func landOSError(err error) error {
+ if err == nil {
+ return nil
+ }
+ return landRefuse("the tree cannot be written on this file system: %v", err)
+}
+
+// landOSWriter makes the errors of writing a file refusals, and not the errors of reading git.
+type landOSWriter struct{ w io.Writer }
+
+func (o landOSWriter) Write(p []byte) (int, error) {
+ n, err := o.w.Write(p)
+ return n, landOSError(err)
}
// landMaterialize writes the entries under dir, reading the contents of the files and links from batch
// in their order (the answer of git-cat-file(1) --batch to the blobs of the listing, gitlinks left
// out). See landMaterializer.add for what it refuses. It sets size and target of the entries.
func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) error {
- m := newLandMaterializer(dir, landCheckoutLimits{})
+ m := newLandMaterializer(context.Background(), dir, landCheckoutLimits{})
for i := range entries {
if err := m.add(&entries[i], batch); err != nil {
return err
@@ -479,21 +521,38 @@ func landMaterialize(entries []landTreeEntry, batch *bufio.Reader, dir string) e
// which would follow a link; a file is created with O_EXCL and O_NOFOLLOW; a link is written as a
// link and never followed.
func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
+ if err := m.ctx.Err(); err != nil {
+ return err
+ }
if !landTreePathValid(e.path) {
return landRefuse("unsafe path %q in the tree", e.path)
}
- if len(e.path) > landMaxTreePath {
- return landRefuse("the tree has a path longer than %d bytes", landMaxTreePath)
+ // The file system's own limits, refused up front: a name of more than 255 bytes, a path that
+ // with the directory of the checkout reaches PATH_MAX.
+ if len(e.path) > landMaxTreePath || len(m.dir)+1+len(e.path) >= landPathMax {
+ return landRefuse("the tree has a path too long to write (limit %d bytes, with the directory of the checkout)", min(landMaxTreePath, landPathMax-len(m.dir)-2))
}
- if m.count++; m.limits.entries > 0 && m.count > m.limits.entries {
- return landRefuse("the tree has more than %d entries (verify.max_tree_entries)", m.limits.entries)
+ if err := m.charge(1); err != nil {
+ return err
}
if e.sized && !e.isGitlink() && m.limits.bytes > 0 && m.bytes+e.listed > m.limits.bytes {
return landRefuse("the tree has more than %d bytes of files (verify.max_tree_bytes)", m.limits.bytes)
}
parts := strings.Split(e.path, "/")
+ if m.limits.depth > 0 && len(parts) > m.limits.depth {
+ return landRefuse("the tree has a path of %d components, more than %d (verify.max_tree_depth)", len(parts), m.limits.depth)
+ }
+ if !e.isGitlink() {
+ // An entry is kept in memory by its path, twice (the entries and the folded names).
+ if err := m.chargePath(2 * int64(len(e.path))); err != nil {
+ return err
+ }
+ }
for _, c := range parts {
+ if len(c) > landNameMax {
+ return landRefuse("the path %q has a name of more than %d bytes", e.path, landNameMax)
+ }
if resolver.IsGitDirName(c) {
return landRefuse("the path %q has a component that names .git, which a checkout does not write", e.path)
}
@@ -544,8 +603,11 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
if _, err := io.ReadFull(batch, content); err != nil {
return fmt.Errorf("%q: %w", e.path, err)
}
+ if bytes.IndexByte(content, 0) >= 0 {
+ return landRefuse("the symbolic link %q has a NUL in its target", e.path)
+ }
if err := os.Symlink(string(content), target); err != nil {
- return err
+ return landOSError(err)
}
e.target, e.size = string(content), size
m.links[e.path] = true
@@ -556,18 +618,22 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
}
f, err := os.OpenFile(target, os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW, perm)
if err != nil {
- return err
+ return landOSError(err)
}
- if _, err := io.CopyN(f, batch, size); err != nil {
+ if _, err := io.CopyN(landOSWriter{f}, batch, size); err != nil {
_ = f.Close()
+ var refusal *landCheckoutRefusal
+ if errors.As(err, &refusal) {
+ return err
+ }
return fmt.Errorf("%q: %w", e.path, err)
}
if err := f.Close(); err != nil {
- return err
+ return landOSError(err)
}
// The mode is the tree's, whatever the umask of the process.
if err := os.Chmod(target, perm); err != nil {
- return err
+ return landOSError(err)
}
e.size = size
}
@@ -577,6 +643,22 @@ func (m *landMaterializer) add(e *landTreeEntry, batch *bufio.Reader) error {
return nil
}
+// charge counts n more entries or directories against verify.max_tree_entries.
+func (m *landMaterializer) charge(n int) error {
+ if m.count += n; m.limits.entries > 0 && m.count > m.limits.entries {
+ return landRefuse("the tree has more than %d entries and directories (verify.max_tree_entries)", m.limits.entries)
+ }
+ return nil
+}
+
+// chargePath counts n bytes of paths kept in memory against landMaxPathBytes.
+func (m *landMaterializer) chargePath(n int64) error {
+ if m.paths += n; m.limits.pathBytes > 0 && m.paths > m.limits.pathBytes {
+ return landRefuse("the paths of the tree take more than %d bytes of memory", m.limits.pathBytes)
+ }
+ return nil
+}
+
// mkdirAll makes the directory of the given components under dir, one at a time. A component that
// exists must be a directory by Lstat, which does not follow a link: a link, or a file, in the way
// is a refusal.
@@ -591,11 +673,18 @@ func (m *landMaterializer) mkdirAll(parts []string) error {
info, err := os.Lstat(current)
switch {
case errors.Is(err, fs.ErrNotExist):
- if err := os.Mkdir(current, 0o755); err != nil {
+ // A directory is an inode and a path kept in memory, charged like the entries.
+ if err := m.charge(1); err != nil {
return err
}
+ if err := m.chargePath(2 * int64(len(key))); err != nil {
+ return err
+ }
+ if err := os.Mkdir(current, 0o755); err != nil {
+ return landOSError(err)
+ }
case err != nil:
- return err
+ return landOSError(err)
case !info.IsDir():
return landRefuse("%q is in the way of a directory of the tree, and is not a directory", key)
}
diff --git a/internal/gitaly/service/operations/land_verify_test.go b/internal/gitaly/service/operations/land_verify_test.go
index 769899d..49faf12 100644
--- a/internal/gitaly/service/operations/land_verify_test.go
+++ b/internal/gitaly/service/operations/land_verify_test.go
@@ -4,12 +4,14 @@ import (
"archive/tar"
"bufio"
"bytes"
+ "context"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"testing"
+ "time"
"github.com/stretchr/testify/require"
@@ -569,11 +571,22 @@ func TestLandCheckoutTree_limits(t *testing.T) {
}
}
- require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{entries: len(entries), bytes: total}), "at the limits")
+ // The entries count, and so does every directory the checkout makes for them (a gitlink is both).
+ written := t.TempDir()
+ require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, written, landCheckoutLimits{}))
+ dirs := 0
+ require.NoError(t, filepath.WalkDir(written, func(p string, d os.DirEntry, err error) error {
+ if err == nil && d.IsDir() && p != written {
+ dirs++
+ }
+ return nil
+ }))
+ count := len(entries) + dirs
+ require.NoError(t, landCheckoutTree(ctx, fx.repo, fx.tree, t.TempDir(), landCheckoutLimits{entries: count, bytes: total}), "at the limits")
var refusal *landCheckoutRefusal
dir := t.TempDir()
- require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{entries: len(entries) - 1}), &refusal)
+ require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{entries: count - 1}), &refusal)
require.Contains(t, refusal.reason, "more than")
require.Contains(t, refusal.reason, "verify.max_tree_entries")
@@ -581,15 +594,15 @@ func TestLandCheckoutTree_limits(t *testing.T) {
require.ErrorAs(t, landCheckoutTree(ctx, fx.repo, fx.tree, dir, landCheckoutLimits{bytes: total - 1}), &refusal)
require.Contains(t, refusal.reason, "verify.max_tree_bytes")
// The bytes were refused from the listing, before the file that goes over was read: less than the tree is there.
- var written int64
+ var bytesWritten int64
require.NoError(t, filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
if err == nil && d.Type().IsRegular() {
info, _ := d.Info()
- written += info.Size()
+ bytesWritten += info.Size()
}
return nil
}))
- require.Less(t, written, total)
+ require.Less(t, bytesWritten, total)
}
// TestLandCheckoutTree_streamsALargeTree: a tree whose listing and blobs are far over the buffers between
@@ -664,3 +677,136 @@ func TestLandVerify_aTreeThatCannotBeCheckedOutIsNotVerified(t *testing.T) {
})
}
}
+
+// shared builds the tree of the attack: a subtree of the given depth, and a root that holds it once per
+// name. A few hundred bytes of objects, that git-ls-tree(1) -r expands to one path per name, each as
+// deep as the subtree.
+func sharedSubtree(t *testing.T, cfg config.Cfg, repoPath string, depth int, names []string) git.ObjectID {
+ t.Helper()
+
+ subtree := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{{Mode: "100644", Path: "leaf", Content: "leaf\n"}})
+ for i := 1; i < depth; i++ {
+ subtree = gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{{Mode: "040000", Path: "d", OID: subtree}})
+ }
+ var root []gittest.TreeEntry
+ for _, name := range names {
+ root = append(root, gittest.TreeEntry{Mode: "040000", Path: name, OID: subtree})
+ }
+ return gittest.WriteTree(t, cfg, repoPath, root)
+}
+
+// inodes counts what a checkout made under dir.
+func inodes(t *testing.T, dir string) int {
+ t.Helper()
+
+ n := 0
+ require.NoError(t, filepath.WalkDir(dir, func(string, os.DirEntry, error) error { n++; return nil }))
+ return n - 1
+}
+
+// TestLandCheckoutTree_aSmallPushCannotBecomeALargeCheckout: a tree of a few hundred bytes that git expands
+// (many names for one deep subtree, or many directories) is refused, closed and early, by the depth, the
+// entries and directories together, and the memory of the paths; what was made under the directory is
+// bounded by the limit, not by the expansion.
+func TestLandCheckoutTree_aSmallPushCannotBecomeALargeCheckout(t *testing.T) {
+ t.Parallel()
+
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ repo := localrepo.NewTestRepo(t, cfg, repoProto)
+ names := make([]string, 300)
+ for i := range names {
+ names[i] = fmt.Sprintf("n%03d", i)
+ }
+ deep := sharedSubtree(t, cfg, repoPath, 200, names[:40])
+ medium := sharedSubtree(t, cfg, repoPath, 30, names)
+
+ var wide []gittest.TreeEntry
+ for i := 0; i < 600; i++ {
+ wide = append(wide, gittest.TreeEntry{Mode: "100644", Path: fmt.Sprintf("w%03d/f", i), Content: "x\n"})
+ }
+ wideTree := gittest.WriteTree(t, cfg, repoPath, nestTreeEntries(t, cfg, repoPath, wide))
+
+ defaults := landCheckoutLimits{entries: config.DefaultGraftResolverVerifyMaxTreeEntries, bytes: config.DefaultGraftResolverVerifyMaxTreeBytes, depth: config.DefaultGraftResolverVerifyMaxTreeDepth, pathBytes: landMaxPathBytes}
+ for name, tc := range map[string]struct {
+ tree git.ObjectID
+ limits landCheckoutLimits
+ reason string
+ maxMaker int
+ }{
+ "the depth, with the defaults": {deep, defaults, "verify.max_tree_depth", 70},
+ "directories count as entries": {medium, landCheckoutLimits{entries: 500, depth: 64}, "verify.max_tree_entries", 500},
+ "the shallow and wide variant": {wideTree, landCheckoutLimits{entries: 1000, depth: 64}, "verify.max_tree_entries", 1000},
+ "the memory of the paths": {medium, landCheckoutLimits{depth: 64, pathBytes: 20000}, "bytes of memory", 400},
+ } {
+ t.Run(name, func(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ start := time.Now()
+ err := landCheckoutTree(ctx, repo, tc.tree, dir, tc.limits)
+ var refusal *landCheckoutRefusal
+ require.ErrorAs(t, err, &refusal)
+ require.Contains(t, refusal.reason, tc.reason)
+ require.Less(t, inodes(t, dir), tc.maxMaker+1, "what the checkout made is bounded by the limit")
+ require.Less(t, time.Since(start), 30*time.Second)
+ })
+ }
+
+ t.Run("the same trees within the limits are written", func(t *testing.T) {
+ t.Parallel()
+
+ require.NoError(t, landCheckoutTree(ctx, repo, medium, t.TempDir(), landCheckoutLimits{entries: 100000, depth: 64, pathBytes: landMaxPathBytes}))
+ })
+}
+
+// TestLandMaterialize_refusesWhatTheFileSystemWouldNot: the limits of the file system are refused up front, or
+// turned into a refusal with the reason when the system call fails: a land with such a tree still gets its
+// package, and not an error of the server.
+func TestLandMaterialize_refusesWhatTheFileSystemWouldNot(t *testing.T) {
+ t.Parallel()
+
+ reg, link := uint32(0o100644), uint32(0o120000)
+ longName := strings.Repeat("n", 256)
+ var longPath []string
+ for i := 0; i < 20; i++ {
+ longPath = append(longPath, strings.Repeat("p", 250))
+ }
+ for name, tc := range map[string]struct {
+ dir func(t *testing.T) string
+ entries []fakeTreeEntry
+ reason string
+ }{
+ "a name of more than 255 bytes": {entries: []fakeTreeEntry{{reg, longName, "x"}}, reason: "more than 255 bytes"},
+ "a path that reaches PATH_MAX": {entries: []fakeTreeEntry{{reg, strings.Join(longPath, "/"), "x"}}, reason: "too long"},
+ "a NUL in the target of a link": {entries: []fakeTreeEntry{{link, "l", "a\x00b"}}, reason: "NUL"},
+ "a system call that fails": {
+ dir: func(t *testing.T) string { return filepath.Join(t.TempDir(), "missing") },
+ entries: []fakeTreeEntry{{reg, "a/b", "x"}},
+ reason: "cannot be written on this file system",
+ },
+ } {
+ t.Run(name, func(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ if tc.dir != nil {
+ dir = tc.dir(t)
+ }
+ var refusal *landCheckoutRefusal
+ require.ErrorAs(t, materializeFake(t, dir, tc.entries...), &refusal)
+ require.Contains(t, refusal.reason, tc.reason)
+ })
+ }
+
+ t.Run("a context that ended stops the next entry", func(t *testing.T) {
+ t.Parallel()
+
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ m := newLandMaterializer(ctx, t.TempDir(), landCheckoutLimits{})
+ entries, contents := fakeEntries(t)
+ require.ErrorIs(t, m.add(&entries[0], batchOf(contents, entries)), context.Canceled)
+ })
+}