Git designed for AI agents.
operations: UserCommitFiles in 5 git processes (WP96)
Federico Eliab677dec44a72
diff --git a/doc/graft/PLAN9.md b/doc/graft/PLAN9.mdindex 2c96993..7ab4946 100644--- a/doc/graft/PLAN9.md+++ b/doc/graft/PLAN9.md@@ -652,3 +652,149 @@ and commit build (95 to 146 ms), stage 2.with the numbers of the window.* **Release.** v0.2.0 is main with phase 4b, the skew fix, its performance fix(`6aecff1`) and the `UserSquash` fix. The rest of this plan goes to v0.3.0.++## 15. WP96: the git processes of `UserCommitFiles` (design, for approval)++Owner since 2026-10-19: m3 (scion-74), taking over p5w. This replaces the sketch of 3.4.++### 15.1 Where the processes come from (measured and traced)++`TestUserCommitFilesProcessCount` (`operations`, counting command factory) counts the+git processes of one request that creates one file:++| Case | Processes | By subcommand |+|---|---|---|+| existing branch, `file.txt` | 10 | rev-parse 3, hash-object 2, for-each-ref 1, ls-tree, commit-tree, cat-file, update-ref |+| existing branch, `a/b/file.txt` | 12 | hash-object 4 (1 blob + 3 trees) |+| new branch (from `start_branch_name`), `file.txt` | 13 | rev-parse 5, for-each-ref 2 |+| new branch, `a/b/file.txt` | 15 | |++Each one, traced to its call site (all through `localrepo.Repo.Exec`):++| | Process | Count | From | Needed? |+|---|---|---|---|---|+| A | `rev-parse --verify refs/heads/<branch>^{commit}` | 1 | `commit_files.go:569` | a read: the request's object reader can answer it |+| B | `for-each-ref --count=1 refs/heads/` (`HasBranches`) | new branch: 1 | `commit_files.go:799` (issue-3294 workaround) | only when the start branch does not resolve |+| C | `rev-parse --verify refs/heads/<start>^{commit}` | new branch: 1 | `commit_files.go:829` | a read |+| D | `rev-parse --verify <oid>^{commit}` (`fetchMissingCommit`) | new branch: 1 | `commit_files.go:843` | redundant: C resolved and peeled it in the same repository |+| E | `hash-object --stdin -w --path=<path>` | 1 per blob | `commit_files.go:655` | yes: `--path` applies the repository's attributes (clean and eol filters) to the content |+| F | `rev-parse --verify <parent>^{tree}` | 1 | `commit_files.go:453` | a read |+| G | `rev-parse --verify <tree>:` | 1 | `tree.go:689` (`ReadTree`) | redundant: its input is a tree ID, which `Info` checks right after |+| H | `cat-file -Z --batch-command --buffer` | 1 | `tree.go:694` | the request's object reader (a cache miss: the quarantine is per request) |+| I | `ls-tree -z -r -t <tree>:` | 1 | `tree.go:407` | reads the whole tree recursively; only the trees on the changed paths are needed |+| J | `hash-object -t tree -w --stdin` | 1 per changed tree (depth + 1) | `tree.go:623` | one process can write every tree |+| K | `commit-tree` | 1 | `commit.go:203` | yes (see 15.2, item 4) |+| L | `for-each-ref --count=1 refs/heads/` | 1 | `commit_files.go:732` | only for `RepoCreated`, which A or C already decides |+| M | `update-ref -z --stdin` | 1 | `update_with_hooks.go:286` | yes |++### 15.2 Design++1. **Reads go through the request's object reader (H).** One `cat-file --batch-command`+ process, taken once per request and kept for it (with the MVCC backend the catfile+ cache is off, so the handle must be held, not looked up per read):+ * A, C and F become `Info("refs/heads/<branch>^{commit}")`,+ `Info("refs/heads/<start>^{commit}")` and `Info("<parent>^{tree}")`. A missing+ object answers `missing`, which maps to the same not-found error as the failed+ `rev-parse` today. Both resolve a name with git's same rules (`get_oid`), so+ no ambiguity rule of `rev-parse` is lost. Probed on git 2.53: a running `cat-file --batch-command`+ resolves `refs/heads/x^{commit}` and `^{tree}`, answers `missing` for an absent+ reference, and sees objects and references written after it started.+ * D becomes `Info("<parent>^{commit}")` on the same reader (the fetch from the start+ repository, when it is missing, is unchanged; the reader then sees the fetched pack). G is+ dropped (F already answered it).+ * I is replaced by a lazy tree: the root is read with `Object()` and parsed (in+ `localrepo`, into the entries `ls-tree -r -t` gives: a six-digit mode and the type+ from the mode's format bits), and a subtree is read only when an action descends into it+ (create, update, delete, move, chmod and create-dir each address one path).+ Untouched subtrees keep their ID, so `Write` skips them as today. The cost no+ longer grows with the size of the repository's tree.+2. **One process writes every tree.** Today's writer, `git hash-object -t tree -w`, in+ `--stdin-paths` mode: the request starts it once, writes each tree's content (the+ same bytes `writeEntries` builds today) to a temporary file of the storage, sends the+ path and reads the ID back, bottom-up (the `CommitWriter` pattern, for trees). The+ bytes, the format checks and the absence of an existence check are those of today, by+ construction. (Amended after the approval: `mktree --batch` was probed. It gives the+ same IDs on valid trees, but it writes trees with duplicate names and invalid modes,+ which `hash-object` refuses as malformed, and it refuses entries whose object is+ missing, which today's writer accepts. Probe:+ `m3-work/wp96/probe_mktree2.sh`.) A malformed tree makes the process exit, so the+ request fails as today; the writer is not reused after an error.+3. **`for-each-ref` only when nothing else decides it.** `RepoCreated` is false when A or+ C resolved a branch; `HasBranches` runs only when neither did (an orphan commit,+ possibly into an empty repository). B runs only when the start branch does not+ resolve, which is where the issue-3294 workaround matters.+4. **Unchanged:** E, one `hash-object --path` per blob: the attributes of the path decide+ the bytes of the blob, and a batched writer cannot apply per-path attributes without a+ `check-attr` process, which costs as much as it saves for one or two files. K, one+ `commit-tree`: the existing `CommitWriter` costs two processes per request (its `git+ config` lookup and its writer) unless it outlives the request. M.++Expected, one created file, any depth, existing or new branch: **5 processes**+(cat-file, hash-object for the blob, hash-object for the trees, commit-tree, update-ref). n files: 4 + n.+An orphan commit into an empty repository: 6 (`for-each-ref`).++### 15.3 What stays identical++* **Objects:** the same blob bytes (same `--path`), tree bytes (the same writer and the+ same input; the differential test checks it) and commit (`commit-tree`, unchanged).+* **Durability:** every write is still a git process built by the command factory, so+ `core.fsync=objects,derived-metadata,reference` and `core.fsyncMethod=fsync` apply to+ the tree writer exactly as today. No object is written by Go, and+ `TestQuarantineWritersKeepTheDurabilityContract` has no new place. The temporary files+ are the writer's input, not objects.+* **Errors:** the same codes and messages for a missing branch, a missing start branch, an+ existing file, a missing file to update, delete or move, and directory conflicts; the+ existing `UserCommitFiles` tests run unchanged.+* **Two answers decided differently, in one corner.** With the start branch in the same+ repository, it is now resolved before `HasBranches`, which runs only if it does not+ resolve; and `RepoCreated` is false without `for-each-ref` when the target or the start+ branch resolved. Both rely on "a resolved `refs/heads/<x>` is a branch". `get_oid` also+ tries `refs/refs/heads/<x>`, `refs/tags/refs/heads/<x>` and `refs/remotes/refs/heads/<x>`+ when `refs/heads/<x>` is absent, so a repository with no branch but such a ref now+ starts from that ref, where today it starts an orphan commit (probe:+ `m3-work/wp96/probe_dwim.sh`; `rev-parse --verify` and `cat-file` resolve it alike). Today the same rules+ already apply to the target branch and, in a repository with branches, to the start+ branch.+* **Kept as today, found by the differential test:** a file created below an existing+ file (`README/x`) walks into the blob entry, which loses its ID; the tree is then+ malformed and both writers refuse it (`writing tree: exit status 128`, an internal error+ rather than an index error). WP96 keeps this; making it an index error is a separate+ change.+* **The object reader goes back to the catfile cache** at the end of the request, on every+ path (a deferred release), as today's `ReadTree` reader does. Its cache key holds the+ quarantine directory, so no other request reuses it, and the cache's TTL ends it.+* **Git features:** `hash-object --stdin-paths` (long-standing, already used by+ `CommitWriter`) and `cat-file --batch-command` (2.36, already used by the object+ reader). The server already needs newer git+ (reftable, `merge-tree --stdin`).++### 15.4 Proof++* `TestUserCommitFilesProcessCount` asserts the processes by subcommand: 5 for the four+ cases above, 7 for three files (4 + n), 6 for the orphan commit into an empty repository.+ A process that comes back fails it.+* `TestApplyActionLazyTreeMatchesRecursiveTree` applies seeded random sequences of the six+ actions to the whole tree with today's writer and to the lazy tree with the tree writer.+ It asserts the same error for every action, the same tree ID or the same refusal for+ every sequence, and that every action type succeeds and every index error occurs below+ the top level. A failure names the seed (`-lazy-tree-seed` replays it). It fails when+ the lazy read is removed from the walk.+* `TestTreeWriter_matchesWriteEntries` covers edge names, git's directory order, every+ mode, missing objects (accepted), and duplicate names, file/directory clashes, invalid+ modes and empty IDs (refused with today's error). Writes after a refusal fail, and one+ process writes every tree. `TestReadTreeLazily_matchesReadTree` checks that the walked+ lazy tree equals `ReadTree`. `TestTreeWriter_quarantine` checks objects that exist only+ in the quarantine.+* `TestUserCommitFilesReleasesProcessesAndFiles` runs failing requests (index error,+ refused tree, missing start branch) and a successful one. After each, evicting the cache+ leaves no cat-file process and the storage's temporary directory holds no tree object+ file. It fails without the deferred release, or without the file removal.+* The existing operations tests.+* `UserCommitFiles` latency at 50 agents (graft-bench, as in 2.2), before and after,+ alternated, in a window the PM grants.++### 15.5 Ownership needed++`internal/gitaly/service/operations/commit_files.go`; `internal/git/localrepo/tree.go` (the+lazy tree and the tree writer) and `refs.go` if a reader-based resolve helper lives+there; tests in those packages and `gittest/counting_command_factory.go` (`Counts`).diff --git a/internal/git/gittest/counting_command_factory.go b/internal/git/gittest/counting_command_factory.goindex 607036a..ae55f92 100644--- a/internal/git/gittest/counting_command_factory.go+++ b/internal/git/gittest/counting_command_factory.go@@ -41,6 +41,18 @@ func (f *CountingCommandFactory) CommandCount(cmd string) uint64 {return c}+// Counts returns a copy of the count of every git command.+func (f *CountingCommandFactory) Counts() map[string]uint64 {+ f.m.Lock()+ defer f.m.Unlock()++ counts := make(map[string]uint64, len(f.counts))+ for cmd, c := range f.counts {+ counts[cmd] = c+ }+ return counts+}+// RequireCommandCount checks if the count for the given command equals the// expected count.func (f *CountingCommandFactory) RequireCommandCount(tb testing.TB, cmd string, expected interface{}) {diff --git a/internal/git/localrepo/refs.go b/internal/git/localrepo/refs.goindex f4205cd..5e5ad92 100644--- a/internal/git/localrepo/refs.go+++ b/internal/git/localrepo/refs.go@@ -11,6 +11,7 @@ import ("github.com/feder-cr/git-ai/internal/command""github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/catfile""github.com/feder-cr/git-ai/internal/git/gitcmd""github.com/feder-cr/git-ai/internal/git/updateref""github.com/feder-cr/git-ai/internal/gitaly/transaction"@@ -73,6 +74,45 @@ func (repo *Repo) ResolveRevision(ctx context.Context, revision git.Revision) (greturn oid, nil}+// RequestObjectReader returns the object reader of a request that resolves revisions and reads+// objects through one `git cat-file --batch-command` process: one process for the whole request,+// where ResolveRevision starts one per revision (PLAN9, WP96). The returned function must be+// called on every path once the request is done with the reader.+func (repo *Repo) RequestObjectReader(ctx context.Context) (catfile.ObjectReader, func(), error) {+ contentReader, cancel, err := repo.catfileCache.ObjectReader(ctx, repo)+ if err != nil {+ return nil, nil, fmt.Errorf("creating object reader: %w", err)+ }++ reader, ok := contentReader.(catfile.ObjectReader)+ if !ok {+ cancel()+ return nil, nil, fmt.Errorf("object reader %T cannot read object info", contentReader)+ }++ return reader, cancel, nil+}++// ResolveRevisionWithReader is ResolveRevision through reader, the request's object reader: the+// same object ID, or git.ErrReferenceNotFound when the revision does not resolve. Unlike+// ResolveRevision it also requires the object to exist, which a revision peeled to an object type+// (`<rev>^{commit}`) requires anyway: pass only such revisions where ResolveRevision is replaced.+func (repo *Repo) ResolveRevisionWithReader(ctx context.Context, reader catfile.ObjectInfoReader, revision git.Revision) (git.ObjectID, error) {+ if revision.String() == "" {+ return "", errors.New("repository cannot contain empty reference name")+ }++ info, err := reader.Info(ctx, revision)+ if err != nil {+ if errors.As(err, &catfile.NotFoundError{}) {+ return "", git.ErrReferenceNotFound+ }+ return "", err+ }++ return info.Oid, nil+}+// GetReference looks up and returns the given reference. Returns a// ReferenceNotFound error if the reference was not found.func (repo *Repo) GetReference(ctx context.Context, reference git.ReferenceName) (git.Reference, error) {diff --git a/internal/git/localrepo/tree.go b/internal/git/localrepo/tree.goindex 1dd3ecb..c068907 100644--- a/internal/git/localrepo/tree.go+++ b/internal/git/localrepo/tree.go@@ -1,15 +1,20 @@package localrepoimport (+ "bufio""bytes""context"+ "encoding/hex""errors""fmt""io"+ "os""path/filepath""sort"+ "strconv""strings"+ "github.com/feder-cr/git-ai/internal/command""github.com/feder-cr/git-ai/internal/git""github.com/feder-cr/git-ai/internal/git/catfile""github.com/feder-cr/git-ai/internal/git/gitcmd"@@ -72,6 +77,94 @@ type TreeEntry struct {Type ObjectType// Entries is a slice of this tree's entries.Entries []*TreeEntry++ // loader, set on a tree that ReadTreeLazily read without its entries, reads them when an+ // operation first descends into the tree. Nil once they are read, and for every tree of+ // ReadTree.+ loader *treeLoader+}++// treeLoader reads the entries of the trees of a tree read by ReadTreeLazily through the object+// reader of the request (PLAN9, WP96): only the trees that the operations descend into are read.+type treeLoader struct {+ ctx context.Context+ reader catfile.ObjectContentReader+ objectHash git.ObjectHash+}++// ensureLoaded reads the entries of t if it is a tree of ReadTreeLazily whose entries were not read+// yet. It does nothing for any other entry.+func (t *TreeEntry) ensureLoaded() error {+ if t.loader == nil {+ return nil+ }++ entries, err := t.loader.read(t.OID)+ if err != nil {+ return err+ }+ t.Entries = entries+ t.loader = nil+ return nil+}++// read returns the entries of the tree oid as ls-tree describes them: a six-digit mode, the type+// derived from the mode and the base name. Its subtrees are left to be read on demand.+func (l *treeLoader) read(oid git.ObjectID) ([]*TreeEntry, error) {+ object, err := l.reader.Object(l.ctx, oid.Revision())+ if err != nil {+ return nil, fmt.Errorf("reading tree %s: %w", oid, err)+ }+ // The object must be consumed fully before the reader serves another request.+ data, err := io.ReadAll(object)+ if err != nil {+ return nil, fmt.Errorf("reading tree %s: %w", oid, err)+ }+ if object.Type != "tree" {+ return nil, fmt.Errorf("reading tree %s: %w", oid, ErrNotTreeish)+ }++ rawSize := l.objectHash.EncodedLen() / 2+ var entries []*TreeEntry+ for len(data) > 0 {+ mode, rest, ok := bytes.Cut(data, []byte{' '})+ if !ok {+ return nil, fmt.Errorf("parsing tree %s: no mode", oid)+ }+ name, rest, ok := bytes.Cut(rest, []byte{0})+ if !ok || len(rest) < rawSize {+ return nil, fmt.Errorf("parsing tree %s: truncated entry", oid)+ }+ entryOID, err := l.objectHash.FromHex(hex.EncodeToString(rest[:rawSize]))+ if err != nil {+ return nil, fmt.Errorf("parsing tree %s: %w", oid, err)+ }+ data = rest[rawSize:]++ // As ls-tree prints it: the octal value with six digits, and the type from its format+ // bits.+ modeValue, err := strconv.ParseUint(string(mode), 8, 32)+ if err != nil {+ return nil, fmt.Errorf("parsing tree %s: invalid mode %q", oid, mode)+ }+ entry := &TreeEntry{+ OID: entryOID,+ Mode: fmt.Sprintf("%06o", modeValue),+ Path: string(name),+ }+ switch modeValue & 0o170000 {+ case 0o040000:+ entry.Type = Tree+ entry.loader = l+ case 0o160000:+ entry.Type = Submodule+ default:+ entry.Type = Blob+ }+ entries = append(entries, entry)+ }++ return entries, nil}// IsBlob returns whether or not the TreeEntry is a blob.@@ -179,6 +272,11 @@ func (t *TreeEntry) Add(Loop:for {+ // A tree of ReadTreeLazily is read when the walk first reaches it.+ if err := currentTree.ensureLoaded(); err != nil {+ return err+ }+// loop through each layer of the tree based on the directory// structure specified by path.// firstComponent is the current directory name, and secondComponent is the rest@@ -259,6 +357,11 @@ func (t *TreeEntry) recurse(currentTree := tfor {+ // A tree of ReadTreeLazily is read when the walk first reaches it.+ if err := currentTree.ensureLoaded(); err != nil {+ return err+ }+// Loop through each layer of the tree based on the directory// structure specified by path.// firstComponent is the current directory name, and secondComponent is the rest@@ -538,6 +641,9 @@ func (t *TreeEntry) walk(dirPath string, callback func(path string, entry *TreeEreturn err}+ if err := t.ensureLoaded(); err != nil {+ return err+ }for _, entry := range t.Entries {if err := entry.walk(nextDirPath, callback); err != nil {return err@@ -550,6 +656,9 @@ func (t *TreeEntry) walk(dirPath string, callback func(path string, entry *TreeE// Walk will walk the whole tree structure and call callback on every entry of// the tree in a depth-first like fashion.func (t *TreeEntry) Walk(callback func(path string, entry *TreeEntry) error) error {+ if err := t.ensureLoaded(); err != nil {+ return err+ }for _, e := range t.Entries {if err := e.walk(t.Path, callback); err != nil {return err@@ -588,9 +697,8 @@ func (t *TreeEntry) Write(return nil}-// writeTreeEntries writes a new tree object from a slice of entries. This function does not verify whether OIDs-// referred to by tree entries actually exist in the repository.-func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.ObjectID, error) {+// treeContent returns the content of the tree object of entries, which it sorts in git's order.+func treeContent(entries []*TreeEntry) (*bytes.Buffer, error) {var tree bytes.Buffersort.Stable(TreeEntriesByPath(entries))@@ -601,18 +709,29 @@ func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.OoidBytes, err := entry.OID.Bytes()if err != nil {- return "", err+ return nil, err}if _, err := tree.WriteString(formattedEntry); err != nil {- return "", err+ return nil, err}if _, err := tree.Write(oidBytes); err != nil {- return "", err+ return nil, err}}+ return &tree, nil+}++// writeTreeEntries writes a new tree object from a slice of entries. This function does not verify whether OIDs+// referred to by tree entries actually exist in the repository.+func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.ObjectID, error) {+ tree, err := treeContent(entries)+ if err != nil {+ return "", err+ }+options := []gitcmd.Option{gitcmd.ValueFlag{Name: "-t", Value: "tree"},gitcmd.Flag{Name: "-w"},@@ -627,7 +746,7 @@ func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.O},gitcmd.WithStdout(&stdout),gitcmd.WithStderr(&stderr),- gitcmd.WithStdin(&tree),+ gitcmd.WithStdin(tree),); err != nil {return "", structerr.New("%w", err).WithMetadata("stderr", stderr.String())}@@ -737,3 +856,223 @@ func (repo *Repo) ReadTree(ctx context.Context, treeish git.Revision, options ..return rootEntry, nil}++// ReadTreeLazily returns the tree treeOID with its direct entries read through reader, the object+// reader of the request. Its subtrees are read only when an operation (Add, Delete, Get, Modify,+// Walk) descends into them, through the same reader, so a request that changes a few paths reads+// a few trees instead of the whole tree (PLAN9, WP96). The entries are those ReadTree with+// WithRecursive gives, for every tree that is read. reader must stay open while the tree is used.+func (repo *Repo) ReadTreeLazily(ctx context.Context, reader catfile.ObjectReader, treeOID git.ObjectID) (*TreeEntry, error) {+ objectHash, err := repo.ObjectHash(ctx)+ if err != nil {+ return nil, fmt.Errorf("detecting object hash: %w", err)+ }++ obj, err := reader.Info(ctx, treeOID.Revision())+ if err != nil {+ return nil, fmt.Errorf("check object type: %w", err)+ }+ if obj.Type != "tree" {+ return nil, ErrNotTreeish+ }++ root := &TreeEntry{+ OID: treeOID,+ Type: Tree,+ Mode: "040000",+ loader: &treeLoader{ctx: ctx, reader: reader, objectHash: objectHash},+ }+ if err := root.ensureLoaded(); err != nil {+ return nil, err+ }++ return root, nil+}++// TreeWriter writes the tree objects of one request through a single `git hash-object -t tree -w+// --stdin-paths` process, started on the first write: today's tree writer, with the same input and+// the same format checks, without a process per tree (PLAN9, WP96). The content of each tree is+// written to a file of the storage's temporary directory (created exclusively, mode 0600, removed+// after the write, whatever happened), whose path the process reads. A write that fails stops the+// process and makes every later write fail: a tree the process refused ends the request, as+// today. Close must be called on every path.+type TreeWriter struct {+ repo *Repo+ ctx context.Context+ tempDir string+ objectHash git.ObjectHash++ cmd *command.Command+ stdout *bufio.Reader+ stderr bytes.Buffer+ cancel context.CancelFunc+ err error+}++var errTreeWriterClosed = errors.New("tree writer is closed")++// NewTreeWriter returns the tree writer of a request on repo. Its process runs under ctx.+func (repo *Repo) NewTreeWriter(ctx context.Context) (*TreeWriter, error) {+ tempDir, err := repo.StorageTempDir()+ if err != nil {+ return nil, fmt.Errorf("getting temporary directory: %w", err)+ }+ objectHash, err := repo.ObjectHash(ctx)+ if err != nil {+ return nil, fmt.Errorf("detecting object hash: %w", err)+ }++ return &TreeWriter{repo: repo, ctx: ctx, tempDir: tempDir, objectHash: objectHash}, nil+}++// write writes the tree of entries and returns its object ID.+func (w *TreeWriter) write(entries []*TreeEntry) (_ git.ObjectID, returnedErr error) {+ if w.err != nil {+ return "", w.err+ }+ defer func() {+ if returnedErr != nil {+ w.err = returnedErr+ w.stop()+ }+ }()++ content, err := treeContent(entries)+ if err != nil {+ return "", err+ }++ // os.CreateTemp opens the file with O_CREATE|O_EXCL and mode 0600.+ file, err := os.CreateTemp(w.tempDir, "tree-object-*")+ if err != nil {+ return "", fmt.Errorf("creating tree object file: %w", err)+ }+ defer func() {+ if err := os.Remove(file.Name()); err != nil && !errors.Is(err, os.ErrNotExist) && returnedErr == nil {+ returnedErr = fmt.Errorf("removing tree object file: %w", err)+ }+ }()+ if _, err := file.Write(content.Bytes()); err != nil {+ _ = file.Close()+ return "", fmt.Errorf("writing tree object file: %w", err)+ }+ if err := file.Close(); err != nil {+ return "", fmt.Errorf("closing tree object file: %w", err)+ }++ if w.cmd == nil {+ if err := w.start(); err != nil {+ return "", err+ }+ }++ if _, err := fmt.Fprintf(w.cmd.Stdin(), "%s\n", file.Name()); err != nil {+ return "", w.processError(fmt.Errorf("sending tree object path: %w", err))+ }+ line, err := w.stdout.ReadString('\n')+ if err != nil {+ return "", w.processError(fmt.Errorf("reading tree object ID: %w", err))+ }++ oid, err := w.objectHash.FromHex(strings.TrimSuffix(line, "\n"))+ if err != nil {+ return "", fmt.Errorf("parsing tree object ID: %w", err)+ }+ return oid, nil+}++// start starts the hash-object process.+func (w *TreeWriter) start() error {+ processCtx, cancel := context.WithCancel(w.ctx)+ cmd, err := w.repo.Exec(processCtx,+ gitcmd.Command{+ Name: "hash-object",+ Flags: []gitcmd.Option{+ gitcmd.ValueFlag{Name: "-t", Value: "tree"},+ gitcmd.Flag{Name: "-w"},+ gitcmd.Flag{Name: "--stdin-paths"},+ },+ },+ gitcmd.WithSetupStdin(),+ gitcmd.WithSetupStdout(),+ gitcmd.WithStderr(&w.stderr),+ )+ if err != nil {+ cancel()+ return fmt.Errorf("starting hash-object: %w", err)+ }++ w.cmd = cmd+ w.stdout = bufio.NewReader(cmd)+ w.cancel = cancel+ return nil+}++// processError reaps the process, which stopped answering, and returns why: its exit status with+// what it printed on stderr, as today's writer reports a refused tree, or err if it exited+// cleanly. Wait closes its stdin first.+func (w *TreeWriter) processError(err error) error {+ waitErr := w.cmd.Wait()+ w.cmd = nil+ w.cancel()+ if waitErr != nil {+ err = waitErr+ }+ return structerr.New("%w", err).WithMetadata("stderr", w.stderr.String())+}++// stop kills and reaps the process, if any.+func (w *TreeWriter) stop() {+ if w.cmd == nil {+ return+ }+ w.cancel()+ // The process was killed, so its exit status carries no information.+ _ = w.cmd.Wait()+ w.cmd = nil+}++// Close ends the process: it reads the end of its input and exits, so every tree it wrote is in the+// object directory when Close returns. Close is idempotent, later calls return nil, and it is safe+// after a failed write, whose process is already reaped. Writes fail after Close.+func (w *TreeWriter) Close() error {+ if w.err == nil {+ w.err = errTreeWriterClosed+ }+ if w.cmd == nil {+ return nil+ }+ defer w.cancel()++ cmd := w.cmd+ w.cmd = nil+ // Wait closes the process's stdin first.+ if err := cmd.Wait(); err != nil {+ return structerr.New("waiting for tree writer: %w", err).WithMetadata("stderr", w.stderr.String())+ }+ return nil+}++// WriteWith is Write through the request's tree writer: the same trees, written depth first when+// their OID is empty, by one process.+func (t *TreeEntry) WriteWith(w *TreeWriter) error {+ if t.OID != "" {+ return nil+ }++ for _, e := range t.Entries {+ if e.Type == Tree && e.OID == "" {+ if err := e.WriteWith(w); err != nil {+ return err+ }+ }+ }++ treeOID, err := w.write(t.Entries)+ if err != nil {+ return fmt.Errorf("writing tree: %w", err)+ }++ t.OID = treeOID+ return nil+}diff --git a/internal/git/localrepo/tree_writer_test.go b/internal/git/localrepo/tree_writer_test.gonew file mode 100644index 0000000..f253042--- /dev/null+++ b/internal/git/localrepo/tree_writer_test.go@@ -0,0 +1,306 @@+package localrepo++import (+ "context"+ "os"+ "strings"+ "testing"++ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/git/quarantine"+ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+)++func setupTreeWriterRepo(t *testing.T) (context.Context, config.Cfg, *Repo, string) {+ t.Helper()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ return ctx, cfg, NewTestRepo(t, cfg, repoProto), repoPath+}++// requireNoTreeObjectFiles asserts that no temporary tree object file is left in the storage's+// temporary directory.+func requireNoTreeObjectFiles(t *testing.T, repo *Repo) {+ t.Helper()++ tempDir, err := repo.StorageTempDir()+ require.NoError(t, err)+ entries, err := os.ReadDir(tempDir)+ require.NoError(t, err)+ for _, entry := range entries {+ require.NotContains(t, entry.Name(), "tree-object-")+ }+}++func cloneEntries(entries []*TreeEntry) []*TreeEntry {+ cloned := make([]*TreeEntry, 0, len(entries))+ for _, entry := range entries {+ entryCopy := *entry+ cloned = append(cloned, &entryCopy)+ }+ return cloned+}++// TestTreeWriter_matchesWriteEntries checks that the tree writer writes the trees today's writer+// writes, with the same IDs, through one process for every tree, and refuses the trees it refuses,+// with the same error (PLAN9, WP96).+func TestTreeWriter_matchesWriteEntries(t *testing.T) {+ t.Parallel()++ ctx, cfg, repo, repoPath := setupTreeWriterRepo(t)++ blob := gittest.WriteBlob(t, cfg, repoPath, []byte("one\n"))+ otherBlob := gittest.WriteBlob(t, cfg, repoPath, []byte("two\n"))+ subtree := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{{Path: "x", Mode: "100644", OID: blob}})+ commit := gittest.WriteCommit(t, cfg, repoPath)+ // An object ID that no object of the repository has.+ missing := gittest.DefaultObjectHash.HashData([]byte("missing object"))++ for _, tc := range []struct {+ desc string+ entries []*TreeEntry+ refused bool+ }{+ {+ desc: "empty",+ },+ {+ desc: "edge names",+ entries: []*TreeEntry{+ {Path: "a", Mode: "100644", Type: Blob, OID: blob},+ {Path: "a.b", Mode: "100644", Type: Blob, OID: otherBlob},+ {Path: "a b", Mode: "100644", Type: Blob, OID: blob},+ {Path: "é té", Mode: "100644", Type: Blob, OID: blob},+ {Path: "-", Mode: "100644", Type: Blob, OID: blob},+ },+ },+ {+ desc: "a directory sorts as its name with a slash",+ entries: []*TreeEntry{+ {Path: "a0", Mode: "100644", Type: Blob, OID: blob},+ {Path: "a", Mode: "040000", Type: Tree, OID: subtree},+ {Path: "a.b", Mode: "100644", Type: Blob, OID: blob},+ {Path: "a-", Mode: "100644", Type: Blob, OID: blob},+ },+ },+ {+ desc: "every mode",+ entries: []*TreeEntry{+ {Path: "regular", Mode: "100644", Type: Blob, OID: blob},+ {Path: "executable", Mode: "100755", Type: Blob, OID: blob},+ {Path: "symlink", Mode: "120000", Type: Blob, OID: otherBlob},+ {Path: "directory", Mode: "040000", Type: Tree, OID: subtree},+ {Path: "submodule", Mode: "160000", Type: Submodule, OID: commit},+ },+ },+ {+ desc: "an object that does not exist is accepted",+ entries: []*TreeEntry{+ {Path: "missing", Mode: "100644", Type: Blob, OID: missing},+ {Path: "missing-submodule", Mode: "160000", Type: Submodule, OID: missing},+ },+ },+ {+ desc: "duplicate names",+ entries: []*TreeEntry{+ {Path: "d", Mode: "100644", Type: Blob, OID: blob},+ {Path: "d", Mode: "100644", Type: Blob, OID: otherBlob},+ },+ refused: true,+ },+ {+ desc: "a file and a directory of the same name",+ entries: []*TreeEntry{+ {Path: "d", Mode: "100644", Type: Blob, OID: blob},+ {Path: "d", Mode: "040000", Type: Tree, OID: subtree},+ },+ refused: true,+ },+ {+ desc: "invalid mode",+ entries: []*TreeEntry{{Path: "bad", Mode: "100600", Type: Blob, OID: blob}},+ refused: true,+ },+ {+ desc: "an entry without object ID",+ entries: []*TreeEntry{{Path: "empty", Mode: "100644", Type: Blob}},+ refused: true,+ },+ } {+ t.Run(tc.desc, func(t *testing.T) {+ expectedOID, expectedErr := repo.writeEntries(ctx, cloneEntries(tc.entries))+ require.Equal(t, tc.refused, expectedErr != nil, "today's writer: %v", expectedErr)++ writer, err := repo.NewTreeWriter(ctx)+ require.NoError(t, err)++ oid, err := writer.write(cloneEntries(tc.entries))+ if tc.refused {+ require.Error(t, err)+ require.Equal(t, expectedErr.Error(), err.Error())++ // The process exited and was reaped; the writer refuses every later write.+ require.Nil(t, writer.cmd)+ _, laterErr := writer.write(cloneEntries(tc.entries[:1]))+ require.Equal(t, err, laterErr)+ } else {+ require.NoError(t, err)+ require.Equal(t, expectedOID, oid)++ // The same process writes the next tree.+ process := writer.cmd+ oid, err = writer.write(cloneEntries(tc.entries))+ require.NoError(t, err)+ require.Equal(t, expectedOID, oid)+ require.Same(t, process, writer.cmd)+ }++ require.NoError(t, writer.Close())+ require.Nil(t, writer.cmd)+ require.NoError(t, writer.Close())+ _, err = writer.write(cloneEntries(tc.entries))+ require.Error(t, err)++ requireNoTreeObjectFiles(t, repo)+ })+ }+}++// TestTreeEntry_WriteWith checks that WriteWith writes the trees Write writes.+func TestTreeEntry_WriteWith(t *testing.T) {+ t.Parallel()++ ctx, cfg, repo, repoPath := setupTreeWriterRepo(t)+ blob := gittest.WriteBlob(t, cfg, repoPath, []byte("content\n"))++ build := func() *TreeEntry {+ root := &TreeEntry{Mode: "040000", Type: Tree}+ for _, path := range []string{"a/b/c", "a/b.c", "a.b/c", "d", "a/é/f"} {+ require.NoError(t, root.Add(path, TreeEntry{Path: path[strings.LastIndex(path, "/")+1:], Mode: "100644", Type: Blob, OID: blob}))+ }+ return root+ }++ expected := build()+ require.NoError(t, expected.Write(ctx, repo))++ actual := build()+ writer, err := repo.NewTreeWriter(ctx)+ require.NoError(t, err)+ require.NoError(t, actual.WriteWith(writer))+ require.NoError(t, writer.Close())++ require.Equal(t, expected, actual)+ requireNoTreeObjectFiles(t, repo)+}++// TestReadTreeLazily_matchesReadTree checks that the lazy tree, once walked, is the tree ReadTree+// reads whole, and that it refuses what ReadTree refuses.+func TestReadTreeLazily_matchesReadTree(t *testing.T) {+ t.Parallel()++ ctx, cfg, repo, repoPath := setupTreeWriterRepo(t)++ blob := gittest.WriteBlob(t, cfg, repoPath, []byte("content\n"))+ commit := gittest.WriteCommit(t, cfg, repoPath)+ inner := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{+ {Path: "file", Mode: "100644", OID: blob},+ {Path: "é", Mode: "100755", OID: blob},+ })+ middle := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{+ {Path: "inner", Mode: "040000", OID: inner},+ {Path: "link", Mode: "120000", OID: blob},+ })+ root := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{+ {Path: "a", Mode: "040000", OID: middle},+ {Path: "a.b", Mode: "100644", OID: blob},+ {Path: "a b", Mode: "040000", OID: inner},+ {Path: "submodule", Mode: "160000", OID: commit},+ })++ expected, err := repo.ReadTree(ctx, root.Revision(), WithRecursive())+ require.NoError(t, err)++ reader, cancel, err := repo.RequestObjectReader(ctx)+ require.NoError(t, err)+ defer cancel()++ lazy, err := repo.ReadTreeLazily(ctx, reader, root)+ require.NoError(t, err)+ // Only the root is read.+ for _, entry := range lazy.Entries {+ require.Nil(t, entry.Entries)+ }++ require.NoError(t, lazy.Walk(func(string, *TreeEntry) error { return nil }))+ require.Equal(t, expected, lazy)++ _, err = repo.ReadTreeLazily(ctx, reader, blob)+ require.Equal(t, ErrNotTreeish, err)++ missing := gittest.DefaultObjectHash.HashData([]byte("missing tree"))+ _, err = repo.ReadTreeLazily(ctx, reader, missing)+ require.ErrorContains(t, err, "check object type: object not found")+}++// TestTreeWriter_quarantine checks that the request's reader and tree writer run on the+// quarantined repository: they write into the quarantine and read what only it holds.+func TestTreeWriter_quarantine(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, _ := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ repo := NewTestRepo(t, cfg, repoProto)++ quarantineDir, cleanup, err := quarantine.New(ctx, repoProto, testhelper.NewLogger(t), config.NewLocator(cfg))+ require.NoError(t, err)+ t.Cleanup(cleanup)+ quarantineRepo := NewTestRepo(t, cfg, quarantineDir.QuarantinedRepo())++ blob, err := quarantineRepo.WriteBlob(ctx, strings.NewReader("quarantined\n"), WriteBlobConfig{})+ require.NoError(t, err)++ tree := &TreeEntry{Mode: "040000", Type: Tree}+ require.NoError(t, tree.Add("dir/file", TreeEntry{Path: "file", Mode: "100644", Type: Blob, OID: blob}))+ writer, err := quarantineRepo.NewTreeWriter(ctx)+ require.NoError(t, err)+ require.NoError(t, tree.WriteWith(writer))+ require.NoError(t, writer.Close())++ // The trees are in the quarantine only.+ for _, oid := range []git.ObjectID{tree.OID, tree.Entries[0].OID} {+ _, err := quarantineRepo.ReadObject(ctx, oid)+ require.NoError(t, err)+ _, err = repo.ReadObject(ctx, oid)+ require.Equal(t, InvalidObjectError(oid.String()), err)+ }++ reader, cancel, err := quarantineRepo.RequestObjectReader(ctx)+ require.NoError(t, err)+ defer cancel()++ resolved, err := quarantineRepo.ResolveRevisionWithReader(ctx, reader, tree.OID.Revision()+"^{tree}")+ require.NoError(t, err)+ require.Equal(t, tree.OID, resolved)++ lazy, err := quarantineRepo.ReadTreeLazily(ctx, reader, tree.OID)+ require.NoError(t, err)+ entry, err := lazy.Get("dir/file")+ require.NoError(t, err)+ require.Equal(t, blob, entry.OID)++ mainReader, cancelMain, err := repo.RequestObjectReader(ctx)+ require.NoError(t, err)+ defer cancelMain()++ _, err = repo.ResolveRevisionWithReader(ctx, mainReader, tree.OID.Revision()+"^{tree}")+ require.Equal(t, git.ErrReferenceNotFound, err)+}diff --git a/internal/gitaly/service/operations/commit_files.go b/internal/gitaly/service/operations/commit_files.goindex 22c0e62..9ca0175 100644--- a/internal/gitaly/service/operations/commit_files.go+++ b/internal/gitaly/service/operations/commit_files.go@@ -11,7 +11,7 @@ import ("strings""github.com/feder-cr/git-ai/internal/git"- "github.com/feder-cr/git-ai/internal/git/gitcmd"+ "github.com/feder-cr/git-ai/internal/git/catfile""github.com/feder-cr/git-ai/internal/git/localrepo""github.com/feder-cr/git-ai/internal/git/remoterepo""github.com/feder-cr/git-ai/internal/gitaly/hook"@@ -439,6 +439,7 @@ func (s *Server) userCommitFilesGit(header *gitalypb.UserCommitFilesRequestHeader,parentCommitOID git.ObjectID,quarantineRepo *localrepo.Repo,+ reader catfile.ObjectReader,repoPath string,actions []commitAction,) (git.ObjectID, error) {@@ -450,8 +451,9 @@ func (s *Server) userCommitFilesGit(var treeish git.ObjectIDif parentCommitOID != "" {- treeish, err = quarantineRepo.ResolveRevision(+ treeish, err = quarantineRepo.ResolveRevisionWithReader(ctx,+ reader,git.Revision(fmt.Sprintf("%s^{tree}", parentCommitOID)),)if err != nil {@@ -465,11 +467,8 @@ func (s *Server) userCommitFilesGit(}if treeish != "" {- treeEntry, err = quarantineRepo.ReadTree(- ctx,- git.Revision(treeish),- localrepo.WithRecursive(),- )+ // Only the trees the actions descend into are read.+ treeEntry, err = quarantineRepo.ReadTreeLazily(ctx, reader, treeish)if err != nil {return "", fmt.Errorf("reading tree: %w", err)}@@ -486,10 +485,7 @@ func (s *Server) userCommitFilesGit(}}- if err := treeEntry.Write(- ctx,- quarantineRepo,- ); err != nil {+ if err := writeTreeEntries(ctx, quarantineRepo, treeEntry); err != nil {return "", fmt.Errorf("writing tree %w", err)}@@ -539,6 +535,22 @@ func (s *Server) userCommitFilesGit()}+// writeTreeEntries writes the trees of tree that changed through one tree writer, which it closes+// on every path: once it returns, every tree it wrote is in the object directory.+func writeTreeEntries(ctx context.Context, repo *localrepo.Repo, tree *localrepo.TreeEntry) (returnedErr error) {+ treeWriter, err := repo.NewTreeWriter(ctx)+ if err != nil {+ return err+ }+ defer func() {+ if err := treeWriter.Close(); err != nil && returnedErr == nil {+ returnedErr = err+ }+ }()++ return tree.WriteWith(treeWriter)+}+func (s *Server) userCommitFiles(ctx context.Context,header *gitalypb.UserCommitFilesRequestHeader,@@ -565,8 +577,16 @@ func (s *Server) userCommitFiles(remoteRepo = nil}+ // One object reader serves every revision and tree the request reads (PLAN9, WP96). It reads+ // the quarantined repository, so it sees the objects the request writes.+ reader, cancelReader, err := quarantineRepo.RequestObjectReader(ctx)+ if err != nil {+ return err+ }+ defer cancelReader()+targetBranchName := git.NewReferenceNameFromBranchName(string(header.GetBranchName()))- targetBranchCommit, err := quarantineRepo.ResolveRevision(ctx, targetBranchName.Revision()+"^{commit}")+ targetBranchCommit, err := quarantineRepo.ResolveRevisionWithReader(ctx, reader, targetBranchName.Revision()+"^{commit}")if err != nil {if !errors.Is(err, git.ErrReferenceNotFound) {return fmt.Errorf("resolve target branch commit: %w", err)@@ -576,10 +596,14 @@ func (s *Server) userCommitFiles(}var parentCommitOID git.ObjectID+ // startBranchResolved is whether the start branch resolved in the repository, which then has+ // a branch.+ var startBranchResolved boolif header.GetStartSha() == "" {- parentCommitOID, err = s.resolveParentCommit(+ parentCommitOID, startBranchResolved, err = s.resolveParentCommit(ctx,quarantineRepo,+ reader,remoteRepo,targetBranchName,targetBranchCommit,@@ -596,7 +620,7 @@ func (s *Server) userCommitFiles(}if parentCommitOID != targetBranchCommit {- if err := s.fetchMissingCommit(ctx, quarantineRepo, remoteRepo, parentCommitOID); err != nil {+ if err := s.fetchMissingCommit(ctx, quarantineRepo, reader, remoteRepo, parentCommitOID); err != nil {return fmt.Errorf("fetch missing commit: %w", err)}}@@ -718,6 +742,7 @@ func (s *Server) userCommitFiles(header,parentCommitOID,quarantineRepo,+ reader,repoPath,actions,)@@ -729,9 +754,13 @@ func (s *Server) userCommitFiles(return err}- hasBranches, err := quarantineRepo.HasBranches(ctx)- if err != nil {- return fmt.Errorf("was repo created: %w", err)+ // A branch that resolved already tells that the repository has branches.+ hasBranches := targetBranchCommit != "" || startBranchResolved+ if !hasBranches {+ hasBranches, err = quarantineRepo.HasBranches(ctx)+ if err != nil {+ return fmt.Errorf("was repo created: %w", err)+ }}var oldRevision git.ObjectID@@ -775,29 +804,52 @@ func sameRepository(repoA, repoB *gitalypb.Repository) bool {repoA.GetRelativePath() == repoB.GetRelativePath()}+// resolveParentCommit returns the parent commit of the new commit, and whether it is the start+// branch of local, which then has a branch.func (s *Server) resolveParentCommit(ctx context.Context,- local gitcmd.Repository,+ local *localrepo.Repo,+ localReader catfile.ObjectInfoReader,remote *gitalypb.Repository,targetBranch git.ReferenceName,targetBranchCommit git.ObjectID,startBranch string,-) (git.ObjectID, error) {+) (git.ObjectID, bool, error) {if remote == nil && startBranch == "" {- return targetBranchCommit, nil+ return targetBranchCommit, false, nil}- repo := local- if remote != nil {- var err error- repo, err = remoterepo.New(ctx, remote, s.conns)- if err != nil {- return "", fmt.Errorf("remote repository: %w", err)+ branch := targetBranch+ if startBranch != "" {+ branch = git.NewReferenceNameFromBranchName(startBranch)+ }+ refish := branch + "^{commit}"++ if remote == nil {+ // The start branch is resolved first, through the request's reader: a repository where+ // it resolves has branches, so the workaround below matters only when it does not.+ commit, err := local.ResolveRevisionWithReader(ctx, localReader, git.Revision(refish))+ if err == nil {+ return commit, true, nil}++ if hasBranches, hasBranchesErr := local.HasBranches(ctx); hasBranchesErr != nil {+ return "", false, fmt.Errorf("has branches: %w", hasBranchesErr)+ } else if !hasBranches {+ // The first branch is being created: see the workaround below.+ return targetBranchCommit, false, nil+ }++ return "", false, fmt.Errorf("resolving refish %q in %T: %w", refish, local, err)+ }++ repo, err := remoterepo.New(ctx, remote, s.conns)+ if err != nil {+ return "", false, fmt.Errorf("remote repository: %w", err)}if hasBranches, err := repo.HasBranches(ctx); err != nil {- return "", fmt.Errorf("has branches: %w", err)+ return "", false, fmt.Errorf("has branches: %w", err)} else if !hasBranches {// GitLab sends requests to UserCommitFiles where target repository// and start repository are the same. If the request hits Gitaly directly,@@ -817,30 +869,25 @@ func (s *Server) resolveParentCommit(// already resolved locally.//// See: https://gitlab.com/gitlab-org/gitaly/-/issues/3294- return targetBranchCommit, nil+ return targetBranchCommit, false, nil}- branch := targetBranch- if startBranch != "" {- branch = git.NewReferenceNameFromBranchName(startBranch)- }- refish := branch + "^{commit}"-commit, err := repo.ResolveRevision(ctx, git.Revision(refish))if err != nil {- return "", fmt.Errorf("resolving refish %q in %T: %w", refish, repo, err)+ return "", false, fmt.Errorf("resolving refish %q in %T: %w", refish, repo, err)}- return commit, nil+ return commit, false, nil}func (s *Server) fetchMissingCommit(ctx context.Context,localRepo *localrepo.Repo,+ localReader catfile.ObjectInfoReader,remoteRepo *gitalypb.Repository,commit git.ObjectID,) error {- if _, err := localRepo.ResolveRevision(ctx, commit.Revision()+"^{commit}"); err != nil {+ if _, err := localRepo.ResolveRevisionWithReader(ctx, localReader, commit.Revision()+"^{commit}"); err != nil {if !errors.Is(err, git.ErrReferenceNotFound) || remoteRepo == nil {return fmt.Errorf("lookup parent commit: %w", err)}diff --git a/internal/gitaly/service/operations/commit_files_cleanup_test.go b/internal/gitaly/service/operations/commit_files_cleanup_test.gonew file mode 100644index 0000000..a4e942e--- /dev/null+++ b/internal/gitaly/service/operations/commit_files_cleanup_test.go@@ -0,0 +1,112 @@+package operations++import (+ "os"+ "strings"+ "testing"++ "github.com/prometheus/client_golang/prometheus/testutil"+ "github.com/stretchr/testify/require"+ "google.golang.org/grpc"++ "github.com/feder-cr/git-ai/internal/featureflag"+ "github.com/feder-cr/git-ai/internal/git/catfile"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/gitaly/config"+ "github.com/feder-cr/git-ai/internal/gitaly/service"+ "github.com/feder-cr/git-ai/internal/gitaly/service/hook"+ "github.com/feder-cr/git-ai/internal/gitaly/service/repository"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+ "github.com/feder-cr/git-ai/internal/testhelper/testserver"+ "github.com/feder-cr/git-ai/proto/go/gitalypb"+)++// TestUserCommitFilesReleasesProcessesAndFiles checks that a UserCommitFiles request, whether it+// fails or succeeds, gives back its object reader and stops its tree writer, and leaves no+// temporary tree object file in the storage (PLAN9, WP96): once the cache evicts its processes,+// no cat-file process is left running.+func TestUserCommitFilesReleasesProcessesAndFiles(t *testing.T) {+ t.Parallel()++ ctx := featureflag.OutgoingCtxWithFeatureFlag(testhelper.Context(t), featureflag.GPGSigning, false)+ cfg := testcfg.Build(t)+ testcfg.BuildGitalySSH(t, cfg)+ testcfg.BuildGitalyHooks(t, cfg)++ var processCache *catfile.ProcessCache+ cfg.SocketPath = testserver.RunGitalyServer(t, cfg, func(srv *grpc.Server, deps *service.Dependencies) {+ processCache = deps.GetCatfileCache().(*catfile.ProcessCache)+ gitalypb.RegisterOperationServiceServer(srv, NewServer(deps))+ gitalypb.RegisterHookServiceServer(srv, hook.NewServer(deps))+ gitalypb.RegisterRepositoryServiceServer(srv, repository.NewServer(deps))+ })+ client, conn := newOperationClient(t, cfg.SocketPath)+ t.Cleanup(func() { conn.Close() })+ ctx = testhelper.MergeOutgoingMetadata(ctx, testcfg.GitalyServersMetadataFromCfg(t, cfg))++ repo, repoPath := gittest.CreateRepository(t, ctx, cfg)+ gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("master"),+ gittest.WithTreeEntries(gittest.TreeEntry{Path: "README", Mode: "100644", Content: "readme\n"}))++ tempDir, err := config.NewLocator(cfg).TempDir(cfg.Storages[0].Name)+ require.NoError(t, err)++ for _, tc := range []struct {+ desc string+ startBranch string+ requests []*gitalypb.UserCommitFilesRequest+ expectedErr string+ }{+ {+ desc: "a file to delete that does not exist",+ requests: []*gitalypb.UserCommitFilesRequest{deleteFileHeaderRequest("missing.txt")},+ expectedErr: "A file with this name doesn't exist",+ },+ {+ desc: "a tree the writer refuses",+ requests: []*gitalypb.UserCommitFilesRequest{createFileHeaderRequest("README/file"), actionContentRequest("content\n")},+ expectedErr: "writing tree writing tree: exit status 128",+ },+ {+ desc: "a start branch that does not exist",+ startBranch: "missing",+ requests: []*gitalypb.UserCommitFilesRequest{createFileHeaderRequest("file"), actionContentRequest("content\n")},+ expectedErr: "resolve parent commit",+ },+ {+ desc: "success",+ requests: []*gitalypb.UserCommitFilesRequest{createFileHeaderRequest("dir/file"), actionContentRequest("content\n")},+ },+ } {+ t.Run(tc.desc, func(t *testing.T) {+ stream, err := client.UserCommitFiles(ctx)+ require.NoError(t, err)+ require.NoError(t, stream.Send(headerRequest(repo, gittest.TestUser, "master", []byte("message"), tc.startBranch, "")))+ for _, request := range tc.requests {+ require.NoError(t, stream.Send(request))+ }+ _, err = stream.CloseAndRecv()+ if tc.expectedErr == "" {+ require.NoError(t, err)+ } else {+ require.ErrorContains(t, err, tc.expectedErr)+ }++ // Every process the request took was given back to the cache or closed: evicting+ // the cache stops them all.+ processCache.Evict()+ require.NoError(t, testutil.CollectAndCompare(processCache, strings.NewReader(`+# HELP gitaly_catfile_processes Gauge of active catfile processes+# TYPE gitaly_catfile_processes gauge+gitaly_catfile_processes 0+`), "gitaly_catfile_processes"))++ entries, err := os.ReadDir(tempDir)+ require.NoError(t, err)+ for _, entry := range entries {+ require.NotContains(t, entry.Name(), "tree-object-")+ }+ })+ }+}diff --git a/internal/gitaly/service/operations/commit_files_lazy_test.go b/internal/gitaly/service/operations/commit_files_lazy_test.gonew file mode 100644index 0000000..4e436cb--- /dev/null+++ b/internal/gitaly/service/operations/commit_files_lazy_test.go@@ -0,0 +1,202 @@+package operations++import (+ "errors"+ "flag"+ "fmt"+ "math/rand/v2"+ "path/filepath"+ "strings"+ "testing"+ "time"++ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/git"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/git/localrepo"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+)++var lazyTreeSeed = flag.Uint64("lazy-tree-seed", 0, "seed of TestApplyActionLazyTreeMatchesRecursiveTree, random when 0")++// TestApplyActionLazyTreeMatchesRecursiveTree applies random sequences of UserCommitFiles actions+// to the tree read whole (ReadTree with WithRecursive, then Write: the path before WP96) and to the+// tree of ReadTreeLazily (then the tree writer: the path after it). Every action must give the same+// error on both, and every sequence the same tree ID. The actions reach every outcome: each action+// type succeeds, and each error the whole tree detected (an existing file or directory, a move+// onto an existing file, a missing file to update, move or delete, a file in the way of a+// directory) occurs, also below the top level. A failure names the seed, which+// -lazy-tree-seed replays.+func TestApplyActionLazyTreeMatchesRecursiveTree(t *testing.T) {+ t.Parallel()++ ctx := testhelper.Context(t)+ cfg := testcfg.Build(t)+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})+ repo := localrepo.NewTestRepo(t, cfg, repoProto)++ seed := *lazyTreeSeed+ if seed == 0 {+ seed = uint64(time.Now().UnixNano())+ }+ t.Logf("seed %d (replay with -lazy-tree-seed=%d)", seed, seed)+ rng := rand.New(rand.NewPCG(seed, 0))++ blobs := gittest.WriteBlobs(t, cfg, repoPath, 4)+ submodule := gittest.WriteCommit(t, cfg, repoPath)+ randomBlob := func() git.ObjectID { return blobs[rng.IntN(len(blobs))] }++ // Few names, so that paths collide: a file where a directory is wanted, and the reverse.+ names := []string{"a", "b", "a.b", "a b", "é"}+ // basePaths are the paths of the base tree's entries, which half of the actions address.+ var basePaths []string+ randomPath := func() string {+ if len(basePaths) > 0 && rng.IntN(2) == 0 {+ return basePaths[rng.IntN(len(basePaths))]+ }+ components := make([]string, 1+rng.IntN(3))+ for i := range components {+ components[i] = names[rng.IntN(len(names))]+ }+ return strings.Join(components, "/")+ }++ randomAction := func() commitAction {+ switch rng.IntN(6) {+ case 0:+ return createFile{Path: randomPath(), OID: randomBlob().String(), ExecutableMode: rng.IntN(2) == 0}+ case 1:+ return updateFile{Path: randomPath(), OID: randomBlob().String()}+ case 2:+ action := moveFile{Path: randomPath(), NewPath: randomPath()}+ if rng.IntN(2) == 0 {+ action.OID = randomBlob().String()+ }+ return action+ case 3:+ return deleteFile{Path: randomPath()}+ case 4:+ return changeFileMode{Path: randomPath(), ExecutableMode: rng.IntN(2) == 0}+ default:+ return createDirectory{Path: randomPath()}+ }+ }++ // underFile is whether a directory of path is a file or a submodule in root: Add would then+ // add path below that entry, which no tree can hold.+ underFile := func(root *localrepo.TreeEntry, path string) bool {+ for dir := filepath.Dir(path); dir != "."; dir = filepath.Dir(dir) {+ if entry, err := root.Get(dir); err == nil && entry.Type != localrepo.Tree {+ return true+ }+ }+ return false+ }++ // writeBaseTree writes a random valid tree with regular, executable, symlink and submodule+ // entries, through today's writer.+ writeBaseTree := func() git.ObjectID {+ basePaths = nil+ root := &localrepo.TreeEntry{Mode: "040000", Type: localrepo.Tree}+ for range 4 + rng.IntN(12) {+ path := randomPath()+ if underFile(root, path) {+ continue+ }+ entry := localrepo.TreeEntry{Path: filepath.Base(path), Type: localrepo.Blob, OID: randomBlob()}+ switch rng.IntN(5) {+ case 0:+ entry.Mode = "100755"+ case 1:+ entry.Mode = "120000"+ case 2:+ entry.Mode, entry.Type, entry.OID = "160000", localrepo.Submodule, submodule+ default:+ entry.Mode = "100644"+ }+ if err := root.Add(path, entry); err != nil {+ require.ErrorIs(t, err, localrepo.ErrEntryExists)+ continue+ }+ basePaths = append(basePaths, path)+ }+ require.NoError(t, root.Write(ctx, repo))+ return root.OID+ }++ reader, cancel, err := repo.RequestObjectReader(ctx)+ require.NoError(t, err)+ defer cancel()++ outcomes := map[string]int{}+ for sequence := range 120 {+ baseOID := writeBaseTree()++ recursive, err := repo.ReadTree(ctx, baseOID.Revision(), localrepo.WithRecursive())+ require.NoError(t, err)+ lazy, err := repo.ReadTreeLazily(ctx, reader, baseOID)+ require.NoError(t, err)++ var trace []string+ for range 12 {+ action := randomAction()+ trace = append(trace, fmt.Sprintf("%T%+v", action, action))++ recursiveErr := applyAction(ctx, action, recursive, repo)+ lazyErr := applyAction(ctx, action, lazy, repo)+ require.Equal(t, recursiveErr, lazyErr, "seed %d, sequence %d, base %s, actions %v", seed, sequence, baseOID, trace)++ outcomes[outcome(action, recursiveErr)]+++ }++ // A file created below a file leaves a tree that no writer accepts: both must refuse it+ // the same way, and write the same tree otherwise.+ recursiveErr := recursive.Write(ctx, repo)+ lazyErr := writeTreeEntries(ctx, repo, lazy)+ require.Equal(t, fmt.Sprint(recursiveErr), fmt.Sprint(lazyErr), "seed %d, sequence %d, base %s, actions %v", seed, sequence, baseOID, trace)+ require.Equal(t, recursiveErr == nil, lazyErr == nil, "seed %d, sequence %d", seed, sequence)+ if recursiveErr != nil {+ outcomes["write: "+recursiveErr.Error()]+++ continue+ }+ outcomes["write ok"]+++ require.Equal(t, recursive.OID, lazy.OID, "seed %d, sequence %d, base %s, actions %v", seed, sequence, baseOID, trace)+ }++ for _, expected := range []string{+ "operations.createFile ok",+ "operations.updateFile ok",+ "operations.moveFile ok",+ "operations.deleteFile ok",+ "operations.changeFileMode ok",+ "operations.createDirectory ok",+ "operations.createFile at depth: A file with this name already exists",+ "operations.createDirectory at depth: A file with this name already exists",+ "operations.createDirectory at depth: A directory with this name already exists",+ "operations.moveFile at depth: A file with this name already exists",+ "operations.moveFile at depth: A file with this name doesn't exist",+ "operations.updateFile at depth: A file with this name doesn't exist",+ "operations.deleteFile at depth: A file with this name doesn't exist",+ "operations.changeFileMode at depth: A file with this name doesn't exist",+ "write ok",+ "write: writing tree: exit status 128",+ } {+ require.Positive(t, outcomes[expected], "seed %d: outcome %q never occurred; outcomes: %v", seed, expected, outcomes)+ }+}++// outcome names the outcome of an action: its type, and either ok or its error, with "at depth"+// when the path that failed is below the top level.+func outcome(action commitAction, err error) string {+ if err == nil {+ return fmt.Sprintf("%T ok", action)+ }++ var indexErr indexError+ if errors.As(err, &indexErr) && strings.Contains(indexErr.path, "/") {+ return fmt.Sprintf("%T at depth: %s", action, err)+ }+ return fmt.Sprintf("%T: %s", action, err)+}diff --git a/internal/gitaly/service/operations/commit_files_processes_test.go b/internal/gitaly/service/operations/commit_files_processes_test.gonew file mode 100644index 0000000..55b27a2--- /dev/null+++ b/internal/gitaly/service/operations/commit_files_processes_test.go@@ -0,0 +1,101 @@+package operations++import (+ "fmt"+ "testing"++ "github.com/stretchr/testify/require"++ "github.com/feder-cr/git-ai/internal/featureflag"+ "github.com/feder-cr/git-ai/internal/git/gittest"+ "github.com/feder-cr/git-ai/internal/testhelper"+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"+ "github.com/feder-cr/git-ai/internal/testhelper/testserver"+)++// TestUserCommitFilesProcessCount counts the git processes of one UserCommitFiles request, by+// subcommand (PLAN9, WP96): one cat-file for every read, one hash-object per blob, one hash-object+// for every tree, commit-tree and update-ref. A for-each-ref runs only when no branch resolved,+// for the orphan commit into an empty repository. A process that comes back fails the test.+func TestUserCommitFilesProcessCount(t *testing.T) {+ t.Parallel()++ for _, tc := range []struct {+ desc string+ emptyRepo bool+ newBranch bool+ paths []string+ expected map[string]uint64+ repoCreated bool+ }{+ {+ desc: "existing branch, top level",+ paths: []string{"file.txt"},+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},+ },+ {+ desc: "existing branch, nested",+ paths: []string{"a/b/file.txt"},+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},+ },+ {+ desc: "new branch, top level",+ newBranch: true,+ paths: []string{"file.txt"},+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},+ },+ {+ desc: "new branch, nested",+ newBranch: true,+ paths: []string{"a/b/file.txt"},+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},+ },+ {+ desc: "three files",+ paths: []string{"one.txt", "a/two.txt", "a/b/three.txt"},+ expected: map[string]uint64{"cat-file": 1, "hash-object": 4, "commit-tree": 1, "update-ref": 1},+ },+ {+ desc: "orphan commit into an empty repository",+ emptyRepo: true,+ paths: []string{"a/file.txt"},+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "for-each-ref": 1, "update-ref": 1},+ repoCreated: true,+ },+ } {+ t.Run(tc.desc, func(t *testing.T) {+ t.Parallel()+ ctx := featureflag.OutgoingCtxWithFeatureFlag(testhelper.Context(t), featureflag.GPGSigning, false)+ cfg := testcfg.Build(t)+ counting := gittest.NewCountingCommandFactory(t, cfg)+ ctx, cfg, client := setupOperationsServiceWithCfg(t, ctx, cfg, testserver.WithGitCommandFactory(counting))++ repo, repoPath := gittest.CreateRepository(t, ctx, cfg)+ if !tc.emptyRepo {+ gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("master"),+ gittest.WithTreeEntries(gittest.TreeEntry{Path: "README", Mode: "100644", Content: "readme\n"}))+ }++ branch, start := "master", ""+ if tc.newBranch {+ branch, start = "feature", "master"+ }++ counting.ResetCount()+ stream, err := client.UserCommitFiles(ctx)+ require.NoError(t, err)+ require.NoError(t, stream.Send(headerRequest(repo, gittest.TestUser, branch, []byte("message"), start, "")))+ for i, path := range tc.paths {+ require.NoError(t, stream.Send(createFileHeaderRequest(path)))+ require.NoError(t, stream.Send(actionContentRequest(fmt.Sprintf("content %d\n", i))))+ }+ resp, err := stream.CloseAndRecv()+ require.NoError(t, err)+ require.Empty(t, resp.GetIndexError())+ require.Empty(t, resp.GetPreReceiveError())+ require.Equal(t, tc.repoCreated, resp.GetBranchUpdate().GetRepoCreated())++ require.Equal(t, tc.expected, counting.Counts())+ })+ }+}