Skip to content

Git designed for AI agents.

operations: UserCommitFiles in 5 git processes (WP96)

Federico Eliab677dec44a72
diff --git a/doc/graft/PLAN9.md b/doc/graft/PLAN9.md
index 2c96993..7ab4946 100644
--- a/doc/graft/PLAN9.md
+++ b/doc/graft/PLAN9.md
@@ -652,3 +652,149 @@ and commit build (95 to 146 ms), stage 2.
with the numbers of the window.
* **Release.** v0.2.0 is main with phase 4b, the skew fix, its performance fix
(`6aecff1`) and the `UserSquash` fix. The rest of this plan goes to v0.3.0.
+
+## 15. WP96: the git processes of `UserCommitFiles` (design, for approval)
+
+Owner since 2026-10-19: m3 (scion-74), taking over p5w. This replaces the sketch of 3.4.
+
+### 15.1 Where the processes come from (measured and traced)
+
+`TestUserCommitFilesProcessCount` (`operations`, counting command factory) counts the
+git processes of one request that creates one file:
+
+| Case | Processes | By subcommand |
+|---|---|---|
+| existing branch, `file.txt` | 10 | rev-parse 3, hash-object 2, for-each-ref 1, ls-tree, commit-tree, cat-file, update-ref |
+| existing branch, `a/b/file.txt` | 12 | hash-object 4 (1 blob + 3 trees) |
+| new branch (from `start_branch_name`), `file.txt` | 13 | rev-parse 5, for-each-ref 2 |
+| new branch, `a/b/file.txt` | 15 | |
+
+Each one, traced to its call site (all through `localrepo.Repo.Exec`):
+
+| | Process | Count | From | Needed? |
+|---|---|---|---|---|
+| A | `rev-parse --verify refs/heads/<branch>^{commit}` | 1 | `commit_files.go:569` | a read: the request's object reader can answer it |
+| B | `for-each-ref --count=1 refs/heads/` (`HasBranches`) | new branch: 1 | `commit_files.go:799` (issue-3294 workaround) | only when the start branch does not resolve |
+| C | `rev-parse --verify refs/heads/<start>^{commit}` | new branch: 1 | `commit_files.go:829` | a read |
+| D | `rev-parse --verify <oid>^{commit}` (`fetchMissingCommit`) | new branch: 1 | `commit_files.go:843` | redundant: C resolved and peeled it in the same repository |
+| E | `hash-object --stdin -w --path=<path>` | 1 per blob | `commit_files.go:655` | yes: `--path` applies the repository's attributes (clean and eol filters) to the content |
+| F | `rev-parse --verify <parent>^{tree}` | 1 | `commit_files.go:453` | a read |
+| G | `rev-parse --verify <tree>:` | 1 | `tree.go:689` (`ReadTree`) | redundant: its input is a tree ID, which `Info` checks right after |
+| H | `cat-file -Z --batch-command --buffer` | 1 | `tree.go:694` | the request's object reader (a cache miss: the quarantine is per request) |
+| I | `ls-tree -z -r -t <tree>:` | 1 | `tree.go:407` | reads the whole tree recursively; only the trees on the changed paths are needed |
+| J | `hash-object -t tree -w --stdin` | 1 per changed tree (depth + 1) | `tree.go:623` | one process can write every tree |
+| K | `commit-tree` | 1 | `commit.go:203` | yes (see 15.2, item 4) |
+| L | `for-each-ref --count=1 refs/heads/` | 1 | `commit_files.go:732` | only for `RepoCreated`, which A or C already decides |
+| M | `update-ref -z --stdin` | 1 | `update_with_hooks.go:286` | yes |
+
+### 15.2 Design
+
+1. **Reads go through the request's object reader (H).** One `cat-file --batch-command`
+ process, taken once per request and kept for it (with the MVCC backend the catfile
+ cache is off, so the handle must be held, not looked up per read):
+ * A, C and F become `Info("refs/heads/<branch>^{commit}")`,
+ `Info("refs/heads/<start>^{commit}")` and `Info("<parent>^{tree}")`. A missing
+ object answers `missing`, which maps to the same not-found error as the failed
+ `rev-parse` today. Both resolve a name with git's same rules (`get_oid`), so
+ no ambiguity rule of `rev-parse` is lost. Probed on git 2.53: a running `cat-file --batch-command`
+ resolves `refs/heads/x^{commit}` and `^{tree}`, answers `missing` for an absent
+ reference, and sees objects and references written after it started.
+ * D becomes `Info("<parent>^{commit}")` on the same reader (the fetch from the start
+ repository, when it is missing, is unchanged; the reader then sees the fetched pack). G is
+ dropped (F already answered it).
+ * I is replaced by a lazy tree: the root is read with `Object()` and parsed (in
+ `localrepo`, into the entries `ls-tree -r -t` gives: a six-digit mode and the type
+ from the mode's format bits), and a subtree is read only when an action descends into it
+ (create, update, delete, move, chmod and create-dir each address one path).
+ Untouched subtrees keep their ID, so `Write` skips them as today. The cost no
+ longer grows with the size of the repository's tree.
+2. **One process writes every tree.** Today's writer, `git hash-object -t tree -w`, in
+ `--stdin-paths` mode: the request starts it once, writes each tree's content (the
+ same bytes `writeEntries` builds today) to a temporary file of the storage, sends the
+ path and reads the ID back, bottom-up (the `CommitWriter` pattern, for trees). The
+ bytes, the format checks and the absence of an existence check are those of today, by
+ construction. (Amended after the approval: `mktree --batch` was probed. It gives the
+ same IDs on valid trees, but it writes trees with duplicate names and invalid modes,
+ which `hash-object` refuses as malformed, and it refuses entries whose object is
+ missing, which today's writer accepts. Probe:
+ `m3-work/wp96/probe_mktree2.sh`.) A malformed tree makes the process exit, so the
+ request fails as today; the writer is not reused after an error.
+3. **`for-each-ref` only when nothing else decides it.** `RepoCreated` is false when A or
+ C resolved a branch; `HasBranches` runs only when neither did (an orphan commit,
+ possibly into an empty repository). B runs only when the start branch does not
+ resolve, which is where the issue-3294 workaround matters.
+4. **Unchanged:** E, one `hash-object --path` per blob: the attributes of the path decide
+ the bytes of the blob, and a batched writer cannot apply per-path attributes without a
+ `check-attr` process, which costs as much as it saves for one or two files. K, one
+ `commit-tree`: the existing `CommitWriter` costs two processes per request (its `git
+ config` lookup and its writer) unless it outlives the request. M.
+
+Expected, one created file, any depth, existing or new branch: **5 processes**
+(cat-file, hash-object for the blob, hash-object for the trees, commit-tree, update-ref). n files: 4 + n.
+An orphan commit into an empty repository: 6 (`for-each-ref`).
+
+### 15.3 What stays identical
+
+* **Objects:** the same blob bytes (same `--path`), tree bytes (the same writer and the
+ same input; the differential test checks it) and commit (`commit-tree`, unchanged).
+* **Durability:** every write is still a git process built by the command factory, so
+ `core.fsync=objects,derived-metadata,reference` and `core.fsyncMethod=fsync` apply to
+ the tree writer exactly as today. No object is written by Go, and
+ `TestQuarantineWritersKeepTheDurabilityContract` has no new place. The temporary files
+ are the writer's input, not objects.
+* **Errors:** the same codes and messages for a missing branch, a missing start branch, an
+ existing file, a missing file to update, delete or move, and directory conflicts; the
+ existing `UserCommitFiles` tests run unchanged.
+* **Two answers decided differently, in one corner.** With the start branch in the same
+ repository, it is now resolved before `HasBranches`, which runs only if it does not
+ resolve; and `RepoCreated` is false without `for-each-ref` when the target or the start
+ branch resolved. Both rely on "a resolved `refs/heads/<x>` is a branch". `get_oid` also
+ tries `refs/refs/heads/<x>`, `refs/tags/refs/heads/<x>` and `refs/remotes/refs/heads/<x>`
+ when `refs/heads/<x>` is absent, so a repository with no branch but such a ref now
+ starts from that ref, where today it starts an orphan commit (probe:
+ `m3-work/wp96/probe_dwim.sh`; `rev-parse --verify` and `cat-file` resolve it alike). Today the same rules
+ already apply to the target branch and, in a repository with branches, to the start
+ branch.
+* **Kept as today, found by the differential test:** a file created below an existing
+ file (`README/x`) walks into the blob entry, which loses its ID; the tree is then
+ malformed and both writers refuse it (`writing tree: exit status 128`, an internal error
+ rather than an index error). WP96 keeps this; making it an index error is a separate
+ change.
+* **The object reader goes back to the catfile cache** at the end of the request, on every
+ path (a deferred release), as today's `ReadTree` reader does. Its cache key holds the
+ quarantine directory, so no other request reuses it, and the cache's TTL ends it.
+* **Git features:** `hash-object --stdin-paths` (long-standing, already used by
+ `CommitWriter`) and `cat-file --batch-command` (2.36, already used by the object
+ reader). The server already needs newer git
+ (reftable, `merge-tree --stdin`).
+
+### 15.4 Proof
+
+* `TestUserCommitFilesProcessCount` asserts the processes by subcommand: 5 for the four
+ cases above, 7 for three files (4 + n), 6 for the orphan commit into an empty repository.
+ A process that comes back fails it.
+* `TestApplyActionLazyTreeMatchesRecursiveTree` applies seeded random sequences of the six
+ actions to the whole tree with today's writer and to the lazy tree with the tree writer.
+ It asserts the same error for every action, the same tree ID or the same refusal for
+ every sequence, and that every action type succeeds and every index error occurs below
+ the top level. A failure names the seed (`-lazy-tree-seed` replays it). It fails when
+ the lazy read is removed from the walk.
+* `TestTreeWriter_matchesWriteEntries` covers edge names, git's directory order, every
+ mode, missing objects (accepted), and duplicate names, file/directory clashes, invalid
+ modes and empty IDs (refused with today's error). Writes after a refusal fail, and one
+ process writes every tree. `TestReadTreeLazily_matchesReadTree` checks that the walked
+ lazy tree equals `ReadTree`. `TestTreeWriter_quarantine` checks objects that exist only
+ in the quarantine.
+* `TestUserCommitFilesReleasesProcessesAndFiles` runs failing requests (index error,
+ refused tree, missing start branch) and a successful one. After each, evicting the cache
+ leaves no cat-file process and the storage's temporary directory holds no tree object
+ file. It fails without the deferred release, or without the file removal.
+* The existing operations tests.
+* `UserCommitFiles` latency at 50 agents (graft-bench, as in 2.2), before and after,
+ alternated, in a window the PM grants.
+
+### 15.5 Ownership needed
+
+`internal/gitaly/service/operations/commit_files.go`; `internal/git/localrepo/tree.go` (the
+lazy tree and the tree writer) and `refs.go` if a reader-based resolve helper lives
+there; tests in those packages and `gittest/counting_command_factory.go` (`Counts`).
diff --git a/internal/git/gittest/counting_command_factory.go b/internal/git/gittest/counting_command_factory.go
index 607036a..ae55f92 100644
--- a/internal/git/gittest/counting_command_factory.go
+++ b/internal/git/gittest/counting_command_factory.go
@@ -41,6 +41,18 @@ func (f *CountingCommandFactory) CommandCount(cmd string) uint64 {
return c
}
+// Counts returns a copy of the count of every git command.
+func (f *CountingCommandFactory) Counts() map[string]uint64 {
+ f.m.Lock()
+ defer f.m.Unlock()
+
+ counts := make(map[string]uint64, len(f.counts))
+ for cmd, c := range f.counts {
+ counts[cmd] = c
+ }
+ return counts
+}
+
// RequireCommandCount checks if the count for the given command equals the
// expected count.
func (f *CountingCommandFactory) RequireCommandCount(tb testing.TB, cmd string, expected interface{}) {
diff --git a/internal/git/localrepo/refs.go b/internal/git/localrepo/refs.go
index f4205cd..5e5ad92 100644
--- a/internal/git/localrepo/refs.go
+++ b/internal/git/localrepo/refs.go
@@ -11,6 +11,7 @@ import (
"github.com/feder-cr/git-ai/internal/command"
"github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/catfile"
"github.com/feder-cr/git-ai/internal/git/gitcmd"
"github.com/feder-cr/git-ai/internal/git/updateref"
"github.com/feder-cr/git-ai/internal/gitaly/transaction"
@@ -73,6 +74,45 @@ func (repo *Repo) ResolveRevision(ctx context.Context, revision git.Revision) (g
return oid, nil
}
+// RequestObjectReader returns the object reader of a request that resolves revisions and reads
+// objects through one `git cat-file --batch-command` process: one process for the whole request,
+// where ResolveRevision starts one per revision (PLAN9, WP96). The returned function must be
+// called on every path once the request is done with the reader.
+func (repo *Repo) RequestObjectReader(ctx context.Context) (catfile.ObjectReader, func(), error) {
+ contentReader, cancel, err := repo.catfileCache.ObjectReader(ctx, repo)
+ if err != nil {
+ return nil, nil, fmt.Errorf("creating object reader: %w", err)
+ }
+
+ reader, ok := contentReader.(catfile.ObjectReader)
+ if !ok {
+ cancel()
+ return nil, nil, fmt.Errorf("object reader %T cannot read object info", contentReader)
+ }
+
+ return reader, cancel, nil
+}
+
+// ResolveRevisionWithReader is ResolveRevision through reader, the request's object reader: the
+// same object ID, or git.ErrReferenceNotFound when the revision does not resolve. Unlike
+// ResolveRevision it also requires the object to exist, which a revision peeled to an object type
+// (`<rev>^{commit}`) requires anyway: pass only such revisions where ResolveRevision is replaced.
+func (repo *Repo) ResolveRevisionWithReader(ctx context.Context, reader catfile.ObjectInfoReader, revision git.Revision) (git.ObjectID, error) {
+ if revision.String() == "" {
+ return "", errors.New("repository cannot contain empty reference name")
+ }
+
+ info, err := reader.Info(ctx, revision)
+ if err != nil {
+ if errors.As(err, &catfile.NotFoundError{}) {
+ return "", git.ErrReferenceNotFound
+ }
+ return "", err
+ }
+
+ return info.Oid, nil
+}
+
// GetReference looks up and returns the given reference. Returns a
// ReferenceNotFound error if the reference was not found.
func (repo *Repo) GetReference(ctx context.Context, reference git.ReferenceName) (git.Reference, error) {
diff --git a/internal/git/localrepo/tree.go b/internal/git/localrepo/tree.go
index 1dd3ecb..c068907 100644
--- a/internal/git/localrepo/tree.go
+++ b/internal/git/localrepo/tree.go
@@ -1,15 +1,20 @@
package localrepo
import (
+ "bufio"
"bytes"
"context"
+ "encoding/hex"
"errors"
"fmt"
"io"
+ "os"
"path/filepath"
"sort"
+ "strconv"
"strings"
+ "github.com/feder-cr/git-ai/internal/command"
"github.com/feder-cr/git-ai/internal/git"
"github.com/feder-cr/git-ai/internal/git/catfile"
"github.com/feder-cr/git-ai/internal/git/gitcmd"
@@ -72,6 +77,94 @@ type TreeEntry struct {
Type ObjectType
// Entries is a slice of this tree's entries.
Entries []*TreeEntry
+
+ // loader, set on a tree that ReadTreeLazily read without its entries, reads them when an
+ // operation first descends into the tree. Nil once they are read, and for every tree of
+ // ReadTree.
+ loader *treeLoader
+}
+
+// treeLoader reads the entries of the trees of a tree read by ReadTreeLazily through the object
+// reader of the request (PLAN9, WP96): only the trees that the operations descend into are read.
+type treeLoader struct {
+ ctx context.Context
+ reader catfile.ObjectContentReader
+ objectHash git.ObjectHash
+}
+
+// ensureLoaded reads the entries of t if it is a tree of ReadTreeLazily whose entries were not read
+// yet. It does nothing for any other entry.
+func (t *TreeEntry) ensureLoaded() error {
+ if t.loader == nil {
+ return nil
+ }
+
+ entries, err := t.loader.read(t.OID)
+ if err != nil {
+ return err
+ }
+ t.Entries = entries
+ t.loader = nil
+ return nil
+}
+
+// read returns the entries of the tree oid as ls-tree describes them: a six-digit mode, the type
+// derived from the mode and the base name. Its subtrees are left to be read on demand.
+func (l *treeLoader) read(oid git.ObjectID) ([]*TreeEntry, error) {
+ object, err := l.reader.Object(l.ctx, oid.Revision())
+ if err != nil {
+ return nil, fmt.Errorf("reading tree %s: %w", oid, err)
+ }
+ // The object must be consumed fully before the reader serves another request.
+ data, err := io.ReadAll(object)
+ if err != nil {
+ return nil, fmt.Errorf("reading tree %s: %w", oid, err)
+ }
+ if object.Type != "tree" {
+ return nil, fmt.Errorf("reading tree %s: %w", oid, ErrNotTreeish)
+ }
+
+ rawSize := l.objectHash.EncodedLen() / 2
+ var entries []*TreeEntry
+ for len(data) > 0 {
+ mode, rest, ok := bytes.Cut(data, []byte{' '})
+ if !ok {
+ return nil, fmt.Errorf("parsing tree %s: no mode", oid)
+ }
+ name, rest, ok := bytes.Cut(rest, []byte{0})
+ if !ok || len(rest) < rawSize {
+ return nil, fmt.Errorf("parsing tree %s: truncated entry", oid)
+ }
+ entryOID, err := l.objectHash.FromHex(hex.EncodeToString(rest[:rawSize]))
+ if err != nil {
+ return nil, fmt.Errorf("parsing tree %s: %w", oid, err)
+ }
+ data = rest[rawSize:]
+
+ // As ls-tree prints it: the octal value with six digits, and the type from its format
+ // bits.
+ modeValue, err := strconv.ParseUint(string(mode), 8, 32)
+ if err != nil {
+ return nil, fmt.Errorf("parsing tree %s: invalid mode %q", oid, mode)
+ }
+ entry := &TreeEntry{
+ OID: entryOID,
+ Mode: fmt.Sprintf("%06o", modeValue),
+ Path: string(name),
+ }
+ switch modeValue & 0o170000 {
+ case 0o040000:
+ entry.Type = Tree
+ entry.loader = l
+ case 0o160000:
+ entry.Type = Submodule
+ default:
+ entry.Type = Blob
+ }
+ entries = append(entries, entry)
+ }
+
+ return entries, nil
}
// IsBlob returns whether or not the TreeEntry is a blob.
@@ -179,6 +272,11 @@ func (t *TreeEntry) Add(
Loop:
for {
+ // A tree of ReadTreeLazily is read when the walk first reaches it.
+ if err := currentTree.ensureLoaded(); err != nil {
+ return err
+ }
+
// loop through each layer of the tree based on the directory
// structure specified by path.
// firstComponent is the current directory name, and secondComponent is the rest
@@ -259,6 +357,11 @@ func (t *TreeEntry) recurse(
currentTree := t
for {
+ // A tree of ReadTreeLazily is read when the walk first reaches it.
+ if err := currentTree.ensureLoaded(); err != nil {
+ return err
+ }
+
// Loop through each layer of the tree based on the directory
// structure specified by path.
// firstComponent is the current directory name, and secondComponent is the rest
@@ -538,6 +641,9 @@ func (t *TreeEntry) walk(dirPath string, callback func(path string, entry *TreeE
return err
}
+ if err := t.ensureLoaded(); err != nil {
+ return err
+ }
for _, entry := range t.Entries {
if err := entry.walk(nextDirPath, callback); err != nil {
return err
@@ -550,6 +656,9 @@ func (t *TreeEntry) walk(dirPath string, callback func(path string, entry *TreeE
// Walk will walk the whole tree structure and call callback on every entry of
// the tree in a depth-first like fashion.
func (t *TreeEntry) Walk(callback func(path string, entry *TreeEntry) error) error {
+ if err := t.ensureLoaded(); err != nil {
+ return err
+ }
for _, e := range t.Entries {
if err := e.walk(t.Path, callback); err != nil {
return err
@@ -588,9 +697,8 @@ func (t *TreeEntry) Write(
return nil
}
-// writeTreeEntries writes a new tree object from a slice of entries. This function does not verify whether OIDs
-// referred to by tree entries actually exist in the repository.
-func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.ObjectID, error) {
+// treeContent returns the content of the tree object of entries, which it sorts in git's order.
+func treeContent(entries []*TreeEntry) (*bytes.Buffer, error) {
var tree bytes.Buffer
sort.Stable(TreeEntriesByPath(entries))
@@ -601,18 +709,29 @@ func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.O
oidBytes, err := entry.OID.Bytes()
if err != nil {
- return "", err
+ return nil, err
}
if _, err := tree.WriteString(formattedEntry); err != nil {
- return "", err
+ return nil, err
}
if _, err := tree.Write(oidBytes); err != nil {
- return "", err
+ return nil, err
}
}
+ return &tree, nil
+}
+
+// writeTreeEntries writes a new tree object from a slice of entries. This function does not verify whether OIDs
+// referred to by tree entries actually exist in the repository.
+func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.ObjectID, error) {
+ tree, err := treeContent(entries)
+ if err != nil {
+ return "", err
+ }
+
options := []gitcmd.Option{
gitcmd.ValueFlag{Name: "-t", Value: "tree"},
gitcmd.Flag{Name: "-w"},
@@ -627,7 +746,7 @@ func (repo *Repo) writeEntries(ctx context.Context, entries []*TreeEntry) (git.O
},
gitcmd.WithStdout(&stdout),
gitcmd.WithStderr(&stderr),
- gitcmd.WithStdin(&tree),
+ gitcmd.WithStdin(tree),
); err != nil {
return "", structerr.New("%w", err).WithMetadata("stderr", stderr.String())
}
@@ -737,3 +856,223 @@ func (repo *Repo) ReadTree(ctx context.Context, treeish git.Revision, options ..
return rootEntry, nil
}
+
+// ReadTreeLazily returns the tree treeOID with its direct entries read through reader, the object
+// reader of the request. Its subtrees are read only when an operation (Add, Delete, Get, Modify,
+// Walk) descends into them, through the same reader, so a request that changes a few paths reads
+// a few trees instead of the whole tree (PLAN9, WP96). The entries are those ReadTree with
+// WithRecursive gives, for every tree that is read. reader must stay open while the tree is used.
+func (repo *Repo) ReadTreeLazily(ctx context.Context, reader catfile.ObjectReader, treeOID git.ObjectID) (*TreeEntry, error) {
+ objectHash, err := repo.ObjectHash(ctx)
+ if err != nil {
+ return nil, fmt.Errorf("detecting object hash: %w", err)
+ }
+
+ obj, err := reader.Info(ctx, treeOID.Revision())
+ if err != nil {
+ return nil, fmt.Errorf("check object type: %w", err)
+ }
+ if obj.Type != "tree" {
+ return nil, ErrNotTreeish
+ }
+
+ root := &TreeEntry{
+ OID: treeOID,
+ Type: Tree,
+ Mode: "040000",
+ loader: &treeLoader{ctx: ctx, reader: reader, objectHash: objectHash},
+ }
+ if err := root.ensureLoaded(); err != nil {
+ return nil, err
+ }
+
+ return root, nil
+}
+
+// TreeWriter writes the tree objects of one request through a single `git hash-object -t tree -w
+// --stdin-paths` process, started on the first write: today's tree writer, with the same input and
+// the same format checks, without a process per tree (PLAN9, WP96). The content of each tree is
+// written to a file of the storage's temporary directory (created exclusively, mode 0600, removed
+// after the write, whatever happened), whose path the process reads. A write that fails stops the
+// process and makes every later write fail: a tree the process refused ends the request, as
+// today. Close must be called on every path.
+type TreeWriter struct {
+ repo *Repo
+ ctx context.Context
+ tempDir string
+ objectHash git.ObjectHash
+
+ cmd *command.Command
+ stdout *bufio.Reader
+ stderr bytes.Buffer
+ cancel context.CancelFunc
+ err error
+}
+
+var errTreeWriterClosed = errors.New("tree writer is closed")
+
+// NewTreeWriter returns the tree writer of a request on repo. Its process runs under ctx.
+func (repo *Repo) NewTreeWriter(ctx context.Context) (*TreeWriter, error) {
+ tempDir, err := repo.StorageTempDir()
+ if err != nil {
+ return nil, fmt.Errorf("getting temporary directory: %w", err)
+ }
+ objectHash, err := repo.ObjectHash(ctx)
+ if err != nil {
+ return nil, fmt.Errorf("detecting object hash: %w", err)
+ }
+
+ return &TreeWriter{repo: repo, ctx: ctx, tempDir: tempDir, objectHash: objectHash}, nil
+}
+
+// write writes the tree of entries and returns its object ID.
+func (w *TreeWriter) write(entries []*TreeEntry) (_ git.ObjectID, returnedErr error) {
+ if w.err != nil {
+ return "", w.err
+ }
+ defer func() {
+ if returnedErr != nil {
+ w.err = returnedErr
+ w.stop()
+ }
+ }()
+
+ content, err := treeContent(entries)
+ if err != nil {
+ return "", err
+ }
+
+ // os.CreateTemp opens the file with O_CREATE|O_EXCL and mode 0600.
+ file, err := os.CreateTemp(w.tempDir, "tree-object-*")
+ if err != nil {
+ return "", fmt.Errorf("creating tree object file: %w", err)
+ }
+ defer func() {
+ if err := os.Remove(file.Name()); err != nil && !errors.Is(err, os.ErrNotExist) && returnedErr == nil {
+ returnedErr = fmt.Errorf("removing tree object file: %w", err)
+ }
+ }()
+ if _, err := file.Write(content.Bytes()); err != nil {
+ _ = file.Close()
+ return "", fmt.Errorf("writing tree object file: %w", err)
+ }
+ if err := file.Close(); err != nil {
+ return "", fmt.Errorf("closing tree object file: %w", err)
+ }
+
+ if w.cmd == nil {
+ if err := w.start(); err != nil {
+ return "", err
+ }
+ }
+
+ if _, err := fmt.Fprintf(w.cmd.Stdin(), "%s\n", file.Name()); err != nil {
+ return "", w.processError(fmt.Errorf("sending tree object path: %w", err))
+ }
+ line, err := w.stdout.ReadString('\n')
+ if err != nil {
+ return "", w.processError(fmt.Errorf("reading tree object ID: %w", err))
+ }
+
+ oid, err := w.objectHash.FromHex(strings.TrimSuffix(line, "\n"))
+ if err != nil {
+ return "", fmt.Errorf("parsing tree object ID: %w", err)
+ }
+ return oid, nil
+}
+
+// start starts the hash-object process.
+func (w *TreeWriter) start() error {
+ processCtx, cancel := context.WithCancel(w.ctx)
+ cmd, err := w.repo.Exec(processCtx,
+ gitcmd.Command{
+ Name: "hash-object",
+ Flags: []gitcmd.Option{
+ gitcmd.ValueFlag{Name: "-t", Value: "tree"},
+ gitcmd.Flag{Name: "-w"},
+ gitcmd.Flag{Name: "--stdin-paths"},
+ },
+ },
+ gitcmd.WithSetupStdin(),
+ gitcmd.WithSetupStdout(),
+ gitcmd.WithStderr(&w.stderr),
+ )
+ if err != nil {
+ cancel()
+ return fmt.Errorf("starting hash-object: %w", err)
+ }
+
+ w.cmd = cmd
+ w.stdout = bufio.NewReader(cmd)
+ w.cancel = cancel
+ return nil
+}
+
+// processError reaps the process, which stopped answering, and returns why: its exit status with
+// what it printed on stderr, as today's writer reports a refused tree, or err if it exited
+// cleanly. Wait closes its stdin first.
+func (w *TreeWriter) processError(err error) error {
+ waitErr := w.cmd.Wait()
+ w.cmd = nil
+ w.cancel()
+ if waitErr != nil {
+ err = waitErr
+ }
+ return structerr.New("%w", err).WithMetadata("stderr", w.stderr.String())
+}
+
+// stop kills and reaps the process, if any.
+func (w *TreeWriter) stop() {
+ if w.cmd == nil {
+ return
+ }
+ w.cancel()
+ // The process was killed, so its exit status carries no information.
+ _ = w.cmd.Wait()
+ w.cmd = nil
+}
+
+// Close ends the process: it reads the end of its input and exits, so every tree it wrote is in the
+// object directory when Close returns. Close is idempotent, later calls return nil, and it is safe
+// after a failed write, whose process is already reaped. Writes fail after Close.
+func (w *TreeWriter) Close() error {
+ if w.err == nil {
+ w.err = errTreeWriterClosed
+ }
+ if w.cmd == nil {
+ return nil
+ }
+ defer w.cancel()
+
+ cmd := w.cmd
+ w.cmd = nil
+ // Wait closes the process's stdin first.
+ if err := cmd.Wait(); err != nil {
+ return structerr.New("waiting for tree writer: %w", err).WithMetadata("stderr", w.stderr.String())
+ }
+ return nil
+}
+
+// WriteWith is Write through the request's tree writer: the same trees, written depth first when
+// their OID is empty, by one process.
+func (t *TreeEntry) WriteWith(w *TreeWriter) error {
+ if t.OID != "" {
+ return nil
+ }
+
+ for _, e := range t.Entries {
+ if e.Type == Tree && e.OID == "" {
+ if err := e.WriteWith(w); err != nil {
+ return err
+ }
+ }
+ }
+
+ treeOID, err := w.write(t.Entries)
+ if err != nil {
+ return fmt.Errorf("writing tree: %w", err)
+ }
+
+ t.OID = treeOID
+ return nil
+}
diff --git a/internal/git/localrepo/tree_writer_test.go b/internal/git/localrepo/tree_writer_test.go
new file mode 100644
index 0000000..f253042
--- /dev/null
+++ b/internal/git/localrepo/tree_writer_test.go
@@ -0,0 +1,306 @@
+package localrepo
+
+import (
+ "context"
+ "os"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/git/quarantine"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+)
+
+func setupTreeWriterRepo(t *testing.T) (context.Context, config.Cfg, *Repo, string) {
+ t.Helper()
+
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ return ctx, cfg, NewTestRepo(t, cfg, repoProto), repoPath
+}
+
+// requireNoTreeObjectFiles asserts that no temporary tree object file is left in the storage's
+// temporary directory.
+func requireNoTreeObjectFiles(t *testing.T, repo *Repo) {
+ t.Helper()
+
+ tempDir, err := repo.StorageTempDir()
+ require.NoError(t, err)
+ entries, err := os.ReadDir(tempDir)
+ require.NoError(t, err)
+ for _, entry := range entries {
+ require.NotContains(t, entry.Name(), "tree-object-")
+ }
+}
+
+func cloneEntries(entries []*TreeEntry) []*TreeEntry {
+ cloned := make([]*TreeEntry, 0, len(entries))
+ for _, entry := range entries {
+ entryCopy := *entry
+ cloned = append(cloned, &entryCopy)
+ }
+ return cloned
+}
+
+// TestTreeWriter_matchesWriteEntries checks that the tree writer writes the trees today's writer
+// writes, with the same IDs, through one process for every tree, and refuses the trees it refuses,
+// with the same error (PLAN9, WP96).
+func TestTreeWriter_matchesWriteEntries(t *testing.T) {
+ t.Parallel()
+
+ ctx, cfg, repo, repoPath := setupTreeWriterRepo(t)
+
+ blob := gittest.WriteBlob(t, cfg, repoPath, []byte("one\n"))
+ otherBlob := gittest.WriteBlob(t, cfg, repoPath, []byte("two\n"))
+ subtree := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{{Path: "x", Mode: "100644", OID: blob}})
+ commit := gittest.WriteCommit(t, cfg, repoPath)
+ // An object ID that no object of the repository has.
+ missing := gittest.DefaultObjectHash.HashData([]byte("missing object"))
+
+ for _, tc := range []struct {
+ desc string
+ entries []*TreeEntry
+ refused bool
+ }{
+ {
+ desc: "empty",
+ },
+ {
+ desc: "edge names",
+ entries: []*TreeEntry{
+ {Path: "a", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "a.b", Mode: "100644", Type: Blob, OID: otherBlob},
+ {Path: "a b", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "é té", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "-", Mode: "100644", Type: Blob, OID: blob},
+ },
+ },
+ {
+ desc: "a directory sorts as its name with a slash",
+ entries: []*TreeEntry{
+ {Path: "a0", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "a", Mode: "040000", Type: Tree, OID: subtree},
+ {Path: "a.b", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "a-", Mode: "100644", Type: Blob, OID: blob},
+ },
+ },
+ {
+ desc: "every mode",
+ entries: []*TreeEntry{
+ {Path: "regular", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "executable", Mode: "100755", Type: Blob, OID: blob},
+ {Path: "symlink", Mode: "120000", Type: Blob, OID: otherBlob},
+ {Path: "directory", Mode: "040000", Type: Tree, OID: subtree},
+ {Path: "submodule", Mode: "160000", Type: Submodule, OID: commit},
+ },
+ },
+ {
+ desc: "an object that does not exist is accepted",
+ entries: []*TreeEntry{
+ {Path: "missing", Mode: "100644", Type: Blob, OID: missing},
+ {Path: "missing-submodule", Mode: "160000", Type: Submodule, OID: missing},
+ },
+ },
+ {
+ desc: "duplicate names",
+ entries: []*TreeEntry{
+ {Path: "d", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "d", Mode: "100644", Type: Blob, OID: otherBlob},
+ },
+ refused: true,
+ },
+ {
+ desc: "a file and a directory of the same name",
+ entries: []*TreeEntry{
+ {Path: "d", Mode: "100644", Type: Blob, OID: blob},
+ {Path: "d", Mode: "040000", Type: Tree, OID: subtree},
+ },
+ refused: true,
+ },
+ {
+ desc: "invalid mode",
+ entries: []*TreeEntry{{Path: "bad", Mode: "100600", Type: Blob, OID: blob}},
+ refused: true,
+ },
+ {
+ desc: "an entry without object ID",
+ entries: []*TreeEntry{{Path: "empty", Mode: "100644", Type: Blob}},
+ refused: true,
+ },
+ } {
+ t.Run(tc.desc, func(t *testing.T) {
+ expectedOID, expectedErr := repo.writeEntries(ctx, cloneEntries(tc.entries))
+ require.Equal(t, tc.refused, expectedErr != nil, "today's writer: %v", expectedErr)
+
+ writer, err := repo.NewTreeWriter(ctx)
+ require.NoError(t, err)
+
+ oid, err := writer.write(cloneEntries(tc.entries))
+ if tc.refused {
+ require.Error(t, err)
+ require.Equal(t, expectedErr.Error(), err.Error())
+
+ // The process exited and was reaped; the writer refuses every later write.
+ require.Nil(t, writer.cmd)
+ _, laterErr := writer.write(cloneEntries(tc.entries[:1]))
+ require.Equal(t, err, laterErr)
+ } else {
+ require.NoError(t, err)
+ require.Equal(t, expectedOID, oid)
+
+ // The same process writes the next tree.
+ process := writer.cmd
+ oid, err = writer.write(cloneEntries(tc.entries))
+ require.NoError(t, err)
+ require.Equal(t, expectedOID, oid)
+ require.Same(t, process, writer.cmd)
+ }
+
+ require.NoError(t, writer.Close())
+ require.Nil(t, writer.cmd)
+ require.NoError(t, writer.Close())
+ _, err = writer.write(cloneEntries(tc.entries))
+ require.Error(t, err)
+
+ requireNoTreeObjectFiles(t, repo)
+ })
+ }
+}
+
+// TestTreeEntry_WriteWith checks that WriteWith writes the trees Write writes.
+func TestTreeEntry_WriteWith(t *testing.T) {
+ t.Parallel()
+
+ ctx, cfg, repo, repoPath := setupTreeWriterRepo(t)
+ blob := gittest.WriteBlob(t, cfg, repoPath, []byte("content\n"))
+
+ build := func() *TreeEntry {
+ root := &TreeEntry{Mode: "040000", Type: Tree}
+ for _, path := range []string{"a/b/c", "a/b.c", "a.b/c", "d", "a/é/f"} {
+ require.NoError(t, root.Add(path, TreeEntry{Path: path[strings.LastIndex(path, "/")+1:], Mode: "100644", Type: Blob, OID: blob}))
+ }
+ return root
+ }
+
+ expected := build()
+ require.NoError(t, expected.Write(ctx, repo))
+
+ actual := build()
+ writer, err := repo.NewTreeWriter(ctx)
+ require.NoError(t, err)
+ require.NoError(t, actual.WriteWith(writer))
+ require.NoError(t, writer.Close())
+
+ require.Equal(t, expected, actual)
+ requireNoTreeObjectFiles(t, repo)
+}
+
+// TestReadTreeLazily_matchesReadTree checks that the lazy tree, once walked, is the tree ReadTree
+// reads whole, and that it refuses what ReadTree refuses.
+func TestReadTreeLazily_matchesReadTree(t *testing.T) {
+ t.Parallel()
+
+ ctx, cfg, repo, repoPath := setupTreeWriterRepo(t)
+
+ blob := gittest.WriteBlob(t, cfg, repoPath, []byte("content\n"))
+ commit := gittest.WriteCommit(t, cfg, repoPath)
+ inner := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{
+ {Path: "file", Mode: "100644", OID: blob},
+ {Path: "é", Mode: "100755", OID: blob},
+ })
+ middle := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{
+ {Path: "inner", Mode: "040000", OID: inner},
+ {Path: "link", Mode: "120000", OID: blob},
+ })
+ root := gittest.WriteTree(t, cfg, repoPath, []gittest.TreeEntry{
+ {Path: "a", Mode: "040000", OID: middle},
+ {Path: "a.b", Mode: "100644", OID: blob},
+ {Path: "a b", Mode: "040000", OID: inner},
+ {Path: "submodule", Mode: "160000", OID: commit},
+ })
+
+ expected, err := repo.ReadTree(ctx, root.Revision(), WithRecursive())
+ require.NoError(t, err)
+
+ reader, cancel, err := repo.RequestObjectReader(ctx)
+ require.NoError(t, err)
+ defer cancel()
+
+ lazy, err := repo.ReadTreeLazily(ctx, reader, root)
+ require.NoError(t, err)
+ // Only the root is read.
+ for _, entry := range lazy.Entries {
+ require.Nil(t, entry.Entries)
+ }
+
+ require.NoError(t, lazy.Walk(func(string, *TreeEntry) error { return nil }))
+ require.Equal(t, expected, lazy)
+
+ _, err = repo.ReadTreeLazily(ctx, reader, blob)
+ require.Equal(t, ErrNotTreeish, err)
+
+ missing := gittest.DefaultObjectHash.HashData([]byte("missing tree"))
+ _, err = repo.ReadTreeLazily(ctx, reader, missing)
+ require.ErrorContains(t, err, "check object type: object not found")
+}
+
+// TestTreeWriter_quarantine checks that the request's reader and tree writer run on the
+// quarantined repository: they write into the quarantine and read what only it holds.
+func TestTreeWriter_quarantine(t *testing.T) {
+ t.Parallel()
+
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ repoProto, _ := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ repo := NewTestRepo(t, cfg, repoProto)
+
+ quarantineDir, cleanup, err := quarantine.New(ctx, repoProto, testhelper.NewLogger(t), config.NewLocator(cfg))
+ require.NoError(t, err)
+ t.Cleanup(cleanup)
+ quarantineRepo := NewTestRepo(t, cfg, quarantineDir.QuarantinedRepo())
+
+ blob, err := quarantineRepo.WriteBlob(ctx, strings.NewReader("quarantined\n"), WriteBlobConfig{})
+ require.NoError(t, err)
+
+ tree := &TreeEntry{Mode: "040000", Type: Tree}
+ require.NoError(t, tree.Add("dir/file", TreeEntry{Path: "file", Mode: "100644", Type: Blob, OID: blob}))
+ writer, err := quarantineRepo.NewTreeWriter(ctx)
+ require.NoError(t, err)
+ require.NoError(t, tree.WriteWith(writer))
+ require.NoError(t, writer.Close())
+
+ // The trees are in the quarantine only.
+ for _, oid := range []git.ObjectID{tree.OID, tree.Entries[0].OID} {
+ _, err := quarantineRepo.ReadObject(ctx, oid)
+ require.NoError(t, err)
+ _, err = repo.ReadObject(ctx, oid)
+ require.Equal(t, InvalidObjectError(oid.String()), err)
+ }
+
+ reader, cancel, err := quarantineRepo.RequestObjectReader(ctx)
+ require.NoError(t, err)
+ defer cancel()
+
+ resolved, err := quarantineRepo.ResolveRevisionWithReader(ctx, reader, tree.OID.Revision()+"^{tree}")
+ require.NoError(t, err)
+ require.Equal(t, tree.OID, resolved)
+
+ lazy, err := quarantineRepo.ReadTreeLazily(ctx, reader, tree.OID)
+ require.NoError(t, err)
+ entry, err := lazy.Get("dir/file")
+ require.NoError(t, err)
+ require.Equal(t, blob, entry.OID)
+
+ mainReader, cancelMain, err := repo.RequestObjectReader(ctx)
+ require.NoError(t, err)
+ defer cancelMain()
+
+ _, err = repo.ResolveRevisionWithReader(ctx, mainReader, tree.OID.Revision()+"^{tree}")
+ require.Equal(t, git.ErrReferenceNotFound, err)
+}
diff --git a/internal/gitaly/service/operations/commit_files.go b/internal/gitaly/service/operations/commit_files.go
index 22c0e62..9ca0175 100644
--- a/internal/gitaly/service/operations/commit_files.go
+++ b/internal/gitaly/service/operations/commit_files.go
@@ -11,7 +11,7 @@ import (
"strings"
"github.com/feder-cr/git-ai/internal/git"
- "github.com/feder-cr/git-ai/internal/git/gitcmd"
+ "github.com/feder-cr/git-ai/internal/git/catfile"
"github.com/feder-cr/git-ai/internal/git/localrepo"
"github.com/feder-cr/git-ai/internal/git/remoterepo"
"github.com/feder-cr/git-ai/internal/gitaly/hook"
@@ -439,6 +439,7 @@ func (s *Server) userCommitFilesGit(
header *gitalypb.UserCommitFilesRequestHeader,
parentCommitOID git.ObjectID,
quarantineRepo *localrepo.Repo,
+ reader catfile.ObjectReader,
repoPath string,
actions []commitAction,
) (git.ObjectID, error) {
@@ -450,8 +451,9 @@ func (s *Server) userCommitFilesGit(
var treeish git.ObjectID
if parentCommitOID != "" {
- treeish, err = quarantineRepo.ResolveRevision(
+ treeish, err = quarantineRepo.ResolveRevisionWithReader(
ctx,
+ reader,
git.Revision(fmt.Sprintf("%s^{tree}", parentCommitOID)),
)
if err != nil {
@@ -465,11 +467,8 @@ func (s *Server) userCommitFilesGit(
}
if treeish != "" {
- treeEntry, err = quarantineRepo.ReadTree(
- ctx,
- git.Revision(treeish),
- localrepo.WithRecursive(),
- )
+ // Only the trees the actions descend into are read.
+ treeEntry, err = quarantineRepo.ReadTreeLazily(ctx, reader, treeish)
if err != nil {
return "", fmt.Errorf("reading tree: %w", err)
}
@@ -486,10 +485,7 @@ func (s *Server) userCommitFilesGit(
}
}
- if err := treeEntry.Write(
- ctx,
- quarantineRepo,
- ); err != nil {
+ if err := writeTreeEntries(ctx, quarantineRepo, treeEntry); err != nil {
return "", fmt.Errorf("writing tree %w", err)
}
@@ -539,6 +535,22 @@ func (s *Server) userCommitFilesGit(
)
}
+// writeTreeEntries writes the trees of tree that changed through one tree writer, which it closes
+// on every path: once it returns, every tree it wrote is in the object directory.
+func writeTreeEntries(ctx context.Context, repo *localrepo.Repo, tree *localrepo.TreeEntry) (returnedErr error) {
+ treeWriter, err := repo.NewTreeWriter(ctx)
+ if err != nil {
+ return err
+ }
+ defer func() {
+ if err := treeWriter.Close(); err != nil && returnedErr == nil {
+ returnedErr = err
+ }
+ }()
+
+ return tree.WriteWith(treeWriter)
+}
+
func (s *Server) userCommitFiles(
ctx context.Context,
header *gitalypb.UserCommitFilesRequestHeader,
@@ -565,8 +577,16 @@ func (s *Server) userCommitFiles(
remoteRepo = nil
}
+ // One object reader serves every revision and tree the request reads (PLAN9, WP96). It reads
+ // the quarantined repository, so it sees the objects the request writes.
+ reader, cancelReader, err := quarantineRepo.RequestObjectReader(ctx)
+ if err != nil {
+ return err
+ }
+ defer cancelReader()
+
targetBranchName := git.NewReferenceNameFromBranchName(string(header.GetBranchName()))
- targetBranchCommit, err := quarantineRepo.ResolveRevision(ctx, targetBranchName.Revision()+"^{commit}")
+ targetBranchCommit, err := quarantineRepo.ResolveRevisionWithReader(ctx, reader, targetBranchName.Revision()+"^{commit}")
if err != nil {
if !errors.Is(err, git.ErrReferenceNotFound) {
return fmt.Errorf("resolve target branch commit: %w", err)
@@ -576,10 +596,14 @@ func (s *Server) userCommitFiles(
}
var parentCommitOID git.ObjectID
+ // startBranchResolved is whether the start branch resolved in the repository, which then has
+ // a branch.
+ var startBranchResolved bool
if header.GetStartSha() == "" {
- parentCommitOID, err = s.resolveParentCommit(
+ parentCommitOID, startBranchResolved, err = s.resolveParentCommit(
ctx,
quarantineRepo,
+ reader,
remoteRepo,
targetBranchName,
targetBranchCommit,
@@ -596,7 +620,7 @@ func (s *Server) userCommitFiles(
}
if parentCommitOID != targetBranchCommit {
- if err := s.fetchMissingCommit(ctx, quarantineRepo, remoteRepo, parentCommitOID); err != nil {
+ if err := s.fetchMissingCommit(ctx, quarantineRepo, reader, remoteRepo, parentCommitOID); err != nil {
return fmt.Errorf("fetch missing commit: %w", err)
}
}
@@ -718,6 +742,7 @@ func (s *Server) userCommitFiles(
header,
parentCommitOID,
quarantineRepo,
+ reader,
repoPath,
actions,
)
@@ -729,9 +754,13 @@ func (s *Server) userCommitFiles(
return err
}
- hasBranches, err := quarantineRepo.HasBranches(ctx)
- if err != nil {
- return fmt.Errorf("was repo created: %w", err)
+ // A branch that resolved already tells that the repository has branches.
+ hasBranches := targetBranchCommit != "" || startBranchResolved
+ if !hasBranches {
+ hasBranches, err = quarantineRepo.HasBranches(ctx)
+ if err != nil {
+ return fmt.Errorf("was repo created: %w", err)
+ }
}
var oldRevision git.ObjectID
@@ -775,29 +804,52 @@ func sameRepository(repoA, repoB *gitalypb.Repository) bool {
repoA.GetRelativePath() == repoB.GetRelativePath()
}
+// resolveParentCommit returns the parent commit of the new commit, and whether it is the start
+// branch of local, which then has a branch.
func (s *Server) resolveParentCommit(
ctx context.Context,
- local gitcmd.Repository,
+ local *localrepo.Repo,
+ localReader catfile.ObjectInfoReader,
remote *gitalypb.Repository,
targetBranch git.ReferenceName,
targetBranchCommit git.ObjectID,
startBranch string,
-) (git.ObjectID, error) {
+) (git.ObjectID, bool, error) {
if remote == nil && startBranch == "" {
- return targetBranchCommit, nil
+ return targetBranchCommit, false, nil
}
- repo := local
- if remote != nil {
- var err error
- repo, err = remoterepo.New(ctx, remote, s.conns)
- if err != nil {
- return "", fmt.Errorf("remote repository: %w", err)
+ branch := targetBranch
+ if startBranch != "" {
+ branch = git.NewReferenceNameFromBranchName(startBranch)
+ }
+ refish := branch + "^{commit}"
+
+ if remote == nil {
+ // The start branch is resolved first, through the request's reader: a repository where
+ // it resolves has branches, so the workaround below matters only when it does not.
+ commit, err := local.ResolveRevisionWithReader(ctx, localReader, git.Revision(refish))
+ if err == nil {
+ return commit, true, nil
}
+
+ if hasBranches, hasBranchesErr := local.HasBranches(ctx); hasBranchesErr != nil {
+ return "", false, fmt.Errorf("has branches: %w", hasBranchesErr)
+ } else if !hasBranches {
+ // The first branch is being created: see the workaround below.
+ return targetBranchCommit, false, nil
+ }
+
+ return "", false, fmt.Errorf("resolving refish %q in %T: %w", refish, local, err)
+ }
+
+ repo, err := remoterepo.New(ctx, remote, s.conns)
+ if err != nil {
+ return "", false, fmt.Errorf("remote repository: %w", err)
}
if hasBranches, err := repo.HasBranches(ctx); err != nil {
- return "", fmt.Errorf("has branches: %w", err)
+ return "", false, fmt.Errorf("has branches: %w", err)
} else if !hasBranches {
// GitLab sends requests to UserCommitFiles where target repository
// and start repository are the same. If the request hits Gitaly directly,
@@ -817,30 +869,25 @@ func (s *Server) resolveParentCommit(
// already resolved locally.
//
// See: https://gitlab.com/gitlab-org/gitaly/-/issues/3294
- return targetBranchCommit, nil
+ return targetBranchCommit, false, nil
}
- branch := targetBranch
- if startBranch != "" {
- branch = git.NewReferenceNameFromBranchName(startBranch)
- }
- refish := branch + "^{commit}"
-
commit, err := repo.ResolveRevision(ctx, git.Revision(refish))
if err != nil {
- return "", fmt.Errorf("resolving refish %q in %T: %w", refish, repo, err)
+ return "", false, fmt.Errorf("resolving refish %q in %T: %w", refish, repo, err)
}
- return commit, nil
+ return commit, false, nil
}
func (s *Server) fetchMissingCommit(
ctx context.Context,
localRepo *localrepo.Repo,
+ localReader catfile.ObjectInfoReader,
remoteRepo *gitalypb.Repository,
commit git.ObjectID,
) error {
- if _, err := localRepo.ResolveRevision(ctx, commit.Revision()+"^{commit}"); err != nil {
+ if _, err := localRepo.ResolveRevisionWithReader(ctx, localReader, commit.Revision()+"^{commit}"); err != nil {
if !errors.Is(err, git.ErrReferenceNotFound) || remoteRepo == nil {
return fmt.Errorf("lookup parent commit: %w", err)
}
diff --git a/internal/gitaly/service/operations/commit_files_cleanup_test.go b/internal/gitaly/service/operations/commit_files_cleanup_test.go
new file mode 100644
index 0000000..a4e942e
--- /dev/null
+++ b/internal/gitaly/service/operations/commit_files_cleanup_test.go
@@ -0,0 +1,112 @@
+package operations
+
+import (
+ "os"
+ "strings"
+ "testing"
+
+ "github.com/prometheus/client_golang/prometheus/testutil"
+ "github.com/stretchr/testify/require"
+ "google.golang.org/grpc"
+
+ "github.com/feder-cr/git-ai/internal/featureflag"
+ "github.com/feder-cr/git-ai/internal/git/catfile"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/gitaly/config"
+ "github.com/feder-cr/git-ai/internal/gitaly/service"
+ "github.com/feder-cr/git-ai/internal/gitaly/service/hook"
+ "github.com/feder-cr/git-ai/internal/gitaly/service/repository"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/internal/testhelper/testserver"
+ "github.com/feder-cr/git-ai/proto/go/gitalypb"
+)
+
+// TestUserCommitFilesReleasesProcessesAndFiles checks that a UserCommitFiles request, whether it
+// fails or succeeds, gives back its object reader and stops its tree writer, and leaves no
+// temporary tree object file in the storage (PLAN9, WP96): once the cache evicts its processes,
+// no cat-file process is left running.
+func TestUserCommitFilesReleasesProcessesAndFiles(t *testing.T) {
+ t.Parallel()
+
+ ctx := featureflag.OutgoingCtxWithFeatureFlag(testhelper.Context(t), featureflag.GPGSigning, false)
+ cfg := testcfg.Build(t)
+ testcfg.BuildGitalySSH(t, cfg)
+ testcfg.BuildGitalyHooks(t, cfg)
+
+ var processCache *catfile.ProcessCache
+ cfg.SocketPath = testserver.RunGitalyServer(t, cfg, func(srv *grpc.Server, deps *service.Dependencies) {
+ processCache = deps.GetCatfileCache().(*catfile.ProcessCache)
+ gitalypb.RegisterOperationServiceServer(srv, NewServer(deps))
+ gitalypb.RegisterHookServiceServer(srv, hook.NewServer(deps))
+ gitalypb.RegisterRepositoryServiceServer(srv, repository.NewServer(deps))
+ })
+ client, conn := newOperationClient(t, cfg.SocketPath)
+ t.Cleanup(func() { conn.Close() })
+ ctx = testhelper.MergeOutgoingMetadata(ctx, testcfg.GitalyServersMetadataFromCfg(t, cfg))
+
+ repo, repoPath := gittest.CreateRepository(t, ctx, cfg)
+ gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("master"),
+ gittest.WithTreeEntries(gittest.TreeEntry{Path: "README", Mode: "100644", Content: "readme\n"}))
+
+ tempDir, err := config.NewLocator(cfg).TempDir(cfg.Storages[0].Name)
+ require.NoError(t, err)
+
+ for _, tc := range []struct {
+ desc string
+ startBranch string
+ requests []*gitalypb.UserCommitFilesRequest
+ expectedErr string
+ }{
+ {
+ desc: "a file to delete that does not exist",
+ requests: []*gitalypb.UserCommitFilesRequest{deleteFileHeaderRequest("missing.txt")},
+ expectedErr: "A file with this name doesn't exist",
+ },
+ {
+ desc: "a tree the writer refuses",
+ requests: []*gitalypb.UserCommitFilesRequest{createFileHeaderRequest("README/file"), actionContentRequest("content\n")},
+ expectedErr: "writing tree writing tree: exit status 128",
+ },
+ {
+ desc: "a start branch that does not exist",
+ startBranch: "missing",
+ requests: []*gitalypb.UserCommitFilesRequest{createFileHeaderRequest("file"), actionContentRequest("content\n")},
+ expectedErr: "resolve parent commit",
+ },
+ {
+ desc: "success",
+ requests: []*gitalypb.UserCommitFilesRequest{createFileHeaderRequest("dir/file"), actionContentRequest("content\n")},
+ },
+ } {
+ t.Run(tc.desc, func(t *testing.T) {
+ stream, err := client.UserCommitFiles(ctx)
+ require.NoError(t, err)
+ require.NoError(t, stream.Send(headerRequest(repo, gittest.TestUser, "master", []byte("message"), tc.startBranch, "")))
+ for _, request := range tc.requests {
+ require.NoError(t, stream.Send(request))
+ }
+ _, err = stream.CloseAndRecv()
+ if tc.expectedErr == "" {
+ require.NoError(t, err)
+ } else {
+ require.ErrorContains(t, err, tc.expectedErr)
+ }
+
+ // Every process the request took was given back to the cache or closed: evicting
+ // the cache stops them all.
+ processCache.Evict()
+ require.NoError(t, testutil.CollectAndCompare(processCache, strings.NewReader(`
+# HELP gitaly_catfile_processes Gauge of active catfile processes
+# TYPE gitaly_catfile_processes gauge
+gitaly_catfile_processes 0
+`), "gitaly_catfile_processes"))
+
+ entries, err := os.ReadDir(tempDir)
+ require.NoError(t, err)
+ for _, entry := range entries {
+ require.NotContains(t, entry.Name(), "tree-object-")
+ }
+ })
+ }
+}
diff --git a/internal/gitaly/service/operations/commit_files_lazy_test.go b/internal/gitaly/service/operations/commit_files_lazy_test.go
new file mode 100644
index 0000000..4e436cb
--- /dev/null
+++ b/internal/gitaly/service/operations/commit_files_lazy_test.go
@@ -0,0 +1,202 @@
+package operations
+
+import (
+ "errors"
+ "flag"
+ "fmt"
+ "math/rand/v2"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/git"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/git/localrepo"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+)
+
+var lazyTreeSeed = flag.Uint64("lazy-tree-seed", 0, "seed of TestApplyActionLazyTreeMatchesRecursiveTree, random when 0")
+
+// TestApplyActionLazyTreeMatchesRecursiveTree applies random sequences of UserCommitFiles actions
+// to the tree read whole (ReadTree with WithRecursive, then Write: the path before WP96) and to the
+// tree of ReadTreeLazily (then the tree writer: the path after it). Every action must give the same
+// error on both, and every sequence the same tree ID. The actions reach every outcome: each action
+// type succeeds, and each error the whole tree detected (an existing file or directory, a move
+// onto an existing file, a missing file to update, move or delete, a file in the way of a
+// directory) occurs, also below the top level. A failure names the seed, which
+// -lazy-tree-seed replays.
+func TestApplyActionLazyTreeMatchesRecursiveTree(t *testing.T) {
+ t.Parallel()
+
+ ctx := testhelper.Context(t)
+ cfg := testcfg.Build(t)
+ repoProto, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{SkipCreationViaService: true})
+ repo := localrepo.NewTestRepo(t, cfg, repoProto)
+
+ seed := *lazyTreeSeed
+ if seed == 0 {
+ seed = uint64(time.Now().UnixNano())
+ }
+ t.Logf("seed %d (replay with -lazy-tree-seed=%d)", seed, seed)
+ rng := rand.New(rand.NewPCG(seed, 0))
+
+ blobs := gittest.WriteBlobs(t, cfg, repoPath, 4)
+ submodule := gittest.WriteCommit(t, cfg, repoPath)
+ randomBlob := func() git.ObjectID { return blobs[rng.IntN(len(blobs))] }
+
+ // Few names, so that paths collide: a file where a directory is wanted, and the reverse.
+ names := []string{"a", "b", "a.b", "a b", "é"}
+ // basePaths are the paths of the base tree's entries, which half of the actions address.
+ var basePaths []string
+ randomPath := func() string {
+ if len(basePaths) > 0 && rng.IntN(2) == 0 {
+ return basePaths[rng.IntN(len(basePaths))]
+ }
+ components := make([]string, 1+rng.IntN(3))
+ for i := range components {
+ components[i] = names[rng.IntN(len(names))]
+ }
+ return strings.Join(components, "/")
+ }
+
+ randomAction := func() commitAction {
+ switch rng.IntN(6) {
+ case 0:
+ return createFile{Path: randomPath(), OID: randomBlob().String(), ExecutableMode: rng.IntN(2) == 0}
+ case 1:
+ return updateFile{Path: randomPath(), OID: randomBlob().String()}
+ case 2:
+ action := moveFile{Path: randomPath(), NewPath: randomPath()}
+ if rng.IntN(2) == 0 {
+ action.OID = randomBlob().String()
+ }
+ return action
+ case 3:
+ return deleteFile{Path: randomPath()}
+ case 4:
+ return changeFileMode{Path: randomPath(), ExecutableMode: rng.IntN(2) == 0}
+ default:
+ return createDirectory{Path: randomPath()}
+ }
+ }
+
+ // underFile is whether a directory of path is a file or a submodule in root: Add would then
+ // add path below that entry, which no tree can hold.
+ underFile := func(root *localrepo.TreeEntry, path string) bool {
+ for dir := filepath.Dir(path); dir != "."; dir = filepath.Dir(dir) {
+ if entry, err := root.Get(dir); err == nil && entry.Type != localrepo.Tree {
+ return true
+ }
+ }
+ return false
+ }
+
+ // writeBaseTree writes a random valid tree with regular, executable, symlink and submodule
+ // entries, through today's writer.
+ writeBaseTree := func() git.ObjectID {
+ basePaths = nil
+ root := &localrepo.TreeEntry{Mode: "040000", Type: localrepo.Tree}
+ for range 4 + rng.IntN(12) {
+ path := randomPath()
+ if underFile(root, path) {
+ continue
+ }
+ entry := localrepo.TreeEntry{Path: filepath.Base(path), Type: localrepo.Blob, OID: randomBlob()}
+ switch rng.IntN(5) {
+ case 0:
+ entry.Mode = "100755"
+ case 1:
+ entry.Mode = "120000"
+ case 2:
+ entry.Mode, entry.Type, entry.OID = "160000", localrepo.Submodule, submodule
+ default:
+ entry.Mode = "100644"
+ }
+ if err := root.Add(path, entry); err != nil {
+ require.ErrorIs(t, err, localrepo.ErrEntryExists)
+ continue
+ }
+ basePaths = append(basePaths, path)
+ }
+ require.NoError(t, root.Write(ctx, repo))
+ return root.OID
+ }
+
+ reader, cancel, err := repo.RequestObjectReader(ctx)
+ require.NoError(t, err)
+ defer cancel()
+
+ outcomes := map[string]int{}
+ for sequence := range 120 {
+ baseOID := writeBaseTree()
+
+ recursive, err := repo.ReadTree(ctx, baseOID.Revision(), localrepo.WithRecursive())
+ require.NoError(t, err)
+ lazy, err := repo.ReadTreeLazily(ctx, reader, baseOID)
+ require.NoError(t, err)
+
+ var trace []string
+ for range 12 {
+ action := randomAction()
+ trace = append(trace, fmt.Sprintf("%T%+v", action, action))
+
+ recursiveErr := applyAction(ctx, action, recursive, repo)
+ lazyErr := applyAction(ctx, action, lazy, repo)
+ require.Equal(t, recursiveErr, lazyErr, "seed %d, sequence %d, base %s, actions %v", seed, sequence, baseOID, trace)
+
+ outcomes[outcome(action, recursiveErr)]++
+ }
+
+ // A file created below a file leaves a tree that no writer accepts: both must refuse it
+ // the same way, and write the same tree otherwise.
+ recursiveErr := recursive.Write(ctx, repo)
+ lazyErr := writeTreeEntries(ctx, repo, lazy)
+ require.Equal(t, fmt.Sprint(recursiveErr), fmt.Sprint(lazyErr), "seed %d, sequence %d, base %s, actions %v", seed, sequence, baseOID, trace)
+ require.Equal(t, recursiveErr == nil, lazyErr == nil, "seed %d, sequence %d", seed, sequence)
+ if recursiveErr != nil {
+ outcomes["write: "+recursiveErr.Error()]++
+ continue
+ }
+ outcomes["write ok"]++
+ require.Equal(t, recursive.OID, lazy.OID, "seed %d, sequence %d, base %s, actions %v", seed, sequence, baseOID, trace)
+ }
+
+ for _, expected := range []string{
+ "operations.createFile ok",
+ "operations.updateFile ok",
+ "operations.moveFile ok",
+ "operations.deleteFile ok",
+ "operations.changeFileMode ok",
+ "operations.createDirectory ok",
+ "operations.createFile at depth: A file with this name already exists",
+ "operations.createDirectory at depth: A file with this name already exists",
+ "operations.createDirectory at depth: A directory with this name already exists",
+ "operations.moveFile at depth: A file with this name already exists",
+ "operations.moveFile at depth: A file with this name doesn't exist",
+ "operations.updateFile at depth: A file with this name doesn't exist",
+ "operations.deleteFile at depth: A file with this name doesn't exist",
+ "operations.changeFileMode at depth: A file with this name doesn't exist",
+ "write ok",
+ "write: writing tree: exit status 128",
+ } {
+ require.Positive(t, outcomes[expected], "seed %d: outcome %q never occurred; outcomes: %v", seed, expected, outcomes)
+ }
+}
+
+// outcome names the outcome of an action: its type, and either ok or its error, with "at depth"
+// when the path that failed is below the top level.
+func outcome(action commitAction, err error) string {
+ if err == nil {
+ return fmt.Sprintf("%T ok", action)
+ }
+
+ var indexErr indexError
+ if errors.As(err, &indexErr) && strings.Contains(indexErr.path, "/") {
+ return fmt.Sprintf("%T at depth: %s", action, err)
+ }
+ return fmt.Sprintf("%T: %s", action, err)
+}
diff --git a/internal/gitaly/service/operations/commit_files_processes_test.go b/internal/gitaly/service/operations/commit_files_processes_test.go
new file mode 100644
index 0000000..55b27a2
--- /dev/null
+++ b/internal/gitaly/service/operations/commit_files_processes_test.go
@@ -0,0 +1,101 @@
+package operations
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/feder-cr/git-ai/internal/featureflag"
+ "github.com/feder-cr/git-ai/internal/git/gittest"
+ "github.com/feder-cr/git-ai/internal/testhelper"
+ "github.com/feder-cr/git-ai/internal/testhelper/testcfg"
+ "github.com/feder-cr/git-ai/internal/testhelper/testserver"
+)
+
+// TestUserCommitFilesProcessCount counts the git processes of one UserCommitFiles request, by
+// subcommand (PLAN9, WP96): one cat-file for every read, one hash-object per blob, one hash-object
+// for every tree, commit-tree and update-ref. A for-each-ref runs only when no branch resolved,
+// for the orphan commit into an empty repository. A process that comes back fails the test.
+func TestUserCommitFilesProcessCount(t *testing.T) {
+ t.Parallel()
+
+ for _, tc := range []struct {
+ desc string
+ emptyRepo bool
+ newBranch bool
+ paths []string
+ expected map[string]uint64
+ repoCreated bool
+ }{
+ {
+ desc: "existing branch, top level",
+ paths: []string{"file.txt"},
+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},
+ },
+ {
+ desc: "existing branch, nested",
+ paths: []string{"a/b/file.txt"},
+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},
+ },
+ {
+ desc: "new branch, top level",
+ newBranch: true,
+ paths: []string{"file.txt"},
+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},
+ },
+ {
+ desc: "new branch, nested",
+ newBranch: true,
+ paths: []string{"a/b/file.txt"},
+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "update-ref": 1},
+ },
+ {
+ desc: "three files",
+ paths: []string{"one.txt", "a/two.txt", "a/b/three.txt"},
+ expected: map[string]uint64{"cat-file": 1, "hash-object": 4, "commit-tree": 1, "update-ref": 1},
+ },
+ {
+ desc: "orphan commit into an empty repository",
+ emptyRepo: true,
+ paths: []string{"a/file.txt"},
+ expected: map[string]uint64{"cat-file": 1, "hash-object": 2, "commit-tree": 1, "for-each-ref": 1, "update-ref": 1},
+ repoCreated: true,
+ },
+ } {
+ t.Run(tc.desc, func(t *testing.T) {
+ t.Parallel()
+ ctx := featureflag.OutgoingCtxWithFeatureFlag(testhelper.Context(t), featureflag.GPGSigning, false)
+ cfg := testcfg.Build(t)
+ counting := gittest.NewCountingCommandFactory(t, cfg)
+ ctx, cfg, client := setupOperationsServiceWithCfg(t, ctx, cfg, testserver.WithGitCommandFactory(counting))
+
+ repo, repoPath := gittest.CreateRepository(t, ctx, cfg)
+ if !tc.emptyRepo {
+ gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("master"),
+ gittest.WithTreeEntries(gittest.TreeEntry{Path: "README", Mode: "100644", Content: "readme\n"}))
+ }
+
+ branch, start := "master", ""
+ if tc.newBranch {
+ branch, start = "feature", "master"
+ }
+
+ counting.ResetCount()
+ stream, err := client.UserCommitFiles(ctx)
+ require.NoError(t, err)
+ require.NoError(t, stream.Send(headerRequest(repo, gittest.TestUser, branch, []byte("message"), start, "")))
+ for i, path := range tc.paths {
+ require.NoError(t, stream.Send(createFileHeaderRequest(path)))
+ require.NoError(t, stream.Send(actionContentRequest(fmt.Sprintf("content %d\n", i))))
+ }
+ resp, err := stream.CloseAndRecv()
+ require.NoError(t, err)
+ require.Empty(t, resp.GetIndexError())
+ require.Empty(t, resp.GetPreReceiveError())
+ require.Equal(t, tc.repoCreated, resp.GetBranchUpdate().GetRepoCreated())
+
+ require.Equal(t, tc.expected, counting.Counts())
+ })
+ }
+}