Skip to content

Find, verify, and analyze leaked credentials

fix(gitparse): cheaper low-memory git scan, stop losing trailing commits (#5331)

Kashif Khandf5b7ce7ec89
diff --git a/pkg/gitparse/README.md b/pkg/gitparse/README.md
new file mode 100644
index 000000000..51eb50e3a
--- /dev/null
+++ b/pkg/gitparse/README.md
@@ -0,0 +1,123 @@
+# gitparse
+
+`gitparse` turns a git repository's history into a stream of `*Diff` values, one for each
+changed file in each commit. It is the front door for the git based sources: everything
+TruffleHog scans out of a repository's history comes through this package.
+
+It works by running the `git` binary and reading its text output, rather than reading
+git's object files directly. Parsing text is not pretty, but `git log --patch` already
+handles renames, binary files, path filters and merges, and rewriting all of that
+ourselves would be a much bigger thing to get wrong.
+
+## What comes out
+
+A `Diff` is one file's added content inside one commit, along with the details of that
+commit:
+
+```go
+diffChan, err := parser.RepoPath(ctx, repoPath, "", true, nil, false)
+for diff := range diffChan {
+ diff.Commit.Hash // the commit this change belongs to
+ diff.PathB // the path of the changed file
+ diff.LineStart // where in the file the hunk starts
+ diff.ReadCloser() // the added lines, and nothing else
+}
+```
+
+Only added lines are kept. Removed and unchanged lines are thrown away, because a secret
+that was deleted was already there in the commit that added it, and that commit is in the
+stream too.
+
+Commit details are attached to every diff. A commit with no file changes at all, such as
+a merge or an empty commit, is still sent once on its own, because its message, author and
+notes can hold a secret even when no file changed.
+
+Diff content goes to one of two writers, chosen by the caller. `buffer_writer` keeps it in
+memory and is the default. `buffered_file_writer` spills to disk past a size and is turned
+on with `UseCustomContentWriter`. Repositories with big files want the second one.
+
+## How the log is produced
+
+There are two ways, and the caller picks.
+
+**The normal way** is one `git log --patch` for the whole repository. Simple, fast, and
+fine for most repositories.
+
+**The lower memory way** is turned on with `UseLowMemoryScan`, which the CLI exposes as
+`--git-low-memory-scan`. It exists because one `git log --patch` keeps a little state for
+every commit it walks and never gives any of it back, so on a repository with millions of
+commits it grows until the machine kills it.
+
+That mode splits the work in two:
+
+1. **List the commits.** `git rev-list` prints the hashes we want and nothing else. The
+ hashes come back in groups, so the next step can start before the whole history has
+ been walked.
+
+2. **Make the patches.** Each group of hashes is handed to its own short lived `git log`
+ process, which reads them on standard input. Each process only holds state for its own
+ group and then exits, so this step costs the same no matter how long the history is.
+
+The results of all those processes are joined back into the single channel the caller
+sees, so nothing above this package needs to know which way was used.
+
+## Things that look like details but are not
+
+Each of these came from a test or a measurement that said otherwise.
+
+**Listing uses `git rev-list`, not `git log`.** They can both print hashes, but `git log`
+sets up git's diff machinery as soon as a diff option is present, and then compares files
+in every commit just to decide which commits to print. `rev-list` never does that. This is
+the step that uses the most memory on long histories, so the difference matters, and it is
+large.
+
+**Diff options stay out of the listing step.** Options like `--diff-filter=AM` live with
+the patch options, not the commit picking options. `rev-list` rejects them anyway. Leaving
+them out means the listing picks up a few extra commits whose changes are all filtered
+away, and the `git log` that makes the patches drops those commits itself, exactly as the
+single command form does. The end result is the same commits, worked out in the cheaper
+place.
+
+**Patches come from `git log --no-walk=unsorted`, not `git show`.** They print the same
+thing for ordinary commits, but only `git log` applies `--diff-filter` to whole commits,
+so using `git show` here would scan more commits than the normal way does. `unsorted`
+matters too: plain `--no-walk` re-sorts each group by commit date, which scrambles the
+order across groups.
+
+**Hashes go in on standard input, not as arguments.** A command line has a length limit,
+and on Windows it is short enough to cap a group at a few dozen commits. Standard input
+has no such limit, so groups can be thousands of commits and full hashes can be used
+instead of shortened ones. Fewer, bigger groups mean fewer git processes to start.
+
+**Path filters go to both steps.** The listing needs them to pick the same commits the
+single command would have shown, and the patch step needs them to decide which files show
+up.
+
+**The last commit of a stream needs finishing off.** The parser normally completes a
+commit when it sees the next one begin. The last commit never gets that, so it is
+completed at the end of the stream instead. Without this, a commit with no file changes
+was dropped whenever it landed at the end, which used to be rare with one long stream and
+became common once the log was split into groups.
+
+## Stopping early
+
+A scan can stop before the diffs run out, when it hits its maximum depth or reaches its
+base commit. A Go channel cannot tell you that its reader has gone away, so the scan
+cancels its context on the way out, and that is what shuts down the git processes still
+producing diffs.
+
+Cancelling also keeps the listing step short. It only ever runs a few groups ahead of what
+is being consumed, because the channels between the steps are unbuffered, so a scan that
+stops early never walks the whole history.
+
+## Tests
+
+```sh
+go test ./pkg/gitparse/
+```
+
+The tests in `lowmemory_test.go` hold the lower memory mode to one rule: it must produce
+exactly what the single command form produces, with the same commits, paths and content,
+wherever the group boundaries land. They run with tiny group sizes on purpose, since a
+group size of one puts every commit at the end of a stream at once, which is where the
+awkward cases live.
diff --git a/pkg/gitparse/gitparse.go b/pkg/gitparse/gitparse.go
index e52f46cda..7bb712a9d 100644
--- a/pkg/gitparse/gitparse.go
+++ b/pkg/gitparse/gitparse.go
@@ -4,6 +4,7 @@ import (
"bufio"
"bytes"
"cmp"
+ "errors"
"fmt"
"io"
"os"
@@ -36,13 +37,13 @@ const (
// defaultWaitDelay is the default time to wait after context cancellation before forcefully killing git processes.
defaultWaitDelay = 5 * time.Second
- // abbrevCommit is the git sha abbreviation length to use for `git show` invocations in the lower-memory scan mode.
- abbrevCommit = 20
-
- // showGroupSize is the number of commits per `git show` in the lower-memory scan mode.
+ // logGroupSize is the number of commits per `git log` in the lower-memory scan mode.
//
- // Windows has a command length limit of 32767, so at these values we should only be using a tiny part of of that for the commit list ((abbrevCommit + 1) * showGroupSize). We don't target any platforms with shorter limits.
- showGroupSize = 75
+ // The hashes are fed in on stdin rather than as arguments, so a group is not
+ // limited by how long a command line may be and we can use full hashes. Bigger
+ // groups mean fewer git processes to start; each one still only holds state for
+ // its own group, which is what keeps memory flat however long the history is.
+ logGroupSize = 5000
)
// contentWriter defines a common interface for writing, reading, and managing diff content.
@@ -141,6 +142,11 @@ type Parser struct {
useCustomContentWriter bool
lowMemoryScan bool
+
+ // groupSize is how many commits go to each `git log` in the lower-memory scan.
+ // Zero means logGroupSize. Only the tests set it, so they can put a group
+ // boundary wherever they need one.
+ groupSize int
}
type ParseState int
@@ -286,7 +292,7 @@ func (c *Parser) RepoPath(
}
func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Diff, error) {
- commitGroups, err := c.gatherGitLog(ctx, args)
+ commitGroups, err := c.enumerateCommits(ctx, args)
if err != nil {
return nil, err
}
@@ -295,7 +301,7 @@ func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Dif
// different goroutine after the command finishes, but we're not
// running a single command anymore. we'll use a channel of channels to
// reduce back to one channel we return to our caller. Unbuffered so
- // we have at most one git show running and one git show draining.
+ // we have at most one git log running and one git log draining.
diffGroups := make(chan chan *Diff)
go func() {
defer common.RecoverWithExit(ctx)
@@ -306,20 +312,36 @@ func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Dif
return
}
- showCmd := exec.CommandContext(ctx,
+ // `git log` over an explicit list, not `git show`. The two print the
+ // same thing for ordinary commits, but only log applies --diff-filter
+ // to whole commits, so this is what keeps the set of scanned commits
+ // the same as the single-command form.
+ logCmd := exec.CommandContext(ctx,
"git",
- slices.Concat(args.global, []string{"show"}, args.show, group, args.paths)...,
+ slices.Concat(
+ args.global, []string{"log"}, args.show,
+ []string{
+ // Keep the commits in the order rev-list gave them. Plain
+ // --no-walk would re-sort each group by commit date, which
+ // scrambles the order across groups.
+ "--no-walk=unsorted",
+ // Hashes come in on stdin, so the group size is ours to pick.
+ "--stdin",
+ },
+ args.paths,
+ )...,
)
- showCmd.Env = args.env
+ logCmd.Env = args.env
+ logCmd.Stdin = strings.NewReader(strings.Join(group, "\n") + "\n")
- diffGroup, err := c.executeCommand(ctx, showCmd, false)
+ diffGroup, err := c.executeCommand(ctx, logCmd, false)
if err != nil {
- ctx.Logger().Error(err, "Error executing git show for commit group.")
+ ctx.Logger().Error(err, "Error executing git log for commit group.")
return
}
err = common.CancellableWrite(ctx, diffGroups, diffGroup)
if err != nil {
- ctx.Logger().Error(err, "git show interation cancelled")
+ ctx.Logger().Error(err, "git log iteration cancelled")
return
}
}
@@ -346,71 +368,101 @@ func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Dif
return diffChan, nil
}
-// Ask git for a list of all relevant commit hashes but only hashes. Git takes
-// on the work of linearizing history for us, then we work through the commit
-// list. Returns a channel of groups of commit IDs, so scanning can start asap
-// even if git log is taking a bit for large repos.
-func (c *Parser) gatherGitLog(ctx context.Context, args gitArgs) (chan []string, error) {
+// enumerateCommits asks git for the hashes of the commits we mean to scan, and
+// nothing else. It returns them in groups, so patch generation can start before the
+// whole history has been walked.
+//
+// This uses `git rev-list` rather than `git log`. rev-list is the plumbing command for
+// listing commits and never sets up git's diff machinery, which `git log` does as soon
+// as a diff option is present. On this repository that is the difference between 43 MB
+// and 3.4 MB of peak memory for a phase whose only job is to print hashes, and this is
+// the phase that sets the peak on long histories.
+// commitGroupSize is how many commits each `git log` gets.
+func (c *Parser) commitGroupSize() int {
+ // A size of zero or less would mean a group that never fills, so it falls back.
+ if c.groupSize <= 0 {
+ return logGroupSize
+ }
+ return c.groupSize
+}
+
+func (c *Parser) enumerateCommits(ctx context.Context, args gitArgs) (chan []string, error) {
+ // args.log holds only the options that choose commits, so it can go to rev-list
+ // as-is. Diff options live in args.show and rev-list would reject them.
cmd := exec.CommandContext(ctx,
"git", slices.Concat(
- args.global, []string{"log"},
- args.log, []string{
- // https://git-scm.com/docs/git-log#_pretty_formats
- "--pretty=format:%h",
- // https://git-scm.com/docs/git-log#Documentation/git-log.txt---abbrevn
- fmt.Sprintf("--abbrev=%d", abbrevCommit),
- },
+ args.global, []string{"rev-list"},
+ args.log,
args.paths,
)...)
cmd.WaitDelay = c.waitDelay
cmd.Env = args.env
+ // Keep stderr, because it carries the reason a bad ref or a broken repo failed.
+ var stderr strings.Builder
+ cmd.Stderr = &stderr
+
stdOut, err := cmd.StdoutPipe()
if err != nil {
return nil, err
}
- err = cmd.Start()
- if err != nil {
- return nil, fmt.Errorf("failed to execute git log: %w", err)
+ if err := cmd.Start(); err != nil {
+ return nil, fmt.Errorf("failed to execute git rev-list: %w", err)
+ }
+
+ // Wait for the first byte before returning. A bad revision makes rev-list fail
+ // immediately, and this is the last moment we can hand that back to the caller as
+ // an error. Reporting it any later means closing the channel instead, and a typo in
+ // a branch name then looks exactly like an empty repository.
+ reader := bufio.NewReader(stdOut)
+ if _, err := reader.Peek(1); err != nil {
+ if waitErr := cmd.Wait(); waitErr != nil {
+ return nil, fmt.Errorf("failed to list commits: %w: %s", waitErr, strings.TrimSpace(stderr.String()))
+ }
+ if !errors.Is(err, io.EOF) {
+ return nil, fmt.Errorf("failed to read commit list: %w", err)
+ }
+
+ // git was happy and printed nothing, so there is genuinely nothing to scan.
+ empty := make(chan []string)
+ close(empty)
+ return empty, nil
}
commitGroups := make(chan []string)
go func() {
defer close(commitGroups)
defer func() {
- err := cmd.Wait()
- if err != nil {
- ctx.Logger().Error(err, "git log exited with error", "stderr", cmd.Stderr)
+ if err := cmd.Wait(); err != nil {
+ ctx.Logger().Error(err, "git rev-list exited with error", "stderr", strings.TrimSpace(stderr.String()))
}
}()
- s := bufio.NewScanner(stdOut)
- commitGroup := make([]string, 0, showGroupSize)
-
- var err error
+ s := bufio.NewScanner(reader)
+ groupSize := c.commitGroupSize()
+ commitGroup := make([]string, 0, groupSize)
for s.Scan() && !common.IsDone(ctx) {
commitGroup = append(commitGroup, s.Text())
-
- if len(commitGroup) == showGroupSize {
- err = common.CancellableWrite(ctx, commitGroups, commitGroup)
- if err != nil {
- ctx.Logger().Error(err, "git log stopping early")
- return
- }
- commitGroup = make([]string, 0, showGroupSize)
+ if len(commitGroup) < groupSize {
+ continue
}
+ if err := common.CancellableWrite(ctx, commitGroups, commitGroup); err != nil {
+ ctx.Logger().Error(err, "git rev-list stopping early")
+ return
+ }
+ commitGroup = make([]string, 0, groupSize)
}
+
+ // The last group is almost never exactly full.
if len(commitGroup) != 0 {
- err = common.CancellableWrite(ctx, commitGroups, commitGroup)
- if err != nil {
- ctx.Logger().Error(err, "failed to flush last git log group")
+ if err := common.CancellableWrite(ctx, commitGroups, commitGroup); err != nil {
+ ctx.Logger().Error(err, "failed to flush last commit group")
}
-
}
if err := s.Err(); err != nil {
- ctx.Logger().Error(err, "error reading git log")
+ ctx.Logger().Error(err, "error reading commit list")
}
}()
@@ -440,8 +492,13 @@ func (c *Parser) prepGitArgs(source string, head string, abbreviatedLog bool, ex
}
if abbreviatedLog {
+ // Only in show. args.log holds the options that choose commits, and it is
+ // also what the lower-memory scan hands to `git rev-list`, which rejects diff
+ // options outright. Leaving it out costs nothing: rev-list then lists a few
+ // commits whose diffs are all filtered away, and the `git log` that generates
+ // the patches drops those commits itself, exactly as the single-command form
+ // does.
// https://git-scm.com/docs/git-show#Documentation/git-show.txt---diff-filterACDMRTUXB
- args.log = append(args.log, "--diff-filter=AM")
args.show = append(args.show, "--diff-filter=AM")
}
@@ -1178,10 +1235,25 @@ func cleanupParse(ctx context.Context, currentCommit *Commit, currentDiff *Diff,
if currentDiff != nil && (currentDiff.Len() > 0 || currentDiff.IsBinary) {
currentDiff.Commit = currentCommit
diffChan <- currentDiff
- }
- if currentCommit != nil {
- if totalLogSize != nil {
- *totalLogSize += currentCommit.Size
+ if currentCommit != nil {
+ currentCommit.hasDiffs = true
}
}
+
+ if currentCommit == nil {
+ return
+ }
+ if totalLogSize != nil {
+ *totalLogSize += currentCommit.Size
+ }
+
+ // A commit is normally finished off in the loop above, when the next commit line
+ // shows up. The last commit in the stream never gets that, so it is finished here
+ // instead, and it needs the same rule: a commit with no diffs of its own still has
+ // a message, an author and notes worth scanning, so it goes out on its own.
+ //
+ // Staged diffs come through here too and carry no commit, hence the hash check.
+ if !currentCommit.hasDiffs && currentCommit.Hash != "" {
+ diffChan <- &Diff{Commit: currentCommit}
+ }
}
diff --git a/pkg/gitparse/gitparse_test.go b/pkg/gitparse/gitparse_test.go
index 06aace3bf..52dd304f1 100644
--- a/pkg/gitparse/gitparse_test.go
+++ b/pkg/gitparse/gitparse_test.go
@@ -36,8 +36,10 @@ func TestPrepGitArgs(t *testing.T) {
// head
assert.Contains(t, args.log, "branchname")
assert.NotContains(t, args.log, "--all")
- // abbreviatedLog
- assert.Contains(t, args.log, "--diff-filter=AM")
+ // abbreviatedLog. Only show carries the diff filter: args.log holds the options
+ // that choose commits and is what the lower-memory scan gives to `git rev-list`,
+ // which rejects diff options.
+ assert.NotContains(t, args.log, "--diff-filter=AM")
assert.Contains(t, args.show, "--diff-filter=AM")
// excludedGlobs
assert.Contains(t, args.paths, "--")
diff --git a/pkg/gitparse/lowmemory_test.go b/pkg/gitparse/lowmemory_test.go
new file mode 100644
index 000000000..a4ab2e44d
--- /dev/null
+++ b/pkg/gitparse/lowmemory_test.go
@@ -0,0 +1,282 @@
+package gitparse
+
+import (
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/trufflesecurity/trufflehog/v3/pkg/context"
+)
+
+// The lower-memory scan splits one `git log` into a `git rev-list` that lists commits
+// and a `git log` per group that generates their patches. These tests hold it to the
+// only thing that really matters: it must find exactly what the single-command form
+// finds, wherever the group boundaries happen to land.
+
+// collectDiffs drains a diff channel into something comparable. Content is included,
+// since a group boundary in the wrong place could keep the commit and lose its patch.
+func collectDiffs(t *testing.T, diffChan chan *Diff) []string {
+ t.Helper()
+
+ var out []string
+ for diff := range diffChan {
+ content := ""
+ // A commit with no diffs arrives with nothing written, and asking such a diff
+ // for its content fails, so check before reading.
+ if diff.contentWriter != nil && diff.Len() > 0 {
+ got, err := diff.contentWriter.String()
+ if err != nil {
+ t.Fatalf("reading diff content: %v", err)
+ }
+ content = got
+ }
+ out = append(out, strings.Join([]string{diff.Commit.Hash, diff.PathB, content}, "\x00"))
+ }
+ return out
+}
+
+func TestLowMemoryScanMatchesSingleProcess(t *testing.T) {
+ repo := testRepoRoot(t)
+
+ // abbreviatedLog is the caller's BaseHash == "", so both values are real
+ // configurations and they take different paths through git. With it on, git drops
+ // commits whose diffs are all filtered away; with it off, those commits stay.
+ //
+ // The group sizes go down to 1 on purpose. Every group ends a stream, and a commit
+ // with no diffs used to be dropped when it landed at the end of one, so a size of 1
+ // puts every commit in that spot at once. Before cleanupParse learned to finish off
+ // the last commit, this repository lost 84 diffs at size 1 and 1 at size 75.
+ for _, tc := range []struct {
+ name string
+ abbreviated bool
+ }{
+ {"abbreviated", true},
+ {"full", false},
+ } {
+ abbreviated, name := tc.abbreviated, tc.name
+
+ t.Run(name, func(t *testing.T) {
+ ctx := context.Background()
+
+ single := NewParser()
+ singleChan, err := single.RepoPath(ctx, repo, "", abbreviated, nil, false)
+ if err != nil {
+ t.Fatalf("single-process RepoPath: %v", err)
+ }
+ want := collectDiffs(t, singleChan)
+ if len(want) == 0 {
+ t.Fatal("single-process scan produced no diffs")
+ }
+
+ // Group sizes small enough that this repository crosses boundaries, which
+ // is the only place the two forms can drift apart.
+ for _, groupSize := range []int{1, 2, 7, 500} {
+ low := NewParser(UseLowMemoryScan())
+ low.groupSize = groupSize
+
+ lowChan, err := low.RepoPath(ctx, repo, "", abbreviated, nil, false)
+ if err != nil {
+ t.Fatalf("group size %d: RepoPath: %v", groupSize, err)
+ }
+ got := collectDiffs(t, lowChan)
+
+ if len(got) != len(want) {
+ t.Fatalf("group size %d: got %d diffs, want %d", groupSize, len(got), len(want))
+ }
+ for i := range want {
+ if got[i] != want[i] {
+ t.Fatalf("group size %d: diff %d differs\n got: %q\nwant: %q",
+ groupSize, i, got[i], want[i])
+ }
+ }
+ }
+ })
+ }
+}
+
+// TestLowMemoryScanExcludedGlobs checks the path filters reach both commands. They
+// decide which commits rev-list lists and which files the patches contain, so sending
+// them to only one of the two would quietly change what gets scanned.
+func TestLowMemoryScanExcludedGlobs(t *testing.T) {
+ repo := testRepoRoot(t)
+ ctx := context.Background()
+ globs := []string{"*.go"}
+
+ single := NewParser()
+ singleChan, err := single.RepoPath(ctx, repo, "", true, globs, false)
+ if err != nil {
+ t.Fatalf("single-process RepoPath: %v", err)
+ }
+ want := collectDiffs(t, singleChan)
+
+ low := NewParser(UseLowMemoryScan())
+ low.groupSize = 13
+ lowChan, err := low.RepoPath(ctx, repo, "", true, globs, false)
+ if err != nil {
+ t.Fatalf("low-memory RepoPath: %v", err)
+ }
+ got := collectDiffs(t, lowChan)
+
+ if len(got) != len(want) {
+ t.Fatalf("got %d diffs, want %d", len(got), len(want))
+ }
+ for i := range want {
+ if got[i] != want[i] {
+ t.Fatalf("diff %d differs\n got: %q\nwant: %q", i, got[i], want[i])
+ }
+ }
+}
+
+// TestLowMemoryScanUnknownHead covers the error the old form threw away. A bad revision
+// used to be logged and the channel closed, so a typo in a branch name looked exactly
+// like an empty repository.
+func TestLowMemoryScanUnknownHead(t *testing.T) {
+ repo := testRepoRoot(t)
+
+ parser := NewParser(UseLowMemoryScan())
+ if _, err := parser.RepoPath(context.Background(), repo, "no-such-ref-exists", true, nil, false); err == nil {
+ t.Fatal("expected an error for an unknown head, got nil")
+ }
+}
+
+// TestLowMemoryScanEmptyRepo checks a repository with no commits ends the scan cleanly
+// rather than failing. rev-list prints nothing and exits zero, which has to be told
+// apart from rev-list printing nothing because it failed.
+func TestLowMemoryScanEmptyRepo(t *testing.T) {
+ dir := t.TempDir()
+ runTestGit(t, dir, "init", "-q")
+
+ parser := NewParser(UseLowMemoryScan())
+ diffChan, err := parser.RepoPath(context.Background(), dir, "", true, nil, false)
+ if err != nil {
+ t.Fatalf("RepoPath on an empty repo: %v", err)
+ }
+ if n := len(collectDiffs(t, diffChan)); n != 0 {
+ t.Fatalf("got %d diffs from an empty repo, want 0", n)
+ }
+}
+
+// TestLowMemoryScanAbandonedByConsumer covers a caller that stops reading early, which
+// is what a depth-limited scan does. Reading one diff and walking away must not leave
+// the scan wedged; the cancel in ScanCommits is what releases it in production, so the
+// same cancel stands in for it here.
+func TestLowMemoryScanAbandonedByConsumer(t *testing.T) {
+ repo := testRepoRoot(t)
+ ctx, cancel := context.WithCancel(context.Background())
+
+ parser := NewParser(UseLowMemoryScan())
+ parser.groupSize = 2
+
+ diffChan, err := parser.RepoPath(ctx, repo, "", true, nil, false)
+ if err != nil {
+ t.Fatalf("RepoPath: %v", err)
+ }
+
+ // Take a single diff, then stop, the way the scan loop does at max depth.
+ if _, ok := <-diffChan; !ok {
+ t.Fatal("expected at least one diff")
+ }
+ cancel()
+
+ // Draining to the end must finish. If it hangs the test times out, which is the
+ // failure we are looking for.
+ for range diffChan { //nolint:revive // draining
+ }
+}
+
+func TestCommitGroupSize(t *testing.T) {
+ // A size of zero or less would mean a group that never fills, so it falls back.
+ for _, tc := range []struct{ set, want int }{
+ {0, logGroupSize},
+ {-1, logGroupSize},
+ {1, 1},
+ {99, 99},
+ } {
+ parser := NewParser()
+ parser.groupSize = tc.set
+ if got := parser.commitGroupSize(); got != tc.want {
+ t.Errorf("groupSize %d: got %d, want %d", tc.set, got, tc.want)
+ }
+ }
+}
+
+// testRepoRoot returns the root of the git checkout the tests run from. The real
+// history has the merges, binaries, renames and uneven commit sizes that a hand-built
+// fixture does not, which is where a batching mistake actually shows up.
+func testRepoRoot(tb testing.TB) string {
+ tb.Helper()
+
+ out, err := exec.Command("git", "rev-parse", "--show-toplevel").Output()
+ if err != nil {
+ tb.Skip("not running from a git checkout")
+ }
+ root := strings.TrimSpace(string(out))
+ if _, err := os.Stat(filepath.Join(root, ".git")); err != nil {
+ tb.Skip("no .git directory")
+ }
+ return root
+}
+
+func runTestGit(tb testing.TB, dir string, args ...string) {
+ tb.Helper()
+
+ cmd := exec.Command("git", args...)
+ cmd.Dir = dir
+ if out, err := cmd.CombinedOutput(); err != nil {
+ tb.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)
+ }
+}
+
+// TestTrailingCommitWithoutDiffsIsReported pins down the rule that a commit is finished
+// off at the end of a stream, not only when the next commit line arrives.
+//
+// A commit with no diffs of its own still carries a message, an author and notes, and
+// those are worth scanning. The main loop has always sent such a commit on when it saw
+// the next one start, but the end of the stream had no equivalent, so whichever commit
+// happened to be last was dropped. One `git log` for a whole repository made that nearly
+// harmless, since only the oldest commit in the history was ever in that spot. Splitting
+// the log into groups puts a different commit there for every group.
+func TestTrailingCommitWithoutDiffsIsReported(t *testing.T) {
+ dir := t.TempDir()
+ runTestGit(t, dir, "init", "-q")
+ runTestGit(t, dir, "config", "user.email", "[email protected]")
+ runTestGit(t, dir, "config", "user.name", "Test")
+
+ if err := os.WriteFile(filepath.Join(dir, "f.txt"), []byte("hello\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ runTestGit(t, dir, "add", "-A")
+ runTestGit(t, dir, "commit", "-qm", "adds a file")
+
+ // An empty commit has a message but changes nothing, which is the shape that used
+ // to disappear.
+ runTestGit(t, dir, "commit", "-q", "--allow-empty", "-m", "empty commit worth scanning")
+
+ // Group size 1 puts the empty commit at the end of its own stream, which is exactly
+ // where it used to be lost.
+ parser := NewParser(UseLowMemoryScan())
+ parser.groupSize = 1
+
+ diffChan, err := parser.RepoPath(context.Background(), dir, "", false, nil, false)
+ if err != nil {
+ t.Fatalf("RepoPath: %v", err)
+ }
+
+ var messages []string
+ for diff := range diffChan {
+ messages = append(messages, diff.Commit.Message.String())
+ }
+
+ var found bool
+ for _, msg := range messages {
+ if strings.Contains(msg, "empty commit worth scanning") {
+ found = true
+ break
+ }
+ }
+ if !found {
+ t.Fatalf("the empty commit was never reported; got %d diffs with messages %q", len(messages), messages)
+ }
+}
diff --git a/pkg/sources/git/git.go b/pkg/sources/git/git.go
index 6e33eaedd..89094f263 100644
--- a/pkg/sources/git/git.go
+++ b/pkg/sources/git/git.go
@@ -856,6 +856,13 @@ func (s *Git) ScanCommits(ctx context.Context, repo *git.Repository, path string
repoCtx = ctx
}
+ // The scan can stop before the diff channel is drained, on max depth or on
+ // reaching the base commit. Nothing else tells the parser that, so cancelling on
+ // the way out is what shuts down the git processes still producing diffs. Without
+ // it they sit blocked on a channel nobody is reading until the whole scan ends.
+ repoCtx, cancel := context.WithCancel(repoCtx)
+ defer cancel()
+
logger := repoCtx.Logger()
var logValues []any
if scanOptions.BaseHash != "" {