Find, verify, and analyze leaked credentials
Go credentialsdevsecopsdynamic-analysisprecommitscanningsecretsecret-managementsecretssecuritysecurity-toolstrufflehogverification
fix(gitparse): cheaper low-memory git scan, stop losing trailing commits (#5331)
Kashif Khandf5b7ce7ec89
diff --git a/pkg/gitparse/README.md b/pkg/gitparse/README.mdnew file mode 100644index 000000000..51eb50e3a--- /dev/null+++ b/pkg/gitparse/README.md@@ -0,0 +1,123 @@+# gitparse++`gitparse` turns a git repository's history into a stream of `*Diff` values, one for each+changed file in each commit. It is the front door for the git based sources: everything+TruffleHog scans out of a repository's history comes through this package.++It works by running the `git` binary and reading its text output, rather than reading+git's object files directly. Parsing text is not pretty, but `git log --patch` already+handles renames, binary files, path filters and merges, and rewriting all of that+ourselves would be a much bigger thing to get wrong.++## What comes out++A `Diff` is one file's added content inside one commit, along with the details of that+commit:++```go+diffChan, err := parser.RepoPath(ctx, repoPath, "", true, nil, false)+for diff := range diffChan {+ diff.Commit.Hash // the commit this change belongs to+ diff.PathB // the path of the changed file+ diff.LineStart // where in the file the hunk starts+ diff.ReadCloser() // the added lines, and nothing else+}+```++Only added lines are kept. Removed and unchanged lines are thrown away, because a secret+that was deleted was already there in the commit that added it, and that commit is in the+stream too.++Commit details are attached to every diff. A commit with no file changes at all, such as+a merge or an empty commit, is still sent once on its own, because its message, author and+notes can hold a secret even when no file changed.++Diff content goes to one of two writers, chosen by the caller. `buffer_writer` keeps it in+memory and is the default. `buffered_file_writer` spills to disk past a size and is turned+on with `UseCustomContentWriter`. Repositories with big files want the second one.++## How the log is produced++There are two ways, and the caller picks.++**The normal way** is one `git log --patch` for the whole repository. Simple, fast, and+fine for most repositories.++**The lower memory way** is turned on with `UseLowMemoryScan`, which the CLI exposes as+`--git-low-memory-scan`. It exists because one `git log --patch` keeps a little state for+every commit it walks and never gives any of it back, so on a repository with millions of+commits it grows until the machine kills it.++That mode splits the work in two:++1. **List the commits.** `git rev-list` prints the hashes we want and nothing else. The+ hashes come back in groups, so the next step can start before the whole history has+ been walked.++2. **Make the patches.** Each group of hashes is handed to its own short lived `git log`+ process, which reads them on standard input. Each process only holds state for its own+ group and then exits, so this step costs the same no matter how long the history is.++The results of all those processes are joined back into the single channel the caller+sees, so nothing above this package needs to know which way was used.++## Things that look like details but are not++Each of these came from a test or a measurement that said otherwise.++**Listing uses `git rev-list`, not `git log`.** They can both print hashes, but `git log`+sets up git's diff machinery as soon as a diff option is present, and then compares files+in every commit just to decide which commits to print. `rev-list` never does that. This is+the step that uses the most memory on long histories, so the difference matters, and it is+large.++**Diff options stay out of the listing step.** Options like `--diff-filter=AM` live with+the patch options, not the commit picking options. `rev-list` rejects them anyway. Leaving+them out means the listing picks up a few extra commits whose changes are all filtered+away, and the `git log` that makes the patches drops those commits itself, exactly as the+single command form does. The end result is the same commits, worked out in the cheaper+place.++**Patches come from `git log --no-walk=unsorted`, not `git show`.** They print the same+thing for ordinary commits, but only `git log` applies `--diff-filter` to whole commits,+so using `git show` here would scan more commits than the normal way does. `unsorted`+matters too: plain `--no-walk` re-sorts each group by commit date, which scrambles the+order across groups.++**Hashes go in on standard input, not as arguments.** A command line has a length limit,+and on Windows it is short enough to cap a group at a few dozen commits. Standard input+has no such limit, so groups can be thousands of commits and full hashes can be used+instead of shortened ones. Fewer, bigger groups mean fewer git processes to start.++**Path filters go to both steps.** The listing needs them to pick the same commits the+single command would have shown, and the patch step needs them to decide which files show+up.++**The last commit of a stream needs finishing off.** The parser normally completes a+commit when it sees the next one begin. The last commit never gets that, so it is+completed at the end of the stream instead. Without this, a commit with no file changes+was dropped whenever it landed at the end, which used to be rare with one long stream and+became common once the log was split into groups.++## Stopping early++A scan can stop before the diffs run out, when it hits its maximum depth or reaches its+base commit. A Go channel cannot tell you that its reader has gone away, so the scan+cancels its context on the way out, and that is what shuts down the git processes still+producing diffs.++Cancelling also keeps the listing step short. It only ever runs a few groups ahead of what+is being consumed, because the channels between the steps are unbuffered, so a scan that+stops early never walks the whole history.++## Tests++```sh+go test ./pkg/gitparse/+```++The tests in `lowmemory_test.go` hold the lower memory mode to one rule: it must produce+exactly what the single command form produces, with the same commits, paths and content,+wherever the group boundaries land. They run with tiny group sizes on purpose, since a+group size of one puts every commit at the end of a stream at once, which is where the+awkward cases live.diff --git a/pkg/gitparse/gitparse.go b/pkg/gitparse/gitparse.goindex e52f46cda..7bb712a9d 100644--- a/pkg/gitparse/gitparse.go+++ b/pkg/gitparse/gitparse.go@@ -4,6 +4,7 @@ import ("bufio""bytes""cmp"+ "errors""fmt""io""os"@@ -36,13 +37,13 @@ const (// defaultWaitDelay is the default time to wait after context cancellation before forcefully killing git processes.defaultWaitDelay = 5 * time.Second- // abbrevCommit is the git sha abbreviation length to use for `git show` invocations in the lower-memory scan mode.- abbrevCommit = 20-- // showGroupSize is the number of commits per `git show` in the lower-memory scan mode.+ // logGroupSize is the number of commits per `git log` in the lower-memory scan mode.//- // Windows has a command length limit of 32767, so at these values we should only be using a tiny part of of that for the commit list ((abbrevCommit + 1) * showGroupSize). We don't target any platforms with shorter limits.- showGroupSize = 75+ // The hashes are fed in on stdin rather than as arguments, so a group is not+ // limited by how long a command line may be and we can use full hashes. Bigger+ // groups mean fewer git processes to start; each one still only holds state for+ // its own group, which is what keeps memory flat however long the history is.+ logGroupSize = 5000)// contentWriter defines a common interface for writing, reading, and managing diff content.@@ -141,6 +142,11 @@ type Parser struct {useCustomContentWriter boollowMemoryScan bool++ // groupSize is how many commits go to each `git log` in the lower-memory scan.+ // Zero means logGroupSize. Only the tests set it, so they can put a group+ // boundary wherever they need one.+ groupSize int}type ParseState int@@ -286,7 +292,7 @@ func (c *Parser) RepoPath(}func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Diff, error) {- commitGroups, err := c.gatherGitLog(ctx, args)+ commitGroups, err := c.enumerateCommits(ctx, args)if err != nil {return nil, err}@@ -295,7 +301,7 @@ func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Dif// different goroutine after the command finishes, but we're not// running a single command anymore. we'll use a channel of channels to// reduce back to one channel we return to our caller. Unbuffered so- // we have at most one git show running and one git show draining.+ // we have at most one git log running and one git log draining.diffGroups := make(chan chan *Diff)go func() {defer common.RecoverWithExit(ctx)@@ -306,20 +312,36 @@ func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Difreturn}- showCmd := exec.CommandContext(ctx,+ // `git log` over an explicit list, not `git show`. The two print the+ // same thing for ordinary commits, but only log applies --diff-filter+ // to whole commits, so this is what keeps the set of scanned commits+ // the same as the single-command form.+ logCmd := exec.CommandContext(ctx,"git",- slices.Concat(args.global, []string{"show"}, args.show, group, args.paths)...,+ slices.Concat(+ args.global, []string{"log"}, args.show,+ []string{+ // Keep the commits in the order rev-list gave them. Plain+ // --no-walk would re-sort each group by commit date, which+ // scrambles the order across groups.+ "--no-walk=unsorted",+ // Hashes come in on stdin, so the group size is ours to pick.+ "--stdin",+ },+ args.paths,+ )...,)- showCmd.Env = args.env+ logCmd.Env = args.env+ logCmd.Stdin = strings.NewReader(strings.Join(group, "\n") + "\n")- diffGroup, err := c.executeCommand(ctx, showCmd, false)+ diffGroup, err := c.executeCommand(ctx, logCmd, false)if err != nil {- ctx.Logger().Error(err, "Error executing git show for commit group.")+ ctx.Logger().Error(err, "Error executing git log for commit group.")return}err = common.CancellableWrite(ctx, diffGroups, diffGroup)if err != nil {- ctx.Logger().Error(err, "git show interation cancelled")+ ctx.Logger().Error(err, "git log iteration cancelled")return}}@@ -346,71 +368,101 @@ func (c *Parser) repoPathLowMemory(ctx context.Context, args gitArgs) (chan *Difreturn diffChan, nil}-// Ask git for a list of all relevant commit hashes but only hashes. Git takes-// on the work of linearizing history for us, then we work through the commit-// list. Returns a channel of groups of commit IDs, so scanning can start asap-// even if git log is taking a bit for large repos.-func (c *Parser) gatherGitLog(ctx context.Context, args gitArgs) (chan []string, error) {+// enumerateCommits asks git for the hashes of the commits we mean to scan, and+// nothing else. It returns them in groups, so patch generation can start before the+// whole history has been walked.+//+// This uses `git rev-list` rather than `git log`. rev-list is the plumbing command for+// listing commits and never sets up git's diff machinery, which `git log` does as soon+// as a diff option is present. On this repository that is the difference between 43 MB+// and 3.4 MB of peak memory for a phase whose only job is to print hashes, and this is+// the phase that sets the peak on long histories.+// commitGroupSize is how many commits each `git log` gets.+func (c *Parser) commitGroupSize() int {+ // A size of zero or less would mean a group that never fills, so it falls back.+ if c.groupSize <= 0 {+ return logGroupSize+ }+ return c.groupSize+}++func (c *Parser) enumerateCommits(ctx context.Context, args gitArgs) (chan []string, error) {+ // args.log holds only the options that choose commits, so it can go to rev-list+ // as-is. Diff options live in args.show and rev-list would reject them.cmd := exec.CommandContext(ctx,"git", slices.Concat(- args.global, []string{"log"},- args.log, []string{- // https://git-scm.com/docs/git-log#_pretty_formats- "--pretty=format:%h",- // https://git-scm.com/docs/git-log#Documentation/git-log.txt---abbrevn- fmt.Sprintf("--abbrev=%d", abbrevCommit),- },+ args.global, []string{"rev-list"},+ args.log,args.paths,)...)cmd.WaitDelay = c.waitDelaycmd.Env = args.env+ // Keep stderr, because it carries the reason a bad ref or a broken repo failed.+ var stderr strings.Builder+ cmd.Stderr = &stderr+stdOut, err := cmd.StdoutPipe()if err != nil {return nil, err}- err = cmd.Start()- if err != nil {- return nil, fmt.Errorf("failed to execute git log: %w", err)+ if err := cmd.Start(); err != nil {+ return nil, fmt.Errorf("failed to execute git rev-list: %w", err)+ }++ // Wait for the first byte before returning. A bad revision makes rev-list fail+ // immediately, and this is the last moment we can hand that back to the caller as+ // an error. Reporting it any later means closing the channel instead, and a typo in+ // a branch name then looks exactly like an empty repository.+ reader := bufio.NewReader(stdOut)+ if _, err := reader.Peek(1); err != nil {+ if waitErr := cmd.Wait(); waitErr != nil {+ return nil, fmt.Errorf("failed to list commits: %w: %s", waitErr, strings.TrimSpace(stderr.String()))+ }+ if !errors.Is(err, io.EOF) {+ return nil, fmt.Errorf("failed to read commit list: %w", err)+ }++ // git was happy and printed nothing, so there is genuinely nothing to scan.+ empty := make(chan []string)+ close(empty)+ return empty, nil}commitGroups := make(chan []string)go func() {defer close(commitGroups)defer func() {- err := cmd.Wait()- if err != nil {- ctx.Logger().Error(err, "git log exited with error", "stderr", cmd.Stderr)+ if err := cmd.Wait(); err != nil {+ ctx.Logger().Error(err, "git rev-list exited with error", "stderr", strings.TrimSpace(stderr.String()))}}()- s := bufio.NewScanner(stdOut)- commitGroup := make([]string, 0, showGroupSize)-- var err error+ s := bufio.NewScanner(reader)+ groupSize := c.commitGroupSize()+ commitGroup := make([]string, 0, groupSize)for s.Scan() && !common.IsDone(ctx) {commitGroup = append(commitGroup, s.Text())-- if len(commitGroup) == showGroupSize {- err = common.CancellableWrite(ctx, commitGroups, commitGroup)- if err != nil {- ctx.Logger().Error(err, "git log stopping early")- return- }- commitGroup = make([]string, 0, showGroupSize)+ if len(commitGroup) < groupSize {+ continue}+ if err := common.CancellableWrite(ctx, commitGroups, commitGroup); err != nil {+ ctx.Logger().Error(err, "git rev-list stopping early")+ return+ }+ commitGroup = make([]string, 0, groupSize)}++ // The last group is almost never exactly full.if len(commitGroup) != 0 {- err = common.CancellableWrite(ctx, commitGroups, commitGroup)- if err != nil {- ctx.Logger().Error(err, "failed to flush last git log group")+ if err := common.CancellableWrite(ctx, commitGroups, commitGroup); err != nil {+ ctx.Logger().Error(err, "failed to flush last commit group")}-}if err := s.Err(); err != nil {- ctx.Logger().Error(err, "error reading git log")+ ctx.Logger().Error(err, "error reading commit list")}}()@@ -440,8 +492,13 @@ func (c *Parser) prepGitArgs(source string, head string, abbreviatedLog bool, ex}if abbreviatedLog {+ // Only in show. args.log holds the options that choose commits, and it is+ // also what the lower-memory scan hands to `git rev-list`, which rejects diff+ // options outright. Leaving it out costs nothing: rev-list then lists a few+ // commits whose diffs are all filtered away, and the `git log` that generates+ // the patches drops those commits itself, exactly as the single-command form+ // does.// https://git-scm.com/docs/git-show#Documentation/git-show.txt---diff-filterACDMRTUXB- args.log = append(args.log, "--diff-filter=AM")args.show = append(args.show, "--diff-filter=AM")}@@ -1178,10 +1235,25 @@ func cleanupParse(ctx context.Context, currentCommit *Commit, currentDiff *Diff,if currentDiff != nil && (currentDiff.Len() > 0 || currentDiff.IsBinary) {currentDiff.Commit = currentCommitdiffChan <- currentDiff- }- if currentCommit != nil {- if totalLogSize != nil {- *totalLogSize += currentCommit.Size+ if currentCommit != nil {+ currentCommit.hasDiffs = true}}++ if currentCommit == nil {+ return+ }+ if totalLogSize != nil {+ *totalLogSize += currentCommit.Size+ }++ // A commit is normally finished off in the loop above, when the next commit line+ // shows up. The last commit in the stream never gets that, so it is finished here+ // instead, and it needs the same rule: a commit with no diffs of its own still has+ // a message, an author and notes worth scanning, so it goes out on its own.+ //+ // Staged diffs come through here too and carry no commit, hence the hash check.+ if !currentCommit.hasDiffs && currentCommit.Hash != "" {+ diffChan <- &Diff{Commit: currentCommit}+ }}diff --git a/pkg/gitparse/gitparse_test.go b/pkg/gitparse/gitparse_test.goindex 06aace3bf..52dd304f1 100644--- a/pkg/gitparse/gitparse_test.go+++ b/pkg/gitparse/gitparse_test.go@@ -36,8 +36,10 @@ func TestPrepGitArgs(t *testing.T) {// headassert.Contains(t, args.log, "branchname")assert.NotContains(t, args.log, "--all")- // abbreviatedLog- assert.Contains(t, args.log, "--diff-filter=AM")+ // abbreviatedLog. Only show carries the diff filter: args.log holds the options+ // that choose commits and is what the lower-memory scan gives to `git rev-list`,+ // which rejects diff options.+ assert.NotContains(t, args.log, "--diff-filter=AM")assert.Contains(t, args.show, "--diff-filter=AM")// excludedGlobsassert.Contains(t, args.paths, "--")diff --git a/pkg/gitparse/lowmemory_test.go b/pkg/gitparse/lowmemory_test.gonew file mode 100644index 000000000..a4ab2e44d--- /dev/null+++ b/pkg/gitparse/lowmemory_test.go@@ -0,0 +1,282 @@+package gitparse++import (+ "os"+ "os/exec"+ "path/filepath"+ "strings"+ "testing"++ "github.com/trufflesecurity/trufflehog/v3/pkg/context"+)++// The lower-memory scan splits one `git log` into a `git rev-list` that lists commits+// and a `git log` per group that generates their patches. These tests hold it to the+// only thing that really matters: it must find exactly what the single-command form+// finds, wherever the group boundaries happen to land.++// collectDiffs drains a diff channel into something comparable. Content is included,+// since a group boundary in the wrong place could keep the commit and lose its patch.+func collectDiffs(t *testing.T, diffChan chan *Diff) []string {+ t.Helper()++ var out []string+ for diff := range diffChan {+ content := ""+ // A commit with no diffs arrives with nothing written, and asking such a diff+ // for its content fails, so check before reading.+ if diff.contentWriter != nil && diff.Len() > 0 {+ got, err := diff.contentWriter.String()+ if err != nil {+ t.Fatalf("reading diff content: %v", err)+ }+ content = got+ }+ out = append(out, strings.Join([]string{diff.Commit.Hash, diff.PathB, content}, "\x00"))+ }+ return out+}++func TestLowMemoryScanMatchesSingleProcess(t *testing.T) {+ repo := testRepoRoot(t)++ // abbreviatedLog is the caller's BaseHash == "", so both values are real+ // configurations and they take different paths through git. With it on, git drops+ // commits whose diffs are all filtered away; with it off, those commits stay.+ //+ // The group sizes go down to 1 on purpose. Every group ends a stream, and a commit+ // with no diffs used to be dropped when it landed at the end of one, so a size of 1+ // puts every commit in that spot at once. Before cleanupParse learned to finish off+ // the last commit, this repository lost 84 diffs at size 1 and 1 at size 75.+ for _, tc := range []struct {+ name string+ abbreviated bool+ }{+ {"abbreviated", true},+ {"full", false},+ } {+ abbreviated, name := tc.abbreviated, tc.name++ t.Run(name, func(t *testing.T) {+ ctx := context.Background()++ single := NewParser()+ singleChan, err := single.RepoPath(ctx, repo, "", abbreviated, nil, false)+ if err != nil {+ t.Fatalf("single-process RepoPath: %v", err)+ }+ want := collectDiffs(t, singleChan)+ if len(want) == 0 {+ t.Fatal("single-process scan produced no diffs")+ }++ // Group sizes small enough that this repository crosses boundaries, which+ // is the only place the two forms can drift apart.+ for _, groupSize := range []int{1, 2, 7, 500} {+ low := NewParser(UseLowMemoryScan())+ low.groupSize = groupSize++ lowChan, err := low.RepoPath(ctx, repo, "", abbreviated, nil, false)+ if err != nil {+ t.Fatalf("group size %d: RepoPath: %v", groupSize, err)+ }+ got := collectDiffs(t, lowChan)++ if len(got) != len(want) {+ t.Fatalf("group size %d: got %d diffs, want %d", groupSize, len(got), len(want))+ }+ for i := range want {+ if got[i] != want[i] {+ t.Fatalf("group size %d: diff %d differs\n got: %q\nwant: %q",+ groupSize, i, got[i], want[i])+ }+ }+ }+ })+ }+}++// TestLowMemoryScanExcludedGlobs checks the path filters reach both commands. They+// decide which commits rev-list lists and which files the patches contain, so sending+// them to only one of the two would quietly change what gets scanned.+func TestLowMemoryScanExcludedGlobs(t *testing.T) {+ repo := testRepoRoot(t)+ ctx := context.Background()+ globs := []string{"*.go"}++ single := NewParser()+ singleChan, err := single.RepoPath(ctx, repo, "", true, globs, false)+ if err != nil {+ t.Fatalf("single-process RepoPath: %v", err)+ }+ want := collectDiffs(t, singleChan)++ low := NewParser(UseLowMemoryScan())+ low.groupSize = 13+ lowChan, err := low.RepoPath(ctx, repo, "", true, globs, false)+ if err != nil {+ t.Fatalf("low-memory RepoPath: %v", err)+ }+ got := collectDiffs(t, lowChan)++ if len(got) != len(want) {+ t.Fatalf("got %d diffs, want %d", len(got), len(want))+ }+ for i := range want {+ if got[i] != want[i] {+ t.Fatalf("diff %d differs\n got: %q\nwant: %q", i, got[i], want[i])+ }+ }+}++// TestLowMemoryScanUnknownHead covers the error the old form threw away. A bad revision+// used to be logged and the channel closed, so a typo in a branch name looked exactly+// like an empty repository.+func TestLowMemoryScanUnknownHead(t *testing.T) {+ repo := testRepoRoot(t)++ parser := NewParser(UseLowMemoryScan())+ if _, err := parser.RepoPath(context.Background(), repo, "no-such-ref-exists", true, nil, false); err == nil {+ t.Fatal("expected an error for an unknown head, got nil")+ }+}++// TestLowMemoryScanEmptyRepo checks a repository with no commits ends the scan cleanly+// rather than failing. rev-list prints nothing and exits zero, which has to be told+// apart from rev-list printing nothing because it failed.+func TestLowMemoryScanEmptyRepo(t *testing.T) {+ dir := t.TempDir()+ runTestGit(t, dir, "init", "-q")++ parser := NewParser(UseLowMemoryScan())+ diffChan, err := parser.RepoPath(context.Background(), dir, "", true, nil, false)+ if err != nil {+ t.Fatalf("RepoPath on an empty repo: %v", err)+ }+ if n := len(collectDiffs(t, diffChan)); n != 0 {+ t.Fatalf("got %d diffs from an empty repo, want 0", n)+ }+}++// TestLowMemoryScanAbandonedByConsumer covers a caller that stops reading early, which+// is what a depth-limited scan does. Reading one diff and walking away must not leave+// the scan wedged; the cancel in ScanCommits is what releases it in production, so the+// same cancel stands in for it here.+func TestLowMemoryScanAbandonedByConsumer(t *testing.T) {+ repo := testRepoRoot(t)+ ctx, cancel := context.WithCancel(context.Background())++ parser := NewParser(UseLowMemoryScan())+ parser.groupSize = 2++ diffChan, err := parser.RepoPath(ctx, repo, "", true, nil, false)+ if err != nil {+ t.Fatalf("RepoPath: %v", err)+ }++ // Take a single diff, then stop, the way the scan loop does at max depth.+ if _, ok := <-diffChan; !ok {+ t.Fatal("expected at least one diff")+ }+ cancel()++ // Draining to the end must finish. If it hangs the test times out, which is the+ // failure we are looking for.+ for range diffChan { //nolint:revive // draining+ }+}++func TestCommitGroupSize(t *testing.T) {+ // A size of zero or less would mean a group that never fills, so it falls back.+ for _, tc := range []struct{ set, want int }{+ {0, logGroupSize},+ {-1, logGroupSize},+ {1, 1},+ {99, 99},+ } {+ parser := NewParser()+ parser.groupSize = tc.set+ if got := parser.commitGroupSize(); got != tc.want {+ t.Errorf("groupSize %d: got %d, want %d", tc.set, got, tc.want)+ }+ }+}++// testRepoRoot returns the root of the git checkout the tests run from. The real+// history has the merges, binaries, renames and uneven commit sizes that a hand-built+// fixture does not, which is where a batching mistake actually shows up.+func testRepoRoot(tb testing.TB) string {+ tb.Helper()++ out, err := exec.Command("git", "rev-parse", "--show-toplevel").Output()+ if err != nil {+ tb.Skip("not running from a git checkout")+ }+ root := strings.TrimSpace(string(out))+ if _, err := os.Stat(filepath.Join(root, ".git")); err != nil {+ tb.Skip("no .git directory")+ }+ return root+}++func runTestGit(tb testing.TB, dir string, args ...string) {+ tb.Helper()++ cmd := exec.Command("git", args...)+ cmd.Dir = dir+ if out, err := cmd.CombinedOutput(); err != nil {+ tb.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)+ }+}++// TestTrailingCommitWithoutDiffsIsReported pins down the rule that a commit is finished+// off at the end of a stream, not only when the next commit line arrives.+//+// A commit with no diffs of its own still carries a message, an author and notes, and+// those are worth scanning. The main loop has always sent such a commit on when it saw+// the next one start, but the end of the stream had no equivalent, so whichever commit+// happened to be last was dropped. One `git log` for a whole repository made that nearly+// harmless, since only the oldest commit in the history was ever in that spot. Splitting+// the log into groups puts a different commit there for every group.+func TestTrailingCommitWithoutDiffsIsReported(t *testing.T) {+ dir := t.TempDir()+ runTestGit(t, dir, "init", "-q")+ runTestGit(t, dir, "config", "user.email", "[email protected]")+ runTestGit(t, dir, "config", "user.name", "Test")++ if err := os.WriteFile(filepath.Join(dir, "f.txt"), []byte("hello\n"), 0o600); err != nil {+ t.Fatal(err)+ }+ runTestGit(t, dir, "add", "-A")+ runTestGit(t, dir, "commit", "-qm", "adds a file")++ // An empty commit has a message but changes nothing, which is the shape that used+ // to disappear.+ runTestGit(t, dir, "commit", "-q", "--allow-empty", "-m", "empty commit worth scanning")++ // Group size 1 puts the empty commit at the end of its own stream, which is exactly+ // where it used to be lost.+ parser := NewParser(UseLowMemoryScan())+ parser.groupSize = 1++ diffChan, err := parser.RepoPath(context.Background(), dir, "", false, nil, false)+ if err != nil {+ t.Fatalf("RepoPath: %v", err)+ }++ var messages []string+ for diff := range diffChan {+ messages = append(messages, diff.Commit.Message.String())+ }++ var found bool+ for _, msg := range messages {+ if strings.Contains(msg, "empty commit worth scanning") {+ found = true+ break+ }+ }+ if !found {+ t.Fatalf("the empty commit was never reported; got %d diffs with messages %q", len(messages), messages)+ }+}diff --git a/pkg/sources/git/git.go b/pkg/sources/git/git.goindex 6e33eaedd..89094f263 100644--- a/pkg/sources/git/git.go+++ b/pkg/sources/git/git.go@@ -856,6 +856,13 @@ func (s *Git) ScanCommits(ctx context.Context, repo *git.Repository, path stringrepoCtx = ctx}+ // The scan can stop before the diff channel is drained, on max depth or on+ // reaching the base commit. Nothing else tells the parser that, so cancelling on+ // the way out is what shuts down the git processes still producing diffs. Without+ // it they sit blocked on a channel nobody is reading until the whole scan ends.+ repoCtx, cancel := context.WithCancel(repoCtx)+ defer cancel()+logger := repoCtx.Logger()var logValues []anyif scanOptions.BaseHash != "" {