Find, verify, and analyze leaked credentials
Go credentialsdevsecopsdynamic-analysisprecommitscanningsecretsecret-managementsecretssecuritysecurity-toolstrufflehogverification
Updated Docker source with new test cases and README (#4481)
Kashif Khanec61ad98fd3c
diff --git a/pkg/sources/docker/README.md b/pkg/sources/docker/README.mdnew file mode 100644index 000000000..c67a44037--- /dev/null+++ b/pkg/sources/docker/README.md@@ -0,0 +1,270 @@+# Docker Source++## Overview++The Docker source enables TruffleHog to scan Docker images for secrets, credentials, and sensitive data. It supports scanning images from multiple sources including Docker registries, local Docker daemon, and tarball files.++## Docker Fundamentals++### What is Docker?++Docker is a containerization platform that packages applications and their dependencies into isolated containers. A Docker image is a read-only template used to create containers, consisting of multiple layers stacked on top of each other.++### Key Docker Terminology++| Term | Description |+|------|-------------|+| **Image** | A read-only template containing application code, runtime, libraries, and dependencies |+| **Layer** | Each modification to an image creates a new layer. Layers are stacked and merged to form the final image |+| **Tag** | A label applied to an image (e.g., `latest`, `v1.0.0`) for version identification |+| **Digest** | A SHA256 hash that uniquely identifies an image or layer |+| **Registry** | A repository for storing and distributing Docker images (e.g., Docker Hub, Quay, GHCR) |+| **Daemon** | The Docker service running on the host that manages containers and images |+| **Tarball** | A compressed archive file containing an exported Docker image |+| **History** | Metadata about how an image was built, including commands executed |++## Features++- **Multiple Image Sources**: Scan images from remote registries, local Docker daemon, or tarball files+- **Layer-by-Layer Scanning**: Examines each layer independently for comprehensive coverage+- **History Metadata Scanning**: Analyzes image build history for exposed secrets in commands+- **Concurrent Processing**: Parallel layer scanning for improved performance+- **Authentication Support**: Multiple authentication methods for private registries+- **File Exclusion**: Configure patterns to skip specific files or directories+- **Size Limits**: Automatically skips files exceeding 50MB to optimize performance++## Configuration++### Connection Types++The Docker source supports several image reference formats:++```go+// Remote registry (default)+"nginx:latest"+"myregistry.com/myapp:v1.0.0"+"gcr.io/project/image@sha256:abcd1234..."++// Local Docker daemon+"docker://nginx:latest"++// Tarball file+"file:///path/to/image.tar"+```+### Authentication Methods++#### 1. Unauthenticated (Public Images)++For public images that don't require authentication:++**YAML Configuration:**+```yaml+sources:+ - type: docker+ name: public-images+ docker:+ unauthenticated: {}+ images:+ - nginx:latest+ - alpine:3.18+```++**CLI Usage:**+```bash+trufflehog docker --image nginx:latest+```++---++#### 2. Basic Authentication++For private registries requiring username and password:++**YAML Configuration:**+```yaml+sources:+ - type: docker+ name: private-registry+ docker:+ basic_auth:+ username: myuser+ password: mypassword+ images:+ - myregistry.com/private-image:latest+ - myregistry.com/another-image:v1.0.0+```++**CLI Usage:**++Trufflehog does not provide basic authentication using username and password through CLI at the moment.++---++#### 3. Bearer Token++For registries using token-based authentication (e.g., Dockerhub registry):++**YAML Configuration:**+```yaml+sources:+ - type: docker+ name: truffle-packages+ docker:+ bearer_token: "ghp_xxxxxxxxxxxxxxxxxxxx"+ images:+ - myorg/myapp:latest+ - myorg/frontend:v2.1.0+```++**CLI Usage:**+```bash+trufflehog docker --image myorg/myapp:latest --bearer-token eyJ_xxxxxxxxxxxxxxxxxxxx+```++---++#### 4. Docker Keychain++Uses credentials from your local Docker configuration (`~/.docker/config.json`):++**YAML Configuration:**+```yaml+sources:+ - type: docker+ name: local-docker-creds+ docker:+ docker_keychain: true+ images:+ - myregistry.com/private-image:latest+ - docker.io/myorg/app:latest+```++**CLI Usage:**+```bash+# First, authenticate with Docker+docker login myregistry.com++# Then scan using stored credentials+trufflehog docker --image myregistry.com/private-image:latest+```++**Prerequisites:**+```bash+# Authenticate with your registry first+docker login+docker login ghcr.io+docker login quay.io++# Verify credentials are stored+cat ~/.docker/config.json+```+++### File Exclusion++Exclude specific files or directories from scanning using glob patterns:++```bash+trufflehog docker --image myregistry.com/private-image:latest --exclude-paths **/*.log+```++## How Image Scanning Works++### Scanning Process++1. **Image Retrieval**: Fetches the image from the specified source (registry, daemon, or file)+2. **History Scanning**: Extracts and scans image configuration history for secrets in build commands+3. **Layer Processing**: Iterates through each layer in parallel+4. **File Extraction**: Decompresses and extracts files from each layer+5. **Content Scanning**: Analyzes file contents for secrets and credentials+6. **Chunk Generation**: Emits chunks of data to the detection engine++### What Gets Scanned++- **Layer Contents**: All files within each image layer+- **Build History**: Commands used to build the image (FROM, RUN, ENV, etc.)+- **Configuration**: Environment variables and labels+- **Metadata**: Image annotations and custom metadata++### What Doesn't Get Scanned++- Files larger than 50MB (configurable limit)+- Files matching exclude patterns+- Empty layers (no content changes)++## Usage Examples++### Scanning a Public Image++```bash+trufflehog docker --image nginx:latest+```++### Scanning Multiple Images++```bash+trufflehog docker --image nginx:latest --image postgres:13 --image redis:alpine+```++### Scanning from Local Docker Daemon++```bash+trufflehog docker --image docker://myapp:local+```++### Scanning a Tarball++```bash+# First, save an image to a tarball+docker save myapp:latest -o myapp.tar++# Then scan it+trufflehog docker --image file:///path/to/myapp.tar+```++### Scanning Private Registry with Authentication++```bash+docker login my-registry.io+trufflehog docker --image my-registry.io/private-app:v1.0.0+```++## Testing Results++### Integration Test Results++| Test Case | Status | Command/Configuration | Registry URL | Notes |+|-----------|--------|----------------------|--------------|-------|+| Scan remote image on DockerHub | ✅ Success | `--image <image_name>` | https://hub.docker.com/ | Public images work without authentication |+| Scan specific tag of image on DockerHub | ✅ Success | `--image <image_name>:<tag_name>` | https://hub.docker.com/ | Tag specification working correctly |+| Scan remote image on Quay.io | ✅ Success | `--image quay.io/prometheus/prometheus` | https://quay.io/search | Public Quay.io registry supported |+| Scan multiple images | ✅ Success | `--image <image_name> --image <image_name>` | Multiple registries | Sequential scanning of multiple images |+| Scan remote image on DockerHub with token | ✅ Success | Generate token using username and password | https://hub.docker.com/ | Basic auth with PAT working |+| Scan private image on Quay | ⏸️ Halted | N/A | https://quay.io/ | RedHat requires paid account for private repos |+| Scan private image on GHCR | ✅ Success | `--image ghcr.io/<image_name>` | https://github.com/packages | GitHub Container Registry |++## Troubleshooting++### Common Issues++**Issue**: Authentication failures with private registries++**Solution**: Ensure credentials are correct and have pull permissions. Use `docker login` first when using Docker Keychain method.++---++**Issue**: Out of memory errors with large images++**Solution**: Reduce concurrency or scan smaller images. Consider increasing available memory.++---++**Issue**: Slow scanning performance++**Solution**: Enable concurrent processing, use local daemon instead of remote registry, or exclude unnecessary directories.++---++**Issue**: Files not being scanned++**Solution**: Check exclude patterns and file size limits. Verify files are under 50MB.diff --git a/pkg/sources/docker/docker.go b/pkg/sources/docker/docker.goindex b3c50e2d9..d47558fb9 100644--- a/pkg/sources/docker/docker.go+++ b/pkg/sources/docker/docker.go@@ -38,7 +38,7 @@ type Source struct {verify boolconcurrency intconn sourcespb.Docker- globFilter *glob.Filter+ globFilter *glob.Filter // Filter for excluding files based on glob patternssources.Progresssources.CommonSourceUnitUnmarshaller}@@ -77,7 +77,7 @@ func (s *Source) Init(_ context.Context, name string, jobId sources.JobID, sourcreturn fmt.Errorf("error unmarshalling connection: %w", err)}- // Extract exclude paths from connection and compile regexes+ // Extract exclude paths from connection and compile glob patternsif paths := s.conn.GetExcludePaths(); len(paths) > 0 {var err errors.globFilter, err = glob.NewGlobFilter(glob.WithExcludeGlobs(paths...))@@ -89,24 +89,27 @@ func (s *Source) Init(_ context.Context, name string, jobId sources.JobID, sourcreturn nil}+// imageInfo holds information about a Docker image being processedtype imageInfo struct {- image v1.Image- base string- tag string+ image v1.Image // The image object from go-containerregistry+ base string // Base name of the image (without tag/digest)+ tag string // Tag or digest of the image}+// historyEntryInfo represents a single entry from the image's build historytype historyEntryInfo struct {- index int- entry v1.History- layerDigest string- base string- tag string+ index int // Position in the history array+ entry v1.History // The history entry containing build commands+ layerDigest string // SHA256 digest of the associated layer (empty for empty layers)+ base string // Base name of the image+ tag string // Tag or digest of the image}+// layerInfo contains identifying information for a Docker image layertype layerInfo struct {- digest v1.Hash- base string- tag string+ digest v1.Hash // SHA256 digest uniquely identifying the layer+ base string // Base name of the image+ tag string // Tag or digest of the image}// Chunks emits data over a channel that is decoded and scanned for secrets.@@ -137,12 +140,14 @@ func (s *Source) Chunks(ctx context.Context, chunksChan chan *sources.Chunk, _ .return nil}+ // Get history entries and associate them with layershistoryEntries, err := getHistoryEntries(ctx, imgInfo, layers)if err != nil {ctx.Logger().Error(err, "error getting image history entries")return nil}+ // Scan each history entry for secrets in build commandsfor _, historyEntry := range historyEntries {if err := s.processHistoryEntry(ctx, historyEntry, chunksChan); err != nil {ctx.Logger().Error(err, "error processing history entry")@@ -153,6 +158,7 @@ func (s *Source) Chunks(ctx context.Context, chunksChan chan *sources.Chunk, _ .ctx.Logger().V(2).Info("scanning image layers")+ // Process each layer concurrentlyfor _, layer := range layers {workers.Go(func() error {if err := s.processLayer(ctx, layer, imgInfo, chunksChan); err != nil {@@ -177,6 +183,7 @@ func (s *Source) Chunks(ctx context.Context, chunksChan chan *sources.Chunk, _ .}// processImage processes an individual image and prepares it for further processing.+// It handles three image source types: remote registry, local daemon, and tarball file.func (s *Source) processImage(ctx context.Context, image string) (imageInfo, error) {var (imgInfo imageInfo@@ -191,15 +198,16 @@ func (s *Source) processImage(ctx context.Context, image string) (imageInfo, errconst filePrefix = "file://"const dockerPrefix = "docker://"- if strings.HasPrefix(image, filePrefix) {- image = strings.TrimPrefix(image, filePrefix)++ // Handle tarball file images+ if image, ok := strings.CutPrefix(image, filePrefix); ok {imgInfo.base = imageimgInfo.image, err = tarball.ImageFromPath(image, nil)if err != nil {return imgInfo, err}- } else if strings.HasPrefix(image, dockerPrefix) {- image = strings.TrimPrefix(image, dockerPrefix)+ // Handle local Docker daemon images+ } else if image, ok := strings.CutPrefix(image, dockerPrefix); ok {imgInfo, imageName, err = s.extractImageNameTagDigest(image)if err != nil {return imgInfo, err@@ -208,6 +216,7 @@ func (s *Source) processImage(ctx context.Context, image string) (imageInfo, errif err != nil {return imgInfo, err}+ // Handle remote registry images (default)} else {imgInfo, imageName, err = s.extractImageNameTagDigest(image)if err != nil {@@ -235,8 +244,10 @@ func (*Source) extractImageNameTagDigest(image string) (imageInfo, name.Referenc)imgInfo.base, imgInfo.tag, hasDigest = baseAndTagFromImage(image)+ // Parse as digest reference (e.g., nginx@sha256:abc123...)if hasDigest {imgName, err = name.NewDigest(image)+ // Parse as tag reference (e.g., nginx:latest)} else {imgName, err = name.NewTag(image)}@@ -267,8 +278,8 @@ func getHistoryEntries(ctx context.Context, imgInfo imageInfo, layers []v1.Layerindex: historyIndex,}- // Associate with a layer if possible -- failing to do this will not affect- // the scan, just remove some traceability.+ // Associate with a layer if possible. Some history entries don't create layers (e.g., ENV, LABEL)+ // Failing to associate won't affect scanning, just reduces traceabilityif !entry.EmptyLayer {if layerIndex < len(layers) {digest, err := layers[layerIndex].Digest()@@ -294,9 +305,10 @@ func getHistoryEntries(ctx context.Context, imgInfo imageInfo, layers []v1.Layer}// processHistoryEntry processes a history entry from the image configuration metadata.+// It scans the CreatedBy field which contains the command used to create that layer.func (s *Source) processHistoryEntry(ctx context.Context, historyInfo historyEntryInfo, chunksChan chan *sources.Chunk) error {- // Make up an identifier for this entry that is moderately sensible. There is- // no file name to use here, so the path tries to be a little descriptive.+ // Create a descriptive identifier for this history entry+ // There's no file name here, so we use a synthetic pathentryPath := fmt.Sprintf("image-metadata:history:%d:created-by", historyInfo.index)chunk := &sources.Chunk{@@ -323,6 +335,7 @@ func (s *Source) processHistoryEntry(ctx context.Context, historyInfo historyEnt}// processLayer processes an individual layer of an image.+// It decompresses the layer and extracts all files for scanning.func (s *Source) processLayer(ctx context.Context, layer v1.Layer, imgInfo imageInfo, chunksChan chan *sources.Chunk) error {layerInfo := layerInfo{base: imgInfo.base,@@ -343,9 +356,10 @@ func (s *Source) processLayer(ctx context.Context, layer v1.Layer, imgInfo image}defer rc.Close()+ // Configure parallel gzip decompression for better performanceconst (- defaultBlockSize = 1 << 24 // 16MB- defaultBlocks = 8+ defaultBlockSize = 1 << 24 // 16MB blocks+ defaultBlocks = 8 // Process 8 blocks in parallel)gzipReader, err := gzip.NewReaderN(rc, defaultBlockSize, defaultBlocks)@@ -354,6 +368,7 @@ func (s *Source) processLayer(ctx context.Context, layer v1.Layer, imgInfo image}defer gzipReader.Close()+ // Layers are tar archives, so we read them as tar filestarReader := tar.NewReader(gzipReader)for {header, err := tarReader.Next()@@ -373,27 +388,31 @@ func (s *Source) processLayer(ctx context.Context, layer v1.Layer, imgInfo imagereturn nil}+// chunkProcessingInfo holds information needed to process a single file from a layertype chunkProcessingInfo struct {- size int64- name string- reader io.Reader- layer layerInfo+ size int64 // Size of the file in bytes+ name string // Full path of the file within the layer+ reader io.Reader // Reader for the file contents+ layer layerInfo // Information about the layer this file belongs to}// processChunk processes an individual chunk of a layer.+// It applies file size limits and exclusion patterns before scanning.func (s *Source) processChunk(ctx context.Context, info chunkProcessingInfo, chunksChan chan *sources.Chunk) error {+ // Skip files that are too large to avoid memory issuesconst filesizeLimitBytes int64 = 50 * 1024 * 1024 // 50MBif info.size > filesizeLimitBytes {ctx.Logger().V(2).Info("skipping file: size exceeds max allowed", "file", info.name, "size", info.size, "limit", filesizeLimitBytes)return nil}- // Check if the file should be excluded+ // Check if the file matches any exclude patternsfilePath := "/" + info.nameif s.isExcluded(ctx, filePath) {return nil}+ // Read the file in chunks to handle large files efficientlychunkReader := sources.NewChunkReader(sources.WithFileSize(int(info.size)))chunkResChan := chunkReader(ctx, info.reader)@@ -429,14 +448,14 @@ func (s *Source) processChunk(ctx context.Context, info chunkProcessingInfo, chureturn nil}-// isExcluded checks if a given filePath should be excluded based on the configured excludePaths and excludeRegexes.+// isExcluded checks if a given filePath should be excluded based on the configured glob patterns.func (s *Source) isExcluded(ctx context.Context, filePath string) bool {if s.globFilter == nil {- return false // No filter configured, so nothing is excluded.+ return false // No filter configured, so nothing is excluded}- // globFilter.ShouldInclude returns true if it's NOT excluded by an exclude glob or if it IS included by an include glob.- // If ShouldInclude is true (passes the filter), it means it was NOT matched by an exclude glob, so it's NOT excluded.- // If ShouldInclude is false (fails the filter), it means it WAS matched by an exclude glob, so it IS excluded.++ // ShouldInclude returns true if the file should be scanned (not excluded)+ // If it returns false, the file matches an exclude patternisIncluded := s.globFilter.ShouldInclude(filePath)if !isIncluded {@@ -445,7 +464,10 @@ func (s *Source) isExcluded(ctx context.Context, filePath string) bool {return !isIncluded}+// remoteOpts configures the options for fetching images from remote registries.+// It sets up HTTP transport and authentication based on the connection configuration.func (s *Source) remoteOpts() ([]remote.Option, error) {+ // Configure HTTP transport with reasonable timeouts and connection poolingdefaultTransport := &http.Transport{Proxy: http.ProxyFromEnvironment,DialContext: (&net.Dialer{@@ -463,6 +485,7 @@ func (s *Source) remoteOpts() ([]remote.Option, error) {var opts []remote.Optionopts = append(opts, remote.WithTransport(defaultTransport))+ // Configure authentication based on credential typeswitch s.conn.GetCredential().(type) {case *sourcespb.Docker_Unauthenticated:return nil, nil@@ -476,6 +499,7 @@ func (s *Source) remoteOpts() ([]remote.Option, error) {Token: s.conn.GetBearerToken(),}))case *sourcespb.Docker_DockerKeychain:+ // Use credentials from ~/.docker/config.jsonopts = append(opts, remote.WithAuthFromKeychain(authn.DefaultKeychain))default:return nil, fmt.Errorf("unknown credential type: %T", s.conn.Credential)@@ -484,6 +508,8 @@ func (s *Source) remoteOpts() ([]remote.Option, error) {return opts, nil}+// baseAndTagFromImage extracts the base name and tag/digest from an image reference string.+// It handles both digest-based references (image@sha256:...) and tag-based references (image:tag).func baseAndTagFromImage(image string) (base, tag string, hasDigest bool) {if base, tag, hasDigest = extractDigest(image); hasDigest {return base, tag, true@@ -493,8 +519,8 @@ func baseAndTagFromImage(image string) (base, tag string, hasDigest bool) {return base, tag, false}-// extractDigest tries to split the image string on the digest delimiter.-// If successful, it means the image has a digest.+// extractDigest tries to split the image string on the digest delimiter (@).+// If successful, it means the image has a digest reference.func extractDigest(image string) (base, tag string, hasDigest bool) {const digestDelim = "@"@@ -514,7 +540,8 @@ func extractTagOrUseDefault(image string) (base, tag string) {parts := strings.Split(image, tagDelim)- // Check if the last part is not a hostname with a port (for weak validation)+ // Check if the last part is a tag (not a hostname with port)+ // We use a weak validation: if it contains "/" it's likely part of a registry hostnameif len(parts) > 1 && !strings.Contains(parts[len(parts)-1], regRepoDelim) {return strings.Join(parts[:len(parts)-1], tagDelim), parts[len(parts)-1]}diff --git a/pkg/sources/docker/docker_test.go b/pkg/sources/docker/docker_test.goindex 0e25cd663..bd66c6baa 100644--- a/pkg/sources/docker/docker_test.go+++ b/pkg/sources/docker/docker_test.go@@ -66,6 +66,102 @@ func TestDockerImageScan(t *testing.T) {assert.Equal(t, 1, historyCounter)}+func TestQuayRegistry(t *testing.T) {+ dockerConn := &sourcespb.Docker{+ Credential: &sourcespb.Docker_Unauthenticated{+ Unauthenticated: &credentialspb.Unauthenticated{},+ },+ Images: []string{"quay.io/prometheus/busybox"}, // https://quay.io/repository/prometheus/busybox+ }++ conn := &anypb.Any{}+ err := conn.MarshalFrom(dockerConn)+ assert.NoError(t, err)++ s := &Source{}+ err = s.Init(context.TODO(), "test source", 0, 0, false, conn, 1)+ assert.NoError(t, err)++ var wg sync.WaitGroup+ chunksChan := make(chan *sources.Chunk, 1)+ chunkCounter := 0+ layerCounter := 0+ historyCounter := 0++ wg.Add(1)+ go func() {+ defer wg.Done()+ for chunk := range chunksChan {+ assert.NotEmpty(t, chunk)+ chunkCounter++++ if isHistoryChunk(t, chunk) {+ historyCounter+++ } else {+ layerCounter+++ }+ }+ }()++ err = s.Chunks(context.TODO(), chunksChan)+ assert.NoError(t, err)++ close(chunksChan)+ wg.Wait()++ assert.Equal(t, 945, chunkCounter)+ assert.Equal(t, 941, layerCounter)+ assert.Equal(t, 4, historyCounter)+}++func TestGHCRRegistry(t *testing.T) {+ dockerConn := &sourcespb.Docker{+ Credential: &sourcespb.Docker_Unauthenticated{+ Unauthenticated: &credentialspb.Unauthenticated{},+ },+ Images: []string{"ghcr.io/trufflesecurity/trufflehog:3.0.0-rc0-amd64"}, // https://github.com/trufflesecurity/trufflehog/pkgs/container/trufflehog+ }++ conn := &anypb.Any{}+ err := conn.MarshalFrom(dockerConn)+ assert.NoError(t, err)++ s := &Source{}+ err = s.Init(context.TODO(), "test source", 0, 0, false, conn, 1)+ assert.NoError(t, err)++ var wg sync.WaitGroup+ chunksChan := make(chan *sources.Chunk, 1)+ chunkCounter := 0+ layerCounter := 0+ historyCounter := 0++ wg.Add(1)+ go func() {+ defer wg.Done()+ for chunk := range chunksChan {+ assert.NotEmpty(t, chunk)+ chunkCounter++++ if isHistoryChunk(t, chunk) {+ historyCounter+++ } else {+ layerCounter+++ }+ }+ }()++ err = s.Chunks(context.TODO(), chunksChan)+ assert.NoError(t, err)++ close(chunksChan)+ wg.Wait()++ assert.Equal(t, 714, chunkCounter)+ assert.Equal(t, 709, layerCounter)+ assert.Equal(t, 5, historyCounter)+}+func TestDockerImageScanWithDigest(t *testing.T) {dockerConn := &sourcespb.Docker{Credential: &sourcespb.Docker_Unauthenticated{