Find, verify, and analyze leaked credentials
Go credentialsdevsecopsdynamic-analysisprecommitscanningsecretsecret-managementsecretssecuritysecurity-toolstrufflehogverification
Add filesystem source documentation (#5285)
Kashif Khan8d77a9dfc077
diff --git a/pkg/sources/filesystem/README.md b/pkg/sources/filesystem/README.mdnew file mode 100644index 000000000..1dd67ef37--- /dev/null+++ b/pkg/sources/filesystem/README.md@@ -0,0 +1,210 @@+# Filesystem Source++## Overview++The Filesystem source lets TruffleHog scan files and directories on local disk for secrets, credentials, and sensitive data. It can scan a single file, walk a whole directory tree, or follow symlinks to files and directories outside the given path.++## Filesystem Fundamentals++### What does this source scan?++This source scans paths on the machine running TruffleHog. A path can point to a single file, a directory, or a symlink. When a directory is given, TruffleHog walks every file inside it, including files in subdirectories.++### Key Terminology++| Term | Description |+|------|-------------|+| **Path** | A file or directory location to scan. You can list more than one |+| **Symlink** | A file that points to another file or directory instead of holding data itself |+| **Symlink Depth** | How many symlinks in a row TruffleHog will follow before it stops, to avoid following a loop forever |+| **Include Paths File** | A file with a list of patterns. Only files matching one of these patterns are scanned |+| **Exclude Paths File** | A file with a list of patterns. Files matching one of these patterns are skipped |+| **Binary File** | A file that is not plain text, such as an image or a compiled program |+| **Resume Info** | Progress information saved during a scan so a long scan can continue where it left off after a restart |++## Features++- **File and Directory Scanning**: Scan a single file or an entire directory tree+- **Symlink Support**: Optionally follow symlinks to files and directories, up to a configured depth+- **Path Filtering**: Include or exclude files using pattern files+- **Binary File Skipping**: Optionally skip files that are not plain text+- **Concurrent Processing**: Files within a directory are scanned in parallel+- **Resumable Scans**: A scan that stops partway through a large directory can pick up from where it stopped++## Configuration++### Scanning Paths++**CLI Usage:**+```bash+trufflehog filesystem /var/log /home/user/project/config.yaml+```++**YAML Configuration:**+```yaml+sources:+- connection:+ '@type': type.googleapis.com/sources.Filesystem+ paths:+ - /var/log+ - /home/user/project/config.yaml+ name: local-scan+ type: SOURCE_TYPE_FILESYSTEM+ verify: true+```++You can also pass directories using the `--directory` flag, which can be repeated:++```bash+trufflehog filesystem --directory /var/log --directory /etc+```++### Following Symlinks++By default, symlinks are not followed. Set a maximum depth to allow TruffleHog to follow them.++**CLI Usage:**+```bash+trufflehog filesystem /home/user/project --max-symlink-depth 5+```++**YAML Configuration:**+```yaml+sources:+- connection:+ '@type': type.googleapis.com/sources.Filesystem+ paths:+ - /home/user/project+ max_symlink_depth: 5+ name: local-scan+ type: SOURCE_TYPE_FILESYSTEM+ verify: true+```++The highest depth allowed is 40. Setting a higher number will cause the scan to fail with an error at startup.++### Skipping Binary Files++**YAML Configuration:**+```yaml+sources:+- connection:+ '@type': type.googleapis.com/sources.Filesystem+ paths:+ - /home/user/project+ skip_binaries: true+ name: local-scan+ type: SOURCE_TYPE_FILESYSTEM+ verify: true+```++### Path Filtering++Use plain text files with one pattern per line to include or exclude files from a scan.++**CLI Usage:**+```bash+trufflehog filesystem /home/user/project --include-paths /home/user/include-patterns.txt --exclude-paths /home/user/exclude-patterns.txt+```++**YAML Configuration:**+```yaml+sources:+- connection:+ '@type': type.googleapis.com/sources.Filesystem+ paths:+ - /home/user/project+ include_paths_file: /home/user/include-patterns.txt+ exclude_paths_file: /home/user/exclude-patterns.txt+ name: local-scan+ type: SOURCE_TYPE_FILESYSTEM+ verify: true+```++## How Scanning Works++### Scanning Process++1. **Path Check**: For each configured path, TruffleHog checks whether it is a file, a directory, or a symlink+2. **Directory Walk**: If it is a directory, every entry inside is visited. Subdirectories are visited the same way, going deeper until every file is reached+3. **Symlink Handling**: If an entry is a symlink and following symlinks is turned on, TruffleHog follows it to the file or directory it points to, up to the configured depth+4. **File Filtering**: Each file is checked against the include and exclude patterns before it is scanned+5. **Binary Check**: If skipping binaries is turned on, files that look like binary files are skipped+6. **Content Scanning**: The file is opened and its content is sent to the detection engine as chunks+7. **Progress Tracking**: The path of the last file scanned in each directory is saved, so a scan that stops can resume from that point++### What Gets Scanned++- Regular files at the given paths+- Files inside directories at the given paths, including subdirectories+- Files reached by following a symlink, when symlink following is turned on++### What Doesn't Get Scanned++- Symlinks, when symlink following is turned off+- Files that go past the configured maximum symlink depth+- Files matching an exclude pattern, or not matching an include pattern, when pattern files are set+- Binary files, when skipping binaries is turned on+- Files that are not regular files, such as device files or sockets++## Usage Examples++### Scanning a Single File++```bash+trufflehog filesystem /home/user/project/config.yaml+```++### Scanning a Directory++```bash+trufflehog filesystem /home/user/project+```++### Scanning Multiple Paths++```bash+trufflehog filesystem /home/user/project /var/log/app.log+```++### Scanning a Directory and Following Symlinks++```bash+trufflehog filesystem /home/user/project --max-symlink-depth 10+```++### Scanning While Skipping Binary Files++```yaml+sources:+- connection:+ '@type': type.googleapis.com/sources.Filesystem+ paths:+ - /home/user/project+ skip_binaries: true+ name: local-scan+ type: SOURCE_TYPE_FILESYSTEM+ verify: true+```++## Troubleshooting++### Common Issues++**Issue**: Symlinks are not being scanned+**Solution**: Set `max_symlink_depth` (or `--max-symlink-depth`) to a value greater than zero. Symlinks are skipped by default.++---++**Issue**: `specified symlink depth exceeds the allowed max`+**Solution**: The maximum allowed symlink depth is 40. Use a smaller value.++---++**Issue**: Some files are missing from scan results+**Solution**: Check the include and exclude pattern files for a pattern that matches those files, and confirm `skip_binaries` is not turned on for text files that look like binary data.++---++**Issue**: A restarted scan seems to repeat or skip files+**Solution**: Resume tracking saves the last file scanned inside each directory. If the directory contents changed between runs, the resume point may no longer line up with the same files.diff --git a/pkg/sources/filesystem/filesystem_test.go b/pkg/sources/filesystem/filesystem_test.goindex c8a6317cb..67b5e0c4c 100644--- a/pkg/sources/filesystem/filesystem_test.go+++ b/pkg/sources/filesystem/filesystem_test.go@@ -761,6 +761,29 @@ func TestComparePathsForResume(t *testing.T) {}}+func TestInit_MaxSymlinkDepthWithinLimit(t *testing.T) {+ ctx := trContext.Background()++ conn, err := anypb.New(&sourcespb.Filesystem{MaxSymlinkDepth: defaultMaxSymlinkDepth})+ require.NoError(t, err)++ s := Source{}+ err = s.Init(ctx, "test max symlink depth within limit", 0, 0, true, conn, 1)+ require.NoError(t, err)+ assert.Equal(t, defaultMaxSymlinkDepth, s.maxSymlinkDepth)+}++func TestInit_MaxSymlinkDepthExceedsLimit(t *testing.T) {+ ctx := trContext.Background()++ conn, err := anypb.New(&sourcespb.Filesystem{MaxSymlinkDepth: defaultMaxSymlinkDepth + 1})+ require.NoError(t, err)++ s := Source{}+ err = s.Init(ctx, "test max symlink depth exceeds limit", 0, 0, true, conn, 1)+ assert.Error(t, err)+}+func TestResumptionWithOutOfSubtreeResumePoint(t *testing.T) {ctx := trContext.Background()