Skip to content

Find, verify, and analyze leaked credentials

7.7 KB
Raw Blame

S3 source

Scans objects in S3 buckets, or in any S3-compatible service.

Scan paths

The source implements both the plain Source interface and SourceUnitEnumChunker, where one unit is one bucket (plus the role used to reach it).

  • Chunks walks every bucket in sequence.
  • Enumerate reports each bucket as a unit, and ChunkUnit scans one bucket.

Both end up in scanBucket -> pageChunker, so per-object behavior is identical either way. The CLI always takes the unit path, because main.go enables source units and the source supports them.

Credentials

Set by which flags are present:

Flags Credentials used
none anonymous, for public buckets only
--cloud-environment the AWS SDK's normal chain: env vars, ~/.aws/credentials, instance metadata
--key and --secret those static credentials
--key, --secret, --session-token those temporary credentials

The default is anonymous, not your local AWS profile. Scanning a private bucket without --cloud-environment or explicit keys gives 403 AccessDenied.

--role-arn assumes a role before scanning, and can be repeated. With roles and no --bucket, the source tries every bucket each role can list, so AccessDenied on buckets a role cannot reach is expected and does not fail the scan.

Choosing buckets

--bucket scans only the named buckets. --ignore-bucket scans everything else. They cannot be combined; doing so fails at startup. Bucket names are matched exactly, not as globs or prefixes.

Choosing objects within a bucket

Four flags, all repeatable, all matched against the object key before the object is downloaded. A skipped object costs no GetObject request.

  • --include-prefix / --exclude-prefix match a literal key prefix. They can be used together to scan a subtree while leaving part of it out, and an exclude match always wins.
  • --include-extension / --exclude-extension match the file extension, written without a leading dot. They cannot be used together, since listing what to scan already excludes everything else. Matching is case insensitive.

Prefixes are literal, so --include-prefix=log matches logs/app.txt and logs-archive/app.txt alike.

The two kinds are combined with AND: an object has to pass both to be scanned. Entries are trimmed, and blank ones are ignored.

Include prefixes are also sent to S3 as ListObjectsV2Input.Prefix. A bucket is listed once per include prefix instead of in full, so the listing covers only the keys they select, and each prefix costs at least one ListObjectsV2 request per bucket even when nothing is under it. A prefix under another, like src/vendor/ under src/, is dropped, since the shorter one already lists it. Directory buckets are listed in full, because they only accept a prefix that ends in /.

Exclude prefixes and extensions have no server-side form, so they are matched after listing. They save GetObject requests but not ListObjectsV2 requests.

Three edge cases worth knowing:

  • Only the last extension counts. path.Ext("backup.tar.gz") is .gz, so --exclude-extension=tar.gz never matches, while --exclude-extension=gz does.
  • A key with no extension, like Makefile, matches no extension entry, so --include-extension skips it and --exclude-extension keeps it.
  • A dotfile is treated as all extension. .env has extension env, so --exclude-extension=env skips it, and any --include-extension list that omits env skips it too. That is easy to do by accident, and .env files are a common source of secrets.

Filters apply to object keys only, not to files inside an archive. --exclude-extension=mp4 skips clip.mp4 in the bucket but not a clip.mp4 packed inside media.zip, since archive contents are filtered by the shared deny list instead. Excluding zip avoids expanding the archive at all.

Prefixes are matched with strings.HasPrefix, so an exclude prefix naming one key also excludes anything that key is a prefix of: --exclude-prefix=foo/bar.txt excludes foo/bar.txt.bak as well. There is no exact-key match.

The same prefixes and extensions apply to every bucket in a scan. There is no way to scope a prefix to one bucket.

If a filter excludes every object in a bucket, the scan logs Scanned no objects in bucket with the count. If nothing is listed under the include prefixes at all, it logs Found no objects under the include prefixes. Either way, a mistyped prefix does not look like a clean scan of an empty bucket.

Objects skipped regardless of configuration

  • Glacier and Glacier Instant Retrieval storage classes
  • Larger than --max-object-size, which defaults to 250 MiB and cannot be raised above it
  • Empty
  • Extensions in the shared deny list in pkg/common/vars.go, which already covers images, audio, and video
  • Keys ending in /, which are directory markers

Debug logs

With debug on for this source (debug: true in the config), the scan logs:

  • V(1): S3 source initialized with the config choices (the endpoint's host only, never credentials), and Found buckets to scan for each role.
  • V(2): Scanning bucket and Finished scanning bucket, with the objects scanned, pages and duration.
  • V(3): Scanning object for each object downloaded.
  • V(4): Skipping bucket and Skipping object with a reason, Listed page for each page, and AWS SDK lines for retries, including S3 throttling (SlowDown).
  • V(5): Finished scanning object with its duration, and the region used for a bucket outside the default region.

Lines carry role, bucket, page_number and key where they apply. An object skipped for one of the reasons above shows its reason here, so a missing finding can be traced to the object it would have come from.

Custom endpoints

--endpoint points the scan at an S3-compatible service instead of AWS S3. Setting it changes three things:

  • The client uses path-style addressing, since these services rarely publish the wildcard DNS that virtual-hosted addressing needs.
  • Per-bucket region discovery is skipped. GetBucketRegion is AWS-only, and a custom endpoint serves its own buckets, so requests sign with --region instead.
  • Object links in finding metadata point at the endpoint rather than amazonaws.com, so they resolve the way the scan read them.

The endpoint is parsed in Init, so a malformed one fails at startup rather than as an SDK error mid-scan. A missing scheme is assumed to be https.

--region sets the region used to sign requests and defaults to us-east-1. On AWS each bucket's real region is discovered automatically, so it mostly matters with a custom endpoint.

Resumption

Checkpointer tracks which objects in the current page of up to 1000 have finished, and records the highest consecutively completed key as StartAfter in Progress.EncodedResumeInfo, along with the bucket and role. An interrupted scan resumes from there.

Only consecutive completions are checkpointed. If objects 0 to 5 and 7 to 8 are done but 6 is not, the checkpoint stops at 5, so resuming may rescan a few objects but never misses one.

With include prefixes the scan lists each in turn. They are listed in key order and none is under another, so keys still arrive in ascending order and the checkpoint stays a single key. Resuming passes it as StartAfter to every prefix, and S3 returns nothing for the prefixes already finished. A checkpoint from a scan that listed the whole bucket resumes the same way.

This matters when adding a new skip to pageChunker: a skipped object still has to call UpdateObjectCompletion. Skipping without it stalls the checkpoint at the first skipped object, and a resumed scan then redoes everything after it.

The OSS CLI does not persist progress between runs, so resumption only takes effect where something stores and returns EncodedResumeInfo.