Skip to content

Find, verify, and analyze leaked credentials

Make a first pass at some structural introduction docs (#4076)

Martin Locklear35e23ee4e2f9
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 786b4a806..2badad299 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -8,6 +8,12 @@ Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/tru
# Resources
+## How things work
+
+It can be a bit daunting diving into the code and wrapping your head around the project from a high level. The following two docs help give that high level overview:
+* [Process Flow](docs/process_flow.md)
+* [Concurrency Overview](docs/concurrency.md)
+
## Adding new secret detectors
We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!
diff --git a/docs/concurrency.md b/docs/concurrency.md
new file mode 100644
index 000000000..15edf73fa
--- /dev/null
+++ b/docs/concurrency.md
@@ -0,0 +1,43 @@
+
+
+## Concurrency
+
+```mermaid
+sequenceDiagram
+ %% Setup the workers
+ participant Main
+ Note over Main: e.startWorkers()<br />kicks off some number<br />of threads per worker type
+ create participant ScannerWorkers
+ Main->>ScannerWorkers: e.startScannerWorkers()
+ Note over ScannerWorkers: ScannerWorkers are primarily<br />responsible for enumerating<br />and chunking a source
+ create participant VerificationOverlapWorkers
+ Main->>VerificationOverlapWorkers: e.startVerificationOverlapWorkers()
+ Note over VerificationOverlapWorkers: VerificationOverlapWorkers<br />handles chunks<br />matched to multiple<br />detectors
+ create participant DetectorWorkers
+ Main->>DetectorWorkers: e.startDetectorWorkers()
+ Note over DetectorWorkers: DetectorWorkers are primarily<br />responsible for running<br />detectors on chunks
+ create participant NotifierWorkers
+ Main->>NotifierWorkers: e.startNotifierWorkers()
+ Note over NotifierWorkers: Primarily responsible for reporting<br />results (typically to the cmd line)
+
+ %% Set up the parallelism
+ par
+ Note over Main,ScannerWorkers: Depending on the type of<br />scan requested, calls one of<br />engine.(ScanGit|ScanGitHub|ScanFileSystem|etc)
+ Main->>ScannerWorkers: e.ChunksChan()<br /><- chunk
+ and
+ Note over ScannerWorkers: Decode chunks and find matching detectors
+ ScannerWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk
+ Note over ScannerWorkers: When multiple detectors match on the<br />same chunk we have to decided _which_<br />detector will verify found secrets
+ ScannerWorkers->>VerificationOverlapWorkers: e.verificationOverlapChunksChan<br /><- verificationOverlapChunk
+ and
+ Note over VerificationOverlapWorkers: Decide which detectors to run on that chunk
+ VerificationOverlapWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk
+ and
+ Note over DetectorWorkers: Run detection (finding secrets),<br />optionally verify them<br />do filtering and enrichment
+ DetectorWorkers->>NotifierWorkers: e.ResultsChan()|e.results<br /><-detectors.ResultWithMetadata
+ and
+ Note over NotifierWorkers: Write results to output
+ end
+
+
+```
\ No newline at end of file
diff --git a/docs/process_flow.md b/docs/process_flow.md
new file mode 100644
index 000000000..1936df69c
--- /dev/null
+++ b/docs/process_flow.md
@@ -0,0 +1,138 @@
+# TruffleHog Process Flows
+
+## Scans
+
+## Data Flow
+
+```mermaid
+flowchart LR
+ SourceDecomposition["`**Source Decomposition**
+
+Breaking up the locations that we are looking _for_ secrets into small chunks`"]
+
+ DetectorMatching{Chunk<br/>to<br/>Detector<br/>Matching}
+
+ SecretDetection["`**Secret Detection**
+
+Finding secrets in these chunks and (optionally) verifying whether they are live`"]
+
+ ResultNotification["`**Result Notification**
+
+Enriching results with metadata and (usually) printing to console`"]
+
+ SourceDecomposition -- chunks --> DetectorMatching
+ DetectorMatching -- matched chunks --> SecretDetection
+ SecretDetection -- results --> ResultNotification
+```
+
+#### Source Decomposition
+
+```mermaid
+flowchart TD
+ subgraph Source
+ direction TB
+ SourceDescription("`**(1)** Sources are top level places we find data/files/text to _scan_`")
+ GitSource["git Source"]
+ GitHubSource["GitHub Source"]
+ FilesystemSource["File System Source"]
+ PostmanSource["Postman Source"]
+ end
+
+ subgraph Unit
+ direction TB
+ UnitDescription("`**(2)** Units are natural subdivisions of Sources, but still quite large`")
+ FilesystemUnit[Directory]
+ GitUnit[Git Repository]
+ end
+
+ subgraph Chunk
+ direction TB
+ ChunkDescription("`**(3)** Chunks are the smallest units that we decompose our chunks into, and are subsequent passed on to detection`")
+ FilesystemChunk[file contents]
+ GitRepositoryChunk["`git log diff hunks`"]
+ PostmanChunk[data chunk]
+ end
+
+
+ SourceDescription -- decomposed into --> UnitDescription
+ UnitDescription -- further decomposed into --> ChunkDescription
+
+
+ GitSource -- cloned locally<br />if not already local --> GitUnit
+ GitHubSource -- cloned locally --> GitUnit
+ PostmanSource -- Most sources\ndon't use units --> PostmanChunk
+ FilesystemSource --> FilesystemUnit
+
+ GitUnit -- git log -p --> GitRepositoryChunk
+ FilesystemUnit --> FilesystemChunk
+
+ style SourceDescription fill:#89553e
+ style UnitDescription fill:#89553e
+ style ChunkDescription fill:#89553e
+```
+
+#### Chunk to Detector Matching
+
+```mermaid
+flowchart LR
+
+
+ KeywordMatching["`**Keyword Matching**
+_(Aho-Corsick)_
+
+Match chunks to detectors based on the presence of specific keywords in the chunk`"]
+
+ chunks --> KeywordMatching --> detectors
+```
+
+#### Secret Detection
+
+```mermaid
+flowchart LR
+
+subgraph Detector
+ direction RL
+ subgraph DetectorDescription[" "]
+ DetectorDescriptionText["`Detectors are the bits that actually check for the existence of a secret in a chunk, and (optionally) verify it`"]
+ ExampleDetectors["`Example Detectors:
+ * AWS
+ * Azure
+ * Twilio`"]
+ end
+
+ subgraph DetectorResponsibility[" "]
+ direction LR
+
+ De-Dupe-Detectors["`**De-Dupe-Detectors**
+
+If multiple detectors keyword-match on the same chunk, we have some logic that chooses which detector will verify found secret (so we don't duplicate verification requests to externa APIs)`"]
+
+ CollectMatches["`**Collect Matches**
+
+Detector specific regexes are run against the matched chunks, resulting in unverified secrets`"]
+ VerifyMatches["`**Verify Matches**
+
+Optionally, observed unverified secrets are verified by attempting to use them against live services`"]
+
+ De-Dupe-Detectors -- deduped detectors --> CollectMatches
+ CollectMatches -- regex matched chunks --> VerifyMatches
+ end
+
+ style DetectorDescription fill:#89553e
+ style DetectorDescriptionText fill:#89553e
+end
+```
+
+#### Result Notification
+
+```mermaid
+flowchart LR
+
+ Dispatcher["`**Dispatcher**
+
+Results, verified or otherwise, are sent to a dispatcher to be sent to whichever place we're updating about the
+results -- usually the command line.`"]
+
+ results --> Dispatcher --> output
+```
+
diff --git a/pkg/engine/engine.go b/pkg/engine/engine.go
index eb29b4ab7..e33018b67 100644
--- a/pkg/engine/engine.go
+++ b/pkg/engine/engine.go
@@ -1,3 +1,6 @@
+// Check the [process flow](docs/process_flow.md) and [concurrency](docs/concurrency.md) docs for
+// something of a structural overview
+
package engine
import (