Find, verify, and analyze leaked credentials
Go credentialsdevsecopsdynamic-analysisprecommitscanningsecretsecret-managementsecretssecuritysecurity-toolstrufflehogverification
Make a first pass at some structural introduction docs (#4076)
Martin Locklear35e23ee4e2f9
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 786b4a806..2badad299 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -8,6 +8,12 @@ Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/tru# Resources+## How things work++It can be a bit daunting diving into the code and wrapping your head around the project from a high level. The following two docs help give that high level overview:+* [Process Flow](docs/process_flow.md)+* [Concurrency Overview](docs/concurrency.md)+## Adding new secret detectorsWe have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!diff --git a/docs/concurrency.md b/docs/concurrency.mdnew file mode 100644index 000000000..15edf73fa--- /dev/null+++ b/docs/concurrency.md@@ -0,0 +1,43 @@+++## Concurrency++```mermaid+sequenceDiagram+ %% Setup the workers+ participant Main+ Note over Main: e.startWorkers()<br />kicks off some number<br />of threads per worker type+ create participant ScannerWorkers+ Main->>ScannerWorkers: e.startScannerWorkers()+ Note over ScannerWorkers: ScannerWorkers are primarily<br />responsible for enumerating<br />and chunking a source+ create participant VerificationOverlapWorkers+ Main->>VerificationOverlapWorkers: e.startVerificationOverlapWorkers()+ Note over VerificationOverlapWorkers: VerificationOverlapWorkers<br />handles chunks<br />matched to multiple<br />detectors+ create participant DetectorWorkers+ Main->>DetectorWorkers: e.startDetectorWorkers()+ Note over DetectorWorkers: DetectorWorkers are primarily<br />responsible for running<br />detectors on chunks+ create participant NotifierWorkers+ Main->>NotifierWorkers: e.startNotifierWorkers()+ Note over NotifierWorkers: Primarily responsible for reporting<br />results (typically to the cmd line)++ %% Set up the parallelism+ par+ Note over Main,ScannerWorkers: Depending on the type of<br />scan requested, calls one of<br />engine.(ScanGit|ScanGitHub|ScanFileSystem|etc)+ Main->>ScannerWorkers: e.ChunksChan()<br /><- chunk+ and+ Note over ScannerWorkers: Decode chunks and find matching detectors+ ScannerWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk+ Note over ScannerWorkers: When multiple detectors match on the<br />same chunk we have to decided _which_<br />detector will verify found secrets+ ScannerWorkers->>VerificationOverlapWorkers: e.verificationOverlapChunksChan<br /><- verificationOverlapChunk+ and+ Note over VerificationOverlapWorkers: Decide which detectors to run on that chunk+ VerificationOverlapWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk+ and+ Note over DetectorWorkers: Run detection (finding secrets),<br />optionally verify them<br />do filtering and enrichment+ DetectorWorkers->>NotifierWorkers: e.ResultsChan()|e.results<br /><-detectors.ResultWithMetadata+ and+ Note over NotifierWorkers: Write results to output+ end+++```\ No newline at end of filediff --git a/docs/process_flow.md b/docs/process_flow.mdnew file mode 100644index 000000000..1936df69c--- /dev/null+++ b/docs/process_flow.md@@ -0,0 +1,138 @@+# TruffleHog Process Flows++## Scans++## Data Flow++```mermaid+flowchart LR+ SourceDecomposition["`**Source Decomposition**++Breaking up the locations that we are looking _for_ secrets into small chunks`"]++ DetectorMatching{Chunk<br/>to<br/>Detector<br/>Matching}++ SecretDetection["`**Secret Detection**++Finding secrets in these chunks and (optionally) verifying whether they are live`"]++ ResultNotification["`**Result Notification**++Enriching results with metadata and (usually) printing to console`"]++ SourceDecomposition -- chunks --> DetectorMatching+ DetectorMatching -- matched chunks --> SecretDetection+ SecretDetection -- results --> ResultNotification+```++#### Source Decomposition++```mermaid+flowchart TD+ subgraph Source+ direction TB+ SourceDescription("`**(1)** Sources are top level places we find data/files/text to _scan_`")+ GitSource["git Source"]+ GitHubSource["GitHub Source"]+ FilesystemSource["File System Source"]+ PostmanSource["Postman Source"]+ end++ subgraph Unit+ direction TB+ UnitDescription("`**(2)** Units are natural subdivisions of Sources, but still quite large`")+ FilesystemUnit[Directory]+ GitUnit[Git Repository]+ end++ subgraph Chunk+ direction TB+ ChunkDescription("`**(3)** Chunks are the smallest units that we decompose our chunks into, and are subsequent passed on to detection`")+ FilesystemChunk[file contents]+ GitRepositoryChunk["`git log diff hunks`"]+ PostmanChunk[data chunk]+ end+++ SourceDescription -- decomposed into --> UnitDescription+ UnitDescription -- further decomposed into --> ChunkDescription+++ GitSource -- cloned locally<br />if not already local --> GitUnit+ GitHubSource -- cloned locally --> GitUnit+ PostmanSource -- Most sources\ndon't use units --> PostmanChunk+ FilesystemSource --> FilesystemUnit++ GitUnit -- git log -p --> GitRepositoryChunk+ FilesystemUnit --> FilesystemChunk++ style SourceDescription fill:#89553e+ style UnitDescription fill:#89553e+ style ChunkDescription fill:#89553e+```++#### Chunk to Detector Matching++```mermaid+flowchart LR+++ KeywordMatching["`**Keyword Matching**+_(Aho-Corsick)_++Match chunks to detectors based on the presence of specific keywords in the chunk`"]++ chunks --> KeywordMatching --> detectors+```++#### Secret Detection++```mermaid+flowchart LR++subgraph Detector+ direction RL+ subgraph DetectorDescription[" "]+ DetectorDescriptionText["`Detectors are the bits that actually check for the existence of a secret in a chunk, and (optionally) verify it`"]+ ExampleDetectors["`Example Detectors:+ * AWS+ * Azure+ * Twilio`"]+ end++ subgraph DetectorResponsibility[" "]+ direction LR++ De-Dupe-Detectors["`**De-Dupe-Detectors**++If multiple detectors keyword-match on the same chunk, we have some logic that chooses which detector will verify found secret (so we don't duplicate verification requests to externa APIs)`"]++ CollectMatches["`**Collect Matches**++Detector specific regexes are run against the matched chunks, resulting in unverified secrets`"]+ VerifyMatches["`**Verify Matches**++Optionally, observed unverified secrets are verified by attempting to use them against live services`"]++ De-Dupe-Detectors -- deduped detectors --> CollectMatches+ CollectMatches -- regex matched chunks --> VerifyMatches+ end++ style DetectorDescription fill:#89553e+ style DetectorDescriptionText fill:#89553e+end+```++#### Result Notification++```mermaid+flowchart LR++ Dispatcher["`**Dispatcher**++Results, verified or otherwise, are sent to a dispatcher to be sent to whichever place we're updating about the+results -- usually the command line.`"]++ results --> Dispatcher --> output+```+diff --git a/pkg/engine/engine.go b/pkg/engine/engine.goindex eb29b4ab7..e33018b67 100644--- a/pkg/engine/engine.go+++ b/pkg/engine/engine.go@@ -1,3 +1,6 @@+// Check the [process flow](docs/process_flow.md) and [concurrency](docs/concurrency.md) docs for+// something of a structural overview+package engineimport (