Skip to content

Building 70 Projects ranging from beginner to advanced so anyone can β€” learn from, build upon, use as a reference, or even copy directly. Owned by my cybersecurity company πŸ‘‡

docs: add Portia secrets scanner design document

CarterPerez-devc6c844db4780
diff --git a/docs/plans/2026-02-20-secrets-scanner-design.md b/docs/plans/2026-02-20-secrets-scanner-design.md
new file mode 100644
index 00000000..03480a2d
--- /dev/null
+++ b/docs/plans/2026-02-20-secrets-scanner-design.md
@@ -0,0 +1,334 @@
+# Portia β€” Secrets Scanner Design Document
+
+**Date**: 2026-02-20
+**Project**: Cybersecurity-Projects / intermediate / secrets-scanner
+**Replaces**: Container Security Scanner (overlap with Docker Security Audit)
+**Module**: `github.com/CarterPerez-dev/portia`
+
+---
+
+## Overview
+
+Portia is a Go CLI tool that scans git repositories and directories for leaked API keys, passwords, tokens, and other secrets. It combines keyword pre-filtering, regex pattern matching, Shannon entropy analysis, and Have I Been Pwned breach verification into a pipeline architecture modeled after production scanners like Gitleaks and TruffleHog.
+
+**Scan targets**: Git history (via go-git in-process) and filesystem directories.
+**Not in scope**: Docker image layer extraction (covered by the existing Docker Security Audit project).
+
+---
+
+## CLI Structure
+
+### Commands
+
+```
+portia scan [path] Scan a directory/file for secrets
+portia git [path] Scan git history for secrets
+portia init Generate .portia.toml config
+portia config rules List all built-in detection rules
+portia config test <rule> Test a specific rule against sample input
+```
+
+### Global Flags
+
+```
+--format string Output format: terminal (default), json, sarif
+--rules string Path to custom rules TOML file
+--verbose/-v Show full finding details (line context, entropy scores)
+--no-color Disable colored output
+--exit-code int Exit code when secrets found (default: 1, 0 for CI soft-fail)
+```
+
+### Scan Flags
+
+```
+portia scan:
+ --exclude []string Glob patterns to skip (e.g., "vendor/**", "*.lock")
+ --max-size string Skip files larger than (default: "1MB")
+ --entropy float Override entropy threshold (default: per-rule)
+ --hibp Enable HIBP password breach checking
+
+portia git:
+ --since string Only scan commits after date (e.g., "7 days ago")
+ --branch string Scan specific branch (default: all)
+ --depth int Max commit depth (default: unlimited)
+ --staged-only Only scan currently staged changes (pre-commit hook mode)
+ --hibp Enable HIBP password breach checking
+```
+
+---
+
+## Pipeline Architecture
+
+```
+Source --> Chunker --> Keyword Filter --> Detector Pool --> Verifier (HIBP) --> Reporter
+```
+
+### Stage 1: Source
+
+Two implementations behind a common interface:
+- **DirSource**: Walks filesystem, respects .gitignore + .portiaignore, skips binaries and large files.
+- **GitSource**: Uses go-git to iterate commits, produces diffs (added lines only).
+
+### Stage 2: Chunker
+
+Splits source output into Chunk structs containing content, file path, line number, and optional git metadata (commit SHA, author, date).
+
+### Stage 3: Keyword Filter
+
+The critical performance optimization. Each rule declares case-insensitive keywords. Before any regex runs, the engine checks whether the chunk contains at least one keyword. This eliminates ~95% of chunks. Only rules whose keywords matched are forwarded to the detector pool for that chunk.
+
+### Stage 4: Detector Pool
+
+Bounded concurrency via errgroup.SetLimit(runtime.NumCPU()). For each chunk that passed keyword filtering, matched rules execute:
+1. Regex match on captured group
+2. Entropy check on captured value (if rule has a threshold)
+3. Stopword/allowlist filtering
+4. Structural validation (assignment operator present, not a template variable)
+
+### Stage 5: Verifier (HIBP)
+
+Post-processing enrichment, only when --hibp flag is set. Classifies each finding by secret type, routes passwords to HIBP k-anonymity API, marks results as verified_breach, not_breached, or skipped.
+
+### Stage 6: Reporter
+
+Three implementations behind a Reporter interface:
+- **TerminalReporter**: Angela-style colored output with severity colors, spinners, summary.
+- **JSONReporter**: Structured JSON to stdout.
+- **SARIFReporter**: SARIF v2.1.0 for GitHub/GitLab integration.
+
+### Concurrency Model
+
+- Channels buffered at 256 to prevent backpressure stalls
+- errgroup.WithContext for detector pool β€” first fatal error cancels everything
+- HIBP lookups run as a separate bounded pool (20 concurrent, 50 req/s rate limit) after scanning completes
+- Context propagation throughout for clean Ctrl+C cancellation
+
+---
+
+## Core Types
+
+### Finding
+
+```
+Finding {
+ RuleID string
+ Description string
+ Severity string // critical, high, medium, low
+ Match string
+ Secret string // The captured secret value (redactable)
+ Entropy float64
+ FilePath string
+ LineNumber int
+ LineContent string
+ CommitSHA string // Empty for dir scan
+ Author string // Empty for dir scan
+ CommitDate time.Time // Empty for dir scan
+ HIBPStatus string // "breached", "clean", "skipped", "unchecked"
+ BreachCount int
+}
+```
+
+### Rule
+
+```
+Rule {
+ ID string
+ Description string
+ Severity string
+ Keywords []string
+ Pattern *regexp.Regexp
+ SecretGroup int
+ Entropy *float64 // nil = skip entropy check
+ Allowlist Allowlist
+ SecretType SecretType // Password, APIKey, Token, PrivateKey, etc.
+}
+```
+
+### Chunk
+
+```
+Chunk {
+ Content string
+ FilePath string
+ LineStart int
+ CommitSHA string
+ Author string
+ CommitDate time.Time
+}
+```
+
+---
+
+## Rule Engine
+
+### 50+ Built-in Rules
+
+| Category | Count | Examples | Severity |
+|---|---|---|---|
+| Cloud providers | ~10 | AWS Access Key, AWS Secret Key, GCP API Key, Azure Client Secret, DigitalOcean PAT | Critical |
+| Source control | ~8 | GitHub PAT (classic), Fine-grained PAT, OAuth, GitLab PAT, Bitbucket App Password | Critical |
+| Payment/SaaS | ~8 | Stripe Secret Key, Stripe Restricted Key, SendGrid, Twilio, Shopify | Critical |
+| Communication | ~5 | Slack Bot/User/App Token, Slack Webhook, Discord Bot Token | High |
+| AI/ML services | ~4 | OpenAI, Anthropic, HuggingFace, Replicate | High |
+| Infrastructure | ~5 | Heroku, Terraform Cloud, Vault Token, PlanetScale, Supabase | Critical |
+| Cryptographic | ~3 | Private Keys (RSA/EC/DSA/OPENSSH), PGP, X.509 | Critical |
+| Authentication | ~4 | JWT, Bearer Token, Basic Auth, OAuth Client Secret | High |
+| Database | ~5 | PostgreSQL URI, MySQL URI, MongoDB URI, Redis URI, Generic DB URI | Critical |
+| Generic | ~3 | Generic API Key (entropy 3.7), Generic Secret (entropy 3.7), Generic Password | Medium |
+
+### False Positive Defenses (5 Layers)
+
+**Layer 1 β€” Keyword pre-filter**: Part of the pipeline. No keyword match = chunk skipped.
+
+**Layer 2 β€” Structural validation**: Generic rules require an assignment operator between keyword and value (=, :, =>, :=, ||).
+
+**Layer 3 β€” Stopwords**: ~1,500 common programming words. If captured secret contains a stopword, finding is dropped.
+
+**Layer 4 β€” Allowlists**:
+- Global path allowlist: go.sum, package-lock.json, pnpm-lock.yaml, *.min.js, vendor/, node_modules/, .git/, dist-info/
+- Global value allowlist: placeholder values (EXAMPLE_KEY, your-api-key-here, xxxx, ${VAR}, {{template}}, os.Getenv(...))
+- Per-rule allowlists for rule-specific exclusions
+
+**Layer 5 β€” Entropy validation**: Post-filter on captured secret value.
+- Base64 charset: 4.5 threshold
+- Hex charset: 3.0 threshold
+- Alphanumeric (generic rules): 3.7 threshold
+- Provider-specific rules with known prefixes: 3.0 threshold
+
+### TOML Override Format (.portia.toml)
+
+Users can disable built-in rules, add path excludes, allowlist values, and define custom rules. Custom rules use the same structure as built-in rules (id, description, severity, keywords, pattern, secret-group, entropy).
+
+---
+
+## HIBP Integration
+
+### K-Anonymity Protocol
+
+1. SHA-1 hash the plaintext password locally
+2. Send only the 5-character hex prefix to api.pwnedpasswords.com/range/{prefix}
+3. Receive ~800 suffixes with breach counts
+4. Match locally β€” the API never learns the password
+
+### Architecture
+
+- **HTTP Client**: hashicorp/go-retryablehttp with custom transport (MaxIdleConnsPerHost: 50, HTTP/2, TLS 1.2+)
+- **Cache**: hashicorp/golang-lru/v2/expirable β€” LRU cache keyed by 5-char prefix, 24-hour TTL
+- **Circuit Breaker**: sony/gobreaker β€” trips after 5 consecutive failures, 30s recovery timeout
+- **Rate Limiter**: golang.org/x/time/rate at 50 req/s with burst of 10
+- **Concurrency**: errgroup.SetLimit(20) for parallel lookups
+
+### Secret Classification Router
+
+Only human-chosen passwords are sent to HIBP. Machine-generated secrets (API keys, tokens, private keys) are skipped based on:
+- Known format prefixes (AKIA, ghp_, sk_live_, xoxb-, SG., sk-)
+- Keyword context (password, passwd, pwd in surrounding code)
+- Entropy heuristic (< 3.5 bits + < 32 chars = likely human-chosen)
+
+### Privacy Safeguards
+
+- Never transmit full hash or plaintext
+- Logger set to nil to prevent logging of /range/{prefix} URLs
+- Results reported as boolean compromised/not-compromised (breach count available in verbose mode only)
+- HTTPS exclusively, TLS 1.2+ enforced
+
+---
+
+## Terminal Output Design
+
+### Color Mapping
+
+- CRITICAL: red
+- HIGH: bold red
+- MEDIUM: yellow
+- LOW: cyan
+- File paths: hi-cyan
+- Rule IDs: blue
+- Secret values: dim (partially redacted)
+- Line numbers: dim italic
+- HIBP breach warnings: bold yellow
+
+### Output Sections
+
+1. Banner (alternating red/blue ASCII art)
+2. Spinner during scan (cyan braille frame + magenta message)
+3. Findings grouped by file, sorted by severity
+4. HIBP breach check section (when --hibp used)
+5. Summary (files scanned, rules evaluated, secrets found, duration)
+
+### Redaction
+
+Secret values are partially redacted by default: first 6 characters shown, rest replaced with "...". Full values shown only with --verbose flag.
+
+---
+
+## Project Structure
+
+```
+PROJECTS/intermediate/secrets-scanner/
+ cmd/portia/main.go
+ internal/
+ cli/ root.go, scan.go, git.go, init.go, config.go
+ engine/ pipeline.go, chunk.go, detector.go, filter.go
+ source/ source.go, directory.go, git.go
+ rules/ registry.go, builtin.go, custom.go, entropy.go
+ hibp/ client.go, cache.go, breaker.go, classify.go
+ report/ reporter.go, terminal.go, json.go, sarif.go
+ ui/ banner.go, color.go, symbol.go, spinner.go
+ config/ config.go
+ pkg/types/ types.go
+ testdata/ repos/, fixtures/, golden/
+ learn/ 00-OVERVIEW.md through 04-CHALLENGES.md
+ .portia.toml, .golangci.yml, .gitignore, Justfile, go.mod, LICENSE, README.md
+```
+
+### Dependencies
+
+```
+github.com/spf13/cobra
+github.com/fatih/color
+github.com/go-git/go-git/v5
+github.com/pelletier/go-toml/v2
+github.com/hashicorp/golang-lru/v2
+github.com/sony/gobreaker/v2
+github.com/hashicorp/go-retryablehttp
+golang.org/x/sync
+golang.org/x/time
+```
+
+---
+
+## Testing Strategy
+
+### Unit Tests
+
+- **rules/**: Table-driven tests for each of the 50+ rules with true positives, true negatives, and known false positive patterns.
+- **engine/**: Pipeline stages tested in isolation (chunker, keyword filter, detector, structural validation).
+- **rules/entropy.go**: Shannon entropy tested against known values from detect-secrets' test suite.
+- **hibp/**: HTTP test server mocks, cache behavior, circuit breaker states, classification routing.
+- **source/**: Directory walker respects gitignore, skips binaries, respects max-size. Git source produces correct diffs.
+- **report/**: JSON and SARIF output compared against golden files.
+- **config/**: TOML parsing, rule merging, disable/override behavior.
+
+### Integration Tests
+
+- testdata/fixtures/ with planted secrets across Python, YAML, .env, JSON, Go, JS files.
+- testdata/repos/ with small git repos containing commits with secrets added then deleted.
+- Golden file tests for JSON and SARIF output.
+
+### CI
+
+- go test -race ./... (mandatory)
+- golangci-lint v2 with Tier 1/2/3 linters
+- gofumpt + golines (80-char max)
+
+---
+
+## Learn Folder
+
+Grounded in three real-world incidents:
+- **Uber 2022**: Hardcoded HCP Vault credentials in PowerShell script on GitHub
+- **Samsung 2022**: Leaked SmartThings secrets via accidental git push
+- **CircleCI 2023**: Rotated all customer secrets after infrastructure breach
+
+Documentation follows the standard learn/ template: 00-OVERVIEW, 01-CONCEPTS, 02-ARCHITECTURE, 03-IMPLEMENTATION, 04-CHALLENGES.