Building 70 Projects ranging from beginner to advanced so anyone can β learn from, build upon, use as a reference, or even copy directly. Owned by my cybersecurity company π
Go aicertificationcloud-engineeringcybercybersecuritycybersecurity-analystcybersecurity-certificationcybersecurity-certificationscybersecurity-educationcybersecurity-jobscybersecurity-portfoliocybersecurity-projectscybersecurity-toolsgrchackinglearningpentestingproject-repositoryprojetcspython
docs: add Portia secrets scanner design document
CarterPerez-devc6c844db4780
diff --git a/docs/plans/2026-02-20-secrets-scanner-design.md b/docs/plans/2026-02-20-secrets-scanner-design.mdnew file mode 100644index 00000000..03480a2d--- /dev/null+++ b/docs/plans/2026-02-20-secrets-scanner-design.md@@ -0,0 +1,334 @@+# Portia β Secrets Scanner Design Document++**Date**: 2026-02-20+**Project**: Cybersecurity-Projects / intermediate / secrets-scanner+**Replaces**: Container Security Scanner (overlap with Docker Security Audit)+**Module**: `github.com/CarterPerez-dev/portia`++---++## Overview++Portia is a Go CLI tool that scans git repositories and directories for leaked API keys, passwords, tokens, and other secrets. It combines keyword pre-filtering, regex pattern matching, Shannon entropy analysis, and Have I Been Pwned breach verification into a pipeline architecture modeled after production scanners like Gitleaks and TruffleHog.++**Scan targets**: Git history (via go-git in-process) and filesystem directories.+**Not in scope**: Docker image layer extraction (covered by the existing Docker Security Audit project).++---++## CLI Structure++### Commands++```+portia scan [path] Scan a directory/file for secrets+portia git [path] Scan git history for secrets+portia init Generate .portia.toml config+portia config rules List all built-in detection rules+portia config test <rule> Test a specific rule against sample input+```++### Global Flags++```+--format string Output format: terminal (default), json, sarif+--rules string Path to custom rules TOML file+--verbose/-v Show full finding details (line context, entropy scores)+--no-color Disable colored output+--exit-code int Exit code when secrets found (default: 1, 0 for CI soft-fail)+```++### Scan Flags++```+portia scan:+ --exclude []string Glob patterns to skip (e.g., "vendor/**", "*.lock")+ --max-size string Skip files larger than (default: "1MB")+ --entropy float Override entropy threshold (default: per-rule)+ --hibp Enable HIBP password breach checking++portia git:+ --since string Only scan commits after date (e.g., "7 days ago")+ --branch string Scan specific branch (default: all)+ --depth int Max commit depth (default: unlimited)+ --staged-only Only scan currently staged changes (pre-commit hook mode)+ --hibp Enable HIBP password breach checking+```++---++## Pipeline Architecture++```+Source --> Chunker --> Keyword Filter --> Detector Pool --> Verifier (HIBP) --> Reporter+```++### Stage 1: Source++Two implementations behind a common interface:+- **DirSource**: Walks filesystem, respects .gitignore + .portiaignore, skips binaries and large files.+- **GitSource**: Uses go-git to iterate commits, produces diffs (added lines only).++### Stage 2: Chunker++Splits source output into Chunk structs containing content, file path, line number, and optional git metadata (commit SHA, author, date).++### Stage 3: Keyword Filter++The critical performance optimization. Each rule declares case-insensitive keywords. Before any regex runs, the engine checks whether the chunk contains at least one keyword. This eliminates ~95% of chunks. Only rules whose keywords matched are forwarded to the detector pool for that chunk.++### Stage 4: Detector Pool++Bounded concurrency via errgroup.SetLimit(runtime.NumCPU()). For each chunk that passed keyword filtering, matched rules execute:+1. Regex match on captured group+2. Entropy check on captured value (if rule has a threshold)+3. Stopword/allowlist filtering+4. Structural validation (assignment operator present, not a template variable)++### Stage 5: Verifier (HIBP)++Post-processing enrichment, only when --hibp flag is set. Classifies each finding by secret type, routes passwords to HIBP k-anonymity API, marks results as verified_breach, not_breached, or skipped.++### Stage 6: Reporter++Three implementations behind a Reporter interface:+- **TerminalReporter**: Angela-style colored output with severity colors, spinners, summary.+- **JSONReporter**: Structured JSON to stdout.+- **SARIFReporter**: SARIF v2.1.0 for GitHub/GitLab integration.++### Concurrency Model++- Channels buffered at 256 to prevent backpressure stalls+- errgroup.WithContext for detector pool β first fatal error cancels everything+- HIBP lookups run as a separate bounded pool (20 concurrent, 50 req/s rate limit) after scanning completes+- Context propagation throughout for clean Ctrl+C cancellation++---++## Core Types++### Finding++```+Finding {+ RuleID string+ Description string+ Severity string // critical, high, medium, low+ Match string+ Secret string // The captured secret value (redactable)+ Entropy float64+ FilePath string+ LineNumber int+ LineContent string+ CommitSHA string // Empty for dir scan+ Author string // Empty for dir scan+ CommitDate time.Time // Empty for dir scan+ HIBPStatus string // "breached", "clean", "skipped", "unchecked"+ BreachCount int+}+```++### Rule++```+Rule {+ ID string+ Description string+ Severity string+ Keywords []string+ Pattern *regexp.Regexp+ SecretGroup int+ Entropy *float64 // nil = skip entropy check+ Allowlist Allowlist+ SecretType SecretType // Password, APIKey, Token, PrivateKey, etc.+}+```++### Chunk++```+Chunk {+ Content string+ FilePath string+ LineStart int+ CommitSHA string+ Author string+ CommitDate time.Time+}+```++---++## Rule Engine++### 50+ Built-in Rules++| Category | Count | Examples | Severity |+|---|---|---|---|+| Cloud providers | ~10 | AWS Access Key, AWS Secret Key, GCP API Key, Azure Client Secret, DigitalOcean PAT | Critical |+| Source control | ~8 | GitHub PAT (classic), Fine-grained PAT, OAuth, GitLab PAT, Bitbucket App Password | Critical |+| Payment/SaaS | ~8 | Stripe Secret Key, Stripe Restricted Key, SendGrid, Twilio, Shopify | Critical |+| Communication | ~5 | Slack Bot/User/App Token, Slack Webhook, Discord Bot Token | High |+| AI/ML services | ~4 | OpenAI, Anthropic, HuggingFace, Replicate | High |+| Infrastructure | ~5 | Heroku, Terraform Cloud, Vault Token, PlanetScale, Supabase | Critical |+| Cryptographic | ~3 | Private Keys (RSA/EC/DSA/OPENSSH), PGP, X.509 | Critical |+| Authentication | ~4 | JWT, Bearer Token, Basic Auth, OAuth Client Secret | High |+| Database | ~5 | PostgreSQL URI, MySQL URI, MongoDB URI, Redis URI, Generic DB URI | Critical |+| Generic | ~3 | Generic API Key (entropy 3.7), Generic Secret (entropy 3.7), Generic Password | Medium |++### False Positive Defenses (5 Layers)++**Layer 1 β Keyword pre-filter**: Part of the pipeline. No keyword match = chunk skipped.++**Layer 2 β Structural validation**: Generic rules require an assignment operator between keyword and value (=, :, =>, :=, ||).++**Layer 3 β Stopwords**: ~1,500 common programming words. If captured secret contains a stopword, finding is dropped.++**Layer 4 β Allowlists**:+- Global path allowlist: go.sum, package-lock.json, pnpm-lock.yaml, *.min.js, vendor/, node_modules/, .git/, dist-info/+- Global value allowlist: placeholder values (EXAMPLE_KEY, your-api-key-here, xxxx, ${VAR}, {{template}}, os.Getenv(...))+- Per-rule allowlists for rule-specific exclusions++**Layer 5 β Entropy validation**: Post-filter on captured secret value.+- Base64 charset: 4.5 threshold+- Hex charset: 3.0 threshold+- Alphanumeric (generic rules): 3.7 threshold+- Provider-specific rules with known prefixes: 3.0 threshold++### TOML Override Format (.portia.toml)++Users can disable built-in rules, add path excludes, allowlist values, and define custom rules. Custom rules use the same structure as built-in rules (id, description, severity, keywords, pattern, secret-group, entropy).++---++## HIBP Integration++### K-Anonymity Protocol++1. SHA-1 hash the plaintext password locally+2. Send only the 5-character hex prefix to api.pwnedpasswords.com/range/{prefix}+3. Receive ~800 suffixes with breach counts+4. Match locally β the API never learns the password++### Architecture++- **HTTP Client**: hashicorp/go-retryablehttp with custom transport (MaxIdleConnsPerHost: 50, HTTP/2, TLS 1.2+)+- **Cache**: hashicorp/golang-lru/v2/expirable β LRU cache keyed by 5-char prefix, 24-hour TTL+- **Circuit Breaker**: sony/gobreaker β trips after 5 consecutive failures, 30s recovery timeout+- **Rate Limiter**: golang.org/x/time/rate at 50 req/s with burst of 10+- **Concurrency**: errgroup.SetLimit(20) for parallel lookups++### Secret Classification Router++Only human-chosen passwords are sent to HIBP. Machine-generated secrets (API keys, tokens, private keys) are skipped based on:+- Known format prefixes (AKIA, ghp_, sk_live_, xoxb-, SG., sk-)+- Keyword context (password, passwd, pwd in surrounding code)+- Entropy heuristic (< 3.5 bits + < 32 chars = likely human-chosen)++### Privacy Safeguards++- Never transmit full hash or plaintext+- Logger set to nil to prevent logging of /range/{prefix} URLs+- Results reported as boolean compromised/not-compromised (breach count available in verbose mode only)+- HTTPS exclusively, TLS 1.2+ enforced++---++## Terminal Output Design++### Color Mapping++- CRITICAL: red+- HIGH: bold red+- MEDIUM: yellow+- LOW: cyan+- File paths: hi-cyan+- Rule IDs: blue+- Secret values: dim (partially redacted)+- Line numbers: dim italic+- HIBP breach warnings: bold yellow++### Output Sections++1. Banner (alternating red/blue ASCII art)+2. Spinner during scan (cyan braille frame + magenta message)+3. Findings grouped by file, sorted by severity+4. HIBP breach check section (when --hibp used)+5. Summary (files scanned, rules evaluated, secrets found, duration)++### Redaction++Secret values are partially redacted by default: first 6 characters shown, rest replaced with "...". Full values shown only with --verbose flag.++---++## Project Structure++```+PROJECTS/intermediate/secrets-scanner/+ cmd/portia/main.go+ internal/+ cli/ root.go, scan.go, git.go, init.go, config.go+ engine/ pipeline.go, chunk.go, detector.go, filter.go+ source/ source.go, directory.go, git.go+ rules/ registry.go, builtin.go, custom.go, entropy.go+ hibp/ client.go, cache.go, breaker.go, classify.go+ report/ reporter.go, terminal.go, json.go, sarif.go+ ui/ banner.go, color.go, symbol.go, spinner.go+ config/ config.go+ pkg/types/ types.go+ testdata/ repos/, fixtures/, golden/+ learn/ 00-OVERVIEW.md through 04-CHALLENGES.md+ .portia.toml, .golangci.yml, .gitignore, Justfile, go.mod, LICENSE, README.md+```++### Dependencies++```+github.com/spf13/cobra+github.com/fatih/color+github.com/go-git/go-git/v5+github.com/pelletier/go-toml/v2+github.com/hashicorp/golang-lru/v2+github.com/sony/gobreaker/v2+github.com/hashicorp/go-retryablehttp+golang.org/x/sync+golang.org/x/time+```++---++## Testing Strategy++### Unit Tests++- **rules/**: Table-driven tests for each of the 50+ rules with true positives, true negatives, and known false positive patterns.+- **engine/**: Pipeline stages tested in isolation (chunker, keyword filter, detector, structural validation).+- **rules/entropy.go**: Shannon entropy tested against known values from detect-secrets' test suite.+- **hibp/**: HTTP test server mocks, cache behavior, circuit breaker states, classification routing.+- **source/**: Directory walker respects gitignore, skips binaries, respects max-size. Git source produces correct diffs.+- **report/**: JSON and SARIF output compared against golden files.+- **config/**: TOML parsing, rule merging, disable/override behavior.++### Integration Tests++- testdata/fixtures/ with planted secrets across Python, YAML, .env, JSON, Go, JS files.+- testdata/repos/ with small git repos containing commits with secrets added then deleted.+- Golden file tests for JSON and SARIF output.++### CI++- go test -race ./... (mandatory)+- golangci-lint v2 with Tier 1/2/3 linters+- gofumpt + golines (80-char max)++---++## Learn Folder++Grounded in three real-world incidents:+- **Uber 2022**: Hardcoded HCP Vault credentials in PowerShell script on GitHub+- **Samsung 2022**: Leaked SmartThings secrets via accidental git push+- **CircleCI 2023**: Rotated all customer secrets after infrastructure breach++Documentation follows the standard learn/ template: 00-OVERVIEW, 01-CONCEPTS, 02-ARCHITECTURE, 03-IMPLEMENTATION, 04-CHALLENGES.