Building 70 Projects ranging from beginner to advanced so anyone can β learn from, build upon, use as a reference, or even copy directly. Owned by my cybersecurity company π
02-ARCHITECTURE.md
System Architecture
High-Level Pipeline
The scanner follows a linear pipeline: CLI parses arguments, the engine orchestrates, scanners extract and detect, and reporters format output.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CLI Layer (Typer) β
β β
β dlp-scan file ./data -f json -o results.json β
β dlp-scan db postgres://user:pass@host/db β
β dlp-scan network capture.pcap β
β dlp-scan report summary results.json β
ββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β ScanEngine β
β β
β Loads config ββΊ Builds DetectorRegistry ββΊ Selects β
β scanner type ββΊ Runs scan ββΊ Routes to reporter β
ββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββ
β
ββββββββββββββΌβββββββββββββ
βΌ βΌ βΌ
ββββββββββββββββ ββββββββββββ ββββββββββββββββ
β FileScanner β βDBScanner β βNetworkScannerβ
β β β β β β
β Walk dirs β β Schema β β PCAP parse β
β Extract text β β introspectβ β TCP reassemblyβ
β Run detectorsβ β Sample β β DNS exfil β
β β β rows β β DPI protocol β
β β β Detect β β Detect β
ββββββββ¬ββββββββ ββββββ¬ββββββ ββββββββ¬ββββββββ
β β β
ββββββββββββββββΌβββββββββββββββ
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DetectorRegistry β
β β
β PatternDetector ββΊ ContextBoost ββΊ CooccurrenceBoost β
β β β
β βββΊ EntropyDetector (parallel) β
β β
β Rules: PII | Financial | Credentials | Health β
ββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Reporter Layer β
β β
β ConsoleReporter ββΊ Rich tables with severity colors β
β JsonReporter ββΊ Structured JSON with metadata β
β SarifReporter ββΊ SARIF 2.1.0 for CI/CD pipelines β
β CsvReporter ββΊ Flat CSV for compliance teams β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Component Breakdown
CLI Layer
Purpose: Parse command-line arguments, propagate global options, route to the correct scan command or report utility.
Files: cli.py, commands/scan.py, commands/report.py
The root Typer app in cli.py defines a callback that captures --config, --verbose, and --version into Click's context object. The scan commands (file, db, network) are defined in commands/scan.py and registered as top-level commands through a register(app) function that calls app.command("file")(scan_file) for each. This avoids nesting under a scan subgroup while keeping the command definitions in their own module.
The report subgroup is a separate Typer instance added via app.add_typer(report_app, name="report"). It provides convert (JSON to other formats) and summary (print Rich table from JSON results).
ScanEngine
Purpose: Single orchestration point that connects config to scanners to reporters.
File: engine.py
The engine takes a ScanConfig and constructs a DetectorRegistry by unpacking detection configuration into individual parameters:
class ScanEngine:
def __init__(self, config: ScanConfig) -> None:
self._config = config
detection = config.detection
allowlist_vals = detection.allowlists.values
self._registry = DetectorRegistry(
enable_patterns=detection.enable_rules,
disable_patterns=detection.disable_rules,
allowlist_values=(
frozenset(allowlist_vals)
if allowlist_vals else None
),
context_window_tokens=(
detection.context_window_tokens
),
)
The engine exposes scan_files, scan_database, and scan_network, each of which constructs the appropriate scanner, runs it, and returns a ScanResult. Report generation uses a REPORTER_MAP dict that maps format strings to reporter classes.
DetectorRegistry
Purpose: Central hub that loads detection rules, filters them by enable/disable globs, and runs the full scoring pipeline against text.
File: detectors/registry.py
The registry loads all rules from four rule modules (PII, Financial, Credentials, Health), filters them using fnmatch.fnmatch against enable/disable patterns, and wraps the survivors in a PatternDetector. When detect() is called:
PatternDetectorruns all regex patterns, validates matches with checksums (Luhn, Mod-97, Mod-11), and filters against the allowlistapply_context_boostscans a token window around each match for relevant keywords and adjusts scores based on proximity_apply_cooccurrence_boostadds a bonus when multiple different PII types appear within 500 characters of each otherEntropyDetectorindependently finds high-entropy regions using a sliding window
Text Input
β
βΌ
βββββββββββββββββββββββββββ
β PatternDetector β
β β
β For each rule: β
β regex.finditer(text) β
β ββΊ allowlist filter β
β ββΊ validator (Luhn, β
β Mod-97, SSN area) β
β ββΊ base_score + boost β
βββββββββββ¬ββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β Context Boost β
β β
β Token window Β±10 tokens β
β Keyword proximity search β
β Distance-weighted boost β
β (0.05 to 0.35) β
βββββββββββ¬ββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β Co-occurrence Boost β
β β
β Different rule_ids β
β within 500 chars ββΊ +0.15β
βββββββββββ¬ββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β Entropy Detector β
β β
β Sliding 256-byte window β
β Shannon H >= 7.2 bits β
β Independent matches β
βββββββββββ¬ββββββββββββββββ
β
βΌ
DetectorMatch[]
Scanners
Purpose: Each scanner handles a different scan surface (files, databases, network) and converts raw data into text that the DetectorRegistry can process.
Files: scanners/file_scanner.py, scanners/db_scanner.py, scanners/network_scanner.py
All scanners follow the same Scanner protocol: a scan(target: str) -> ScanResult method. They share a common flow: iterate over targets, extract text, run detection, convert matches to findings via match_to_finding in scoring.py (which handles severity classification, compliance lookup, remediation, and redaction in one call), and aggregate into a ScanResult.
FileScanner walks a directory tree, applies extension and exclusion filters, dispatches each file to the appropriate extractor based on extension, and runs the detector on each TextChunk. The extension-to-extractor mapping is built once by _build_extension_map, which iterates over all extractor instances and indexes by their supported_extensions.
DatabaseScanner connects via URI scheme detection (postgres, mysql, mongodb, sqlite), introspects the schema to find text-type columns, samples rows using database-native sampling (TABLESAMPLE BERNOULLI for PostgreSQL, RAND() for MySQL, $sample for MongoDB), and scans column values.
NetworkScanner reads PCAP files via read_pcap, feeds packets into a FlowTracker for TCP reassembly, and processes DNS traffic inline through parse_dns and DnsExfilDetector. Each packet payload is also checked by detect_base64_payload for encoded data. After packet iteration, the scanner reassembles TCP flows, identifies the application protocol via identify_protocol, extracts text with protocol awareness (parse_http for HTTP bodies and sensitive headers, skip encrypted TLS/SSH, UTF-8 decode for everything else), and runs detection on the extracted text.
Extractors
Purpose: Convert binary and structured file formats into uniform TextChunk objects that carry both the extracted text and a Location describing where it came from.
Files: extractors/plaintext.py, extractors/pdf.py, extractors/office.py, extractors/structured.py, extractors/archive.py, extractors/email.py
All extractors implement the Extractor protocol: extract(path) -> list[TextChunk] and supported_extensions -> frozenset[str].
βββββββββββββββββββββββββββββββββββββββββββββββββ
β Extractor Protocol β
β extract(path) -> list[TextChunk] β
β supported_extensions -> frozenset[str] β
βββββββββββββββββββββββββββββββββββββββββββββββββ
β
ββββββ΄βββββ¬βββββββββββ¬βββββββββββ¬βββββββ
βΌ βΌ βΌ βΌ βΌ
Plaintext PDF Office Structured Archive
.txt .log .pdf .docx .csv .json .zip
.cfg .py .xlsx .xml .yaml .tar.gz
.html .md .xls .parquet .tar.bz2
.ts .go .avro
... .tsv
The PlaintextExtractor chunks files into 500-line blocks to keep memory bounded. Binary format extractors (PDF via PyMuPDF, DOCX via python-docx, XLSX via openpyxl) each return one TextChunk per page/sheet/section. The archive extractor recurses into compressed files up to a configurable depth with zip bomb protection (compression ratio threshold check).
Reporters
Purpose: Take a ScanResult and serialize it into the requested output format.
Files: reporters/console.py, reporters/json_report.py, reporters/sarif.py, reporters/csv_report.py
Each reporter has a generate(result) -> str method. The ConsoleReporter also has a display(result) method for Rich-formatted terminal output with severity-colored tables.
The JSON reporter outputs a structured document with scan_metadata, findings, and summary sections. The SARIF reporter produces a SARIF 2.1.0 document with tool.driver.rules, mapping severity levels through SARIF_SEVERITY_MAP (critical/high to "error", medium to "warning", low to "note"). The CSV reporter flattens findings into rows.
Data Models
Core Models
@dataclass(frozen=True, slots=True)
class Location:
source_type: str
uri: str
line: int | None = None
column: int | None = None
byte_offset: int | None = None
table_name: str | None = None
column_name: str | None = None
sheet_name: str | None = None
@dataclass(slots=True)
class Finding:
finding_id: str
rule_id: str
rule_name: str
severity: Severity
confidence: float
location: Location
redacted_snippet: str
compliance_frameworks: list[str]
remediation: str
detected_at: datetime
@dataclass(slots=True)
class ScanResult:
scan_id: str
tool_version: str
scan_started_at: datetime
scan_completed_at: datetime | None
targets_scanned: int
findings: list[Finding]
errors: list[str]
Location is frozen because it represents a fact about where something was found. Finding is mutable because fields like finding_id and detected_at get defaults from factory functions. ScanResult aggregates findings and provides computed properties (findings_by_severity, findings_by_rule, findings_by_framework) that group counts for summary reporting.
The TextChunk dataclass carries extracted text paired with its Location, forming the bridge between extractors and detectors. Every text fragment knows exactly where it came from, which lets findings carry precise location information through the pipeline.
Detection Models
@dataclass(frozen=True, slots=True)
class DetectionRule:
rule_id: str
rule_name: str
pattern: re.Pattern[str]
base_score: float
context_keywords: list[str]
validator: Callable[[str], bool] | None
compliance_frameworks: list[str]
@dataclass(frozen=True, slots=True)
class DetectorMatch:
rule_id: str
rule_name: str
start: int
end: int
matched_text: str
score: float
context_keywords: list[str]
compliance_frameworks: list[str]
DetectionRule is a specification: the regex pattern to match, the base confidence score, optional checksum validator, and context keywords. DetectorMatch is a result: what was found, where in the text, and the current score after validation. The score field gets modified through the boost pipeline (context, co-occurrence) before being mapped to a Severity level and placed into a Finding.
Configuration Architecture
ββββββββββββββββββββββββββββββββββββββββββββββ
β .dlp-scanner.yml β
β β
β scan: β
β file: { max_file_size_mb, recursive } β
β database: { sample_percentage } β
β network: { bpf_filter, max_packets } β
β detection: β
β min_confidence, enable_rules, β
β disable_rules, allowlists β
β compliance: { frameworks } β
β output: { format, redaction_style } β
β logging: { level, json_output } β
ββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββ
β load_config(path) -> ScanConfig β
β β
β 1. Check CLI --config flag β
β 2. Search candidates: β
β .dlp-scanner.yml β
β .dlp-scanner.yaml β
β ~/.dlp-scanner.yml β
β 3. Parse YAML via ruamel.yaml β
β 4. Validate with Pydantic 2.x models β
β 5. Return ScanConfig with defaults β
ββββββββββββββββββββββββββββββββββββββββββββββ
Every configuration value has a constant default defined in constants.py. The Pydantic models in config.py use these constants as field defaults, so a completely empty config file produces a working scanner. Constrained-choice fields (severity_threshold, format, redaction_style) use Literal types defined in constants.py (e.g., Literal["critical", "high", "medium", "low"]), so Pydantic rejects invalid values at parse time rather than silently accepting a typo. The config loader uses ruamel.yaml (not PyYAML) because it preserves comments and handles YAML 1.2.
The YAML structure uses a scan: top-level key to group scanner-specific config, while detection:, compliance:, output:, and logging: sit at root level. This mirrors how users think about configuration: "how to scan" vs. "what to detect" vs. "how to report".
Data Flow: File Scan
Step-by-step walkthrough of dlp-scan file ./data -f json:
1. Typer parses args
βββΊ main() callback stores config_path="" and verbose=False in ctx.obj
2. scan_file() receives ctx, target="./data", format="json"
βββΊ _run_scan() validates format, loads config, sets logging to WARNING
(WARNING for machine-readable formats keeps stdout clean)
3. ScanEngine(config) constructs DetectorRegistry
βββΊ Registry loads 29 rules from PII/Financial/Credential/Health modules
βββΊ Filters through enable_rules=["*"], disable_rules=[]
4. engine.scan_files("./data")
βββΊ FileScanner.scan() creates ScanResult, walks directory
5. For each file in ./data/**/*:
βββΊ Check extension against include_extensions
βββΊ Check path against exclude_patterns
βββΊ Check file size against max_file_size_mb
βββΊ Select extractor by extension (e.g. .csv -> CsvExtractor)
βββΊ extractor.extract(path) -> list[TextChunk]
6. For each TextChunk:
βββΊ registry.detect(chunk.text) -> list[DetectorMatch]
βββΊ PatternDetector: regex match + allowlist + validator
βββΊ apply_context_boost: keyword proximity scoring
βββΊ _apply_cooccurrence_boost: multi-PII bonus
βββΊ EntropyDetector: high-entropy region detection
7. For each DetectorMatch above min_confidence:
βββΊ match_to_finding(match, text, location, redaction_style)
βββΊ score_to_severity(match.score) -> Severity
βββΊ get_frameworks_for_rule(match.rule_id) -> compliance list
βββΊ get_remediation_for_rule(match.rule_id) -> guidance string
βββΊ redact(chunk.text, start, end, style) -> snippet
βββΊ Append Finding to ScanResult
8. Back in _run_scan():
βββΊ engine.generate_report(result, "json")
βββΊ JsonReporter().generate(result) -> JSON string
βββΊ typer.echo(output) -> stdout
Design Patterns
Protocol-Based Polymorphism
The codebase uses Python's typing.Protocol instead of abstract base classes for extension points. The Extractor, Scanner, and Detector protocols define structural interfaces without requiring inheritance.
class Extractor(Protocol):
def extract(self, path: str) -> list[TextChunk]: ...
@property
def supported_extensions(self) -> frozenset[str]: ...
Any class with matching method signatures satisfies the protocol. This means you can add a new extractor (say, for .pptx files) without importing the base module. The type checker verifies compliance; the runtime never checks inheritance.
Why not ABCs: Abstract base classes force an import dependency and mandate super().__init__() chains. Protocols are lighter and match Python's duck typing philosophy. Since extractors are stateless (no shared state or lifecycle), there is nothing an ABC would provide beyond the type contract.
Registry Pattern
The DetectorRegistry centralizes rule management: loading, filtering, and execution. Individual rule modules (pii.py, financial.py, credentials.py, health.py) each export a list of DetectionRule objects. The registry merges them into ALL_RULES, applies glob filtering, and wraps the result in a PatternDetector.
This keeps rule definitions declarative. Adding a new rule is a matter of appending a DetectionRule to the appropriate list. The registry handles filtering and execution without rule authors needing to understand the scoring pipeline.
Command Registration Pattern
CLI commands are defined in commands/scan.py as plain functions and registered on the root app through a register(app) function:
def register(app: typer.Typer) -> None:
app.command("file")(scan_file)
app.command("db")(scan_db)
app.command("network")(scan_network)
This achieves top-level commands (dlp-scan file, not dlp-scan scan file) while keeping the command logic out of cli.py. The _run_scan helper deduplicates the shared logic (config loading, format validation, output routing) across all three scan types.
Compliance Mapping
The compliance module maps rule IDs to regulatory frameworks and remediation guidance using two static dictionaries:
RULE_FRAMEWORK_MAP: rule_id -> [frameworks]
RULE_REMEDIATION_MAP: rule_id -> guidance string
Rule IDs match actual detection rules (e.g., FIN_CREDIT_CARD_VISA, FIN_CREDIT_CARD_MC, not a generic FIN_CREDIT_CARD). Network exfiltration indicators (NET_DNS_EXFIL_*, NET_ENCODED_*) are also mapped. Every rule has a remediation entry with specific guidance text; unknown rules fall back to a generic default.
When a DetectorMatch is converted to a Finding via match_to_finding in scoring.py, the function calls get_frameworks_for_rule and get_remediation_for_rule to decorate the finding with compliance metadata. If the detection rule itself also carries compliance_frameworks, both sets are merged.
This design keeps detection rules independent of compliance logic. The PII module does not need to know that HIPAA cares about SSNs. The compliance module owns that mapping, and it can be updated independently when regulations change.
Redaction Pipeline
matched text
β
βΌ
style == "none"? βyesββΊ raw snippet with context
β no
βΌ
style == "full"? βyesββΊ [REDACTED] with context
β no
βΌ
_partial_redact()
β
ββ 9+ digit number ββΊ *****6789 (mask all but last 4)
ββ email address ββΊ j****@example.com
ββ generic string ββΊ keep last 25%
β
βΌ
_build_snippet()
β
ββ Β±20 chars context ββΊ "...SSN: *****6789 for..."
Partial redaction is the default because it gives analysts enough to identify the data type and triage priority without exposing the full sensitive value. The last 4 digits of SSNs and credit cards are considered non-sensitive by PCI-DSS (you can print them on receipts), so partial redaction for those types is compliant.
Network Analysis Architecture
ββββββββββββββββββββββββββββββββββββββββββββββ
β PCAP File β
β (.pcap or .pcapng) β
ββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββ
β pcap.read_pcap() β
β β
β dpkt.pcap.Reader / dpkt.pcapng.Reader β
β Parse Ethernet -> IP -> TCP/UDP β
β Yield PacketInfo(src_ip, dst_ip, β
β src_port, dst_port, payload, β
β tcp_seq, tcp_flags) β
ββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βββββββββ΄ββββββββ
βΌ βΌ
βββββββββββββββ βββββββββββββββββ
βFlowTracker β βDnsExfilDetectorβ
β β β β
βTrack by β βLabel length β
β4-tuple key β βcheck (>50) β
β β β β
βReassemble β βSubdomain β
βTCP streams β βentropy (>4.0) β
βby seq num β β β
β β βQNAME length β
βDedup retx β βcheck (>100) β
ββββββββ¬βββββββ β β
β βTXT volume β
βΌ βratio check β
βββββββββββββββ βββββββββ¬ββββββββ
βProtocol ID β β
β(DPI) β βΌ
β β ExfilIndicator[]
βHTTP: method β
β prefix β
βTLS: \x16\x03β
βSSH: SSH- β
βSMTP: 220 β
ββββββββ¬βββββββ
β
βΌ
Reassembled text
sent to DetectorRegistry
The flow tracker creates bidirectional flow keys by sorting the forward and reverse 4-tuples, so (A, B, 80, 12345) and (B, A, 12345, 80) map to the same flow. TCP reassembly sorts segments by sequence number and deduplicates retransmissions. Without reassembly, a credit card number split across two TCP segments would be missed.
The DNS exfiltration detector runs independently of the regex-based detectors. It analyzes DNS queries for encoding signals: base64-like entropy in subdomain labels, abnormally long labels, long QNAMEs, and suspicious TXT query volume ratios. The OilRig APT campaign used exactly these patterns to exfiltrate stolen documents through DNS tunneling to C2 infrastructure.
Error Handling Strategy
Errors are collected, not thrown. Each scanner appends error messages to ScanResult.errors and continues scanning the remaining targets. The CLI checks result.errors after the scan completes and exits with code 1 if any errors occurred, but the partial results are still reported.
This "collect and continue" approach means a single corrupt PDF in a directory of 10,000 files does not abort the scan. The Equifax breach investigation found that scanning tools that failed on individual files often left entire directories unscanned, which is why modern DLP tools treat extraction failures as warnings rather than fatal errors.
Performance Considerations
File scanning is I/O-bound. The scanner processes files sequentially to avoid overwhelming disk I/O. Text extraction for binary formats (PDF, Office) can be CPU-intensive, but these files are typically a small fraction of the total.
Detection scales linearly with text length times rule count. With 29 rules and an average text chunk of 500 lines, a single detection pass takes microseconds. The entropy detector is more expensive due to its sliding window, so it only runs when enabled and only against high-level text chunks (not individual regex matches).
Memory stays bounded through chunking. The plaintext extractor reads 500 lines at a time. Archive extraction enforces depth limits and zip bomb ratio checks.
Key Files Reference
cli.py- Entry point, global options, Typer appengine.py- Orchestration, connects config to scanners to reportersconfig.py- Pydantic models, YAML loading, config searchconstants.py- All magic numbers, thresholds, type literalsmodels.py- Finding, Location, ScanResult, TextChunkcompliance.py- Rule-to-framework mapping, severity classificationscoring.py- Shared match-to-finding conversion for all scannersredaction.py- Partial/full/none redaction strategiesdetectors/registry.py- Rule loading, filtering, scoring pipelinedetectors/pattern.py- Regex matching with allowlist and checksum validationdetectors/context.py- Keyword proximity boost, co-occurrence boostdetectors/entropy.py- Shannon entropy detection, sliding windowdetectors/rules/- Rule definitions (pii, financial, credentials, health)extractors/- Text extraction from 14+ file formatsscanners/- File, database, network scan implementationsnetwork/- PCAP parsing, flow tracking, DPI, DNS exfiltrationreporters/- Console, JSON, SARIF, CSV outputcommands/- CLI command implementations (scan, report)
Next Steps
Now that you understand the architecture:
- Read 03-IMPLEMENTATION.md for the code walkthrough
- Try modifying a detection rule in
detectors/rules/pii.pyto see how the scoring pipeline responds