Building 70 Projects ranging from beginner to advanced so anyone can β learn from, build upon, use as a reference, or even copy directly. Owned by my cybersecurity company π
βββββββ βββ βββββββ ββββββββ βββββββ ββββββ ββββ βββ
βββββββββββ ββββββββ βββββββββββββββββββββββββββββ βββ
βββ ββββββ βββββββββββββββββββββββββ ββββββββββββββ βββ
βββ ββββββ βββββββ βββββββββββββββββ ββββββββββββββββββ
βββββββββββββββββββ βββββββββββββββββββ ββββββ ββββββ
βββββββ βββββββββββ ββββββββ ββββββββββ ββββββ βββββ
Data Loss Prevention scanner for files, databases, and network traffic.
This is a quick overview. Security theory, architecture, and full walkthroughs are in the learn modules.
What It Does
- Scans files (PDF, DOCX, XLSX, CSV, JSON, XML, YAML, Parquet, Avro, archives, emails) for PII, credentials, financial data, and PHI
- Scans databases (PostgreSQL, MySQL, MongoDB, SQLite) with schema introspection and sampling
- Scans network captures (PCAP/PCAPNG) with protocol parsing, TCP reassembly, and DNS exfiltration detection
- Confidence scoring pipeline: regex match, checksum validation (Luhn, Mod-97, Mod-11), context keyword proximity, entity co-occurrence
- Maps findings to compliance frameworks (HIPAA, PCI-DSS, GDPR, CCPA, SOX, GLBA, FERPA)
- Reports in console (Rich tables), JSON, SARIF 2.1.0, or CSV
Quick Start
bash install.sh
dlp-scan file ./data
[!TIP] This project uses
justas a command runner. Typejustto see all available commands.Install:
curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin
Usage
dlp-scan file ./data/employees/ # scan a directory
dlp-scan file ./report.pdf -f json # scan a file, JSON output
dlp-scan db postgres://user:pass@host/db # scan PostgreSQL
dlp-scan db sqlite:///path/to/local.db # scan SQLite
dlp-scan network capture.pcap # scan network traffic
dlp-scan file ./data -f sarif -o results.sarif # SARIF for CI/CD
dlp-scan report convert results.json -f csv # convert report format
dlp-scan report summary results.json # print summary stats
Global Options
--config, -c Path to YAML config file
--verbose, -v Enable debug logging
--version Show version
Output Formats
| Format | Flag | Use Case |
|---|---|---|
| Console | -f console |
Interactive review with Rich tables |
| JSON | -f json |
Structured analysis and archival |
| SARIF | -f sarif |
GitHub code scanning, CI/CD integration |
| CSV | -f csv |
Compliance team export, spreadsheet import |
Stack
Language: Python 3.12+
CLI: Typer 0.15+ with Rich integration
Detection: Regex + checksum validators + Shannon entropy + context keyword scoring
File Formats: PyMuPDF, python-docx, openpyxl, xlrd, defusedxml, lxml, pyarrow, fastavro, extract-msg
Databases: asyncpg (PostgreSQL), aiomysql (MySQL), pymongo async (MongoDB), aiosqlite (SQLite)
Network: dpkt (PCAP parsing), TCP reassembly, DPI protocol identification, DNS exfiltration heuristics
Config: Pydantic 2.10+ models with YAML config loading (ruamel.yaml)
Quality: ruff, mypy (strict), yapf, pytest + hypothesis, structlog
Configuration
Copy .dlp-scanner.yml to your project root and customize. Key settings:
detection:
min_confidence: 0.20 # minimum score to report
enable_rules: ["*"] # glob patterns for rule IDs
allowlists:
values: ["123-45-6789"] # suppress known test values
output:
format: "console" # console, json, sarif, csv
redaction_style: "partial" # partial, full, none
Learn
This project includes step-by-step learning materials covering security theory, architecture, and implementation.
| Module | Topic |
|---|---|
| 00 - Overview | Prerequisites and quick start |
| 01 - Concepts | DLP theory and real-world breaches |
| 02 - Architecture | System design and data flow |
| 03 - Implementation | Code walkthrough |
| 04 - Challenges | Extension ideas and exercises |