reconFTW is a tool designed to perform automated recon on a target domain by running the best set of tools to perform scanning and finding out vulnerabilities
Shell bug-bountybugbountybugbounty-tooldnshackingnucleiosintpenetration-testingpentestpentest-toolpentestingreconreconnaissancescannersecuritysecurity-toolssubdomainsubdomain-enumerationvulnerabilities
docs: create roadmap (5 phases)
six2dez7d06758f7bcd
diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.mdindex 08c074f..eb0063b 100644--- a/.planning/REQUIREMENTS.md+++ b/.planning/REQUIREMENTS.md@@ -80,28 +80,28 @@ Which phases cover which requirements. Updated during roadmap creation.| Requirement | Phase | Status ||-------------|-------|--------|-| RESIL-01 | TBD | Pending |-| RESIL-02 | TBD | Pending |-| RESIL-03 | TBD | Pending |-| SEC-01 | TBD | Pending |-| SEC-02 | TBD | Pending |-| SEC-03 | TBD | Pending |-| SEC-04 | TBD | Pending |-| PERF-01 | TBD | Pending |-| PERF-02 | TBD | Pending |-| TEST-01 | TBD | Pending |-| TEST-02 | TBD | Pending |-| TEST-03 | TBD | Pending |-| DOCS-01 | TBD | Pending |-| DOCS-02 | TBD | Pending |-| FIX-01 | TBD | Pending |-| FIX-02 | TBD | Pending |+| RESIL-01 | Phase 1 | Pending |+| RESIL-02 | Phase 1 | Pending |+| RESIL-03 | Phase 1 | Pending |+| SEC-01 | Phase 2 | Pending |+| SEC-02 | Phase 2 | Pending |+| SEC-03 | Phase 2 | Pending |+| SEC-04 | Phase 2 | Pending |+| PERF-01 | Phase 3 | Pending |+| PERF-02 | Phase 1 | Pending |+| TEST-01 | Phase 4 | Pending |+| TEST-02 | Phase 4 | Pending |+| TEST-03 | Phase 4 | Pending |+| DOCS-01 | Phase 5 | Pending |+| DOCS-02 | Phase 5 | Pending |+| FIX-01 | Phase 2 | Pending |+| FIX-02 | Phase 3 | Pending |**Coverage:**- v1 requirements: 16 total-- Mapped to phases: 0 (pending roadmap)-- Unmapped: 16 ⚠️+- Mapped to phases: 16 ✓+- Unmapped: 0---*Requirements defined: 2026-05-13*-*Last updated: 2026-05-13 after initial definition*+*Last updated: 2026-05-13 after roadmap creation (traceability populated)*diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.mdnew file mode 100644index 0000000..4782f0f--- /dev/null+++ b/.planning/ROADMAP.md@@ -0,0 +1,135 @@+# Roadmap: reconFTW (Audit Hardening Milestone)++## Overview++This milestone closes the 2026-03 audit gaps captured in `.planning/codebase/CONCERNS.md`. Five phases drive 16 v1 requirements across resilience, security, performance, test coverage, and documentation. The sequencing places resilience and security fixes first (foundation), then performance caps and scope coherence (behavioural correctness), then test coverage that asserts the new behaviour, then documentation alignment. Each phase groups work by concern and shared code-touch surface so a single PR/plan modifies one logical area, minimising merge churn.++## Phases++**Phase Numbering:**+- Integer phases (1, 2, 3): Planned milestone work+- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)++Decimal phases appear between their surrounding integers in numeric order.++- [ ] **Phase 1: Resilient Resume & Timeout Safety** — `.inprogress` sentinel, disk-full mid-run detection, parallel job timeouts, DNS timeout defaults+- [ ] **Phase 2: Security Quoting & Supply-Chain Hygiene** — Eval removal, curl quoting, `AXIOM_EXTRA_ARGS` array refactor, installer SHA256, mantra path fix+- [ ] **Phase 3: Concurrency Caps & Scope Unification** — Per-tool thread caps, `is_in_scope_host` vs `domain_match_regex` reconciliation with cross-check tests+- [ ] **Phase 4: Test Coverage Reinforcement** — `parallel_funcs` batch behaviour, mocked end-to-end pipeline, axiom failover+- [ ] **Phase 5: Configuration & Documentation Alignment** — Surface hidden tunables in `reconftw.cfg`, resolve disk-space default mismatch++## Phase Details++### Phase 1: Resilient Resume & Timeout Safety+**Goal**: Long-running scans survive interruptions, disk pressure, and stuck tools without silently producing truncated output or wasting hours of re-work on resume.+**Depends on**: Nothing (first phase)+**Requirements**: RESIL-01, RESIL-02, RESIL-03, PERF-02+**Success Criteria** (what must be TRUE):+ 1. Interrupting a recon mid-`sub_brute` and re-running with `PRESERVE=true` produces a clear `.inprogress_sub_brute` indicator and re-executes only that function — completed functions retain `.<func>` checkpoints and are skipped+ 2. A scan that exhausts disk space mid-run aborts with a clear `ENOSPC` error message rather than producing zero-byte or truncated output files+ 3. Setting `PARALLEL_JOB_TIMEOUT_SECONDS=600` in `reconftw.cfg` terminates any single `parallel_funcs` child exceeding 10 minutes via `kill -TERM`, allowing the batch to continue+ 4. Fresh installs of `reconftw.cfg` ship with non-zero `DNS_BRUTE_TIMEOUT=6h` and `DNS_RESOLVE_TIMEOUT=4h` defaults, and a stuck DNS run aborts with a logged timeout rather than blocking indefinitely+**Plans**: TBD++Plans:+- [ ] 01-01: `.inprogress` sentinel lifecycle in `start_func`/`end_func` with resume detection (RESIL-01)+- [ ] 01-02: Disk-full mid-run guard — periodic `df` check + `ENOSPC` trap (RESIL-02)+- [ ] 01-03: `PARALLEL_JOB_TIMEOUT_SECONDS` enforcement in `lib/parallel.sh` plus DNS timeout defaults in `reconftw.cfg` (RESIL-03, PERF-02)++### Phase 2: Security Quoting & Supply-Chain Hygiene+**Goal**: Eliminate the remaining `eval` injection vector, lock down unquoted notification curl calls, make `AXIOM_EXTRA_ARGS` tokenisation explicit, and integrity-check installer bootstrappers.+**Depends on**: Phase 1+**Requirements**: SEC-01, SEC-02, SEC-03, SEC-04, FIX-01+**Success Criteria** (what must be TRUE):+ 1. `grep -n "eval " lib/common.sh modules/*.sh` returns no `eval` calls under `safe_count()` and all callers pass file paths — the `else` branch at `lib/common.sh:760` is gone+ 2. `sendToNotify()` in `modules/core.sh` quotes every variable passed to `curl` (`"$discord_url"`, `"${slack_channel}"`, `"${1}"`); shellcheck reports no new findings against `modules/core.sh:1399-1414`+ 3. `AXIOM_EXTRA_ARGS` is consumed as `AXIOM_EXTRA_ARGS_ARR=()` across `modules/subdomains.sh` and `modules/web.sh`; a value containing intentional whitespace tokens (e.g., `--rate-limit 100`) tokenises correctly and an accidentally-quoted single-token value does not split+ 4. `install.sh` calls `verify_sha256()` against pinned hashes for `rustup-init.sh` and `uv/install.sh`; aborts with a clear error on mismatch; a `tools.lock` (or equivalent manifest) pins versions for `nuclei`, `httpx`, `ffuf`, `puredns`, `subfinder`+ 5. Running `axiom-exec mantra` against a remote node installs `github.com/brosck/mantra@latest` (lowercase) and the binary runs — the previous 404 / module-not-found on case-sensitive GitHub paths is gone+**Plans**: TBD++Plans:+- [ ] 02-01: `safe_count()` eval removal + caller migration to file-path form (SEC-01)+- [ ] 02-02: `sendToNotify()` quoting refactor + `modules/web.sh:2340` mantra path fix (SEC-02, FIX-01)+- [ ] 02-03: `AXIOM_EXTRA_ARGS_ARR` array refactor + installer SHA256 verification + tools manifest (SEC-03, SEC-04)++### Phase 3: Concurrency Caps & Scope Unification+**Goal**: Bound multi-core thread scaling so large hosts don't overwhelm targets, and reconcile the two scope-checking implementations so output is consistent across code paths.+**Depends on**: Phase 2+**Requirements**: PERF-01, FIX-02+**Success Criteria** (what must be TRUE):+ 1. On a 64-core machine the computed values of `FFUF_THREADS`, `HTTPX_THREADS`, `HTTPX_UNCOMMONPORTS_THREADS`, `KATANA_THREADS`, `DALFOX_THREADS`, `DNSTAKE_THREADS` never exceed their respective `*_THREADS_MAX` caps from `reconftw.cfg`+ 2. The thread-cap variables (`FFUF_THREADS_MAX`, etc.) appear as commented-out reference entries in `reconftw.cfg` so users can tune them without reading source code+ 3. `is_in_scope_host()` and `domain_match_regex()` agree on edge cases — trailing dots, wildcards, empty input, IDN — verified by a cross-check test in `tests/unit/` that feeds the same corpus through both functions and asserts identical accept/reject decisions+**Plans**: TBD++Plans:+- [ ] 03-01: Per-tool thread-count caps with `*_THREADS_MAX` config knobs and `min()` enforcement (PERF-01)+- [ ] 03-02: Scope-check unification — pick canonical implementation + cross-check bats test (FIX-02)++### Phase 4: Test Coverage Reinforcement+**Goal**: Lock in the resilience and parallel changes from Phase 1 with regression tests, and add coverage for the two highest-priority untested areas (end-to-end module handoff and axiom failover).+**Depends on**: Phase 3+**Requirements**: TEST-01, TEST-02, TEST-03+**Success Criteria** (what must be TRUE):+ 1. `bats tests/unit/test_parallel.bats` covers barrier sync, heartbeat polling cadence, the failure-counter increment path, and the new `PARALLEL_JOB_TIMEOUT_SECONDS` kill path — at least 4 new `@test` blocks, all green in `make unit-fast`+ 2. A new bats integration test (e.g., `tests/integration/test_pipeline_handoff.bats`) exercises `subdomains.sh` → `web.sh` → `vulns.sh` against mocked tool outputs and asserts the `webs.txt` shape and scope-filter output remain stable+ 3. A new bats test covers the axiom failover path: `axiom_disable_runtime` invocation on launch failure, partial-fleet fallback to local mode, and `AXIOM_AUTO_FIX_HOSTKEY` repair logic — running locally without an axiom fleet+ 4. `make test` (or the equivalent CI target) reports the new test files in its summary and CI passes on the `dev` branch with the additions+**Plans**: TBD++Plans:+- [ ] 04-01: `parallel_funcs` unit tests — barrier, heartbeat, failure counter, timeout kill path (TEST-01)+- [ ] 04-02: Mocked end-to-end pipeline integration test + axiom failover test (TEST-02, TEST-03)++### Phase 5: Configuration & Documentation Alignment+**Goal**: Surface every previously-undocumented config knob in `reconftw.cfg` and resolve the disk-space default mismatch so the config file is authoritative.+**Depends on**: Phase 4+**Requirements**: DOCS-01, DOCS-02+**Success Criteria** (what must be TRUE):+ 1. `reconftw.cfg` contains commented-out reference entries for `PARALLEL_MAX_JOBS`, `ALLOW_TRANSFER`, `RAISE_ULIMIT`, `ULIMIT_TARGET`, and `AXIOM_EXTRA_ARGS` semantics; a user grepping `reconftw.cfg` for any of these names finds documentation rather than empty results+ 2. `MIN_DISK_SPACE_GB` has a single source of truth — either `reconftw.cfg:39` lifts to `5` to match `modes.sh`, or `modes.sh:23` drops the `:-5` fallback; the value reported by `check_disk_space()` matches the value documented in `reconftw.cfg`+ 3. Running `shellcheck reconftw.cfg` and `bats tests/unit/test_config.bats` (or equivalent) passes with no new findings tied to the documentation additions+**Plans**: TBD++Plans:+- [ ] 05-01: Surface hidden tunables + resolve disk-space mismatch in `reconftw.cfg` and `modes.sh` (DOCS-01, DOCS-02)++## Progress++**Execution Order:**+Phases execute in numeric order: 1 → 2 → 3 → 4 → 5++| Phase | Plans Complete | Status | Completed |+|-------|----------------|--------|-----------|+| 1. Resilient Resume & Timeout Safety | 0/3 | Not started | - |+| 2. Security Quoting & Supply-Chain Hygiene | 0/3 | Not started | - |+| 3. Concurrency Caps & Scope Unification | 0/2 | Not started | - |+| 4. Test Coverage Reinforcement | 0/2 | Not started | - |+| 5. Configuration & Documentation Alignment | 0/1 | Not started | - |++## Coverage++**v1 requirements:** 16 total, 16 mapped (100%)++| Requirement | Phase |+|-------------|-------|+| RESIL-01 | Phase 1 |+| RESIL-02 | Phase 1 |+| RESIL-03 | Phase 1 |+| PERF-02 | Phase 1 |+| SEC-01 | Phase 2 |+| SEC-02 | Phase 2 |+| SEC-03 | Phase 2 |+| SEC-04 | Phase 2 |+| FIX-01 | Phase 2 |+| PERF-01 | Phase 3 |+| FIX-02 | Phase 3 |+| TEST-01 | Phase 4 |+| TEST-02 | Phase 4 |+| TEST-03 | Phase 4 |+| DOCS-01 | Phase 5 |+| DOCS-02 | Phase 5 |++---+*Roadmap created: 2026-05-13*diff --git a/.planning/STATE.md b/.planning/STATE.mdnew file mode 100644index 0000000..c5e667b--- /dev/null+++ b/.planning/STATE.md@@ -0,0 +1,79 @@+# Project State++## Project Reference++See: .planning/PROJECT.md (updated 2026-05-13)++**Core value:** Run one command, get a complete recon picture of a target — passive, active, and vulnerability layers — with resumable checkpoints, structured output, and zero-touch tool orchestration.+**Current focus:** Phase 1 — Resilient Resume & Timeout Safety++## Current Position++Phase: 1 of 5 (Resilient Resume & Timeout Safety)+Plan: 0 of 3 in current phase+Status: Ready to plan+Last activity: 2026-05-13 — Roadmap created from CONCERNS.md audit findings++Progress: [░░░░░░░░░░] 0%++## Performance Metrics++**Velocity:**+- Total plans completed: 0+- Average duration: —+- Total execution time: 0.0 hours++**By Phase:**++| Phase | Plans | Total | Avg/Plan |+|-------|-------|-------|----------|+| 1. Resilient Resume & Timeout Safety | 0/3 | — | — |+| 2. Security Quoting & Supply-Chain Hygiene | 0/3 | — | — |+| 3. Concurrency Caps & Scope Unification | 0/2 | — | — |+| 4. Test Coverage Reinforcement | 0/2 | — | — |+| 5. Configuration & Documentation Alignment | 0/1 | — | — |++**Recent Trend:**+- Last 5 plans: —+- Trend: —++*Updated after each plan completion*++## Accumulated Context++### Decisions++Decisions are logged in PROJECT.md Key Decisions table.+Recent decisions affecting current work:++- Init: Audit-mode milestone first (not features) — 2026-03 audit surfaced concrete reliability/security gaps that block confident feature work+- Init: CONCERNS.md drives v1 Requirements — audit inventory already classifies severity; using it as the backlog avoids re-deriving the same list+- Roadmap: Tests sequenced AFTER resilience changes (Phase 4 follows Phase 1) so tests assert the new `.inprogress`/timeout behaviour, not the old+- Roadmap: Docs in last phase (Phase 5) so disk-space alignment reflects the settled `MIN_DISK_SPACE_GB` decision rather than pre-deciding it++### Pending Todos++None yet.++### Blockers/Concerns++None yet.++## Deferred Items++Items acknowledged and carried forward from previous milestone close:++| Category | Item | Status | Deferred At |+|----------|------|--------|-------------|+| Architecture | ARCH-01: split `modules/web.sh` (2965 lines) | v2 backlog | 2026-05-13 |+| Architecture | ARCH-02: file-based secret handling (avoid CLI exposure in `ps aux`) | v2 backlog | 2026-05-13 |+| Scaling | SCALE-01: memory-aware permutation throttling | v2 backlog | 2026-05-13 |+| Scaling | SCALE-02: resolver-file health gate for puredns | v2 backlog | 2026-05-13 |+| Observability | OBS-01: surface venv health in startup summary | v2 backlog | 2026-05-13 |+| Observability | OBS-02: structured JSONL events at every module boundary by default | v2 backlog | 2026-05-13 |++## Session Continuity++Last session: 2026-05-13+Stopped at: Roadmap and STATE initialised; 16/16 requirements mapped to 5 phases+Resume file: Nonediff --git a/CLAUDE.md b/CLAUDE.mdnew file mode 100644index 0000000..865143f--- /dev/null+++ b/CLAUDE.md@@ -0,0 +1,411 @@+<!-- GSD:project-start source:PROJECT.md -->+## Project++**reconFTW**++reconFTW is a comprehensive bash-based reconnaissance automation framework used by bug bounty hunters, penetration testers, and security researchers. It orchestrates 70+ external security tools (Go, Python, Rust) across subdomain enumeration, web probing, OSINT, and vulnerability scanning, producing structured per-target output trees with optional Axiom distributed execution, AI reporting, monitor/incremental mode, and Slack/Telegram/Discord notifications.++**Core Value:** Run one command, get a complete recon picture of a target — passive, active, and vulnerability layers — with resumable checkpoints, structured output, and zero-touch tool orchestration.++### Constraints++- **Tech stack**: Bash 4.3+ — Required for `wait -n`, `mapfile`, associative arrays. macOS users must have Homebrew bash; auto re-exec is best-effort.+- **External tools**: 70+ runtime dependencies — Most install via `go install @latest` (no version pinning), which is convenient but a known supply-chain risk.+- **Single process**: All modules sourced into one shell — No subshell isolation between modules; all state shared via globals. Workflow functions must save/restore globals they override (see `passive()` pattern).+- **Resume semantics**: Checkpoint files are touch-once at `end_func` — Interrupted functions re-run from scratch on next invocation; partial outputs are not detected.+- **Single-operator**: Designed for one user per target run — No locking, no multi-user state, no concurrent runs against the same target dir.+- **Output stability**: `Recon/<domain>/` tree is a public contract — Subdirectory names and filenames are consumed by downstream pipelines, scripts, and parsers; renames are breaking changes.+- **macOS compatibility**: GNU coreutils + GNU sed + GNU getopt required — System BSD versions are not supported.+- **CI budget**: Integration-full is weekly cron — Unit + smoke are per-push; adding heavy integration tests must respect this split.+<!-- GSD:project-end -->++<!-- GSD:stack-start source:codebase/STACK.md -->+## Technology Stack++## Languages+- Bash 4+ - All core framework logic (`reconftw.sh`, `modules/*.sh`, `lib/*.sh`, `install.sh`)+- Python 3.7+ - Python-backed tools (each runs in isolated `uv venv`): `dorks_hunter`, `CMSeeK`, `EmailHarvester`, `Spoofy`, `SSTImap`, `gato`, `regulator`, `reconftw_ai`, `getjswords.py`+- Go (latest, min ~1.21) - Primary binary language for ~55 security tools installed via `go install @latest`+## Runtime+- Linux (Debian/Ubuntu/RHEL/Arch) or macOS (Apple Silicon / Intel)+- Docker: Ubuntu 24.04 base image (`Docker/Dockerfile`)+- ARM64/aarch64, ARMv6l/v7l, and x86_64 all supported for Go binary installs+- `getopt` (GNU getopt required on macOS via `brew install gnu-getopt`)+- `nproc` / `sysctl -n hw.ncpu` for CPU core auto-detection+- `timeout` / `gtimeout` (macOS Homebrew `coreutils`)+- `gnu-sed` (macOS requires `brew install gnu-sed`)+- `gnu-coreutils` (macOS requires `brew install coreutils`)+- Go tools: `go install` (`GOPATH=$HOME/go`, `GOROOT=/usr/local/go`)+- Python tools: `uv tool install` from GitHub (most) or PyPI (`fray`)+- Python repo venvs: `uv venv venv && uv pip install -r requirements.txt`+- Lockfile: None (always installs `@latest`)+## Frameworks+- No framework — pure Bash with sourced module files+- Module loading order: `lib/validation.sh` → `lib/common.sh` → `lib/ui.sh` → `lib/parallel.sh` → `modules/utils.sh` → `modules/core.sh` → `modules/osint.sh` → `modules/subdomains.sh` → `modules/web.sh` → `modules/vulns.sh` → `modules/axiom.sh` → `modules/modes.sh`+- bats-core (Bash Automated Testing System)+- GNU make (`Makefile`) — test, lint, format targets+- shellcheck (error-level) — `make lint`, pre-commit hook+- shfmt (4-space indent, `-bn`, `-ci`) — `make fmt`, pre-commit hook+- pre-commit hooks defined in `.pre-commit-config.yaml`+- semgrep: `.github/workflows/semgrep.yml` (CI only)+## Key Dependencies+### Go Tools (installed via `go install @latest`)+- `subfinder` (projectdiscovery/subfinder) — passive multi-source subdomain discovery+- `github-subdomains` (gwen001/github-subdomains) — GitHub-based subdomain search+- `gitlab-subdomains` (gwen001/gitlab-subdomains) — GitLab-based subdomain search+- `dnstake` (pwnesia/dnstake) — subdomain takeover detection+- `puredns` (d3mondev/puredns) — mass DNS resolution with wildcard filtering+- `dnsx` (projectdiscovery/dnsx) — DNS toolkit+- `massdns` (blechschmidt/massdns) — via repo clone + build+- `dsieve` (trickest/dsieve) — subdomain filtering+- `enumerepo` (trickest/enumerepo) — GitHub org repo enumeration+- `gotator` (Josue87/gotator) — subdomain permutations+- `analyticsrelationships` (Josue87/analyticsrelationships) — Google Analytics pivoting+- `roboxtractor` (Josue87/roboxtractor) — robots.txt extractor+- `crt` (cemulus/crt) — crt.sh search+- `asnmap` (projectdiscovery/asnmap) — ASN-to-CIDR mapping+- `mapcidr` (projectdiscovery/mapcidr) — CIDR manipulation+- `smap` (s0md3v/smap) — passive Shodan-powered port scan+- `tlsx` (projectdiscovery/tlsx) — TLS certificate harvesting+- `hakip2host` (hakluke/hakip2host) — reverse IP lookup+- `cdncheck` (projectdiscovery/cdncheck) — CDN/WAF IP classification+- `hakoriginfinder` (hakluke/hakoriginfinder) — origin IP discovery behind CDN+- `inscope` (tomnomnom/hacks/inscope) — scope filtering+- `csprecon` (edoardottt/csprecon) — CSP-based subdomain discovery+- `favirecon` (edoardottt/favirecon) — favicon-based tech recon+- `httpx` (projectdiscovery/httpx) — multi-probe HTTP toolkit+- `katana` (projectdiscovery/katana) — web crawler+- `ffuf` (ffuf/ffuf) — web fuzzer+- `subjs` (lc/subjs) — JavaScript URL extractor+- `Gxss` (KathanP19/Gxss) — reflected XSS param finder+- `jsluice` (BishopFox/jsluice) — JS secret/URL extractor+- `sourcemapper` (denandz/sourcemapper) — JS source map extractor+- `mantra` (brosck/mantra) — JS/secret scanner+- `urlfinder` (projectdiscovery/urlfinder) — URL discovery+- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — via uv+- `nmapurls` (sdcampbell/nmapurls) — URL extraction from Nmap XML+- `naabu` (projectdiscovery/naabu) — fast port scanner+- `VhostFinder` (wdahlenburg/VhostFinder) — virtual host discovery+- `shortscan` (bitquark/shortscan) — IIS short filename scanner+- `nuclei` (projectdiscovery/nuclei) — template-based scanner+- `dalfox` (hahwul/dalfox) — XSS scanner+- `crlfuzz` (dwisiswant0/crlfuzz) — CRLF injection scanner+- `Web-Cache-Vulnerability-Scanner` (Hackmanit) — web cache poisoning+- `TInjA` (Hackmanit/TInjA) — SSTI scanner+- `toxicache` (xhzeem/toxicache) — web cache poisoning+- `second-order` (mhmdiaa/second-order) — broken link/second-order injection+- `s3scanner` (sa7mon/s3scanner) — S3/GCS/Azure Blob misconfiguration+- `misconfig-mapper` (intigriti/misconfig-mapper) — third-party misconfiguration+- `sj` (BishopFox/sj) — Swagger/OpenAPI analysis+- `grpcurl` (fullstorydev/grpcurl) — gRPC reflection scanner+- `nerva` (praetorian-inc/nerva) — service fingerprinting+- `brutus` (praetorian-inc/brutus) — credential spraying+- `julius` (praetorian-inc/julius) — LLM endpoint probe+- `titus` (praetorian-inc/titus) — secrets engine+- `notify` (projectdiscovery/notify) — multi-channel notifications+- `interactsh-client` (projectdiscovery/interactsh) — OOB callback server+- `gf` (tomnomnom/gf) — URL pattern grep+- `anew` (tomnomnom/anew) — append new lines only+- `unfurl` (tomnomnom/unfurl) — URL parser+- `qsreplace` (tomnomnom/qsreplace) — querystring replacer+- `gitdorks_go` (damit5/gitdorks_go) — GitHub dork search+- `github-endpoints` (gwen001/github-endpoints) — GitHub endpoint discovery+- `cent` (xm1k3/cent) — nuclei template manager+- `trufflehog` (trufflesecurity/trufflehog) — secrets scanner (via `go install`)+- `brutespray` (x90skysn3k/brutespray) — service credential spraying+### Python Tools (installed via `uv tool install`)+- `dnsvalidator` (vortexau/dnsvalidator) — DNS resolver validation+- `interlace` (pry0cc/interlace) — parallel command runner+- `wafw00f` (EnableSecurity/wafw00f) — WAF fingerprinting+- `commix` (commixproject/commix) — command injection scanner+- `waymore` (xnl-h4ck3r/waymore) — passive URL collection+- `urless` (xnl-h4ck3r/urless) — URL deduplication+- `ghauri` (r0oth3x49/ghauri) — SQLi scanner (optional)+- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — deep link finder+- `xnldorker` (xnl-h4ck3r/xnldorker) — Google dorker+- `porch-pirate` (MandConsultingGroup/porch-pirate) — Postman API leaks+- `p1radup` (iambouali/p1radup) — URL deduplication+- `subwiz` (hadriansecurity/subwiz) — ML-based subdomain prediction+- `arjun` (s0md3v/Arjun) — parameter discovery+- `gqlspection` (doyensec/GQLSpection) — GraphQL deep introspection+- `postleaksNg` (six2dez/postleaksNG) — Postman public leak search+- `cewler` (roys/cewler) — web wordlist generator+- `fray` (dalisecurity/fray) — WAF-aware payload testing (PyPI)+### Repo-Clone Tools (Python venvs, run via `venv/bin/python3`)+- `dorks_hunter` (six2dez/dorks_hunter) — Google dork automation+- `CMSeeK` (Tuhinshubhra/CMSeeK) — CMS fingerprinting+- `cloud_enum` (initstring/cloud_enum) — AWS/GCP/Azure bucket enumeration+- `EmailHarvester` (maldevel/EmailHarvester) — email harvesting+- `SwaggerSpy` (UndeadSec/SwaggerSpy) — Swagger endpoint leak detection+- `LeakSearch` (JoelGMSec/LeakSearch) — credential leak search+- `Spoofy` (MattKeeley/Spoofy) — email spoofing check+- `msftrecon` (Arcanum-Sec/msftrecon) — Microsoft tenant recon+- `Scopify` (Arcanum-Sec/Scopify) — scope management+- `regulator` (cramppet/regulator) — regex-based subdomain permutations+- `SSTImap` (vladko312/SSTImap) — SSTI scanner (alternative engine)+- `gato` (praetorian-inc/gato) — GitHub Actions audit+### Repo-Clone Tools (Go build)+- `ghleaks` (dinosn/ghleaks) — GitHub-wide secret search+- `nomore403` (devploit/nomore403) — 403 bypass+- `ffufPostprocessing` (Damian89/ffufPostprocessing) — ffuf result analysis+- `JSA` (w9w/JSA) — JS analysis+- `ultimate-nmap-parser` (shifty0g/ultimate-nmap-parser) — Nmap XML parser+### System-Level Tools (apt/brew/yum)+- `nmap` — active port scanning+- `massdns` — DNS resolver (also cloned + built from source)+- `jq` — JSON processing throughout all modules+- `exiftool` (perl-Image-ExifTool) — metadata extraction+- `whois` — domain registration lookup+- `sqlmap` — SQL injection (system or via repo clone)+- `testssl.sh` (testssl/testssl.sh) — TLS/SSL misconfiguration testing+- `medusa` — credential brute-force (system install)+- `shodan` CLI — installed via `uv tool install shodan`+### Rust Tools+- `smugglex` (Cargo) — HTTP request smuggling detection+- Rustup installed from `https://sh.rustup.rs`+## Configuration+- `reconftw.cfg` — sourced after CLI parsing; all feature flags, rate limits, timeouts, wordlist paths, API keys, thread counts+- `secrets.cfg` (gitignored, auto-sourced) — API keys and tokens separated from main config+- `secrets.cfg.example` — template showing all supported secret vars+- Feature flags: `OSINT=true`, `SUBDOMAINS_GENERAL=true`, `VULNS_GENERAL=false`, etc.+- Rate limits: `HTTPX_RATELIMIT=150`, `NUCLEI_RATELIMIT=150`, `FFUF_RATELIMIT=0`+- Thread counts: auto-scaled via `AVAILABLE_CORES=$(nproc)` with multipliers per tool+- Timeouts: per-tool in seconds or minutes (`CMSSCAN_TIMEOUT=3600`, `SUBFINDER_ENUM_TIMEOUT=180`)+- Wordlist paths: `fuzz_wordlist`, `lfi_wordlist`, `subs_wordlist` etc. under `${WORDLISTS_DIR}`+- Output: `EXPORT_FORMAT`, `AI_REPORT_TYPE`, `ASSET_STORE`+- GNU `getopt` long options, parsed in `reconftw.sh` while/case loop+- All CLI overrides use `CLI_*` pattern and are re-applied after `reconftw.cfg` is sourced+- Full list from `getopt` call: `domain`, `list`, `recon`, `subdomains`, `passive`, `all`, `web`, `osint`, `zen`, `deep`, `help`, `vps`, `vps-count`, `ai`, `check-tools`, `health-check`, `quick-rescan`, `incremental`, `adaptive-rate`, `dry-run`, `parallel`, `no-parallel`, `monitor`, `monitor-interval`, `monitor-cycles`, `refresh-cache`, `gen-resolvers`, `force`, `export`, `report-only`, `no-report`, `parallel-log`, `quiet`, `verbose`, `no-color`, `log-format`, `show-cache`, `banner`, `no-banner`, `legal`+- `SHODAN_API_KEY`, `WHOISXML_API`, `PDCP_API_KEY`, `XSS_SERVER`, `COLLAB_SERVER` — preferred over config file+- `GOROOT`, `GOPATH`, `PATH` — extended by `reconftw.cfg` for Go and Rust binaries+- `LOGFILE` — per-target log path+- `config/reconftw_full.cfg` — full-scan preset+- `config/reconftw_quick.cfg` — quick-scan preset+- `config/reconftw_stealth.cfg` — low-noise preset+## Build+- Default: `go1.23.6` (fetches latest from `https://go.dev/VERSION?m=text`)+- Installed to `/usr/local/go`; set `install_golang=false` in config to skip+- Minimum: Python 3.7 (enforced in `install_yum()`)+- Virtual environments per tool via `uv venv`+- Root venv at `.venv/` for `getjswords.py` and similar helpers+## Platform Requirements+- Bash ≥ 4.3 (for `wait -n` used in `lib/parallel.sh`)+- Go ≥ 1.21 (tools use SIV module paths like `/v2`, `/v3`)+- Python ≥ 3.7+- `uv` package manager+- Rust / Cargo (for `smugglex`)+- GNU coreutils, getopt, sed (macOS only via Homebrew)+- ~5GB free disk space for Go cache, tools, and repos+- ~1GB RAM minimum (Go compilation)+- Base image: `ubuntu:24.04`+- Build arg `INSTALL_AXIOM=true` (default) installs axiom fleet tooling+- Ports 85-90 exposed (for headless browser tooling)+- Runs as root (required for raw socket operations by some tools)+- Health check: `./reconftw.sh --health-check`+<!-- GSD:stack-end -->++<!-- GSD:conventions-start source:CONVENTIONS.md -->+## Conventions++## Shell Settings+## Source Guard Pattern+- `lib/common.sh`: `[[ -n "$_COMMON_SH_LOADED" ]] && return 0`+- `lib/parallel.sh`: `[[ -n "$_PARALLEL_SH_LOADED" ]] && return 0`+- `lib/ui.sh`: `[[ -n "${_UI_SH_LOADED:-}" ]] && return 0`+## Function Naming+- Public module functions: `snake_case` prefixed with module context (`sub_passive`, `sub_crt`, `geo_info`)+- Private helpers: `_snake_case` prefix (`_print_status`, `_print_error`, `_print_module_start`, `_parallel_emit_job_output`)+- UI layer: `ui_` prefix (`ui_init`, `ui_header`, `ui_summary`, `ui_batch_end`)+- Lifecycle wrappers: `start_func` / `end_func` (call these at top/bottom of every recon function)+- Validation functions: `validate_*` / `sanitize_*` (defined in `lib/validation.sh` and `modules/utils.sh`)+## Variable Naming+- Config flags: `SUBPASSIVE`, `SUBCRT`, `PARALLEL_MODE`, `OUTPUT_VERBOSITY`+- Runtime state: `LOGFILE`, `SCRIPTPATH`, `DIFF`, `DRY_RUN`, `AXIOM`+- Error codes: `E_SUCCESS=0`, `E_INVALID_DOMAIN=20`, `E_INVALID_IP=21` (readonly, defined in `lib/validation.sh`)+## CLI Flag Pattern+## Output / UI Conventions+- `OUTPUT_VERBOSITY=0` (quiet): only errors/FAIL printed+- `OUTPUT_VERBOSITY=1` (normal, default): OK/WARN/FAIL/SKIP status lines+- `OUTPUT_VERBOSITY=2` (verbose): all of the above + INFO messages + start_func messages+## Function Lifecycle (start_func / end_func)+- `start_func name desc` — logs to LOGFILE, sets per-function start timestamp, emits INFO at verbosity >= 2+- `end_func message name [status]` — touches checkpoint file, calculates elapsed time, calls `_print_status`+- `skip_notification reason` — emits SKIP/CACHE badge; reasons: `"disabled"`, `"mode"`, `"processed"`, `"processed-visible"`, `"noinput"`+## File Checkpointing (Resumability)+- `end_func` creates the checkpoint: `touch "$called_fn_dir/.${fn}"`+- DIFF mode (`DIFF=true`) bypasses checkpoint — forces re-execution+- Helper `should_run()` in `lib/common.sh` provides a cleaner gate: `if should_run "FLAG_VAR"; then`+## Error Handling+- `E_SUCCESS=0`, `E_GENERAL=1`, `E_MISSING_DEP=2`, `E_INVALID_INPUT=3`+## Validation Functions+| Function | Purpose |+|----------|---------|+| `validate_domain()` | RFC domain check + injection character rejection |+| `validate_ipv4()` | Octet range validation |+| `validate_integer()` | Numeric range check |+| `validate_boolean()` | Accepts `true`/`false` only (not `1`/`0`/`yes`/`no`) |+| `validate_file_readable()` | Exists + readable + is-a-file |+| `sanitize_interlace_input()` | Removes shell metacharacters from input files (canonical in `lib/validation.sh`) |+| `sanitize_domain()` | Strips URL components, lowercases, rejects injection (in `modules/utils.sh`) |+| `is_in_scope_host()` | Anchored hostname scope check (prevents substring false positives) |+| `filter_in_scope_urls()` | Python3-based URL scope check (scheme, userinfo, host) |+## Path and CWD Conventions+## Parallel Execution+## Import / Sourcing Order+## Logging+- All tool output redirected to `$LOGFILE`: `command ... 2>>"$LOGFILE" >/dev/null`+- Structured JSON logging via `log_json level func message [key=val]` (optional, `STRUCTURED_LOGGING=true`)+- `redact_secrets()` scrubs `REDACT_VARS` and `REGISTERED_SECRETS` from log lines+- `register_secret "$value"` must be called before logging any secret value+## Comments+<!-- GSD:conventions-end -->++<!-- GSD:architecture-start source:ARCHITECTURE.md -->+## Architecture++## System Overview+```text+```+## Component Responsibilities+| Component | Responsibility | File |+|-----------|----------------|------|+| Entry point & CLI parser | getopt argument parsing, config sourcing, mode dispatch | `reconftw.sh` |+| Mode orchestration | `start`/`end`, workflow functions (`recon`, `passive`, `all`, `vulns`, `osint`, `subs_menu`, `webs_menu`, `zen_menu`, `monitor_mode`) | `modules/modes.sh` |+| Function lifecycle | `start_func`/`end_func`, checkpointing, logging, notifications, reporting, plugins, health check | `modules/core.sh` |+| Subdomain enumeration | All `sub_*` functions, `subtakeover`, `zonetransfer`, `s3buckets`, `geo_info` | `modules/subdomains.sh` |+| Web analysis | `webprobe_full`, `screenshot`, `nuclei_check`, `fuzz`, `jschecks`, `urlchecks`, `waf_checks`, and 20+ others | `modules/web.sh` |+| Vulnerability scanning | `xss`, `ssrf_checks`, `sqli`, `crlf_checks`, `lfi`, `ssti`, `smuggling`, `fuzzparams`, `nuclei_dast`, and others | `modules/vulns.sh` |+| OSINT collection | `domain_info`, `ip_info`, `emails`, `google_dorks`, `github_leaks`, `github_actions_audit`, `cloud_enum_scan`, etc. | `modules/osint.sh` |+| Shared utilities | `run_command`, `sed_i`, `deleteOutScoped`, `validate_config`, `cache_*`, `checkpoint_*`, `circuit_breaker_*`, rate-limit adaption | `modules/utils.sh` |+| Axiom/distributed mode | `axiom_launch`, `axiom_shutdown`, `axiom_selected`, `resolvers_update`, `ipcidr_target` | `modules/axiom.sh` |+| Parallel execution | `parallel_funcs`, `_throttle_jobs`, job heartbeat, progress live display, log mode output | `lib/parallel.sh` |+| Input validation/sanitization | `sanitize_domain`, `sanitize_ip`, `validate_domain`, `validate_integer`, `_sanitize_list_entry` | `lib/validation.sh` |+| Shared file/counter utilities | `ensure_dirs`, `ensure_webs_all`, `safe_backup`, `count_lines`, `safe_count`, incident tracking | `lib/common.sh` |+| UI presentation layer | `_print_status`, `_print_msg`, `_print_section`, `_print_rule`, `ui_header`, `ui_summary`, TTY detection, color management, JSONL output | `lib/ui.sh` |+| Configuration | All runtime settings (~350 variables); sourced after CLI parse | `reconftw.cfg` |+## Pattern Overview+- Single process: `reconftw.sh` sources all libraries and modules at startup; every function lives in the same shell environment+- No subshell isolation between modules — all state is shared via global variables+- File-based checkpointing: `called_fn_dir/.funcname` sentinel files prevent re-running completed functions across invocations+- CLI-over-config: `reconftw.cfg` provides defaults; CLI flags set `CLI_*` variables that are re-applied after config sourcing to guarantee they cannot be overwritten+- All external tool invocations go through `run_command()` which handles dry-run mode, adaptive rate limiting, axiom dispatch, and debug logging+## Layers+- Purpose: Bootstrap, macOS re-exec, module loading, getopt CLI parsing, config sourcing, CLI override re-application, mode dispatch+- Location: `reconftw.sh`+- Contains: `normalize_vps_count_args()`, the main `while/case` getopt loop, the config `source` sequence, CLI override if-blocks, the final `case $opt_mode` dispatch+- Depends on: All libraries (sourced first), all modules (sourced second)+- Used by: End user / CI+- Purpose: Reusable utilities with no side effects; loadable independently for tests+- Location: `lib/validation.sh`, `lib/common.sh`, `lib/ui.sh`, `lib/parallel.sh`+- Contains: Input sanitization, file helpers, UI/color/progress, parallel job management+- Depends on: Nothing (source-guarded with `_*_LOADED` pattern)+- Used by: All modules and reconftw.sh+- Purpose: Implement all scanning, analysis, and orchestration functions+- Location: `modules/`+- Contains: All recon, vuln, OSINT, web, subdomain functions+- Depends on: Libraries (always loaded first), `reconftw.cfg` variables, external tools on PATH+- Used by: modes.sh orchestrates all others; reconftw.sh dispatches to modes.sh+- Purpose: Default runtime values for ~350 flags/paths/limits; can be overridden by `secrets.cfg` and custom config+- Location: `reconftw.cfg`, optionally `secrets.cfg`, optionally `$CUSTOM_CONFIG`+- Contains: Module enable/disable flags, tool flags, API key env-var references, paths, parallelism settings, verbosity, Axiom settings+- Depends on: Nothing+- Used by: Sourced by `reconftw.sh` between CLI parse and CLI override re-application+- Purpose: Store per-target findings in a stable directory hierarchy+- Location: `Recon/<domain>/` (created at `start()` time by `modules/modes.sh`)+- Contains: Standard subdirectories listed below+- Depends on: `start()` in modes.sh creates the directory tree+## Data Flow+### Primary Recon Request Path (`-r` / `--recon`)+### Function Execution Path (every leaf module function)+### Parallel Execution Path+### Axiom Distributed Scan Flow+- Global bash variables throughout (no encapsulation). Config vars, target vars (`domain`, `dir`, `called_fn_dir`, `LOGFILE`), and result counters are all globals+- `passive()` saves/restores module-enable globals before overriding them (`modules/modes.sh:549-611`)+## Key Abstractions+- Purpose: Lifecycle wrapper around every leaf scanning function+- Examples: Used in every function in `modules/subdomains.sh`, `modules/web.sh`, `modules/vulns.sh`, `modules/osint.sh`+- Pattern: `start_func "${FUNCNAME[0]}" "description"` at top; `end_func "output path" "${FUNCNAME[0]}"` at bottom; creates checkpoint file on end+- Purpose: Prevent re-running completed functions across multiple invocations of the same target+- Location: `Recon/<domain>/.called_fn/.funcname`+- Pattern: Each function tests `[[ ! -f "$called_fn_dir/.${FUNCNAME[0]}" ]] || [[ $DIFF == true ]]`; `end_func` writes the sentinel via `touch "$called_fn_dir/.${fn}"`+- Purpose: Universal external-tool gate for dry-run preview, axiom dispatch, adaptive rate limiting, and debug logging+- Location: `modules/utils.sh:468`+- Pattern: All tool calls inside module functions use `run_command <binary> <args>` rather than direct invocation+- Purpose: Allow modules to be sourced multiple times (test re-sourcing, `--source-only`) without re-executing+- Pattern: `[[ -n "$_FOO_LOADED" ]] && return 0` at top of each lib file (`lib/common.sh:6`, `lib/parallel.sh:6`, `lib/ui.sh:5`, `lib/validation.sh` — validation uses error-code guards instead)+- Purpose: Run independent module functions concurrently up to `PARALLEL_MAX_JOBS`+- Pattern: `parallel_funcs N func_a func_b func_c` — each function spawned as a background subshell; used in `recon()`, `osint()`, `vulns()` for independent groups+- Purpose: Transparent axiom/local fallback wrapper — if axiom fails during a module, retries locally+- Location: `modules/modes.sh:656`+- Pattern: All module calls inside `subs_menu`, `webs_menu`, `recon`, `passive` use this wrapper+## Entry Points+- Location: `reconftw.sh`+- Triggers: Direct execution (`./reconftw.sh -d example.com -r`)+- Responsibilities: Bootstrap, all module loading, CLI parse, config source, mode dispatch+- Location: `reconftw.sh:123-125`+- Triggers: `./reconftw.sh --source-only` (used by bats test `setup()` blocks)+- Responsibilities: Sources all modules without executing any recon+- Location: `modules/modes.sh:13`+- Triggers: Called at the top of most workflow functions (`recon`, `subs_menu`, `passive`, `osint`, `zen_menu`)+- Responsibilities: Create output directory tree, init LOGFILE, init cache/incremental/DNS/plugins, set global `dir` and `called_fn_dir`+- Location: `modules/modes.sh:286`+- Triggers: Called at the bottom of most workflow functions+- Responsibilities: AI report, cleanup, Faraday, screenshot diffs, plugin events, hotlist, `export_reports()`, timing summary+## Output Directory Structure+```+```+## Verbosity and Output Controls+- `0` (quiet): Only errors and final summary printed to terminal; banner suppressed+- `1` (normal, default): Errors + warnings printed; `notification()` info/good suppressed+- `2` (verbose): All `notification()` calls, PID info, full parallel output, `start_func` messages, `print_timing_summary`+- `summary`: One badge line per completed parallel job+- `tail`: Last `PARALLEL_TAIL_LINES` (default 20, doubled on failure) from each job's log+- `full`: Complete captured stdout from each job+- `jsonl-strict`: Forces `OUTPUT_VERBOSITY=0`, emits only machine-readable JSONL+## Architectural Constraints+- **Threading:** Single-threaded bash with optional background subshells via `parallel_funcs`; `wait -n` (bash 4.3+) used for job throttling in `_throttle_jobs`+- **Global state:** All config vars, `domain`, `dir`, `called_fn_dir`, `LOGFILE`, `start`, `runtime`, `DIFF`, `AXIOM`, and hundreds of module-enable flags are module-level globals; any sourced function can read or mutate them+- **Circular imports:** None by design — reconftw.sh sources libs first, then modules in explicit dependency order (`utils.sh` → `core.sh` → `osint.sh` → `subdomains.sh` → `web.sh` → `vulns.sh` → `axiom.sh` → `modes.sh`)+- **macOS bash version:** reconftw.sh re-execs itself under Homebrew bash ≥ 4 on macOS (system bash is 3.2); `lib/parallel.sh` requires bash 4.3+ for `wait -n`+- **Working directory:** `start()` calls `cd "$dir"` (the per-target output dir) before any module function runs; all relative paths inside modules resolve against the target dir. `reconftw.sh` captures `startdir=${PWD}` before this+- **No subshell isolation per module:** Modules are sourced functions, not subprocess commands. A `return` inside a module returns from the function; an `exit` would kill the whole shell+## Anti-Patterns+### Direct external tool calls without `run_command`+### Writing checkpoint files manually+### Skipping the `[[ ! -f "$called_fn_dir/.${FUNCNAME[0]}" ]] || [[ $DIFF == true ]]` guard+### Overriding config globals without save/restore in workflow functions+## Error Handling+- ERR trap in `start()` logs function name, line number, and command to `$LOGFILE` and calls `explain_err()` (`modules/modes.sh:140`)+- Non-zero exit from `parallel_funcs` increments `RECON_OSINT_PARALLEL_FAILURES` and sets `RECON_PARTIAL_RUN=true`+- `run_module_with_axiom_failover` catches axiom mid-run failures and retries locally+- Circuit-breaker helpers (`circuit_breaker_is_open`, `circuit_breaker_record_failure`) in `modules/utils.sh:1190` for persistent tool failures+## Cross-Cutting Concerns+<!-- GSD:architecture-end -->++<!-- GSD:skills-start source:skills/ -->+## Project Skills++No project skills found. Add skills to any of: `.claude/skills/`, `.agents/skills/`, `.cursor/skills/`, `.github/skills/`, or `.codex/skills/` with a `SKILL.md` index file.+<!-- GSD:skills-end -->++<!-- GSD:workflow-start source:GSD defaults -->+## GSD Workflow Enforcement++Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.++Use these entry points:+- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks+- `/gsd-debug` for investigation and bug fixing+- `/gsd-execute-phase` for planned phase work++Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.+<!-- GSD:workflow-end -->++++<!-- GSD:profile-start -->+## Developer Profile++> Profile not yet configured. Run `/gsd-profile-user` to generate your developer profile.+> This section is managed by `generate-claude-profile` -- do not edit manually.+<!-- GSD:profile-end -->