Autonomous OODA-loop pentesting agent. Hermes brain, Kali + Playwright tools over MCP, ATT&CK as the Orient language.
Python ai-agentsautonomous-agentskali-linuxllm-agentsoffensive-securityooda-looppentestingplaywrightsecurity
feat(kali): add dns_recon (DNSDumpster) and webtech_scan (whatweb+webanalyze+wpscan) tools
G0dsp6a72b14c2de0
diff --git a/.env.example b/.env.exampleindex 1c5d967..38f7b3a 100644--- a/.env.example+++ b/.env.example@@ -123,3 +123,15 @@ PROWLER_STATUS=PROWLER_REGIONS=PROWLER_SERVICES=PROWLER_TIMEOUT=++# --- Recon / fingerprint APIs (OPTIONAL) --------------------------------------+# DNSDumpster API key for the `dns_recon` tool (DNS + attack-surface recon: subdomains,+# IPs, ASN/PTR and HTTP/HTTPS banners for a domain). Get a free key from your account at+# https://dnsdumpster.com/ (free tier: 50 records/domain; Plus raises limits + pagination).+# Without it, `dns_recon` fails fail-closed and reports the missing key.+DNSDUMPSTER_API_KEY=++# WPScan API token for the `webtech_scan` tool. OPTIONAL: wpscan enumerates WordPress+# (core/plugins/theme versions) WITHOUT a token; the token only adds known-vulnerability+# (CVE) data to the results. Get one at https://wpscan.com/profile (free tier: 25 req/day).+WPSCAN_API_TOKEN=diff --git a/docker-compose.yml b/docker-compose.ymlindex 194137e..f8f5469 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -51,6 +51,12 @@ services:PROWLER_REGIONS: "${PROWLER_REGIONS-${AWS_REGION:-us-east-1}}"PROWLER_SERVICES: "${PROWLER_SERVICES-iam s3}"PROWLER_TIMEOUT: "${PROWLER_TIMEOUT:-1800}"+ # --- Recon / fingerprint APIs (optional) ---+ # DNSDumpster API key for the `dns_recon` tool; WPScan token for `webtech_scan`+ # (optional: enables CVE data). Both empty = those tools run without them+ # (dns_recon fails fail-closed without a key; webtech_scan still enumerates).+ DNSDUMPSTER_API_KEY: "${DNSDUMPSTER_API_KEY:-}"+ WPSCAN_API_TOKEN: "${WPSCAN_API_TOKEN:-}"# --- Scope-guard (OPT-IN, off by default) ---# The scope-guard validates the DESTINATION of kali_run/kali_exec BEFORE executing,# deterministically (code, not prompt). It activates ONLY if a scope file exists atdiff --git a/docker/kali/Dockerfile b/docker/kali/Dockerfileindex fc74e8a..c8eadf6 100644--- a/docker/kali/Dockerfile+++ b/docker/kali/Dockerfile@@ -75,8 +75,39 @@ RUN apt-get update \sqlmap \gobuster \impacket-scripts \+ wpscan \&& rm -rf /var/lib/apt/lists/*+# webanalyze: motor de fingerprints de Wappalyzer (port en Go; el CLI oficial de Wappalyzer+# está descontinuado). Se usa junto a whatweb + wpscan en la tool `webtech_scan` para cruzar+# tecnologías/versiones. No está en apt: se baja el binario de release y sus fingerprints+# (`technologies.json`) a una ruta FIJA (`/opt/webanalyze`), que `webtech_scan` pasa con+# `-apps`. Versión fijada; checksum verificado contra el `checksums.txt` de la release.+ARG WEBANALYZE_VERSION=v0.4.6+RUN set -eux; \+ arch="$(dpkg --print-architecture)"; \+ case "$arch" in \+ amd64) wa_arch="x86_64" ;; \+ arm64) wa_arch="arm64" ;; \+ *) echo "[kali-mcp] webanalyze: arch $arch no soportada; se reportará ausente." && exit 0 ;; \+ esac; \+ base="https://github.com/rverton/webanalyze/releases/download/${WEBANALYZE_VERSION}"; \+ ver="${WEBANALYZE_VERSION#v}"; \+ tarball="webanalyze_Linux_${wa_arch}.tar.gz"; \+ cd /tmp; \+ curl -fsSL -o "$tarball" "${base}/${tarball}"; \+ curl -fsSL -o checksums.txt "${base}/webanalyze_${ver}_checksums.txt"; \+ grep " ${tarball}\$" checksums.txt | sha256sum -c -; \+ tar -xzf "$tarball" webanalyze; \+ install -m 0755 webanalyze /usr/local/bin/webanalyze; \+ mkdir -p /opt/webanalyze; \+ ( cd /opt/webanalyze && webanalyze -update ); \+ test -f /opt/webanalyze/technologies.json; \+ rm -f /tmp/"$tarball" /tmp/checksums.txt /tmp/webanalyze++# Ruta fija de los fingerprints de webanalyze que lee `webtech_scan` (tool `webtech_scan`).+ENV WEBANALYZE_APPS=/opt/webanalyze/technologies.json+WORKDIR /opt/kali-mcp# Virtualenv dedicado con Python 3.13 (compatible con Prowler). Ponerlo primero en@@ -138,6 +169,15 @@ COPY docker/kali/aws_iam_simulate.py ./aws_iam_simulate.py# Runner de SHELL CRUDA (tool `kali_exec`): ejecuta un comando arbitrario de la Kali vía# `sh -c` reutilizando tool_streaming (STDLIB PURA). Lo importa kali_mcp_server.COPY docker/kali/shell_exec.py ./shell_exec.py+# Cliente de la API de DNSDumpster (tool `dns_recon`): recon DNS + superficie de ataque de+# un dominio (hosts/IPs/ASN/PTR/banners) con rate limit y paginación. STDLIB PURA (urllib).+# Lo importa kali_mcp_server (import dnsdumpster_client); sin este COPY el servidor aborta+# al arrancar con ModuleNotFoundError.+COPY docker/kali/dnsdumpster_client.py ./dnsdumpster_client.py+# Fingerprint web por cruce whatweb+webanalyze+wpscan (tool `webtech_scan`): fusiona las+# versiones que ve cada herramienta y marca conflictos. STDLIB PURA. Lo importa+# kali_mcp_server (import webtech_scan as webtech); DEBE estar en la imagen.+COPY docker/kali/webtech_scan.py ./webtech_scan.py# Store de hallazgos por engagement (tools `record_finding`/`read_findings`): append# estructurado a un JSONL sobre el plano de artefactos (STDLIB PURA, reutiliza# artifact_store para confinar la ruta). Lo importa kali_mcp_server.diff --git a/docker/kali/dnsdumpster_client.py b/docker/kali/dnsdumpster_client.pynew file mode 100644index 0000000..b688469--- /dev/null+++ b/docker/kali/dnsdumpster_client.py@@ -0,0 +1,369 @@+"""Cliente de la API de DNSDumpster (STDLIB PURA) — recon DNS + superficie de ataque.++Es la LÓGICA testeable del runner ``dns_recon`` que ``kali_mcp_server`` expone como+``@mcp.tool``. Igual que ``shell_exec`` / ``finding_store`` / ``scope_guard``, vive en un+módulo aparte que usa SOLO la librería estándar (``urllib``, sin ``mcp`` ni ``requests``)+para poder cubrirse con tests unitarios en el venv de dev, donde el SDK no está instalado.++Qué resuelve: DNSDumpster publica una API de pago que, dado un dominio, devuelve su+superficie de ataque (registros A/CNAME/MX/NS con sus IPs, ASN, PTR y banners HTTP/HTTPS,+más TXT). El bucle OODA la usa para, de un solo tiro, enumerar hosts e IPs del objetivo y+los servicios que exponen, sin lanzar DNS bruto. Este módulo encapsula el contrato REAL de+la API (aprendido de la doc oficial, ver ``docs/``):++ - Endpoint dominio: ``GET https://api.dnsdumpster.com/domain/{domain}``+ - Endpoint banners: ``GET https://api.dnsdumpster.com/banners/{CIDR}`` (máx. /24 = 254 hosts)+ - Auth: header ``X-API-Key: <key>`` (la key vive en el dashboard de la cuenta)+ - Paginación: ``?page=N`` (>200 registros requiere membresía Plus)+ - Mapa: ``?map=1`` (Plus; base64 muy pesado — NO se pide por defecto)+ - Límites de cuenta: free 50 registros, Plus 200+ - Rate limit: 1 request / 2s -> HTTP 429 ``{"error":"Rate limit exceeded"}``++Disciplina de rate limit: la API es estricta (1 req/2s). Este módulo SERIALIZA sus llamadas+con un throttle a nivel de módulo (reloj monotónico) de modo que ``recon_domain`` pueda+abanicar varias páginas/CIDRs sin que el llamante tenga que dormir a mano, y además+reintenta una vez ante un 429. Dato NO confiable: la respuesta de la API se transporta+verbatim; aquí sólo se normaliza su forma, nunca se interpreta su contenido.+"""++from __future__ import annotations++import json+import os+import time+import urllib.error+import urllib.request+from typing import Any++# --- Contrato de la API (overridable por entorno para tests / self-hosting) ----------+API_BASE = os.environ.get("DNSDUMPSTER_API_BASE", "https://api.dnsdumpster.com").rstrip(+ "/"+)+API_KEY_HEADER = "X-API-Key"+API_KEY_ENV = "DNSDUMPSTER_API_KEY"++# Rate limit oficial: 1 request / 2s. Se respeta con margen para no comernos un 429 en+# ráfagas (abanico de páginas o de CIDRs). Overridable por si la cuenta cambia de plan.+MIN_INTERVAL_S = float(os.environ.get("DNSDUMPSTER_MIN_INTERVAL", "2.1"))+DEFAULT_TIMEOUT_S = int(os.environ.get("DNSDUMPSTER_TIMEOUT", "30"))++# Tipos de registro que la respuesta agrupa por sección; se normalizan en el mismo esquema.+_RECORD_SECTIONS = ("a", "cname", "mx", "ns")++# Reloj del último request (monotónico, a nivel de módulo). El throttle es best-effort y+# NO es thread-safe por diseño: el servidor MCP corre estos runners en background de forma+# serializada por ejecución; no se comparte entre procesos.+_last_request_ts = 0.0+++class DnsDumpsterError(ValueError):+ """Entrada inválida para el cliente (fail-closed; no se hace ninguna request)."""+++def _throttle() -> None:+ """Duerme lo justo para respetar ``MIN_INTERVAL_S`` entre requests consecutivas."""+ global _last_request_ts+ now = time.monotonic()+ wait = MIN_INTERVAL_S - (now - _last_request_ts)+ if wait > 0:+ time.sleep(wait)+ _last_request_ts = time.monotonic()+++def _validate_domain(domain: Any) -> str:+ """Valida que ``domain`` sea un string no vacío y sin esquema/espacios.++ NO resuelve ni comprueba que exista: sólo rechaza fail-closed lo que no sirve como+ path de la API (no-string, vacío, con ``/`` o espacios) para no construir una URL rota.+ """+ if not isinstance(domain, str) or not domain.strip():+ raise DnsDumpsterError("domain inválido: se espera un dominio no vacío.")+ d = domain.strip().lower()+ if d.startswith(("http://", "https://")):+ d = d.split("://", 1)[1]+ d = d.strip("/")+ if "/" in d or " " in d or not d:+ raise DnsDumpsterError(f"domain inválido: {domain!r}")+ return d+++def _resolve_key(api_key: str | None) -> str:+ """Devuelve la API key explícita o la del entorno, o lanza fail-closed."""+ key = (api_key or os.environ.get(API_KEY_ENV, "")).strip()+ if not key:+ raise DnsDumpsterError(+ f"Falta la API key de DNSDumpster: pásala o exporta {API_KEY_ENV}."+ )+ return key+++def _get(path: str, api_key: str, timeout: int) -> dict[str, Any]:+ """GET autenticado a ``API_BASE + path`` con throttle y manejo de errores HTTP.++ Returns un dict de estado uniforme (nunca lanza por un error HTTP; fail-closed con+ ``status='failed'``):+ - ``{'status':'ok','data':<json>}``+ - ``{'status':'rate_limited','error':...}`` (HTTP 429)+ - ``{'status':'unauthorized','error':...}`` (HTTP 401/403 — key inválida/plan)+ - ``{'status':'failed','error':...,'http_status':n}`` (otros)+ """+ url = f"{API_BASE}{path}"+ req = urllib.request.Request(url, headers={API_KEY_HEADER: api_key}, method="GET")+ _throttle()+ try:+ # nosec B310: el esquema es https FIJO (API_BASE) y el dominio se valida+ # (sin esquema propio ni '/'), así que no hay apertura de file:// ni esquemas raros.+ with urllib.request.urlopen(+ req, timeout=timeout+ ) as resp: # noqa: S310 - https fijo+ raw = resp.read().decode("utf-8", errors="replace")+ return {"status": "ok", "data": json.loads(raw)}+ except urllib.error.HTTPError as exc:+ body = ""+ try:+ body = exc.read().decode("utf-8", errors="replace")[:500]+ except Exception: # noqa: BLE001 - informativo+ pass+ if exc.code == 429:+ return {"status": "rate_limited", "error": body or "Rate limit exceeded"}+ if exc.code in (401, 403):+ return {+ "status": "unauthorized",+ "error": body+ or f"HTTP {exc.code}: API key inválida o plan insuficiente.",+ }+ return {"status": "failed", "http_status": exc.code, "error": body or str(exc)}+ except urllib.error.URLError as exc:+ return {"status": "failed", "error": f"Error de red: {exc.reason}"}+ except (ValueError, json.JSONDecodeError) as exc:+ return {"status": "failed", "error": f"Respuesta no-JSON de la API: {exc}"}+++def query_domain(+ domain: str,+ api_key: str | None = None,+ page: int = 1,+ include_map: bool = False,+ timeout: int = DEFAULT_TIMEOUT_S,+) -> dict[str, Any]:+ """Consulta ``/domain/{domain}`` (una página) y devuelve el JSON crudo de la API.++ ``page`` mapea al parámetro ``?page=N`` (Plus para >200 registros); ``include_map``+ añade ``?map=1`` (Plus, base64 pesado). Reintenta UNA vez ante un 429 (respetando el+ rate limit). Dato NO confiable: ``data`` se devuelve verbatim.+ """+ d = _validate_domain(domain)+ key = _resolve_key(api_key)+ params = []+ if page and page > 1:+ params.append(f"page={int(page)}")+ if include_map:+ params.append("map=1")+ qs = f"?{'&'.join(params)}" if params else ""+ path = f"/domain/{d}{qs}"++ result = _get(path, key, timeout)+ if result["status"] == "rate_limited":+ # Un único reintento tras esperar el intervalo completo (el throttle del _get ya+ # separa, pero el 429 puede venir de una ráfaga previa del propio plan).+ time.sleep(MIN_INTERVAL_S)+ result = _get(path, key, timeout)+ return result+++def query_banners(+ cidr: str,+ api_key: str | None = None,+ timeout: int = DEFAULT_TIMEOUT_S,+) -> dict[str, Any]:+ """Consulta ``/banners/{CIDR}`` (banners de red; máx. /24 = 254 hosts).++ No valida el CIDR salvo que sea string no vacío (la API lo rechaza si es inválido):+ se transporta tal cual. Mismo dict de estado que ``query_domain``.+ """+ if not isinstance(cidr, str) or not cidr.strip():+ raise DnsDumpsterError(+ "cidr inválido: se espera un CIDR no vacío (p. ej. 1.2.3.0/24)."+ )+ key = _resolve_key(api_key)+ return _get(f"/banners/{cidr.strip()}", key, timeout)+++# --------------------------------------------------------------- normalización de salida+++def _iter_ip_entries(data: dict[str, Any]) -> list[dict[str, Any]]:+ """Aplana A/CNAME/MX/NS en una lista de registros {record_type, host, ip, ...}.++ La respuesta agrupa por sección y cada host lleva una lista ``ips``; aquí se desdobla+ a una fila por IP, conservando el tipo de registro y los campos de banner si existen.+ Robusto a secciones ausentes o con forma inesperada (dato NO confiable).+ """+ rows: list[dict[str, Any]] = []+ for section in _RECORD_SECTIONS:+ entries = data.get(section)+ if not isinstance(entries, list):+ continue+ for entry in entries:+ if not isinstance(entry, dict):+ continue+ host = entry.get("host", "")+ ips = entry.get("ips")+ if not isinstance(ips, list):+ continue+ for ipobj in ips:+ if not isinstance(ipobj, dict):+ continue+ banners = (+ ipobj.get("banners")+ if isinstance(ipobj.get("banners"), dict)+ else {}+ )+ http = (+ banners.get("http") if isinstance(banners.get("http"), dict) else {}+ )+ https = (+ banners.get("https")+ if isinstance(banners.get("https"), dict)+ else {}+ )+ apps = []+ for b in (http, https):+ for a in (+ b.get("apps", []) if isinstance(b.get("apps"), list) else []+ ):+ if a and a not in apps:+ apps.append(a)+ rows.append(+ {+ "record_type": section.upper(),+ "host": host,+ "ip": ipobj.get("ip", ""),+ "asn": ipobj.get("asn", ""),+ "asn_name": ipobj.get("asn_name", ""),+ "asn_range": ipobj.get("asn_range", ""),+ "country": ipobj.get("country", ""),+ "country_code": ipobj.get("country_code", ""),+ "ptr": ipobj.get("ptr", ""),+ "http_server": http.get("server", ""),+ "http_title": http.get("title", ""),+ "https_cn": https.get("cn", ""),+ "https_server": https.get("server", ""),+ "apps": apps,+ }+ )+ return rows+++def extract_surface(data: dict[str, Any]) -> dict[str, Any]:+ """Normaliza el JSON de ``/domain`` en una superficie de ataque consumible.++ Devuelve ``{hosts:[...], ips:[...], cidrs:[...], txt:[...], counts:{...}}`` donde:+ - ``hosts`` : una fila por (host, IP) con ASN/PTR/banners (``_iter_ip_entries``).+ - ``ips`` : lista DEDUPLICADA de IPs vistas (para abanicar escaneos aguas abajo).+ - ``cidrs`` : lista DEDUPLICADA de ``asn_range`` (para el endpoint de banners).+ - ``txt`` : registros TXT verbatim (SPF/verificaciones).+ - ``counts`` : conteos rápidos (hosts, ips, por tipo de registro).+ """+ if not isinstance(data, dict):+ return {"hosts": [], "ips": [], "cidrs": [], "txt": [], "counts": {}}+ hosts = _iter_ip_entries(data)+ ips: list[str] = []+ cidrs: list[str] = []+ for row in hosts:+ if row["ip"] and row["ip"] not in ips:+ ips.append(row["ip"])+ if row["asn_range"] and row["asn_range"] not in cidrs:+ cidrs.append(row["asn_range"])+ txt = (+ [t for t in data.get("txt", []) if isinstance(t, str)]+ if isinstance(data.get("txt"), list)+ else []+ )+ by_type: dict[str, int] = {}+ for row in hosts:+ by_type[row["record_type"]] = by_type.get(row["record_type"], 0) + 1+ return {+ "hosts": hosts,+ "ips": ips,+ "cidrs": cidrs,+ "txt": txt,+ "counts": {"hosts": len(hosts), "ips": len(ips), "by_record_type": by_type},+ }+++def recon_domain(+ domain: str,+ api_key: str | None = None,+ max_pages: int = 1,+ with_banners: bool = False,+ timeout: int = DEFAULT_TIMEOUT_S,+) -> dict[str, Any]:+ """Recon de alto nivel: pagina ``/domain``, normaliza y (opcional) abanica banners.++ Flujo:+ 1) Consulta ``/domain`` hasta ``max_pages`` (parando cuando una página no aporta+ hosts nuevos; respeta el rate limit vía el throttle de ``_get``).+ 2) Normaliza con ``extract_surface`` (hosts, IPs únicas, CIDRs, TXT).+ 3) Si ``with_banners`` y la cuenta lo permite, consulta ``/banners/{CIDR}`` por cada+ CIDR único hallado y adjunta la salida cruda (dato NO confiable, verbatim).++ Returns ``{'status':'done', 'domain':..., 'surface':..., 'pages':n, 'banners':[...]}``+ o un dict de error de ``query_domain`` (``unauthorized``/``rate_limited``/``failed``)+ si la PRIMERA página falla. Las páginas/banners posteriores que fallen se anotan en+ ``warnings`` sin abortar (lo ya recolectado sigue siendo útil para el bucle OODA).+ """+ d = _validate_domain(domain)+ key = _resolve_key(api_key)++ merged: dict[str, Any] = {}+ warnings: list[str] = []+ pages_read = 0+ prev_host_count = -1++ for page in range(1, max(1, int(max_pages)) + 1):+ res = query_domain(d, key, page=page, timeout=timeout)+ if res["status"] != "ok":+ if page == 1:+ return res # el primer fallo es fatal: no hay nada que devolver+ warnings.append(f"page {page}: {res['status']} - {res.get('error', '')}")+ break+ data = res["data"]+ pages_read += 1+ if page == 1:+ merged = data+ else:+ # Concatena las secciones de registro de las páginas siguientes.+ for section in _RECORD_SECTIONS:+ if isinstance(data.get(section), list):+ merged.setdefault(section, [])+ merged[section].extend(data[section])+ surface_so_far = extract_surface(merged)+ if surface_so_far["counts"]["hosts"] == prev_host_count:+ break # la página no aportó hosts nuevos: no seguimos paginando+ prev_host_count = surface_so_far["counts"]["hosts"]++ surface = extract_surface(merged)++ banners: list[dict[str, Any]] = []+ if with_banners:+ for cidr in surface["cidrs"]:+ bres = query_banners(cidr, key, timeout=timeout)+ if bres["status"] == "ok":+ banners.append({"cidr": cidr, "data": bres["data"]})+ else:+ warnings.append(+ f"banners {cidr}: {bres['status']} - {bres.get('error', '')}"+ )++ out: dict[str, Any] = {+ "status": "done",+ "domain": d,+ "pages": pages_read,+ "surface": surface,+ }+ if with_banners:+ out["banners"] = banners+ if warnings:+ out["warnings"] = warnings+ return outdiff --git a/docker/kali/kali_mcp_server.py b/docker/kali/kali_mcp_server.pyindex e908044..edd8d0d 100644--- a/docker/kali/kali_mcp_server.py+++ b/docker/kali/kali_mcp_server.py@@ -56,6 +56,18 @@ import aws_simulate# no hay gate de autorización (es responsabilidad del operador).import shell_exec+# Cliente de la API de DNSDumpster (STDLIB PURA, sin `mcp`/`requests`): recon DNS ++# superficie de ataque (hosts, IPs, ASN, PTR, banners) de un dominio, con rate limit+# (1 req/2s) y paginación. La tool `dns_recon` es una capa fina sobre `recon_domain`.+import dnsdumpster_client++# Fingerprinting web por CRUCE de whatweb + webanalyze + wpscan (STDLIB PURA): fusiona las+# versiones que ven las tres herramientas y marca conflictos. La tool `webtech_scan`+# delega en `run_webtech`. Cada binario es opcional: si no está, se omite (no se finge).+# Se importa con alias para que la tool `@mcp.tool def webtech_scan(...)` no sombree al+# módulo (ambos se llamarían igual en el espacio de nombres global).+import webtech_scan as webtech+# Store de hallazgos por engagement (STDLIB PURA): append estructurado a un JSONL sobre# el plano de artefactos (``/artifacts``). Las tools `record_finding`/`read_findings` son# una capa fina sobre ``finding_store.append``/``read``. Es el registro durable del que@@ -159,6 +171,19 @@ KALI_RUN_TOOL_NAME = "kali_run"RECORD_FINDING_TOOL_NAME = "record_finding"READ_FINDINGS_STORE_TOOL_NAME = "read_findings"+# Tool de recon DNS + superficie de ataque vía la API de DNSDumpster (BLOQUEANTE como+# `kali_run`): dado un dominio, enumera hosts/subdominios, IPs únicas, ASN/PTR y banners+# HTTP/HTTPS de un tiro, respetando el rate limit de la API (1 req/2s). Pensada para+# alimentar el abanico de escaneos aguas abajo (p. ej. `webtech_scan` por cada host/IP).+# Requiere `DNSDUMPSTER_API_KEY` en el entorno (o pasada como argumento).+DNS_RECON_TOOL_NAME = "dns_recon"++# Tool de fingerprint web por CRUCE de 3 herramientas (BLOQUEANTE): corre whatweb ++# webanalyze + wpscan contra una URL y fusiona qué tecnologías/versiones ve cada una,+# marcando conflictos de versión. wpscan sólo aplica a WordPress (si no lo es, lo reporta+# sin error). Cada binario es opcional: el que no esté se omite y se anota en `skipped`.+WEBTECH_SCAN_TOOL_NAME = "webtech_scan"+# El endpoint MCP se monta en la raíz para que KALI_MCP_URL=http://kali:8900# (sin path) funcione tal cual con el cliente (streamable-HTTP del SDK oficial).mcp = FastMCP("kali-mcp", host=HOST, port=PORT, streamable_http_path="/")@@ -197,12 +222,23 @@ def server_info() -> dict[str, Any]:# Disponibilidad REAL por collector (which de prowler/scout/cloudsplaining); no se# finge en arm64: lo que no esté se reporta ausente y su collection falla claro.collectors = collector_runner.available_collectors()+ # Disponibilidad REAL de los binarios de recon/fingerprint (no se finge en arm64 ni en+ # imágenes recortadas): lo que no esté se reporta ausente y su tool lo omite/avisa.+ recon_tools = {+ "whatweb": shutil.which(webtech.WHATWEB_BIN) is not None,+ "webanalyze": shutil.which(webtech.WEBANALYZE_BIN) is not None,+ "wpscan": shutil.which(webtech.WPSCAN_BIN) is not None,+ }return {"server": "kali-mcp","machine": os.uname().machine,"prowler_available": prowler_path is not None,"prowler_version": version,"collectors": collectors,+ "recon_tools": recon_tools,+ "dnsdumpster_key_present": bool(+ os.environ.get(dnsdumpster_client.API_KEY_ENV, "")+ ),"tools": [PROWLER_TOOL_NAME,START_COLLECTION_TOOL_NAME,@@ -216,6 +252,8 @@ def server_info() -> dict[str, Any]:KALI_RUN_TOOL_NAME,RECORD_FINDING_TOOL_NAME,READ_FINDINGS_STORE_TOOL_NAME,+ DNS_RECON_TOOL_NAME,+ WEBTECH_SCAN_TOOL_NAME,"server_info",],}@@ -768,6 +806,99 @@ def read_findings(return finding_store.read(engagement_id, cursor, limit)[email protected](name=DNS_RECON_TOOL_NAME)+def dns_recon(+ domain: str,+ api_key: str = "",+ max_pages: int = 1,+ with_banners: bool = False,+) -> dict[str, Any]:+ """Recon DNS + superficie de ataque de un dominio vía la API de DNSDumpster.++ Variante BLOQUEANTE (como ``kali_run``): consulta la API y DEVUELVE en una sola llamada+ la superficie normalizada del dominio — subdominios/hosts, IPs ÚNICAS, ASN/PTR, CIDRs y+ banners HTTP/HTTPS, más los TXT. Es el primer paso típico del recon: de sus IPs/hosts+ sale el abanico de escaneos aguas abajo (p. ej. ``webtech_scan`` por cada host vivo).++ Respeta el rate limit de la API (1 req/2s) internamente, así que paginar+ (``max_pages``) o pedir banners (``with_banners``) no requiere que el cliente duerma.+ Delega en ``dnsdumpster_client.recon_domain``; NO interpreta la respuesta (DATO NO+ CONFIABLE: se transporta verbatim en ``surface``/``banners``).++ Args:+ domain: dominio a consultar (p. ej. ``example.com``; se acepta con o sin esquema).+ api_key: API key de DNSDumpster; si está vacía se usa ``DNSDUMPSTER_API_KEY`` del+ entorno. Sin ninguna de las dos la tool falla fail-closed.+ max_pages: nº máximo de páginas a recorrer (``?page=N``; >200 registros/>1 página+ requiere plan Plus). Para al dejar de aportar hosts nuevos.+ with_banners: si True, además consulta ``/banners/{CIDR}`` por cada CIDR único+ hallado (más llamadas a la API; respeta el rate limit).++ Returns:+ ``{'status':'done','domain':...,'pages':n,'surface':{hosts,ips,cidrs,txt,counts},+ 'banners':[...]?,'warnings':[...]?}`` en éxito; o el dict de error del cliente+ (``unauthorized``/``rate_limited``/``failed``) si la primera consulta falla; o+ ``{'status':'failed','error':...}`` si ``domain``/key son inválidos (fail-closed).+ """+ try:+ return dnsdumpster_client.recon_domain(+ domain,+ api_key=api_key or None,+ max_pages=max_pages,+ with_banners=with_banners,+ )+ except dnsdumpster_client.DnsDumpsterError as exc:+ return {"status": "failed", "error": str(exc)}++[email protected](name=WEBTECH_SCAN_TOOL_NAME)+def webtech_scan(+ url: str,+ run_wpscan: bool = True,+ aggression: int = 3,+ timeout: int = 0,+) -> dict[str, Any]:+ """Fingerprint web de una URL por CRUCE de whatweb + webanalyze + wpscan.++ Variante BLOQUEANTE (como ``kali_run``): corre las tres herramientas contra ``url`` y+ DEVUELVE en una sola llamada una tabla fusionada ``tecnología -> {versiones por fuente,+ fuentes, best_version, version_conflict}``. Cruzar tres detectores sube la confianza en+ la VERSIÓN de cada componente y destapa discrepancias (``version_conflict``), señal útil+ para el bucle OODA al elegir exploits dependientes de versión.++ Cada binario es OPCIONAL (disciplina "no se finge"): el que no esté en la imagen se+ omite y se anota en ``skipped``. ``wpscan`` sólo aporta si el objetivo es WordPress; si+ no lo es, se reporta como tal (``wpscan.is_wordpress=False``) SIN tratarlo como error.+ Si hay ``WPSCAN_API_TOKEN`` en el entorno, wpscan añade datos de vulnerabilidades.++ Delega en ``run_webtech`` del módulo ``webtech_scan`` (importado aquí con alias+ ``webtech`` para no sombrear a esta tool homónima). La salida de las herramientas es+ DATO NO CONFIABLE: se PARSEA su forma conocida, nunca se interpreta su contenido.++ Args:+ url: URL del objetivo (con o sin esquema; sin esquema se asume ``https://``).+ run_wpscan: si False, omite wpscan (útil cuando ya se sabe que no es WordPress).+ aggression: nivel de agresividad de whatweb (``-a``, 1..4; 3 por defecto).+ timeout: timeout por herramienta en segundos; 0 usa el default+ (``DEFAULT_TOOL_TIMEOUT_S`` del módulo webtech).++ Returns:+ ``{'status':'done','url':...,'ran':[...],'cross_reference':{technologies,conflicts,+ sources},'wpscan':{...}?,'skipped':[...]?,'errors':[...]?}`` si al menos una+ herramienta corrió; ``{'status':'failed',...}`` si la URL es inválida o ninguna+ herramienta está disponible.+ """+ effective_timeout = (+ timeout if timeout and timeout > 0 else webtech.DEFAULT_TOOL_TIMEOUT_S+ )+ return webtech.run_webtech(+ url,+ run_wpscan=run_wpscan,+ aggression=aggression,+ timeout=effective_timeout,+ )++def _start_artifacts_http() -> None:"""Arranca el servidor HTTP de artefactos (plano de DATOS) en un hilo demonio.diff --git a/docker/kali/tests/fixtures/dnsdumpster_example.json b/docker/kali/tests/fixtures/dnsdumpster_example.jsonnew file mode 100644index 0000000..68f154a--- /dev/null+++ b/docker/kali/tests/fixtures/dnsdumpster_example.json@@ -0,0 +1,93 @@+{+ "a": [+ {+ "host": "www.example.com",+ "ips": [+ {+ "ip": "192.0.2.10",+ "asn": "64500",+ "asn_name": "EXAMPLE-AS",+ "asn_range": "192.0.2.0/24",+ "country": "United States",+ "country_code": "US",+ "ptr": "www.example.com",+ "banners": {+ "ip": "192.0.2.10",+ "http": {"apps": ["nginx"], "server": "nginx", "title": "Example Domain"},+ "https": {+ "alt_n": ["example.com", "www.example.com"],+ "apps": ["nginx", "PHP"],+ "cn": "example.com",+ "server": "nginx",+ "title": "Example Domain"+ }+ }+ }+ ]+ },+ {+ "host": "blog.example.com",+ "ips": [+ {+ "ip": "192.0.2.11",+ "asn": "64500",+ "asn_name": "EXAMPLE-AS",+ "asn_range": "192.0.2.0/24",+ "country": "United States",+ "country_code": "US",+ "ptr": "",+ "banners": {}+ }+ ]+ }+ ],+ "cname": [],+ "mx": [+ {+ "host": "mail.example.com",+ "ips": [+ {+ "ip": "198.51.100.20",+ "asn": "64501",+ "asn_name": "MAIL-AS",+ "asn_range": "198.51.100.0/24",+ "country": "United States",+ "country_code": "US",+ "ptr": "mail.example.com"+ }+ ]+ }+ ],+ "ns": [+ {+ "host": "ns1.example.net",+ "ips": [+ {+ "ip": "203.0.113.30",+ "asn": "64502",+ "asn_name": "NS-AS",+ "asn_range": "203.0.113.0/24",+ "country": "United States",+ "country_code": "US",+ "ptr": "ns1.example.net"+ }+ ]+ },+ {+ "host": "ns2.example.net",+ "ips": [+ {+ "ip": "203.0.113.31",+ "asn": "64502",+ "asn_name": "NS-AS",+ "asn_range": "203.0.113.0/24",+ "country": "United States",+ "country_code": "US",+ "ptr": "ns2.example.net"+ }+ ]+ }+ ],+ "total_a_recs": 2,+ "txt": ["v=spf1 -all", "google-site-verification=EXAMPLEtoken"]+}diff --git a/docker/kali/tests/fixtures/webanalyze_example.json b/docker/kali/tests/fixtures/webanalyze_example.jsonnew file mode 100644index 0000000..90b661b--- /dev/null+++ b/docker/kali/tests/fixtures/webanalyze_example.json@@ -0,0 +1,36 @@+{+ "hostname": "https://www.example.com",+ "matches": [+ {+ "app": {+ "cats": ["1", "11"],+ "category_names": ["CMS", "Blogs"],+ "website": "https://wordpress.org",+ "implies": ["PHP", "MySQL"]+ },+ "app_name": "WordPress",+ "matches": [["meta generator"]],+ "version": "6.4.8"+ },+ {+ "app": {+ "cats": ["59"],+ "category_names": ["JavaScript libraries"],+ "website": "https://jquery.com"+ },+ "app_name": "jQuery",+ "matches": [],+ "version": "3.6.0"+ },+ {+ "app": {+ "cats": ["22"],+ "category_names": ["Web servers"],+ "website": "https://nginx.org"+ },+ "app_name": "Nginx",+ "matches": [],+ "version": ""+ }+ ]+}diff --git a/docker/kali/tests/fixtures/whatweb_example.json b/docker/kali/tests/fixtures/whatweb_example.jsonnew file mode 100644index 0000000..8b45812--- /dev/null+++ b/docker/kali/tests/fixtures/whatweb_example.json@@ -0,0 +1,24 @@+[+ {+ "target": "https://example.com",+ "http_status": 301,+ "request_config": {"headers": {"User-Agent": "WhatWeb/0.6.4"}},+ "plugins": {+ "HTTPServer": {"string": ["nginx"]},+ "RedirectLocation": {"string": ["https://www.example.com/"]}+ }+ },+ {+ "target": "https://www.example.com/",+ "http_status": 200,+ "request_config": {"headers": {"User-Agent": "WhatWeb/0.6.4"}},+ "plugins": {+ "HTTPServer": {"string": ["nginx"]},+ "MetaGenerator": {"string": ["WordPress 6.4.8"]},+ "jQuery": {"version": ["3.6.0"]},+ "PHP": {"version": ["8.1.2"]},+ "HTML5": {},+ "Title": {"string": ["Example Domain"]}+ }+ }+]diff --git a/docker/kali/tests/fixtures/wpscan_not_wordpress.json b/docker/kali/tests/fixtures/wpscan_not_wordpress.jsonnew file mode 100644index 0000000..551dbe1--- /dev/null+++ b/docker/kali/tests/fixtures/wpscan_not_wordpress.json@@ -0,0 +1,7 @@+{+ "db_files_updated": [],+ "db_update_started": true,+ "db_update_finished": true,+ "target_url": "https://www.example.com/",+ "scan_aborted": "The remote website is up, but does not seem to be running WordPress."+}diff --git a/docker/kali/tests/fixtures/wpscan_wordpress.json b/docker/kali/tests/fixtures/wpscan_wordpress.jsonnew file mode 100644index 0000000..21d4105--- /dev/null+++ b/docker/kali/tests/fixtures/wpscan_wordpress.json@@ -0,0 +1,34 @@+{+ "target_url": "https://blog.example.com/",+ "effective_url": "https://blog.example.com/",+ "version": {+ "number": "6.4.8",+ "status": "insecure",+ "confidence": 100,+ "interesting_entries": ["https://blog.example.com/, Match: 'WordPress 6.4.8'"]+ },+ "main_theme": {+ "slug": "twentytwentyfour",+ "style_name": "Twenty Twenty-Four",+ "version": {"number": "1.2", "confidence": 80}+ },+ "plugins": {+ "contact-form-7": {+ "slug": "contact-form-7",+ "version": {"number": "5.9.3", "confidence": 100}+ },+ "revslider": {+ "slug": "revslider",+ "version": {"number": "6.6.0", "confidence": 90}+ },+ "wordpress-seo": {+ "slug": "wordpress-seo",+ "version": {"number": "22.6", "confidence": 100}+ }+ },+ "interesting_findings": [+ {"url": "https://blog.example.com/xmlrpc.php", "type": "xmlrpc", "to_s": "XML-RPC seems to be enabled"},+ {"url": "https://blog.example.com/readme.html", "type": "readme", "to_s": "WordPress readme found"}+ ],+ "scan_aborted": null+}diff --git a/docker/kali/tests/test_dnsdumpster_client.py b/docker/kali/tests/test_dnsdumpster_client.pynew file mode 100644index 0000000..83ff712--- /dev/null+++ b/docker/kali/tests/test_dnsdumpster_client.py@@ -0,0 +1,219 @@+"""Tests stdlib-only del cliente DNSDumpster (`dnsdumpster_client.py`).++Ejercitan la LÓGICA del cliente SIN tocar la red (``_get`` se sustituye por un fake o se+monkeypatchea ``urllib.request.urlopen``). Fijan que:+ - ``_validate_domain`` normaliza (quita esquema/barras) y rechaza fail-closed la basura;+ - ``_resolve_key`` toma la key explícita o la del entorno y falla sin ella;+ - ``extract_surface`` aplana A/CNAME/MX/NS a filas (host, IP, ASN, banners), deduplica+ IPs y CIDRs y recoge TXT — comprobado contra un fixture SINTÉTICO (example.com, IPs de+ rango documentación TEST-NET) con la forma real de la API;+ - el manejo de estados HTTP: 200->ok, 429->rate_limited (con reintento), 401/403->+ unauthorized, otros->failed, cuerpo no-JSON->failed;+ - ``recon_domain`` devuelve el error de la primera página si falla, y agrega páginas.+"""++from __future__ import annotations++import json+import sys+from pathlib import Path++import pytest++sys.path.insert(0, str(Path(__file__).resolve().parent.parent))++import dnsdumpster_client as dd # noqa: E402++FIXTURES = Path(__file__).resolve().parent / "fixtures"+++def _load(name: str) -> dict:+ return json.loads((FIXTURES / name).read_text(encoding="utf-8"))+++# --------------------------------------------------------------- validación de entrada+++ "raw,expected",+ [+ ("example.com", "example.com"),+ ("https://example.com/", "example.com"),+ ("HTTP://Example.COM", "example.com"),+ (" example.com ", "example.com"),+ ],+)+def test_validate_domain_normaliza(raw, expected):+ assert dd._validate_domain(raw) == expected++[email protected]("bad", ["", " ", None, 123, "a b.com", "foo.com/path"])+def test_validate_domain_rechaza(bad):+ with pytest.raises(dd.DnsDumpsterError):+ dd._validate_domain(bad)+++def test_resolve_key_explicita_o_entorno(monkeypatch):+ assert dd._resolve_key("abc") == "abc"+ monkeypatch.setenv(dd.API_KEY_ENV, "fromenv")+ assert dd._resolve_key(None) == "fromenv"+ monkeypatch.delenv(dd.API_KEY_ENV, raising=False)+ with pytest.raises(dd.DnsDumpsterError):+ dd._resolve_key(None)+++# --------------------------------------------------------------- normalización de salida+++def test_extract_surface_fixture():+ data = _load("dnsdumpster_example.json")+ surface = dd.extract_surface(data)+ # El fixture trae hosts en A/MX/NS; se aplanan a una fila por IP.+ assert surface["counts"]["hosts"] == 5+ assert len(surface["ips"]) == len(set(surface["ips"])) # deduplicadas+ assert all(ip for ip in surface["ips"]) # sin vacíos+ # Debe haber registros A y NS (el fixture usa ambos).+ types = surface["counts"]["by_record_type"]+ assert "A" in types and "NS" in types+ # Las filas llevan los campos normalizados esperados.+ row = surface["hosts"][0]+ assert set(["record_type", "host", "ip", "asn", "ptr", "apps"]).issubset(row)+ # TXT se recoge verbatim (el fixture publica SPF + verificación).+ assert "v=spf1 -all" in surface["txt"]+++def test_extract_surface_banners_y_apps():+ data = {+ "a": [+ {+ "host": "web.example.com",+ "ips": [+ {+ "ip": "192.0.2.4",+ "asn": "64500",+ "asn_name": "EXAMPLE",+ "asn_range": "192.0.2.0/24",+ "country": "US",+ "country_code": "US",+ "ptr": "web.example.com",+ "banners": {+ "http": {+ "apps": ["Nginx"],+ "server": "nginx",+ "title": "Home",+ },+ "https": {+ "apps": ["Nginx", "PHP"],+ "cn": "example.com",+ "server": "nginx",+ },+ },+ }+ ],+ }+ ],+ "txt": ["v=spf1 -all"],+ }+ surface = dd.extract_surface(data)+ assert surface["ips"] == ["192.0.2.4"]+ assert surface["cidrs"] == ["192.0.2.0/24"]+ row = surface["hosts"][0]+ assert row["http_server"] == "nginx"+ assert row["https_cn"] == "example.com"+ assert "PHP" in row["apps"] and "Nginx" in row["apps"] # apps fusionadas http+https+++def test_extract_surface_robusto_a_basura():+ # Secciones ausentes, tipos inesperados: no revienta, devuelve vacío coherente.+ assert dd.extract_surface({})["ips"] == []+ assert dd.extract_surface({"a": "no-es-lista"})["hosts"] == []+ assert dd.extract_surface(None)["hosts"] == []+++# --------------------------------------------------------------- manejo HTTP (fake urlopen)+++class _FakeResp:+ def __init__(self, body: str):+ self._body = body.encode("utf-8")++ def read(self):+ return self._body++ def __enter__(self):+ return self++ def __exit__(self, *a):+ return False+++def test_get_ok(monkeypatch):+ monkeypatch.setattr(dd, "_throttle", lambda: None)+ monkeypatch.setattr(+ dd.urllib.request, "urlopen", lambda req, timeout: _FakeResp('{"a":[]}')+ )+ res = dd._get("/domain/x", "k", 10)+ assert res["status"] == "ok" and res["data"] == {"a": []}+++def _make_httperror(code: int, body: str):+ import urllib.error++ err = urllib.error.HTTPError("http://x", code, "err", {}, None)+ err.read = lambda: body.encode("utf-8") # type: ignore[assignment]+ return err+++def test_get_status_codes(monkeypatch):+ monkeypatch.setattr(dd, "_throttle", lambda: None)++ def raise_code(code, body=""):+ def _urlopen(req, timeout):+ raise _make_httperror(code, body)++ return _urlopen++ monkeypatch.setattr(+ dd.urllib.request, "urlopen", raise_code(429, '{"error":"Rate limit exceeded"}')+ )+ assert dd._get("/domain/x", "k", 10)["status"] == "rate_limited"++ monkeypatch.setattr(dd.urllib.request, "urlopen", raise_code(403))+ assert dd._get("/domain/x", "k", 10)["status"] == "unauthorized"++ monkeypatch.setattr(dd.urllib.request, "urlopen", raise_code(500, "boom"))+ assert dd._get("/domain/x", "k", 10)["status"] == "failed"+++def test_get_cuerpo_no_json(monkeypatch):+ monkeypatch.setattr(dd, "_throttle", lambda: None)+ monkeypatch.setattr(+ dd.urllib.request,+ "urlopen",+ lambda req, timeout: _FakeResp("<html>no json</html>"),+ )+ assert dd._get("/domain/x", "k", 10)["status"] == "failed"+++# --------------------------------------------------------------- recon_domain (alto nivel)+++def test_recon_domain_primer_fallo_es_fatal(monkeypatch):+ monkeypatch.setattr(+ dd,+ "query_domain",+ lambda *a, **k: {"status": "unauthorized", "error": "bad key"},+ )+ out = dd.recon_domain("example.com", api_key="k")+ assert out["status"] == "unauthorized"+++def test_recon_domain_ok_normaliza(monkeypatch):+ data = _load("dnsdumpster_example.json")+ monkeypatch.setattr(+ dd, "query_domain", lambda *a, **k: {"status": "ok", "data": data}+ )+ out = dd.recon_domain("example.com", api_key="k", max_pages=1)+ assert out["status"] == "done"+ assert out["domain"] == "example.com"+ assert out["surface"]["counts"]["hosts"] == 5diff --git a/docker/kali/tests/test_server_recon_tools.py b/docker/kali/tests/test_server_recon_tools.pynew file mode 100644index 0000000..3609b99--- /dev/null+++ b/docker/kali/tests/test_server_recon_tools.py@@ -0,0 +1,67 @@+"""Tests de FORMA del cableado de `dns_recon` / `webtech_scan` en el servidor MCP.++Restricción del entorno (igual que ``test_kali_run``): ``kali_mcp_server.py`` exige+``mcp<2`` (FastMCP 1.x) y en el venv dev no está, así que NO se importa el servidor: se+inspecciona su TEXTO fuente. La LÓGICA de ambas tools vive en ``dnsdumpster_client`` /+``webtech_scan`` y se cubre en sus propios tests; aquí sólo se fija que el servidor:+ - importa los módulos (``webtech_scan`` con ALIAS para no sombrear a la tool homónima);+ - declara las constantes de nombre de tool y las registra en ``server_info``;+ - expone wrappers ``@mcp.tool`` que DELEGAN en la lógica (no la reimplementan);+ - reporta la disponibilidad REAL de binarios y la presencia de la API key.+"""++from __future__ import annotations++from pathlib import Path++_SERVER_PATH = Path(__file__).resolve().parent.parent / "kali_mcp_server.py"+++def _source() -> str:+ return _SERVER_PATH.read_text(encoding="utf-8")+++def test_imports_de_modulos():+ source = _source()+ assert "import dnsdumpster_client" in source+ # El módulo webtech_scan se importa con alias para no colisionar con la tool.+ assert "import webtech_scan as webtech" in source+++def test_constantes_de_tool():+ source = _source()+ assert 'DNS_RECON_TOOL_NAME = "dns_recon"' in source+ assert 'WEBTECH_SCAN_TOOL_NAME = "webtech_scan"' in source+++def test_wrappers_registrados_y_delegan():+ source = _source()+ assert "@mcp.tool(name=DNS_RECON_TOOL_NAME)" in source+ assert "def dns_recon(" in source+ assert "dnsdumpster_client.recon_domain(" in source++ assert "@mcp.tool(name=WEBTECH_SCAN_TOOL_NAME)" in source+ assert "def webtech_scan(" in source+ # Delega usando el ALIAS del módulo, no el nombre de la función (evita el shadowing).+ assert "webtech.run_webtech(" in source+++def test_server_info_lista_tools_y_disponibilidad():+ source = _source()+ assert "DNS_RECON_TOOL_NAME," in source+ assert "WEBTECH_SCAN_TOOL_NAME," in source+ assert '"recon_tools":' in source+ assert '"dnsdumpster_key_present":' in source+ # La disponibilidad se calcula con shutil.which sobre los binarios del módulo webtech.+ assert "shutil.which(webtech.WHATWEB_BIN)" in source+ assert "shutil.which(webtech.WPSCAN_BIN)" in source+++def test_no_hay_shadowing_del_modulo():+ # Dentro del cuerpo de la tool NO debe usarse `webtech_scan.` (sería la función, no el+ # módulo): toda referencia al módulo va por el alias `webtech.`.+ source = _source()+ body_start = source.index("def webtech_scan(")+ body = source[body_start:]+ assert "webtech_scan.run_webtech" not in body+ assert "webtech_scan.DEFAULT_TOOL_TIMEOUT_S" not in bodydiff --git a/docker/kali/tests/test_webtech_scan.py b/docker/kali/tests/test_webtech_scan.pynew file mode 100644index 0000000..d82c97f--- /dev/null+++ b/docker/kali/tests/test_webtech_scan.py@@ -0,0 +1,162 @@+"""Tests stdlib-only del fingerprinting web por cruce (`webtech_scan.py`).++Ejercitan los PARSERS puros y el CRUCE contra fixtures SINTÉTICOS (example.com) con la+forma real de cada herramienta: whatweb (``--log-json``), webanalyze (``-output json``) y+wpscan (``--format json``, tanto "no es WordPress" como WordPress detectado). Fijan que:+ - ``parse_whatweb`` extrae plugins+versiones del array ``--log-json``;+ - ``parse_webanalyze`` extrae app_name/version/categorías del ``-output json``;+ - ``parse_wpscan`` distingue "no es WordPress" (``scan_aborted``) de WP detectado+ (core+plugins+tema con versión) y normaliza a ``techs``;+ - ``cross_reference`` fusiona por nombre, agrega versiones por fuente, y marca+ ``version_conflict`` cuando dos fuentes discrepan y acuerdo cuando coinciden;+ - ``run_webtech`` falla fail-closed con URL inválida y omite (no finge) una herramienta+ ausente.+"""++from __future__ import annotations++import sys+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent.parent))++import webtech_scan as wt # noqa: E402++FIXTURES = Path(__file__).resolve().parent / "fixtures"+++def _read(name: str) -> str:+ return (FIXTURES / name).read_text(encoding="utf-8")+++# --------------------------------------------------------------- parsers+++def test_parse_whatweb_fixture():+ techs = wt.parse_whatweb(_read("whatweb_example.json"))+ names = {t["name"] for t in techs}+ assert "HTTPServer" in names and "MetaGenerator" in names+ assert all(t["source"] == "whatweb" for t in techs)+ # El fixture trae plugins versionados (jQuery/PHP): su versión se extrae.+ by_name = {t["name"]: t["version"] for t in techs}+ assert by_name.get("jQuery") == "3.6.0"+++def test_parse_whatweb_versiones():+ raw = '[{"plugins": {"jQuery": {"version": ["3.4.1"]}, "PHP": {"version": ["8.1.2"]}, "HTML5": {}}}]'+ techs = {t["name"]: t["version"] for t in wt.parse_whatweb(raw)}+ assert techs["jQuery"] == "3.4.1"+ assert techs["PHP"] == "8.1.2"+ assert techs["HTML5"] == "" # sin versión pero presente+++def test_parse_whatweb_json_roto():+ assert wt.parse_whatweb("no json") == []+ assert wt.parse_whatweb("") == []+++def test_parse_webanalyze_fixture():+ techs = wt.parse_webanalyze(_read("webanalyze_example.json"))+ names = {t["name"] for t in techs}+ assert "WordPress" in names+ assert all(t["source"] == "webanalyze" for t in techs)+ # Las categorías se conservan (WordPress -> CMS) y la versión se extrae.+ wp = next(t for t in techs if t["name"] == "WordPress")+ assert wp["version"] == "6.4.8"+ assert any("CMS" in c for c in wp.get("categories", []))+++def test_parse_wpscan_no_es_wordpress_fixture():+ res = wt.parse_wpscan(_read("wpscan_not_wordpress.json"))+ assert res["is_wordpress"] is False+ assert (+ "WordPress" in res["aborted_reason"] or "does not seem" in res["aborted_reason"]+ )+ assert res["techs"] == []+++def test_parse_wpscan_wordpress_detectado():+ res = wt.parse_wpscan(_read("wpscan_wordpress.json"))+ assert res["is_wordpress"] is True+ assert res["wp_version"] == "6.4.8"+ assert "twentytwentyfour" in res["theme"]+ slugs = {p["slug"] for p in res["plugins"]}+ assert {"contact-form-7", "revslider", "wordpress-seo"}.issubset(slugs)+ # normaliza core+plugins+tema a techs con prefijo plugin:/theme:+ tech_names = {t["name"] for t in res["techs"]}+ assert "WordPress" in tech_names+ assert "plugin:contact-form-7" in tech_names+ cf7 = next(t for t in res["techs"] if t["name"] == "plugin:contact-form-7")+ assert cf7["version"] == "5.9.3"+++# --------------------------------------------------------------- cruce de versiones+++def test_cross_reference_acuerdo():+ # Dos fuentes ven jQuery con la MISMA versión: sin conflicto, best = esa versión.+ a = [{"name": "jQuery", "version": "3.4.1", "source": "whatweb"}]+ b = [{"name": "jquery", "version": "3.4.1", "source": "webanalyze"}]+ out = wt.cross_reference(a, b)+ jq = next(t for t in out["technologies"] if t["name"].lower() == "jquery")+ assert jq["version_conflict"] is False+ assert jq["best_version"] == "3.4.1"+ assert set(jq["sources"]) == {"whatweb", "webanalyze"}+ assert out["conflicts"] == []+++def test_cross_reference_conflicto():+ # Dos fuentes discrepan en la versión de jQuery: se marca conflicto y se listan ambas.+ a = [{"name": "jQuery", "version": "3.4.1", "source": "whatweb"}]+ b = [{"name": "jQuery", "version": "3.5.0", "source": "webanalyze"}]+ out = wt.cross_reference(a, b)+ jq = next(t for t in out["technologies"] if t["name"].lower() == "jquery")+ assert jq["version_conflict"] is True+ assert set(out["conflicts"][0]["versions"]) == {"3.4.1", "3.5.0"}+++def test_cross_reference_orden_por_num_fuentes():+ # La tech vista por más fuentes va primero.+ a = [{"name": "WordPress", "version": "6.4.8", "source": "whatweb"}]+ b = [{"name": "WordPress", "version": "6.4.8", "source": "wpscan"}]+ c = [{"name": "Nginx", "version": "1.25.3", "source": "whatweb"}]+ out = wt.cross_reference(a, b, c)+ assert out["technologies"][0]["name"] == "WordPress" # 2 fuentes > 1+++def test_cross_reference_tres_fuentes():+ ww = wt.parse_whatweb(_read("whatweb_example.json"))+ wa = wt.parse_webanalyze(_read("webanalyze_example.json"))+ wp = wt.parse_wpscan(_read("wpscan_not_wordpress.json"))["techs"]+ out = wt.cross_reference(ww, wa, wp)+ assert "whatweb" in out["sources"] and "webanalyze" in out["sources"]+ assert len(out["technologies"]) > 0+ # whatweb y webanalyze comparten el nombre "jQuery" con la misma versión -> se fusiona+ # en una sola entrada vista por dos fuentes, sin conflicto.+ jq = next(t for t in out["technologies"] if t["name"].lower() == "jquery")+ assert set(jq["sources"]) == {"whatweb", "webanalyze"}+ assert jq["version_conflict"] is False+ assert jq["best_version"] == "3.6.0"+++# --------------------------------------------------------------- orquestador+++def test_run_webtech_url_invalida():+ out = wt.run_webtech("")+ assert out["status"] == "failed"+++def test_run_webtech_sin_binarios(monkeypatch):+ # Si ninguna herramienta está disponible, falla claro (no finge).+ monkeypatch.setattr(wt, "_which", lambda b: None)+ out = wt.run_webtech("https://example.com")+ assert out["status"] == "failed"+ assert set(out["skipped"]) >= {"whatweb", "webanalyze"}+++def test_run_webtech_normaliza_url(monkeypatch):+ # Sin esquema -> se asume https; con ninguna tool disponible seguimos validando la URL.+ monkeypatch.setattr(wt, "_which", lambda b: None)+ out = wt.run_webtech("example.com")+ assert out["url"] == "https://example.com"diff --git a/docker/kali/webtech_scan.py b/docker/kali/webtech_scan.pynew file mode 100644index 0000000..f1c999b--- /dev/null+++ b/docker/kali/webtech_scan.py@@ -0,0 +1,464 @@+"""Fingerprinting web por CRUCE de whatweb + webanalyze + wpscan (STDLIB PURA).++Es la LÓGICA testeable del runner ``webtech_scan`` que ``kali_mcp_server`` expone como+``@mcp.tool``. Igual que ``shell_exec`` / ``dnsdumpster_client`` / ``scope_guard``, vive en+un módulo aparte con SOLO la librería estándar (sin ``mcp``) para cubrirse con tests+unitarios en el venv de dev.++Qué resuelve: tres herramientas ven la pila tecnológica de un sitio con sesgos distintos.+Ninguna sola es de fiar para la VERSIÓN de un componente; cruzarlas sube la confianza y+destapa discrepancias. Este módulo ejecuta las tres (cada una OPCIONAL: si el binario no+está se omite y se reporta, nunca se finge) y FUSIONA sus salidas en una tabla única+``tecnología -> {versiones por fuente, fuentes que la vieron, acuerdo/conflicto}``:++ - whatweb : fingerprint amplio y rápido; versiones en ``plugins.<x>.version``.+ - webanalyze : motor de fingerprints de Wappalyzer (port en Go; el CLI oficial de+ Wappalyzer está descontinuado). Categorías + versión por app.+ - wpscan : especialista WordPress. Si el objetivo NO es WordPress lo dice+ (``scan_aborted``) y se reporta como tal (NO es un error); si lo es,+ aporta versión del core, plugins/temas con versión y hallazgos.++Cruce de versiones: por cada tecnología se recogen TODAS las versiones observadas por+fuente. Si dos fuentes dan versiones distintas para el mismo componente se marca+``version_conflict`` (señal para el bucle OODA de que la versión no es concluyente). La+versión "mejor" es la más específica concordante.++Dato NO confiable: la salida de las tres herramientas se transporta verbatim; aquí sólo se+PARSEA su forma conocida, nunca se interpreta su contenido. Las funciones ``parse_*`` y+``cross_reference`` son PURAS (string JSON -> dict) y son el núcleo cubierto por tests; el+orquestador ``run_webtech`` ejecuta los binarios y delega en ellas.+"""++from __future__ import annotations++import json+import os+import shutil+import subprocess+import tempfile+from typing import Any++# Timeout por herramienta (segundos). whatweb/webanalyze son rápidos; wpscan puede tardar.+DEFAULT_TOOL_TIMEOUT_S = int(os.environ.get("WEBTECH_TOOL_TIMEOUT", "300"))++# Nombres de binario (overridable por entorno para contenedores con rutas no estándar).+WHATWEB_BIN = os.environ.get("WHATWEB_BIN", "whatweb")+WEBANALYZE_BIN = os.environ.get("WEBANALYZE_BIN", "webanalyze")+WPSCAN_BIN = os.environ.get("WPSCAN_BIN", "wpscan")++# webanalyze lee sus fingerprints de ``technologies.json`` en el CWD por defecto. En el+# contenedor el fichero se baja a una ruta fija (ver Dockerfile) y se expone aquí para+# pasarlo con ``-apps``; si está vacío, webanalyze usa su búsqueda por defecto.+WEBANALYZE_APPS = os.environ.get("WEBANALYZE_APPS", "")+++class WebTechError(ValueError):+ """Entrada inválida para el runner (fail-closed; no se ejecuta nada)."""+++def _validate_url(url: Any) -> str:+ """Valida que ``url`` sea http(s) no vacía, o lanza ``WebTechError``."""+ if not isinstance(url, str) or not url.strip():+ raise WebTechError("url inválida: se espera una URL http(s) no vacía.")+ u = url.strip()+ if not u.startswith(("http://", "https://")):+ u = "https://" + u+ return u+++# --------------------------------------------------------------- parsers (PUROS)+++def parse_whatweb(raw: str) -> list[dict[str, Any]]:+ """Parsea el ``--log-json`` de whatweb en ``[{name, version, source='whatweb'}]``.++ whatweb emite un ARRAY (una entrada por salto de URL); cada una lleva ``plugins`` como+ dict ``nombre -> {version:[...], string:[...]}``. Se extrae nombre+versión de cada+ plugin. Robusto a JSON vacío/erróneo (devuelve lista vacía): dato NO confiable.+ """+ out: list[dict[str, Any]] = []+ seen: set[tuple[str, str]] = set()+ try:+ docs = json.loads(raw)+ except (ValueError, json.JSONDecodeError):+ return out+ if isinstance(docs, dict):+ docs = [docs]+ if not isinstance(docs, list):+ return out+ for doc in docs:+ plugins = doc.get("plugins") if isinstance(doc, dict) else None+ if not isinstance(plugins, dict):+ continue+ for name, info in plugins.items():+ versions = info.get("version", []) if isinstance(info, dict) else []+ if not isinstance(versions, list):+ versions = [versions]+ versions = [str(v) for v in versions if v not in (None, "", [])]+ if versions:+ for v in versions:+ key = (name, v)+ if key not in seen:+ seen.add(key)+ out.append({"name": name, "version": v, "source": "whatweb"})+ else:+ key = (name, "")+ if key not in seen:+ seen.add(key)+ out.append({"name": name, "version": "", "source": "whatweb"})+ return out+++def parse_webanalyze(raw: str) -> list[dict[str, Any]]:+ """Parsea el ``-output json`` de webanalyze en ``[{name, version, categories, source}]``.++ webanalyze emite un objeto ``{hostname, matches:[{app_name, version, app:{category_names}}]}``+ (o un array de ellos si se le pasan varios hosts). Robusto a formas inesperadas.+ """+ out: list[dict[str, Any]] = []+ seen: set[tuple[str, str]] = set()+ try:+ data = json.loads(raw)+ except (ValueError, json.JSONDecodeError):+ return out+ docs = data if isinstance(data, list) else [data]+ for doc in docs:+ matches = doc.get("matches") if isinstance(doc, dict) else None+ if not isinstance(matches, list):+ continue+ for m in matches:+ if not isinstance(m, dict):+ continue+ name = m.get("app_name") or m.get("app") or ""+ if not isinstance(name, str) or not name:+ continue+ version = str(m.get("version") or "")+ cats = []+ app = m.get("app")+ if isinstance(app, dict) and isinstance(app.get("category_names"), list):+ cats = [str(c) for c in app["category_names"]]+ key = (name, version)+ if key not in seen:+ seen.add(key)+ out.append(+ {+ "name": name,+ "version": version,+ "categories": cats,+ "source": "webanalyze",+ }+ )+ return out+++def parse_wpscan(raw: str) -> dict[str, Any]:+ """Parsea el ``--format json`` de wpscan en un resumen WordPress estructurado.++ Devuelve ``{is_wordpress, aborted_reason, wp_version, theme, plugins:[...],+ interesting_count, techs:[...]}`` donde ``techs`` normaliza core + plugins + tema a+ ``[{name, version, source='wpscan'}]`` para el cruce. Si wpscan abortó porque el sitio+ no es WordPress, ``is_wordpress=False`` y ``aborted_reason`` lleva el motivo (NO es un+ error). Robusto a JSON vacío/erróneo.+ """+ result: dict[str, Any] = {+ "is_wordpress": False,+ "aborted_reason": "",+ "wp_version": "",+ "theme": "",+ "plugins": [],+ "interesting_count": 0,+ "techs": [],+ }+ try:+ data = json.loads(raw)+ except (ValueError, json.JSONDecodeError):+ result["aborted_reason"] = "salida de wpscan no parseable"+ return result+ if not isinstance(data, dict):+ return result++ aborted = data.get("scan_aborted")+ if aborted:+ result["aborted_reason"] = str(aborted)+ return result++ techs: list[dict[str, Any]] = []+ version = data.get("version")+ if isinstance(version, dict) and version.get("number"):+ result["is_wordpress"] = True+ result["wp_version"] = str(version["number"])+ techs.append(+ {"name": "WordPress", "version": result["wp_version"], "source": "wpscan"}+ )++ main_theme = data.get("main_theme")+ if isinstance(main_theme, dict):+ tname = main_theme.get("slug") or main_theme.get("style_name") or ""+ tver = (+ str(main_theme.get("version", {}).get("number", ""))+ if isinstance(main_theme.get("version"), dict)+ else ""+ )+ if tname:+ result["is_wordpress"] = True+ result["theme"] = f"{tname} {tver}".strip()+ techs.append(+ {"name": f"theme:{tname}", "version": tver, "source": "wpscan"}+ )++ plugins = data.get("plugins")+ if isinstance(plugins, dict):+ for slug, pinfo in plugins.items():+ if not isinstance(pinfo, dict):+ continue+ result["is_wordpress"] = True+ pver = ""+ if isinstance(pinfo.get("version"), dict):+ pver = str(pinfo["version"].get("number", ""))+ result["plugins"].append({"slug": slug, "version": pver})+ techs.append(+ {"name": f"plugin:{slug}", "version": pver, "source": "wpscan"}+ )++ interesting = data.get("interesting_findings")+ if isinstance(interesting, list):+ result["interesting_count"] = len(interesting)+ # Un target vivo con findings pero sin version/plugins detectados sigue siendo WP+ # si wpscan no abortó: lo marcamos para no reportar un falso "no es WordPress".+ if interesting and not result["is_wordpress"] and not result["aborted_reason"]:+ result["is_wordpress"] = True++ result["techs"] = techs+ return result+++def _norm(name: str) -> str:+ """Clave de fusión case-insensitive y sin espacios de borde para un nombre de tech."""+ return name.strip().lower()+++def cross_reference(*tech_lists: list[dict[str, Any]]) -> dict[str, Any]:+ """Fusiona varias listas ``[{name, version, source}]`` cruzando versiones por fuente.++ Devuelve ``{technologies:[...], conflicts:[...], sources:[...]}`` donde cada entrada de+ ``technologies`` es::++ {name, sources:[...], versions:{source: [v,...]}, best_version, version_conflict}++ ``version_conflict`` es True si dos fuentes reportan versiones NO vacías distintas para+ el mismo componente (señal de versión no concluyente). ``best_version`` es la versión+ concordante más específica (la más larga entre las observadas; vacío si ninguna fuente+ dio versión). El cruce es por nombre normalizado (case-insensitive).+ """+ merged: dict[str, dict[str, Any]] = {}+ all_sources: list[str] = []++ for tech_list in tech_lists:+ for tech in tech_list:+ name = tech.get("name", "")+ if not name:+ continue+ source = tech.get("source", "?")+ version = str(tech.get("version", "") or "")+ if source not in all_sources:+ all_sources.append(source)+ key = _norm(name)+ entry = merged.setdefault(+ key,+ {"name": name, "sources": [], "versions": {}, "categories": []},+ )+ if source not in entry["sources"]:+ entry["sources"].append(source)+ entry["versions"].setdefault(source, [])+ if version and version not in entry["versions"][source]:+ entry["versions"][source].append(version)+ cats = tech.get("categories")+ if isinstance(cats, list):+ for c in cats:+ if c and c not in entry["categories"]:+ entry["categories"].append(c)++ technologies: list[dict[str, Any]] = []+ conflicts: list[dict[str, Any]] = []+ for entry in merged.values():+ distinct_versions = sorted(+ {v for versions in entry["versions"].values() for v in versions}+ )+ conflict = len(distinct_versions) > 1+ best = max(distinct_versions, key=len) if distinct_versions else ""+ tech_out = {+ "name": entry["name"],+ "sources": entry["sources"],+ "versions": entry["versions"],+ "best_version": best,+ "version_conflict": conflict,+ }+ if entry["categories"]:+ tech_out["categories"] = entry["categories"]+ technologies.append(tech_out)+ if conflict:+ conflicts.append(+ {+ "name": entry["name"],+ "versions": distinct_versions,+ "by_source": entry["versions"],+ }+ )++ # Orden estable: primero las vistas por más fuentes, luego alfabético.+ technologies.sort(key=lambda t: (-len(t["sources"]), t["name"].lower()))+ return {+ "technologies": technologies,+ "conflicts": conflicts,+ "sources": all_sources,+ }+++# --------------------------------------------------------------- orquestador (ejecuta)+++def _which(bin_name: str) -> str | None:+ return shutil.which(bin_name)+++def _run(cmd: list[str], timeout: int) -> tuple[int, str, str]:+ """Ejecuta ``cmd`` sin shell con timeout; devuelve ``(rc, stdout, stderr)``.++ No lanza por returncode ni por timeout: un fallo de la herramienta es información, no+ una excepción. Dato NO confiable: stdout/stderr verbatim.+ """+ try:+ proc = subprocess.run( # noqa: S603 - binarios fijos, args construidos aquí+ cmd, capture_output=True, text=True, timeout=timeout+ )+ return proc.returncode, proc.stdout or "", proc.stderr or ""+ except subprocess.TimeoutExpired:+ return -1, "", f"timeout tras {timeout}s"+ except FileNotFoundError:+ return -1, "", "binario no encontrado"+++def run_webtech(+ url: str,+ run_wpscan: bool = True,+ aggression: int = 3,+ timeout: int = DEFAULT_TOOL_TIMEOUT_S,+) -> dict[str, Any]:+ """Ejecuta whatweb + webanalyze + (opcional) wpscan y CRUZA sus versiones.++ Cada herramienta es opcional: si su binario no está, se omite y se anota en ``skipped``+ (disciplina "no se finge"). Flujo por herramienta: ejecutar -> capturar JSON (fichero o+ stdout) -> parsear con la función pura correspondiente. Finalmente ``cross_reference``+ fusiona las tres fuentes.++ Returns ``{'status':'done', url, cross_reference:{...}, wpscan:{...}, ran:[...],+ skipped:[...], errors:[...]}``. ``status`` es ``'done'`` aunque alguna herramienta+ falle (lo recolectado por las demás sigue siendo útil); sólo es ``'failed'`` si la URL+ es inválida o NINGUNA herramienta pudo ejecutarse.+ """+ try:+ u = _validate_url(url)+ except WebTechError as exc:+ return {"status": "failed", "url": url, "error": str(exc)}+ ran: list[str] = []+ skipped: list[str] = []+ errors: list[str] = []+ whatweb_techs: list[dict[str, Any]] = []+ webanalyze_techs: list[dict[str, Any]] = []+ wpscan_summary: dict[str, Any] = {}++ workdir = tempfile.mkdtemp(prefix="webtech-")+ try:+ # --- whatweb (--log-json a fichero) ---+ if _which(WHATWEB_BIN):+ out_f = os.path.join(workdir, "whatweb.json")+ rc, _so, se = _run(+ [+ WHATWEB_BIN,+ "--quiet",+ f"--log-json={out_f}",+ "-a",+ str(aggression),+ u,+ ],+ timeout,+ )+ try:+ with open(out_f, encoding="utf-8", errors="replace") as fh:+ whatweb_techs = parse_whatweb(fh.read())+ ran.append("whatweb")+ except OSError:+ errors.append(f"whatweb: sin salida (rc={rc}) {se[:120]}")+ else:+ skipped.append("whatweb")++ # --- webanalyze (-output json a stdout) ---+ if _which(WEBANALYZE_BIN):+ wa_cmd = [WEBANALYZE_BIN, "-host", u, "-output", "json", "-crawl", "0"]+ if WEBANALYZE_APPS:+ wa_cmd += ["-apps", WEBANALYZE_APPS]+ rc, so, se = _run(wa_cmd, timeout)+ webanalyze_techs = parse_webanalyze(so)+ if webanalyze_techs or rc == 0:+ ran.append("webanalyze")+ else:+ errors.append(f"webanalyze: sin matches (rc={rc}) {se[:120]}")+ else:+ skipped.append("webanalyze")++ # --- wpscan (--format json a fichero) ---+ if run_wpscan:+ if _which(WPSCAN_BIN):+ out_f = os.path.join(workdir, "wpscan.json")+ cmd = [+ WPSCAN_BIN,+ "--url",+ u,+ "--format",+ "json",+ "--no-banner",+ "--random-user-agent",+ "-o",+ out_f,+ ]+ token = os.environ.get("WPSCAN_API_TOKEN", "").strip()+ if token:+ cmd += ["--api-token", token]+ rc, _so, se = _run(cmd, timeout)+ try:+ with open(out_f, encoding="utf-8", errors="replace") as fh:+ wpscan_summary = parse_wpscan(fh.read())+ ran.append("wpscan")+ except OSError:+ errors.append(f"wpscan: sin salida (rc={rc}) {se[:120]}")+ else:+ skipped.append("wpscan")++ if not ran:+ return {+ "status": "failed",+ "url": u,+ "error": "ninguna herramienta de fingerprint disponible",+ "skipped": skipped,+ "errors": errors,+ }++ wpscan_techs = wpscan_summary.get("techs", []) if wpscan_summary else []+ crossref = cross_reference(whatweb_techs, webanalyze_techs, wpscan_techs)++ out: dict[str, Any] = {+ "status": "done",+ "url": u,+ "ran": ran,+ "cross_reference": crossref,+ }+ if wpscan_summary:+ # resumen WP sin la lista ``techs`` (ya está fusionada en cross_reference)+ out["wpscan"] = {k: v for k, v in wpscan_summary.items() if k != "techs"}+ if skipped:+ out["skipped"] = skipped+ if errors:+ out["errors"] = errors+ return out+ finally:+ shutil.rmtree(workdir, ignore_errors=True)diff --git a/docs/usage.md b/docs/usage.mdindex fc97c48..cfdf8b1 100644--- a/docs/usage.md+++ b/docs/usage.md@@ -82,6 +82,48 @@ For CI scripts or plain `curl`, Hermes exposes an OpenAI-compatible API server d(the same endpoints `vulnerator-mcp` wraps). Full reference — access modes, the runs flow,`chat/completions`, and the request/response contract — in [`http-api.md`](http-api.md).+# Recon & fingerprint tools++The Kali MCP server (`kali-mcp`) ships two dedicated recon tools alongside the raw shell+(`kali_run`/`kali_exec`). They are **blocking** tools: one call in, structured result out — no+`started → poll` dance. `server_info` reports which binaries are present (`recon_tools`) and+whether the DNSDumpster key is set (`dnsdumpster_key_present`).++## `dns_recon` — DNS + attack surface (DNSDumpster API)++Given a domain, enumerates its attack surface in one call: subdomains/hosts, **unique IPs**,+ASN/PTR and HTTP/HTTPS banners, plus TXT records. It is the usual first recon step — its IPs and+hosts feed the downstream fan-out (e.g. `webtech_scan` per live host).++- **Key:** set `DNSDUMPSTER_API_KEY` in `.env` (free key at <https://dnsdumpster.com/>; the+ installer offers to store it). Without a key the tool fails fail-closed and says so.+- **Rate limit:** the API allows 1 request / 2 s; the client throttles internally and retries once+ on HTTP 429, so paginating (`max_pages`) or fetching banners (`with_banners`) needs no manual+ sleeps. Free tier returns up to 50 records/domain; pagination beyond 200 needs a Plus plan.+- **Returns:** `{status, domain, pages, surface:{hosts, ips, cidrs, txt, counts}, banners?, warnings?}`.+ The API response is untrusted data — it is normalized in shape, never interpreted.++## `webtech_scan` — technology/version fingerprint by cross-reference++Runs **whatweb + webanalyze + wpscan** against a URL and **merges** what each one sees into a single+`technology → {versions per source, sources, best_version, version_conflict}` table. Cross-checking+three detectors raises confidence in a component's version and flags disagreements+(`version_conflict`) — a useful signal before choosing a version-dependent exploit.++- **Optional binaries, no faking:** any tool missing from the image is skipped and listed under+ `skipped` (the image installs all three). `wpscan` only contributes on WordPress; on a non-WP+ site it reports `wpscan.is_wordpress=false` **as a normal result, not an error**.+- **WPScan token (optional):** set `WPSCAN_API_TOKEN` in `.env` to add known-vulnerability (CVE)+ data; wpscan still enumerates core/plugin/theme versions without it.+- **Wappalyzer note:** the official Wappalyzer CLI is discontinued, so the Wappalyzer *fingerprint+ engine* is provided via [`webanalyze`](https://github.com/rverton/webanalyze) (Go port). Its+ fingerprint DB lives at `/opt/webanalyze/technologies.json` in the image.+- **Cross-reference caveat:** merging is by (normalized) technology name. whatweb and webanalyze use+ different naming vocabularies for some techs (e.g. whatweb encodes a CMS inside `MetaGenerator`'s+ value while webanalyze names the app directly), so not every detection auto-merges between those+ two. The cross-reference is most valuable where tools share a name (jQuery, nginx, PHP, WordPress,+ …) and for surfacing version conflicts.+# Practice against a lab target (Juice Shop)To practice the `web-pentest` skill without going out to the internet or touching real systems,diff --git a/install.sh b/install.shindex eabdd3f..b754d54 100755--- a/install.sh+++ b/install.sh@@ -113,6 +113,36 @@ read_openrouter_key() {export OPENROUTER_API_KEY}+# --- 2b. Recon / fingerprint API keys (OPTIONAL) -----------------------------+# These are NOT required to bring the stack up. They enable two recon tools:+# - DNSDUMPSTER_API_KEY -> the `dns_recon` tool (DNS + attack-surface recon).+# - WPSCAN_API_TOKEN -> CVE data in the `webtech_scan` tool (wpscan works without it).+# Resolution order per key: exported environment variable > interactive prompt (Enter to+# skip) > empty. A non-interactive run just leaves whatever the environment provided.+read_recon_keys() {+ # DNSDumpster+ if [ -z "${DNSDUMPSTER_API_KEY:-}" ] && [ -t 0 ]; then+ echo+ info "OPTIONAL: DNSDumpster API key for the 'dns_recon' tool (DNS + attack surface)."+ info "Free key at https://dnsdumpster.com/ — press Enter to skip."+ printf 'DNSDUMPSTER_API_KEY (optional): '+ read -rs DNSDUMPSTER_API_KEY+ echo+ fi+ export DNSDUMPSTER_API_KEY="${DNSDUMPSTER_API_KEY:-}"++ # WPScan+ if [ -z "${WPSCAN_API_TOKEN:-}" ] && [ -t 0 ]; then+ echo+ info "OPTIONAL: WPScan API token for 'webtech_scan' (adds CVE data; wpscan runs without it)."+ info "Free token at https://wpscan.com/profile — press Enter to skip."+ printf 'WPSCAN_API_TOKEN (optional): '+ read -rs WPSCAN_API_TOKEN+ echo+ fi+ export WPSCAN_API_TOKEN="${WPSCAN_API_TOKEN:-}"+}+# --- 3. Secret generation ----------------------------------------------------gen_secret() { openssl rand -hex 32; }@@ -123,13 +153,18 @@ gen_secret() { openssl rand -hex 32; }# avoiding any injection coming from the user-supplied API key.build_env() {local openrouter_key="$1" hermes_key="$2" mcp_token="$3"+ # Optional recon keys (may be empty); passed through to .env as-is.+ local dnsdumpster_key="${4:-}" wpscan_token="${5:-}"- info "Building .env from .env.example (only the three required keys are filled in)..."+ info "Building .env from .env.example (required keys filled in; optional recon keys if provided)..."- awk -v ork="$openrouter_key" -v hk="$hermes_key" -v mt="$mcp_token" '+ awk -v ork="$openrouter_key" -v hk="$hermes_key" -v mt="$mcp_token" \+ -v ddk="$dnsdumpster_key" -v wpt="$wpscan_token" '/^OPENROUTER_API_KEY=/ { print "OPENROUTER_API_KEY=" ork; next }/^HERMES_API_SERVER_KEY=/ { print "HERMES_API_SERVER_KEY=" hk; next }/^VULNERATOR_MCP_INBOUND_TOKEN=/ { print "VULNERATOR_MCP_INBOUND_TOKEN=" mt; next }+ /^DNSDUMPSTER_API_KEY=/ { print "DNSDUMPSTER_API_KEY=" ddk; next }+ /^WPSCAN_API_TOKEN=/ { print "WPSCAN_API_TOKEN=" wpt; next }{ print }' "$ENV_EXAMPLE" > "$ENV_FILE.tmp"@@ -168,11 +203,15 @@ main() {read_openrouter_key[ -n "${OPENROUTER_API_KEY:-}" ] || die "OPENROUTER_API_KEY is empty. It is required in OpenRouter mode (the default)."+ # Optional recon/fingerprint API keys (Enter to skip; empty is fine).+ read_recon_keys+local hermes_key mcp_tokenhermes_key="$(gen_secret)"mcp_token="$(gen_secret)"- build_env "$OPENROUTER_API_KEY" "$hermes_key" "$mcp_token"+ build_env "$OPENROUTER_API_KEY" "$hermes_key" "$mcp_token" \+ "${DNSDUMPSTER_API_KEY:-}" "${WPSCAN_API_TOKEN:-}"if [ "$NO_UP" -eq 1 ]; thenecho@@ -190,6 +229,8 @@ main() {echo " - OPENROUTER_API_KEY (the key you provided)"echo " - HERMES_API_SERVER_KEY (auto-generated, 32-byte hex)"echo " - VULNERATOR_MCP_INBOUND_TOKEN (auto-generated, 32-byte hex)"+ [ -n "${DNSDUMPSTER_API_KEY:-}" ] && echo " - DNSDUMPSTER_API_KEY (optional; enables dns_recon)"+ [ -n "${WPSCAN_API_TOKEN:-}" ] && echo " - WPSCAN_API_TOKEN (optional; CVE data in webtech_scan)"echoinfo "Open the interactive TUI once the containers are healthy:"echo " docker compose exec -it ooda-agent hermes chat"