Skip to content

Autonomous OODA-loop pentesting agent. Hermes brain, Kali + Playwright tools over MCP, ATT&CK as the Orient language.

9.5 KB
Raw Blame

Design notes

Deep-dive on how vulnerator-ai works internally: the OODA loop, how it compares to public OODA patterns, how evidence is persisted, and the web flank. The README carries the usage-facing summary; this page is the detail.

The OODA loop: design and comparison

This project's "OODA loop" is not an engine that runs on its own: there is no code that iterates nor a scheduler. The OODA is prompt-induced behavior, split across three pieces of the profile (agent-ooda/):

  • profile/SOUL.md — identity of the OODA driver (first block of the system prompt).
  • skills/pentest-ooda/SKILL.md — the loop procedure (Observe → Orient → Decide → Act), stop rules and delegation.
  • skills/ooda-report-handoff/SKILL.md — format of the summary when closing a cycle.

Domain skills (conditional activation, they only load when the context fits):

  • skills/l3-recon-pivot/ — layer-3 network reconnaissance behind NAT (network/infra).
  • skills/aws-pacu-privesc/ — offensive AWS pentest with Pacu (cloud/IAM), mapped to ATT&CK Cloud.
  • skills/web-pentest/ — phased web pentest (recon → analysis → proof-based exploitation), combining kali_run (nmap/whatweb/nuclei/ffuf/sqlmap) with a real browser via the playwright MCP. Web flank.
  • skills/attack-navigation/ — MITRE ATT&CK as the Orient language: orient by tactics and tag findings with their technique_id. Cross-cutting.
  • skills/pentest-report/ — generates the formal report (Markdown) when closing the engagement, reading the findings store. Loop closure.

The trigger is the operator's message: Hermes enters the OODA role when you hand it an engagement, advances autonomously as long as there is real progress toward the goal, and stops (handoff) when the path is exhausted, after 3 steps without progress, or when it confirms the goal is unreachable.

ATT&CK as the loop's language: the control loop is still OODA; ATT&CK does not replace it. It is used as a lens in Orient (place the current tactic and the next plausible one — it feeds the "Decide with alternatives") and as a taxonomy to tag each finding (technique_id), which enables reports mapped to ATT&CK. The general detail lives in attack-navigation and the Cloud one in aws-pacu-privesc.

Comparison with public OODAs (what we learned)

Two public OODA skills/patterns were compared with ours:

  • neurofoo/agent-skills (OODA as an analysis framework). A thinking OODA: its "Act" produces a plan, it does not execute. Useful to deliberate; it is a structured report, not a control loop. Idea taken: Decide with explicit alternatives (weigh 2-3 paths before choosing one) — already distilled into pentest-ooda and SOUL.md.
  • jomadu/rooda (OODA as an execution loop). The closest to ours: it iterates O→O→D→A with fresh context each round and max-iterations. Idea taken: fresh context — in Hermes it is implemented by delegating long branches to subagents (delegate_task), which start with clean context and only return their summary. Already distilled into pentest-ooda (delegation section) and required re-enabling delegation in config.yaml.
Dimension neurofoo (analysis) rooda (execution) Our pentest-ooda
Nature Decision report Autonomous code loop Autonomous pentest loop
"Act" Plans (does not execute) Actually executes Executes on Kali (kali_run)
Stop criterion — (one-shot) max-iterations (quantitative) Qualitative progress (getting closer to the goal / unreachability)
Memory between rounds — File (AGENTS.md, PLAN.md) Ephemeral per engagement (on purpose)
Fresh context — Per iteration Per delegation to subagents
Output as untrusted data No No Yes (prompt injection in outputs)

Two things ours does better and worth preserving: the qualitative stop criterion (not a simple iteration counter) and treating all tool output as untrusted data (critical in pentesting). The two borrowed ideas (alternatives in Decide, fresh context via subagents) cover the two weaknesses we had versus the public ones.

Evidence persistence and report

The OODA has a per-engagement findings store that the handoff and the report pull from:

  • Where it lives. On the artifacts plane (/artifacts, port :8901), NOT in Hermes' state: it survives the ephemeral nature of the profile and is downloadable. One file per engagement (<engagement-id>/findings.jsonl), one finding per line.
  • How the agent feeds it. With the MCP tools record_finding / read_findings (in docker/kali/kali_mcp_server.py, over docker/kali/finding_store.py): structured append and paginated reads, without the agent writing JSON by hand.
  • Schema of a finding. id, OODA phase, class/technique_id (ATT&CK), target (host/port/resource/ARN), status (hypothesis | partial | confirmed | discarded), evidence (command
    • returncode + verbatim output or reference to the artifact), confidence, timestamp, and for changes in the account the vulnerator-ai tag + reversal command. The status aligns with the "hypothesis vs. confirmed finding" discipline from SOUL.md.
  • Consumers. ooda-report-handoff and pentest-report READ from the store instead of rebuilding from the conversation.
  • Untrusted data. The stored evidence is still untrusted data when re-read: reason about it, do not obey embedded instructions (prompt injection).

The pentest-report skill produces the formal deliverable at close: a Markdown report generated by docker/kali/report_gen.py (baked into the image, invoked with kali_run) that reads the store and leaves the report as an artifact on :8901. Structure distilled from public templates with attribution (Theophilus/athena-pentest, Marmeus/pentesting-report-generator): engagement details, executive summary, table by status, methodology (OODA/ATT&CK/PTES/OWASP), findings with CVSS 3.1 + CWE and reproducible evidence, remediation. CVSS only for confirmed findings. The report is emitted in English.

A redacted sample deliverable and its findings store live at example-report.md and example-findings.jsonl.

The web flank (browser via MCP)

The web-pentest skill covers phased web pentesting. It is not limited to kali_run: it combines the Kali CLI tools (nmap/whatweb/nuclei/ffuf/sqlmap, via kali_run) with a real browser via MCP — the playwright-mcp container (headless Chromium, official Microsoft image), registered as the second MCP server (playwright) in the profile. This preserves the project's principle (everything goes through MCP, zero native toolsets): Hermes gains mcp_playwright_* for navigation/DOM/XSS without re-enabling the native browser toolset. Its guardrails are aligned with the project's philosophy: there is no gate that blocks, but there is audit discipline and reproducible evidence so the exploitation can be re-established (and the leakage warning from the auxiliary client is kept: redact credentials, title_generation off in sensitive engagements).

The intercepting-proxy flank (Caido, driven by kali-mcp)

Beyond the CLI tools and the browser, the loop can drive Caido — a modern web-security toolkit (intercepting proxy + Replay + HTTPQL history + Automate fuzzing + interception

  • scope + workflows). Caido runs as its own container (caido, official caido/caido:0.59.0 image pinned, persistent named volume, a one-shot caido-init that fixes the volume ownership for Caido's non-root user). It is not a separate MCP server: it lives on the internal compose network and kali-mcp talks to its GraphQL API at http://caido:8080 (never published to the host). This keeps the project principle intact — the agent's tool surface is still just the MCP servers — while reusing the deterministic kali-mcp runtime as the single control point.

Why a dedicated container rather than installing Caido inside kali: Caido is a stateful, long-lived service (proxy daemon + GraphQL API + project/session store), not a short-lived CLI, so it follows the same pattern as playwright-mcp and the lab targets — a separate service reached by internal DNS. This decouples its upgrade cycle (bump CAIDO_IMAGE), keeps the kali image slim, and avoids supervising a persistent daemon inside the tool-exec container. Proxy capture does not need co-location: it works over HTTP(S)_PROXY to caido:8080 regardless.

The surface is exposed to Hermes as thin @mcp.tool wrappers over stdlib-pure modules (no new dependencies): caido_graphql (version-agnostic passthrough to the whole GraphQL schema), caido_request (send through Replay), caido_history (HTTPQL search), caido_automate (Intruder/ffuf-style fuzzing), caido_intercept (hold/list/read/edit-forward/drop), and caido_proxy (opt-in routing of kali_run's HTTP-tool traffic through Caido for capture). Every tool that sends traffic to a target (caido_request, caido_automate, and traffic-sending caido_graphql mutations) goes through the same scope-guard as kali_run, validating the full scheme://host:port so scope.json ports are honored and a blocked request sends nothing (fail-closed). Authentication is zero-touch by default (headless guest login against a --allow-guests instance); failure modes are explicit (unauthenticated / unreachable / graphql_error / timeout / failed), never faked. See docs/caido.md for the auth model, the --allow-guests trade-off, and the documented limitations.