Fully autonomous AI Agents system capable of performing complex penetration testing tasks
Go ai-agentsai-security-toolanthropicautonomous-agentsgolanggptgraphqlmulti-agent-systemoffensive-securityopen-sourceopenaipenetration-testingpenetration-testing-toolsreactsecurity-automationsecurity-testingsecurity-toolsself-hosted
docs: add headless crawler integration RFC
mason5052364fbd4010a1
diff --git a/examples/proposals/headless_crawler_integration.md b/examples/proposals/headless_crawler_integration.mdnew file mode 100644index 00000000..406d5ff1--- /dev/null+++ b/examples/proposals/headless_crawler_integration.md@@ -0,0 +1,472 @@+# RFC: Headless Crawler Integration for PentAGI++## Summary++Issue [#336](https://github.com/vxcontrol/pentagi/issues/336) asks for+a headless crawler / URL discovery capability. The current web testing+flow can brute-force directories with dictionary tools such as `ffuf`,+but it cannot crawl an application to discover its real routes, links,+forms, parameters, and JavaScript endpoints. Dictionary fuzzing guesses+paths from a wordlist; crawling observes the paths the application+actually exposes. The two are complementary, and PentAGI currently has+no first-class, scoped, structured crawler capability.++This RFC is a design proposal only. It does not add runtime code, a new+tool handler, a crawler client, GraphQL schema, database migrations,+frontend UI, installer logic, provider configuration, Docker image+changes, Docker Compose changes, `.env.example` entries, or generated+files. It proposes how PentAGI could expose crawler-style URL discovery+as an optional capability so maintainers can review scope, safety, and+the artifact model before any implementation lands.++The proposed direction is a tool-agnostic "crawler backend" (or+"discovery tool") abstraction over candidate tools such as `katana`,+`crawlergo`, `rad`, and `jsfinder`, disabled by default, producing+normalized discovery artifacts (URLs, forms, parameters, JavaScript+endpoints, status codes, source page, depth, and scope decision) that+other PentAGI tools, agents, and reports can consume.++This work is deliberately kept separate from the BrowserOS MCP browser+backend proposed in issue+[#342](https://github.com/vxcontrol/pentagi/issues/342). That proposal+targets stateful, interactive browser control (navigation, click,+fill/type, form submission, login-page handling). This RFC targets+URL/route/form/link discovery and passive or semi-passive site mapping,+not interactive session automation. The two can complement each other,+and JavaScript-heavy crawling is the natural overlap point, but their+contracts are different: a crawler returns a discovery artifact, while+an interactive browser drives a session.++## Goals++- Add a reviewable design for an optional crawler / URL discovery+ capability before any runtime work begins.+- Treat the crawler as a tool-agnostic backend with one or more+ selectable discovery tools, not a single hardcoded CLI.+- Produce normalized, structured discovery artifacts that capture URLs,+ HTTP methods, status codes, forms, parameters, JavaScript endpoints,+ the source page, crawl depth, and the in-scope/out-of-scope decision.+- Let PentAGI agents use crawler output to seed `ffuf`/`dirsearch`-style+ content discovery, guide browser checks, reduce repeated manual+ enumeration, and enrich reports.+- Keep the capability secure by default: disabled unless enabled, scoped+ to the authorized target, rate limited, and bounded by depth, page,+ and request limits.+- Reuse PentAGI's existing URL classification and screenshot/evidence+ paths where practical instead of inventing parallel infrastructure.+- Keep crawler discovery clearly separated from the BrowserOS MCP+ interactive browser backend.++## Non-Goals++- This RFC does not implement crawler runtime code, a crawler tool+ handler, or a crawler client.+- This RFC does not add Docker image changes, Docker Compose services,+ installer behavior, `.env.example` entries, new environment variables,+ GraphQL types, REST endpoints, database tables, generated files,+ provider configuration, or frontend settings.+- This RFC does not replace `ffuf`, `dirsearch`, or any existing content+ discovery workflow. Crawling and dictionary fuzzing are complementary.+- This RFC does not provide stateful interactive browser automation such+ as clicking through UI, typing into fields, or driving multi-step+ login flows. That direction belongs to the BrowserOS MCP browser+ backend in issue #342.+- This RFC does not pick a mandatory default crawler tool. `katana`,+ `crawlergo`, `rad`, and `jsfinder` are framed as candidates.+- This RFC does not propose credentialed crawling or automatic active+ form submission as default behavior.+- This RFC does not choose the final storage shape for crawler+ configuration or artifacts.++## Current Browser and Discovery Behavior++PentAGI already has two relevant building blocks, but neither performs+structured crawling.++The browser tool in `backend/pkg/tools/browser.go` is scraper-backed and+request/response oriented. It operates on a single URL at a time and+supports the `markdown`, `html`, and `links` actions, with an optional+screenshot per call through the configured scraper. `SCRAPER_PUBLIC_URL`+and `SCRAPER_PRIVATE_URL` select public or private scraper routing based+on whether the target resolves to a private or public address, and the+tool already classifies local zones and binary URLs to avoid fetching+non-HTML resources. The `links` action returns the links found on one+fetched page, but the browser tool does not recursively follow links,+build a site map, deduplicate routes, extract forms or parameters,+discover endpoints referenced only from JavaScript, or enforce a crawl+scope or budget. It is single-page extraction, not crawling.++The terminal tool lets the pentester agent run arbitrary command-line+tools inside the isolated pentest container. The pentester prompt's web+testing guidance already lists crawler-adjacent tools (for example+`ffuf`, `gobuster`, `dirsearch`, `feroxbuster`, `httpx`, `katana`,+`hakrawler`, `waybackurls`, and `gau`) as examples the agent may use,+with a reminder to verify availability and install missing tools in the+current image. This means a crawler such as `katana` can in principle be+invoked today, but only as an ad-hoc terminal command. Its output is+unstructured stdout, it is not scoped or budgeted by PentAGI, the+results are not normalized into a reusable artifact, and nothing feeds+the discovered routes back into `ffuf`, the browser tool, or the report.++The gap from issue #336 is therefore not "no tool can crawl" but "there+is no first-class, scoped, structured crawler capability." Discovery+today depends on whichever CLI the agent happens to run, returns+free-form text, and cannot be reliably reused across subtasks.++## Proposed Crawler Capability++A future implementation can introduce a crawler / URL discovery+capability that is conceptually parallel to the existing browser tool:+it wraps a discovery backend, accepts one or more seed URLs plus a+scope, and returns a normalized discovery artifact instead of free-form+text.++Proposed shape of the capability:++- Input: one or more seed URLs, an explicit scope (allowed hosts or+ scope entries), a crawl mode, and budget limits (maximum depth, pages,+ requests, duration, and request rate).+- Crawl mode candidates:+ - `passive`: derive URLs from already-collected sources such as link+ extraction, archived URL feeds, or saved responses, without sending+ new requests to the target beyond what is already authorized.+ - `static`: request and parse HTML and linked resources without a+ full browser engine.+ - `headless`: render pages with a headless browser engine so that+ JavaScript-built routes, single-page-application views, and+ dynamically generated links can be discovered.+- Output: a structured discovery artifact (see Artifacts and Reporting)+ that records each discovered URL with its method, status code, source+ page, depth, parameters, forms, JavaScript endpoints, the backend that+ found it, and the scope decision.+- Agent-facing result: a concise text summary for the agent (counts,+ notable routes, limits reached) plus the structured artifact stored+ through PentAGI's existing artifact and memory paths, mirroring how the+ browser tool returns text while saving screenshots.++The capability is observation-focused. It maps what exists; it does not+log in, submit credentials, or perform stateful multi-step interaction.+Where a target genuinely requires JavaScript rendering or authenticated+interaction to reveal routes, that overlaps with the BrowserOS MCP+backend (issue #342); this RFC treats such cases as a handoff boundary,+not as crawler scope creep.++## Candidate Tools++The tools named in issue #336 cover complementary discovery styles. They+are candidates for a pluggable backend, not mandated defaults.++- `katana` (ProjectDiscovery): fast crawler with both a standard and a+ headless mode, configurable depth and scope, form and endpoint+ awareness, and structured (JSON/JSONL) output. It is already named in+ the pentester prompt's web testing list, which makes it a natural+ first candidate for a structured backend.+- `crawlergo`: a Chromium-based dynamic crawler aimed at+ JavaScript-heavy applications. It exercises the DOM to surface routes+ and requests that static crawling misses and emits a request list that+ can seed further testing.+- `rad`: a Chromium-based crawler focused on browser-driven crawling+ with scope controls, useful where rendering is required to enumerate+ application routes.+- `jsfinder`: an endpoint/URL extractor that mines JavaScript files for+ references to API paths and routes. It is semi-passive and complements+ crawlers by recovering endpoints that are present in scripts but never+ linked from rendered pages.++Adjacent tools already referenced by the pentester prompt, such as+`hakrawler`, `waybackurls`, and `gau`, could be modeled as additional+passive discovery sources behind the same abstraction. The RFC does not+require adopting all of them; it requires that the abstraction not be+hardwired to one CLI.++## Tool Selection Model++The capability should expose a `crawler backend` (equivalently a+`discovery tool`) abstraction so that the agent-facing tool surface is+stable while the underlying CLI can vary.++Proposed selection model:++- Default is off. No crawler backend is selected and no crawl runs+ unless an operator explicitly enables at least one backend.+- Operators register which discovery tools are available and enabled in+ the deployment, since tool availability depends on the pentest image.+- Each backend declares the crawl modes it can serve (`passive`,+ `static`, `headless`) and any tool-specific limits.+- The agent (or a future planner step) selects an enabled backend and+ mode appropriate to the subtask. If a requested mode has no enabled+ backend, the capability reports that clearly instead of silently+ falling back to an unsafe or out-of-scope behavior.+- All backends normalize into the same discovery artifact schema so that+ downstream consumers do not depend on which CLI produced the result.++No single tool is promoted to a required default. `katana` is a+reasonable first candidate because it already appears in PentAGI's tool+guidance and supports structured output, but the RFC frames it as one+option among several, not a mandate.++## Agent Workflow++A structured crawler changes how the pentester agent approaches a web+target. Today the agent fuzzes directories and may run an ad-hoc crawler+whose text output is hard to reuse. With a structured capability, a web+subtask can run discovery once, persist the artifact, and let later+subtasks consume it.++Illustrative workflow:++1. Early in a web engagement, the pentester agent runs crawler discovery+ against the authorized target within scope and budget limits.+2. The normalized artifact is stored through PentAGI's artifact and+ long-term memory paths so it is reusable across subtasks rather than+ re-derived.+3. Discovered directories, file extensions, and parameter names seed+ `ffuf`/`dirsearch`-style content discovery, making dictionary fuzzing+ more targeted instead of purely wordlist-driven.+4. Interesting routes (login, upload, admin, API, parameterized+ endpoints) guide focused browser checks, and, where interactive+ rendering or login is required, hand off to the BrowserOS MCP backend+ (issue #342).+5. Parameterized URLs and JavaScript endpoints inform vulnerability+ testing tools the agent already uses (for example `nuclei` or+ `sqlmap`) by giving them concrete inputs.+6. The final report is enriched with a route/endpoint inventory derived+ from the artifact, reducing repeated manual enumeration and giving+ reviewers a clear map of the attack surface.++This fits PentAGI's existing flow/task/subtask model and result tools+(for example `hack_result` and `report_result`) and its RAG memory,+without introducing hidden lifecycle state. The crawl is an explicit+tool call with a visible, inspectable artifact.++## Artifacts and Reporting++Crawler output should become a normalized artifact rather than raw+stdout. The schema below is illustrative, not a final contract.++Per-entry fields:++- `url`: the discovered URL.+- `method`: HTTP method observed or inferred (for example GET or POST).+- `status_code`: response status when the crawler requested the URL.+- `source_page`: the page the URL was discovered from.+- `depth`: crawl depth at which the URL was found.+- `scope_decision`: whether the URL was treated as in scope or out of+ scope, and why.+- `parameters`: query or body parameter names associated with the URL.+- `forms`: form action, method, and input field names.+- `js_endpoints`: endpoints referenced from JavaScript for this page.+- `discovered_by`: which backend (for example `katana` or `jsfinder`)+ produced the entry.++An illustrative entry:++```json+{+ "url": "https://target.example/admin/login",+ "method": "GET",+ "status_code": 200,+ "source_page": "https://target.example/",+ "depth": 1,+ "scope_decision": "in_scope",+ "parameters": ["redirect", "lang"],+ "forms": [+ {+ "action": "https://target.example/admin/login",+ "method": "POST",+ "inputs": ["username", "password", "csrf_token"]+ }+ ],+ "js_endpoints": ["https://target.example/api/v1/session"],+ "discovered_by": "katana"+}+```++A run-level summary should accompany the entries: seed URLs, scope+configuration, backend and mode used, counts (pages crawled, URLs,+forms, parameters, JavaScript endpoints), which limits were reached, and+how many entries were dropped as out of scope. Artifacts should be+stored through the same flow-scoped artifact, screenshot, and reporting+paths the browser tool already uses where practical. If the evidence+chain proposal in+[evidence_chain.md](evidence_chain.md) is implemented later, a crawl run+and its artifact are a natural source of toolcall receipts.++## Scope and Safety++Crawling reaches more of a target than single-page scraping, so the+capability must be secure by default and explicitly scoped.++Required safety properties:++- Disabled or explicit by default. No crawl runs unless an operator+ enables a backend and the agent invokes it deliberately.+- Obey flow target scope. Crawling must respect the authorized target+ scope of the flow and must not wander to unrelated hosts.+- Depth, page, and request limits. Every crawl is bounded by maximum+ depth, maximum pages, maximum total requests, and maximum duration.+- Rate limiting. Request rate and concurrency are capped to avoid+ hammering the target or tripping protective controls.+- Same-origin and allowed-host controls. Off-origin and off-allowlist+ links are recorded as out of scope by default rather than followed.+- SSRF and private-network protection. The capability should reuse the+ browser tool's URL classification so link-local, loopback, metadata,+ private, and reserved targets are not crawled unless explicitly+ authorized for the engagement.+- `robots.txt` is treated as an operator policy question, not a hard+ rule. Authorized engagements differ on whether to honor `robots.txt`,+ so the policy (honor, ignore, or record only) should be configurable+ and recorded in the artifact, and the default should be conservative.+ This RFC does not mandate either honoring or ignoring it.+- Avoid crawling outside authorized targets. Out-of-scope discovery is+ recorded for awareness but not actively requested.+- No credentialed crawling by default. The crawler should not assume it+ can use stored credentials, cookies, or authenticated sessions.+- Active form submission requires separate approval or policy. Passive+ discovery of forms is in scope; submitting forms (which mutates target+ state) is a separate, gated capability and is not default behavior.++These properties keep the capability aligned with PentAGI's lawful+pentesting posture: it maps the authorized attack surface, it does not+broaden scope, automate accounts, or take state-changing actions without+an explicit decision.++## Configuration Sketch++This sketch is illustrative only. It is not a proposed `.env.example`+change and does not choose the final storage shape.++```yaml+crawler:+ enabled: false # off by default; operator opt-in+ default_backend: none # candidate values: none, katana, crawlergo, rad+ backends:+ katana:+ enabled: false+ modes: [static, headless] # advertised crawl modes+ crawlergo:+ enabled: false+ modes: [headless]+ rad:+ enabled: false+ modes: [headless]+ jsfinder:+ enabled: false+ modes: [passive] # JavaScript endpoint extraction+ scope:+ follow: same_origin # same_origin | allowlist | scope_entries+ allowed_hosts:+ - target.example+ robots_policy: record_only # honor | ignore | record_only+ limits:+ max_depth: 3+ max_pages: 500+ max_requests: 2000+ max_duration_seconds: 600+ requests_per_second: 5+ max_concurrency: 5+ credentials:+ allow_credentialed_crawl: false # no authenticated crawling by default+ active:+ allow_form_submission: false # passive discovery only by default+ require_approval_for_submission: true+```++Implementation notes for a future PR:++- The crawler backend should be selectable so the agent-facing tool+ surface stays stable while the underlying CLI can change.+- Tool identifiers and flags here are illustrative. A real+ implementation must validate the canonical tool invocation and output+ format for each backend before normalizing it.+- Scope and SSRF checks should integrate with PentAGI's existing URL+ classification rather than re-implementing target analysis.+- The final storage mechanism (mounted YAML, database-backed settings,+ existing tool configuration, or another maintainer-approved shape) is+ intentionally left open.++## Failure Modes++A crawler capability should degrade safely and visibly.++Expected failure modes:++- The selected backend is not installed or not available in the current+ pentest image.+- The target is unreachable, blocks the crawler, or rate-limits it.+- A JavaScript-heavy site yields little in `static` mode and needs a+ `headless` backend that is not enabled.+- The crawl hits a trap such as an infinite calendar, faceted search, or+ session-id-in-URL pattern and approaches its budget without progress.+- A link points outside the authorized scope.+- The backend emits a very large result set beyond configured limits.+- The backend returns malformed or unexpected output that cannot be+ normalized.++Recommended behavior:++- Return a clear discovery tool error or partial result to the agent and+ continue the agent loop when possible.+- Record the backend, mode, target, scope decision, and which limit was+ reached.+- Fail closed for scope, allowlist, SSRF, and approval decisions rather+ than retrying with a broader behavior.+- Prefer returning a bounded partial artifact over hanging or crawling+ past configured limits.+- Do not retry in a tight loop; rely on bounded budgets and, if retry is+ added later, on backoff.++## Open Questions++- Should the crawler be exposed as a new first-class tool, as an+ extension of the existing browser tool, or wrapped around terminal+ execution of the chosen CLI?+- Which backend(s) should be vendored into the pentest image by default,+ given that tool availability drives what the agent can actually run?+- Should `headless` crawling reuse a browser engine shared with the+ BrowserOS MCP backend (issue #342), or run as a standalone crawler+ process?+- Where should crawler configuration and artifacts live so operators can+ inspect scope and results without leaking target details?+- How should the discovery artifact integrate with the report,+ long-term memory, and the evidence chain proposal?+- What default rate limits, depth, and page budgets are safe for+ authorized engagements without being so low that discovery is useless?+- Should semi-passive sources (`waybackurls`, `gau`, `hakrawler`) be+ modeled behind the same abstraction or kept as separate passive tools?+- What is the right approval model and UX for the optional active+ form-submission capability?+- How should `robots.txt` policy default be chosen, and should it be set+ per deployment, per flow, or per crawl?++## Incremental Milestones++1. Docs-only RFC.+ - Land this proposal so maintainers can review scope, safety, the+ backend abstraction, and the artifact model before runtime work+ begins.++2. Artifact schema and single-backend prototype.+ - Define the normalized discovery artifact and run summary.+ - Wire one backend (for example `katana` in `static` mode) behind the+ abstraction, scoped and budget-limited, without exposing it to+ agents yet.++3. Scoped discovery tool for the pentester agent.+ - Expose crawler discovery as an explicit, disabled-by-default tool+ with scope, depth, page, request, rate, and duration limits.+ - Persist the artifact through existing artifact and memory paths.++4. Downstream integration.+ - Use the artifact to seed `ffuf`/`dirsearch`, guide browser checks,+ inform parameterized vulnerability testing, and enrich reports.++5. Optional headless and semi-passive backends.+ - Add `headless` backends (for example `crawlergo` or `rad`) and+ JavaScript endpoint extraction (`jsfinder`) for JavaScript-heavy+ targets.+ - Keep `robots.txt` policy explicit, keep credentialed crawling and+ active form submission off by default, and gate any state-changing+ action behind approval.++Refs #336