Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai
Python agentsagentskillscitationsclaude-codecodexcodex-plugincodex-skilldeep-researchdeep-research-agentdeepresearchmcp-serverresearchresearch-agentresearch-tools
fix(security): SSRF gate refuses URLs a browser parses to a different host (#182)
Jordan Gibbsd98ae1de73e7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cda7a1a..d3d663d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,7 @@- **The pipeline runs on the OpenAI Codex CLI.** `hyperresearch install . --target codex` installs the entry skill at `.agents/skills/hyperresearch/`, invoked as `$hyperresearch <query>`, along with the 18 step procedures as plain files under `.hyperresearch/codex/steps/`, the subagent roster as custom agents in `.codex/agents/*.toml`, a Stop hook in `.codex/hooks.json`, and an `AGENTS.md` block. The step procedures are plain files because Codex caps the skill listing and has no skill-to-skill invocation. `--target all` installs both platforms. `--global --target codex` writes the skill to `~/.agents/skills/` and the agents to `~/.codex/agents/`. `--target claude` is the default and its output is byte-identical to 0.12.0. Differences on Codex: there is no browser lane, so blocked fetches stay queued and are listed for the user; the Read+Edit tool locks are instructions rather than enforcement; and the Stop hook (`hyperresearch run stop-gate`) blocks the session from ending mid-pipeline. The Stop hook exists because an earlier Codex attempt answered inline and never touched the vault. Codex's sandbox defaults to read-only with no network, so a session needs `--sandbox workspace-write -c sandbox_workspace_write.network_access=true`. hyperresearch documents these flags and never writes to Codex's config.- **The repo is a Codex plugin marketplace too.** `.codex-plugin/plugin.json` and `.agents/plugins/marketplace.json` publish the same `deep-research` bootstrap skill as the Claude Code plugin: `codex plugin marketplace add jordan-gibbs/hyperresearch`, then `codex plugin add hyperresearch@hyperresearch`. The skill no longer assumes Claude Code. On Codex it runs `install --target codex` and then reads the entry skill directly, because Codex discovers skills when a session starts. It gains `license` and `compatibility` frontmatter and an `agents/openai.yaml`. Codex namespaces plugin skills, so from the plugin it is `$hyperresearch:deep-research`; installed standalone with `npx skills add jordan-gibbs/hyperresearch -a codex` (to `.agents/skills/deep-research/`) it is `$deep-research`. The `openai.yaml` starter prompt names the skill in words rather than by a `$` mention that would only match one of those. The README now leads with Claude Code and Codex install paths side by side.- **`run resume` follows the tier step 1 declared.** A run is initialized with the installed gear's step list, and step 1 can reclassify it (usually to `light`) in `prompt-decomposition.json`. The ship gate already honoured that tier; `resume_position` did not, so a light-tier run that finished step 2 was told to resume at step 3, a step its tier skips. `run resume`, `run status` and the Codex Stop gate now use the declared tier's steps. Found during the first live Codex run.+- **The SSRF gate refuses URLs a browser reads differently.** `check_url` learns the host from urlparse and httpx (RFC 3986), but the crawl4ai lane hands the URL to Chromium (WHATWG). For `http://127.0.0.1\@evil.com/` the gate checked `evil.com` while Chromium, which reads `\` as `/`, connected to 127.0.0.1; the final-URL recheck only ran after the request was sent. The gate now refuses a backslash in the authority, a percent-encoded hostname (`127.0.0.%31`, which Chromium decodes), and a host whose last label is numeric unless it is a canonical dotted quad (`127.1`, `0x7f.1`, `2130706433`: Chromium parses these as IPv4 in its own grammar, getaddrinfo varies by platform). Tabs, newlines and other control characters inside the URL were already refused by the httpx parse. A backslash in the path, query or fragment is still allowed: it cannot move the host. The check is in `check_url`, so every lane gets it.- **Codex bench harness rebuilt (`codex_bench/`).** Each query runs in its own git-initialized project set up by `install --target codex`, with the prompt in `research/prompt.txt` and the inline-citation wrapper contract. The session is one `codex exec` under the workspace-write-plus-network sandbox, with JSONL events captured. `result.json` records whether the pipeline ran: vault note count, manifest steps done and not done, tier, subagent spawns, sandbox rejections, and `run verify`. An inline answer that skipped the vault is reported as `PIPELINE_SKIPPED`. `--dry-run` sets up the run dir and prints the exact `codex exec` command without running it. There are no dollar fields.- **Run spend counters and count ceilings (#92).** `hpr run step --status done` and `hpr run reconcile` count tagged notes and distinct fetched URLs linked to those notes in the vault DB, including HTML and PDFs. Optional `hpr run init --max-sources` / `--max-notes` ceilings block at step boundaries. `run resume` or `run step --status done --force` can clear a count block up to three times; the same count stays clear until it increases, and a fourth clear rejects the step. Agent-reported `estimated_usd` stays unchanged; `--budget` stays inert unless something calls `hpr run spend`. Real runs get count ceilings only when the entry skill passes them to `run init`.- **The MCP `fetch_url` tool no longer blocks the server during a fetch (#118, reported by @AmirF194).** It was a sync tool, so FastMCP ran it on the event loop's thread, and the crawl4ai provider joined a helper thread there for the whole fetch; the server answered nothing else until it returned. `fetch_url` is now async and runs only the provider's `fetch` in a worker thread. The duplicate check and the note and `sources` writes stay on the loop thread, because the vault's sqlite connection only works on the thread that opened it. `fetch_and_save` is unchanged for its callers; it is now `prepare_fetch` -> `prov.fetch` -> `save_fetched`, which the tool calls in pieces. `fetch_many` is not an MCP tool and only the CLI batch path uses it.diff --git a/src/hyperresearch/web/safe_http.py b/src/hyperresearch/web/safe_http.pyindex e104e5b..b90951d 100644--- a/src/hyperresearch/web/safe_http.py+++ b/src/hyperresearch/web/safe_http.py@@ -27,6 +27,7 @@ Returns a small :class:`SafeResponse` dataclass instead of anfrom __future__ import annotationsimport ipaddress+import reimport socketfrom dataclasses import dataclass, fieldfrom urllib.parse import urlparse@@ -224,6 +225,66 @@ def _parse_allowlist(# _WHATWG_C0_CONTROL_OR_SPACE, a private name)._C0_CONTROL_OR_SPACE = "".join(chr(c) for c in range(0x20)) + " "+# A host label the WHATWG URL parser reads as an IPv4 number: all decimal+# digits, or 0x/0X plus hex digits (the "ends in a number" check).+_WHATWG_NUMERIC_LABEL_RE = re.compile(r"[0-9]+|0[xX][0-9a-fA-F]*")+++def _check_parser_agreement(stripped_url: str, hostname: str) -> None:+ """Refuse URLs whose host a WHATWG parser (Chromium) reads differently.++ The gate learns the host from urlparse and httpx, which follow RFC 3986.+ The crawl4ai lane hands the URL to Chromium, which follows the WHATWG URL+ standard, so any spelling the two families disagree on lets the gate+ check one host while the browser connects to another. Such URLs are+ refused outright; a legitimate page never needs them.++ ``stripped_url`` is the URL with leading C0 controls and spaces removed;+ ``hostname`` is httpx's raw host for it.+ """+ # For http(s) WHATWG treats "\" exactly like "/": it ends the authority.+ # RFC 3986 parsers keep it as an ordinary character, so for+ # "http://127.0.0.1\@evil.com/" they report host evil.com while Chromium+ # connects to 127.0.0.1. Scan the raw authority as the RFC parsers delimit+ # it (from after "//" to the first "/", "?" or "#"); a backslash there is+ # a disagreement. One after that point is in the path, query or fragment+ # under both readings and cannot move the host, so it stays allowed.+ rest = stripped_url.partition(":")[2]+ if rest.startswith("//"):+ authority = re.split(r"[/?#]", rest[2:], maxsplit=1)[0]+ if "\\" in authority:+ raise SafeHTTPError(+ f"refusing URL {stripped_url!r}: backslash in the authority "+ "(browsers read it as '/', which changes the host)"+ )+ if ":" in hostname:+ # IPv6 literal. WHATWG parses it as an address with no percent+ # decoding, and the address gate classifies the same literal.+ return+ # WHATWG percent-decodes the host ("127.0.0.%31" is 127.0.0.1 to+ # Chromium); httpx and getaddrinfo take "%" literally.+ if "%" in hostname:+ raise SafeHTTPError(+ f"refusing URL {stripped_url!r}: percent-encoded hostname {hostname!r} "+ "(browsers decode it to a different host)"+ )+ # WHATWG parses a host whose last label is numeric as an IPv4 address in+ # its own grammar (octal, hex, fewer than four parts: "0x7f.1",+ # "2130706433"), while getaddrinfo's treatment of those forms varies by+ # platform and can fall through to DNS. Only the canonical dotted quad,+ # which every parser reads the same way, is let through.+ labels = hostname.split(".")+ if len(labels) > 1 and labels[-1] == "":+ labels.pop()+ if _WHATWG_NUMERIC_LABEL_RE.fullmatch(labels[-1]):+ try:+ ipaddress.IPv4Address(hostname)+ except ValueError:+ raise SafeHTTPError(+ f"refusing URL {stripped_url!r}: non-canonical numeric host "+ f"{hostname!r} (browsers parse it as an IPv4 address)"+ ) from None+def check_url(url: str, allow_private_hosts: tuple[str, ...] = ()) -> None:"""Validate the URL is safe to fetch. Raises :class:`SafeHTTPError`.@@ -261,15 +322,19 @@ def check_url(url: str, allow_private_hosts: tuple[str, ...] = ()) -> None:# reads " http://host/" as a relative URL with no host. Strip the same# characters so both parsers see one host, and refuse a URL httpx still# finds no host in rather than look up an empty name.+ stripped = url.lstrip(_C0_CONTROL_OR_SPACE)try:- hostname = httpx.URL(url.lstrip(_C0_CONTROL_OR_SPACE)).raw_host.decode("ascii")+ hostname = httpx.URL(stripped).raw_host.decode("ascii")except (httpx.InvalidURL, UnicodeError) as exc:# UnicodeError: httpx cannot encode a lone surrogate in the path,# query or fragment, which is what a non-UTF-8 byte in a command-line- # argument becomes.+ # argument becomes. InvalidURL also covers tabs, newlines and other+ # control characters anywhere in the URL, which WHATWG would strip+ # or percent-encode.raise SafeHTTPError(f"refusing unparseable URL {url!r}: {exc}") from excif not hostname:raise SafeHTTPError(f"refusing URL with no hostname: {url!r}")+ _check_parser_agreement(stripped, hostname)allowed_hosts, allowed_nets = _parse_allowlist(allow_private_hosts)if _normalize_hostname(hostname) in allowed_hosts:returndiff --git a/tests/test_web/test_safe_http.py b/tests/test_web/test_safe_http.pyindex 488fbd9..57b4fb3 100644--- a/tests/test_web/test_safe_http.py+++ b/tests/test_web/test_safe_http.py@@ -136,18 +136,134 @@ def test_check_url_rejects_parse_only_loopback_bypasses(url):],)def test_check_url_rejects_resolver_dependent_loopback_bypasses(url):- """`ipaddress.ip_address` rejects these numeric encodings, so they fall to- the resolver, which normalizes them to 127.0.0.1. getaddrinfo is stubbed- so the test is deterministic and offline. This pins the contract that such- forms are *resolved and gated*, not parsed as public — a regression that- made `_resolve` parse the literal itself (e.g. via inet_aton) would need to- keep blocking them."""+ """`ipaddress.ip_address` rejects these encodings. The numeric ones are+ refused before any lookup (a browser parses them as IPv4 in its own+ grammar); the rest fall to the resolver, which normalizes them to+ 127.0.0.1. getaddrinfo is stubbed so the test is deterministic and+ offline. Either way they are never parsed as public."""with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:gai.return_value = [(socket.AF_INET, None, None, "", ("127.0.0.1", 0))]- with pytest.raises(SafeHTTPError, match="non-public address"):+ with pytest.raises(SafeHTTPError, match=r"non-public address|numeric host"):check_url(url)+# ---------------------------------------------------------------------------+# check_url — RFC 3986 (urlparse, httpx) vs WHATWG (Chromium) disagreements.+# The crawl4ai lane hands the URL to Chromium, so a spelling the parsers+# disagree on lets the gate approve one host while the browser connects to+# another. getaddrinfo is stubbed to a public address: if the gate fell+# through to resolution, these would be allowed.+# ---------------------------------------------------------------------------+++def public_dns():+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:+ gai.return_value = [(socket.AF_INET, None, None, "", ("93.184.216.34", 0))]+ yield gai+++ "url",+ [+ "http://127.0.0.1\\@evil.com/", # urlparse/httpx: evil.com; Chromium: 127.0.0.1+ "HTTPS://127.0.0.1\\@evil.com/",+ "http://127.0.0.1:80\\@evil.com/",+ "http://[email protected]\\@evil.com/",+ "http://169.254.169.254\\@evil.com/latest/meta-data/",+ "http://127.0.0.1\\@evil.com", # no path at all+ " http://127.0.0.1\\@evil.com/", # after the leading-whitespace strip+ ],+)+def test_check_url_refuses_backslash_in_authority(url, public_dns):+ with pytest.raises(SafeHTTPError, match="backslash in the authority"):+ check_url(url)+++def test_check_url_backslash_then_private_userinfo_host_still_refused():+ """The mirror image: the RFC parsers see 127.0.0.1, Chromium sees+ evil.com. Fetching it would be harmless in the browser, but the gate+ must not let it through on any lane."""+ with pytest.raises(SafeHTTPError):+ check_url("http://evil.com\\@127.0.0.1/")+++ "url",+ [+ "https://example.com/path\\with\\backslash",+ "https://example.com/a?q=C:\\dir\\file",+ "https://example.com/#frag\\x",+ "https://example.com?\\@127.0.0.1",+ ],+)+def test_check_url_allows_backslash_after_the_authority(url, public_dns):+ """Past the first '/', '?' or '#', every parser is in the path, query or+ fragment. Chromium turns a path backslash into '/', which changes the+ path but cannot move the host, so these stay fetchable."""+ check_url(url)+++ "url",+ ["http:\\\\127.0.0.1/", "http:/\\127.0.0.1/", "http:/127.0.0.1/", "http:127.0.0.1/"],+)+def test_check_url_refuses_whatwg_only_authority_forms(url):+ """WHATWG finds a host in all of these (it skips any mix of slashes and+ backslashes after a special scheme); the RFC parsers find none."""+ with pytest.raises(SafeHTTPError, match="no hostname"):+ check_url(url)+++ "url",+ [+ "http://127.0.0.1\[email protected]/", # WHATWG strips tab/newline anywhere+ "http://127.0.0.1\[email protected]/",+ "http://127.0.0.1\r\\@evil.com/",+ "http://127.0.0.1\[email protected]/",+ ],+)+def test_check_url_refuses_control_characters_inside_the_url(url, public_dns):+ with pytest.raises(SafeHTTPError, match="unparseable"):+ check_url(url)+++ "url", ["http://127.0.0.%31/", "http://%31%32%37.0.0.1/", "http://%6cocalhost/"]+)+def test_check_url_refuses_percent_encoded_host(url, public_dns):+ """Chromium percent-decodes the host; httpx and getaddrinfo do not."""+ with pytest.raises(SafeHTTPError, match="percent-encoded hostname"):+ check_url(url)+++ "url",+ [+ "http://127.1/",+ "http://0x7f.1/",+ "http://0177.0.0.1/",+ "http://127.0.0.1./",+ "http://example.0x10/",+ "http://1.2.3.4.5/",+ ],+)+def test_check_url_refuses_non_canonical_numeric_host(url, public_dns):+ """WHATWG reads a host whose last label is numeric as IPv4 in its own+ grammar; getaddrinfo's reading varies by platform and may fall to DNS.+ httpx itself rejects some of these ("Invalid IPv4")."""+ with pytest.raises(SafeHTTPError, match=r"numeric host|Invalid IPv4"):+ check_url(url)+++ "url", ["http://8.8.8.8/", "http://example.com/", "http://a1.example.com/", "http://x.com123/"]+)+def test_check_url_numeric_rule_spares_ordinary_hosts(url, public_dns):+ check_url(url)++def test_check_url_uppercase_http_scheme_is_normalized_then_ip_gated():"""An uppercase HTTP scheme must not fail open: it normalizes to http andthe loopback host is still refused by the address gate, not waved through