Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

fix(security): SSRF gate refuses URLs a browser parses to a different host (#182)

Jordan Gibbsd98ae1de73e7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index cda7a1a..d3d663d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@
- **The pipeline runs on the OpenAI Codex CLI.** `hyperresearch install . --target codex` installs the entry skill at `.agents/skills/hyperresearch/`, invoked as `$hyperresearch <query>`, along with the 18 step procedures as plain files under `.hyperresearch/codex/steps/`, the subagent roster as custom agents in `.codex/agents/*.toml`, a Stop hook in `.codex/hooks.json`, and an `AGENTS.md` block. The step procedures are plain files because Codex caps the skill listing and has no skill-to-skill invocation. `--target all` installs both platforms. `--global --target codex` writes the skill to `~/.agents/skills/` and the agents to `~/.codex/agents/`. `--target claude` is the default and its output is byte-identical to 0.12.0. Differences on Codex: there is no browser lane, so blocked fetches stay queued and are listed for the user; the Read+Edit tool locks are instructions rather than enforcement; and the Stop hook (`hyperresearch run stop-gate`) blocks the session from ending mid-pipeline. The Stop hook exists because an earlier Codex attempt answered inline and never touched the vault. Codex's sandbox defaults to read-only with no network, so a session needs `--sandbox workspace-write -c sandbox_workspace_write.network_access=true`. hyperresearch documents these flags and never writes to Codex's config.
- **The repo is a Codex plugin marketplace too.** `.codex-plugin/plugin.json` and `.agents/plugins/marketplace.json` publish the same `deep-research` bootstrap skill as the Claude Code plugin: `codex plugin marketplace add jordan-gibbs/hyperresearch`, then `codex plugin add hyperresearch@hyperresearch`. The skill no longer assumes Claude Code. On Codex it runs `install --target codex` and then reads the entry skill directly, because Codex discovers skills when a session starts. It gains `license` and `compatibility` frontmatter and an `agents/openai.yaml`. Codex namespaces plugin skills, so from the plugin it is `$hyperresearch:deep-research`; installed standalone with `npx skills add jordan-gibbs/hyperresearch -a codex` (to `.agents/skills/deep-research/`) it is `$deep-research`. The `openai.yaml` starter prompt names the skill in words rather than by a `$` mention that would only match one of those. The README now leads with Claude Code and Codex install paths side by side.
- **`run resume` follows the tier step 1 declared.** A run is initialized with the installed gear's step list, and step 1 can reclassify it (usually to `light`) in `prompt-decomposition.json`. The ship gate already honoured that tier; `resume_position` did not, so a light-tier run that finished step 2 was told to resume at step 3, a step its tier skips. `run resume`, `run status` and the Codex Stop gate now use the declared tier's steps. Found during the first live Codex run.
+- **The SSRF gate refuses URLs a browser reads differently.** `check_url` learns the host from urlparse and httpx (RFC 3986), but the crawl4ai lane hands the URL to Chromium (WHATWG). For `http://127.0.0.1\@evil.com/` the gate checked `evil.com` while Chromium, which reads `\` as `/`, connected to 127.0.0.1; the final-URL recheck only ran after the request was sent. The gate now refuses a backslash in the authority, a percent-encoded hostname (`127.0.0.%31`, which Chromium decodes), and a host whose last label is numeric unless it is a canonical dotted quad (`127.1`, `0x7f.1`, `2130706433`: Chromium parses these as IPv4 in its own grammar, getaddrinfo varies by platform). Tabs, newlines and other control characters inside the URL were already refused by the httpx parse. A backslash in the path, query or fragment is still allowed: it cannot move the host. The check is in `check_url`, so every lane gets it.
- **Codex bench harness rebuilt (`codex_bench/`).** Each query runs in its own git-initialized project set up by `install --target codex`, with the prompt in `research/prompt.txt` and the inline-citation wrapper contract. The session is one `codex exec` under the workspace-write-plus-network sandbox, with JSONL events captured. `result.json` records whether the pipeline ran: vault note count, manifest steps done and not done, tier, subagent spawns, sandbox rejections, and `run verify`. An inline answer that skipped the vault is reported as `PIPELINE_SKIPPED`. `--dry-run` sets up the run dir and prints the exact `codex exec` command without running it. There are no dollar fields.
- **Run spend counters and count ceilings (#92).** `hpr run step --status done` and `hpr run reconcile` count tagged notes and distinct fetched URLs linked to those notes in the vault DB, including HTML and PDFs. Optional `hpr run init --max-sources` / `--max-notes` ceilings block at step boundaries. `run resume` or `run step --status done --force` can clear a count block up to three times; the same count stays clear until it increases, and a fourth clear rejects the step. Agent-reported `estimated_usd` stays unchanged; `--budget` stays inert unless something calls `hpr run spend`. Real runs get count ceilings only when the entry skill passes them to `run init`.
- **The MCP `fetch_url` tool no longer blocks the server during a fetch (#118, reported by @AmirF194).** It was a sync tool, so FastMCP ran it on the event loop's thread, and the crawl4ai provider joined a helper thread there for the whole fetch; the server answered nothing else until it returned. `fetch_url` is now async and runs only the provider's `fetch` in a worker thread. The duplicate check and the note and `sources` writes stay on the loop thread, because the vault's sqlite connection only works on the thread that opened it. `fetch_and_save` is unchanged for its callers; it is now `prepare_fetch` -> `prov.fetch` -> `save_fetched`, which the tool calls in pieces. `fetch_many` is not an MCP tool and only the CLI batch path uses it.
diff --git a/src/hyperresearch/web/safe_http.py b/src/hyperresearch/web/safe_http.py
index e104e5b..b90951d 100644
--- a/src/hyperresearch/web/safe_http.py
+++ b/src/hyperresearch/web/safe_http.py
@@ -27,6 +27,7 @@ Returns a small :class:`SafeResponse` dataclass instead of an
from __future__ import annotations
import ipaddress
+import re
import socket
from dataclasses import dataclass, field
from urllib.parse import urlparse
@@ -224,6 +225,66 @@ def _parse_allowlist(
# _WHATWG_C0_CONTROL_OR_SPACE, a private name).
_C0_CONTROL_OR_SPACE = "".join(chr(c) for c in range(0x20)) + " "
+# A host label the WHATWG URL parser reads as an IPv4 number: all decimal
+# digits, or 0x/0X plus hex digits (the "ends in a number" check).
+_WHATWG_NUMERIC_LABEL_RE = re.compile(r"[0-9]+|0[xX][0-9a-fA-F]*")
+
+
+def _check_parser_agreement(stripped_url: str, hostname: str) -> None:
+ """Refuse URLs whose host a WHATWG parser (Chromium) reads differently.
+
+ The gate learns the host from urlparse and httpx, which follow RFC 3986.
+ The crawl4ai lane hands the URL to Chromium, which follows the WHATWG URL
+ standard, so any spelling the two families disagree on lets the gate
+ check one host while the browser connects to another. Such URLs are
+ refused outright; a legitimate page never needs them.
+
+ ``stripped_url`` is the URL with leading C0 controls and spaces removed;
+ ``hostname`` is httpx's raw host for it.
+ """
+ # For http(s) WHATWG treats "\" exactly like "/": it ends the authority.
+ # RFC 3986 parsers keep it as an ordinary character, so for
+ # "http://127.0.0.1\@evil.com/" they report host evil.com while Chromium
+ # connects to 127.0.0.1. Scan the raw authority as the RFC parsers delimit
+ # it (from after "//" to the first "/", "?" or "#"); a backslash there is
+ # a disagreement. One after that point is in the path, query or fragment
+ # under both readings and cannot move the host, so it stays allowed.
+ rest = stripped_url.partition(":")[2]
+ if rest.startswith("//"):
+ authority = re.split(r"[/?#]", rest[2:], maxsplit=1)[0]
+ if "\\" in authority:
+ raise SafeHTTPError(
+ f"refusing URL {stripped_url!r}: backslash in the authority "
+ "(browsers read it as '/', which changes the host)"
+ )
+ if ":" in hostname:
+ # IPv6 literal. WHATWG parses it as an address with no percent
+ # decoding, and the address gate classifies the same literal.
+ return
+ # WHATWG percent-decodes the host ("127.0.0.%31" is 127.0.0.1 to
+ # Chromium); httpx and getaddrinfo take "%" literally.
+ if "%" in hostname:
+ raise SafeHTTPError(
+ f"refusing URL {stripped_url!r}: percent-encoded hostname {hostname!r} "
+ "(browsers decode it to a different host)"
+ )
+ # WHATWG parses a host whose last label is numeric as an IPv4 address in
+ # its own grammar (octal, hex, fewer than four parts: "0x7f.1",
+ # "2130706433"), while getaddrinfo's treatment of those forms varies by
+ # platform and can fall through to DNS. Only the canonical dotted quad,
+ # which every parser reads the same way, is let through.
+ labels = hostname.split(".")
+ if len(labels) > 1 and labels[-1] == "":
+ labels.pop()
+ if _WHATWG_NUMERIC_LABEL_RE.fullmatch(labels[-1]):
+ try:
+ ipaddress.IPv4Address(hostname)
+ except ValueError:
+ raise SafeHTTPError(
+ f"refusing URL {stripped_url!r}: non-canonical numeric host "
+ f"{hostname!r} (browsers parse it as an IPv4 address)"
+ ) from None
+
def check_url(url: str, allow_private_hosts: tuple[str, ...] = ()) -> None:
"""Validate the URL is safe to fetch. Raises :class:`SafeHTTPError`.
@@ -261,15 +322,19 @@ def check_url(url: str, allow_private_hosts: tuple[str, ...] = ()) -> None:
# reads " http://host/" as a relative URL with no host. Strip the same
# characters so both parsers see one host, and refuse a URL httpx still
# finds no host in rather than look up an empty name.
+ stripped = url.lstrip(_C0_CONTROL_OR_SPACE)
try:
- hostname = httpx.URL(url.lstrip(_C0_CONTROL_OR_SPACE)).raw_host.decode("ascii")
+ hostname = httpx.URL(stripped).raw_host.decode("ascii")
except (httpx.InvalidURL, UnicodeError) as exc:
# UnicodeError: httpx cannot encode a lone surrogate in the path,
# query or fragment, which is what a non-UTF-8 byte in a command-line
- # argument becomes.
+ # argument becomes. InvalidURL also covers tabs, newlines and other
+ # control characters anywhere in the URL, which WHATWG would strip
+ # or percent-encode.
raise SafeHTTPError(f"refusing unparseable URL {url!r}: {exc}") from exc
if not hostname:
raise SafeHTTPError(f"refusing URL with no hostname: {url!r}")
+ _check_parser_agreement(stripped, hostname)
allowed_hosts, allowed_nets = _parse_allowlist(allow_private_hosts)
if _normalize_hostname(hostname) in allowed_hosts:
return
diff --git a/tests/test_web/test_safe_http.py b/tests/test_web/test_safe_http.py
index 488fbd9..57b4fb3 100644
--- a/tests/test_web/test_safe_http.py
+++ b/tests/test_web/test_safe_http.py
@@ -136,18 +136,134 @@ def test_check_url_rejects_parse_only_loopback_bypasses(url):
],
)
def test_check_url_rejects_resolver_dependent_loopback_bypasses(url):
- """`ipaddress.ip_address` rejects these numeric encodings, so they fall to
- the resolver, which normalizes them to 127.0.0.1. getaddrinfo is stubbed
- so the test is deterministic and offline. This pins the contract that such
- forms are *resolved and gated*, not parsed as public — a regression that
- made `_resolve` parse the literal itself (e.g. via inet_aton) would need to
- keep blocking them."""
+ """`ipaddress.ip_address` rejects these encodings. The numeric ones are
+ refused before any lookup (a browser parses them as IPv4 in its own
+ grammar); the rest fall to the resolver, which normalizes them to
+ 127.0.0.1. getaddrinfo is stubbed so the test is deterministic and
+ offline. Either way they are never parsed as public."""
with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
gai.return_value = [(socket.AF_INET, None, None, "", ("127.0.0.1", 0))]
- with pytest.raises(SafeHTTPError, match="non-public address"):
+ with pytest.raises(SafeHTTPError, match=r"non-public address|numeric host"):
check_url(url)
+# ---------------------------------------------------------------------------
+# check_url — RFC 3986 (urlparse, httpx) vs WHATWG (Chromium) disagreements.
+# The crawl4ai lane hands the URL to Chromium, so a spelling the parsers
+# disagree on lets the gate approve one host while the browser connects to
+# another. getaddrinfo is stubbed to a public address: if the gate fell
+# through to resolution, these would be allowed.
+# ---------------------------------------------------------------------------
+
+
+def public_dns():
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("93.184.216.34", 0))]
+ yield gai
+
+
+ "url",
+ [
+ "http://127.0.0.1\\@evil.com/", # urlparse/httpx: evil.com; Chromium: 127.0.0.1
+ "HTTPS://127.0.0.1\\@evil.com/",
+ "http://127.0.0.1:80\\@evil.com/",
+ "http://[email protected]\\@evil.com/",
+ "http://169.254.169.254\\@evil.com/latest/meta-data/",
+ "http://127.0.0.1\\@evil.com", # no path at all
+ " http://127.0.0.1\\@evil.com/", # after the leading-whitespace strip
+ ],
+)
+def test_check_url_refuses_backslash_in_authority(url, public_dns):
+ with pytest.raises(SafeHTTPError, match="backslash in the authority"):
+ check_url(url)
+
+
+def test_check_url_backslash_then_private_userinfo_host_still_refused():
+ """The mirror image: the RFC parsers see 127.0.0.1, Chromium sees
+ evil.com. Fetching it would be harmless in the browser, but the gate
+ must not let it through on any lane."""
+ with pytest.raises(SafeHTTPError):
+ check_url("http://evil.com\\@127.0.0.1/")
+
+
+ "url",
+ [
+ "https://example.com/path\\with\\backslash",
+ "https://example.com/a?q=C:\\dir\\file",
+ "https://example.com/#frag\\x",
+ "https://example.com?\\@127.0.0.1",
+ ],
+)
+def test_check_url_allows_backslash_after_the_authority(url, public_dns):
+ """Past the first '/', '?' or '#', every parser is in the path, query or
+ fragment. Chromium turns a path backslash into '/', which changes the
+ path but cannot move the host, so these stay fetchable."""
+ check_url(url)
+
+
+ "url",
+ ["http:\\\\127.0.0.1/", "http:/\\127.0.0.1/", "http:/127.0.0.1/", "http:127.0.0.1/"],
+)
+def test_check_url_refuses_whatwg_only_authority_forms(url):
+ """WHATWG finds a host in all of these (it skips any mix of slashes and
+ backslashes after a special scheme); the RFC parsers find none."""
+ with pytest.raises(SafeHTTPError, match="no hostname"):
+ check_url(url)
+
+
+ "url",
+ [
+ "http://127.0.0.1\[email protected]/", # WHATWG strips tab/newline anywhere
+ "http://127.0.0.1\[email protected]/",
+ "http://127.0.0.1\r\\@evil.com/",
+ "http://127.0.0.1\[email protected]/",
+ ],
+)
+def test_check_url_refuses_control_characters_inside_the_url(url, public_dns):
+ with pytest.raises(SafeHTTPError, match="unparseable"):
+ check_url(url)
+
+
+ "url", ["http://127.0.0.%31/", "http://%31%32%37.0.0.1/", "http://%6cocalhost/"]
+)
+def test_check_url_refuses_percent_encoded_host(url, public_dns):
+ """Chromium percent-decodes the host; httpx and getaddrinfo do not."""
+ with pytest.raises(SafeHTTPError, match="percent-encoded hostname"):
+ check_url(url)
+
+
+ "url",
+ [
+ "http://127.1/",
+ "http://0x7f.1/",
+ "http://0177.0.0.1/",
+ "http://127.0.0.1./",
+ "http://example.0x10/",
+ "http://1.2.3.4.5/",
+ ],
+)
+def test_check_url_refuses_non_canonical_numeric_host(url, public_dns):
+ """WHATWG reads a host whose last label is numeric as IPv4 in its own
+ grammar; getaddrinfo's reading varies by platform and may fall to DNS.
+ httpx itself rejects some of these ("Invalid IPv4")."""
+ with pytest.raises(SafeHTTPError, match=r"numeric host|Invalid IPv4"):
+ check_url(url)
+
+
+ "url", ["http://8.8.8.8/", "http://example.com/", "http://a1.example.com/", "http://x.com123/"]
+)
+def test_check_url_numeric_rule_spares_ordinary_hosts(url, public_dns):
+ check_url(url)
+
+
def test_check_url_uppercase_http_scheme_is_normalized_then_ip_gated():
"""An uppercase HTTP scheme must not fail open: it normalizes to http and
the loopback host is still refused by the address gate, not waved through