Skip to content

Convert Claude Code or Codex into the most intelligent Deep Research Agent. Collect, search, and synthesize web research into a persistent, searchable wiki that builds on itself. Hosted API + MCP: hyperresearch.ai

security: SSRF gate + size caps for fetches (builtin, crawl4ai entry points, PDFs, images) (#53)

Francois du Plessis54c7d0205046
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b4e5611..f3572c7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,13 @@
- **The PreToolUse reminder now reaches the model (#94).** The installed hook wrote its reminder to stderr and exited 0, and for PreToolUse that channel is discarded — so the "check the research base before searching the web" nudge shipped since 0.x had never once been delivered. It is emitted as `hookSpecificOutput.additionalContext` now, which is the documented injection channel. The matcher is narrowed to `WebSearch|WebFetch`: on `Glob` and `Grep` the advice is noise, and delivering it there for the first time would have made every local vault operation pay for a reminder about the web.
+### SSRF gate + size caps for fetches (builtin, crawl4ai entry points, PDF and image downloads)
+
+- **The builtin provider, the crawl4ai entry points, and PDF and image downloads now go through an SSRF gate** (`web/safe_http.py`): http/https only; hostnames resolving to private, loopback, link-local, multicast, reserved, CGNAT (RFC 6598), or unspecified addresses are refused (cloud metadata, RFC1918, `[::1]`, `100.64.0.0/10`, …). Not covered: the fixed-host API calls in `core/embed.py` and `core/scholar.py`, which take no attacker-influenced URLs. On the `safe_get` paths (builtin provider, PDF and image downloads) redirects are followed manually and every hop is revalidated. The browser lanes (crawl4ai `fetch()`, `fetch_many()`, and the visible-window path) are gated at entry AND re-checked on the URL the browser actually landed on — the browser follows redirects internally, so the entry check alone cannot vouch for the destination. A post-hoc recheck cannot stop the request that already fired (blind SSRF survives it), but it keeps private-network content out of the vault; the same-host case skips the extra DNS lookup. Refused URLs in a batch are logged and skipped, not fatal. The hostname check is best-effort against DNS rebinding (httpx re-resolves at connect time); the residual window is documented in the module docstring.
+- **`[fetch] allow_private_hosts`** — escape hatch for self-hosted mirrors and intranet sources: hostnames (exact, case-insensitive) or CIDRs that may be fetched despite resolving to private space. Empty by default; consulted by `check_url` everywhere, including redirect hops and the final-URL recheck. A malformed CIDR entry errors loudly instead of being silently ignored.
+- **Response-size caps**, streamed and enforced mid-body so a lying or chunked server cannot exhaust memory. Configurable via new `[fetch]` settings `max_html_bytes` (10 MiB), `max_pdf_bytes` (25 MiB), `max_image_bytes` (2 MiB). A gate refusal on an image download is printed, not silently swallowed.
+- **No unverified-TLS retry.** A certificate failure on a PDF fetch is a refusal that names the existing `pdf_verify_tls = false` opt-out — never an automatic verify-off retry, which a MITM could force with a bad cert. The refusal is its own exception type (`CertVerificationError`), and a cert-refused PDF is never handed to the browser fallback lane: that lane runs with TLS errors ignored, so "falling back" there would be the automatic unverified retry by another name. In a batch it is a loud skip instead.
+
### Scholarly discovery: eight sources through one client layer
Academic discovery used to be four URL templates rendered into the agent's instructions by `core/agent_docs.py`, which the model was trusted to assemble and call by hand. No retry, no rate limiting, no dedup, no offline tests — and one of those templates shipped `[email protected]`, a shared placeholder on every install, which is exactly the anti-pattern the open-access resolver refuses to commit for Unpaywall. It is now a real package.
diff --git a/src/hyperresearch/cli/fetch.py b/src/hyperresearch/cli/fetch.py
index d400ed7..7258152 100644
--- a/src/hyperresearch/cli/fetch.py
+++ b/src/hyperresearch/cli/fetch.py
@@ -10,7 +10,7 @@ from urllib.parse import urlparse
import typer
-from hyperresearch.cli._output import console, output
+from hyperresearch.cli._output import console, err_console, output
from hyperresearch.core.config import AssetSettings, FetchSettings
from hyperresearch.models.output import error, success
@@ -587,6 +587,8 @@ def fetch(
saved_assets = _save_assets(
conn, result, note_id, assets_dir,
settings=vault.config.assets, image_timeout_s=vault.config.fetch.image_timeout_s,
+ max_image_bytes=vault.config.fetch.max_image_bytes,
+ allow_private_hosts=vault.config.fetch.allow_private_hosts,
)
data = {
@@ -666,10 +668,18 @@ def _save_assets(
assets_dir: Path,
settings: AssetSettings | None = None,
image_timeout_s: int | None = None,
+ max_image_bytes: int | None = None,
+ allow_private_hosts: tuple[str, ...] | None = None,
) -> list[dict]:
"""Save screenshot and images to assets dir, record in DB. Returns list of saved asset info."""
settings = settings or AssetSettings()
- timeout_s = image_timeout_s if image_timeout_s is not None else FetchSettings().image_timeout_s
+ fetch_defaults = FetchSettings()
+ timeout_s = image_timeout_s if image_timeout_s is not None else fetch_defaults.image_timeout_s
+ image_cap = max_image_bytes if max_image_bytes is not None else fetch_defaults.max_image_bytes
+ allow_private = (
+ allow_private_hosts if allow_private_hosts is not None
+ else fetch_defaults.allow_private_hosts
+ )
saved: list[dict] = []
now = datetime.now(UTC).isoformat()
@@ -713,6 +723,7 @@ def _save_assets(
asset_info = _download_image(
conn, note_id, img_url, alt, assets_dir, now,
min_image_bytes=settings.min_image_bytes, timeout_s=timeout_s,
+ max_image_bytes=image_cap, allow_private_hosts=allow_private,
)
if asset_info:
saved.append(asset_info)
@@ -726,10 +737,10 @@ def _save_assets(
def _download_image(
conn, note_id: str, img_url: str, alt: str, assets_dir: Path, now: str,
min_image_bytes: int = 50_000, timeout_s: int = 15,
+ max_image_bytes: int | None = None,
+ allow_private_hosts: tuple[str, ...] = (),
) -> dict | None:
"""Download a single image. Returns asset info dict or None if skipped."""
- import urllib.request
-
# Generate filename from URL
parsed = urlparse(img_url)
url_path = parsed.path.rstrip("/")
@@ -751,11 +762,24 @@ def _download_image(
file_path = assets_dir / f"{clean_name}-{counter}{ext}"
counter += 1
+ from hyperresearch.web.safe_http import MAX_BYTES_IMAGE, SafeHTTPError, safe_get
+
try:
- req = urllib.request.Request(img_url, headers={"User-Agent": "hyperresearch/0.1"})
- with urllib.request.urlopen(req, timeout=timeout_s) as resp:
- data = resp.read()
- content_type = resp.headers.get("Content-Type", "")
+ resp = safe_get(
+ img_url,
+ max_bytes=max_image_bytes if max_image_bytes is not None else MAX_BYTES_IMAGE,
+ timeout=timeout_s,
+ allow_private_hosts=allow_private_hosts,
+ )
+ data = resp.content
+ content_type = resp.headers.get("content-type", "")
+ except SafeHTTPError as exc:
+ # A gate refusal (SSRF / size cap) is a security event, not a skip —
+ # it must be visible, unlike routine 404s and timeouts below. Printed
+ # to stderr: this helper has no json_output flag, and stdout must stay
+ # pure JSON in --json mode.
+ err_console.print(f" [yellow]Image refused by fetch gate:[/] {img_url} — {exc}")
+ return None
except Exception:
return None
diff --git a/src/hyperresearch/core/config.py b/src/hyperresearch/core/config.py
index 49f8c60..5a99c87 100644
--- a/src/hyperresearch/core/config.py
+++ b/src/hyperresearch/core/config.py
@@ -18,6 +18,18 @@ class FetchSettings:
# cert-broken mirrors you explicitly trust.
pdf_verify_tls: bool = True
min_pdf_bytes: int = 100
+ # Response-size caps enforced by the SSRF gate (web/safe_http.py).
+ # Defaults mirror the MAX_BYTES_* constants there.
+ max_html_bytes: int = 10 * 1024 * 1024
+ max_pdf_bytes: int = 25 * 1024 * 1024
+ max_image_bytes: int = 2 * 1024 * 1024
+ # SSRF-gate escape hatch: hostnames (exact, case-insensitive) or IP
+ # networks in CIDR form ("10.8.0.0/16", "192.168.1.20") that may be
+ # fetched even though they resolve to private/reserved addresses —
+ # for self-hosted mirrors and intranet sources. Empty by default:
+ # adding an entry is an explicit act by someone who controls the
+ # address space.
+ allow_private_hosts: tuple[str, ...] = ()
# Smart-wait DOM-stability loop (shared by headless and visible paths)
wait_initial_ms: int = 2000
poll_interval_ms: int = 500
diff --git a/src/hyperresearch/core/fetcher.py b/src/hyperresearch/core/fetcher.py
index e365458..0b9406e 100644
--- a/src/hyperresearch/core/fetcher.py
+++ b/src/hyperresearch/core/fetcher.py
@@ -175,6 +175,8 @@ def fetch_and_save(
saved_assets = _save_assets(
conn, result, note_id, assets_dir,
settings=vault.config.assets, image_timeout_s=vault.config.fetch.image_timeout_s,
+ max_image_bytes=vault.config.fetch.max_image_bytes,
+ allow_private_hosts=vault.config.fetch.allow_private_hosts,
)
return {
diff --git a/src/hyperresearch/web/base.py b/src/hyperresearch/web/base.py
index 4e944ee..ebf5570 100644
--- a/src/hyperresearch/web/base.py
+++ b/src/hyperresearch/web/base.py
@@ -194,7 +194,7 @@ def get_provider(
if name is None or name == "builtin":
from hyperresearch.web.builtin import BuiltinProvider
- return BuiltinProvider()
+ return BuiltinProvider(settings=settings)
if name == "crawl4ai":
try:
diff --git a/src/hyperresearch/web/builtin.py b/src/hyperresearch/web/builtin.py
index c359fea..b90a076 100644
--- a/src/hyperresearch/web/builtin.py
+++ b/src/hyperresearch/web/builtin.py
@@ -1,4 +1,4 @@
-"""Builtin web provider — uses httpx + beautifulsoup4 if available, else bare urllib."""
+"""Builtin web provider — SSRF-gated fetching via safe_http, beautifulsoup4 extraction if available."""
from __future__ import annotations
@@ -52,6 +52,11 @@ class BuiltinProvider:
name = "builtin"
+ def __init__(self, settings=None):
+ from hyperresearch.core.config import FetchSettings
+
+ self._settings = settings or FetchSettings()
+
def fetch(self, url: str) -> WebResult:
html, final_url = self._download(url)
title, content = self._extract(html)
@@ -70,23 +75,18 @@ class BuiltinProvider:
)
def _download(self, url: str) -> tuple[str, str]:
- """Download URL, return (html, final_url). Tries httpx first, falls back to urllib."""
- try:
- import httpx
-
- with httpx.Client(follow_redirects=True, timeout=30) as client:
- resp = client.get(url, headers={"User-Agent": "hyperresearch/0.1"})
- resp.raise_for_status()
- return resp.text, str(resp.url)
- except ImportError:
- pass
-
- import urllib.request
-
- req = urllib.request.Request(url, headers={"User-Agent": "hyperresearch/0.1"})
- with urllib.request.urlopen(req, timeout=30) as resp:
- html = resp.read().decode("utf-8", errors="replace")
- return html, resp.url or url
+ """Download URL, return (html, final_url). All requests go through the
+ SSRF gate in :mod:`hyperresearch.web.safe_http`."""
+ from hyperresearch.web.safe_http import safe_get
+
+ resp = safe_get(
+ url,
+ max_bytes=self._settings.max_html_bytes,
+ allow_private_hosts=self._settings.allow_private_hosts,
+ )
+ if resp.status_code >= 400:
+ raise RuntimeError(f"HTTP {resp.status_code} fetching {url}")
+ return resp.text, resp.url
def _extract(self, html: str) -> tuple[str, str]:
"""Extract title and clean text from HTML. Tries bs4 first, falls back to stdlib."""
diff --git a/src/hyperresearch/web/crawl4ai_provider.py b/src/hyperresearch/web/crawl4ai_provider.py
index 225dcf9..e2f33fd 100644
--- a/src/hyperresearch/web/crawl4ai_provider.py
+++ b/src/hyperresearch/web/crawl4ai_provider.py
@@ -122,6 +122,63 @@ def _import_pymupdf():
return None
+_PDF_FETCH_HEADERS = {
+ "User-Agent": (
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+ "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+ ),
+ "Accept": "application/pdf,application/octet-stream;q=0.9,*/*;q=0.8",
+}
+
+
+def _is_cert_error(exc: Exception) -> bool:
+ import ssl
+
+ # httpx nests the ssl error two causes deep (httpx.ConnectError ->
+ # httpcore.ConnectError -> SSLCertVerificationError), so walk the chain
+ # rather than checking only the direct cause. String match as fallback.
+ cause: BaseException | None = exc
+ for _ in range(5):
+ if cause is None:
+ break
+ if isinstance(cause, ssl.SSLCertVerificationError):
+ return True
+ cause = cause.__cause__
+ return "CERTIFICATE_VERIFY_FAILED" in str(exc)
+
+
+def _safe_get_pdf(url: str, settings: FetchSettings):
+ """SSRF-gated, size-capped PDF download.
+
+ TLS verification follows ``pdf_verify_tls`` (default on) with NO
+ automatic unverified retry: a MITM can serve a bad certificate
+ precisely to force such a retry, which would turn verify-by-default
+ into something the attacker controls. Mirrors with known-broken
+ certificates are handled by the explicit, user-declared
+ ``pdf_verify_tls = false`` opt-out.
+ """
+ from hyperresearch.web.safe_http import CertVerificationError, safe_get
+
+ try:
+ return safe_get(url, max_bytes=settings.max_pdf_bytes,
+ timeout=settings.pdf_timeout_s,
+ headers=_PDF_FETCH_HEADERS,
+ verify=settings.pdf_verify_tls,
+ allow_private_hosts=settings.allow_private_hosts)
+ except Exception as exc:
+ if settings.pdf_verify_tls and _is_cert_error(exc):
+ # Refuse, but say how to opt out for a trusted cert-broken mirror.
+ # Raised as its own type: the browser lane runs with TLS errors
+ # ignored, so treating this like any failed PDF would hand the
+ # URL to a lane that amounts to an automatic unverified retry.
+ raise CertVerificationError(
+ f"certificate verification failed for {url!r}: {exc}. "
+ "If this host is a known cert-broken mirror you trust, set "
+ "pdf_verify_tls = false under [fetch] in config.toml."
+ ) from exc
+ raise
+
+
def _fetch_pdf(url: str, settings: FetchSettings | None = None) -> WebResult | None:
"""Download a PDF and extract text using pymupdf. Returns None if extraction fails.
@@ -135,7 +192,7 @@ def _fetch_pdf(url: str, settings: FetchSettings | None = None) -> WebResult | N
settings = settings or FetchSettings()
- import httpx
+ from hyperresearch.web.safe_http import CertVerificationError, SafeHTTPError
try:
# Convert arXiv abs links to PDF links
@@ -144,14 +201,17 @@ def _fetch_pdf(url: str, settings: FetchSettings | None = None) -> WebResult | N
if not url.endswith(".pdf"):
url += ".pdf"
- resp = httpx.get(url, follow_redirects=True, timeout=settings.pdf_timeout_s,
- verify=settings.pdf_verify_tls, headers={
- "User-Agent": (
- "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
- "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
- ),
- "Accept": "application/pdf,application/octet-stream;q=0.9,*/*;q=0.8",
- })
+ try:
+ resp = _safe_get_pdf(url, settings)
+ except CertVerificationError:
+ # Propagates: callers must not fold a cert refusal into the
+ # generic PDF-failed None, whose fallback is the TLS-ignoring
+ # browser lane.
+ raise
+ except SafeHTTPError as exc:
+ _pdf_log().warning("refused PDF fetch for %s: %s", url, exc)
+ return None
+
if resp.status_code != 200:
_pdf_log().warning("PDF fetch for %s returned HTTP %s", url, resp.status_code)
return None
@@ -212,11 +272,43 @@ def _fetch_pdf(url: str, settings: FetchSettings | None = None) -> WebResult | N
raw_content_type="application/pdf",
)
+ except CertVerificationError:
+ # Second guard: the catch-all below must not convert a cert refusal
+ # into the generic None either.
+ raise
except Exception as e:
_pdf_log().warning("PDF extraction failed for %s: %s", url, e)
return None
+def _check_final_url(entry_url: str, final_url: str | None, settings: FetchSettings) -> None:
+ """Re-validate the URL the browser actually ended up on.
+
+ The browser follows redirects internally, so the entry-point gate only
+ vouches for where navigation STARTED. Refusing after the fact cannot
+ stop a request that already fired (blind SSRF survives), but it keeps
+ private-network content out of the vault. Skipped when the final host
+ equals the entry host — the entry gate already vouched for it, and the
+ extra ``getaddrinfo`` would be pure cost on the no-redirect common case.
+ """
+ from urllib.parse import urlparse
+
+ from hyperresearch.web.safe_http import SafeHTTPError, check_url
+
+ if not final_url or final_url == entry_url:
+ return
+ entry_host = urlparse(entry_url).hostname
+ parsed_final = urlparse(final_url)
+ if parsed_final.hostname == entry_host and parsed_final.scheme.lower() in ("http", "https"):
+ return
+ try:
+ check_url(final_url, settings.allow_private_hosts)
+ except SafeHTTPError as exc:
+ raise SafeHTTPError(
+ f"browser navigation for {entry_url!r} ended at refused URL: {exc}"
+ ) from exc
+
+
class Crawl4AIProvider:
name = "crawl4ai"
@@ -283,6 +375,15 @@ class Crawl4AIProvider:
return AsyncWebCrawler(crawler_strategy=strategy, config=self._browser_config)
def fetch(self, url: str) -> WebResult:
+ # SSRF gate: reject private/loopback/etc. before any browser or http
+ # call. PDF redirects are re-checked hop by hop via safe_get; the
+ # browser lanes drive their own navigation, so they get the
+ # entry-point check here plus a final-URL recheck after the fact
+ # (_check_final_url).
+ from hyperresearch.web.safe_http import check_url
+
+ check_url(url, self._settings.allow_private_hosts)
+
# PDF detection: fetch directly with httpx, extract text with pymupdf
if _is_pdf_url(url):
result = _fetch_pdf(url, self._settings)
@@ -333,6 +434,10 @@ class Crawl4AIProvider:
await context.close()
+ # This lane runs with ignore_https_errors and follows redirects on
+ # its own, so where it LANDED needs the same gate as where it started.
+ _check_final_url(url, final_url, self._settings)
+
# Convert HTML to markdown using crawl4ai's markdown generator
# Prefer fit_markdown (main content, no nav/footer chrome) over raw_markdown.
md_result = self._md_generator.generate_markdown(html, base_url=final_url)
@@ -357,6 +462,12 @@ class Crawl4AIProvider:
async def _fetch_async(self, url: str) -> WebResult:
async with self._make_crawler() as crawler:
result = await crawler.arun(url=url, config=self._run_config)
+ # crawl4ai's result.url is the REQUESTED url even after the
+ # browser followed redirects; the landing url is redirected_url
+ # (hermetic-tier proven — rechecking result.url alone is blind).
+ _check_final_url(
+ url, getattr(result, "redirected_url", None) or result.url, self._settings
+ )
metadata = result.metadata or {}
# result.markdown is a MarkdownGenerationResult with .raw_markdown,
@@ -412,9 +523,23 @@ class Crawl4AIProvider:
return asyncio.run(self._fetch_many_async(urls))
async def _fetch_many_async(self, urls: list[str]) -> list[WebResult]:
+ # SSRF gate: same entry-point check as fetch(). Refused URLs are
+ # skipped (logged), not fatal — one hostile URL must not kill the
+ # whole batch.
+ from hyperresearch.web.safe_http import CertVerificationError, SafeHTTPError, check_url
+
+ log = logging.getLogger("hyperresearch.web")
+ allowed_urls = []
+ for url in urls:
+ try:
+ check_url(url, self._settings.allow_private_hosts)
+ allowed_urls.append(url)
+ except SafeHTTPError as exc:
+ log.warning("refused batch fetch for %s: %s", url, exc)
+
# Split: PDFs go direct, rest go through browser
- pdf_urls = [u for u in urls if _is_pdf_url(u)]
- html_urls = [u for u in urls if not _is_pdf_url(u)]
+ pdf_urls = [u for u in allowed_urls if _is_pdf_url(u)]
+ html_urls = [u for u in allowed_urls if not _is_pdf_url(u)]
web_results = []
@@ -422,10 +547,22 @@ class Crawl4AIProvider:
# None (parser failure, non-PDF body served at a .pdf URL) we fall
# back to the browser path — mirrors the single-fetch behaviour in
# fetch() so batch callers don't silently lose academic PDFs that
- # need JS-rendered landing pages.
+ # need JS-rendered landing pages. Cert refusals are the exception:
+ # the browser lane ignores TLS errors, so falling back would be an
+ # automatic unverified retry. A batch has no per-URL failure
+ # channel, so the skip is logged loudly instead.
failed_pdf_urls = []
for url in pdf_urls:
- pdf_result = _fetch_pdf(url, self._settings)
+ try:
+ pdf_result = _fetch_pdf(url, self._settings)
+ except CertVerificationError as exc:
+ log.warning(
+ "SKIPPED (TLS certificate invalid): %s -- a potentially "
+ "valuable source was not fetched. To include it, set "
+ "pdf_verify_tls = false under [fetch] in config.toml. (%s)",
+ url, exc,
+ )
+ continue
if pdf_result is not None:
web_results.append(pdf_result)
else:
@@ -440,6 +577,13 @@ class Crawl4AIProvider:
for cr, url in zip(results, browser_urls, strict=False):
if not cr.success:
continue
+ try:
+ _check_final_url(
+ url, getattr(cr, "redirected_url", None) or cr.url, self._settings
+ )
+ except SafeHTTPError as exc:
+ log.warning("dropping batch result for %s: %s", url, exc)
+ continue
metadata = cr.metadata or {}
md = cr.markdown
if md and hasattr(md, "fit_markdown"):
@@ -453,7 +597,13 @@ class Crawl4AIProvider:
# Post-fetch binary check — browser may have fetched a PDF inline
if content and _looks_like_binary(content, self._gates):
- pdf_result = _fetch_pdf(url, self._settings)
+ try:
+ pdf_result = _fetch_pdf(url, self._settings)
+ except CertVerificationError as exc:
+ # Keep the batch alive, but don't store the binary
+ # garbage the browser got either.
+ log.warning("SKIPPED (TLS certificate invalid): %s (%s)", url, exc)
+ continue
if pdf_result is not None:
web_results.append(pdf_result)
continue
diff --git a/src/hyperresearch/web/safe_http.py b/src/hyperresearch/web/safe_http.py
new file mode 100644
index 0000000..2f39106
--- /dev/null
+++ b/src/hyperresearch/web/safe_http.py
@@ -0,0 +1,267 @@
+"""SSRF-safe HTTP helper for outbound fetches.
+
+Centralizes the network egress used by web providers and the image
+downloader. Every request:
+
+- Validates the URL scheme (http/https only).
+- Resolves the hostname and refuses any address that is private,
+ loopback, link-local, multicast, reserved, or unspecified — so an
+ attacker page cannot trick a fetcher into reaching `169.254.169.254`
+ (cloud metadata), `127.0.0.1`, RFC1918, or `file://`.
+- Follows redirects MANUALLY, re-validating each hop against the same
+ gate.
+- Streams the response into a fixed-size buffer so a hostile server
+ cannot exhaust memory with an infinite or zip-bombed body.
+
+The address check is BEST-EFFORT against DNS rebinding: the gate
+resolves the hostname to validate it, but httpx re-resolves at connect
+time, so a low-TTL record that flips between the two lookups can still
+reach a private address. Closing that window needs the validated IP
+pinned into the actual connection, which httpx does not support without
+a custom transport; the residual risk is accepted and documented here.
+
+Returns a small :class:`SafeResponse` dataclass instead of an
+``httpx.Response`` so callers don't depend on httpx internals.
+"""
+
+from __future__ import annotations
+
+import ipaddress
+import socket
+from dataclasses import dataclass, field
+from urllib.parse import urlparse
+
+import httpx
+
+_ALLOWED_SCHEMES = frozenset({"http", "https"})
+
+DEFAULT_TIMEOUT = 30.0
+DEFAULT_MAX_REDIRECTS = 5
+DEFAULT_USER_AGENT = "hyperresearch/0.1"
+
+# Caller-intent size caps. Pick the smallest that fits the body type;
+# the gate refuses anything larger. These are the DEFAULTS — callers with
+# a FetchSettings in scope should pass its max_html_bytes / max_pdf_bytes
+# / max_image_bytes fields instead, which are user-configurable and
+# mirror these values.
+MAX_BYTES_HTML = 10 * 1024 * 1024
+MAX_BYTES_PDF = 25 * 1024 * 1024
+MAX_BYTES_IMAGE = 2 * 1024 * 1024
+
+
+class SafeHTTPError(Exception):
+ """The SSRF gate refused this request, or the response exceeded a limit."""
+
+
+class CertVerificationError(SafeHTTPError):
+ """TLS certificate verification failed and verification is required.
+
+ Raised distinctly so callers can surface the refusal (and its
+ ``pdf_verify_tls = false`` opt-out) instead of treating it like any
+ other failed fetch — in particular, a cert-refused URL must never be
+ retried through a lane that ignores TLS errors.
+ """
+
+
+@dataclass
+class SafeResponse:
+ """Minimal response wrapper. Avoids leaking httpx types to callers."""
+
+ url: str
+ status_code: int
+ headers: dict[str, str]
+ content: bytes = field(repr=False)
+
+ @property
+ def text(self) -> str:
+ charset = "utf-8"
+ ct = self.headers.get("content-type", "")
+ if "charset=" in ct.lower():
+ charset = ct.lower().split("charset=", 1)[1].split(";", 1)[0].strip() or "utf-8"
+ try:
+ return self.content.decode(charset, errors="replace")
+ except LookupError:
+ return self.content.decode("utf-8", errors="replace")
+
+
+# Carrier-grade NAT (RFC 6598). Not covered by any ipaddress is_* property
+# (checked through 3.11.9), yet never a public destination.
+_CGNAT_NET = ipaddress.ip_network("100.64.0.0/10")
+# 6to4 (RFC 3056): a 2002:AABB:CCDD::/48 prefix embeds the IPv4 address
+# A.B.C.D, so 2002:7f00:1:: is loopback wearing an IPv6 coat. The transport
+# is deprecated and rarely routed, but the ipaddress module does not flag it
+# and the embedded address is what matters. IPv4-mapped (::ffff:a.b.c.d) is
+# already caught because ipaddress unwraps it for the is_* properties.
+_6TO4_NET = ipaddress.ip_network("2002::/16")
+
+
+def _is_blocked_ip(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
+ if ip.version == 6 and ip in _6TO4_NET:
+ packed = int(ip) >> 80 & 0xFFFFFFFF
+ return _is_blocked_ip(ipaddress.IPv4Address(packed))
+ return (
+ ip.is_private
+ or ip.is_loopback
+ or ip.is_link_local
+ or ip.is_multicast
+ or ip.is_reserved
+ or ip.is_unspecified
+ or (ip.version == 4 and ip in _CGNAT_NET)
+ )
+
+
+def _resolve(host: str) -> list[ipaddress.IPv4Address | ipaddress.IPv6Address]:
+ try:
+ return [ipaddress.ip_address(host.strip("[]"))]
+ except ValueError:
+ pass
+ try:
+ infos = socket.getaddrinfo(host, None)
+ except socket.gaierror as exc:
+ raise SafeHTTPError(f"DNS lookup failed for {host!r}: {exc}") from exc
+ addrs: list[ipaddress.IPv4Address | ipaddress.IPv6Address] = []
+ for info in infos:
+ try:
+ addrs.append(ipaddress.ip_address(info[4][0]))
+ except (ValueError, IndexError):
+ continue
+ return addrs
+
+
+def _parse_allowlist(
+ entries: tuple[str, ...] | list[str],
+) -> tuple[frozenset[str], list[ipaddress.IPv4Network | ipaddress.IPv6Network]]:
+ """Split ``allow_private_hosts`` entries into hostnames and networks.
+
+ An entry that parses as an IP network (a bare address counts as a
+ /32 or /128) matches by resolved address; anything else matches the
+ URL hostname exactly, case-insensitively. A malformed entry raises
+ :class:`SafeHTTPError` naming it — silently ignoring one would turn
+ a typo into a lockout that looks like an SSRF refusal.
+ """
+ hostnames: set[str] = set()
+ networks: list[ipaddress.IPv4Network | ipaddress.IPv6Network] = []
+ for entry in entries:
+ entry = entry.strip()
+ if not entry:
+ continue
+ try:
+ networks.append(ipaddress.ip_network(entry, strict=False))
+ except ValueError:
+ if "/" in entry: # meant as a CIDR, but malformed
+ raise SafeHTTPError(
+ f"invalid allow_private_hosts entry {entry!r}: not a valid CIDR"
+ ) from None
+ hostnames.add(entry.lower())
+ return frozenset(hostnames), networks
+
+
+def check_url(url: str, allow_private_hosts: tuple[str, ...] = ()) -> None:
+ """Validate the URL is safe to fetch. Raises :class:`SafeHTTPError`.
+
+ ``allow_private_hosts`` (the ``[fetch]`` config field of the same name)
+ lists hostnames or CIDRs that are fetchable despite resolving to
+ private/reserved addresses — self-hosted mirrors, intranet wikis.
+ Scheme and DNS checks still apply to allowlisted hosts.
+
+ Best-effort: validation resolves the hostname here, but the connection
+ made afterwards re-resolves it (see the module docstring on DNS
+ rebinding).
+ """
+ parsed = urlparse(url)
+ if parsed.scheme.lower() not in _ALLOWED_SCHEMES:
+ raise SafeHTTPError(
+ f"refusing URL with scheme {parsed.scheme!r}; only http/https are allowed"
+ )
+ if not parsed.hostname:
+ raise SafeHTTPError(f"refusing URL with no hostname: {url!r}")
+ allowed_hosts, allowed_nets = _parse_allowlist(allow_private_hosts)
+ if parsed.hostname.lower() in allowed_hosts:
+ return
+ addrs = _resolve(parsed.hostname)
+ if not addrs:
+ raise SafeHTTPError(f"refusing URL {url!r}: hostname did not resolve")
+ for addr in addrs:
+ if _is_blocked_ip(addr) and not any(
+ addr.version == net.version and addr in net for net in allowed_nets
+ ):
+ raise SafeHTTPError(
+ f"refusing URL {url!r}: hostname {parsed.hostname} resolves to "
+ f"non-public address {addr}"
+ )
+
+
+def safe_get(
+ url: str,
+ *,
+ max_bytes: int,
+ timeout: float = DEFAULT_TIMEOUT,
+ max_redirects: int = DEFAULT_MAX_REDIRECTS,
+ headers: dict[str, str] | None = None,
+ verify: bool | str = True,
+ allow_private_hosts: tuple[str, ...] = (),
+ transport: httpx.BaseTransport | None = None,
+) -> SafeResponse:
+ """GET ``url`` with SSRF + size protection.
+
+ ``max_bytes`` is required so callers consciously pick a cap. Pass the
+ matching ``FetchSettings`` field when one is in scope, or a
+ ``MAX_BYTES_*`` constant otherwise.
+
+ ``allow_private_hosts`` is threaded into the per-hop URL checks, so a
+ redirect landing on an allowlisted private host is honored the same
+ way a direct fetch of it would be.
+
+ ``transport`` is a test seam: it lets the suite exercise the redirect
+ revalidation and size-cap logic against ``httpx.MockTransport``
+ without opening sockets. Production callers must leave it None.
+ """
+ if max_bytes <= 0:
+ raise ValueError("max_bytes must be > 0")
+
+ request_headers = {"User-Agent": DEFAULT_USER_AGENT}
+ if headers:
+ request_headers.update(headers)
+
+ current_url = url
+ with httpx.Client(
+ follow_redirects=False, timeout=timeout, verify=verify, transport=transport
+ ) as client:
+ for _ in range(max_redirects + 1):
+ check_url(current_url, allow_private_hosts)
+ declared = None
+ with client.stream("GET", current_url, headers=request_headers) as resp:
+ if resp.is_redirect:
+ location = resp.headers.get("location")
+ if not location:
+ return _materialize(resp, b"", current_url)
+ current_url = str(httpx.URL(current_url).join(location))
+ continue
+
+ declared = resp.headers.get("content-length")
+ if declared and declared.isdigit() and int(declared) > max_bytes:
+ raise SafeHTTPError(
+ f"refusing response from {current_url!r}: "
+ f"Content-Length {declared} exceeds cap {max_bytes}"
+ )
+
+ body = bytearray()
+ for chunk in resp.iter_bytes():
+ body.extend(chunk)
+ if len(body) > max_bytes:
+ raise SafeHTTPError(
+ f"refusing response from {current_url!r}: "
+ f"body exceeds cap {max_bytes} bytes"
+ )
+ return _materialize(resp, bytes(body), current_url)
+
+ raise SafeHTTPError(f"too many redirects (>{max_redirects}) starting at {url!r}")
+
+
+def _materialize(resp: httpx.Response, body: bytes, final_url: str) -> SafeResponse:
+ return SafeResponse(
+ url=final_url,
+ status_code=resp.status_code,
+ headers={k.lower(): v for k, v in resp.headers.items()},
+ content=body,
+ )
diff --git a/tests/test_web/test_fetch_many_gate.py b/tests/test_web/test_fetch_many_gate.py
new file mode 100644
index 0000000..95a5cd3
--- /dev/null
+++ b/tests/test_web/test_fetch_many_gate.py
@@ -0,0 +1,74 @@
+"""fetch_many must apply the same SSRF gate as fetch().
+
+It is the main fetch path (every wave-1 fetcher batch goes through it), so a
+gate that only covered fetch() would leave the bulk of outbound traffic
+unprotected. Offline: only .pdf URLs are used and _fetch_pdf is stubbed to
+SUCCEED, so the browser lane is never touched — since v0.9.1 a *failed* PDF
+falls back to the browser (that path is covered in test_fetch_many_fallback
+and test_final_url_recheck).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+
+import hyperresearch.web.crawl4ai_provider as provider
+from hyperresearch.web.base import WebResult
+
+
+def _run(coro):
+ """Run ``coro`` to completion in an isolated thread with its own loop.
+
+ A prior test in the full suite can leave this thread's event-loop state as
+ "running" (seen intermittently on Python 3.11 once crawl4ai's async
+ machinery has been exercised), which makes a plain ``asyncio.run()`` here
+ raise "cannot be called from a running event loop". A fresh thread has no
+ running loop, so running the coroutine there is immune to that leaked state.
+ Logging still lands in caplog: handlers are process-global, and the thread
+ is joined inside the caplog context below.
+ """
+ import threading
+
+ box: dict = {}
+
+ def target() -> None:
+ try:
+ box["result"] = asyncio.run(coro)
+ except Exception as exc:
+ box["error"] = exc
+
+ thread = threading.Thread(target=target)
+ thread.start()
+ thread.join()
+ if "error" in box:
+ raise box["error"]
+ return box["result"]
+
+
+def test_fetch_many_skips_refused_urls_and_fetches_the_rest(monkeypatch, caplog):
+ fetched: list[str] = []
+
+ def fake_fetch_pdf(url, settings=None):
+ fetched.append(url)
+ return WebResult(url=url, title="ok", content="pdf text")
+
+ monkeypatch.setattr(provider, "_fetch_pdf", fake_fetch_pdf)
+
+ inst = provider.Crawl4AIProvider.__new__(provider.Crawl4AIProvider)
+ inst._settings = provider.FetchSettings()
+
+ urls = [
+ "http://127.0.0.1/internal.pdf", # loopback — refused
+ "http://169.254.169.254/meta.pdf", # cloud metadata — refused
+ "http://8.8.8.8/paper.pdf", # public literal — allowed
+ ]
+ with caplog.at_level(logging.WARNING, logger="hyperresearch.web"):
+ results = _run(inst._fetch_many_async(urls))
+
+ assert [r.url for r in results] == ["http://8.8.8.8/paper.pdf"]
+ assert fetched == ["http://8.8.8.8/paper.pdf"], (
+ "refused URLs must be skipped, allowed ones still fetched"
+ )
+ assert "refused batch fetch" in caplog.text
+ assert "127.0.0.1" in caplog.text
diff --git a/tests/test_web/test_fetch_settings.py b/tests/test_web/test_fetch_settings.py
index b47856e..bb0c90b 100644
--- a/tests/test_web/test_fetch_settings.py
+++ b/tests/test_web/test_fetch_settings.py
@@ -2,9 +2,12 @@
from __future__ import annotations
+from unittest.mock import patch
+
import pytest
from hyperresearch.core.config import FetchSettings, JunkGates
+from hyperresearch.web.safe_http import SafeResponse
crawl4ai_provider = pytest.importorskip(
"hyperresearch.web.crawl4ai_provider",
@@ -76,3 +79,52 @@ class TestGetProviderThreading:
)
assert prov._run_config.page_timeout == 12345
assert prov._gates.binary_garbage_ratio == 0.9
+
+
+class TestByteCapThreading:
+ """The configurable size caps only protect anything if the FetchSettings
+ field actually reaches safe_get's max_bytes. safe_get's own enforcement is
+ tested in test_safe_http; these pin the wiring from config to gate, which
+ is what a refactor to the hardcoded MAX_BYTES_* constants would silently
+ break."""
+
+ def test_html_cap_reaches_safe_get(self):
+ from hyperresearch.web.builtin import BuiltinProvider
+
+ seen = {}
+
+ def spy(url, *, max_bytes, **kwargs):
+ seen["max_bytes"] = max_bytes
+ return SafeResponse(url=url, status_code=200, headers={}, content=b"<html></html>")
+
+ prov = BuiltinProvider(settings=FetchSettings(max_html_bytes=1234))
+ with patch("hyperresearch.web.safe_http.safe_get", spy):
+ prov._download("http://8.8.8.8/page")
+ assert seen["max_bytes"] == 1234
+
+ def test_pdf_cap_reaches_safe_get(self):
+ seen = {}
+
+ def spy(url, *, max_bytes, **kwargs):
+ seen["max_bytes"] = max_bytes
+ return SafeResponse(url=url, status_code=200, headers={}, content=b"%PDF-")
+
+ with patch("hyperresearch.web.safe_http.safe_get", spy):
+ crawl4ai_provider._safe_get_pdf("http://8.8.8.8/paper.pdf", FetchSettings(max_pdf_bytes=777))
+ assert seen["max_bytes"] == 777
+
+
+class TestAllowPrivateHostsConfig:
+ def test_default_is_empty(self):
+ """Fresh installs get the full gate; admitting a private host is an
+ explicit config act."""
+ assert FetchSettings().allow_private_hosts == ()
+
+ def test_toml_list_parses_to_tuple(self):
+ from hyperresearch.core.config import _build_section
+
+ s = _build_section(
+ FetchSettings,
+ {"allow_private_hosts": ["10.8.0.0/16", "mirror.internal"]},
+ )
+ assert s.allow_private_hosts == ("10.8.0.0/16", "mirror.internal")
diff --git a/tests/test_web/test_final_url_recheck.py b/tests/test_web/test_final_url_recheck.py
new file mode 100644
index 0000000..cd8f90f
--- /dev/null
+++ b/tests/test_web/test_final_url_recheck.py
@@ -0,0 +1,194 @@
+"""Final-URL recheck + cert-refusal routing in the crawl4ai lanes.
+
+The browser follows redirects on its own, so the entry-point SSRF gate only
+vouches for where navigation started; these tests pin (a) that where it
+LANDED is re-checked on every lane, and (b) that a TLS-cert refusal on a PDF
+is never "retried" through the browser lane, which ignores TLS errors.
+All offline: fake crawler, no browser, no network.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from unittest.mock import patch
+
+import pytest
+
+from hyperresearch.core.config import FetchSettings
+from hyperresearch.web.safe_http import CertVerificationError, SafeHTTPError
+
+provider = pytest.importorskip(
+ "hyperresearch.web.crawl4ai_provider",
+ reason="crawl4ai extra not installed",
+)
+
+
+def _run(coro):
+ """Run ``coro`` in an isolated thread with its own loop (immune to the
+ leaked running-loop state crawl4ai can leave behind on 3.11)."""
+ import threading
+
+ box: dict = {}
+
+ def target() -> None:
+ try:
+ box["result"] = asyncio.run(coro)
+ except Exception as exc:
+ box["error"] = exc
+
+ thread = threading.Thread(target=target)
+ thread.start()
+ thread.join()
+ if "error" in box:
+ raise box["error"]
+ return box["result"]
+
+
+class _FakeCR:
+ """Stand-in for a crawl4ai result; ``final_url`` simulates the browser
+ having been redirected away from the requested URL. Mirrors real
+ crawl4ai semantics: ``url`` stays the REQUESTED url, the landing url is
+ ``redirected_url`` (hermetic-tier proven)."""
+
+ def __init__(self, requested: str, final_url: str | None = None):
+ self.success = True
+ self.url = requested
+ self.redirected_url = final_url or requested
+ self.markdown = f"browser text for {requested}"
+ self.metadata = {"title": "T"}
+ self.media = {}
+ self.links = {}
+ self.screenshot = None
+ self.html = "<html></html>"
+
+
+class _FakeCrawler:
+ def __init__(self, captured: list[str], redirects: dict[str, str]):
+ self._captured = captured
+ self._redirects = redirects
+
+ async def __aenter__(self):
+ return self
+
+ async def __aexit__(self, *exc):
+ return False
+
+ async def arun(self, url, config):
+ self._captured.append(url)
+ return _FakeCR(url, self._redirects.get(url))
+
+ async def arun_many(self, urls, config):
+ self._captured.extend(urls)
+ return [_FakeCR(u, self._redirects.get(u)) for u in urls]
+
+
+def _bare_provider(
+ captured: list[str],
+ monkeypatch,
+ redirects: dict[str, str] | None = None,
+ settings: FetchSettings | None = None,
+) -> object:
+ inst = provider.Crawl4AIProvider.__new__(provider.Crawl4AIProvider)
+ inst._settings = settings or provider.FetchSettings()
+ inst._gates = provider.JunkGates()
+ inst._run_config = object()
+ inst._headless = True
+ inst._data_dir = None
+ monkeypatch.setattr(
+ inst, "_make_crawler", lambda: _FakeCrawler(captured, redirects or {})
+ )
+ return inst
+
+
+# ---------------------------------------------------------------------------
+# _check_final_url — the shared helper all three lanes wire in
+# ---------------------------------------------------------------------------
+
+
+def test_same_host_final_url_skips_re_resolution():
+ """No redirect off-host means no second getaddrinfo — the entry gate
+ already vouched for this hostname."""
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.side_effect = AssertionError("must not re-resolve same-host final URL")
+ provider._check_final_url(
+ "http://a.example/x", "http://a.example/y", FetchSettings()
+ )
+
+
+def test_unchanged_or_missing_final_url_is_a_noop():
+ provider._check_final_url("http://a.example/x", "http://a.example/x", FetchSettings())
+ provider._check_final_url("http://a.example/x", None, FetchSettings())
+
+
+def test_cross_host_private_final_url_is_refused():
+ with pytest.raises(SafeHTTPError, match="ended at refused URL"):
+ provider._check_final_url(
+ "http://8.8.8.8/start", "http://127.0.0.1/secret", FetchSettings()
+ )
+
+
+def test_final_url_recheck_honors_the_allowlist():
+ provider._check_final_url(
+ "http://8.8.8.8/start",
+ "http://192.168.1.20/paper",
+ FetchSettings(allow_private_hosts=("192.168.1.20",)),
+ )
+
+
+# ---------------------------------------------------------------------------
+# Lane wiring — single fetch and batch
+# ---------------------------------------------------------------------------
+
+
+def test_single_fetch_refuses_result_that_landed_private(monkeypatch):
+ captured: list[str] = []
+ inst = _bare_provider(
+ captured, monkeypatch,
+ redirects={"http://8.8.8.8/a": "http://127.0.0.1/admin"},
+ )
+ with pytest.raises(SafeHTTPError, match="ended at refused URL"):
+ _run(inst._fetch_async("http://8.8.8.8/a"))
+
+
+def test_batch_drops_only_the_result_that_landed_private(monkeypatch):
+ captured: list[str] = []
+ inst = _bare_provider(
+ captured, monkeypatch,
+ redirects={"http://8.8.8.8/bad": "http://127.0.0.1/admin"},
+ )
+ results = _run(
+ inst._fetch_many_async(["http://8.8.8.8/bad", "http://8.8.8.8/good"])
+ )
+ assert [r.url for r in results] == ["http://8.8.8.8/good"]
+
+
+# ---------------------------------------------------------------------------
+# Cert refusals never reach the TLS-ignoring browser lane
+# ---------------------------------------------------------------------------
+
+
+def _raise_cert(url, settings=None):
+ raise CertVerificationError(f"certificate verification failed for {url!r}")
+
+
+def test_batch_cert_refused_pdf_is_skipped_not_browsered(monkeypatch):
+ """A cert-refused PDF must be a loud skip — handing it to the browser
+ lane (ignore_https_errors) would be an automatic unverified retry."""
+ monkeypatch.setattr(provider, "_fetch_pdf", _raise_cert)
+ captured: list[str] = []
+ inst = _bare_provider(captured, monkeypatch)
+
+ results = _run(inst._fetch_many_async(["http://8.8.8.8/paper.pdf"]))
+
+ assert results == []
+ assert captured == [] # browser lane never entered
+
+
+def test_single_fetch_cert_refusal_propagates_without_browser(monkeypatch):
+ monkeypatch.setattr(provider, "_fetch_pdf", _raise_cert)
+ captured: list[str] = []
+ inst = _bare_provider(captured, monkeypatch)
+
+ with pytest.raises(CertVerificationError):
+ inst.fetch("http://8.8.8.8/paper.pdf")
+ assert captured == []
diff --git a/tests/test_web/test_pdf_diagnostics.py b/tests/test_web/test_pdf_diagnostics.py
index cafdc27..9528bf3 100644
--- a/tests/test_web/test_pdf_diagnostics.py
+++ b/tests/test_web/test_pdf_diagnostics.py
@@ -6,7 +6,8 @@ the platform — every PDF on every domain silently fell through to the browser
lane, arrived as binary, and was discarded as junk, with nothing logged to say
why. These tests pin the diagnostics, not just the happy path.
-Offline: httpx is stubbed, no network is touched.
+Offline: the download layer (`_safe_get_pdf` / `safe_get`) is stubbed, no
+network is touched.
"""
from __future__ import annotations
@@ -28,12 +29,10 @@ class _Resp:
@pytest.fixture
def stub_httpx(monkeypatch):
- """Replace httpx.get with a canned response."""
+ """Replace the SSRF-gated download call with a canned response."""
def _install(resp: _Resp):
- import httpx
-
- monkeypatch.setattr(httpx, "get", lambda *a, **k: resp)
+ monkeypatch.setattr(provider, "_safe_get_pdf", lambda url, settings: resp)
return _install
@@ -156,3 +155,77 @@ def test_scanned_pdf_without_text_layer_explains_itself(stub_httpx, caplog):
assert "no extractable text layer" in caplog.text
assert "OCR" in caplog.text
+
+
+def _cert_error() -> Exception:
+ import ssl
+
+ import httpx
+
+ err = httpx.ConnectError("TLS handshake failed")
+ err.__cause__ = ssl.SSLCertVerificationError("CERTIFICATE_VERIFY_FAILED")
+ return err
+
+
+def test_cert_error_refuses_with_no_unverified_retry(monkeypatch):
+ """A certificate error is a refusal, never an automatic unverified retry —
+ a MITM can serve a bad cert precisely to force such a retry. The refusal
+ names the pdf_verify_tls opt-out so a trusted cert-broken mirror stays
+ reachable by explicit operator decision."""
+ from hyperresearch.web.safe_http import SafeHTTPError
+
+ calls: list[bool] = []
+
+ def fake_safe_get(url, *, max_bytes, timeout=None, headers=None, verify=True,
+ allow_private_hosts=()):
+ calls.append(verify)
+ raise _cert_error()
+
+ monkeypatch.setattr("hyperresearch.web.safe_http.safe_get", fake_safe_get)
+
+ with pytest.raises(SafeHTTPError, match="pdf_verify_tls"):
+ provider._safe_get_pdf(
+ "https://broken-cert.example.edu/paper.pdf", provider.FetchSettings()
+ )
+
+ assert calls == [True], "exactly one verified attempt, no unverified retry"
+
+
+def test_non_cert_error_propagates_untranslated(monkeypatch):
+ """A refused connection is not a certificate problem — it must propagate
+ as-is, without the pdf_verify_tls hint."""
+ import httpx
+
+ calls: list[bool] = []
+
+ def fake_safe_get(url, *, max_bytes, timeout=None, headers=None, verify=True,
+ allow_private_hosts=()):
+ calls.append(verify)
+ raise httpx.ConnectError("connection refused")
+
+ monkeypatch.setattr("hyperresearch.web.safe_http.safe_get", fake_safe_get)
+
+ with pytest.raises(httpx.ConnectError):
+ provider._safe_get_pdf("https://down.example.com/x.pdf", provider.FetchSettings())
+
+ assert calls == [True]
+
+
+def test_pdf_verify_tls_false_fetches_unverified(monkeypatch):
+ """An explicit `pdf_verify_tls = false` in config is an operator decision:
+ fetch unverified directly. SSRF/size gates still apply (same safe_get)."""
+ calls: list[bool] = []
+
+ def fake_safe_get(url, *, max_bytes, timeout=None, headers=None, verify=True,
+ allow_private_hosts=()):
+ calls.append(verify)
+ return _Resp(b"%PDF- direct")
+
+ monkeypatch.setattr("hyperresearch.web.safe_http.safe_get", fake_safe_get)
+
+ resp = provider._safe_get_pdf(
+ "https://mirror.example.org/x.pdf", provider.FetchSettings(pdf_verify_tls=False)
+ )
+
+ assert calls == [False], "config opt-out must fetch unverified on the first attempt"
+ assert resp.content == b"%PDF- direct"
diff --git a/tests/test_web/test_safe_http.py b/tests/test_web/test_safe_http.py
new file mode 100644
index 0000000..c7a5487
--- /dev/null
+++ b/tests/test_web/test_safe_http.py
@@ -0,0 +1,384 @@
+"""Tests for the SSRF gate in hyperresearch.web.safe_http."""
+
+from __future__ import annotations
+
+import socket
+from unittest.mock import patch
+
+import httpx
+import pytest
+
+from hyperresearch.web.safe_http import (
+ MAX_BYTES_HTML,
+ SafeHTTPError,
+ check_url,
+ safe_get,
+)
+
+# ---------------------------------------------------------------------------
+# check_url — scheme and address validation
+# ---------------------------------------------------------------------------
+
+
+ "url",
+ [
+ "file:///etc/passwd",
+ "gopher://example.com/",
+ "ftp://example.com/x",
+ "data:text/html,<script>alert(1)</script>",
+ "javascript:alert(1)",
+ ],
+)
+def test_check_url_rejects_non_http_schemes(url):
+ with pytest.raises(SafeHTTPError, match="scheme"):
+ check_url(url)
+
+
+def test_check_url_rejects_no_hostname():
+ with pytest.raises(SafeHTTPError, match="no hostname"):
+ check_url("http:///path")
+
+
+ "url",
+ [
+ "http://127.0.0.1/",
+ "http://127.1.2.3/",
+ "http://10.0.0.1/",
+ "http://10.255.255.255/",
+ "http://172.16.0.1/",
+ "http://172.31.255.254/",
+ "http://192.168.1.1/",
+ "http://169.254.169.254/latest/meta-data/", # AWS IMDS
+ "http://0.0.0.0/",
+ "http://[::1]/",
+ "http://[fe80::1]/",
+ "http://[fc00::1]/",
+ ],
+)
+def test_check_url_rejects_private_and_loopback_ips(url):
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url(url)
+
+
+ "url",
+ [
+ "http://8.8.8.8/",
+ "https://1.1.1.1/",
+ "http://93.184.216.34/", # example.com at time of writing; literal so no DNS
+ ],
+)
+def test_check_url_allows_public_ip_literals(url):
+ check_url(url) # should not raise
+
+
+def test_check_url_dns_resolved_private_rejected():
+ """Hostnames that resolve to private addresses are refused — even when
+ the literal looks public."""
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("10.0.0.5", 0))]
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url("http://example.com/x")
+
+
+def test_check_url_dns_resolved_mixed_any_private_rejected():
+ """If even ONE of the resolved addresses is private, refuse."""
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [
+ (socket.AF_INET, None, None, "", ("8.8.8.8", 0)),
+ (socket.AF_INET, None, None, "", ("10.0.0.5", 0)),
+ ]
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url("http://example.com/x")
+
+
+def test_check_url_dns_failure_raises():
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.side_effect = socket.gaierror("nope")
+ with pytest.raises(SafeHTTPError, match="DNS lookup failed"):
+ check_url("http://nonexistent.invalid/x")
+
+
+# ---------------------------------------------------------------------------
+# check_url — encoded-loopback bypass vectors. These are the forms an attacker
+# reaches for once the plain "http://127.0.0.1/" literal is blocked. Grouped by
+# whether they resolve purely from the string (parse-only) or need the resolver.
+# ---------------------------------------------------------------------------
+
+
+ "url",
+ [
+ "http://[::ffff:127.0.0.1]/", # IPv4-mapped IPv6 — blocked as is_private
+ "http://[64:ff9b::7f00:1]/", # NAT64 well-known prefix — blocked as is_reserved
+ "http://[email protected]/", # userinfo trick: hostname is 127.0.0.1, not 8.8.8.8
+ ],
+)
+def test_check_url_rejects_parse_only_loopback_bypasses(url):
+ """These encode loopback/reserved space without a DNS lookup — the IP
+ literal or the real hostname is right there in the URL. If the address
+ classifier stops covering mapped/NAT64/reserved ranges, or urlparse's
+ hostname extraction is second-guessed, one of these silently reaches an
+ internal address."""
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url(url)
+
+
+ "url",
+ [
+ "http://2130706433/", # decimal 127.0.0.1
+ "http://0x7f000001/", # hex 127.0.0.1
+ "http://017700000001/", # octal 127.0.0.1
+ "http://localhost./", # trailing-dot hostname
+ ],
+)
+def test_check_url_rejects_resolver_dependent_loopback_bypasses(url):
+ """`ipaddress.ip_address` rejects these numeric encodings, so they fall to
+ the resolver, which normalizes them to 127.0.0.1. getaddrinfo is stubbed
+ so the test is deterministic and offline. This pins the contract that such
+ forms are *resolved and gated*, not parsed as public — a regression that
+ made `_resolve` parse the literal itself (e.g. via inet_aton) would need to
+ keep blocking them."""
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("127.0.0.1", 0))]
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url(url)
+
+
+def test_check_url_uppercase_http_scheme_is_normalized_then_ip_gated():
+ """An uppercase HTTP scheme must not fail open: it normalizes to http and
+ the loopback host is still refused by the address gate, not waved through
+ as an unrecognized scheme."""
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url("HTTP://127.0.0.1/")
+
+
+def test_check_url_uppercase_disallowed_scheme_still_rejected():
+ """Scheme matching is case-folded on the reject path too — FILE:// is
+ refused exactly like file://, not accidentally allowed by case."""
+ with pytest.raises(SafeHTTPError, match="scheme"):
+ check_url("FILE:///etc/passwd")
+
+
+# ---------------------------------------------------------------------------
+# safe_get — refuses bad URLs without even attempting a connection
+# ---------------------------------------------------------------------------
+
+
+def test_safe_get_refuses_private_ip_without_network():
+ """No network mocking needed — the SSRF gate must reject before any
+ socket is opened."""
+ with pytest.raises(SafeHTTPError):
+ safe_get("http://127.0.0.1/x", max_bytes=MAX_BYTES_HTML)
+
+
+def test_safe_get_refuses_file_scheme():
+ with pytest.raises(SafeHTTPError, match="scheme"):
+ safe_get("file:///etc/passwd", max_bytes=MAX_BYTES_HTML)
+
+
+def test_safe_get_requires_positive_max_bytes():
+ with pytest.raises(ValueError):
+ safe_get("http://example.com/", max_bytes=0)
+
+
+# ---------------------------------------------------------------------------
+# Redirect revalidation and size caps — exercised offline via the transport
+# seam (httpx.MockTransport), so no socket is ever opened. All URLs use
+# public IP literals so check_url never needs DNS.
+# ---------------------------------------------------------------------------
+
+
+def _transport(routes: dict[str, httpx.Response]) -> httpx.MockTransport:
+ """Map url-path -> canned response."""
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ return routes[request.url.path]
+
+ return httpx.MockTransport(handler)
+
+
+def test_redirect_to_private_address_is_refused():
+ """Every redirect hop goes back through the SSRF gate — a public host
+ must not be able to bounce the fetcher into loopback/RFC1918."""
+ transport = _transport({
+ "/start": httpx.Response(302, headers={"location": "http://127.0.0.1/secret"}),
+ })
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ safe_get("http://8.8.8.8/start", max_bytes=1024, transport=transport)
+
+
+def test_relative_redirect_is_resolved_and_followed():
+ transport = _transport({
+ "/a": httpx.Response(301, headers={"location": "/b"}),
+ "/b": httpx.Response(200, content=b"made it"),
+ })
+ resp = safe_get("http://8.8.8.8/a", max_bytes=1024, transport=transport)
+ assert resp.status_code == 200
+ assert resp.content == b"made it"
+ assert resp.url == "http://8.8.8.8/b"
+
+
+def test_too_many_redirects_refused():
+ transport = _transport({
+ "/loop": httpx.Response(302, headers={"location": "/loop"}),
+ })
+ with pytest.raises(SafeHTTPError, match="too many redirects"):
+ safe_get("http://8.8.8.8/loop", max_bytes=1024, transport=transport)
+
+
+def test_declared_content_length_over_cap_refused():
+ """An honest Content-Length over the cap is refused up front, before the
+ body is streamed. Response(content=...) auto-sets the header; the match
+ on "Content-Length" proves the header check fired, not the stream cap."""
+ transport = _transport({
+ "/big": httpx.Response(200, content=b"x" * 2048),
+ })
+ with pytest.raises(SafeHTTPError, match="Content-Length"):
+ safe_get("http://8.8.8.8/big", max_bytes=1024, transport=transport)
+
+
+def test_streamed_body_over_cap_refused():
+ """A body that exceeds the cap mid-stream (no Content-Length header, as
+ with chunked encoding) is cut off — a lying or chunked server cannot
+ exhaust memory. Uses a raw byte stream because Response(content=...)
+ would set Content-Length and trip the earlier header check instead."""
+
+ class _ChunkStream(httpx.SyncByteStream):
+ def __iter__(self):
+ for _ in range(4):
+ yield b"x" * 512
+
+ transport = _transport({
+ "/liar": httpx.Response(200, stream=_ChunkStream()),
+ })
+ with pytest.raises(SafeHTTPError, match="body exceeds cap"):
+ safe_get("http://8.8.8.8/liar", max_bytes=1024, transport=transport)
+
+
+def test_body_at_cap_passes():
+ transport = _transport({
+ "/fits": httpx.Response(200, content=b"x" * 1024),
+ })
+ resp = safe_get("http://8.8.8.8/fits", max_bytes=1024, transport=transport)
+ assert resp.content == b"x" * 1024
+
+
+# ---------------------------------------------------------------------------
+# CGNAT (RFC 6598) — 100.64.0.0/10 is not covered by any ipaddress is_*
+# property, so only the explicit network check refuses it
+# ---------------------------------------------------------------------------
+
+
[email protected]("url", ["http://100.64.0.1/", "http://100.127.255.254/"])
+def test_check_url_rejects_cgnat_range(url):
+ """Shared address space is never a public destination, yet it passes
+ every ipaddress ``is_*`` property (checked through 3.11.9)."""
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url(url)
+
+
[email protected]("url", ["http://100.63.255.255/", "http://100.128.0.1/"])
+def test_check_url_allows_cgnat_neighbors(url):
+ """The block is exactly /10 — addresses either side stay fetchable."""
+ check_url(url)
+
+
+# ---------------------------------------------------------------------------
+# 6to4 (RFC 3056) — 2002:AABB:CCDD::/48 embeds the IPv4 address A.B.C.D, so
+# the check must decode it; ipaddress does not flag the prefix on its own
+# ---------------------------------------------------------------------------
+
+
+ "url",
+ [
+ "http://[2002:7f00:1::]/", # 127.0.0.1
+ "http://[2002:c0a8:101::]/", # 192.168.1.1
+ "http://[2002:a9fe:a9fe::]/", # 169.254.169.254, cloud metadata
+ "http://[2002:6440:1::]/", # 100.64.0.1, CGNAT
+ ],
+)
+def test_check_url_rejects_6to4_wrapping_private_ipv4(url):
+ """A deprecated transport, but the embedded address is what would be
+ reached, and a literal in the URL skips DNS entirely."""
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url(url)
+
+
+def test_check_url_allows_6to4_wrapping_public_ipv4():
+ """2002:0808:0808:: embeds 8.8.8.8 and stays fetchable — the block is on
+ the embedded address, not the prefix."""
+ check_url("http://[2002:808:808::]/")
+
+
+# ---------------------------------------------------------------------------
+# allow_private_hosts — the [fetch] escape hatch for intranet mirrors
+# ---------------------------------------------------------------------------
+
+
+def test_allowlist_cidr_admits_private_address():
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("10.0.0.5", 0))]
+ check_url("http://mirror.example/x", allow_private_hosts=("10.0.0.0/8",))
+
+
+def test_allowlist_cidr_is_not_a_blanket_waiver():
+ """Allowlisting one private network must not admit OTHER private space."""
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("192.168.1.5", 0))]
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url("http://mirror.example/x", allow_private_hosts=("10.0.0.0/8",))
+
+
+def test_allowlist_bare_ip_entry_admits_exactly_one_host():
+ check_url("http://192.168.1.20/wiki", allow_private_hosts=("192.168.1.20",))
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url("http://192.168.1.21/wiki", allow_private_hosts=("192.168.1.20",))
+
+
+def test_allowlist_hostname_admits_by_name_case_insensitively():
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("10.0.0.5", 0))]
+ check_url("http://Papers.INTERNAL/x", allow_private_hosts=("papers.internal",))
+
+
+def test_allowlist_hostname_is_exact_not_a_suffix_match():
+ """``papers.internal`` must not admit ``evil-papers.internal``."""
+ with patch("hyperresearch.web.safe_http.socket.getaddrinfo") as gai:
+ gai.return_value = [(socket.AF_INET, None, None, "", ("10.0.0.5", 0))]
+ with pytest.raises(SafeHTTPError, match="non-public address"):
+ check_url(
+ "http://evil-papers.internal/x",
+ allow_private_hosts=("papers.internal",),
+ )
+
+
+def test_allowlist_does_not_bypass_the_scheme_gate():
+ with pytest.raises(SafeHTTPError, match="scheme"):
+ check_url("ftp://papers.internal/x", allow_private_hosts=("papers.internal",))
+
+
+def test_allowlist_malformed_cidr_is_a_loud_error():
+ """A typo'd CIDR must not silently degrade into "entry ignored" — that
+ failure mode looks exactly like an SSRF refusal of the mirror it was
+ supposed to admit."""
+ with pytest.raises(SafeHTTPError, match="invalid allow_private_hosts"):
+ check_url("http://8.8.8.8/", allow_private_hosts=("10.0.0.0/33",))
+
+
+def test_safe_get_redirect_to_allowlisted_private_host_is_followed():
+ """The allowlist is threaded into per-hop revalidation: a public host
+ redirecting onto an allowlisted mirror is followed, not refused."""
+ transport = _transport({
+ "/start": httpx.Response(302, headers={"location": "http://192.168.1.20/paper"}),
+ "/paper": httpx.Response(200, content=b"mirror content"),
+ })
+ resp = safe_get(
+ "http://8.8.8.8/start", max_bytes=1024, transport=transport,
+ allow_private_hosts=("192.168.1.20",),
+ )
+ assert resp.content == b"mirror content"