Skip to content

Universal SEO skill for Claude Code. 26 sub-skills + 19 sub-agents covering technical SEO, E-E-A-T, schema, GEO/AEO, agent readiness (Lighthouse Agentic Browsing, WebMCP, llms.txt), backlinks, local SEO, e-commerce, international SEO, Google APIs, and PDF/Excel reporting. 9 optional extensions for live SEO data.

feat(fetch): IPv6-only hosts, fetch timing and a wall-clock deadline

Agrici Daniel26507e00f467
diff --git a/SECURITY.md b/SECURITY.md
index 4d022fe..6032704 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -57,7 +57,7 @@ claude-seo is a research and audit toolkit that runs on a user's workstation. It
- **Playwright + Chromium DNS rebinding.** Chromium does its own DNS resolution inside the renderer process. claude-seo's Python-layer DNS pin (`url_safety._pin_dns`) cannot reach it. The Playwright `route()` handler re-validates every subresource host (`make_safe_playwright_route_handler`), which closes the common case, but a true rebinding attacker can still race Chromium's resolver after our pre-flight returns. Mitigation: do not point `/seo` skills at untrusted sites with high-frequency redirects.
- **CONNECT proxies weaken DNS pinning.** Behind a configured HTTP proxy (`HTTPS_PROXY` and friends) the proxy resolves the final host itself, so `url_safety`'s pin degrades to the pre-flight check in `validate_url_strict`; the proxy host itself is exempted from the pinned scope only after passing the same hostname blocklist and `is_safe_ip` policy as an audit target, so a proxy on loopback, RFC 1918, RFC 6598, link-local, or a metadata address is refused rather than exempted.
-- **IPv6-only audit targets.** The strict validator queries `family=AF_INET` for the initial resolution. Hosts with AAAA records only will surface as "DNS resolution failed". This is **fail-closed** by design: we'd rather refuse than connect to an unvalidated IPv6 endpoint. Tracked for a future patch (full dual-stack pinning, similar to the Playwright handler which already uses `AF_UNSPEC`).
+- **IPv6 audit targets.** The strict validator resolves both address families and refuses a host if any A or AAAA record is non-public. The pin is IPv4 when the host has an A record and IPv6 otherwise, with no fallback between families. NAT64 (64:ff9b::/96, 64:ff9b:1::/48), 6to4, Teredo, site-local, documentation, ORCHID and zone-scoped addresses are refused. One exception: on a DNS64 network, a synthesized AAAA record under 64:ff9b::/96 is ignored when the host also has an A record, because the pin is then IPv4 and that address is never dialled. A host with no A record gets no such pass, and redirect targets and browser subresources still refuse the prefix.
- **Windows file permissions.** `os.fchmod(fd, 0o600)` is a no-op on Windows for non-ACL filesystems. Users on Windows should rely on per-user directory ACLs instead of POSIX mode bits. `scripts/backlinks_auth.py` additionally makes a best-effort `icacls <path> /inheritance:r /grant:r "%USERNAME%:F"` call after every write to (and load-time remediation of) `~/.config/claude-seo/backlinks-api.json`, restricting it to the current user on NTFS. This is best-effort only: a failure (missing `icacls`, a non-NTFS volume, a restricted shell) is logged as a warning and does not block the credential read or write, so on a failure the file's effective protection on Windows falls back to whatever the containing directory's ACLs already provide.
- **RFC 6598 (100.64.0.0/10) is refused by default.** Cloud providers serve instance metadata in this range, so `url_safety` treats it like RFC 1918. Tailscale also allocates from it, so a staging site reached over Tailscale is refused under the default policy. The escape hatch is `CLAUDE_SEO_LOCAL_TARGETS`, a comma-separated allowlist of `host` or `host:port` entries (for example `localhost:3000,127.0.0.1:8080,100.101.102.103`, the last being a Tailscale address). Its scope is deliberately narrow:
diff --git a/scripts/fetch_page.py b/scripts/fetch_page.py
index 0238f9f..25b71d2 100755
--- a/scripts/fetch_page.py
+++ b/scripts/fetch_page.py
@@ -12,6 +12,21 @@ Usage:
python fetch_page.py https://example.com --json
python fetch_page.py https://example.com --render auto # SPA-aware
python fetch_page.py https://example.com --render always # force render
+ python fetch_page.py https://example.com --max-time 20 # total budget
+
+Timing
+======
+A raw fetch reports three durations in milliseconds: ``ttfb_ms``,
+``response_time_ms`` and ``fetch_duration_ms`` (see ``fetch_page``). They are
+one measurement taken from this machine, over this network, at this moment.
+They are not Core Web Vitals, not a field metric and not what a visitor's
+browser sees: use the CrUX and PageSpeed scripts for those. Treat them as a
+hint that a server is slow to answer, never as a score.
+
+The whole fetch, redirects and body included, also has a wall-clock budget
+(``--max-time``). ``--timeout`` limits each socket read on its own, so
+without the budget a server that sends one byte every few seconds could hold
+a run for hours.
"""
from __future__ import annotations
@@ -20,6 +35,7 @@ import argparse
import json
import os
import sys
+import time
from typing import Optional
try:
@@ -36,7 +52,7 @@ from url_safety import ( # noqa: E402
URLSafetyError,
decode_body,
decode_response_text,
- read_bounded,
+ read_bounded_until,
safe_requests_session,
validate_url_strict,
)
@@ -69,6 +85,24 @@ _decode_response_content = decode_response_text
# Largest body read, counted after decompression (shared with render_page).
MAX_PAGE_BYTES = _MAX_PAGE_BYTES
+# Timing keys of a raw fetch, in the order they are documented.
+_TIMING_KEYS = ("ttfb_ms", "response_time_ms", "fetch_duration_ms")
+
+
+def _default_max_time(timeout: float) -> float:
+ """Wall-clock budget for one fetch when the caller names none.
+
+ Twice the per-read timeout, and never under a minute: 60 seconds at the
+ default ``--timeout 30``. Tying it to the timeout means a caller who
+ raises the timeout for a slow server is not cut short by a fixed budget;
+ the floor keeps a short timeout from making the whole fetch impatient.
+ """
+ return max(60.0, 2.0 * timeout)
+
+
+def _ms(seconds: float) -> int:
+ return int(round(seconds * 1000))
+
def fetch_page(
url: str,
@@ -77,13 +111,14 @@ def fetch_page(
max_redirects: int = 5,
user_agent: Optional[str] = None,
max_bytes: int = MAX_PAGE_BYTES,
+ max_time: Optional[float] = None,
) -> dict:
"""
Fetch a web page and return response details.
SSRF protection is delegated to url_safety.validate_url_strict +
- safe_requests_session, which resolves DNS once, validates every A
- record against private/loopback/reserved ranges, and pins the
+ safe_requests_session, which resolves DNS once, validates every A and
+ AAAA record against private/loopback/reserved ranges, and pins the
connection so the resolver cannot rebind between checks.
Args:
@@ -95,11 +130,39 @@ def fetch_page(
max_bytes: Stop reading the body past this many bytes (after
decompression); ``too_large`` is then True and ``content`` holds
the part read.
+ max_time: Wall-clock budget in seconds for the whole fetch, every
+ redirect hop and the body included. Default: twice ``timeout``,
+ at least 60. It is checked when the final response's headers
+ arrive and after every piece of the body. When it passes,
+ ``error`` says so and ``content`` holds the part read. A stall
+ inside a connect, a header read or a redirect's own body is
+ still limited by ``timeout`` per socket read, not by this.
Returns:
Dictionary with url, status_code, content, headers, redirect_chain,
- redirect_details, too_large, and error.
+ redirect_details, too_large, error, and three timings in
+ milliseconds, each None when it was not reached:
+
+ - ``ttfb_ms``: time to first byte of the final response, from
+ sending that request (a new connection and its TLS handshake
+ included) to the end of its headers.
+ - ``response_time_ms``: the final request alone, from sending it to
+ the last body byte read. Redirect hops do not inflate it.
+ - ``fetch_duration_ms``: the whole call, URL validation and every
+ redirect hop included. Set on every outcome, errors too.
+
+ The timings are measured from this machine. They are not Core Web
+ Vitals and not a field metric.
"""
+ started = time.monotonic()
+ if max_time is None:
+ max_time = _default_max_time(timeout)
+ deadline = started + max_time
+
+ def finish() -> dict:
+ result["fetch_duration_ms"] = _ms(time.monotonic() - started)
+ return result
+
result: dict = {
"url": url,
"status_code": None,
@@ -109,6 +172,9 @@ def fetch_page(
"redirect_details": [],
"too_large": False,
"error": None,
+ "ttfb_ms": None,
+ "response_time_ms": None,
+ "fetch_duration_ms": None,
}
# Normalize scheme-less inputs (e.g. "example.com") before validation.
@@ -119,7 +185,7 @@ def fetch_page(
norm_url, _pinned_ip = validate_url_strict(url)
except URLSafetyError as exc:
result["error"] = f"url_safety: {exc}"
- return result
+ return finish()
result["url"] = norm_url
@@ -137,12 +203,27 @@ def fetch_page(
allow_redirects=follow_redirects,
stream=True,
)
- # Streamed so the body is read under the cap, inside the pinned
- # session; .content must not be read after this.
+ headers_at = time.monotonic()
+ # Streamed so the body is read under the cap and the deadline,
+ # inside the pinned session; .content must not be read after this.
try:
- raw, result["too_large"] = read_bounded(response, max_bytes)
+ raw, result["too_large"], timed_out = read_bounded_until(
+ response, max_bytes, deadline
+ )
finally:
response.close()
+ body_at = time.monotonic()
+
+ # requests times each request from sending it to the end of its
+ # headers, so the final response's ``elapsed`` leaves the hops out.
+ ttfb = response.elapsed.total_seconds()
+ result["ttfb_ms"] = _ms(ttfb)
+ result["response_time_ms"] = _ms(ttfb + body_at - headers_at)
+ if timed_out:
+ result["error"] = (
+ f"Fetch exceeded the {max_time:g} second wall-clock deadline; "
+ "content holds the part read before it"
+ )
result["url"] = response.url
result["status_code"] = response.status_code
@@ -171,7 +252,7 @@ def fetch_page(
# rebinding-pinned session is asked to chase it.
result["error"] = f"url_safety: {e}"
- return result
+ return finish()
def _non_negative_int(value: str) -> int:
@@ -181,6 +262,13 @@ def _non_negative_int(value: str) -> int:
return parsed
+def _positive_seconds(value: str) -> float:
+ parsed = float(value)
+ if not 0 < parsed < float("inf"):
+ raise argparse.ArgumentTypeError("must be a number of seconds above zero")
+ return parsed
+
+
def _as_render_result(result: dict, *, mode_used: str) -> dict:
"""Map a raw ``fetch_page()`` result onto the ``render_page()`` contract.
@@ -219,6 +307,7 @@ def _as_render_result(result: dict, *, mode_used: str) -> dict:
"mode_used": mode_used,
"too_large": bool(result.get("too_large")),
"error": result.get("error"),
+ **{key: result.get(key) for key in _TIMING_KEYS},
}
@@ -231,6 +320,11 @@ def _emit_json(result: dict, output: Optional[str], *, max_text: int = 0) -> Non
"""
from render_page import _json_summary
+ # A rendered result reports render_ms, not the raw-fetch timings; they
+ # are null there so both paths keep one key set.
+ for key in _TIMING_KEYS:
+ result.setdefault(key, None)
+
output_written = False
if output and not result.get("error"):
with open(output, "w", encoding="utf-8") as f:
@@ -258,6 +352,17 @@ def main():
),
)
parser.add_argument("--timeout", "-t", type=int, default=30, help="Timeout in seconds")
+ parser.add_argument(
+ "--max-time",
+ type=_positive_seconds,
+ default=None,
+ help=(
+ "Wall-clock budget in seconds for the whole fetch, redirects and "
+ "body included (default: twice --timeout, at least 60). When it "
+ "passes, the fetch stops with an error and the part read. "
+ "Raw HTTP only: --render is limited by --timeout."
+ ),
+ )
parser.add_argument("--no-redirects", action="store_true", help="Don't follow redirects")
parser.add_argument("--user-agent", help="Custom User-Agent string")
parser.add_argument(
@@ -319,6 +424,7 @@ def main():
timeout=args.timeout,
follow_redirects=not args.no_redirects,
user_agent=ua,
+ max_time=args.max_time,
)
if args.json:
diff --git a/scripts/url_safety.py b/scripts/url_safety.py
index 24af86b..45404aa 100644
--- a/scripts/url_safety.py
+++ b/scripts/url_safety.py
@@ -17,9 +17,11 @@ validate_url(url) -> bool
google_auth.py.
validate_url_strict(url) -> tuple[str, str]
- Resolves the hostname via socket.getaddrinfo, validates every returned
- A record against the safety predicate, and returns
- ``(normalized_url, pinned_ipv4)``. Raises ``URLSafetyError`` if any
+ Resolves the hostname via socket.getaddrinfo in both address families,
+ validates every returned A and AAAA record against the safety
+ predicate, and returns ``(normalized_url, pinned_ip)``. The pinned
+ address is IPv4 whenever the host has an A record, and IPv6 only for a
+ host that publishes AAAA records alone. Raises ``URLSafetyError`` if any
resolved IP is non-public. Use this whenever the caller is about to
open a network connection so DNS rebinding cannot swap a public IP
for a private one between checks.
@@ -41,7 +43,10 @@ safe_requests_session(url) -> context manager yielding requests.Session
is_safe_ip(ip_str) -> bool
True iff the address parses as IPv4/IPv6 and is none of:
- private, loopback, reserved, link-local, multicast, unspecified.
+ private, loopback, reserved, link-local, multicast, unspecified,
+ an IPv6 transition or special-purpose range (NAT64, 6to4, Teredo,
+ site-local, documentation, ORCHID), or an address with a zone
+ identifier.
URLSafetyError
ValueError subclass raised by the strict validator and pinning helpers.
@@ -91,12 +96,17 @@ import os
import re
import socket
import threading
+import time
from contextlib import contextmanager
from typing import Iterator, Optional
from urllib.parse import urlparse
try:
import requests
+
+ # urllib3 is the HTTP layer under requests, so it installs with it.
+ from urllib3 import exceptions as _urllib3_exceptions
+ from urllib3.response import BaseHTTPResponse as _Urllib3Response
except ImportError as exc: # pragma: no cover - hard dependency
raise RuntimeError(
"scripts/url_safety.py requires the 'requests' package. "
@@ -203,21 +213,56 @@ def _reject_authority_confusion(url: str, parsed) -> None:
# hand out 100.64/10 addresses for internal services.
_SHARED_ADDRESS_SPACE = ipaddress.ip_network("100.64.0.0/10")
+# IPv6 ranges that are never a public audit target. The ``ipaddress`` flags
+# used below already cover most of them, but those tables change between
+# Python releases (2001:20::/28 is no longer ``is_private``, 2002::/16 only
+# became so recently) and no flag covers site-local, so the policy is spelled
+# out here and holds on every interpreter. The ranges that embed an IPv4
+# address (NAT64, 6to4, Teredo, IPv4-compatible) are refused whole rather
+# than judged by that address. The one place a NAT64 address is passed over
+# is the DNS64 case in ``validate_url_strict``, where it is never dialled.
+_NAT64_WELL_KNOWN = ipaddress.ip_network("64:ff9b::/96")
+_UNSAFE_IPV6_NETWORKS = tuple(
+ ipaddress.ip_network(network)
+ for network in (
+ "::/96", # IPv4-compatible (deprecated); holds :: and ::1
+ "64:ff9b::/96", # NAT64 well-known prefix (RFC 6052)
+ "64:ff9b:1::/48", # NAT64 local-use prefix (RFC 8215)
+ "2001::/32", # Teredo
+ "2001:10::/28", # ORCHID (deprecated)
+ "2001:20::/28", # ORCHIDv2
+ "2001:db8::/32", # documentation
+ "2002::/16", # 6to4
+ "3fff::/20", # documentation (RFC 9637)
+ "fc00::/7", # unique-local; holds AWS IMDS fd00:ec2::254
+ "fe80::/10", # link-local
+ "fec0::/10", # site-local (deprecated)
+ "ff00::/8", # multicast
+ )
+)
+
def is_safe_ip(ip_str: str) -> bool:
"""Return True iff ``ip_str`` is a public unicast address.
IPv4-mapped IPv6 (``::ffff:127.0.0.1``) is unwrapped and judged as the
embedded IPv4 address, so every IPv4 rule below applies to it too.
- IPv6 unique-local (``fc00::/7``) and link-local (``fe80::/10``) are
- also rejected.
+ Every other IPv6 address is checked against ``_UNSAFE_IPV6_NETWORKS``
+ as well as the flags below. An address with a zone identifier
+ (``fe80::1%eth0``) is refused: a zone only means something on the
+ machine that names it.
"""
try:
ip = ipaddress.ip_address(ip_str)
except ValueError:
return False
- if ip.version == 6 and ip.ipv4_mapped is not None:
- ip = ip.ipv4_mapped
+ if ip.version == 6:
+ if ip.scope_id is not None:
+ return False
+ if ip.ipv4_mapped is not None:
+ ip = ip.ipv4_mapped
+ elif any(ip in network for network in _UNSAFE_IPV6_NETWORKS):
+ return False
return not (
ip.is_private
or (ip.version == 4 and ip in _SHARED_ADDRESS_SPACE)
@@ -290,6 +335,8 @@ def _is_allowlistable_ip(ip_str: str) -> bool:
ip = ipaddress.ip_address(ip_str)
except ValueError:
return False
+ if ip.version == 6 and ip.scope_id is not None:
+ return False # Zone identifier, refused as in is_safe_ip.
if ip.version == 6 and ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
if str(ip) in _NEVER_ALLOWLISTABLE_HOSTNAMES:
@@ -468,19 +515,41 @@ def validate_url(url: str) -> bool:
return allowlisted and _is_allowlistable_ip(hostname)
+def _is_dns64_synthesized(ip_str: str) -> bool:
+ """True for an address in the NAT64 well-known prefix, 64:ff9b::/96.
+
+ The local-use prefix 64:ff9b:1::/48 and anything with a zone identifier
+ do not count: neither is what a DNS64 resolver hands out by default.
+ """
+ try:
+ ip = ipaddress.ip_address(ip_str)
+ except ValueError:
+ return False
+ return ip.version == 6 and ip.scope_id is None and ip in _NAT64_WELL_KNOWN
+
+
def validate_url_strict(url: str) -> tuple[str, str]:
"""
Resolve and validate the URL's hostname.
- Returns ``(url, pinned_ipv4)`` on success. Raises ``URLSafetyError`` if:
+ Returns ``(url, pinned_ip)`` on success. Raises ``URLSafetyError`` if:
- Scheme is invalid or hostname is missing.
- Hostname is hard-blocked.
- DNS resolution fails.
- - Any A record resolves to a non-public IP (DNS rebinding refused).
-
- Multi-A-record handling: every returned record must be public. A
- hostname with one public and one private A record is refused so an
- attacker cannot race the resolver between validate and connect.
+ - Any A or AAAA record resolves to a non-public IP (DNS rebinding
+ refused).
+
+ Multi-record handling: every returned record must be public, in both
+ address families. A hostname with one public and one private record
+ (two A records, or a public A beside a private AAAA) is refused so an
+ attacker cannot race the resolver between validate and connect. The one
+ record passed over is a DNS64 AAAA under 64:ff9b::/96 on a host that
+ also has an A record: the pin is then IPv4 and that address is unused.
+
+ The pinned address is the host's IPv4 address when it has one, so a
+ dual-stack host is reached exactly as before. An IPv6 address is pinned
+ only for a host that publishes AAAA records alone, or for an IPv6
+ literal, which keeps its brackets in the returned URL.
"""
parsed = urlparse(url)
_reject_authority_confusion(url, parsed)
@@ -518,21 +587,36 @@ def validate_url_strict(url: str) -> tuple[str, str]:
addrinfo = socket.getaddrinfo(
hostname,
port,
- family=socket.AF_INET,
+ family=socket.AF_UNSPEC,
type=socket.SOCK_STREAM,
)
except (socket.gaierror, UnicodeError) as exc:
raise URLSafetyError(f"DNS resolution failed for {hostname}: {exc}") from exc
- resolved_ips = sorted({info[4][0] for info in addrinfo})
+ # IPv4 sorts first, so the pin below is IPv6 only when there is no A
+ # record. Every address in the list is still checked, whichever is pinned.
+ resolved_ips = sorted(
+ {info[4][0] for info in addrinfo}, key=lambda ip: (":" in ip, ip)
+ )
if not resolved_ips:
- raise URLSafetyError(f"No A records for {hostname}")
+ raise URLSafetyError(f"No address records for {hostname}")
+
+ # On a DNS64 network the resolver adds, to a name that has an A record
+ # and no AAAA, a synthesized AAAA record under 64:ff9b::/96. Judged like any
+ # other record it would refuse every IPv4-only site, which resolved fine
+ # while this lookup asked for A records alone. With an A record present
+ # the pin is IPv4, so that AAAA is never dialled: it is skipped here, not
+ # trusted. ``is_safe_ip`` still refuses the prefix everywhere else, and a
+ # host with no A record gets no such pass.
+ has_a_record = ":" not in resolved_ips[0]
for ip_str in resolved_ips:
if is_safe_ip(ip_str):
continue
if allowlisted and _is_allowlistable_ip(ip_str):
continue # Named in CLAUDE_SEO_LOCAL_TARGETS; see that block.
+ if has_a_record and _is_dns64_synthesized(ip_str):
+ continue
raise URLSafetyError(
f"DNS rebinding refused: {hostname} resolves to "
f"non-public IP {ip_str}"
@@ -695,6 +779,12 @@ def _pin_dns(
other port the allowlist names for that host; see
:func:`_local_pin_allows_port`.
+ The pin answers in the family of ``pinned_ip``: an ``AF_INET`` tuple for
+ an IPv4 address, an ``AF_INET6`` tuple with the four-part sockaddr for an
+ IPv6 one. A lookup that asks for the other family only is refused, so a
+ machine without IPv6 fails closed on an IPv6-only host instead of being
+ handed an address it cannot connect to.
+
``exempt_hosts`` are resolved by the real resolver without the
fall-through check. It carries the configured HTTP proxy, if any: the
proxy is the process's own egress, and requests must resolve it to open
@@ -726,6 +816,7 @@ def _pin_dns(
# Such a pin must not answer for the host's other ports, or a redirect
# from the allowed dev server reaches every service on that address.
port_bound = not is_safe_ip(pinned_ip)
+ pinned_family = socket.AF_INET6 if ":" in pinned_ip else socket.AF_INET
def patched(host, requested_port, *args, **kwargs):
# Branch 1: the originally-pinned host returns the validated IP
@@ -738,18 +829,22 @@ def _pin_dns(
f"non-public host {host}; validated port is {port}",
)
family = kwargs.get("family", args[0] if args else 0)
- if family in (0, socket.AF_UNSPEC, socket.AF_INET):
+ if family in (0, socket.AF_UNSPEC, pinned_family):
+ sockaddr = (pinned_ip, requested_port or port)
+ if pinned_family == socket.AF_INET6:
+ sockaddr += (0, 0) # flowinfo, scope_id
return [(
- socket.AF_INET,
+ pinned_family,
socket.SOCK_STREAM,
socket.IPPROTO_TCP,
"",
- (pinned_ip, requested_port or port),
+ sockaddr,
)]
+ pinned_kind = "IPv6" if pinned_family == socket.AF_INET6 else "IPv4"
raise socket.gaierror(
socket.EAI_FAIL,
f"url_safety: address family {family} refused for pinned "
- f"IPv4 host {host}",
+ f"{pinned_kind} host {host}",
)
if host and host.lower() in exempt_hosts:
@@ -952,18 +1047,73 @@ def read_bounded(response, max_bytes: int) -> tuple[bytes, bool]:
before the chunk that crossed the cap, and whether the cap was hit.
Decode the result with :func:`decode_body`, not ``response.content``.
"""
- chunks = []
+ raw, too_large, _timed_out = _read_capped(
+ response.iter_content(chunk_size=65536), max_bytes, None
+ )
+ return raw, too_large
+
+
+def read_bounded_until(
+ response, max_bytes: int, deadline: float
+) -> tuple[bytes, bool, bool]:
+ """:func:`read_bounded` with a wall-clock deadline as well.
+
+ ``deadline`` is a ``time.monotonic()`` value. It is checked before the
+ first read and after every piece of the body, and the body is taken as
+ each socket read delivers it, so a server that drips a byte at a time is
+ cut off at the deadline plus at most one socket read timeout. Returns the
+ bytes read so far, whether the size cap was hit, and whether the deadline
+ passed.
+ """
+ return _read_capped(_iter_as_received(response), max_bytes, deadline)
+
+
+def _read_capped(
+ chunks, max_bytes: int, deadline: Optional[float]
+) -> tuple[bytes, bool, bool]:
+ """Join ``chunks`` until the size cap or the deadline stops the read."""
+ parts: list[bytes] = []
size = 0
- too_large = False
- for chunk in response.iter_content(chunk_size=65536):
- if not chunk:
- continue
- size += len(chunk)
- if size > max_bytes:
- too_large = True
- break
- chunks.append(chunk)
- return b"".join(chunks), too_large
+ if deadline is not None and time.monotonic() >= deadline:
+ return b"", False, True
+ for chunk in chunks:
+ if chunk:
+ size += len(chunk)
+ if size > max_bytes:
+ return b"".join(parts), True, False
+ parts.append(chunk)
+ if deadline is not None and time.monotonic() >= deadline:
+ return b"".join(parts), False, True
+ return b"".join(parts), False, False
+
+
+def _iter_as_received(response, chunk_size: int = 65536) -> Iterator[bytes]:
+ """Yield a streamed body as each socket read delivers it.
+
+ ``iter_content(n)`` blocks until ``n`` bytes have arrived, so a slow drip
+ could hold one 64 KiB chunk for hours between two deadline checks.
+ urllib3's ``read1`` returns what a single read brings, still decompressed
+ and at most ``chunk_size`` bytes. Errors are mapped to the requests
+ exceptions ``iter_content`` raises.
+ """
+ raw = getattr(response, "raw", None)
+ if not isinstance(raw, _Urllib3Response): # A test double, not a socket.
+ yield from response.iter_content(chunk_size=chunk_size)
+ return
+ try:
+ while True:
+ data = raw.read1(chunk_size, decode_content=True)
+ if not data:
+ return
+ yield data
+ except _urllib3_exceptions.ProtocolError as exc:
+ raise requests.exceptions.ChunkedEncodingError(exc) from exc
+ except _urllib3_exceptions.DecodeError as exc:
+ raise requests.exceptions.ContentDecodingError(exc) from exc
+ except _urllib3_exceptions.ReadTimeoutError as exc:
+ raise requests.exceptions.ConnectionError(exc) from exc
+ except _urllib3_exceptions.SSLError as exc:
+ raise requests.exceptions.SSLError(exc) from exc
# Deepest JSON nesting a fetched document may have. Real discovery and
@@ -1240,7 +1390,7 @@ def _cli() -> None:
parser.add_argument(
"--strict",
action="store_true",
- help="Run DNS resolution and refuse on any non-public A record.",
+ help="Run DNS resolution and refuse on any non-public A or AAAA record.",
)
parser.add_argument(
"--json",
diff --git a/tests/test_fetch_page_cli.py b/tests/test_fetch_page_cli.py
index 182bf1d..b12a14a 100644
--- a/tests/test_fetch_page_cli.py
+++ b/tests/test_fetch_page_cli.py
@@ -79,6 +79,9 @@ def _expected_json(raw_source: dict, *, mode: str, output_written: bool, max_tex
raw_source if mode != "never" else fetch_page._as_render_result(raw_source, mode_used="raw")
)
summary = render_page._json_summary(normalized, max_text=max_text)
+ # Raw-fetch timings: null on a rendered result, so both paths share keys.
+ for key in fetch_page._TIMING_KEYS:
+ summary.setdefault(key, None)
summary["output_written"] = output_written
return json.loads(json.dumps(summary, default=str))
diff --git a/tests/test_fetch_page_timing.py b/tests/test_fetch_page_timing.py
new file mode 100644
index 0000000..8a7ac69
--- /dev/null
+++ b/tests/test_fetch_page_timing.py
@@ -0,0 +1,444 @@
+"""fetch_page.py: the wall-clock deadline and the timing fields.
+
+``--timeout`` limits each socket read on its own, so a server that sends one
+byte every few seconds was never cut off. The deadline covers the whole
+fetch. The slow servers here are real local stubs, reached through the narrow
+CLAUDE_SEO_LOCAL_TARGETS allowlist the way ``test_redirect_body_cap.py`` does;
+no network is used.
+"""
+
+from __future__ import annotations
+
+import gzip
+import http.server
+import io
+import json
+import socket
+import sys
+import threading
+import time
+from contextlib import contextmanager
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+import requests
+from urllib3.response import HTTPResponse
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
+import fetch_page # noqa: E402
+import render_page # noqa: E402
+import url_safety # noqa: E402
+
+DRIP_BYTES = 400
+DRIP_INTERVAL = 0.02 # the whole drip takes about 8 seconds if nothing stops it
+TIMING_KEYS = ("ttfb_ms", "response_time_ms", "fetch_duration_ms")
+
+
+class _Handler(http.server.BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def do_GET(self): # noqa: N802 (http.server API)
+ if self.path == "/drip":
+ self._headers(200, {"Content-Length": str(DRIP_BYTES)})
+ self._drip(lambda: self.wfile.write(b"x"))
+ elif self.path == "/drip-chunked":
+ self._headers(200, {"Transfer-Encoding": "chunked"})
+ self._drip(lambda: self.wfile.write(b"1\r\nx\r\n"))
+ elif self.path == "/slow-redirect":
+ time.sleep(0.2)
+ self._headers(302, {"Location": "/slow-final", "Content-Length": "0"})
+ elif self.path == "/slow-final":
+ time.sleep(0.1)
+ self._headers(200, {"Content-Length": "5"})
+ time.sleep(0.15)
+ self.wfile.write(b"final")
+ else:
+ self._headers(200, {"Content-Length": "5"})
+ self.wfile.write(b"plain")
+
+ def _headers(self, status: int, headers: dict) -> None:
+ self.send_response(status)
+ self.send_header("Content-Type", "text/html; charset=utf-8")
+ self.send_header("Connection", "close")
+ for name, value in headers.items():
+ self.send_header(name, value)
+ self.end_headers()
+ self.close_connection = True
+
+ def _drip(self, send_one) -> None:
+ try:
+ for _ in range(DRIP_BYTES):
+ if self.server.stop.is_set():
+ return
+ send_one()
+ time.sleep(DRIP_INTERVAL)
+ except OSError:
+ pass # The client hung up, which is the point of the test.
+
+ def log_message(self, *args):
+ pass
+
+
+def server(monkeypatch):
+ httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
+ httpd.daemon_threads = True
+ httpd.stop = threading.Event()
+ thread = threading.Thread(target=httpd.serve_forever, daemon=True)
+ thread.start()
+ port = httpd.server_address[1]
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", f"127.0.0.1:{port}")
+ monkeypatch.setattr(url_safety.requests.utils, "getproxies", lambda: {})
+ try:
+ yield f"http://127.0.0.1:{port}"
+ finally:
+ httpd.stop.set()
+ httpd.shutdown()
+ httpd.server_close()
+
+
+# ---------------------------------------------------------------------------
+# The deadline against a real slow server
+# ---------------------------------------------------------------------------
+
+
[email protected]("path", ["/drip", "/drip-chunked"])
+def test_dripping_body_is_cut_off_at_the_deadline(server, path):
+ """One byte every 20 ms never trips the 5 second read timeout. With a
+ Content-Length body, a 64 KiB chunked read would also sit through the
+ whole drip before the deadline was looked at again."""
+ began = time.monotonic()
+ result = fetch_page.fetch_page(f"{server}{path}", timeout=5, max_time=0.4)
+ took = time.monotonic() - began
+
+ assert took < 4, f"the drip held the fetch for {took:.1f}s"
+ assert "0.4 second wall-clock deadline" in result["error"]
+ assert result["status_code"] == 200
+ assert result["too_large"] is False
+ # The partial body is returned, the way the size cap returns its part.
+ assert 0 < len(result["content"]) < DRIP_BYTES
+ assert set(result["content"]) == {"x"}
+ assert result["fetch_duration_ms"] >= 400
+ assert result["response_time_ms"] >= 300
+ assert result["ttfb_ms"] is not None
+
+
+def test_fast_page_is_untouched_by_the_deadline(server):
+ result = fetch_page.fetch_page(f"{server}/plain", timeout=5)
+ assert result["error"] is None
+ assert result["content"] == "plain"
+ assert result["too_large"] is False
+
+
+def test_timing_fields_separate_the_final_request_from_the_redirect_hops(server):
+ """The stub waits 0.2 s before the redirect, 0.1 s before the final
+ headers and 0.15 s before the final body. Only lower bounds are asserted:
+ a loaded machine can be slower, never faster."""
+ result = fetch_page.fetch_page(f"{server}/slow-redirect", timeout=5)
+
+ assert result["error"] is None
+ assert result["content"] == "final"
+ assert result["redirect_details"] == [{"url": f"{server}/slow-redirect", "status_code": 302}]
+ for key in TIMING_KEYS:
+ assert isinstance(result[key], int), key
+ # Time to first byte of the final response: its 0.1 s wait, not the hop's.
+ assert result["ttfb_ms"] >= 95
+ # The final request alone: headers, then the 0.15 s before the body.
+ assert result["response_time_ms"] >= result["ttfb_ms"] + 140
+ # Every hop: the 0.2 s redirect is in this number and in no other.
+ assert result["fetch_duration_ms"] >= result["response_time_ms"] + 190
+
+
+# ---------------------------------------------------------------------------
+# read_bounded_until, with a clock the test controls
+# ---------------------------------------------------------------------------
+
+
+class _Clock:
+ def __init__(self) -> None:
+ self.now = 0.0
+
+ def monotonic(self) -> float:
+ return self.now
+
+
+def _ticking_response(clock: _Clock, chunks: list, calls: list | None = None):
+ """A response double whose every chunk takes one second to arrive."""
+ def iter_content(chunk_size=None):
+ if calls is not None:
+ calls.append(chunk_size)
+ for chunk in chunks:
+ clock.now += 1.0
+ yield chunk
+
+ return SimpleNamespace(raw=None, iter_content=iter_content)
+
+
+def clock(monkeypatch):
+ fake = _Clock()
+ monkeypatch.setattr(url_safety, "time", fake)
+ return fake
+
+
+def test_read_stops_at_the_deadline_and_keeps_what_arrived(clock):
+ response = _ticking_response(clock, [b"ab"] * 10)
+ assert url_safety.read_bounded_until(response, 1000, 3.0) == (b"ababab", False, True)
+
+
+def test_read_past_the_deadline_never_starts(clock):
+ calls: list = []
+ clock.now = 5.0
+ response = _ticking_response(clock, [b"ab"], calls)
+ assert url_safety.read_bounded_until(response, 1000, 5.0) == (b"", False, True)
+ assert calls == []
+
+
+def test_complete_body_inside_the_deadline_is_not_flagged(clock):
+ response = _ticking_response(clock, [b"ab", b"", b"cd"])
+ assert url_safety.read_bounded_until(response, 1000, 60.0) == (b"abcd", False, False)
+
+
+def test_size_cap_still_wins_when_it_is_hit_first(clock):
+ response = _ticking_response(clock, [b"ab"] * 10)
+ assert url_safety.read_bounded_until(response, 3, 60.0) == (b"ab", True, False)
+
+
+def test_read_bounded_keeps_its_two_value_contract_and_has_no_deadline(clock):
+ response = _ticking_response(clock, [b"ab"] * 10)
+ clock.now = 10_000.0
+ assert url_safety.read_bounded(response, 1000) == (b"ab" * 10, False)
+
+
+def _streamed_response(body: bytes, *, gzipped: bool = False, headers: dict | None = None,
+ fp=None) -> requests.Response:
+ """A streamed requests.Response built the way the HTTP adapter builds one."""
+ all_headers = {"Content-Type": "text/html; charset=utf-8", **(headers or {})}
+ payload = body
+ if gzipped:
+ payload = gzip.compress(body)
+ all_headers["Content-Encoding"] = "gzip"
+ raw = HTTPResponse(
+ body=fp or io.BytesIO(payload), headers=all_headers, status=200, preload_content=False
+ )
+ resp = requests.Response()
+ resp.status_code = 200
+ resp.headers.update(all_headers)
+ resp.raw = raw
+ resp.url = "https://example.com/"
+ return resp
+
+
+def test_deadline_read_still_counts_decompressed_bytes():
+ far = time.monotonic() + 60
+ bomb = _streamed_response(b"A" * (8 * 1024 * 1024), gzipped=True)
+ raw, too_large, timed_out = url_safety.read_bounded_until(bomb, 1024 * 1024, far)
+ assert too_large is True and timed_out is False
+ assert 0 < len(raw) <= 1024 * 1024
+
+ page = _streamed_response(b"B" * 300_000, gzipped=True)
+ assert url_safety.read_bounded_until(page, 1024 * 1024, far) == (b"B" * 300_000, False, False)
+
+
+class _TimingOutBody(io.BytesIO):
+ def read1(self, *args):
+ raise socket.timeout("timed out")
+
+ read = read1
+
+
+def test_deadline_read_raises_the_same_requests_errors_as_iter_content():
+ """fetch_page catches requests exceptions only, so a urllib3 error from
+ the direct read must not escape as itself."""
+ far = time.monotonic() + 60
+ stalled = _streamed_response(b"", fp=_TimingOutBody())
+ with pytest.raises(requests.exceptions.ConnectionError, match="Read timed out"):
+ url_safety.read_bounded_until(stalled, 1000, far)
+
+ short = _streamed_response(b"abc", headers={"Content-Length": "10"})
+ with pytest.raises(requests.exceptions.ChunkedEncodingError):
+ url_safety.read_bounded_until(short, 1000, far)
+
+ garbage = _streamed_response(b"not gzip", headers={"Content-Encoding": "gzip"})
+ with pytest.raises(requests.exceptions.ContentDecodingError):
+ url_safety.read_bounded_until(garbage, 1000, far)
+
+
+# ---------------------------------------------------------------------------
+# fetch_page(): budget, result keys, error paths
+# ---------------------------------------------------------------------------
+
+
+def _patched_fetch(monkeypatch, response: requests.Response, **kwargs) -> dict:
+ class _Session:
+ max_redirects = 5
+
+ def get(self, url, **kw):
+ return response
+
+ @contextmanager
+ def fake_session(_url):
+ yield _Session()
+
+ monkeypatch.setattr(fetch_page, "validate_url_strict", lambda url: (url, "93.184.216.34"))
+ monkeypatch.setattr(fetch_page, "safe_requests_session", fake_session)
+ return fetch_page.fetch_page("https://example.com/", **kwargs)
+
+
+ "timeout,expected", [(30, 60.0), (10, 60.0), (1, 60.0), (45, 90.0), (120, 240.0)]
+)
+def test_default_budget_is_twice_the_timeout_and_at_least_a_minute(timeout, expected):
+ assert fetch_page._default_max_time(timeout) == expected
+
+
+def test_budget_spent_before_the_body_leaves_the_body_unread(monkeypatch):
+ """Redirect hops count: if they used the budget, the body is not started."""
+ page = _streamed_response(b"<html>never read</html>")
+ result = _patched_fetch(monkeypatch, page, max_time=1e-9)
+ assert "wall-clock deadline" in result["error"]
+ assert result["content"] == ""
+ assert result["status_code"] == 200
+ assert page.raw.tell() == 0
+
+
+def test_default_budget_is_used_when_none_is_given(monkeypatch):
+ seen: list = []
+
+ def tiny(timeout):
+ seen.append(timeout)
+ return 1e-9
+
+ monkeypatch.setattr(fetch_page, "_default_max_time", tiny)
+ result = _patched_fetch(monkeypatch, _streamed_response(b"<html></html>"), timeout=17)
+ assert seen == [17]
+ assert "wall-clock deadline" in result["error"]
+
+
+def test_result_carries_integer_timings_on_success(monkeypatch):
+ result = _patched_fetch(monkeypatch, _streamed_response(b"<html>ok</html>"))
+ assert result["error"] is None
+ assert result["content"] == "<html>ok</html>"
+ for key in TIMING_KEYS:
+ assert isinstance(result[key], int) and result[key] >= 0, key
+
+
+def test_refused_url_reports_only_the_total(monkeypatch):
+ monkeypatch.delenv("CLAUDE_SEO_LOCAL_TARGETS", raising=False)
+ result = fetch_page.fetch_page("http://127.0.0.1:1/")
+ assert result["error"].startswith("url_safety:")
+ assert result["ttfb_ms"] is None
+ assert result["response_time_ms"] is None
+ assert isinstance(result["fetch_duration_ms"], int)
+
+
+def test_read_error_reports_only_the_total(monkeypatch):
+ stalled = _streamed_response(b"", fp=_TimingOutBody())
+ result = _patched_fetch(monkeypatch, stalled)
+ assert result["error"].startswith("Connection error:")
+ assert result["ttfb_ms"] is None
+ assert isinstance(result["fetch_duration_ms"], int)
+
+
+def test_timing_is_labelled_as_a_local_measurement():
+ for doc in (fetch_page.__doc__, fetch_page.fetch_page.__doc__):
+ assert "from this machine" in doc
+ assert "not Core Web Vitals" in " ".join(doc.split())
+ assert "field metric" in doc
+
+
+# ---------------------------------------------------------------------------
+# CLI: --max-time and the JSON keys
+# ---------------------------------------------------------------------------
+
+_RAW = {
+ "url": "https://example.com/",
+ "status_code": 200,
+ "content": "<p>partial",
+ "headers": {},
+ "redirect_chain": [],
+ "redirect_details": [],
+ "too_large": False,
+ "error": None,
+ "ttfb_ms": 120,
+ "response_time_ms": 340,
+ "fetch_duration_ms": 910,
+}
+
+
+def _run_cli(monkeypatch, capsys, argv: list, result: dict):
+ seen: dict = {}
+
+ def fake_fetch(*args, **kwargs):
+ seen.update(kwargs)
+ return dict(result)
+
+ monkeypatch.setattr(fetch_page, "fetch_page", fake_fetch)
+ monkeypatch.setattr(sys, "argv", ["fetch_page.py", "https://example.com", *argv])
+ code = 0
+ try:
+ fetch_page.main()
+ except SystemExit as exc:
+ code = exc.code
+ return code, seen, capsys.readouterr()
+
+
+def test_cli_passes_max_time_through(monkeypatch, capsys):
+ _code, seen, _out = _run_cli(monkeypatch, capsys, ["--max-time", "7.5"], _RAW)
+ assert seen["max_time"] == 7.5
+ _code, seen, _out = _run_cli(monkeypatch, capsys, [], _RAW)
+ assert seen["max_time"] is None # fetch_page() then applies the default
+
+
[email protected]("value", ["0", "-5", "nan", "inf", "soon"])
+def test_cli_rejects_a_budget_that_is_not_a_positive_number(monkeypatch, capsys, value):
+ code, seen, captured = _run_cli(monkeypatch, capsys, ["--max-time", value], _RAW)
+ assert code == 2
+ assert seen == {}
+ assert "--max-time" in captured.err
+
+
+def test_cli_json_reports_the_timings_of_a_raw_fetch(monkeypatch, capsys):
+ code, _seen, captured = _run_cli(monkeypatch, capsys, ["--json"], _RAW)
+ payload = json.loads(captured.out)
+ assert code == 0
+ assert {key: payload[key] for key in TIMING_KEYS} == {
+ "ttfb_ms": 120,
+ "response_time_ms": 340,
+ "fetch_duration_ms": 910,
+ }
+
+
+def test_cli_json_keeps_one_key_set_for_a_rendered_fetch(monkeypatch, capsys):
+ """A rendered result has render_ms instead; the raw timings are null."""
+ rendered = {
+ "url": "https://example.com/", "status_code": 200, "content": "<p>x</p>",
+ "raw_content": "<p>x</p>", "render_ms": 12.5, "mode_used": "rendered", "error": None,
+ }
+ monkeypatch.setattr(render_page, "render_page", lambda *a, **kw: dict(rendered))
+ monkeypatch.setattr(
+ sys, "argv", ["fetch_page.py", "https://example.com", "--render", "always", "--json"]
+ )
+ with pytest.raises(SystemExit) as exc:
+ fetch_page.main()
+ payload = json.loads(capsys.readouterr().out)
+ assert exc.value.code == 0
+ assert {key: payload[key] for key in TIMING_KEYS} == dict.fromkeys(TIMING_KEYS)
+ assert payload["render_ms"] == 12.5
+
+
+def test_cli_deadline_is_an_error_that_keeps_the_partial_content(monkeypatch, capsys, tmp_path):
+ cut = dict(_RAW, error="Fetch exceeded the 5 second wall-clock deadline; "
+ "content holds the part read before it")
+ output = tmp_path / "page.html"
+ code, _seen, captured = _run_cli(
+ monkeypatch, capsys, ["--json", "--output", str(output)], cut
+ )
+ payload = json.loads(captured.out)
+ assert code == 1
+ assert "wall-clock deadline" in payload["error"]
+ assert payload["content"] == "<p>partial"
+ # A cut-off page is never saved as if it were the page.
+ assert payload["output_written"] is False
+ assert not output.exists()
diff --git a/tests/test_url_safety_ipv6.py b/tests/test_url_safety_ipv6.py
new file mode 100644
index 0000000..7bc1897
--- /dev/null
+++ b/tests/test_url_safety_ipv6.py
@@ -0,0 +1,819 @@
+"""
+IPv6 in scripts/url_safety.py: the address policy, dual-family resolution,
+and a DNS pin that answers in the pinned address's own family.
+
+The resolver and ``socket.socket`` are replaced throughout, and the end-to-end
+cases drive the real ``requests`` and ``urllib3`` stack the way
+``test_url_safety_port_pin.py`` does. No test here touches the network.
+"""
+
+from __future__ import annotations
+
+import io
+import ipaddress
+import os
+import socket
+import sys
+from unittest.mock import patch
+
+import pytest
+
+_SCRIPTS = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "scripts")
+if _SCRIPTS not in sys.path:
+ sys.path.insert(0, _SCRIPTS)
+
+requests = pytest.importorskip("requests")
+import url_safety # noqa: E402
+import urllib3.util.connection as urllib3_connection # noqa: E402
+
+PUBLIC_V4 = "93.184.216.34"
+PUBLIC_V6 = "2606:4700:4700::1111"
+OTHER_PUBLIC_V6 = "2001:4860:4860::8888"
+
+
+def _info(ip: str, port) -> tuple:
+ """One getaddrinfo row, shaped the way the real resolver shapes it."""
+ if ":" in ip:
+ return (socket.AF_INET6, socket.SOCK_STREAM, 6, "", (ip, port, 0, 0))
+ return (socket.AF_INET, socket.SOCK_STREAM, 6, "", (ip, port))
+
+
+def _resolver(records: dict, families: list | None = None):
+ """getaddrinfo stand-in: ``records`` maps a name to its A and AAAA answers."""
+ def fake_getaddrinfo(host, port, *args, **kwargs):
+ if families is not None:
+ families.append(kwargs.get("family", args[0] if args else 0))
+ if host not in records:
+ raise socket.gaierror(socket.EAI_NONAME, f"unmocked host {host!r}")
+ return [_info(ip, port or 0) for ip in records[host]]
+
+ return fake_getaddrinfo
+
+
[email protected](autouse=True)
+def _isolated(monkeypatch):
+ """No developer allowlist, no developer proxy."""
+ monkeypatch.delenv("CLAUDE_SEO_LOCAL_TARGETS", raising=False)
+ monkeypatch.setattr(url_safety.requests.utils, "getproxies", lambda: {})
+
+
+# ---------------------------------------------------------------------------
+# is_safe_ip: every IPv6 range that is not a public audit target
+# ---------------------------------------------------------------------------
+
+
+ "ip",
+ [
+ pytest.param("::1", id="loopback"),
+ pytest.param("::", id="unspecified"),
+ pytest.param("fe80::1", id="link-local"),
+ pytest.param("febf::1", id="link-local-upper"),
+ pytest.param("fc00::1", id="unique-local-fc"),
+ pytest.param("fd12:3456:789a::1", id="unique-local-fd"),
+ pytest.param("fec0::1", id="site-local"),
+ pytest.param("feff::1", id="site-local-upper"),
+ pytest.param("ff02::1", id="multicast-link"),
+ pytest.param("ff0e::1", id="multicast-global"),
+ pytest.param("::ffff:127.0.0.1", id="mapped-loopback"),
+ pytest.param("::ffff:10.0.0.1", id="mapped-rfc1918"),
+ pytest.param("::ffff:169.254.169.254", id="mapped-metadata"),
+ pytest.param("::ffff:100.64.0.1", id="mapped-rfc6598"),
+ pytest.param("::127.0.0.1", id="compatible-loopback"),
+ pytest.param("::8.8.8.8", id="compatible-public"),
+ pytest.param("64:ff9b::10.0.0.1", id="nat64-private"),
+ pytest.param("64:ff9b::169.254.169.254", id="nat64-metadata"),
+ pytest.param("64:ff9b::8.8.8.8", id="nat64-public"),
+ pytest.param("64:ff9b:1::1", id="nat64-local-use"),
+ pytest.param("64:ff9b:1:ffff::8.8.8.8", id="nat64-local-use-upper"),
+ pytest.param("2002:7f00:1::1", id="6to4-loopback"),
+ pytest.param("2002:a00:1::1", id="6to4-rfc1918"),
+ pytest.param("2002:808:808::1", id="6to4-public"),
+ pytest.param("2001::1", id="teredo"),
+ pytest.param("2001:0:4136:e378:8000:63bf:3fff:fdd2", id="teredo-real-shape"),
+ pytest.param("2001:db8::1", id="documentation"),
+ pytest.param("3fff::1", id="documentation-rfc9637"),
+ pytest.param("2001:10::1", id="orchid"),
+ pytest.param("2001:1f:ffff::1", id="orchid-upper"),
+ pytest.param("2001:20::1", id="orchid-v2"),
+ pytest.param("2001:2f:ffff::1", id="orchid-v2-upper"),
+ pytest.param("fd00:ec2::254", id="aws-imds"),
+ ],
+)
+def test_is_safe_ip_refuses_ipv6_special_ranges(ip: str) -> None:
+ assert url_safety.is_safe_ip(ip) is False
+
+
+ "network", [str(network) for network in url_safety._UNSAFE_IPV6_NETWORKS]
+)
+def test_every_unsafe_ipv6_network_is_refused_at_both_ends(network: str) -> None:
+ parsed = ipaddress.ip_network(network)
+ assert url_safety.is_safe_ip(str(parsed[0])) is False
+ assert url_safety.is_safe_ip(str(parsed[-1])) is False
+
+
+def test_the_unsafe_ipv6_table_names_every_range_the_policy_promises() -> None:
+ """The flags in ``ipaddress`` move between Python releases; the table is
+ what the policy rests on, so a range cannot drop out of it unnoticed."""
+ assert {str(network) for network in url_safety._UNSAFE_IPV6_NETWORKS} == {
+ "::/96",
+ "64:ff9b::/96",
+ "64:ff9b:1::/48",
+ "2001::/32",
+ "2001:10::/28",
+ "2001:20::/28",
+ "2001:db8::/32",
+ "2002::/16",
+ "3fff::/20",
+ "fc00::/7",
+ "fe80::/10",
+ "fec0::/10",
+ "ff00::/8",
+ }
+
+
+ "ip",
+ [
+ "fe80::1%eth0",
+ "fe80::1%1",
+ "::1%lo",
+ f"{PUBLIC_V6}%eth0", # public address, still machine-local with a zone
+ "::ffff:8.8.8.8%eth0",
+ "fe80::1%",
+ ],
+)
+def test_is_safe_ip_refuses_zone_identifiers(ip: str) -> None:
+ assert url_safety.is_safe_ip(ip) is False
+ assert url_safety._is_allowlistable_ip(ip) is False
+
+
+ "ip,expected",
+ [
+ ("::ffff:8.8.8.8", True),
+ ("::ffff:1.1.1.1", True),
+ ("::ffff:127.0.0.1", False),
+ ("::ffff:192.168.1.1", False),
+ ("::ffff:100.100.100.200", False),
+ ("::ffff:0.0.0.0", False),
+ ("::ffff:224.0.0.1", False),
+ ],
+)
+def test_ipv4_mapped_ipv6_is_judged_by_the_embedded_ipv4(ip: str, expected: bool) -> None:
+ assert url_safety.is_safe_ip(ip) is expected
+
+
+ "ip",
+ [
+ PUBLIC_V6,
+ OTHER_PUBLIC_V6,
+ "2620:fe::fe",
+ "2a00:1450:4001:81b::200e",
+ "2001:200::1", # first block past the IETF 2001::/23 assignments
+ "2001:db9::1", # one past the documentation range
+ "2003::1", # one past 6to4
+ ],
+)
+def test_is_safe_ip_still_accepts_public_ipv6(ip: str) -> None:
+ assert url_safety.is_safe_ip(ip) is True
+
+
+# ---------------------------------------------------------------------------
+# validate_url and validate_url_strict: IPv6 literals
+# ---------------------------------------------------------------------------
+
+
+ "url,expected",
+ [
+ (f"http://[{PUBLIC_V6}]/", True),
+ (f"https://[{PUBLIC_V6}]:8443/path?q=1", True),
+ ("http://[::ffff:8.8.8.8]/", True),
+ ("http://[::1]/", False),
+ ("http://[::]/", False),
+ ("http://[fe80::1]/", False),
+ ("http://[fd00::1]/", False),
+ ("http://[fec0::1]/", False),
+ ("http://[ff02::1]/", False),
+ ("http://[::ffff:127.0.0.1]/", False),
+ ("http://[::127.0.0.1]/", False),
+ ("http://[64:ff9b::7f00:1]/", False),
+ ("http://[64:ff9b:1::1]/", False),
+ ("http://[2002:7f00:1::1]/", False),
+ ("http://[2001::1]/", False),
+ ("http://[2001:db8::1]/", False),
+ ("http://[2001:10::1]/", False),
+ ("http://[2001:20::1]/", False),
+ ("http://[fd00:ec2::254]/latest/meta-data/", False),
+ ],
+)
+def test_validate_url_ipv6_literals(url: str, expected: bool) -> None:
+ assert url_safety.validate_url(url) is expected
+ if expected:
+ assert url_safety.validate_url_strict(url)[0] == url
+ else:
+ with pytest.raises(url_safety.URLSafetyError):
+ url_safety.validate_url_strict(url)
+
+
+ "url",
+ [
+ "http://[fe80::1%25eth0]/",
+ "http://[fe80::1%eth0]/",
+ f"http://[{PUBLIC_V6}%25eth0]/",
+ f"http://[{PUBLIC_V6}%eth0]:8080/",
+ ],
+)
+def test_zone_identifier_in_a_url_is_refused(url: str) -> None:
+ assert url_safety.validate_url(url) is False
+ with pytest.raises(url_safety.URLSafetyError):
+ url_safety.validate_url_strict(url)
+
+
+def test_strict_keeps_the_brackets_of_a_public_ipv6_literal() -> None:
+ url = f"https://[{PUBLIC_V6}]:8443/path?q=1"
+ with patch.object(
+ url_safety.socket, "getaddrinfo", side_effect=AssertionError("literal resolved")
+ ):
+ assert url_safety.validate_url_strict(url) == (url, PUBLIC_V6)
+
+
+# ---------------------------------------------------------------------------
+# validate_url_strict: both families resolved, every record checked
+# ---------------------------------------------------------------------------
+
+
+def test_strict_resolves_both_families_in_one_lookup() -> None:
+ families: list = []
+ resolver = _resolver({"example.com": [PUBLIC_V4]}, families)
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ url_safety.validate_url_strict("https://example.com/")
+ assert families == [socket.AF_UNSPEC]
+
+
+def test_strict_pins_ipv6_for_a_host_with_only_aaaa_records() -> None:
+ """Before this, an AAAA-only host failed with 'DNS resolution failed'."""
+ resolver = _resolver({"v6only.example": [OTHER_PUBLIC_V6, PUBLIC_V6]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ url, pinned = url_safety.validate_url_strict("https://v6only.example/")
+ assert url == "https://v6only.example/"
+ assert pinned == OTHER_PUBLIC_V6 # lowest address, as for A records
+
+
+ "answers",
+ [
+ [PUBLIC_V4, PUBLIC_V6],
+ [PUBLIC_V6, PUBLIC_V4],
+ [PUBLIC_V6, OTHER_PUBLIC_V6, "93.184.216.99", PUBLIC_V4],
+ ],
+)
+def test_strict_prefers_ipv4_for_a_dual_stack_host(answers) -> None:
+ """Whatever order the resolver uses, the pin is the address it always was."""
+ resolver = _resolver({"dual.example": answers})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ assert url_safety.validate_url_strict("https://dual.example/")[1] == PUBLIC_V4
+
+
+ "answers,refused",
+ [
+ ([PUBLIC_V4, "fd00::1"], "fd00::1"), # public A, private AAAA
+ ([PUBLIC_V4, "::1"], "::1"),
+ ([PUBLIC_V4, "fe80::1"], "fe80::1"),
+ ([PUBLIC_V4, "fec0::1"], "fec0::1"),
+ ([PUBLIC_V4, "::ffff:10.0.0.1"], "::ffff:10.0.0.1"),
+ ([PUBLIC_V4, "fc00::1"], "fc00::1"),
+ ([PUBLIC_V4, "64:ff9b:1::1"], "64:ff9b:1::1"), # local-use NAT64
+ ([PUBLIC_V4, "2002:a00:1::1"], "2002:a00:1::1"),
+ ([PUBLIC_V4, "fd00:ec2::254"], "fd00:ec2::254"),
+ ([PUBLIC_V6, "10.0.0.7"], "10.0.0.7"), # public AAAA, private A
+ ([PUBLIC_V6, "169.254.169.254"], "169.254.169.254"),
+ ([PUBLIC_V6, "fd00::1"], "fd00::1"), # two AAAA, one private
+ ([PUBLIC_V4, PUBLIC_V6, "fd00::1"], "fd00::1"),
+ ],
+)
+def test_strict_refuses_a_host_with_any_unsafe_record_in_either_family(
+ answers, refused
+) -> None:
+ """The multi-A-record rule, across both families: one bad record refuses
+ the host, even though the address that would be pinned is public."""
+ resolver = _resolver({"mixed.example": answers})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP") as excinfo:
+ url_safety.validate_url_strict("https://mixed.example/")
+ assert str(excinfo.value).endswith(f"non-public IP {refused}")
+
+
+ "address",
+ [
+ "::1", "fe80::1", "fd00::1", "fec0::1", "ff02::1", "::ffff:127.0.0.1",
+ "::10.0.0.1", "64:ff9b::808:808", "64:ff9b:1::1", "2002:808:808::1",
+ "2001::1", "2001:db8::1", "2001:10::1", "2001:20::1", "fd00:ec2::254",
+ "fe80::1%eth0", f"{PUBLIC_V6}%eth0",
+ ],
+)
+def test_strict_refuses_an_aaaa_only_host_in_an_unsafe_range(address: str) -> None:
+ resolver = _resolver({"v6only.example": [address]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP"):
+ url_safety.validate_url_strict("https://v6only.example/")
+
+
+# ---------------------------------------------------------------------------
+# DNS64: a synthesized 64:ff9b::/96 AAAA beside an A record is passed over
+# ---------------------------------------------------------------------------
+
+
+ "nat64",
+ [
+ "64:ff9b::5db8:d822", # the same public IPv4, as DNS64 synthesizes it
+ "64:ff9b::808:808", # another public IPv4
+ "64:ff9b::7f00:1", # loopback embedded: still never dialled
+ "64:ff9b::a9fe:a9fe", # 169.254.169.254 embedded
+ ],
+)
+def test_dns64_aaaa_beside_an_a_record_is_ignored_and_ipv4_is_pinned(nat64) -> None:
+ """An IPv4-only site on a DNS64 network: one real A record plus one
+ synthesized AAAA. It resolved before this lookup asked for both families
+ and must keep resolving, pinned to the A record."""
+ resolver = _resolver({"v4only.example": [nat64, PUBLIC_V4]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ url, pinned = url_safety.validate_url_strict("https://v4only.example/")
+ assert (url, pinned) == ("https://v4only.example/", PUBLIC_V4)
+
+ # The pin for that host answers in IPv4 only, so the AAAA has no way in.
+ with url_safety._pin_dns("v4only.example", pinned, 443):
+ assert socket.getaddrinfo("v4only.example", 443) == [
+ (socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (PUBLIC_V4, 443))
+ ]
+ with pytest.raises(socket.gaierror, match="refused for pinned IPv4 host"):
+ socket.getaddrinfo("v4only.example", 443, socket.AF_INET6)
+
+
+ "answers,refused",
+ [
+ (["10.0.0.7", "64:ff9b::a00:7"], "10.0.0.7"), # private A
+ (["127.0.0.1", "64:ff9b::7f00:1"], "127.0.0.1"),
+ ([PUBLIC_V4, "10.0.0.7", "64:ff9b::5db8:d822"], "10.0.0.7"),
+ ([PUBLIC_V4, "64:ff9b::5db8:d822", "fc00::1"], "fc00::1"),
+ ([PUBLIC_V4, "64:ff9b::5db8:d822", "64:ff9b:1::1"], "64:ff9b:1::1"),
+ ([PUBLIC_V4, "64:ff9b::5db8:d822%eth0"], "64:ff9b::5db8:d822%eth0"),
+ ],
+)
+def test_dns64_pass_covers_nothing_but_the_well_known_prefix(answers, refused) -> None:
+ """The A records are judged as always, and every other unsafe AAAA,
+ the local-use NAT64 prefix included, still refuses the host."""
+ resolver = _resolver({"mixed.example": answers})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP") as excinfo:
+ url_safety.validate_url_strict("https://mixed.example/")
+ assert str(excinfo.value).endswith(f"non-public IP {refused}")
+
+
+ "answers",
+ [
+ ["64:ff9b::5db8:d822"],
+ ["64:ff9b::5db8:d822", "64:ff9b::808:808"],
+ ["64:ff9b::5db8:d822", PUBLIC_V6], # no A record, so no pass
+ ["64:ff9b::5db8:d822", "::ffff:8.8.8.8"], # IPv4-mapped is not an A record
+ ],
+)
+def test_host_without_an_a_record_gets_no_dns64_pass(answers) -> None:
+ resolver = _resolver({"v6only.example": answers})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(
+ url_safety.URLSafetyError, match="non-public IP 64:ff9b::5db8:d822"
+ ):
+ url_safety.validate_url_strict("https://v6only.example/")
+
+
+def test_nat64_stays_refused_everywhere_but_the_dns64_case() -> None:
+ """is_safe_ip is as strict as before: literals, redirect targets inside
+ the pinned scope and the Playwright handler all still refuse the prefix."""
+ assert url_safety.is_safe_ip("64:ff9b::5db8:d822") is False
+ assert url_safety.validate_url("http://[64:ff9b::5db8:d822]/") is False
+ with pytest.raises(url_safety.URLSafetyError, match="Blocked IP literal"):
+ url_safety.validate_url_strict("http://[64:ff9b::5db8:d822]/")
+
+ resolver = _resolver({"redirected.example": [PUBLIC_V4, "64:ff9b::5db8:d822"]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with url_safety._pin_dns("pinned.example", PUBLIC_V4, 443):
+ with pytest.raises(socket.gaierror, match="non-public IP"):
+ socket.getaddrinfo("redirected.example", 443)
+
+
+def test_strict_reports_a_host_with_no_records() -> None:
+ with patch.object(url_safety.socket, "getaddrinfo", return_value=[]):
+ with pytest.raises(url_safety.URLSafetyError, match="No address records"):
+ url_safety.validate_url_strict("https://empty.example/")
+
+
+# ---------------------------------------------------------------------------
+# CLAUDE_SEO_LOCAL_TARGETS with IPv6
+# ---------------------------------------------------------------------------
+
+
+def test_allowlisted_ipv6_loopback_literal_is_pinned_as_ipv6(monkeypatch) -> None:
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")
+ assert url_safety.validate_url_strict("http://[::1]:3000/") == (
+ "http://[::1]:3000/",
+ "::1",
+ )
+ with pytest.raises(url_safety.URLSafetyError, match="Blocked hostname"):
+ url_safety.validate_url_strict("http://[::1]:3001/")
+
+
+def test_allowlisted_localhost_with_both_families_still_pins_ipv4(monkeypatch) -> None:
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "localhost:3000")
+ resolver = _resolver({"localhost": ["::1", "127.0.0.1"]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ assert url_safety.validate_url_strict("http://localhost:3000/")[1] == "127.0.0.1"
+
+
+def test_allowlisted_name_may_resolve_to_unique_local_ipv6(monkeypatch) -> None:
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "staging.internal:8080")
+ resolver = _resolver({"staging.internal": ["fd00::5"]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ assert url_safety.validate_url_strict("http://staging.internal:8080/")[1] == "fd00::5"
+
+
+ "address", ["fe80::1", "fd00:ec2::254", "ff02::1", "::", "fd00::5%eth0", "64:ff9b::a00:1"]
+)
+def test_allowlist_never_admits_these_ipv6_addresses(monkeypatch, address) -> None:
+ """Link-local, metadata, multicast, unspecified, zoned and NAT64 answers
+ stay refused for a listed name, exactly as 169.254.169.254 does."""
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "staging.internal:8080")
+ resolver = _resolver({"staging.internal": [address]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(url_safety.URLSafetyError, match="DNS rebinding refused"):
+ url_safety.validate_url_strict("http://staging.internal:8080/")
+
+
+def test_allowlist_does_not_forgive_an_unsafe_aaaa_on_an_unlisted_host(monkeypatch) -> None:
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "staging.internal:8080")
+ resolver = _resolver({"other.example": [PUBLIC_V4, "fd00::5"]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP fd00::5"):
+ url_safety.validate_url_strict("http://other.example:8080/")
+
+
+# ---------------------------------------------------------------------------
+# _pin_dns: the tuple matches the pinned family, the caller's family is honoured
+# ---------------------------------------------------------------------------
+
+
+ "args,kwargs",
+ [
+ ((), {}),
+ ((socket.AF_UNSPEC,), {}),
+ ((socket.AF_INET6,), {}),
+ ((socket.AF_INET6, socket.SOCK_STREAM), {}),
+ ((), {"family": socket.AF_UNSPEC}),
+ ((), {"family": socket.AF_INET6, "type": socket.SOCK_STREAM}),
+ ],
+)
+def test_ipv6_pin_returns_an_af_inet6_row_with_a_four_part_sockaddr(args, kwargs) -> None:
+ with url_safety._pin_dns("v6only.example", PUBLIC_V6, 443):
+ rows = socket.getaddrinfo("v6only.example", 443, *args, **kwargs)
+ assert rows == [
+ (socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (PUBLIC_V6, 443, 0, 0))
+ ]
+
+
+ "args,kwargs", [((socket.AF_INET,), {}), ((), {"family": socket.AF_INET})]
+)
+def test_ipv6_pin_fails_closed_for_an_ipv4_only_lookup(args, kwargs) -> None:
+ """No AF_INET row carrying an IPv6 address, and no fall-back lookup."""
+ with patch.object(
+ url_safety.socket, "getaddrinfo", side_effect=AssertionError("real resolver used")
+ ):
+ with url_safety._pin_dns("v6only.example", PUBLIC_V6, 443):
+ with pytest.raises(socket.gaierror, match="refused for pinned IPv6 host"):
+ socket.getaddrinfo("v6only.example", 443, *args, **kwargs)
+
+
+ "args,kwargs", [((socket.AF_INET6,), {}), ((), {"family": socket.AF_INET6})]
+)
+def test_ipv4_pin_fails_closed_for_an_ipv6_only_lookup(args, kwargs) -> None:
+ with url_safety._pin_dns("example.com", PUBLIC_V4, 443):
+ with pytest.raises(socket.gaierror, match="refused for pinned IPv4 host"):
+ socket.getaddrinfo("example.com", 443, *args, **kwargs)
+
+
[email protected]("family", [None, socket.AF_UNSPEC, socket.AF_INET])
+def test_ipv4_pin_row_is_unchanged(family) -> None:
+ kwargs = {} if family is None else {"family": family}
+ with url_safety._pin_dns("example.com", PUBLIC_V4, 443):
+ rows = socket.getaddrinfo("example.com", 8443, **kwargs)
+ assert rows == [
+ (socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (PUBLIC_V4, 8443))
+ ]
+
+
[email protected]("requested_port", [80, 443, 8080, None, 0])
+def test_public_ipv6_pin_answers_for_any_port(requested_port) -> None:
+ with url_safety._pin_dns("v6only.example", PUBLIC_V6, 80):
+ sockaddr = socket.getaddrinfo("v6only.example", requested_port)[0][4]
+ assert sockaddr == (PUBLIC_V6, requested_port or 80, 0, 0)
+
+
+def test_ipv6_local_pin_is_bound_to_the_validated_port(monkeypatch) -> None:
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")
+ with url_safety._pin_dns("::1", "::1", 3000):
+ assert socket.getaddrinfo("::1", 3000)[0][4] == ("::1", 3000, 0, 0)
+ assert socket.getaddrinfo("::1", None)[0][4] == ("::1", 3000, 0, 0)
+ for port in (6379, 80, 443, 3001, "redis"):
+ with pytest.raises(socket.gaierror, match="validated port is 3000"):
+ socket.getaddrinfo("::1", port)
+
+
+def test_ipv6_local_pin_is_port_bound_without_an_allowlist() -> None:
+ with url_safety._pin_dns("staging.internal", "fd00::5", 8080):
+ assert socket.getaddrinfo("staging.internal", 8080)[0][4] == ("fd00::5", 8080, 0, 0)
+ with pytest.raises(socket.gaierror, match="validated port is 8080"):
+ socket.getaddrinfo("staging.internal", 9200)
+
+
+def test_bare_ipv6_allowlist_entry_keeps_matching_any_port(monkeypatch) -> None:
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "fd00::5")
+ with url_safety._pin_dns("fd00::5", "fd00::5", 80):
+ assert socket.getaddrinfo("fd00::5", 8080)[0][4] == ("fd00::5", 8080, 0, 0)
+
+
+ "address", ["::1", "fd00::1", "fec0::1", "64:ff9b::a00:1", "2002:a00:1::1", "fe80::1%eth0"]
+)
+def test_redirect_target_on_an_unsafe_ipv6_address_is_refused_in_the_pinned_scope(
+ address,
+) -> None:
+ resolver = _resolver({"redirected.example": [PUBLIC_V4, address]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with url_safety._pin_dns("pinned.example", PUBLIC_V6, 443):
+ with pytest.raises(socket.gaierror, match="non-public IP"):
+ socket.getaddrinfo("redirected.example", 443)
+
+
+def test_redirect_target_on_public_ipv6_resolves_in_the_pinned_scope() -> None:
+ resolver = _resolver({"elsewhere.example": [PUBLIC_V6]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with url_safety._pin_dns("pinned.example", PUBLIC_V4, 443):
+ assert socket.getaddrinfo("elsewhere.example", 443)[0][4][0] == PUBLIC_V6
+
+
+# ---------------------------------------------------------------------------
+# Proxy validation picks up the same IPv6 policy
+# ---------------------------------------------------------------------------
+
+
[email protected]("proxy_host", ["::1", "fd00::1", "fec0::1", "64:ff9b::808:808", "2001:20::1"])
+def test_proxy_on_a_non_public_ipv6_literal_is_refused(proxy_host) -> None:
+ with pytest.raises(url_safety.URLSafetyError, match="Refusing configured HTTP proxy"):
+ url_safety._assert_proxy_host_is_public(proxy_host)
+
+
+def test_proxy_name_with_an_unsafe_aaaa_record_is_refused() -> None:
+ resolver = _resolver({"proxy.corp.example": ["8.8.4.4", "fec0::1"]})
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):
+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP fec0::1"):
+ url_safety._assert_proxy_host_is_public("proxy.corp.example")
+
+
+# ---------------------------------------------------------------------------
+# End to end through requests and urllib3
+# ---------------------------------------------------------------------------
+
+
+def _redirect(location: str) -> bytes:
+ return (
+ f"HTTP/1.1 302 Found\r\nLocation: {location}\r\n"
+ "Content-Length: 0\r\nConnection: close\r\n\r\n"
+ ).encode("ascii")
+
+
+def _ok(body: str) -> bytes:
+ return (
+ f"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n"
+ f"Content-Length: {len(body)}\r\nConnection: close\r\n\r\n{body}"
+ ).encode("ascii")
+
+
+class _FakeNet:
+ """Canned HTTP servers keyed by ``(ip, port)``, plus what reached them."""
+
+ def __init__(self, hosts: dict, servers: dict) -> None:
+ self.hosts = hosts
+ self.servers = servers
+ self.connections: list = [] # (socket family, sockaddr)
+ self.sent: list = []
+
+ def getaddrinfo(self, host, port, *args, **kwargs):
+ if host in self.hosts:
+ ips = self.hosts[host]
+ else:
+ try:
+ ipaddress.ip_address(host)
+ except ValueError:
+ raise socket.gaierror(socket.EAI_NONAME, f"unmocked host {host!r}") from None
+ ips = [host]
+ return [_info(ip, port or 0) for ip in ips]
+
+ def socket(self, family=socket.AF_INET, *args, **kwargs):
+ return _FakeSocket(self, family)
+
+
+class _FakeSocket:
+ """The slice of ``socket.socket`` that urllib3 and http.client use."""
+
+ def __init__(self, net: _FakeNet, family) -> None:
+ self._net = net
+ self._family = family
+ self._key = None
+
+ def connect(self, address) -> None:
+ self._net.connections.append((self._family, address))
+ key = (address[0], address[1])
+ if key not in self._net.servers:
+ raise ConnectionRefusedError(111, "no fake server on this port")
+ self._key = key
+
+ def makefile(self, *args, **kwargs):
+ return io.BytesIO(self._net.servers[self._key])
+
+ def sendall(self, data) -> None:
+ self._net.sent.append(bytes(data))
+
+ def setsockopt(self, *args) -> None:
+ pass
+
+ def settimeout(self, value) -> None:
+ pass
+
+ def shutdown(self, how) -> None:
+ pass
+
+ def close(self) -> None:
+ pass
+
+
+def _get(url: str, net: _FakeNet, *, gai_family=socket.AF_UNSPEC):
+ """``safe_requests_get`` against the fake network. ``gai_family`` is what
+ urllib3 asks the resolver for: AF_UNSPEC on a machine with IPv6, AF_INET
+ on one without. It is set here so the result does not depend on the host."""
+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=net.getaddrinfo), \
+ patch.object(url_safety.socket, "socket", side_effect=net.socket), \
+ patch.object(urllib3_connection, "allowed_gai_family", return_value=gai_family):
+ return url_safety.safe_requests_get(url, timeout=5)
+
+
+def test_ipv6_only_host_is_fetched_over_ipv6() -> None:
+ net = _FakeNet({"v6only.example": [PUBLIC_V6]}, {(PUBLIC_V6, 80): _ok("over ipv6")})
+
+ response = _get("http://v6only.example/page", net)
+
+ assert response.status_code == 200
+ assert response.content == b"over ipv6"
+ assert net.connections == [(socket.AF_INET6, (PUBLIC_V6, 80, 0, 0))]
+ assert b"Host: v6only.example\r\n" in b"".join(net.sent)
+
+
+def test_ipv6_literal_keeps_its_brackets_in_the_url_and_host_header() -> None:
+ url = f"http://[{PUBLIC_V6}]:8080/page"
+ net = _FakeNet({}, {(PUBLIC_V6, 8080): _ok("literal")})
+
+ response = _get(url, net)
+
+ assert response.content == b"literal"
+ assert response.url == url
+ assert response.request.url == url
+ assert net.connections == [(socket.AF_INET6, (PUBLIC_V6, 8080, 0, 0))]
+ assert f"Host: [{PUBLIC_V6}]:8080\r\n".encode("ascii") in b"".join(net.sent)
+
+
+def test_dual_stack_host_still_connects_over_ipv4() -> None:
+ net = _FakeNet(
+ {"dual.example": [PUBLIC_V6, PUBLIC_V4]},
+ {(PUBLIC_V4, 80): _ok("over ipv4"), (PUBLIC_V6, 80): _ok("over ipv6")},
+ )
+
+ assert _get("http://dual.example/", net).content == b"over ipv4"
+ assert net.connections == [(socket.AF_INET, (PUBLIC_V4, 80))]
+
+
+def test_ipv4_only_site_on_a_dns64_network_is_fetched_over_ipv4() -> None:
+ """Even with a server answering at the synthesized address, only the
+ pinned A record is dialled."""
+ nat64 = "64:ff9b::5db8:d822"
+ net = _FakeNet(
+ {"v4only.example": [nat64, PUBLIC_V4]},
+ {(PUBLIC_V4, 80): _ok("over ipv4"), (nat64, 80): _ok("over nat64")},
+ )
+
+ assert _get("http://v4only.example/", net).content == b"over ipv4"
+ assert net.connections == [(socket.AF_INET, (PUBLIC_V4, 80))]
+
+
+def test_ipv6_only_host_fails_closed_on_a_machine_without_ipv6() -> None:
+ """urllib3 asks for AF_INET only there. The pin refuses instead of
+ handing back an IPv6 address in an IPv4 row, and nothing is connected."""
+ net = _FakeNet({"v6only.example": [PUBLIC_V6]}, {(PUBLIC_V6, 80): _ok("over ipv6")})
+
+ with pytest.raises(requests.exceptions.ConnectionError, match="refused for pinned IPv6 host"):
+ _get("http://v6only.example/", net, gai_family=socket.AF_INET)
+
+ assert net.connections == []
+
+
+def test_ipv6_host_redirect_to_another_port_is_followed_when_public() -> None:
+ net = _FakeNet(
+ {"v6only.example": [PUBLIC_V6]},
+ {
+ (PUBLIC_V6, 80): _redirect("http://v6only.example:8080/app"),
+ (PUBLIC_V6, 8080): _ok("public app"),
+ },
+ )
+
+ assert _get("http://v6only.example/", net).content == b"public app"
+ assert net.connections == [
+ (socket.AF_INET6, (PUBLIC_V6, 80, 0, 0)),
+ (socket.AF_INET6, (PUBLIC_V6, 8080, 0, 0)),
+ ]
+
+
+def test_redirect_to_another_port_on_an_allowed_ipv6_local_host_is_refused(monkeypatch) -> None:
+ """The port-bound rule for local targets, unchanged, on an IPv6 pin."""
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")
+ net = _FakeNet(
+ {},
+ {
+ ("::1", 3000): _redirect("http://[::1]:6379/"),
+ ("::1", 6379): _ok("internal service"),
+ },
+ )
+
+ with pytest.raises(requests.exceptions.ConnectionError, match="validated port"):
+ _get("http://[::1]:3000/", net)
+
+ assert net.connections == [(socket.AF_INET6, ("::1", 3000, 0, 0))]
+
+
+ "location",
+ [
+ "http://[0:0:0:0:0:0:0:1]:6379/", # the same address, written out
+ "http://[::ffff:127.0.0.1]:6379/", # its IPv4-mapped neighbour
+ "http://localhost:6379/", # a name for it
+ "http://127.0.0.1:6379/",
+ ],
+)
+def test_another_spelling_of_the_allowed_ipv6_host_does_not_escape_the_port_rule(
+ monkeypatch, location
+) -> None:
+ """A redirect cannot reach another port by naming the pinned loopback
+ address differently: any spelling that is not the pinned host is a new
+ host, and a new host on a non-public address is refused."""
+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")
+ servers = {("::1", 3000): _redirect(location)}
+ for ip in ("::1", "127.0.0.1", "::ffff:127.0.0.1", "::ffff:7f00:1"):
+ servers[(ip, 6379)] = _ok("internal service")
+ net = _FakeNet({"localhost": ["::1", "127.0.0.1"]}, servers)
+
+ with pytest.raises(
+ requests.exceptions.ConnectionError, match="non-public IP|validated port"
+ ):
+ _get("http://[::1]:3000/", net)
+
+ assert net.connections == [(socket.AF_INET6, ("::1", 3000, 0, 0))]
+
+
+ "location", ["http://[::1]:3000/", "http://[fd00::5]/", "http://[fec0::1]/", "http://internal.example/"]
+)
+def test_public_ipv6_host_redirecting_to_a_local_ipv6_target_is_refused(location) -> None:
+ net = _FakeNet(
+ {"v6only.example": [PUBLIC_V6], "internal.example": [PUBLIC_V4, "fd00::5"]},
+ {
+ (PUBLIC_V6, 80): _redirect(location),
+ ("::1", 3000): _ok("dev server"),
+ ("fd00::5", 80): _ok("internal"),
+ ("fec0::1", 80): _ok("internal"),
+ (PUBLIC_V4, 80): _ok("internal"),
+ },
+ )
+
+ with pytest.raises(requests.exceptions.ConnectionError, match="non-public IP"):
+ _get("http://v6only.example/", net)
+
+ assert net.connections == [(socket.AF_INET6, (PUBLIC_V6, 80, 0, 0))]