Universal SEO skill for Claude Code. 26 sub-skills + 19 sub-agents covering technical SEO, E-E-A-T, schema, GEO/AEO, agent readiness (Lighthouse Agentic Browsing, WebMCP, llms.txt), backlinks, local SEO, e-commerce, international SEO, Google APIs, and PDF/Excel reporting. 9 optional extensions for live SEO data.
feat(fetch): IPv6-only hosts, fetch timing and a wall-clock deadline
Agrici Daniel26507e00f467
diff --git a/SECURITY.md b/SECURITY.mdindex 4d022fe..6032704 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -57,7 +57,7 @@ claude-seo is a research and audit toolkit that runs on a user's workstation. It- **Playwright + Chromium DNS rebinding.** Chromium does its own DNS resolution inside the renderer process. claude-seo's Python-layer DNS pin (`url_safety._pin_dns`) cannot reach it. The Playwright `route()` handler re-validates every subresource host (`make_safe_playwright_route_handler`), which closes the common case, but a true rebinding attacker can still race Chromium's resolver after our pre-flight returns. Mitigation: do not point `/seo` skills at untrusted sites with high-frequency redirects.- **CONNECT proxies weaken DNS pinning.** Behind a configured HTTP proxy (`HTTPS_PROXY` and friends) the proxy resolves the final host itself, so `url_safety`'s pin degrades to the pre-flight check in `validate_url_strict`; the proxy host itself is exempted from the pinned scope only after passing the same hostname blocklist and `is_safe_ip` policy as an audit target, so a proxy on loopback, RFC 1918, RFC 6598, link-local, or a metadata address is refused rather than exempted.-- **IPv6-only audit targets.** The strict validator queries `family=AF_INET` for the initial resolution. Hosts with AAAA records only will surface as "DNS resolution failed". This is **fail-closed** by design: we'd rather refuse than connect to an unvalidated IPv6 endpoint. Tracked for a future patch (full dual-stack pinning, similar to the Playwright handler which already uses `AF_UNSPEC`).+- **IPv6 audit targets.** The strict validator resolves both address families and refuses a host if any A or AAAA record is non-public. The pin is IPv4 when the host has an A record and IPv6 otherwise, with no fallback between families. NAT64 (64:ff9b::/96, 64:ff9b:1::/48), 6to4, Teredo, site-local, documentation, ORCHID and zone-scoped addresses are refused. One exception: on a DNS64 network, a synthesized AAAA record under 64:ff9b::/96 is ignored when the host also has an A record, because the pin is then IPv4 and that address is never dialled. A host with no A record gets no such pass, and redirect targets and browser subresources still refuse the prefix.- **Windows file permissions.** `os.fchmod(fd, 0o600)` is a no-op on Windows for non-ACL filesystems. Users on Windows should rely on per-user directory ACLs instead of POSIX mode bits. `scripts/backlinks_auth.py` additionally makes a best-effort `icacls <path> /inheritance:r /grant:r "%USERNAME%:F"` call after every write to (and load-time remediation of) `~/.config/claude-seo/backlinks-api.json`, restricting it to the current user on NTFS. This is best-effort only: a failure (missing `icacls`, a non-NTFS volume, a restricted shell) is logged as a warning and does not block the credential read or write, so on a failure the file's effective protection on Windows falls back to whatever the containing directory's ACLs already provide.- **RFC 6598 (100.64.0.0/10) is refused by default.** Cloud providers serve instance metadata in this range, so `url_safety` treats it like RFC 1918. Tailscale also allocates from it, so a staging site reached over Tailscale is refused under the default policy. The escape hatch is `CLAUDE_SEO_LOCAL_TARGETS`, a comma-separated allowlist of `host` or `host:port` entries (for example `localhost:3000,127.0.0.1:8080,100.101.102.103`, the last being a Tailscale address). Its scope is deliberately narrow:diff --git a/scripts/fetch_page.py b/scripts/fetch_page.pyindex 0238f9f..25b71d2 100755--- a/scripts/fetch_page.py+++ b/scripts/fetch_page.py@@ -12,6 +12,21 @@ Usage:python fetch_page.py https://example.com --jsonpython fetch_page.py https://example.com --render auto # SPA-awarepython fetch_page.py https://example.com --render always # force render+ python fetch_page.py https://example.com --max-time 20 # total budget++Timing+======+A raw fetch reports three durations in milliseconds: ``ttfb_ms``,+``response_time_ms`` and ``fetch_duration_ms`` (see ``fetch_page``). They are+one measurement taken from this machine, over this network, at this moment.+They are not Core Web Vitals, not a field metric and not what a visitor's+browser sees: use the CrUX and PageSpeed scripts for those. Treat them as a+hint that a server is slow to answer, never as a score.++The whole fetch, redirects and body included, also has a wall-clock budget+(``--max-time``). ``--timeout`` limits each socket read on its own, so+without the budget a server that sends one byte every few seconds could hold+a run for hours."""from __future__ import annotations@@ -20,6 +35,7 @@ import argparseimport jsonimport osimport sys+import timefrom typing import Optionaltry:@@ -36,7 +52,7 @@ from url_safety import ( # noqa: E402URLSafetyError,decode_body,decode_response_text,- read_bounded,+ read_bounded_until,safe_requests_session,validate_url_strict,)@@ -69,6 +85,24 @@ _decode_response_content = decode_response_text# Largest body read, counted after decompression (shared with render_page).MAX_PAGE_BYTES = _MAX_PAGE_BYTES+# Timing keys of a raw fetch, in the order they are documented.+_TIMING_KEYS = ("ttfb_ms", "response_time_ms", "fetch_duration_ms")+++def _default_max_time(timeout: float) -> float:+ """Wall-clock budget for one fetch when the caller names none.++ Twice the per-read timeout, and never under a minute: 60 seconds at the+ default ``--timeout 30``. Tying it to the timeout means a caller who+ raises the timeout for a slow server is not cut short by a fixed budget;+ the floor keeps a short timeout from making the whole fetch impatient.+ """+ return max(60.0, 2.0 * timeout)+++def _ms(seconds: float) -> int:+ return int(round(seconds * 1000))+def fetch_page(url: str,@@ -77,13 +111,14 @@ def fetch_page(max_redirects: int = 5,user_agent: Optional[str] = None,max_bytes: int = MAX_PAGE_BYTES,+ max_time: Optional[float] = None,) -> dict:"""Fetch a web page and return response details.SSRF protection is delegated to url_safety.validate_url_strict +- safe_requests_session, which resolves DNS once, validates every A- record against private/loopback/reserved ranges, and pins the+ safe_requests_session, which resolves DNS once, validates every A and+ AAAA record against private/loopback/reserved ranges, and pins theconnection so the resolver cannot rebind between checks.Args:@@ -95,11 +130,39 @@ def fetch_page(max_bytes: Stop reading the body past this many bytes (afterdecompression); ``too_large`` is then True and ``content`` holdsthe part read.+ max_time: Wall-clock budget in seconds for the whole fetch, every+ redirect hop and the body included. Default: twice ``timeout``,+ at least 60. It is checked when the final response's headers+ arrive and after every piece of the body. When it passes,+ ``error`` says so and ``content`` holds the part read. A stall+ inside a connect, a header read or a redirect's own body is+ still limited by ``timeout`` per socket read, not by this.Returns:Dictionary with url, status_code, content, headers, redirect_chain,- redirect_details, too_large, and error.+ redirect_details, too_large, error, and three timings in+ milliseconds, each None when it was not reached:++ - ``ttfb_ms``: time to first byte of the final response, from+ sending that request (a new connection and its TLS handshake+ included) to the end of its headers.+ - ``response_time_ms``: the final request alone, from sending it to+ the last body byte read. Redirect hops do not inflate it.+ - ``fetch_duration_ms``: the whole call, URL validation and every+ redirect hop included. Set on every outcome, errors too.++ The timings are measured from this machine. They are not Core Web+ Vitals and not a field metric."""+ started = time.monotonic()+ if max_time is None:+ max_time = _default_max_time(timeout)+ deadline = started + max_time++ def finish() -> dict:+ result["fetch_duration_ms"] = _ms(time.monotonic() - started)+ return result+result: dict = {"url": url,"status_code": None,@@ -109,6 +172,9 @@ def fetch_page("redirect_details": [],"too_large": False,"error": None,+ "ttfb_ms": None,+ "response_time_ms": None,+ "fetch_duration_ms": None,}# Normalize scheme-less inputs (e.g. "example.com") before validation.@@ -119,7 +185,7 @@ def fetch_page(norm_url, _pinned_ip = validate_url_strict(url)except URLSafetyError as exc:result["error"] = f"url_safety: {exc}"- return result+ return finish()result["url"] = norm_url@@ -137,12 +203,27 @@ def fetch_page(allow_redirects=follow_redirects,stream=True,)- # Streamed so the body is read under the cap, inside the pinned- # session; .content must not be read after this.+ headers_at = time.monotonic()+ # Streamed so the body is read under the cap and the deadline,+ # inside the pinned session; .content must not be read after this.try:- raw, result["too_large"] = read_bounded(response, max_bytes)+ raw, result["too_large"], timed_out = read_bounded_until(+ response, max_bytes, deadline+ )finally:response.close()+ body_at = time.monotonic()++ # requests times each request from sending it to the end of its+ # headers, so the final response's ``elapsed`` leaves the hops out.+ ttfb = response.elapsed.total_seconds()+ result["ttfb_ms"] = _ms(ttfb)+ result["response_time_ms"] = _ms(ttfb + body_at - headers_at)+ if timed_out:+ result["error"] = (+ f"Fetch exceeded the {max_time:g} second wall-clock deadline; "+ "content holds the part read before it"+ )result["url"] = response.urlresult["status_code"] = response.status_code@@ -171,7 +252,7 @@ def fetch_page(# rebinding-pinned session is asked to chase it.result["error"] = f"url_safety: {e}"- return result+ return finish()def _non_negative_int(value: str) -> int:@@ -181,6 +262,13 @@ def _non_negative_int(value: str) -> int:return parsed+def _positive_seconds(value: str) -> float:+ parsed = float(value)+ if not 0 < parsed < float("inf"):+ raise argparse.ArgumentTypeError("must be a number of seconds above zero")+ return parsed++def _as_render_result(result: dict, *, mode_used: str) -> dict:"""Map a raw ``fetch_page()`` result onto the ``render_page()`` contract.@@ -219,6 +307,7 @@ def _as_render_result(result: dict, *, mode_used: str) -> dict:"mode_used": mode_used,"too_large": bool(result.get("too_large")),"error": result.get("error"),+ **{key: result.get(key) for key in _TIMING_KEYS},}@@ -231,6 +320,11 @@ def _emit_json(result: dict, output: Optional[str], *, max_text: int = 0) -> Non"""from render_page import _json_summary+ # A rendered result reports render_ms, not the raw-fetch timings; they+ # are null there so both paths keep one key set.+ for key in _TIMING_KEYS:+ result.setdefault(key, None)+output_written = Falseif output and not result.get("error"):with open(output, "w", encoding="utf-8") as f:@@ -258,6 +352,17 @@ def main():),)parser.add_argument("--timeout", "-t", type=int, default=30, help="Timeout in seconds")+ parser.add_argument(+ "--max-time",+ type=_positive_seconds,+ default=None,+ help=(+ "Wall-clock budget in seconds for the whole fetch, redirects and "+ "body included (default: twice --timeout, at least 60). When it "+ "passes, the fetch stops with an error and the part read. "+ "Raw HTTP only: --render is limited by --timeout."+ ),+ )parser.add_argument("--no-redirects", action="store_true", help="Don't follow redirects")parser.add_argument("--user-agent", help="Custom User-Agent string")parser.add_argument(@@ -319,6 +424,7 @@ def main():timeout=args.timeout,follow_redirects=not args.no_redirects,user_agent=ua,+ max_time=args.max_time,)if args.json:diff --git a/scripts/url_safety.py b/scripts/url_safety.pyindex 24af86b..45404aa 100644--- a/scripts/url_safety.py+++ b/scripts/url_safety.py@@ -17,9 +17,11 @@ validate_url(url) -> boolgoogle_auth.py.validate_url_strict(url) -> tuple[str, str]- Resolves the hostname via socket.getaddrinfo, validates every returned- A record against the safety predicate, and returns- ``(normalized_url, pinned_ipv4)``. Raises ``URLSafetyError`` if any+ Resolves the hostname via socket.getaddrinfo in both address families,+ validates every returned A and AAAA record against the safety+ predicate, and returns ``(normalized_url, pinned_ip)``. The pinned+ address is IPv4 whenever the host has an A record, and IPv6 only for a+ host that publishes AAAA records alone. Raises ``URLSafetyError`` if anyresolved IP is non-public. Use this whenever the caller is about toopen a network connection so DNS rebinding cannot swap a public IPfor a private one between checks.@@ -41,7 +43,10 @@ safe_requests_session(url) -> context manager yielding requests.Sessionis_safe_ip(ip_str) -> boolTrue iff the address parses as IPv4/IPv6 and is none of:- private, loopback, reserved, link-local, multicast, unspecified.+ private, loopback, reserved, link-local, multicast, unspecified,+ an IPv6 transition or special-purpose range (NAT64, 6to4, Teredo,+ site-local, documentation, ORCHID), or an address with a zone+ identifier.URLSafetyErrorValueError subclass raised by the strict validator and pinning helpers.@@ -91,12 +96,17 @@ import osimport reimport socketimport threading+import timefrom contextlib import contextmanagerfrom typing import Iterator, Optionalfrom urllib.parse import urlparsetry:import requests++ # urllib3 is the HTTP layer under requests, so it installs with it.+ from urllib3 import exceptions as _urllib3_exceptions+ from urllib3.response import BaseHTTPResponse as _Urllib3Responseexcept ImportError as exc: # pragma: no cover - hard dependencyraise RuntimeError("scripts/url_safety.py requires the 'requests' package. "@@ -203,21 +213,56 @@ def _reject_authority_confusion(url: str, parsed) -> None:# hand out 100.64/10 addresses for internal services._SHARED_ADDRESS_SPACE = ipaddress.ip_network("100.64.0.0/10")+# IPv6 ranges that are never a public audit target. The ``ipaddress`` flags+# used below already cover most of them, but those tables change between+# Python releases (2001:20::/28 is no longer ``is_private``, 2002::/16 only+# became so recently) and no flag covers site-local, so the policy is spelled+# out here and holds on every interpreter. The ranges that embed an IPv4+# address (NAT64, 6to4, Teredo, IPv4-compatible) are refused whole rather+# than judged by that address. The one place a NAT64 address is passed over+# is the DNS64 case in ``validate_url_strict``, where it is never dialled.+_NAT64_WELL_KNOWN = ipaddress.ip_network("64:ff9b::/96")+_UNSAFE_IPV6_NETWORKS = tuple(+ ipaddress.ip_network(network)+ for network in (+ "::/96", # IPv4-compatible (deprecated); holds :: and ::1+ "64:ff9b::/96", # NAT64 well-known prefix (RFC 6052)+ "64:ff9b:1::/48", # NAT64 local-use prefix (RFC 8215)+ "2001::/32", # Teredo+ "2001:10::/28", # ORCHID (deprecated)+ "2001:20::/28", # ORCHIDv2+ "2001:db8::/32", # documentation+ "2002::/16", # 6to4+ "3fff::/20", # documentation (RFC 9637)+ "fc00::/7", # unique-local; holds AWS IMDS fd00:ec2::254+ "fe80::/10", # link-local+ "fec0::/10", # site-local (deprecated)+ "ff00::/8", # multicast+ )+)+def is_safe_ip(ip_str: str) -> bool:"""Return True iff ``ip_str`` is a public unicast address.IPv4-mapped IPv6 (``::ffff:127.0.0.1``) is unwrapped and judged as theembedded IPv4 address, so every IPv4 rule below applies to it too.- IPv6 unique-local (``fc00::/7``) and link-local (``fe80::/10``) are- also rejected.+ Every other IPv6 address is checked against ``_UNSAFE_IPV6_NETWORKS``+ as well as the flags below. An address with a zone identifier+ (``fe80::1%eth0``) is refused: a zone only means something on the+ machine that names it."""try:ip = ipaddress.ip_address(ip_str)except ValueError:return False- if ip.version == 6 and ip.ipv4_mapped is not None:- ip = ip.ipv4_mapped+ if ip.version == 6:+ if ip.scope_id is not None:+ return False+ if ip.ipv4_mapped is not None:+ ip = ip.ipv4_mapped+ elif any(ip in network for network in _UNSAFE_IPV6_NETWORKS):+ return Falsereturn not (ip.is_privateor (ip.version == 4 and ip in _SHARED_ADDRESS_SPACE)@@ -290,6 +335,8 @@ def _is_allowlistable_ip(ip_str: str) -> bool:ip = ipaddress.ip_address(ip_str)except ValueError:return False+ if ip.version == 6 and ip.scope_id is not None:+ return False # Zone identifier, refused as in is_safe_ip.if ip.version == 6 and ip.ipv4_mapped is not None:ip = ip.ipv4_mappedif str(ip) in _NEVER_ALLOWLISTABLE_HOSTNAMES:@@ -468,19 +515,41 @@ def validate_url(url: str) -> bool:return allowlisted and _is_allowlistable_ip(hostname)+def _is_dns64_synthesized(ip_str: str) -> bool:+ """True for an address in the NAT64 well-known prefix, 64:ff9b::/96.++ The local-use prefix 64:ff9b:1::/48 and anything with a zone identifier+ do not count: neither is what a DNS64 resolver hands out by default.+ """+ try:+ ip = ipaddress.ip_address(ip_str)+ except ValueError:+ return False+ return ip.version == 6 and ip.scope_id is None and ip in _NAT64_WELL_KNOWN++def validate_url_strict(url: str) -> tuple[str, str]:"""Resolve and validate the URL's hostname.- Returns ``(url, pinned_ipv4)`` on success. Raises ``URLSafetyError`` if:+ Returns ``(url, pinned_ip)`` on success. Raises ``URLSafetyError`` if:- Scheme is invalid or hostname is missing.- Hostname is hard-blocked.- DNS resolution fails.- - Any A record resolves to a non-public IP (DNS rebinding refused).-- Multi-A-record handling: every returned record must be public. A- hostname with one public and one private A record is refused so an- attacker cannot race the resolver between validate and connect.+ - Any A or AAAA record resolves to a non-public IP (DNS rebinding+ refused).++ Multi-record handling: every returned record must be public, in both+ address families. A hostname with one public and one private record+ (two A records, or a public A beside a private AAAA) is refused so an+ attacker cannot race the resolver between validate and connect. The one+ record passed over is a DNS64 AAAA under 64:ff9b::/96 on a host that+ also has an A record: the pin is then IPv4 and that address is unused.++ The pinned address is the host's IPv4 address when it has one, so a+ dual-stack host is reached exactly as before. An IPv6 address is pinned+ only for a host that publishes AAAA records alone, or for an IPv6+ literal, which keeps its brackets in the returned URL."""parsed = urlparse(url)_reject_authority_confusion(url, parsed)@@ -518,21 +587,36 @@ def validate_url_strict(url: str) -> tuple[str, str]:addrinfo = socket.getaddrinfo(hostname,port,- family=socket.AF_INET,+ family=socket.AF_UNSPEC,type=socket.SOCK_STREAM,)except (socket.gaierror, UnicodeError) as exc:raise URLSafetyError(f"DNS resolution failed for {hostname}: {exc}") from exc- resolved_ips = sorted({info[4][0] for info in addrinfo})+ # IPv4 sorts first, so the pin below is IPv6 only when there is no A+ # record. Every address in the list is still checked, whichever is pinned.+ resolved_ips = sorted(+ {info[4][0] for info in addrinfo}, key=lambda ip: (":" in ip, ip)+ )if not resolved_ips:- raise URLSafetyError(f"No A records for {hostname}")+ raise URLSafetyError(f"No address records for {hostname}")++ # On a DNS64 network the resolver adds, to a name that has an A record+ # and no AAAA, a synthesized AAAA record under 64:ff9b::/96. Judged like any+ # other record it would refuse every IPv4-only site, which resolved fine+ # while this lookup asked for A records alone. With an A record present+ # the pin is IPv4, so that AAAA is never dialled: it is skipped here, not+ # trusted. ``is_safe_ip`` still refuses the prefix everywhere else, and a+ # host with no A record gets no such pass.+ has_a_record = ":" not in resolved_ips[0]for ip_str in resolved_ips:if is_safe_ip(ip_str):continueif allowlisted and _is_allowlistable_ip(ip_str):continue # Named in CLAUDE_SEO_LOCAL_TARGETS; see that block.+ if has_a_record and _is_dns64_synthesized(ip_str):+ continueraise URLSafetyError(f"DNS rebinding refused: {hostname} resolves to "f"non-public IP {ip_str}"@@ -695,6 +779,12 @@ def _pin_dns(other port the allowlist names for that host; see:func:`_local_pin_allows_port`.+ The pin answers in the family of ``pinned_ip``: an ``AF_INET`` tuple for+ an IPv4 address, an ``AF_INET6`` tuple with the four-part sockaddr for an+ IPv6 one. A lookup that asks for the other family only is refused, so a+ machine without IPv6 fails closed on an IPv6-only host instead of being+ handed an address it cannot connect to.+``exempt_hosts`` are resolved by the real resolver without thefall-through check. It carries the configured HTTP proxy, if any: theproxy is the process's own egress, and requests must resolve it to open@@ -726,6 +816,7 @@ def _pin_dns(# Such a pin must not answer for the host's other ports, or a redirect# from the allowed dev server reaches every service on that address.port_bound = not is_safe_ip(pinned_ip)+ pinned_family = socket.AF_INET6 if ":" in pinned_ip else socket.AF_INETdef patched(host, requested_port, *args, **kwargs):# Branch 1: the originally-pinned host returns the validated IP@@ -738,18 +829,22 @@ def _pin_dns(f"non-public host {host}; validated port is {port}",)family = kwargs.get("family", args[0] if args else 0)- if family in (0, socket.AF_UNSPEC, socket.AF_INET):+ if family in (0, socket.AF_UNSPEC, pinned_family):+ sockaddr = (pinned_ip, requested_port or port)+ if pinned_family == socket.AF_INET6:+ sockaddr += (0, 0) # flowinfo, scope_idreturn [(- socket.AF_INET,+ pinned_family,socket.SOCK_STREAM,socket.IPPROTO_TCP,"",- (pinned_ip, requested_port or port),+ sockaddr,)]+ pinned_kind = "IPv6" if pinned_family == socket.AF_INET6 else "IPv4"raise socket.gaierror(socket.EAI_FAIL,f"url_safety: address family {family} refused for pinned "- f"IPv4 host {host}",+ f"{pinned_kind} host {host}",)if host and host.lower() in exempt_hosts:@@ -952,18 +1047,73 @@ def read_bounded(response, max_bytes: int) -> tuple[bytes, bool]:before the chunk that crossed the cap, and whether the cap was hit.Decode the result with :func:`decode_body`, not ``response.content``."""- chunks = []+ raw, too_large, _timed_out = _read_capped(+ response.iter_content(chunk_size=65536), max_bytes, None+ )+ return raw, too_large+++def read_bounded_until(+ response, max_bytes: int, deadline: float+) -> tuple[bytes, bool, bool]:+ """:func:`read_bounded` with a wall-clock deadline as well.++ ``deadline`` is a ``time.monotonic()`` value. It is checked before the+ first read and after every piece of the body, and the body is taken as+ each socket read delivers it, so a server that drips a byte at a time is+ cut off at the deadline plus at most one socket read timeout. Returns the+ bytes read so far, whether the size cap was hit, and whether the deadline+ passed.+ """+ return _read_capped(_iter_as_received(response), max_bytes, deadline)+++def _read_capped(+ chunks, max_bytes: int, deadline: Optional[float]+) -> tuple[bytes, bool, bool]:+ """Join ``chunks`` until the size cap or the deadline stops the read."""+ parts: list[bytes] = []size = 0- too_large = False- for chunk in response.iter_content(chunk_size=65536):- if not chunk:- continue- size += len(chunk)- if size > max_bytes:- too_large = True- break- chunks.append(chunk)- return b"".join(chunks), too_large+ if deadline is not None and time.monotonic() >= deadline:+ return b"", False, True+ for chunk in chunks:+ if chunk:+ size += len(chunk)+ if size > max_bytes:+ return b"".join(parts), True, False+ parts.append(chunk)+ if deadline is not None and time.monotonic() >= deadline:+ return b"".join(parts), False, True+ return b"".join(parts), False, False+++def _iter_as_received(response, chunk_size: int = 65536) -> Iterator[bytes]:+ """Yield a streamed body as each socket read delivers it.++ ``iter_content(n)`` blocks until ``n`` bytes have arrived, so a slow drip+ could hold one 64 KiB chunk for hours between two deadline checks.+ urllib3's ``read1`` returns what a single read brings, still decompressed+ and at most ``chunk_size`` bytes. Errors are mapped to the requests+ exceptions ``iter_content`` raises.+ """+ raw = getattr(response, "raw", None)+ if not isinstance(raw, _Urllib3Response): # A test double, not a socket.+ yield from response.iter_content(chunk_size=chunk_size)+ return+ try:+ while True:+ data = raw.read1(chunk_size, decode_content=True)+ if not data:+ return+ yield data+ except _urllib3_exceptions.ProtocolError as exc:+ raise requests.exceptions.ChunkedEncodingError(exc) from exc+ except _urllib3_exceptions.DecodeError as exc:+ raise requests.exceptions.ContentDecodingError(exc) from exc+ except _urllib3_exceptions.ReadTimeoutError as exc:+ raise requests.exceptions.ConnectionError(exc) from exc+ except _urllib3_exceptions.SSLError as exc:+ raise requests.exceptions.SSLError(exc) from exc# Deepest JSON nesting a fetched document may have. Real discovery and@@ -1240,7 +1390,7 @@ def _cli() -> None:parser.add_argument("--strict",action="store_true",- help="Run DNS resolution and refuse on any non-public A record.",+ help="Run DNS resolution and refuse on any non-public A or AAAA record.",)parser.add_argument("--json",diff --git a/tests/test_fetch_page_cli.py b/tests/test_fetch_page_cli.pyindex 182bf1d..b12a14a 100644--- a/tests/test_fetch_page_cli.py+++ b/tests/test_fetch_page_cli.py@@ -79,6 +79,9 @@ def _expected_json(raw_source: dict, *, mode: str, output_written: bool, max_texraw_source if mode != "never" else fetch_page._as_render_result(raw_source, mode_used="raw"))summary = render_page._json_summary(normalized, max_text=max_text)+ # Raw-fetch timings: null on a rendered result, so both paths share keys.+ for key in fetch_page._TIMING_KEYS:+ summary.setdefault(key, None)summary["output_written"] = output_writtenreturn json.loads(json.dumps(summary, default=str))diff --git a/tests/test_fetch_page_timing.py b/tests/test_fetch_page_timing.pynew file mode 100644index 0000000..8a7ac69--- /dev/null+++ b/tests/test_fetch_page_timing.py@@ -0,0 +1,444 @@+"""fetch_page.py: the wall-clock deadline and the timing fields.++``--timeout`` limits each socket read on its own, so a server that sends one+byte every few seconds was never cut off. The deadline covers the whole+fetch. The slow servers here are real local stubs, reached through the narrow+CLAUDE_SEO_LOCAL_TARGETS allowlist the way ``test_redirect_body_cap.py`` does;+no network is used.+"""++from __future__ import annotations++import gzip+import http.server+import io+import json+import socket+import sys+import threading+import time+from contextlib import contextmanager+from pathlib import Path+from types import SimpleNamespace++import pytest+import requests+from urllib3.response import HTTPResponse++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))+import fetch_page # noqa: E402+import render_page # noqa: E402+import url_safety # noqa: E402++DRIP_BYTES = 400+DRIP_INTERVAL = 0.02 # the whole drip takes about 8 seconds if nothing stops it+TIMING_KEYS = ("ttfb_ms", "response_time_ms", "fetch_duration_ms")+++class _Handler(http.server.BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"++ def do_GET(self): # noqa: N802 (http.server API)+ if self.path == "/drip":+ self._headers(200, {"Content-Length": str(DRIP_BYTES)})+ self._drip(lambda: self.wfile.write(b"x"))+ elif self.path == "/drip-chunked":+ self._headers(200, {"Transfer-Encoding": "chunked"})+ self._drip(lambda: self.wfile.write(b"1\r\nx\r\n"))+ elif self.path == "/slow-redirect":+ time.sleep(0.2)+ self._headers(302, {"Location": "/slow-final", "Content-Length": "0"})+ elif self.path == "/slow-final":+ time.sleep(0.1)+ self._headers(200, {"Content-Length": "5"})+ time.sleep(0.15)+ self.wfile.write(b"final")+ else:+ self._headers(200, {"Content-Length": "5"})+ self.wfile.write(b"plain")++ def _headers(self, status: int, headers: dict) -> None:+ self.send_response(status)+ self.send_header("Content-Type", "text/html; charset=utf-8")+ self.send_header("Connection", "close")+ for name, value in headers.items():+ self.send_header(name, value)+ self.end_headers()+ self.close_connection = True++ def _drip(self, send_one) -> None:+ try:+ for _ in range(DRIP_BYTES):+ if self.server.stop.is_set():+ return+ send_one()+ time.sleep(DRIP_INTERVAL)+ except OSError:+ pass # The client hung up, which is the point of the test.++ def log_message(self, *args):+ pass+++def server(monkeypatch):+ httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler)+ httpd.daemon_threads = True+ httpd.stop = threading.Event()+ thread = threading.Thread(target=httpd.serve_forever, daemon=True)+ thread.start()+ port = httpd.server_address[1]+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", f"127.0.0.1:{port}")+ monkeypatch.setattr(url_safety.requests.utils, "getproxies", lambda: {})+ try:+ yield f"http://127.0.0.1:{port}"+ finally:+ httpd.stop.set()+ httpd.shutdown()+ httpd.server_close()+++# ---------------------------------------------------------------------------+# The deadline against a real slow server+# ---------------------------------------------------------------------------++[email protected]("path", ["/drip", "/drip-chunked"])+def test_dripping_body_is_cut_off_at_the_deadline(server, path):+ """One byte every 20 ms never trips the 5 second read timeout. With a+ Content-Length body, a 64 KiB chunked read would also sit through the+ whole drip before the deadline was looked at again."""+ began = time.monotonic()+ result = fetch_page.fetch_page(f"{server}{path}", timeout=5, max_time=0.4)+ took = time.monotonic() - began++ assert took < 4, f"the drip held the fetch for {took:.1f}s"+ assert "0.4 second wall-clock deadline" in result["error"]+ assert result["status_code"] == 200+ assert result["too_large"] is False+ # The partial body is returned, the way the size cap returns its part.+ assert 0 < len(result["content"]) < DRIP_BYTES+ assert set(result["content"]) == {"x"}+ assert result["fetch_duration_ms"] >= 400+ assert result["response_time_ms"] >= 300+ assert result["ttfb_ms"] is not None+++def test_fast_page_is_untouched_by_the_deadline(server):+ result = fetch_page.fetch_page(f"{server}/plain", timeout=5)+ assert result["error"] is None+ assert result["content"] == "plain"+ assert result["too_large"] is False+++def test_timing_fields_separate_the_final_request_from_the_redirect_hops(server):+ """The stub waits 0.2 s before the redirect, 0.1 s before the final+ headers and 0.15 s before the final body. Only lower bounds are asserted:+ a loaded machine can be slower, never faster."""+ result = fetch_page.fetch_page(f"{server}/slow-redirect", timeout=5)++ assert result["error"] is None+ assert result["content"] == "final"+ assert result["redirect_details"] == [{"url": f"{server}/slow-redirect", "status_code": 302}]+ for key in TIMING_KEYS:+ assert isinstance(result[key], int), key+ # Time to first byte of the final response: its 0.1 s wait, not the hop's.+ assert result["ttfb_ms"] >= 95+ # The final request alone: headers, then the 0.15 s before the body.+ assert result["response_time_ms"] >= result["ttfb_ms"] + 140+ # Every hop: the 0.2 s redirect is in this number and in no other.+ assert result["fetch_duration_ms"] >= result["response_time_ms"] + 190+++# ---------------------------------------------------------------------------+# read_bounded_until, with a clock the test controls+# ---------------------------------------------------------------------------+++class _Clock:+ def __init__(self) -> None:+ self.now = 0.0++ def monotonic(self) -> float:+ return self.now+++def _ticking_response(clock: _Clock, chunks: list, calls: list | None = None):+ """A response double whose every chunk takes one second to arrive."""+ def iter_content(chunk_size=None):+ if calls is not None:+ calls.append(chunk_size)+ for chunk in chunks:+ clock.now += 1.0+ yield chunk++ return SimpleNamespace(raw=None, iter_content=iter_content)+++def clock(monkeypatch):+ fake = _Clock()+ monkeypatch.setattr(url_safety, "time", fake)+ return fake+++def test_read_stops_at_the_deadline_and_keeps_what_arrived(clock):+ response = _ticking_response(clock, [b"ab"] * 10)+ assert url_safety.read_bounded_until(response, 1000, 3.0) == (b"ababab", False, True)+++def test_read_past_the_deadline_never_starts(clock):+ calls: list = []+ clock.now = 5.0+ response = _ticking_response(clock, [b"ab"], calls)+ assert url_safety.read_bounded_until(response, 1000, 5.0) == (b"", False, True)+ assert calls == []+++def test_complete_body_inside_the_deadline_is_not_flagged(clock):+ response = _ticking_response(clock, [b"ab", b"", b"cd"])+ assert url_safety.read_bounded_until(response, 1000, 60.0) == (b"abcd", False, False)+++def test_size_cap_still_wins_when_it_is_hit_first(clock):+ response = _ticking_response(clock, [b"ab"] * 10)+ assert url_safety.read_bounded_until(response, 3, 60.0) == (b"ab", True, False)+++def test_read_bounded_keeps_its_two_value_contract_and_has_no_deadline(clock):+ response = _ticking_response(clock, [b"ab"] * 10)+ clock.now = 10_000.0+ assert url_safety.read_bounded(response, 1000) == (b"ab" * 10, False)+++def _streamed_response(body: bytes, *, gzipped: bool = False, headers: dict | None = None,+ fp=None) -> requests.Response:+ """A streamed requests.Response built the way the HTTP adapter builds one."""+ all_headers = {"Content-Type": "text/html; charset=utf-8", **(headers or {})}+ payload = body+ if gzipped:+ payload = gzip.compress(body)+ all_headers["Content-Encoding"] = "gzip"+ raw = HTTPResponse(+ body=fp or io.BytesIO(payload), headers=all_headers, status=200, preload_content=False+ )+ resp = requests.Response()+ resp.status_code = 200+ resp.headers.update(all_headers)+ resp.raw = raw+ resp.url = "https://example.com/"+ return resp+++def test_deadline_read_still_counts_decompressed_bytes():+ far = time.monotonic() + 60+ bomb = _streamed_response(b"A" * (8 * 1024 * 1024), gzipped=True)+ raw, too_large, timed_out = url_safety.read_bounded_until(bomb, 1024 * 1024, far)+ assert too_large is True and timed_out is False+ assert 0 < len(raw) <= 1024 * 1024++ page = _streamed_response(b"B" * 300_000, gzipped=True)+ assert url_safety.read_bounded_until(page, 1024 * 1024, far) == (b"B" * 300_000, False, False)+++class _TimingOutBody(io.BytesIO):+ def read1(self, *args):+ raise socket.timeout("timed out")++ read = read1+++def test_deadline_read_raises_the_same_requests_errors_as_iter_content():+ """fetch_page catches requests exceptions only, so a urllib3 error from+ the direct read must not escape as itself."""+ far = time.monotonic() + 60+ stalled = _streamed_response(b"", fp=_TimingOutBody())+ with pytest.raises(requests.exceptions.ConnectionError, match="Read timed out"):+ url_safety.read_bounded_until(stalled, 1000, far)++ short = _streamed_response(b"abc", headers={"Content-Length": "10"})+ with pytest.raises(requests.exceptions.ChunkedEncodingError):+ url_safety.read_bounded_until(short, 1000, far)++ garbage = _streamed_response(b"not gzip", headers={"Content-Encoding": "gzip"})+ with pytest.raises(requests.exceptions.ContentDecodingError):+ url_safety.read_bounded_until(garbage, 1000, far)+++# ---------------------------------------------------------------------------+# fetch_page(): budget, result keys, error paths+# ---------------------------------------------------------------------------+++def _patched_fetch(monkeypatch, response: requests.Response, **kwargs) -> dict:+ class _Session:+ max_redirects = 5++ def get(self, url, **kw):+ return response++ @contextmanager+ def fake_session(_url):+ yield _Session()++ monkeypatch.setattr(fetch_page, "validate_url_strict", lambda url: (url, "93.184.216.34"))+ monkeypatch.setattr(fetch_page, "safe_requests_session", fake_session)+ return fetch_page.fetch_page("https://example.com/", **kwargs)+++ "timeout,expected", [(30, 60.0), (10, 60.0), (1, 60.0), (45, 90.0), (120, 240.0)]+)+def test_default_budget_is_twice_the_timeout_and_at_least_a_minute(timeout, expected):+ assert fetch_page._default_max_time(timeout) == expected+++def test_budget_spent_before_the_body_leaves_the_body_unread(monkeypatch):+ """Redirect hops count: if they used the budget, the body is not started."""+ page = _streamed_response(b"<html>never read</html>")+ result = _patched_fetch(monkeypatch, page, max_time=1e-9)+ assert "wall-clock deadline" in result["error"]+ assert result["content"] == ""+ assert result["status_code"] == 200+ assert page.raw.tell() == 0+++def test_default_budget_is_used_when_none_is_given(monkeypatch):+ seen: list = []++ def tiny(timeout):+ seen.append(timeout)+ return 1e-9++ monkeypatch.setattr(fetch_page, "_default_max_time", tiny)+ result = _patched_fetch(monkeypatch, _streamed_response(b"<html></html>"), timeout=17)+ assert seen == [17]+ assert "wall-clock deadline" in result["error"]+++def test_result_carries_integer_timings_on_success(monkeypatch):+ result = _patched_fetch(monkeypatch, _streamed_response(b"<html>ok</html>"))+ assert result["error"] is None+ assert result["content"] == "<html>ok</html>"+ for key in TIMING_KEYS:+ assert isinstance(result[key], int) and result[key] >= 0, key+++def test_refused_url_reports_only_the_total(monkeypatch):+ monkeypatch.delenv("CLAUDE_SEO_LOCAL_TARGETS", raising=False)+ result = fetch_page.fetch_page("http://127.0.0.1:1/")+ assert result["error"].startswith("url_safety:")+ assert result["ttfb_ms"] is None+ assert result["response_time_ms"] is None+ assert isinstance(result["fetch_duration_ms"], int)+++def test_read_error_reports_only_the_total(monkeypatch):+ stalled = _streamed_response(b"", fp=_TimingOutBody())+ result = _patched_fetch(monkeypatch, stalled)+ assert result["error"].startswith("Connection error:")+ assert result["ttfb_ms"] is None+ assert isinstance(result["fetch_duration_ms"], int)+++def test_timing_is_labelled_as_a_local_measurement():+ for doc in (fetch_page.__doc__, fetch_page.fetch_page.__doc__):+ assert "from this machine" in doc+ assert "not Core Web Vitals" in " ".join(doc.split())+ assert "field metric" in doc+++# ---------------------------------------------------------------------------+# CLI: --max-time and the JSON keys+# ---------------------------------------------------------------------------++_RAW = {+ "url": "https://example.com/",+ "status_code": 200,+ "content": "<p>partial",+ "headers": {},+ "redirect_chain": [],+ "redirect_details": [],+ "too_large": False,+ "error": None,+ "ttfb_ms": 120,+ "response_time_ms": 340,+ "fetch_duration_ms": 910,+}+++def _run_cli(monkeypatch, capsys, argv: list, result: dict):+ seen: dict = {}++ def fake_fetch(*args, **kwargs):+ seen.update(kwargs)+ return dict(result)++ monkeypatch.setattr(fetch_page, "fetch_page", fake_fetch)+ monkeypatch.setattr(sys, "argv", ["fetch_page.py", "https://example.com", *argv])+ code = 0+ try:+ fetch_page.main()+ except SystemExit as exc:+ code = exc.code+ return code, seen, capsys.readouterr()+++def test_cli_passes_max_time_through(monkeypatch, capsys):+ _code, seen, _out = _run_cli(monkeypatch, capsys, ["--max-time", "7.5"], _RAW)+ assert seen["max_time"] == 7.5+ _code, seen, _out = _run_cli(monkeypatch, capsys, [], _RAW)+ assert seen["max_time"] is None # fetch_page() then applies the default++[email protected]("value", ["0", "-5", "nan", "inf", "soon"])+def test_cli_rejects_a_budget_that_is_not_a_positive_number(monkeypatch, capsys, value):+ code, seen, captured = _run_cli(monkeypatch, capsys, ["--max-time", value], _RAW)+ assert code == 2+ assert seen == {}+ assert "--max-time" in captured.err+++def test_cli_json_reports_the_timings_of_a_raw_fetch(monkeypatch, capsys):+ code, _seen, captured = _run_cli(monkeypatch, capsys, ["--json"], _RAW)+ payload = json.loads(captured.out)+ assert code == 0+ assert {key: payload[key] for key in TIMING_KEYS} == {+ "ttfb_ms": 120,+ "response_time_ms": 340,+ "fetch_duration_ms": 910,+ }+++def test_cli_json_keeps_one_key_set_for_a_rendered_fetch(monkeypatch, capsys):+ """A rendered result has render_ms instead; the raw timings are null."""+ rendered = {+ "url": "https://example.com/", "status_code": 200, "content": "<p>x</p>",+ "raw_content": "<p>x</p>", "render_ms": 12.5, "mode_used": "rendered", "error": None,+ }+ monkeypatch.setattr(render_page, "render_page", lambda *a, **kw: dict(rendered))+ monkeypatch.setattr(+ sys, "argv", ["fetch_page.py", "https://example.com", "--render", "always", "--json"]+ )+ with pytest.raises(SystemExit) as exc:+ fetch_page.main()+ payload = json.loads(capsys.readouterr().out)+ assert exc.value.code == 0+ assert {key: payload[key] for key in TIMING_KEYS} == dict.fromkeys(TIMING_KEYS)+ assert payload["render_ms"] == 12.5+++def test_cli_deadline_is_an_error_that_keeps_the_partial_content(monkeypatch, capsys, tmp_path):+ cut = dict(_RAW, error="Fetch exceeded the 5 second wall-clock deadline; "+ "content holds the part read before it")+ output = tmp_path / "page.html"+ code, _seen, captured = _run_cli(+ monkeypatch, capsys, ["--json", "--output", str(output)], cut+ )+ payload = json.loads(captured.out)+ assert code == 1+ assert "wall-clock deadline" in payload["error"]+ assert payload["content"] == "<p>partial"+ # A cut-off page is never saved as if it were the page.+ assert payload["output_written"] is False+ assert not output.exists()diff --git a/tests/test_url_safety_ipv6.py b/tests/test_url_safety_ipv6.pynew file mode 100644index 0000000..7bc1897--- /dev/null+++ b/tests/test_url_safety_ipv6.py@@ -0,0 +1,819 @@+"""+IPv6 in scripts/url_safety.py: the address policy, dual-family resolution,+and a DNS pin that answers in the pinned address's own family.++The resolver and ``socket.socket`` are replaced throughout, and the end-to-end+cases drive the real ``requests`` and ``urllib3`` stack the way+``test_url_safety_port_pin.py`` does. No test here touches the network.+"""++from __future__ import annotations++import io+import ipaddress+import os+import socket+import sys+from unittest.mock import patch++import pytest++_SCRIPTS = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "scripts")+if _SCRIPTS not in sys.path:+ sys.path.insert(0, _SCRIPTS)++requests = pytest.importorskip("requests")+import url_safety # noqa: E402+import urllib3.util.connection as urllib3_connection # noqa: E402++PUBLIC_V4 = "93.184.216.34"+PUBLIC_V6 = "2606:4700:4700::1111"+OTHER_PUBLIC_V6 = "2001:4860:4860::8888"+++def _info(ip: str, port) -> tuple:+ """One getaddrinfo row, shaped the way the real resolver shapes it."""+ if ":" in ip:+ return (socket.AF_INET6, socket.SOCK_STREAM, 6, "", (ip, port, 0, 0))+ return (socket.AF_INET, socket.SOCK_STREAM, 6, "", (ip, port))+++def _resolver(records: dict, families: list | None = None):+ """getaddrinfo stand-in: ``records`` maps a name to its A and AAAA answers."""+ def fake_getaddrinfo(host, port, *args, **kwargs):+ if families is not None:+ families.append(kwargs.get("family", args[0] if args else 0))+ if host not in records:+ raise socket.gaierror(socket.EAI_NONAME, f"unmocked host {host!r}")+ return [_info(ip, port or 0) for ip in records[host]]++ return fake_getaddrinfo++[email protected](autouse=True)+def _isolated(monkeypatch):+ """No developer allowlist, no developer proxy."""+ monkeypatch.delenv("CLAUDE_SEO_LOCAL_TARGETS", raising=False)+ monkeypatch.setattr(url_safety.requests.utils, "getproxies", lambda: {})+++# ---------------------------------------------------------------------------+# is_safe_ip: every IPv6 range that is not a public audit target+# ---------------------------------------------------------------------------+++ "ip",+ [+ pytest.param("::1", id="loopback"),+ pytest.param("::", id="unspecified"),+ pytest.param("fe80::1", id="link-local"),+ pytest.param("febf::1", id="link-local-upper"),+ pytest.param("fc00::1", id="unique-local-fc"),+ pytest.param("fd12:3456:789a::1", id="unique-local-fd"),+ pytest.param("fec0::1", id="site-local"),+ pytest.param("feff::1", id="site-local-upper"),+ pytest.param("ff02::1", id="multicast-link"),+ pytest.param("ff0e::1", id="multicast-global"),+ pytest.param("::ffff:127.0.0.1", id="mapped-loopback"),+ pytest.param("::ffff:10.0.0.1", id="mapped-rfc1918"),+ pytest.param("::ffff:169.254.169.254", id="mapped-metadata"),+ pytest.param("::ffff:100.64.0.1", id="mapped-rfc6598"),+ pytest.param("::127.0.0.1", id="compatible-loopback"),+ pytest.param("::8.8.8.8", id="compatible-public"),+ pytest.param("64:ff9b::10.0.0.1", id="nat64-private"),+ pytest.param("64:ff9b::169.254.169.254", id="nat64-metadata"),+ pytest.param("64:ff9b::8.8.8.8", id="nat64-public"),+ pytest.param("64:ff9b:1::1", id="nat64-local-use"),+ pytest.param("64:ff9b:1:ffff::8.8.8.8", id="nat64-local-use-upper"),+ pytest.param("2002:7f00:1::1", id="6to4-loopback"),+ pytest.param("2002:a00:1::1", id="6to4-rfc1918"),+ pytest.param("2002:808:808::1", id="6to4-public"),+ pytest.param("2001::1", id="teredo"),+ pytest.param("2001:0:4136:e378:8000:63bf:3fff:fdd2", id="teredo-real-shape"),+ pytest.param("2001:db8::1", id="documentation"),+ pytest.param("3fff::1", id="documentation-rfc9637"),+ pytest.param("2001:10::1", id="orchid"),+ pytest.param("2001:1f:ffff::1", id="orchid-upper"),+ pytest.param("2001:20::1", id="orchid-v2"),+ pytest.param("2001:2f:ffff::1", id="orchid-v2-upper"),+ pytest.param("fd00:ec2::254", id="aws-imds"),+ ],+)+def test_is_safe_ip_refuses_ipv6_special_ranges(ip: str) -> None:+ assert url_safety.is_safe_ip(ip) is False+++ "network", [str(network) for network in url_safety._UNSAFE_IPV6_NETWORKS]+)+def test_every_unsafe_ipv6_network_is_refused_at_both_ends(network: str) -> None:+ parsed = ipaddress.ip_network(network)+ assert url_safety.is_safe_ip(str(parsed[0])) is False+ assert url_safety.is_safe_ip(str(parsed[-1])) is False+++def test_the_unsafe_ipv6_table_names_every_range_the_policy_promises() -> None:+ """The flags in ``ipaddress`` move between Python releases; the table is+ what the policy rests on, so a range cannot drop out of it unnoticed."""+ assert {str(network) for network in url_safety._UNSAFE_IPV6_NETWORKS} == {+ "::/96",+ "64:ff9b::/96",+ "64:ff9b:1::/48",+ "2001::/32",+ "2001:10::/28",+ "2001:20::/28",+ "2001:db8::/32",+ "2002::/16",+ "3fff::/20",+ "fc00::/7",+ "fe80::/10",+ "fec0::/10",+ "ff00::/8",+ }+++ "ip",+ [+ "fe80::1%eth0",+ "fe80::1%1",+ "::1%lo",+ f"{PUBLIC_V6}%eth0", # public address, still machine-local with a zone+ "::ffff:8.8.8.8%eth0",+ "fe80::1%",+ ],+)+def test_is_safe_ip_refuses_zone_identifiers(ip: str) -> None:+ assert url_safety.is_safe_ip(ip) is False+ assert url_safety._is_allowlistable_ip(ip) is False+++ "ip,expected",+ [+ ("::ffff:8.8.8.8", True),+ ("::ffff:1.1.1.1", True),+ ("::ffff:127.0.0.1", False),+ ("::ffff:192.168.1.1", False),+ ("::ffff:100.100.100.200", False),+ ("::ffff:0.0.0.0", False),+ ("::ffff:224.0.0.1", False),+ ],+)+def test_ipv4_mapped_ipv6_is_judged_by_the_embedded_ipv4(ip: str, expected: bool) -> None:+ assert url_safety.is_safe_ip(ip) is expected+++ "ip",+ [+ PUBLIC_V6,+ OTHER_PUBLIC_V6,+ "2620:fe::fe",+ "2a00:1450:4001:81b::200e",+ "2001:200::1", # first block past the IETF 2001::/23 assignments+ "2001:db9::1", # one past the documentation range+ "2003::1", # one past 6to4+ ],+)+def test_is_safe_ip_still_accepts_public_ipv6(ip: str) -> None:+ assert url_safety.is_safe_ip(ip) is True+++# ---------------------------------------------------------------------------+# validate_url and validate_url_strict: IPv6 literals+# ---------------------------------------------------------------------------+++ "url,expected",+ [+ (f"http://[{PUBLIC_V6}]/", True),+ (f"https://[{PUBLIC_V6}]:8443/path?q=1", True),+ ("http://[::ffff:8.8.8.8]/", True),+ ("http://[::1]/", False),+ ("http://[::]/", False),+ ("http://[fe80::1]/", False),+ ("http://[fd00::1]/", False),+ ("http://[fec0::1]/", False),+ ("http://[ff02::1]/", False),+ ("http://[::ffff:127.0.0.1]/", False),+ ("http://[::127.0.0.1]/", False),+ ("http://[64:ff9b::7f00:1]/", False),+ ("http://[64:ff9b:1::1]/", False),+ ("http://[2002:7f00:1::1]/", False),+ ("http://[2001::1]/", False),+ ("http://[2001:db8::1]/", False),+ ("http://[2001:10::1]/", False),+ ("http://[2001:20::1]/", False),+ ("http://[fd00:ec2::254]/latest/meta-data/", False),+ ],+)+def test_validate_url_ipv6_literals(url: str, expected: bool) -> None:+ assert url_safety.validate_url(url) is expected+ if expected:+ assert url_safety.validate_url_strict(url)[0] == url+ else:+ with pytest.raises(url_safety.URLSafetyError):+ url_safety.validate_url_strict(url)+++ "url",+ [+ "http://[fe80::1%25eth0]/",+ "http://[fe80::1%eth0]/",+ f"http://[{PUBLIC_V6}%25eth0]/",+ f"http://[{PUBLIC_V6}%eth0]:8080/",+ ],+)+def test_zone_identifier_in_a_url_is_refused(url: str) -> None:+ assert url_safety.validate_url(url) is False+ with pytest.raises(url_safety.URLSafetyError):+ url_safety.validate_url_strict(url)+++def test_strict_keeps_the_brackets_of_a_public_ipv6_literal() -> None:+ url = f"https://[{PUBLIC_V6}]:8443/path?q=1"+ with patch.object(+ url_safety.socket, "getaddrinfo", side_effect=AssertionError("literal resolved")+ ):+ assert url_safety.validate_url_strict(url) == (url, PUBLIC_V6)+++# ---------------------------------------------------------------------------+# validate_url_strict: both families resolved, every record checked+# ---------------------------------------------------------------------------+++def test_strict_resolves_both_families_in_one_lookup() -> None:+ families: list = []+ resolver = _resolver({"example.com": [PUBLIC_V4]}, families)+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ url_safety.validate_url_strict("https://example.com/")+ assert families == [socket.AF_UNSPEC]+++def test_strict_pins_ipv6_for_a_host_with_only_aaaa_records() -> None:+ """Before this, an AAAA-only host failed with 'DNS resolution failed'."""+ resolver = _resolver({"v6only.example": [OTHER_PUBLIC_V6, PUBLIC_V6]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ url, pinned = url_safety.validate_url_strict("https://v6only.example/")+ assert url == "https://v6only.example/"+ assert pinned == OTHER_PUBLIC_V6 # lowest address, as for A records+++ "answers",+ [+ [PUBLIC_V4, PUBLIC_V6],+ [PUBLIC_V6, PUBLIC_V4],+ [PUBLIC_V6, OTHER_PUBLIC_V6, "93.184.216.99", PUBLIC_V4],+ ],+)+def test_strict_prefers_ipv4_for_a_dual_stack_host(answers) -> None:+ """Whatever order the resolver uses, the pin is the address it always was."""+ resolver = _resolver({"dual.example": answers})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ assert url_safety.validate_url_strict("https://dual.example/")[1] == PUBLIC_V4+++ "answers,refused",+ [+ ([PUBLIC_V4, "fd00::1"], "fd00::1"), # public A, private AAAA+ ([PUBLIC_V4, "::1"], "::1"),+ ([PUBLIC_V4, "fe80::1"], "fe80::1"),+ ([PUBLIC_V4, "fec0::1"], "fec0::1"),+ ([PUBLIC_V4, "::ffff:10.0.0.1"], "::ffff:10.0.0.1"),+ ([PUBLIC_V4, "fc00::1"], "fc00::1"),+ ([PUBLIC_V4, "64:ff9b:1::1"], "64:ff9b:1::1"), # local-use NAT64+ ([PUBLIC_V4, "2002:a00:1::1"], "2002:a00:1::1"),+ ([PUBLIC_V4, "fd00:ec2::254"], "fd00:ec2::254"),+ ([PUBLIC_V6, "10.0.0.7"], "10.0.0.7"), # public AAAA, private A+ ([PUBLIC_V6, "169.254.169.254"], "169.254.169.254"),+ ([PUBLIC_V6, "fd00::1"], "fd00::1"), # two AAAA, one private+ ([PUBLIC_V4, PUBLIC_V6, "fd00::1"], "fd00::1"),+ ],+)+def test_strict_refuses_a_host_with_any_unsafe_record_in_either_family(+ answers, refused+) -> None:+ """The multi-A-record rule, across both families: one bad record refuses+ the host, even though the address that would be pinned is public."""+ resolver = _resolver({"mixed.example": answers})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP") as excinfo:+ url_safety.validate_url_strict("https://mixed.example/")+ assert str(excinfo.value).endswith(f"non-public IP {refused}")+++ "address",+ [+ "::1", "fe80::1", "fd00::1", "fec0::1", "ff02::1", "::ffff:127.0.0.1",+ "::10.0.0.1", "64:ff9b::808:808", "64:ff9b:1::1", "2002:808:808::1",+ "2001::1", "2001:db8::1", "2001:10::1", "2001:20::1", "fd00:ec2::254",+ "fe80::1%eth0", f"{PUBLIC_V6}%eth0",+ ],+)+def test_strict_refuses_an_aaaa_only_host_in_an_unsafe_range(address: str) -> None:+ resolver = _resolver({"v6only.example": [address]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP"):+ url_safety.validate_url_strict("https://v6only.example/")+++# ---------------------------------------------------------------------------+# DNS64: a synthesized 64:ff9b::/96 AAAA beside an A record is passed over+# ---------------------------------------------------------------------------+++ "nat64",+ [+ "64:ff9b::5db8:d822", # the same public IPv4, as DNS64 synthesizes it+ "64:ff9b::808:808", # another public IPv4+ "64:ff9b::7f00:1", # loopback embedded: still never dialled+ "64:ff9b::a9fe:a9fe", # 169.254.169.254 embedded+ ],+)+def test_dns64_aaaa_beside_an_a_record_is_ignored_and_ipv4_is_pinned(nat64) -> None:+ """An IPv4-only site on a DNS64 network: one real A record plus one+ synthesized AAAA. It resolved before this lookup asked for both families+ and must keep resolving, pinned to the A record."""+ resolver = _resolver({"v4only.example": [nat64, PUBLIC_V4]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ url, pinned = url_safety.validate_url_strict("https://v4only.example/")+ assert (url, pinned) == ("https://v4only.example/", PUBLIC_V4)++ # The pin for that host answers in IPv4 only, so the AAAA has no way in.+ with url_safety._pin_dns("v4only.example", pinned, 443):+ assert socket.getaddrinfo("v4only.example", 443) == [+ (socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (PUBLIC_V4, 443))+ ]+ with pytest.raises(socket.gaierror, match="refused for pinned IPv4 host"):+ socket.getaddrinfo("v4only.example", 443, socket.AF_INET6)+++ "answers,refused",+ [+ (["10.0.0.7", "64:ff9b::a00:7"], "10.0.0.7"), # private A+ (["127.0.0.1", "64:ff9b::7f00:1"], "127.0.0.1"),+ ([PUBLIC_V4, "10.0.0.7", "64:ff9b::5db8:d822"], "10.0.0.7"),+ ([PUBLIC_V4, "64:ff9b::5db8:d822", "fc00::1"], "fc00::1"),+ ([PUBLIC_V4, "64:ff9b::5db8:d822", "64:ff9b:1::1"], "64:ff9b:1::1"),+ ([PUBLIC_V4, "64:ff9b::5db8:d822%eth0"], "64:ff9b::5db8:d822%eth0"),+ ],+)+def test_dns64_pass_covers_nothing_but_the_well_known_prefix(answers, refused) -> None:+ """The A records are judged as always, and every other unsafe AAAA,+ the local-use NAT64 prefix included, still refuses the host."""+ resolver = _resolver({"mixed.example": answers})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP") as excinfo:+ url_safety.validate_url_strict("https://mixed.example/")+ assert str(excinfo.value).endswith(f"non-public IP {refused}")+++ "answers",+ [+ ["64:ff9b::5db8:d822"],+ ["64:ff9b::5db8:d822", "64:ff9b::808:808"],+ ["64:ff9b::5db8:d822", PUBLIC_V6], # no A record, so no pass+ ["64:ff9b::5db8:d822", "::ffff:8.8.8.8"], # IPv4-mapped is not an A record+ ],+)+def test_host_without_an_a_record_gets_no_dns64_pass(answers) -> None:+ resolver = _resolver({"v6only.example": answers})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(+ url_safety.URLSafetyError, match="non-public IP 64:ff9b::5db8:d822"+ ):+ url_safety.validate_url_strict("https://v6only.example/")+++def test_nat64_stays_refused_everywhere_but_the_dns64_case() -> None:+ """is_safe_ip is as strict as before: literals, redirect targets inside+ the pinned scope and the Playwright handler all still refuse the prefix."""+ assert url_safety.is_safe_ip("64:ff9b::5db8:d822") is False+ assert url_safety.validate_url("http://[64:ff9b::5db8:d822]/") is False+ with pytest.raises(url_safety.URLSafetyError, match="Blocked IP literal"):+ url_safety.validate_url_strict("http://[64:ff9b::5db8:d822]/")++ resolver = _resolver({"redirected.example": [PUBLIC_V4, "64:ff9b::5db8:d822"]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with url_safety._pin_dns("pinned.example", PUBLIC_V4, 443):+ with pytest.raises(socket.gaierror, match="non-public IP"):+ socket.getaddrinfo("redirected.example", 443)+++def test_strict_reports_a_host_with_no_records() -> None:+ with patch.object(url_safety.socket, "getaddrinfo", return_value=[]):+ with pytest.raises(url_safety.URLSafetyError, match="No address records"):+ url_safety.validate_url_strict("https://empty.example/")+++# ---------------------------------------------------------------------------+# CLAUDE_SEO_LOCAL_TARGETS with IPv6+# ---------------------------------------------------------------------------+++def test_allowlisted_ipv6_loopback_literal_is_pinned_as_ipv6(monkeypatch) -> None:+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")+ assert url_safety.validate_url_strict("http://[::1]:3000/") == (+ "http://[::1]:3000/",+ "::1",+ )+ with pytest.raises(url_safety.URLSafetyError, match="Blocked hostname"):+ url_safety.validate_url_strict("http://[::1]:3001/")+++def test_allowlisted_localhost_with_both_families_still_pins_ipv4(monkeypatch) -> None:+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "localhost:3000")+ resolver = _resolver({"localhost": ["::1", "127.0.0.1"]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ assert url_safety.validate_url_strict("http://localhost:3000/")[1] == "127.0.0.1"+++def test_allowlisted_name_may_resolve_to_unique_local_ipv6(monkeypatch) -> None:+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "staging.internal:8080")+ resolver = _resolver({"staging.internal": ["fd00::5"]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ assert url_safety.validate_url_strict("http://staging.internal:8080/")[1] == "fd00::5"+++ "address", ["fe80::1", "fd00:ec2::254", "ff02::1", "::", "fd00::5%eth0", "64:ff9b::a00:1"]+)+def test_allowlist_never_admits_these_ipv6_addresses(monkeypatch, address) -> None:+ """Link-local, metadata, multicast, unspecified, zoned and NAT64 answers+ stay refused for a listed name, exactly as 169.254.169.254 does."""+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "staging.internal:8080")+ resolver = _resolver({"staging.internal": [address]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(url_safety.URLSafetyError, match="DNS rebinding refused"):+ url_safety.validate_url_strict("http://staging.internal:8080/")+++def test_allowlist_does_not_forgive_an_unsafe_aaaa_on_an_unlisted_host(monkeypatch) -> None:+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "staging.internal:8080")+ resolver = _resolver({"other.example": [PUBLIC_V4, "fd00::5"]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP fd00::5"):+ url_safety.validate_url_strict("http://other.example:8080/")+++# ---------------------------------------------------------------------------+# _pin_dns: the tuple matches the pinned family, the caller's family is honoured+# ---------------------------------------------------------------------------+++ "args,kwargs",+ [+ ((), {}),+ ((socket.AF_UNSPEC,), {}),+ ((socket.AF_INET6,), {}),+ ((socket.AF_INET6, socket.SOCK_STREAM), {}),+ ((), {"family": socket.AF_UNSPEC}),+ ((), {"family": socket.AF_INET6, "type": socket.SOCK_STREAM}),+ ],+)+def test_ipv6_pin_returns_an_af_inet6_row_with_a_four_part_sockaddr(args, kwargs) -> None:+ with url_safety._pin_dns("v6only.example", PUBLIC_V6, 443):+ rows = socket.getaddrinfo("v6only.example", 443, *args, **kwargs)+ assert rows == [+ (socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (PUBLIC_V6, 443, 0, 0))+ ]+++ "args,kwargs", [((socket.AF_INET,), {}), ((), {"family": socket.AF_INET})]+)+def test_ipv6_pin_fails_closed_for_an_ipv4_only_lookup(args, kwargs) -> None:+ """No AF_INET row carrying an IPv6 address, and no fall-back lookup."""+ with patch.object(+ url_safety.socket, "getaddrinfo", side_effect=AssertionError("real resolver used")+ ):+ with url_safety._pin_dns("v6only.example", PUBLIC_V6, 443):+ with pytest.raises(socket.gaierror, match="refused for pinned IPv6 host"):+ socket.getaddrinfo("v6only.example", 443, *args, **kwargs)+++ "args,kwargs", [((socket.AF_INET6,), {}), ((), {"family": socket.AF_INET6})]+)+def test_ipv4_pin_fails_closed_for_an_ipv6_only_lookup(args, kwargs) -> None:+ with url_safety._pin_dns("example.com", PUBLIC_V4, 443):+ with pytest.raises(socket.gaierror, match="refused for pinned IPv4 host"):+ socket.getaddrinfo("example.com", 443, *args, **kwargs)++[email protected]("family", [None, socket.AF_UNSPEC, socket.AF_INET])+def test_ipv4_pin_row_is_unchanged(family) -> None:+ kwargs = {} if family is None else {"family": family}+ with url_safety._pin_dns("example.com", PUBLIC_V4, 443):+ rows = socket.getaddrinfo("example.com", 8443, **kwargs)+ assert rows == [+ (socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (PUBLIC_V4, 8443))+ ]++[email protected]("requested_port", [80, 443, 8080, None, 0])+def test_public_ipv6_pin_answers_for_any_port(requested_port) -> None:+ with url_safety._pin_dns("v6only.example", PUBLIC_V6, 80):+ sockaddr = socket.getaddrinfo("v6only.example", requested_port)[0][4]+ assert sockaddr == (PUBLIC_V6, requested_port or 80, 0, 0)+++def test_ipv6_local_pin_is_bound_to_the_validated_port(monkeypatch) -> None:+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")+ with url_safety._pin_dns("::1", "::1", 3000):+ assert socket.getaddrinfo("::1", 3000)[0][4] == ("::1", 3000, 0, 0)+ assert socket.getaddrinfo("::1", None)[0][4] == ("::1", 3000, 0, 0)+ for port in (6379, 80, 443, 3001, "redis"):+ with pytest.raises(socket.gaierror, match="validated port is 3000"):+ socket.getaddrinfo("::1", port)+++def test_ipv6_local_pin_is_port_bound_without_an_allowlist() -> None:+ with url_safety._pin_dns("staging.internal", "fd00::5", 8080):+ assert socket.getaddrinfo("staging.internal", 8080)[0][4] == ("fd00::5", 8080, 0, 0)+ with pytest.raises(socket.gaierror, match="validated port is 8080"):+ socket.getaddrinfo("staging.internal", 9200)+++def test_bare_ipv6_allowlist_entry_keeps_matching_any_port(monkeypatch) -> None:+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "fd00::5")+ with url_safety._pin_dns("fd00::5", "fd00::5", 80):+ assert socket.getaddrinfo("fd00::5", 8080)[0][4] == ("fd00::5", 8080, 0, 0)+++ "address", ["::1", "fd00::1", "fec0::1", "64:ff9b::a00:1", "2002:a00:1::1", "fe80::1%eth0"]+)+def test_redirect_target_on_an_unsafe_ipv6_address_is_refused_in_the_pinned_scope(+ address,+) -> None:+ resolver = _resolver({"redirected.example": [PUBLIC_V4, address]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with url_safety._pin_dns("pinned.example", PUBLIC_V6, 443):+ with pytest.raises(socket.gaierror, match="non-public IP"):+ socket.getaddrinfo("redirected.example", 443)+++def test_redirect_target_on_public_ipv6_resolves_in_the_pinned_scope() -> None:+ resolver = _resolver({"elsewhere.example": [PUBLIC_V6]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with url_safety._pin_dns("pinned.example", PUBLIC_V4, 443):+ assert socket.getaddrinfo("elsewhere.example", 443)[0][4][0] == PUBLIC_V6+++# ---------------------------------------------------------------------------+# Proxy validation picks up the same IPv6 policy+# ---------------------------------------------------------------------------++[email protected]("proxy_host", ["::1", "fd00::1", "fec0::1", "64:ff9b::808:808", "2001:20::1"])+def test_proxy_on_a_non_public_ipv6_literal_is_refused(proxy_host) -> None:+ with pytest.raises(url_safety.URLSafetyError, match="Refusing configured HTTP proxy"):+ url_safety._assert_proxy_host_is_public(proxy_host)+++def test_proxy_name_with_an_unsafe_aaaa_record_is_refused() -> None:+ resolver = _resolver({"proxy.corp.example": ["8.8.4.4", "fec0::1"]})+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=resolver):+ with pytest.raises(url_safety.URLSafetyError, match="non-public IP fec0::1"):+ url_safety._assert_proxy_host_is_public("proxy.corp.example")+++# ---------------------------------------------------------------------------+# End to end through requests and urllib3+# ---------------------------------------------------------------------------+++def _redirect(location: str) -> bytes:+ return (+ f"HTTP/1.1 302 Found\r\nLocation: {location}\r\n"+ "Content-Length: 0\r\nConnection: close\r\n\r\n"+ ).encode("ascii")+++def _ok(body: str) -> bytes:+ return (+ f"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n"+ f"Content-Length: {len(body)}\r\nConnection: close\r\n\r\n{body}"+ ).encode("ascii")+++class _FakeNet:+ """Canned HTTP servers keyed by ``(ip, port)``, plus what reached them."""++ def __init__(self, hosts: dict, servers: dict) -> None:+ self.hosts = hosts+ self.servers = servers+ self.connections: list = [] # (socket family, sockaddr)+ self.sent: list = []++ def getaddrinfo(self, host, port, *args, **kwargs):+ if host in self.hosts:+ ips = self.hosts[host]+ else:+ try:+ ipaddress.ip_address(host)+ except ValueError:+ raise socket.gaierror(socket.EAI_NONAME, f"unmocked host {host!r}") from None+ ips = [host]+ return [_info(ip, port or 0) for ip in ips]++ def socket(self, family=socket.AF_INET, *args, **kwargs):+ return _FakeSocket(self, family)+++class _FakeSocket:+ """The slice of ``socket.socket`` that urllib3 and http.client use."""++ def __init__(self, net: _FakeNet, family) -> None:+ self._net = net+ self._family = family+ self._key = None++ def connect(self, address) -> None:+ self._net.connections.append((self._family, address))+ key = (address[0], address[1])+ if key not in self._net.servers:+ raise ConnectionRefusedError(111, "no fake server on this port")+ self._key = key++ def makefile(self, *args, **kwargs):+ return io.BytesIO(self._net.servers[self._key])++ def sendall(self, data) -> None:+ self._net.sent.append(bytes(data))++ def setsockopt(self, *args) -> None:+ pass++ def settimeout(self, value) -> None:+ pass++ def shutdown(self, how) -> None:+ pass++ def close(self) -> None:+ pass+++def _get(url: str, net: _FakeNet, *, gai_family=socket.AF_UNSPEC):+ """``safe_requests_get`` against the fake network. ``gai_family`` is what+ urllib3 asks the resolver for: AF_UNSPEC on a machine with IPv6, AF_INET+ on one without. It is set here so the result does not depend on the host."""+ with patch.object(url_safety.socket, "getaddrinfo", side_effect=net.getaddrinfo), \+ patch.object(url_safety.socket, "socket", side_effect=net.socket), \+ patch.object(urllib3_connection, "allowed_gai_family", return_value=gai_family):+ return url_safety.safe_requests_get(url, timeout=5)+++def test_ipv6_only_host_is_fetched_over_ipv6() -> None:+ net = _FakeNet({"v6only.example": [PUBLIC_V6]}, {(PUBLIC_V6, 80): _ok("over ipv6")})++ response = _get("http://v6only.example/page", net)++ assert response.status_code == 200+ assert response.content == b"over ipv6"+ assert net.connections == [(socket.AF_INET6, (PUBLIC_V6, 80, 0, 0))]+ assert b"Host: v6only.example\r\n" in b"".join(net.sent)+++def test_ipv6_literal_keeps_its_brackets_in_the_url_and_host_header() -> None:+ url = f"http://[{PUBLIC_V6}]:8080/page"+ net = _FakeNet({}, {(PUBLIC_V6, 8080): _ok("literal")})++ response = _get(url, net)++ assert response.content == b"literal"+ assert response.url == url+ assert response.request.url == url+ assert net.connections == [(socket.AF_INET6, (PUBLIC_V6, 8080, 0, 0))]+ assert f"Host: [{PUBLIC_V6}]:8080\r\n".encode("ascii") in b"".join(net.sent)+++def test_dual_stack_host_still_connects_over_ipv4() -> None:+ net = _FakeNet(+ {"dual.example": [PUBLIC_V6, PUBLIC_V4]},+ {(PUBLIC_V4, 80): _ok("over ipv4"), (PUBLIC_V6, 80): _ok("over ipv6")},+ )++ assert _get("http://dual.example/", net).content == b"over ipv4"+ assert net.connections == [(socket.AF_INET, (PUBLIC_V4, 80))]+++def test_ipv4_only_site_on_a_dns64_network_is_fetched_over_ipv4() -> None:+ """Even with a server answering at the synthesized address, only the+ pinned A record is dialled."""+ nat64 = "64:ff9b::5db8:d822"+ net = _FakeNet(+ {"v4only.example": [nat64, PUBLIC_V4]},+ {(PUBLIC_V4, 80): _ok("over ipv4"), (nat64, 80): _ok("over nat64")},+ )++ assert _get("http://v4only.example/", net).content == b"over ipv4"+ assert net.connections == [(socket.AF_INET, (PUBLIC_V4, 80))]+++def test_ipv6_only_host_fails_closed_on_a_machine_without_ipv6() -> None:+ """urllib3 asks for AF_INET only there. The pin refuses instead of+ handing back an IPv6 address in an IPv4 row, and nothing is connected."""+ net = _FakeNet({"v6only.example": [PUBLIC_V6]}, {(PUBLIC_V6, 80): _ok("over ipv6")})++ with pytest.raises(requests.exceptions.ConnectionError, match="refused for pinned IPv6 host"):+ _get("http://v6only.example/", net, gai_family=socket.AF_INET)++ assert net.connections == []+++def test_ipv6_host_redirect_to_another_port_is_followed_when_public() -> None:+ net = _FakeNet(+ {"v6only.example": [PUBLIC_V6]},+ {+ (PUBLIC_V6, 80): _redirect("http://v6only.example:8080/app"),+ (PUBLIC_V6, 8080): _ok("public app"),+ },+ )++ assert _get("http://v6only.example/", net).content == b"public app"+ assert net.connections == [+ (socket.AF_INET6, (PUBLIC_V6, 80, 0, 0)),+ (socket.AF_INET6, (PUBLIC_V6, 8080, 0, 0)),+ ]+++def test_redirect_to_another_port_on_an_allowed_ipv6_local_host_is_refused(monkeypatch) -> None:+ """The port-bound rule for local targets, unchanged, on an IPv6 pin."""+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")+ net = _FakeNet(+ {},+ {+ ("::1", 3000): _redirect("http://[::1]:6379/"),+ ("::1", 6379): _ok("internal service"),+ },+ )++ with pytest.raises(requests.exceptions.ConnectionError, match="validated port"):+ _get("http://[::1]:3000/", net)++ assert net.connections == [(socket.AF_INET6, ("::1", 3000, 0, 0))]+++ "location",+ [+ "http://[0:0:0:0:0:0:0:1]:6379/", # the same address, written out+ "http://[::ffff:127.0.0.1]:6379/", # its IPv4-mapped neighbour+ "http://localhost:6379/", # a name for it+ "http://127.0.0.1:6379/",+ ],+)+def test_another_spelling_of_the_allowed_ipv6_host_does_not_escape_the_port_rule(+ monkeypatch, location+) -> None:+ """A redirect cannot reach another port by naming the pinned loopback+ address differently: any spelling that is not the pinned host is a new+ host, and a new host on a non-public address is refused."""+ monkeypatch.setenv("CLAUDE_SEO_LOCAL_TARGETS", "[::1]:3000")+ servers = {("::1", 3000): _redirect(location)}+ for ip in ("::1", "127.0.0.1", "::ffff:127.0.0.1", "::ffff:7f00:1"):+ servers[(ip, 6379)] = _ok("internal service")+ net = _FakeNet({"localhost": ["::1", "127.0.0.1"]}, servers)++ with pytest.raises(+ requests.exceptions.ConnectionError, match="non-public IP|validated port"+ ):+ _get("http://[::1]:3000/", net)++ assert net.connections == [(socket.AF_INET6, ("::1", 3000, 0, 0))]+++ "location", ["http://[::1]:3000/", "http://[fd00::5]/", "http://[fec0::1]/", "http://internal.example/"]+)+def test_public_ipv6_host_redirecting_to_a_local_ipv6_target_is_refused(location) -> None:+ net = _FakeNet(+ {"v6only.example": [PUBLIC_V6], "internal.example": [PUBLIC_V4, "fd00::5"]},+ {+ (PUBLIC_V6, 80): _redirect(location),+ ("::1", 3000): _ok("dev server"),+ ("fd00::5", 80): _ok("internal"),+ ("fec0::1", 80): _ok("internal"),+ (PUBLIC_V4, 80): _ok("internal"),+ },+ )++ with pytest.raises(requests.exceptions.ConnectionError, match="non-public IP"):+ _get("http://v6only.example/", net)++ assert net.connections == [(socket.AF_INET6, (PUBLIC_V6, 80, 0, 0))]