Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

security(ssrf): block the IBM Cloud Classic metadata hostname in every mode (#6892)

LearningCircuit90fe60b11cb0
diff --git a/SECURITY.md b/SECURITY.md
index 08ec310c24..c7a9a5d422 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -225,7 +225,17 @@ The block also catches IPv6-wrapped forms of the IPv4 metadata IPs. When an IPv6
Both `ssrf_validator.is_ip_blocked` and `NotificationURLValidator.validate_service_url` enforce this absolutely, including under `allow_private_ips=True`. The latter flag is an operator opt-in for self-hosted webhooks on internal networks (RFC1918, CGNAT, loopback, IPv6 ULA); it does NOT extend to metadata IPs, NAT64-wrapped metadata, or the link-local range. Both validators delegate to the same `is_ip_blocked` helper to keep the absolute-block invariant in lockstep. The two differ on link-local: `NotificationURLValidator` blocks the *entire* link-local range (IPv4 `169.254.0.0/16`, IPv6 `fe80::/10`) for every host-bearing notification destination, `http(s)://` included, even under `allow_private_ips=True` (it passes `block_link_local=True` to `is_ip_blocked`), because cloud metadata can be served across link-local beyond the always-blocked literals -- see "Notification Webhook SSRF" above. Egress-policy host classification, `PRIVATE_ONLY` content fetching (`egress/fetch.py`), search-result full-content fetching (`full_search.py`) and its content-cache check (`search_engine_base.py`), and the request-boundary custom LLM endpoint check (`url_utils.is_safe_custom_llm_endpoint`) also pass `block_link_local=True`. The private search-engine instance-URL checks (SearXNG, Paperless, Elasticsearch) do not: link-local is allowed there under the private-engine opt-in (see `docs/CONFIGURATION.md`). `is_ip_blocked`'s own default is unchanged: link-local remains allowed under `allow_private_ips=True` for callers that do not pass the flag — notably the LLM / embedding provider guard described under "LLM Provider URL Validation" below.
-Future contributors must not remove entries from this set. Adding a new cloud provider's metadata IP is encouraged when a new public-cloud target appears.
+`ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES` is the companion list for metadata endpoints reached by **hostname**. It is the single source of truth (`ssrf_validator.is_metadata_hostname`) consulted by `validate_url` (and so by its callers such as `assert_base_url_safe` and `safe_requests`), the egress policy's `evaluate_url` and host classification, `NotificationURLValidator` (http(s) URLs and the host-bearing plugin schemes such as `json://`, `xml://`, `form://`, `gotify://`, `ntfy://`; token-style schemes such as `discord://`, `slack://` or `tgram://` carry no host to check), and the search-engine URL settings validator. It blocks under every flag and scope. Matching is exact after lower-casing, percent-decoding, stripping trailing dots and, for a non-ASCII name, every ASCII form an IDNA encoder produces for it: the folding `getaddrinfo` applies (stdlib `idna` codec, IDNA 2003), the `idna` package's IDNA 2008 encoding that the urllib3-based clients (`requests`) and `httpx` use, and its UTS #46 mapping as defence in depth. Those HTTP clients do not apply UTS #46: they refuse fullwidth letters and soft hyphens themselves (`httpx` does accept the ideographic full stops), so the folding matters mainly for code that hands the name straight to the resolver. So `API.Service.SoftLayer.com.`, `api。service。softlayer。com` and fullwidth `api.service.softlayer.com` are blocked, a public name that only IDNA 2008 accepts (such as a right-to-left label ending in a digit) is judged by the name the HTTP clients connect to, and a non-ASCII name that none of these encoders can encode is refused (fail closed; such a name may still be accepted by a client with yet another IDNA encoder, and is refused deliberately). A different name such as `api.softlayer.com` is not blocked:
+
+| Hostname | Provider |
+| --- | --- |
+| `metadata.google.internal` | GCP (resolves to `169.254.169.254`, which the IP list already blocks) |
+| `metadata.goog` | GCP (same) |
+| `api.service.softlayer.com` | IBM Cloud classic infrastructure (`SoftLayer_Resource_Metadata`) |
+
+`api.service.softlayer.com` resolves to `10.0.80.88`, an RFC1918 address, so without the name block it passed every check whenever private addresses were allowed. `10.0.80.88` is deliberately **not** in the always-blocked IP list: users may run a legitimate LAN service (such as a local LLM server) at that address. The trade-off is that the IBM protection is a hostname block only. A different DNS name, or the literal `10.0.80.88`, that reaches the same service is still allowed when private addresses are allowed. Use egress firewalling on IBM Cloud classic hosts if that matters. (`161.26.0.6` is IBM's package mirror / NTP address, not a metadata endpoint, and is not listed.)
+
+Future contributors must not remove entries from these sets. Adding a new cloud provider's metadata IP or hostname is encouraged when a new public-cloud target appears.
### IPv6 Transition Prefix Block List
@@ -251,7 +261,7 @@ URL rejection log lines route through `ssrf_validator.redact_url_for_log` to dro
Operator-configured LLM endpoints (`llm.ollama.url`, `llm.lmstudio.url`, `llm.llamacpp.url`, `llm.openai_endpoint.url`) are validated against the same SSRF rules as outbound HTTP via `ssrf_validator.assert_base_url_safe`. Validation runs immediately after `normalize_url` and before the LangChain SDK constructor (`ChatOpenAI` / `ChatOllama`), so a misconfigured or hostile URL fails fast rather than silently routing every inference call at internal services.
-The guard uses `allow_localhost=True, allow_private_ips=True` because the legitimate destinations for these providers are localhost (Ollama, LM Studio, llama.cpp) and RFC1918 (Docker / private network deployments). The `ALWAYS_BLOCKED_METADATA_IPS` set still fires under those flags, so cloud-credential endpoints stay blocked regardless of the operator's settings.
+The guard uses `allow_localhost=True, allow_private_ips=True` because the legitimate destinations for these providers are localhost (Ollama, LM Studio, llama.cpp) and RFC1918 (Docker / private network deployments). The `ALWAYS_BLOCKED_METADATA_IPS` set and the `ALWAYS_BLOCKED_METADATA_HOSTNAMES` name block still fire under those flags, so the listed cloud-credential endpoints stay blocked regardless of the operator's settings. The IBM Cloud classic entry is a name block only (see "Cloud Metadata Endpoint Block List"): another DNS name, or the literal RFC1918 address, that reaches the same service is not blocked.
The guard does **not** pass `block_link_local=True`, so link-local destinations (IPv4 `169.254.0.0/16`, IPv6 `fe80::/10`) are accepted for provider URLs. The reason is Podman's link-local mapping: rootless Podman 5.3+ (pasta) maps `host.containers.internal` to `169.254.1.2`, and under Podman the `host-gateway` value the bundled `docker-compose.yml` uses for `host.docker.internal` resolves to the same address, so a host-side Ollama / LM Studio backend is reached over link-local. Docker's own `host-gateway` and Docker Desktop's `host.docker.internal` are not link-local. Documented metadata literals in `ALWAYS_BLOCKED_METADATA_IPS`, including the link-local ones, stay blocked.
@@ -283,9 +293,10 @@ Operators needing a hard boundary **must** layer OS-level controls: network name
| Prompt-injection-induced URL fetches | The agent `fetch_content` tool now calls `evaluate_url()` and raises `PolicyDeniedError` on denial. Subagents propagate the policy context. |
| PRIVATE_ONLY chain break | `ssrf_validator.policy_aware_validate_url()` lets the user's local lab deployments (Ollama on 127.0.0.1, SearXNG on 192.168.x) actually be reached under `PRIVATE_ONLY` without forcing the operator to set `SSRF_ALLOW_PRIVATE_IPS=1` globally. |
| Cache-hit policy bypass | `SearchCache._get_query_hash` incorporates the active scope; a `scope=BOTH` cache entry isn't returned to a later `scope=PRIVATE_ONLY` query. |
-| NAT64 wrap of cloud metadata | `_classify_host` consults `is_nat64_wrapped_metadata_ip` before `is_private_ip`, so `64:ff9b::169.254.169.254` classifies as public (not as the link-local it superficially appears to be). |
+| NAT64 wrap of cloud metadata | Host classification treats an IPv6 address that carries a metadata IPv4 (NAT64 with any prefix, IPv4-mapped / -compatible / -translated, 6to4, Teredo) as the metadata endpoint it reaches, so `64:ff9b::169.254.169.254` is never local (not the link-local it superficially appears to be) and an engine configured with it is refused; `evaluate_url` refuses it in every scope. |
| DNS race / process-global socket timeout | DNS resolution runs inside a single-shot `ThreadPoolExecutor` with `Future.result(timeout=2.0)` — no `socket.setdefaulttimeout()` mutation of process-global state. The worker is abandoned via `shutdown(wait=False)` so a hung lookup can't block past the timeout. |
-| Cloud-metadata fetch under any scope | `evaluate_url()` rejects cloud-metadata IPs (`169.254.169.254`, ECS, IPv4-mapped forms) regardless of scope. They classify as link-local, so STRICT/PRIVATE_ONLY would otherwise *allow* them — notably via the audit-hook net which calls `evaluate_url` on raw `socket.connect` targets (bypassing the SSRF validator the fetch PEPs run first). |
+| Cloud-metadata fetch under any scope | `evaluate_url()` (the fetch-time check, which the audit-hook net also runs on raw `socket.connect` targets, bypassing the SSRF validator the fetch PEPs run first) refuses a cloud-metadata destination in every scope, before any scope or `llm.allowed_local_hostnames` logic: the addresses in `ALWAYS_BLOCKED_METADATA_IPS` and the hostnames in `ALWAYS_BLOCKED_METADATA_HOSTNAMES` (GCP, IBM Cloud classic). Without that, the IPs (link-local) and the IBM name (resolves to RFC1918) would classify as local and STRICT/PRIVATE_ONLY would *allow* them. It reads the URL host broadly, checking every spelling a resolver or HTTP client may connect to: percent-decoded once, not decoded, and decoded until stable; trailing dots removed; zone IDs ignored; the IDNA-folded forms of a non-ASCII host; the dotted quad of an octal / hex / integer / short IPv4 notation. It also refuses an IPv6 address that carries a metadata IPv4 where IPv4-mapped, -compatible or -translated, NAT64 (well-known or network-specific prefix), 6to4 or Teredo addressing puts it, and refuses a host with whitespace or control characters as malformed. These checks read the host text and do no DNS lookup: a different DNS name that merely resolves to a metadata address is classified by its DNS answer (never local) and refused by the SSRF validator on the connection path. |
+| Odd host spellings in host classification | Host classification (the require-local LLM / embeddings checks, the Elasticsearch / Paperless endpoint check, the `llm.allowed_local_hostnames` override and that setting's save-time validator) reads every host through one function, `policy._canonical_host`: a host containing both `:` and `%` as written is unclassified before any decoding; any other host is percent-decoded exactly once, case folded, one trailing root dot dropped. **Only plain DNS names and canonical IP literals are classified; everything else is unclassified.** A plain DNS name is ASCII letters, digits, hyphens and underscores; a canonical IP literal is one `ipaddress` parses as written (dotted-quad IPv4, or IPv6, bracketed or not, without a zone ID). Integer / hex / octal / short IPv4 notations, non-ASCII or IDN spellings, an IPv6 zone ID in any spelling (checked before decoding, so `fd00:ec2::254%31` is not read as the different address `fd00:ec2::2541`), a `%` left after the one decode (double encoding) and whitespace or control characters are unclassified, and so is every cloud-metadata name or address, including an IPv6 wrap of a metadata IPv4. An unclassified host is never local and never makes an engine eligible as public: an Elasticsearch or Paperless endpoint on one is refused under every protected scope, and an `llm.allowed_local_hostnames` entry of that kind is ignored at run time and rejected at save (as is a new entry containing `%`). Entries are compared in canonical form, so one written with capitals or a trailing dot now takes effect. **Behaviour change:** an endpoint in an unusual but working spelling is no longer classified — `search.engine.web.elasticsearch.default_params.hosts`, `search.engine.web.paperless.default_params.api_url`, the `llm.<provider>.url` checked under `llm.require_local_endpoint`, `embeddings.ollama.url` / `embeddings.openai.base_url` checked under `embeddings.require_local`, and `llm.allowed_local_hostnames` entries. Write the host as a plain DNS name (an internationalised name in its `xn--` form) or a standard IP address; those classify as before. `evaluate_url` keeps its broader reading for fetch decisions (see the row above): a URL host without a canonical spelling is classified there by its decoded text as before, but never matched against an `llm.allowed_local_hostnames` entry. |
| Private collection data → cloud model | Each collection carries a public/private flag (default **private**). A private collection is excluded under PUBLIC_ONLY (including Adaptive with a public-engine primary; a public, locally stored collection as primary resolves Adaptive to the internal BOTH scope, where private collections stay eligible) and forces local LLM/embeddings inference under PRIVATE_ONLY / Adaptive-private. The **Adaptive** scope (default) derives the effective scope from the primary engine, so a private-collection primary resolves the run to PRIVATE_ONLY and forces local inference automatically (best effort: locality is judged by endpoint address, see the caveat below). |
#### Caveats
diff --git a/changelog.d/6869.security.md b/changelog.d/6869.security.md
new file mode 100644
index 0000000000..a79eb66808
--- /dev/null
+++ b/changelog.d/6869.security.md
@@ -0,0 +1,5 @@
+Cloud-metadata hostnames are now blocked from one shared list (`ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES`), under every flag and egress scope. The block applies in `validate_url` (and so in its callers such as `assert_base_url_safe` and `safe_requests`), in the egress policy's `evaluate_url` and host classification, in the notification URL validator (http(s) URLs and host-bearing plugin schemes such as `json://` or `ntfy://`), and in the search-engine URL settings validator. The list adds IBM Cloud classic's metadata endpoint `api.service.softlayer.com` to the existing GCP names. Letter case, trailing dots, percent-encoding and IDNA look-alike spellings do not bypass it: ideographic or fullwidth full stops and fullwidth letters fold to the listed name the way the system resolver (`getaddrinfo`) folds them, a non-ASCII name is also checked in the IDNA 2008 form the HTTP clients connect to, and a non-ASCII name that no IDNA encoding (IDNA 2003, IDNA 2008 or UTS #46) accepts is refused. Previously the IBM name resolved to a private address (`10.0.80.88`) and passed whenever private addresses were allowed. Limit: this is a hostname block only. `10.0.80.88` itself is deliberately not always-blocked, because users may run their own LAN services (such as a local LLM server) at that address. A different DNS name, or the literal address, that reaches the IBM service is still allowed when private addresses are allowed.
+
+Egress host classification now accepts only plain DNS names and canonical IP literals; every other host spelling is unclassified. This applies to the require-local LLM and embeddings checks, the Elasticsearch and Paperless endpoint checks, `llm.allowed_local_hostnames` and its save-time validator, which all read a host through one function. A host that contains both `:` and `%` as written (an IPv6 address with a zone ID, plain or percent-encoded) is unclassified before any decoding, so an encoded zone ID is never read as part of the address; any other host is percent-decoded exactly once, case folded, one trailing dot dropped, and then accepted only as a name of ASCII letters, digits, hyphens and underscores, or as an IP address written the standard way (dotted-quad IPv4, or IPv6 without a zone ID). An unclassified host, and every cloud-metadata name or address (IPv6 forms that wrap a metadata IPv4 included), is never local and never makes an engine eligible as public: an Elasticsearch or Paperless search engine configured with one is refused under every protected egress scope instead of being treated as a public engine, and such an `llm.allowed_local_hostnames` entry is ignored and rejected at save. Previously the policy recognised metadata spellings one at a time, and some unusual spellings slipped through. Behaviour change: endpoints written in an unusual spelling that used to work, such as an integer or hex IPv4 address, an internationalised (non-ASCII) host name or an IPv6 address with a zone ID, are now refused by these checks. This affects `search.engine.web.elasticsearch.default_params.hosts`, `search.engine.web.paperless.default_params.api_url`, the provider URL checked under `llm.require_local_endpoint` (`llm.<provider>.url`), `embeddings.ollama.url` and `embeddings.openai.base_url` under `embeddings.require_local`, and `llm.allowed_local_hostnames`. Write the host as a plain DNS name (an internationalised name in its `xn--` form) or a standard IP address; those classify as before. `llm.allowed_local_hostnames` entries are now compared case-insensitively and ignoring a trailing dot, so an entry written with capitals now takes effect; a new or changed entry containing `%` is rejected at save, and an unchanged list is no longer re-validated when other settings are saved. The fetch-time URL check keeps its broader handling and now also refuses a metadata address carried by an IPv6 address in any NAT64, 6to4 or Teredo form, and a host containing control characters.
+
+The Elasticsearch engine now SSRF-validates every configured host before it builds its client, and its availability probe connects only to a host that passed the same check. Each entry is judged as the scheme, host and port the client connects to: a URL string is parsed with the client's own parser (urllib3) and also validated as written; a host dictionary must give an `http` or `https` scheme and, if it gives a port, an integer from 1 to 65535, and its `host` value is validated as given. The host must be a plain host name or IP address without `%` (so no percent-encoding or IPv6 zone ID), must resolve, and must not be a cloud-metadata name or address; localhost and private addresses, link-local included, stay allowed for this self-hosted engine. A configuration in which any host fails is refused as a whole (fail-closed), including a host that cannot be resolved or `0.0.0.0`, and the engine is dropped with a logged error.
diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md
index 5561a3fc82..143f470b3e 100644
--- a/docs/CONFIGURATION.md
+++ b/docs/CONFIGURATION.md
@@ -38,9 +38,9 @@ They are not available in the Web UI because they are needed to start the applic
| `LDR_POLICY_ALLOW_UNPROTECTED_EGRESS` | Boolean | `False` | No | | Allow users to select the UNPROTECTED egress scope. Disabled by default; this is an environment-only operator gate and cannot be changed through the user-writable settings API. Hard SSRF and cloud-metadata protections remain active when enabled. | Security | |
| `LDR_RESEARCH_LIBRARY_ALLOW_FILESYSTEM_PDF_STORAGE` | Boolean | `False` | No | | Allow users to select the UNENCRYPTED 'filesystem' PDF storage mode for library-downloaded PDFs. Disabled by default; this is an environment-only operator gate and cannot be changed through the user-writable settings API. Filesystem mode writes fetched third-party PDFs as PLAINTEXT to a shared library directory (cleartext storage of sensitive information, CWE-312). When off, the 'filesystem' option is withheld from the settings UI and any stored or environment value of 'filesystem' is coerced to the encrypted 'database' mode at write time. Previously-written plaintext files remain readable. Enable only when the library directory lives on an operator-controlled encrypted volume. | Security | |
| `LDR_RESEARCH_LIBRARY_ALLOW_SHARED_LIBRARY` | Boolean | `False` | No | | Allow shared-library mode, which drops the per-user library directory boundary so all users' downloaded PDFs live in one shared directory. Disabled by default; this is an environment-only operator gate and cannot be enabled through the user-writable settings API. The shared_library flag is otherwise user-editable; leaving shared mode user-toggleable would let an account drop its per-user directory boundary. The storage_path is now operator-only, but legacy saved paths remain in place. When off, the per-user subdirectory is always enforced regardless of the user's shared_library setting. Enable only on single-tenant or mutually-trusted deployments. | Security | |
-| `LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS` | Boolean | `False` | No | | Allow user-editable PUBLIC search-engine URLs (e.g. the SearXNG instance_url) to point at private / loopback / link-local addresses. Environment-only operator gate so it cannot be flipped through the user-writable settings API. Disabled by default: a public-nature engine (its data source is the public internet) pointed at an internal host lets any authenticated user turn a research run into an internal port scan / service probe, breaking the PUBLIC_ONLY egress promise. Enable only when you self-host SearXNG on localhost/LAN and accept that engine fetches may reach your private network. To approve only specific origins instead of all private addresses, use the finer-grained LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST. Docker deployments instead pin the URL directly via LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL, which is trusted as operator-provisioned. Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS) stay blocked regardless of this flag. | Security | |
-| `LDR_SEARCH_ALLOW_PRIVATE_RESULT_FETCH` | Boolean | `False` | No | | Allow full-content fetching (search.snippets_only off) to follow the RESULT URLs a PUBLIC search engine returns into private / loopback addresses, e.g. an internal wiki that a self-hosted SearXNG instance indexes. Distinct from, and required in addition to, the instance-URL approval (LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS, LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST or an env-locked instance URL): that approval only covers reaching the engine itself, and because a public engine proxies the public web, extending it to the URLs the engine returns would let a poisoned index turn result fetching into a scan of the LDR host's private network. Environment-only operator gate so it cannot be flipped through the user-writable settings API. Disabled by default and not set by the bundled docker-compose.yml. Link-local addresses (169.254.0.0/16, fe80::/10) and the cloud-metadata endpoints in ALWAYS_BLOCKED_METADATA_IPS stay blocked regardless of this flag. Takes effect only under the unprotected egress scope: under every other scope, the default adaptive scope included, private result URLs stay blocked even with this flag set. | Security | |
-| `LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST` | String | `None` | No | | Comma-separated list of exact URL origins (scheme://host[:port]) that user-editable PUBLIC search-engine URLs (e.g. the SearXNG instance_url) may use even though they are private / loopback / link-local — e.g. 'http://localhost:8080,http://192.168.1.5:8888'. Finer-grained alternative to LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true: private egress is enabled for an engine only when its configured URL matches a listed origin (the engine's request chain, including redirects, then runs with private access — like the blanket flag, but conditional on the match), rather than for any private URL. Matching is exact scheme + host + port (default ports 80/443 implied, host case-insensitive, path/query ignored); no wildcards or CIDR ranges. Entries that fail to parse are ignored. Environment-only operator setting so it cannot be edited through the user-writable settings API. Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS) and non-http(s) schemes stay blocked even if listed. | Security | |
+| `LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS` | Boolean | `False` | No | | Allow user-editable PUBLIC search-engine URLs (e.g. the SearXNG instance_url) to point at private / loopback / link-local addresses. Environment-only operator gate so it cannot be flipped through the user-writable settings API. Disabled by default: a public-nature engine (its data source is the public internet) pointed at an internal host lets any authenticated user turn a research run into an internal port scan / service probe, breaking the PUBLIC_ONLY egress promise. Enable only when you self-host SearXNG on localhost/LAN and accept that engine fetches may reach your private network. To approve only specific origins instead of all private addresses, use the finer-grained LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST. Docker deployments instead pin the URL directly via LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL, which is trusted as operator-provisioned. Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS and ALWAYS_BLOCKED_METADATA_HOSTNAMES) stay blocked regardless of this flag. | Security | |
+| `LDR_SEARCH_ALLOW_PRIVATE_RESULT_FETCH` | Boolean | `False` | No | | Allow full-content fetching (search.snippets_only off) to follow the RESULT URLs a PUBLIC search engine returns into private / loopback addresses, e.g. an internal wiki that a self-hosted SearXNG instance indexes. Distinct from, and required in addition to, the instance-URL approval (LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS, LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST or an env-locked instance URL): that approval only covers reaching the engine itself, and because a public engine proxies the public web, extending it to the URLs the engine returns would let a poisoned index turn result fetching into a scan of the LDR host's private network. Environment-only operator gate so it cannot be flipped through the user-writable settings API. Disabled by default and not set by the bundled docker-compose.yml. Link-local addresses (169.254.0.0/16, fe80::/10) and the cloud-metadata endpoints in ALWAYS_BLOCKED_METADATA_IPS and ALWAYS_BLOCKED_METADATA_HOSTNAMES stay blocked regardless of this flag. Takes effect only under the unprotected egress scope: under every other scope, the default adaptive scope included, private result URLs stay blocked even with this flag set. | Security | |
+| `LDR_SEARCH_PRIVATE_ENGINE_URL_ALLOWLIST` | String | `None` | No | | Comma-separated list of exact URL origins (scheme://host[:port]) that user-editable PUBLIC search-engine URLs (e.g. the SearXNG instance_url) may use even though they are private / loopback / link-local — e.g. 'http://localhost:8080,http://192.168.1.5:8888'. Finer-grained alternative to LDR_SEARCH_ALLOW_PRIVATE_ENGINE_URLS=true: private egress is enabled for an engine only when its configured URL matches a listed origin (the engine's request chain, including redirects, then runs with private access — like the blanket flag, but conditional on the match), rather than for any private URL. Matching is exact scheme + host + port (default ports 80/443 implied, host case-insensitive, path/query ignored); no wildcards or CIDR ranges. Entries that fail to parse are ignored. Environment-only operator setting so it cannot be edited through the user-writable settings API. Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS and ALWAYS_BLOCKED_METADATA_HOSTNAMES) and non-http(s) schemes stay blocked even if listed. | Security | |
| `LDR_SECURITY_ALLOW_NAT64` | Boolean | `False` | No | | Allow outbound traffic to NAT64 prefixes (64:ff9b::/96 RFC 6052 well-known and 64:ff9b:1::/48 RFC 8215 local-use). Disabled by default to close the IPv6-wrapped SSRF bypass class — on hosts configured with NAT64 routes, attacker-supplied URLs can wrap cloud-metadata or RFC1918 destinations through these prefixes. Enable only on IPv6-only deployments (DNS64+NAT64) where outbound IPv4 traffic is synthesized through this prefix and the operator has accepted the residual SSRF risk. 6to4 (2002::/16), Teredo (2001::/32), the discard prefix (100::/64), the deprecated IPv4-Compatible IPv6 form (::/96, except ::1 loopback which follows the loopback/private flags), and the IPv4-Translated/SIIT form (::ffff:0:0:0/96) remain unconditionally blocked because they have no live legitimate use in 2026. The cloud-metadata block (ALWAYS_BLOCKED_METADATA_IPS) still applies via embedded-IPv4 extraction — see SECURITY.md. | Security | |
| `LDR_SECURITY_CORS_ALLOWED_ORIGINS` | String | `None` | No | | Allowed CORS origins for API routes (comma-separated). Use '*' for all origins, empty for same-origin only. Example: 'https://example.com,https://app.example.com' | Security | |
| `LDR_SECURITY_WEBSOCKET_ALLOWED_ORIGINS` | String | `None` | No | | Allowed origins for WebSocket/Socket.IO connections (comma-separated). Unset or empty means same-origin only (default); use '*' to allow all origins. Example: 'https://example.com,https://app.example.com' | Security | |
diff --git a/docs/NOTIFICATIONS.md b/docs/NOTIFICATIONS.md
index d3f8854ec1..192b57583b 100644
--- a/docs/NOTIFICATIONS.md
+++ b/docs/NOTIFICATIONS.md
@@ -303,7 +303,7 @@ The "Test" button in the notifications settings page returns the validator's rea
**"Blocked cloud-metadata / link-local IP address: \<host\>"**
- **Cause**: A host-bearing notification URL names a cloud-metadata or link-local endpoint. Literal addresses are checked directly; hostnames get one bounded validation-time resolution and are checked again at send time. Fixed-destination token/topic modes (Discord, Slack, ntfy cloud, Matrix t2bot and Telegram) are not resolved as user-supplied hosts.
-- **Always blocked**: AWS IMDS / ECS / EKS Pod Identity, Azure IMDS and WireServer, GCP, OCI, Scaleway, Akamai / Linode, OpenStack / Hetzner, DigitalOcean, AlibabaCloud, Tencent — as plain addresses (IPv4 and IPv6) and, for the IPv4 ones, wrapped through any NAT64 prefix. No env var re-opens these. See [SECURITY.md](../SECURITY.md#cloud-metadata-endpoint-block-list). The rest of the link-local range is blocked for every notification URL as well; `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS` does not reopen it.
+- **Always blocked**: AWS IMDS / ECS / EKS Pod Identity, Azure IMDS and WireServer, GCP, OCI, Scaleway, Akamai / Linode, OpenStack / Hetzner, DigitalOcean, AlibabaCloud, Tencent, Oracle Compute Classic — as plain addresses (IPv4 and IPv6) and, for the IPv4 ones, wrapped through any NAT64 prefix. The GCP and IBM Cloud classic metadata hostnames are blocked by name as well (reported as `"Blocked cloud-metadata hostname: <host>"`). No env var re-opens these. See [SECURITY.md](../SECURITY.md#cloud-metadata-endpoint-block-list). The rest of the link-local range is blocked for every notification URL as well; `LDR_NOTIFICATIONS_ALLOW_PRIVATE_IPS` does not reopen it.
- **Fix**: Choose a different webhook destination. Metadata endpoints expose IAM/instance credentials and are never legitimate webhook targets.
**"Blocked unsafe protocol: \<scheme\>"** / **"Unsupported protocol: \<scheme\>"**
diff --git a/docs/SearXNG-Setup.md b/docs/SearXNG-Setup.md
index 0d16ff5257..3d9490a9b5 100644
--- a/docs/SearXNG-Setup.md
+++ b/docs/SearXNG-Setup.md
@@ -206,9 +206,12 @@ Regardless of which option is used:
- **Cloud metadata endpoints** (AWS IMDS / ECS / EKS Pod Identity, Azure IMDS
and WireServer, GCP, OCI, Scaleway, Akamai / Linode, OpenStack / Hetzner,
- DigitalOcean, AlibabaCloud, Tencent Cloud — see
- `ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS`) are always blocked to prevent
- credential theft in cloud environments
+ DigitalOcean, AlibabaCloud, Tencent Cloud, Oracle Compute Classic — see
+ `ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS` — plus the GCP and IBM Cloud
+ classic metadata hostnames in
+ `ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES`) are always blocked to
+ prevent credential theft in cloud environments. The IBM entry is a hostname
+ block only: another name for the same private address is not covered
- Only `http`/`https` URLs are accepted — other schemes are always refused
- The approval covers the **instance URL only**. It does not let full-content
fetching follow the *result* URLs SearXNG returns into your private network;
diff --git a/src/local_deep_research/security/egress/README.md b/src/local_deep_research/security/egress/README.md
index 63efc3d146..1163d3d270 100644
--- a/src/local_deep_research/security/egress/README.md
+++ b/src/local_deep_research/security/egress/README.md
@@ -195,7 +195,7 @@ The UI orchestrator that calls `warnings.py` lives at `web/warning_checks/__init
| `policy.egress_scope` | `adaptive` | The scope (table above). |
| `llm.require_local_endpoint` | `false` | Refuse non-local LLM endpoints. |
| `embeddings.require_local` | `false` | Refuse cloud embeddings / uncached HF download. |
-| `llm.allowed_local_hostnames` | `[]` | Hostnames treated as local by the classifier (public hosts rejected at save time). |
+| `llm.allowed_local_hostnames` | `[]` | Hostnames treated as local by the classifier, compared in canonical form (case and a trailing dot do not matter; a stored entry is read percent-decoded once, like a URL host). A changed list is validated at save time: entries containing `%`, entries that are not a plain DNS name or canonical IP address, cloud-metadata endpoints, and hosts that resolve to public addresses are rejected; an unchanged list is not re-validated. At run time, entries that are not a canonical spelling and cloud-metadata endpoints are ignored; a public host already stored is still honoured. |
| `collections.is_public` (per collection) | `false` (private) | Collection egress classification. |
| `app.warnings.dismiss_egress_policy` / `…dismiss_cloud_llm` / `…dismiss_cloud_embeddings` | `false` | Per-banner dismiss flags (separate so dismissing one never hides the critical others). |
@@ -334,7 +334,7 @@ Each cell is the decision produced by the real `evaluate_*` logic / PEP for that
| **Library download** (`evaluate_url`, download_service) | private→private-host only; public→public-host only | any classified host ALLOWED | public host ALLOWED; private BLOCKED | private host ALLOWED; **public BLOCKED** | private host ALLOWED; public BLOCKED |
| **Notification webhook** (`evaluate_url`, notifications/manager) | private→private-host only; public→public-host only | any classified http/https host ALLOWED | public webhook ALLOWED; private BLOCKED | private webhook ALLOWED; **public webhook BLOCKED** | private webhook ALLOWED; public BLOCKED |
| **Raw socket** (audit hook → `evaluate_url`) | private→ARMED (public BLOCKED, best effort); public→INACTIVE | INACTIVE (passthrough) | INACTIVE (passthrough) | **ARMED** — public-address socket.connect BLOCKED on threads carrying the run's context (best effort, see below) | ARMED — public-address socket.connect BLOCKED on threads carrying the run's context (best effort, see below) |
-| **Cloud-metadata IP** (169.254.169.254 / NAT64) | BLOCKED | BLOCKED | BLOCKED | BLOCKED | BLOCKED |
+| **Cloud-metadata IP / hostname** (169.254.169.254 / NAT64 / `ALWAYS_BLOCKED_METADATA_HOSTNAMES`) | BLOCKED | BLOCKED | BLOCKED | BLOCKED | BLOCKED |
The URL-fetch rows show the `evaluate_url` decision where that check runs. Under `strict`, search-result and document fetches normally reach neither public nor private hosts. Private hosts are normally refused because LDR's fetch-time SSRF checks relax private addresses only under `private_only`. `LDR_SEARCH_ALLOW_PRIVATE_RESULT_FETCH` for an approved public engine such as a self-hosted SearXNG does not change that under `strict` or any other protected scope: the full-content wrapper applies that grant only under `unprotected` (see the Full-content fetch row above; link-local and cloud metadata stay blocked even there). The SSRF checks do not cover every non-public policy range (198.18.0.0/15, the documentation ranges and 240.0.0.0/4 pass them); the full-content wrapper refuses those result URLs under every scope but `unprotected`. The policy now treats 198.18.0.0/15 as unknown because it might be a real benchmark host or a public hostname mapped by fake-IP DNS (Clash, sing-box): protected policy-checked fetch paths refuse it, as does the `strict` / `private_only` socket guard. Other policy-private ranges retain their existing classification (see "Known limitations" in `docs/egress-modes.md`). Public hosts are normally stopped by the policy check or, on fetch paths without one, by the socket guard (best effort).
@@ -346,7 +346,8 @@ The URL-fetch rows show the `evaluate_url` decision where that check runs. Under
| **public_only stays open to public sources** | public engines pass `evaluate_engine` (private collections BLOCKED via `scope_mismatch_public_only`); `evaluate_url` allows public hosts, blocks private; inference is NOT scope-forced (toggles still apply); the audit hook stays INACTIVE (only PRIVATE_ONLY/STRICT arm it) so legitimate local infra traffic — local Ollama, settings DB — is never falsely blocked. |
| **adaptive just matches the primary engine** | `_resolve_adaptive_scope`: private primary → PRIVATE_ONLY, public primary → PUBLIC_ONLY, unknown → BOTH; a public, locally stored collection primary (public *and* local) also → BOTH, where private collections stay eligible. A registered local retriever primary also resolves PRIVATE_ONLY. Uses the engine's **static class flags** (`is_public`/`is_local` on the Python class), not the network location of the engine's server — so a locally-hosted SearXNG (class `is_public=True`) resolves to PUBLIC_ONLY. Resolved once at run start; the stored `EgressContext` carries the concrete scope, not ADAPTIVE. |
| **A private RAG collection as ADAPTIVE primary forces local inference** | `_resolve_collection_is_public` defaults private → `classify_engine` returns local → adaptive resolves PRIVATE_ONLY → forces local LLM + embeddings (best effort: judged by provider and endpoint address, so a local relay to a hosted model is not detected). Mark the collection `is_public` to opt into cloud inference. |
-| **Known cloud-metadata addresses are normally refused by the policy checks under every scope** | `evaluate_url` runs `is_ip_blocked(..., allow_private_ips=True)` before scope logic; `_classify_host` applies the same metadata block on BOTH the literal-IP and DNS-resolved paths; NAT64-wrapped metadata is reclassified PUBLIC. |
+| **Known cloud-metadata addresses are normally refused by the policy checks under every scope** | `evaluate_url` (the fetch-time check, which the audit-hook net also runs on raw `socket.connect` targets) refuses them before any scope or `local_hostnames` logic (`_url_host_refusal`). It reads the URL host broadly: `_policy_host_forms` yields the host percent-decoded once, not decoded, and decoded until stable, with trailing dots and IPv6 brackets removed, plus the IDNA folding of a non-ASCII host and the dotted quad of an octal / hex / integer / short IPv4 notation. Each form is checked with `is_ip_blocked(..., allow_private_ips=True)` (the always-blocked set, IPv4-mapped and zone-scoped forms included, and the other blocked transition ranges) and by `_is_metadata_destination` (`ssrf_validator.is_metadata_hostname` for the GCP and IBM Cloud classic names, and `_is_metadata_ip`, which also finds a metadata IPv4 carried by an IPv6 address in any NAT64 position, well-known or network-specific prefix, or in 6to4 / Teredo form). A host with whitespace or control characters is refused as malformed. These are string checks: a different DNS name that merely resolves to a metadata address is classified by its answer (never local) and refused by the SSRF validator on the connection path. The IBM entry is a name block only: another name for the same RFC1918 address is not covered. |
+| **Host classification accepts only canonical host spellings** | Every classification caller (`_classify_host`, and through it the require-local LLM / embeddings checks; `_classify_engine_url` for the Elasticsearch / Paperless endpoint fail-up and vector-store endpoints; the `local_hostnames` override; `validate_allowed_local_hostnames`) reads a host through one function, `_canonical_host`, passing it as parsed, not decoded: a host containing both `:` and `%` as written (an IPv6 zone ID, plain or encoded) is unclassified before any decoding, so a decode cannot splice an encoded zone into the address; any other host is percent-decoded exactly once, lower-cased, and one trailing root dot is dropped. Only a plain DNS name (ASCII letters, digits, hyphens and underscores) or an IP literal `ipaddress` parses as written (dotted-quad IPv4; IPv6, bracketed or not, without a zone ID) is classified. Everything else is unclassified, and so is every cloud-metadata name or address (`_is_metadata_host`, IPv6 wraps included): `_classify_host` returns False (not local), `_classify_engine_url` treats the entry as unknown so the engine is refused under every protected scope instead of failing up to public, and a `local_hostnames` entry of either kind is ignored (and refused at save). `local_hostnames` entries are compared in canonical form, so case and a trailing dot do not matter. `evaluate_url` keeps its broader reading for the fetch decision: a URL host without a canonical spelling is classified by its decoded text, as before, but is never matched against a `local_hostnames` entry. |
| **An engine self-checks scope at run time, even off the factory path** | `BaseSearchEngine._verify_egress_scope()` runs at the top of `run()` (and inside `CollectionSearchEngine.search()` / `LibraryRAGSearchEngine.search()`, which bypass `run()`), re-evaluating `evaluate_engine` against the engine's stored snapshot and **raising `PolicyDeniedError`** on a mismatch. Memoized per snapshot identity + (scope, primary) values so the hot path pays the evaluation once. A defense-in-depth backstop behind the factory PEP for engines built by direct instantiation; it cannot deny anything the factory would have allowed. |
| **Direct MCP searches still arm the socket-level net** | `mcp/server.py::_egress_audit_net()` builds the run's `EgressContext` from the request snapshot and arms the PEP-578 audit hook around `engine.run()`, so a direct MCP search (which never goes through `AdvancedSearchSystem`) still gets the raw-socket backstop under PRIVATE_ONLY/STRICT. Fails open to a no-op when the policy is unevaluable — the factory PEP remains primary. |
diff --git a/src/local_deep_research/security/egress/policy.py b/src/local_deep_research/security/egress/policy.py
index cd6cb25d74..47ae1f8a67 100644
--- a/src/local_deep_research/security/egress/policy.py
+++ b/src/local_deep_research/security/egress/policy.py
@@ -19,6 +19,7 @@ from __future__ import annotations
import hashlib
import ipaddress
import json
+import re
import socket
import threading
from collections import OrderedDict
@@ -267,6 +268,223 @@ def _is_nat64_wrapped_metadata(hostname: str) -> bool:
return False
+# A DNS label as host classification accepts it: ASCII letters, digits,
+# hyphens and underscores, 1-63 characters, not starting or ending with a
+# hyphen. Underscores are outside LDH but appear in container and compose
+# service names (the bundled docker-compose.yml names its Ollama container
+# ``ollama_service``); no resolver or HTTP client reads one as anything but
+# a literal character.
+_DNS_LABEL_RE = re.compile(r"[a-z0-9_](?:[a-z0-9_-]{0,61}[a-z0-9_])?")
+
+# A final label ``inet_aton`` reads as a number (decimal, octal or 0x-hex).
+# A name ending in one is an IPv4 address in a shorthand notation, not a DNS
+# name; real top-level domains are never numeric.
+_NUMERIC_LABEL_RE = re.compile(r"[0-9]+|0x[0-9a-f]*")
+
+
+def _canonical_host(hostname: object) -> Optional[str]:
+ """The canonical spelling of ``hostname`` for host classification, or
+ None when it has none.
+
+ The single host canonicaliser for every classification caller
+ (``_classify_host`` and through it the require-local LLM / embeddings
+ checks and the engine URL classification, the
+ ``llm.allowed_local_hostnames`` override, and the settings validator for
+ that list). Callers pass the host as parsed or configured, NOT
+ percent-decoded: it is decoded here exactly once, as the HTTP clients
+ do before connecting, and lower-cased (``str.lower``, as ``urlsplit``
+ does for a URL host). A single trailing root dot is then dropped, except
+ from a bracketed IPv6 literal.
+
+ Before any decoding, a host that contains both ``:`` and ``%`` as
+ written (an IPv6 literal with a zone ID, bracketed or not) returns
+ None. ``%`` is both the zone delimiter and the escape character, so
+ such a host has two readings: an HTTP client takes what follows the
+ ``%`` as the zone, while one decode can splice an encoded zone into the
+ address (``fd00:ec2::254%31`` would decode to the different address
+ ``fd00:ec2::2541``, and ``fd12::1%3a1`` to ``fd12::1:1``).
+
+ The result is accepted only if it is ASCII and either
+
+ * an IP literal ``ipaddress.ip_address`` parses as written: an IPv4
+ address only in dotted-quad decimal, an IPv6 address bracketed or
+ not but without a trailing dot, returned in its standard (compressed)
+ form; or
+ * a plain DNS name: dot-separated labels of ASCII letters, digits,
+ hyphens and underscores (``_DNS_LABEL_RE``), at most 253 characters,
+ whose last label is not numeric, so that ``inet_aton`` cannot read it
+ as an IPv4 address.
+
+ Everything else returns None: an IPv6 zone ID in any spelling (above), a
+ ``%`` left after the one decode (a second level of percent-encoding), any
+ non-ASCII character (IDN, fullwidth or ideographic spellings that need
+ IDNA folding; checked on the decoded text before case folding, so a
+ character ``str.lower`` turns into ASCII, the KELVIN SIGN, is refused
+ too), whitespace or control characters, empty labels, and the
+ integer / hex / octal / short IPv4 notations. Classification treats such
+ a host as unclassified instead of guessing which destination a resolver
+ or client would make of it. Pure string logic: no DNS, never raises.
+ """
+ if not isinstance(hostname, str) or not hostname:
+ return None
+ # An IPv6-style host carrying a zone delimiter, checked as written: no
+ # decoding may turn an encoded zone into part of the address. Brackets
+ # contain neither character, so this holds bracketed or not.
+ if ":" in hostname and "%" in hostname:
+ return None
+ decoded = unquote(hostname)
+ # ASCII-only is checked BEFORE case folding: str.lower maps the KELVIN
+ # SIGN (U+212A) to an ASCII "k", so a host spelt with it would pass as
+ # a plain DNS name.
+ if not decoded.isascii() or "%" in decoded:
+ return None
+ host = decoded.lower()
+ if host.startswith("[") and host.endswith("]"):
+ try:
+ address = ipaddress.ip_address(host[1:-1])
+ except ValueError:
+ return None
+ return str(address) if address.version == 6 else None
+ trailing_dot = host.endswith(".")
+ if trailing_dot:
+ host = host[:-1]
+ if not host:
+ return None
+ try:
+ address = ipaddress.ip_address(host)
+ except ValueError:
+ address = None
+ if address is not None:
+ if address.version == 6:
+ # A root dot means nothing after an IPv6 literal; the resolver
+ # would look the text up as a name.
+ return None if trailing_dot else str(address)
+ # str() equality refuses a zero-padded IPv4 octet on any Python
+ # version.
+ return host if str(address) == host else None
+ if len(host) > 253:
+ return None
+ labels = host.split(".")
+ if not all(_DNS_LABEL_RE.fullmatch(label) for label in labels):
+ return None
+ if _NUMERIC_LABEL_RE.fullmatch(labels[-1]):
+ return None
+ try:
+ socket.inet_aton(host)
+ except (OSError, ValueError, TypeError, UnicodeError):
+ return host
+ # The platform resolver reads it as an IPv4 address after all.
+ return None
+
+
+def _is_metadata_ip(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool:
+ """True iff ``ip`` is a cloud-metadata address
+ (``ALWAYS_BLOCKED_METADATA_IPS``), ignoring an IPv6 zone ID, or an IPv6
+ address that carries one where an IPv4-embedding mechanism puts the
+ IPv4 address it reaches: the RFC 6052 positions for every NAT64 prefix
+ length, well-known or network-specific (the /96 position is also where
+ IPv4-mapped, IPv4-compatible and IPv4-translated addresses carry it),
+ 6to4 and both Teredo addresses. Pure: no DNS, never raises.
+ """
+ from ..ssrf_validator import (
+ _NAT64_V4_BYTE_INDICES,
+ ALWAYS_BLOCKED_METADATA_IPS,
+ )
+
+ if isinstance(ip, ipaddress.IPv6Address) and ip.scope_id:
+ ip = ipaddress.IPv6Address(ip.packed)
+ if str(ip) in ALWAYS_BLOCKED_METADATA_IPS:
+ return True
+ if not isinstance(ip, ipaddress.IPv6Address):
+ return False
+ packed = ip.packed
+ embedded = [
+ ipaddress.IPv4Address(bytes(packed[i] for i in positions))
+ for positions in _NAT64_V4_BYTE_INDICES.values()
+ ]
+ if ip.sixtofour is not None:
+ embedded.append(ip.sixtofour)
+ if ip.teredo is not None:
+ embedded.extend(ip.teredo)
+ return any(str(v4) in ALWAYS_BLOCKED_METADATA_IPS for v4 in embedded)
+
+
+def _is_metadata_host(host: str) -> bool:
+ """True iff the CANONICAL host ``host`` (``_canonical_host`` output)
+ names a cloud-metadata endpoint: a listed metadata hostname
+ (``ALWAYS_BLOCKED_METADATA_HOSTNAMES``) or an address ``_is_metadata_ip``
+ recognises. No normalisation happens here; pass a canonical host."""
+ from ..ssrf_validator import ALWAYS_BLOCKED_METADATA_HOSTNAMES
+
+ if host in ALWAYS_BLOCKED_METADATA_HOSTNAMES:
+ return True
+ try:
+ address = ipaddress.ip_address(host)
+ except ValueError:
+ return False
+ return _is_metadata_ip(address)
+
+
+def _classifiable_host(hostname: object) -> Optional[str]:
+ """``_canonical_host(hostname)``, or None when the host has no canonical
+ spelling or names a cloud-metadata endpoint. A None here means
+ "unclassified": never local, and never eligible as a public endpoint
+ either. Takes the host as parsed or configured, not percent-decoded."""
+ host = _canonical_host(hostname)
+ if host is None or _is_metadata_host(host):
+ return None
+ return host
+
+
+def _declared_local_hosts(ctx: EgressContext) -> frozenset[str]:
+ """The ``llm.allowed_local_hostnames`` entries classification honours,
+ in canonical form. An entry without a canonical spelling, or one that
+ names a cloud-metadata endpoint, is ignored (``_classifiable_host``)."""
+ return frozenset(
+ host
+ for host in map(_classifiable_host, ctx.local_hostnames)
+ if host is not None
+ )
+
+
+def _is_metadata_destination(hostname: str) -> bool:
+ """True iff ``evaluate_url`` should refuse ``hostname`` as a
+ cloud-metadata destination: the fetch-time check, deliberately broader
+ than classification.
+
+ ``hostname`` is the host as ``urlsplit`` returns it (not decoded). It is
+ expanded by ``_policy_host_forms`` into every spelling the resolver may
+ connect to (percent-decoded once, not decoded, and decoded until stable;
+ trailing dots removed; IDNA-folded forms of a non-ASCII host; the dotted
+ quad of an octal / hex / integer / short IPv4 notation), and refused
+ when any form is a listed metadata hostname (``is_metadata_hostname``,
+ checked on every form, which also fails closed on a non-ASCII name no
+ IDNA encoder can encode) or an address ``_is_metadata_ip`` recognises,
+ zone-scoped IPv6 included. String checks only, no DNS: a
+ name that merely resolves to a metadata address is left to the SSRF
+ validator at the connection point.
+
+ Classification does not use this. It accepts only canonical hosts
+ (``_canonical_host``), so a spelling recognised only here is
+ unclassified there anyway, and it checks the canonical form with
+ ``_is_metadata_host``.
+ """
+ from ..ssrf_validator import is_metadata_hostname
+
+ if not isinstance(hostname, str) or not hostname:
+ return False
+ for candidate in _policy_host_forms(hostname):
+ if is_metadata_hostname(candidate):
+ return True
+ try:
+ address = ipaddress.ip_address(candidate)
+ except ValueError:
+ continue
+ if _is_metadata_ip(address):
+ return True
+ return False
+
+
def _resolve_with_timeout(hostname: str) -> Optional[list]:
"""Resolve ``hostname`` via getaddrinfo with a bounded timeout.
@@ -369,10 +587,48 @@ def _cache_classification(
def _classify_host(
hostname: str, ctx: EgressContext, allow_dns: bool = True
) -> Optional[bool]:
- """Classify a hostname as local (True) / public (False) / unknown (None).
+ """Classify a host as local (True) / public (False) / unknown (None).
+
+ ``hostname`` is the host as parsed from the URL or configured value,
+ NOT percent-decoded: ``_canonical_host`` decodes it exactly once. Only a
+ plain DNS name or a canonical IP literal is classified. Any other
+ spelling (residual ``%``, IPv6 zone ID, non-ASCII, control characters,
+ integer / hex / octal / short IPv4) and any cloud-metadata name or
+ address, in canonical spelling or wrapped in an IPv6 form, returns
+ False: "not local", which every require-local caller relies on. It does
+ NOT mean "public, therefore eligible": a caller that reads False that way
+ (the engine URL fail-up in ``_classify_engine_url``) checks
+ ``_classifiable_host`` first and treats None from it as unclassified.
+
+ A canonical host is local when it is listed in
+ ``llm.allowed_local_hostnames`` (compared in canonical form, so case and
+ a trailing dot do not matter; an entry without a canonical spelling is
+ ignored), else it is classified by ``_classify_destination``: by its
+ literal address, or by every address DNS returns for it. Falls back to
+ public on DNS timeout (fail-safe).
+ """
+ if not hostname:
+ return None
+ host = _classifiable_host(hostname)
+ if host is None:
+ return False
+ if host in _declared_local_hosts(ctx):
+ return True
+ return _classify_destination(host, ctx, allow_dns=allow_dns)
+
+
+def _classify_destination(
+ hostname: str, ctx: EgressContext, allow_dns: bool = True
+) -> Optional[bool]:
+ """Classify ``hostname`` by its literal address or its DNS answers:
+ local (True) / public (False) / unknown (None).
+
+ No normalisation and no ``local_hostnames`` override: ``_classify_host``
+ calls this with a canonical host after both, and ``evaluate_url`` calls
+ it for a host without a canonical spelling, which it classifies as it
+ always has (see ``_classify_url_host``).
Uses the run-scoped DNS cache to avoid repeated ``getaddrinfo`` lookups.
- Falls back to public on DNS timeout (fail-safe).
Lock discipline: cache reads and writes are guarded by ``ctx._lock``;
the DNS lookup itself runs OUTSIDE the lock so concurrent subagent
@@ -381,19 +637,12 @@ def _classify_host(
classification is stable for the run even under concurrent disagreeing
lookups.
"""
- if not hostname:
- return None
-
# Cache hit — read under the lock so concurrent writers don't tear
# the dict.
with ctx._lock:
if hostname in ctx._dns_cache:
return ctx._dns_cache[hostname]
- # User-declared local hostnames override DNS classification.
- if hostname in ctx.local_hostnames:
- return _cache_classification(ctx, hostname, True)
-
# NAT64-wrapped metadata IPs (e.g. 64:ff9b::169.254.169.254) classify
# as link-local by is_private_ip but actually wrap AWS/GCE instance
# metadata. Force them to PUBLIC so STRICT/PRIVATE_ONLY don't allow
@@ -1154,6 +1403,9 @@ def classify_engine(
# Neither eligibility is safe to infer. Deny this engine in
# protected scopes while Adaptive retains its restrictive
# PRIVATE_ONLY fallback for a non-contained private primary.
+ # An endpoint whose host is a cloud-metadata name or address, or
+ # has no canonical spelling (see _canonical_host), lands here
+ # too, so it is refused in every protected scope.
return EngineClassification(is_public=False, is_local=False)
return EngineClassification(is_public=is_public, is_local=is_local)
@@ -1369,10 +1621,16 @@ def _classify_engine_url(
engine opts in (``allow_host_dicts``). The opt-in also splits a string
value into the nodes the engine connects to, and treats as unknown a URL
whose host the client's parser reads differently, and a URL or dictionary
- host that is not ASCII (the resolver and the client encode such a name
- differently) or contains ``%`` (HTTP client normalization is
+ host that contains ``%`` (HTTP client normalization is
version-dependent). Other consumers keep single URL-string parsing
and percent-decoding.
+
+ Every host is canonicalised by ``_classifiable_host`` (one
+ percent-decode, one trailing root dot, case). A host that is not a
+ plain DNS name or a canonical IP literal, or that names a cloud-metadata
+ endpoint, is unknown, so an engine configured with one is refused under
+ every protected scope rather than classified local or failed up to
+ public.
"""
value = _get_setting_value(settings_snapshot, url_setting, None)
if not value:
@@ -1406,13 +1664,13 @@ def _classify_engine_url(
# The SDK uses the bare host, not a URL: retain IPv6 colons and
# never reinterpret a path, userinfo, or query as an authority.
hostname = entry.get("host")
- if not isinstance(hostname, str) or not hostname:
- any_unknown = True
- continue
- hostname = hostname.lower().rstrip(".")
- if not hostname or any(
- char.isspace() or ord(char) < 32 or char in "/?#@[]"
- for char in hostname
+ if (
+ not isinstance(hostname, str)
+ or not hostname
+ or any(
+ char.isspace() or ord(char) < 32 or char in "/?#@[]"
+ for char in hostname
+ )
):
any_unknown = True
continue
@@ -1449,28 +1707,36 @@ def _classify_engine_url(
# reads, so a different one here was never classified.
any_unknown = True
continue
- if not allow_host_dicts:
- # Other consumers decode, as evaluate_url does: requests
- # percent-decodes the host before connecting, so
- # "192%2e168%2e1%2e1" is 192.168.1.1. Elasticsearch's HTTP
- # client has varied across versions, so percent-bearing
- # hosts are withheld below.
- hostname = unquote(hostname)
else:
if allow_host_dicts:
any_unknown = True
continue
- if allow_host_dicts and (not hostname.isascii() or "%" in hostname):
+ if allow_host_dicts and "%" in hostname:
# Both branches meet here, so a dictionary host and a URL host are
# judged alike, as written. urllib3 2.8 decodes unreserved host
- # escapes that 2.7 passed to DNS unchanged. Keep percent-bearing
- # hosts unknown under either behavior, including scoped IPv6
- # literals such as ``fe80::1%eth0``. The resolver also encodes a
- # non-ASCII name with IDNA 2003 (``faß`` is looked up as
- # ``fass``) but the client's urllib3 uses IDNA 2008
- # (``xn--fa-hia``). Either way the name classified here need not
- # be the one the client connects to. An ``xn--`` spelling is
- # plain ASCII and is classified as written.
+ # escapes that 2.7 passed to DNS unchanged, so for Elasticsearch
+ # even a host that decodes to a canonical one is ambiguous: keep
+ # every percent-bearing host unknown, under either behaviour.
+ any_unknown = True
+ continue
+ # The host is passed as parsed, NOT decoded: _classifiable_host
+ # percent-decodes it exactly once (other consumers' HTTP clients
+ # decode it before connecting, so "192%2e168%2e1%2e1" is
+ # 192.168.1.1), ignores one trailing root dot and folds case. A host
+ # that is not then a plain DNS name or a canonical IP literal
+ # (residual "%", IPv6 zone ID, non-ASCII, control characters,
+ # integer / hex / octal / short IPv4), or that names a cloud-metadata
+ # endpoint, is unclassified. _classify_host would report it as not
+ # local (False), which this function reads as "public" and would use
+ # to fail a local engine UP, making it eligible under PUBLIC_ONLY:
+ # leave it unknown instead, so classify_engine withholds the engine
+ # in every protected scope. Vector-store callers only accept True,
+ # so this changes nothing for them. A non-ASCII name is unknown here
+ # for a second reason as well: the resolver encodes it with IDNA
+ # 2003 ("faß" is looked up as "fass") but the Elasticsearch client's
+ # urllib3 with IDNA 2008 ("xn--fa-hia"). An "xn--" spelling is plain
+ # ASCII and is classified as written.
+ if _classifiable_host(hostname) is None:
any_unknown = True
continue
result = _classify_host(hostname, ctx, allow_dns=allow_dns)
@@ -1623,12 +1889,12 @@ def evaluate_llm_endpoint(
try:
parsed = urlsplit(url_value)
- # Percent-decode the host before classifying — the HTTP client
- # decodes it before connect, so "http://127%2e0%2e0%2e1:11434"
- # must read as the local 127.0.0.1 (matches evaluate_url and
- # _classify_engine_url). Without this a legitimate percent-encoded
- # local endpoint is wrongly classified public and denied.
- hostname = unquote(parsed.hostname) if parsed.hostname else None
+ # Passed as parsed, NOT decoded: _classify_host percent-decodes
+ # the host exactly once (the HTTP client decodes it before
+ # connect, so "http://127%2e0%2e0%2e1:11434" reads as the local
+ # 127.0.0.1) and refuses to call any host local that is not a
+ # plain DNS name or a canonical IP literal after that decode.
+ hostname = parsed.hostname
except Exception:
return Decision(False, "url_malformed")
@@ -1671,7 +1937,8 @@ def evaluate_embeddings(
if not url:
return Decision(True, "provider_local_default")
parsed = urlsplit(url)
- host = unquote(parsed.hostname) if parsed.hostname else None
+ # As parsed: _classify_host decodes the host once itself.
+ host = parsed.hostname
classification = _classify_host(host, ctx) if host else None
return (
Decision(True, "provider_local")
@@ -1685,7 +1952,8 @@ def evaluate_embeddings(
)
if base_url:
parsed = urlsplit(base_url)
- host = unquote(parsed.hostname) if parsed.hostname else None
+ # As parsed: _classify_host decodes the host once itself.
+ host = parsed.hostname
classification = _classify_host(host, ctx) if host else None
if classification is True:
return Decision(True, "provider_local_endpoint")
@@ -1703,13 +1971,15 @@ _DANGEROUS_SCHEMES = frozenset(
{"javascript", "data", "file", "vbscript", "about"}
)
-# Cloud-metadata endpoints reachable by HOSTNAME rather than literal IP. GCP's
-# IMDS answers on metadata.google.internal / metadata.goog (resolving to
-# 169.254.169.254); is_ip_blocked only catches the IP literal, so these names
-# need an explicit block to honor the "metadata is NEVER permitted, regardless
-# of scope" invariant in evaluate_url. AWS/Azure/Alibaba IMDS have no such
-# hostname (IP only) and are covered by the is_ip_blocked / alt-encoding path.
-_METADATA_HOSTNAMES = frozenset({"metadata.google.internal", "metadata.goog"})
+# Cloud-metadata endpoints reachable by HOSTNAME rather than literal IP (GCP
+# metadata.google.internal / metadata.goog, IBM Cloud classic
+# api.service.softlayer.com) are blocked in evaluate_url via
+# _is_metadata_destination (ssrf_validator.is_metadata_hostname), and in host
+# classification via _is_metadata_host. ssrf_validator's
+# ALWAYS_BLOCKED_METADATA_HOSTNAMES set is the single source of truth shared
+# with validate_url and the notification validator. is_ip_blocked only catches
+# IP literals, so the name check is needed to honor the "metadata is NEVER
+# permitted, regardless of scope" invariant in evaluate_url.
def _normalize_alt_ipv4(host: str) -> Optional[str]:
@@ -1726,10 +1996,10 @@ def _normalize_alt_ipv4(host: str) -> Optional[str]:
parses canonical notation) and reads as an allowed public host.
"""
try:
- import socket
-
return socket.inet_ntoa(socket.inet_aton(host))
- except (OSError, UnicodeError, TypeError):
+ except (OSError, UnicodeError, TypeError, ValueError):
+ # ValueError: an embedded NUL character. Never raises, so a caller
+ # (evaluate_url's metadata check) cannot fail on any input.
return None
@@ -1757,6 +2027,82 @@ def _quota_ctx(ctx: EgressContext) -> EgressContext:
return active if active is not None else ctx
+def _url_host_refusal(hostname: str) -> Optional[str]:
+ """The reason ``evaluate_url`` refuses a URL host before classifying it
+ (``"blocked_metadata_ip"`` or ``"url_malformed"``), or None. Shared with
+ ``host_not_classified_public`` so the two cannot disagree. ``hostname``
+ is the host as ``urlsplit`` returns it (not decoded).
+
+ HTTP client libraries (requests/urllib3) percent-DECODE the host in the
+ netloc before the socket connect, but urlsplit().hostname preserves the
+ encoding, and trailing dots are insignificant to the resolver. The
+ checks below therefore look at every spelling the resolver may connect
+ to (``_policy_host_forms``: decoded once, undecoded, decoded until
+ stable, trailing dots stripped, the IDNA-folded forms of a non-ASCII
+ host, the dotted quad of an octal / hex / integer / short IPv4
+ notation). This broad expansion only ever adds refusals; whether a host
+ is local or public is left to ``_classify_url_host``.
+ """
+ host_forms = _policy_host_forms(hostname)
+
+ # Cloud-metadata endpoints (AWS/GCE/Azure IMDS at 169.254.169.254 etc.)
+ # are NEVER permitted, regardless of scope. They classify as
+ # link-local/private, so STRICT and PRIVATE_ONLY would otherwise ALLOW
+ # them — a credential-theft path for prompt-injected fetches and, more
+ # importantly, for the audit-hook net which calls evaluate_url directly
+ # on raw socket.connect targets (bypassing the SSRF validator that the
+ # explicit fetch PEPs run first).
+ #
+ # is_ip_blocked(allow_localhost=True, allow_private_ips=True) is True
+ # for the always-blocked metadata set (IPv4-mapped forms unwrapped, zone
+ # IDs ignored) and NAT64 wraps of it, and also for every other address
+ # in the SSRF validator's blocked ranges that is neither loopback nor
+ # one of the private ranges it allows: 0.0.0.0/8, the IPv6 unspecified
+ # address, 6to4, Teredo, the IPv4-compatible and IPv4-translated forms,
+ # the IPv6 discard prefix, and the NAT64 prefixes (when the operator
+ # allows NAT64, only a wrap whose embedded IPv4 is itself blocked). All
+ # of those are refused under the same blocked_metadata_ip reason. It
+ # only parses IP notation ipaddress accepts, so each form is checked.
+ try:
+ from ..ssrf_validator import is_ip_blocked
+
+ for form in host_forms:
+ if is_ip_blocked(
+ form, allow_localhost=True, allow_private_ips=True
+ ):
+ return "blocked_metadata_ip"
+ except Exception: # noqa: silent-exception - defensive, see below
+ # Non-IP host or helper error: fall through to the metadata check
+ # below and then to normal classification (the DNS path has its
+ # own handling).
+ pass
+
+ # Metadata endpoints reachable by hostname (GCP, IBM Cloud classic) in
+ # any of those spellings, and metadata addresses wherever an IPv6
+ # address may embed them (every RFC 6052 NAT64 position,
+ # network-specific prefixes included, 6to4, Teredo), which is_ip_blocked
+ # does not look for. IBM's name resolves to RFC1918 (10.0.80.88), which
+ # STRICT / PRIVATE_ONLY would otherwise admit as a local host. Refused
+ # in every scope (the SSRF validator also blocks the name on the actual
+ # fetch, but evaluate_url's own guarantee must not depend on it).
+ # Deliberately OUTSIDE the try/except above, so a helper exception
+ # cannot swallow this refusal: _is_metadata_destination never raises
+ # (is_metadata_hostname fails closed), and evaluate_url's outer handler
+ # denies with internal_error if anything here does.
+ if _is_metadata_destination(hostname):
+ return "blocked_metadata_ip"
+
+ # A host that is, or decodes to, text with whitespace, a control
+ # character or a backslash names no destination an HTTP client can
+ # reach (getaddrinfo refuses an embedded NUL outright): malformed in
+ # every scope rather than classified.
+ from ..ssrf_validator import RFC_FORBIDDEN_URL_CHARS_RE
+
+ if any(RFC_FORBIDDEN_URL_CHARS_RE.search(form) for form in host_forms):
+ return "url_malformed"
+ return None
+
+
def evaluate_url(
url: str, ctx: EgressContext, *, record_denial: bool = True
) -> Decision:
@@ -1799,65 +2145,12 @@ def evaluate_url(
if not parsed.hostname:
return deny(ctx, url, "no_hostname")
- # HTTP client libraries (requests/urllib3) percent-DECODE the host
- # in the netloc before the socket connect, but urlsplit().hostname
- # preserves the encoding. Classifying the encoded form lets
- # "http://192%2e168%2e1%2e1/" read as an unresolvable public host
- # (DNS fails) while the client actually connects to the private
- # 192.168.1.1 — a scope bypass under PUBLIC_ONLY. Classify the
- # DECODED host so the policy sees the real connect target.
- # Trailing dots are insignificant to the resolver (getaddrinfo strips
- # them), so "169.254.169.254." and "metadata.google.internal." must be
- # classified identically to their bare forms — otherwise a trailing
- # dot dodges the metadata checks below. Mirrors the SSRF validator's
- # rstrip("."). Shared with ``host_not_classified_public`` via
- # ``_policy_host`` so both classify the same host.
- host = _policy_host(parsed.hostname)
-
- # Cloud-metadata endpoints (AWS/GCE/Azure IMDS at 169.254.169.254
- # etc.) are NEVER permitted, regardless of scope. They classify as
- # link-local/private, so STRICT and PRIVATE_ONLY would otherwise
- # ALLOW them — a credential-theft path for prompt-injected fetches
- # and, more importantly, for the audit-hook net which calls
- # evaluate_url directly on raw socket.connect targets (bypassing the
- # SSRF validator that the explicit fetch PEPs run first). Reuse
- # is_ip_blocked(allow_private_ips=True), which returns True only for
- # the always-blocked metadata set (+ NAT64 wraps) and also unwraps
- # IPv4-mapped IPv6 forms.
- try:
- from ..ssrf_validator import is_ip_blocked
-
- if is_ip_blocked(
- host, allow_localhost=True, allow_private_ips=True
- ):
- return deny(ctx, url, "blocked_metadata_ip")
- # is_ip_blocked only parses canonical IP notation, so an
- # alternate-encoded metadata literal (octal/hex/integer) slips
- # through above. Normalize it to the dotted-quad the resolver
- # would connect to and re-check, so the "metadata IPs are NEVER
- # permitted, regardless of scope" invariant holds for those forms
- # too — and they get the explicit blocked_metadata_ip reason
- # rather than leaking into the scope-mismatch bucket.
- alt = _normalize_alt_ipv4(host)
- if (
- alt is not None
- and alt != host
- and is_ip_blocked(
- alt, allow_localhost=True, allow_private_ips=True
- )
- ):
- return deny(ctx, url, "blocked_metadata_ip")
- # Metadata endpoints reachable by hostname (GCP) — is_ip_blocked
- # can't see these because they aren't IP literals. Block by name so
- # the invariant holds under PUBLIC_ONLY/BOTH too (the SSRF validator
- # backstops the actual fetch, but evaluate_url's own guarantee must
- # not depend on it).
- if host.lower() in _METADATA_HOSTNAMES:
- return deny(ctx, url, "blocked_metadata_ip")
- except Exception: # noqa: silent-exception - defensive, see below
- # Non-IP host or helper error: fall through to normal
- # classification (DNS path has its own handling).
- pass
+ # Cloud-metadata endpoints, other always-blocked addresses and
+ # malformed hosts are refused in every scope, before any scope or
+ # local_hostnames logic (see _url_host_refusal).
+ refusal = _url_host_refusal(parsed.hostname)
+ if refusal is not None:
+ return deny(ctx, url, refusal)
# UNPROTECTED: egress protection disabled — any host is allowed. This
# gate is deliberately AFTER the SSRF / cloud-metadata block above
@@ -1866,7 +2159,9 @@ def evaluate_url(
if ctx.scope == EgressScope.UNPROTECTED:
return Decision(True, "egress_unprotected")
- classification = _classify_host(host, ctx)
+ # Shared with ``host_not_classified_public`` so both classify the
+ # same host the same way.
+ classification = _classify_url_host(parsed.hostname, ctx)
if ctx.scope == EgressScope.STRICT:
# Under STRICT, URLs whose host is private are allowed; public
@@ -1916,23 +2211,24 @@ def host_not_classified_public(url: str, ctx: EgressContext) -> bool:
of course (the full-content search wrapper, whose result URLs come from
an index it does not control, under every scope but ``unprotected``)
drop them before ``evaluate_url`` without each one spending the run's
- quota. It uses the SAME host normalisation and the SAME
- ``_classify_host`` (and run-scoped DNS cache) as ``evaluate_url``, so
- the set it returns True for is exactly the set ``evaluate_url`` refuses
- as ``scope_mismatch_public_only`` under ``public_only``, not a second,
- differently defined list. (The SSRF validator's ``PRIVATE_IP_RANGES``
- is narrower: it admits 198.18.0.0/15, the TEST-NETs, 192.0.0.0/24,
- 240.0.0.0/4 and 2001:db8::/32, and with NAT64 allowed any NAT64 address
- whose embedded IPv4 it admits. The 198.18.0.0/15 addresses (including
+ quota. It uses the SAME refusal checks (``_url_host_refusal``) and the
+ SAME classification (``_classify_url_host``, and the run-scoped DNS
+ cache) as ``evaluate_url``, so the set it returns True for is exactly
+ the set ``evaluate_url`` refuses as ``scope_mismatch_public_only`` under
+ ``public_only``, not a second, differently defined list. (The SSRF
+ validator's ``PRIVATE_IP_RANGES`` is narrower: it admits 198.18.0.0/15,
+ the TEST-NETs, 192.0.0.0/24, 240.0.0.0/4 and 2001:db8::/32, and with
+ NAT64 allowed any NAT64 address whose embedded IPv4 it admits. The 198.18.0.0/15 addresses (including
fake-IP DNS answers) are unclassified here; the other listed ranges
classify private. IPv4-mapped and NAT64 addresses classify as their
embedded IPv4.)
Returns False for anything ``evaluate_url`` refuses for another reason
- first (malformed URL, non-http(s) scheme, no host); a cloud-metadata
- host classifies public, so it is left to ``evaluate_url``'s
- ``blocked_metadata_ip`` denial. An internal error returns True (fails
- toward refusal, which records no denial either).
+ first (malformed URL or host, non-http(s) scheme, no host, and a
+ cloud-metadata or other always-blocked host, ``_url_host_refusal``), so
+ that refusal is left to ``evaluate_url``, which counts a metadata
+ denial. An internal error returns True (fails toward refusal, which
+ records no denial either).
"""
if not isinstance(url, str) or not url:
return False
@@ -1944,34 +2240,142 @@ def host_not_classified_public(url: str, ctx: EgressContext) -> bool:
if parsed.scheme.lower() not in ("http", "https") or not hostname:
return False
try:
- return _classify_host(_policy_host(hostname), ctx) is not False
+ if _url_host_refusal(hostname) is not None:
+ return False
+ return _classify_url_host(hostname, ctx) is not False
except Exception: # noqa: silent-exception - fail toward refusal
return True
def _policy_host(hostname: str) -> str:
- """The host ``evaluate_url`` classifies: percent-decoded (HTTP clients
- decode it before connecting) and without trailing dots (insignificant
- to the resolver); see the comments at its call site there."""
+ """``evaluate_url``'s broad reading of a URL host: percent-decoded once
+ (HTTP clients decode it before connecting) and without trailing dots
+ (insignificant to the resolver). The first of ``_policy_host_forms``,
+ and the host ``_classify_url_host`` classifies when it has no canonical
+ spelling. Classification of a canonical host uses ``_canonical_host``
+ instead."""
return unquote(hostname).rstrip(".")
+def _classify_url_host(hostname: str, ctx: EgressContext) -> Optional[bool]:
+ """How ``evaluate_url`` and ``host_not_classified_public`` classify the
+ host of a URL being fetched: local (True) / public (False) / unknown
+ (None). ``hostname`` is the host as ``urlsplit`` returns it.
+
+ A host with a canonical spelling (``_canonical_host``) is classified by
+ ``_classify_host``, exactly as the require-local and engine checks
+ classify it; a cloud-metadata host comes back False (not local), and
+ ``evaluate_url`` has refused it before asking.
+
+ A host without one (IDN or fullwidth spelling, IPv6 zone ID, integer /
+ hex / octal / short IPv4, a residual ``%``) is still classified as the
+ fetch check always has, by ``_classify_destination`` on its
+ ``_policy_host`` reading: the fetch check keeps its broader handling.
+ Only the ``llm.allowed_local_hostnames`` override is not applied to it:
+ an entry
+ without a canonical spelling is ignored everywhere, and a URL host that
+ matches one exactly is unknown (None) here rather than local, so that
+ the ignored entry cannot make the host classify public either. And when
+ the host as written, NOT decoded, is itself an IP literal (an HTTP
+ client reads ``%`` in an IPv6 literal as the zone delimiter, so
+ ``fd12::1%10`` is that address on interface 10, while the decoded
+ reading is not an address at all), the two readings must agree or the
+ host is unknown. Each of these can only turn a classification into
+ None, which no scope admits, so this path is never looser than before.
+ """
+ if _canonical_host(hostname) is not None:
+ return _classify_host(hostname, ctx)
+ host = _policy_host(hostname)
+ if not host or host in ctx.local_hostnames:
+ return None
+ result = _classify_destination(host, ctx)
+ undecoded = hostname.rstrip(".")
+ if undecoded != host:
+ try:
+ ipaddress.ip_address(undecoded)
+ except ValueError:
+ return result
+ if _classify_destination(undecoded, ctx) is not result:
+ return None
+ return result
+
+
+def _policy_host_forms(hostname: str) -> tuple[str, ...]:
+ """The spellings of ``hostname`` ``evaluate_url`` checks for a metadata
+ destination (directly and through ``_is_metadata_destination``). Broad
+ on purpose: every extra form can only add a denial. Host classification
+ does not use it; it accepts only canonical hosts (``_canonical_host``).
+ ``hostname`` is the host as ``urlsplit`` returns it (not decoded).
+
+ The first element is ``_policy_host(hostname)`` (percent-decoded once,
+ trailing dots removed), without IPv6 brackets. When the host contains
+ ``%``, it is followed by the same host NOT decoded (an HTTP client reads
+ ``%`` in an IPv6 literal as the zone delimiter: ``fd00:ec2::254%10`` is
+ that address on interface 10) and by the host decoded until no escape
+ is left. A non-ASCII host is followed by the ASCII forms the resolver
+ or an HTTP client may turn it into (``ssrf_validator._idna_folded_names``:
+ IDNA 2003 folding of fullwidth digits and letters and ideographic full
+ stops, and the IDNA 2008 / UTS #46 encodings), and every form that is an
+ IPv4 address in an alternate notation ``inet_aton`` (and so the
+ resolver) accepts — octal, hex, integer or a short form — is followed by
+ its dotted quad (``_normalize_alt_ipv4``). Never raises.
+ """
+ from ..ssrf_validator import _idna_folded_names
+
+ host = _policy_host(hostname)
+ if host.startswith("[") and host.endswith("]"):
+ host = host[1:-1]
+ forms = [host]
+ undecoded = hostname.rstrip(".")
+ if undecoded.startswith("[") and undecoded.endswith("]"):
+ undecoded = undecoded[1:-1]
+ if undecoded and undecoded not in forms:
+ forms.append(undecoded)
+ # And decoded until stable (bounded): no HTTP client decodes twice, but
+ # a host that only decodes to a metadata address after a second pass is
+ # never a legitimate destination either, so refusing it costs nothing.
+ repeated = host
+ for _ in range(4):
+ decoded = unquote(repeated)
+ if decoded == repeated:
+ break
+ repeated = decoded
+ repeated = repeated.rstrip(".")
+ if repeated.startswith("[") and repeated.endswith("]"):
+ repeated = repeated[1:-1]
+ if repeated and repeated not in forms:
+ forms.append(repeated)
+ if not host.isascii():
+ # None (not encodable) adds nothing; is_metadata_hostname fails
+ # closed on such a name.
+ for folded in _idna_folded_names(host) or ():
+ folded = folded.lower().rstrip(".")
+ if folded and folded not in forms:
+ forms.append(folded)
+ for form in list(forms):
+ alt = _normalize_alt_ipv4(form)
+ if alt is not None and alt not in forms:
+ forms.append(alt)
+ return tuple(forms)
+
+
def url_host_is_declared_local(url: object, ctx: EgressContext) -> bool:
"""Whether the URL host has the run's operator-declared local status.
The declared name must retain its classification when DNS pinning checks
the connect-time address; an address-only URL would lose that override.
- Apply the same hostname normalization as ``evaluate_url``.
+ Uses the same rule as ``_classify_host``: the host's canonical spelling
+ (``_canonical_host``) is one of the canonical, non-metadata entries
+ (``_declared_local_hosts``). A host without a canonical spelling is
+ never declared local.
"""
if not ctx.local_hostnames or not isinstance(url, str) or not url:
return False
try:
- host = urlsplit(url).hostname
+ host = _canonical_host(urlsplit(url).hostname)
except Exception:
return False
- if not host:
- return False
- return _policy_host(host) in ctx.local_hostnames
+ return host is not None and host in _declared_local_hosts(ctx)
def _record_denial(ctx: EgressContext, url: object, reason: str) -> Decision:
diff --git a/src/local_deep_research/security/egress/validators.py b/src/local_deep_research/security/egress/validators.py
index c674c6f3f5..b18bd98394 100644
--- a/src/local_deep_research/security/egress/validators.py
+++ b/src/local_deep_research/security/egress/validators.py
@@ -24,7 +24,9 @@ from .policy import (
EgressContext,
EgressScope,
PolicyDeniedError,
+ _canonical_host,
_classify_host,
+ _is_metadata_host,
_resolve_with_timeout,
parse_user_egress_scope,
)
@@ -60,19 +62,94 @@ def validate_egress_scope(form_data, _all_db_settings):
def validate_allowed_local_hostnames(form_data, all_db_settings):
- """Reject public hostnames being added to llm.allowed_local_hostnames.
+ """Reject entries in llm.allowed_local_hostnames that would not, or must
+ not, make a host local.
The default-settings description for this key claims "Public hostnames
added here are rejected at save time", but until now no code actually
- did the rejection. This guard resolves each entry via the same host
- classifier the policy uses, and refuses any that resolve to public
- addresses. A hostname that fails to resolve (DNS down) is accepted —
- fail-open on transient lookup errors so the user can recover.
+ did the rejection. Each entry is judged exactly as host classification
+ reads it (``policy._canonical_host``: decoded once, case folded, one
+ trailing dot ignored), and refused when it
+
+ * contains ``%`` (percent-encoding or an IPv6 zone ID): no hostname
+ needs it, and refusing it keeps the saved text and the name it
+ matches the same (classification reads a stored entry decoded once);
+ * is not a plain DNS name or a canonical IP address (non-ASCII
+ characters, whitespace or control characters, an integer / hex /
+ octal / short IPv4 notation) -- classification ignores such an entry,
+ so accepting it would only hide that it has no effect;
+ * names a cloud-metadata endpoint (``policy._is_metadata_host``), which
+ can never be local; or
+ * resolves to public addresses, via the same host classifier the policy
+ uses. A hostname that fails to resolve (DNS down) is accepted --
+ fail-open on transient lookup errors so the user can recover.
+
+ Only a changed value is validated: one equal to the stored value,
+ entry for entry, is accepted unchecked
+ (``_allowed_local_hostnames_unchanged``). Never raises: a missing
+ ``form_data`` (None) has nothing to validate, and an unexpected error is
+ reported as a validation error (fail closed) rather than escaping into
+ the save route.
"""
key = "llm.allowed_local_hostnames"
- if key not in form_data:
- return None
- value = form_data[key]
+ try:
+ if not form_data or key not in form_data:
+ return None
+ value = form_data[key]
+ # Validate-on-change, as for the engine URLs: the All-Settings tab
+ # resubmits every setting, so an unchanged list (say, one holding an
+ # entry saved before this validator became stricter) must not block
+ # an unrelated save. Runtime classification ignores such an entry
+ # anyway. A changed list is validated in full.
+ if _allowed_local_hostnames_unchanged(value, all_db_settings, key):
+ return None
+ return _check_allowed_local_hostnames(key, value)
+ except Exception:
+ return {
+ "key": key,
+ "error": (
+ "allowed_local_hostnames could not be validated; check that "
+ "it is a JSON array of plain hostnames or IP addresses"
+ ),
+ }
+
+
+def _allowed_local_hostnames_entries(value):
+ """The entries of an ``llm.allowed_local_hostnames`` value (a list, or
+ its JSON text), or None when it is neither."""
+ import json as _json
+
+ value = unwrap_setting(value)
+ if isinstance(value, str):
+ try:
+ value = _json.loads(value) if value.strip() else []
+ except ValueError:
+ return None
+ return value if isinstance(value, list) else None
+
+
+def _allowed_local_hostnames_unchanged(submitted, all_db_settings, key):
+ """True when the submitted ``llm.allowed_local_hostnames`` value holds
+ exactly the stored entries, in the same order (a list and its JSON text
+ compare equal). Fails toward validating: no stored value, or a value on
+ either side that is not a list, counts as changed."""
+ stored = _stored_engine_url_value(all_db_settings, key)
+ if stored is _NO_STORED_VALUE:
+ return False
+ entries = _allowed_local_hostnames_entries(submitted)
+ return entries is not None and entries == _allowed_local_hostnames_entries(
+ stored
+ )
+
+
+def _display_entry(entry: str) -> str:
+ """An entry as quoted in an error message: ``repr`` shows whitespace and
+ control characters a plain rendering would hide, bounded in length."""
+ text = repr(entry)
+ return text if len(text) <= 80 else text[:77] + "..."
+
+
+def _check_allowed_local_hostnames(key, value):
# Setting is JSON-typed; the save pipeline may hand us a list or a
# JSON string. Decode defensively.
if isinstance(value, str):
@@ -108,10 +185,26 @@ def validate_allowed_local_hostnames(form_data, all_db_settings):
)
rejected = []
+ metadata = []
+ invalid = []
for entry in value:
if not isinstance(entry, str) or not entry.strip():
continue
- hostname = entry.strip().lower()
+ # Judged exactly as classification reads the entry, as written:
+ # _canonical_host decodes it once and folds case itself. An entry
+ # with "%" is refused before that, so a saved entry is always the
+ # text it matches.
+ hostname = None if "%" in entry else _canonical_host(entry)
+ if hostname is None:
+ invalid.append(_display_entry(entry))
+ continue
+ # A cloud-metadata name or address can never be declared local
+ # (classification ignores such an entry), whatever it resolves to.
+ # Report it as what it is rather than as a public address: IBM
+ # Cloud classic's metadata name resolves to an RFC1918 address.
+ if _is_metadata_host(hostname):
+ metadata.append(hostname)
+ continue
try:
# Distinguish "could not resolve" from "resolved to a public IP".
# _classify_host collapses BOTH to False (its documented fail-safe
@@ -132,16 +225,32 @@ def validate_allowed_local_hostnames(form_data, all_db_settings):
continue
if classification is False:
rejected.append(hostname)
+ errors = []
+ if invalid:
+ errors.append(
+ "These entries are not valid hostnames for this list: "
+ f"{', '.join(invalid)}. Only plain DNS names (ASCII letters, "
+ "digits, hyphens, underscores and dots, written without "
+ "percent-encoding) and standard IP addresses (dotted-quad IPv4, "
+ "or IPv6 without a zone ID) are accepted; any other spelling "
+ "would be ignored. Write the name in that form (an "
+ "internationalised name in its xn-- form) or remove it."
+ )
+ if metadata:
+ errors.append(
+ "These entries are cloud-metadata endpoints, which are blocked "
+ "under every scope and can never be treated as local: "
+ f"{', '.join(metadata)}. Remove them."
+ )
if rejected:
- return {
- "key": key,
- "error": (
- "These hostnames resolve to PUBLIC addresses and would "
- "let the policy treat external hosts as local: "
- f"{', '.join(rejected)}. Remove them, or use the SSRF "
- "allowlist instead."
- ),
- }
+ errors.append(
+ "These hostnames resolve to PUBLIC addresses and would "
+ "let the policy treat external hosts as local: "
+ f"{', '.join(rejected)}. Remove them, or use the SSRF "
+ "allowlist instead."
+ )
+ if errors:
+ return {"key": key, "error": " ".join(errors)}
return None
@@ -586,7 +695,11 @@ def _engine_url_ssrf_error(key, value, *, allow_private: bool):
return None
url = value.strip()
- from ..ssrf_validator import RFC_FORBIDDEN_URL_CHARS_RE, is_ip_blocked
+ from ..ssrf_validator import (
+ RFC_FORBIDDEN_URL_CHARS_RE,
+ is_ip_blocked,
+ is_metadata_hostname,
+ )
if RFC_FORBIDDEN_URL_CHARS_RE.search(url):
return {
@@ -612,6 +725,17 @@ def _engine_url_ssrf_error(key, value, *, allow_private: bool):
host = host[1:-1]
host = host.rstrip(".")
+ # Cloud-metadata hostname (GCP, IBM Cloud classic): refused regardless of
+ # opt-in. IBM's name resolves to RFC1918, so the IP checks below would
+ # pass it when allow_private is set. is_metadata_hostname also fails
+ # closed on a non-ASCII name no IDNA encoder (IDNA 2003, IDNA 2008 or
+ # UTS #46) can encode; that name is refused too, but reported as what it
+ # is rather than as metadata.
+ if is_metadata_hostname(host):
+ if not _idna_encodable(host):
+ return _engine_url_error(key, "invalid_host")
+ return _engine_url_error(key, "metadata")
+
def _block_reason(ip_str) -> Optional[str]:
# Cloud-metadata / always-blocked set: refused regardless of opt-in.
if is_ip_blocked(ip_str, allow_localhost=True, allow_private_ips=True):
@@ -647,12 +771,37 @@ def _engine_url_ssrf_error(key, value, *, allow_private: bool):
return None
+def _idna_encodable(host: str) -> bool:
+ """Whether any IDNA encoder a resolver or HTTP client uses can encode
+ ``host`` as a DNS name.
+
+ ``ssrf_validator._idna_folded_names`` tries the stdlib ``idna`` codec
+ (IDNA 2003, what ``socket.getaddrinfo`` uses), the ``idna`` package's
+ IDNA 2008 encoding (what urllib3 and ``httpx`` use) and its UTS #46
+ mapping. A name none of them accepts (for example one with a label
+ over 63 octets) cannot be looked up through any of them;
+ ``is_metadata_hostname`` refuses such a name (fails closed), and this
+ lets the error message say why. A name one of them accepts is
+ encodable.
+ """
+ from ..ssrf_validator import _idna_folded_names
+
+ name = host.lower()
+ return name.isascii() or _idna_folded_names(name) is not None
+
+
def _engine_url_error(key, reason):
if reason == "metadata":
message = (
f"{key} points at a cloud-metadata / always-blocked address, "
"which is never a permitted destination."
)
+ elif reason == "invalid_host":
+ message = (
+ f"{key} has an invalid or unsupported hostname: no IDNA "
+ "encoding (IDNA 2003, IDNA 2008 or UTS #46) turns it into a DNS "
+ "name."
+ )
else:
message = (
f"{key} points at a private, loopback, or link-local address. A "
diff --git a/src/local_deep_research/security/notification_validator.py b/src/local_deep_research/security/notification_validator.py
index 09c1bea64c..72606affcc 100644
--- a/src/local_deep_research/security/notification_validator.py
+++ b/src/local_deep_research/security/notification_validator.py
@@ -30,7 +30,11 @@ from .legacy_ipv4 import (
is_percent_encoded_numeric_ipv4_host,
)
from .log_sanitizer import redact_and_bound_for_log
-from .ssrf_validator import RFC_FORBIDDEN_URL_CHARS_RE, redact_url_for_log
+from .ssrf_validator import (
+ RFC_FORBIDDEN_URL_CHARS_RE,
+ is_metadata_hostname,
+ redact_url_for_log,
+)
# (#6938) Cap for a request-supplied hostname or scheme in a log line, and
# in the rejection messages this module returns, which callers log
@@ -632,6 +636,15 @@ class NotificationURLValidator:
``block_link_local`` forces the whole link-local range blocked even
under ``allow_private_ips=True`` (plugin-scheme IMDS guard).
"""
+ # Cloud-metadata endpoints reachable by NAME (GCP, IBM Cloud classic)
+ # are blocked at every level and in every partition (http(s) and
+ # plugin schemes both reach here via _is_private_ip). IBM's
+ # api.service.softlayer.com resolves to RFC1918, so the IP checks
+ # below would pass it under allow_private_ips=True. Shared with
+ # validate_url via ssrf_validator.is_metadata_hostname.
+ if is_metadata_hostname(hostname):
+ return True
+
# Localhost-string shortcuts only apply when the operator hasn't
# opted into private-IP reachability. With allow_private_ips=True
# we let the IP path (DNS-resolved or literal) make the decision
@@ -1206,6 +1219,15 @@ class NotificationURLValidator:
allow_nat64=True,
block_link_local=block_link_local,
):
+ if is_metadata_hostname(effective_host):
+ # Blocked by name (GCP, IBM Cloud classic), not by
+ # address: IBM's name resolves to an RFC1918 address.
+ logger.warning("Blocked cloud-metadata notification host")
+ return (
+ False,
+ "Blocked cloud-metadata hostname: "
+ f"{redact_and_bound_for_log(effective_host, _LOG_HOST_MAX_CHARS)}",
+ )
logger.warning(
"Blocked metadata or link-local notification host"
)
diff --git a/src/local_deep_research/security/safe_requests.py b/src/local_deep_research/security/safe_requests.py
index 7fa27d0e53..4f9429a66c 100644
--- a/src/local_deep_research/security/safe_requests.py
+++ b/src/local_deep_research/security/safe_requests.py
@@ -493,8 +493,10 @@ def safe_get(
like SearXNG or Ollama in containerized environments.
Note: cloud metadata endpoints (AWS / Azure / GCP / OCI /
Scaleway / Akamai / OpenStack / Hetzner / DigitalOcean /
- AlibabaCloud / Tencent) are ALWAYS blocked — see
- ``ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS``.
+ AlibabaCloud / Tencent / Oracle Compute Classic, and the GCP /
+ IBM Cloud classic metadata hostnames) are ALWAYS blocked — see
+ ``ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS`` and
+ ``ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES``.
require_https: Refuse the fetch (and any redirect hop) whose URL
is not https://. Opt-in per caller for integrity-sensitive
downloads (e.g. the journal-quality datasets): a hop to
@@ -712,8 +714,10 @@ def safe_post(
like SearXNG or Ollama in containerized environments.
Note: cloud metadata endpoints (AWS / Azure / GCP / OCI /
Scaleway / Akamai / OpenStack / Hetzner / DigitalOcean /
- AlibabaCloud / Tencent) are ALWAYS blocked — see
- ``ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS``.
+ AlibabaCloud / Tencent / Oracle Compute Classic, and the GCP /
+ IBM Cloud classic metadata hostnames) are ALWAYS blocked — see
+ ``ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS`` and
+ ``ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES``.
**kwargs: Additional arguments to pass to requests.post()
Returns:
@@ -951,8 +955,10 @@ class SafeSession(requests.Session):
Ollama in containerized environments.
Note: cloud metadata endpoints (AWS / Azure / GCP / OCI /
Scaleway / Akamai / OpenStack / Hetzner / DigitalOcean /
- AlibabaCloud / Tencent) are ALWAYS blocked — see
- ``ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS``.
+ AlibabaCloud / Tencent / Oracle Compute Classic, and the
+ GCP / IBM Cloud classic metadata hostnames) are ALWAYS
+ blocked — see ``ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS``
+ and ``ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES``.
block_link_local: Keep the whole link-local range (IPv4
``169.254.0.0/16``, IPv6 ``fe80::/10`` and their NAT64-wrapped
forms) blocked EVEN under ``allow_private_ips=True``, on the
diff --git a/src/local_deep_research/security/ssrf_validator.py b/src/local_deep_research/security/ssrf_validator.py
index 8d48db8b09..f95e3de145 100644
--- a/src/local_deep_research/security/ssrf_validator.py
+++ b/src/local_deep_research/security/ssrf_validator.py
@@ -9,7 +9,7 @@ import ipaddress
import re
import socket
from contextvars import ContextVar
-from urllib.parse import urlparse
+from urllib.parse import unquote, urlparse
from typing import Optional
from loguru import logger
from urllib3.exceptions import LocationParseError
@@ -84,6 +84,145 @@ ALWAYS_BLOCKED_METADATA_IPS = frozenset(
}
)
+# Cloud-provider metadata endpoints reachable by HOSTNAME — always blocked,
+# under every flag combination, by every validator that classifies a
+# destination host (``validate_url`` and therefore ``assert_base_url_safe`` /
+# ``safe_requests``; the egress policy's ``evaluate_url``; the notification
+# validator; the search-engine URL settings validator). This is the single
+# source of truth; callers use ``is_metadata_hostname`` rather than keeping
+# their own list.
+#
+# * ``metadata.google.internal`` / ``metadata.goog`` — GCP IMDS. These resolve
+# to 169.254.169.254, so the IP block above already catches them after DNS;
+# the name block is defense in depth (e.g. a hosts-file / resolver override).
+# * ``api.service.softlayer.com`` — IBM Cloud classic infrastructure
+# (SoftLayer) API, which serves ``SoftLayer_Resource_Metadata`` to classic
+# instances over the private network. It resolves to ``10.0.80.88``, an
+# RFC1918 address, so it is reachable whenever ``allow_private_ips=True``
+# and the IP checks cannot stop it. ``10.0.80.88`` is deliberately NOT in
+# ``ALWAYS_BLOCKED_METADATA_IPS``: an always-block on an RFC1918 address
+# would break users who legitimately run a service (e.g. a local LLM
+# server) at that address on their own LAN. KNOWN LIMIT: this is a name
+# block only. A different DNS name (or a literal ``10.0.80.88``) that
+# reaches the same service is NOT blocked under ``allow_private_ips=True``.
+# (``161.26.0.6`` is NOT IBM metadata — it is IBM's shared package mirror /
+# NTP address — and is not listed anywhere here.)
+#
+# Matching is exact on the normalized name (see ``is_metadata_hostname``), so
+# subdomains such as ``metadata.google.internal.attacker.example`` are not
+# matched by name; they are judged by what they resolve to.
+ALWAYS_BLOCKED_METADATA_HOSTNAMES = frozenset(
+ {
+ "metadata.google.internal", # GCP IMDS
+ "metadata.goog", # GCP IMDS
+ "api.service.softlayer.com", # IBM Cloud classic (SoftLayer) metadata
+ }
+)
+
+
+# Characters IDNA treats as label separators (RFC 3490 section 3.1): the
+# ASCII full stop plus the ideographic, fullwidth and halfwidth ideographic
+# full stops. Mapped to "." before trailing-dot stripping so a trailing
+# "\u3002" is as insignificant as a trailing ".".
+_IDNA_DOT_TRANSLATION = str.maketrans(
+ {"\u3002": ".", "\uff0e": ".", "\uff61": "."}
+)
+
+
+def _idna_folded_names(name: str) -> Optional[list]:
+ """ASCII forms a resolver or HTTP client may turn a non-ASCII ``name``
+ into, or None when no IDNA encoder can encode it.
+
+ * ``socket.getaddrinfo`` encodes a ``str`` host with Python's ``idna``
+ codec (IDNA 2003 nameprep). That codec folds fullwidth letters to
+ ASCII, treats the U+3002 / U+FF0E / U+FF61 full stops as ".", and
+ drops soft hyphens and zero-width characters, so
+ ``api\u3002service\u3002softlayer\u3002com`` and
+ ``\uff41\uff50\uff49.service.softlayer.com`` resolve as
+ ``api.service.softlayer.com``. Code that hands such a name to the
+ resolver (raw sockets, Apprise plugins) reaches the listed host.
+ * urllib3 (and so ``requests``) and ``httpx`` encode the lower-cased
+ name with the ``idna`` package in IDNA 2008 mode, without the UTS #46
+ mapping. They refuse fullwidth letters and soft hyphens outright
+ (``httpx`` does accept the ideographic full stops as separators), but
+ they accept names the IDNA 2003 codec rejects, such as a
+ right-to-left label ending in a digit.
+ * The ``idna`` package's UTS #46 form is included as defence in depth
+ for any client that applies that mapping.
+
+ Every form an encoder produces is returned; an encoder that rejects the
+ name (or the absent ``idna`` package) adds none. Returns None only when
+ no encoder produces a form: no client can then look the name up, and
+ the caller fails CLOSED on that.
+ """
+ folded = []
+ try:
+ folded.append(name.encode("idna").decode("ascii"))
+ except (UnicodeError, ValueError):
+ # The resolver cannot encode it; an HTTP client's IDNA 2008 encoder
+ # may still, so the forms below are what is left to check.
+ pass
+ try:
+ import idna as _idna_pkg
+ except ImportError: # pragma: no cover - idna ships with requests/httpx
+ _idna_pkg = None
+ if _idna_pkg is not None:
+ for encode in (
+ lambda n: _idna_pkg.encode(n.lower()),
+ lambda n: _idna_pkg.encode(n, uts46=True),
+ ):
+ try:
+ folded.append(encode(name).decode("ascii"))
+ except Exception: # noqa: silent-exception - encoder rejects it
+ # This encoder refuses the name, so a client using it
+ # cannot connect to anything by it: no form to match.
+ pass
+ return folded or None
+
+
+def is_metadata_hostname(host: object) -> bool:
+ """Return True if ``host`` names an always-blocked cloud-metadata endpoint.
+
+ Normalizes the way the resolver / HTTP client would before matching:
+ percent-decoding, surrounding whitespace, case, IPv6 brackets, any
+ trailing root dots (``getaddrinfo`` ignores them) and, for a non-ASCII
+ name, every ASCII form an IDNA encoder produces for it
+ (``_idna_folded_names``): the folding ``getaddrinfo`` applies (stdlib
+ ``idna`` codec, IDNA 2003), the ``idna`` package's IDNA 2008 encoding
+ that urllib3 and ``httpx`` use, and its UTS #46 mapping as defence in
+ depth. So ``API.Service.SoftLayer.com.``,
+ ``api\u3002service\u3002softlayer\u3002com`` and fullwidth
+ ``\uff41\uff50\uff49.service.softlayer.com`` all match
+ ``api.service.softlayer.com``, while a public name that only IDNA 2008
+ accepts (a right-to-left label ending in a digit, which the IDNA 2003
+ codec rejects) is judged by the name those HTTP clients connect to.
+
+ Fails CLOSED: a non-ASCII name that none of those encoders can encode
+ returns True. No client can look such a name up through them, and one
+ with yet another encoder, or a proxy that resolves names itself, is
+ refused here deliberately. Non-string input returns False.
+ """
+ if not isinstance(host, str):
+ return False
+ name = unquote(host).strip().lower()
+ if name.startswith("[") and name.endswith("]"):
+ name = name[1:-1]
+ name = name.translate(_IDNA_DOT_TRANSLATION).rstrip(".")
+ if name in ALWAYS_BLOCKED_METADATA_HOSTNAMES:
+ return True
+ if name.isascii():
+ # IDNA leaves ASCII labels unchanged apart from case, which is
+ # already folded, so no other spelling can reach a listed name.
+ return False
+ folded = _idna_folded_names(name)
+ if folded is None:
+ return True
+ return any(
+ f.lower().rstrip(".") in ALWAYS_BLOCKED_METADATA_HOSTNAMES
+ for f in folded
+ )
+
+
# Link-local ranges. When ``block_link_local`` is set (notification
# validation and send, egress-policy host classification, PRIVATE_ONLY
# content fetching, search-result full-content fetching and the
@@ -236,8 +375,8 @@ def is_ip_blocked(
LLM backend reached via ``host.containers.internal``.
Note: cloud metadata endpoints in ``ALWAYS_BLOCKED_METADATA_IPS``
(AWS / Azure / GCP / OCI / Scaleway / Akamai / OpenStack /
- Hetzner / DigitalOcean / AlibabaCloud / Tencent) are ALWAYS
- blocked regardless of these flags.
+ Hetzner / DigitalOcean / AlibabaCloud / Tencent / Oracle
+ Compute Classic) are ALWAYS blocked regardless of these flags.
block_link_local: When True, the entire link-local range — IPv4
``169.254.0.0/16`` and IPv6 ``fe80::/10`` — is treated as blocked
EVEN under ``allow_private_ips=True``. Set by the notification
@@ -312,8 +451,10 @@ def is_ip_blocked(
# ALWAYS block cloud-metadata endpoints - critical SSRF target
# for credential theft (AWS IMDS/ECS/EKS, Azure IMDS/WireServer,
# GCP, OCI, Scaleway, Akamai, OpenStack/Hetzner, DigitalOcean,
- # AlibabaCloud, Tencent Cloud). These are never legitimate
- # destinations regardless of allow_localhost / allow_private_ips.
+ # AlibabaCloud, Tencent Cloud, Oracle Compute Classic). These are
+ # never legitimate destinations regardless of allow_localhost /
+ # allow_private_ips. Endpoints reached by NAME (GCP, IBM Cloud
+ # classic) are blocked by is_metadata_hostname, not here.
if str(ip) in ALWAYS_BLOCKED_METADATA_IPS:
return True
@@ -441,8 +582,11 @@ def validate_url(
1. URL scheme is allowed (http/https only)
2. The host is not percent-encoded (any ``%`` outside a bracketed IP
literal is refused, whatever the private-IP policy)
- 3. Hostname is not an internal/private IP address
- 4. Hostname does not resolve to an internal/private IP
+ 3. Hostname is not a cloud-metadata hostname
+ (``ALWAYS_BLOCKED_METADATA_HOSTNAMES``, under every flag)
+ 4. Hostname is not an internal/private IP address (the cloud-metadata
+ addresses in ``ALWAYS_BLOCKED_METADATA_IPS`` under every flag)
+ 5. Hostname does not resolve to an internal/private IP
Args:
url: URL to validate
@@ -457,8 +601,10 @@ def validate_url(
LLM backend reached via ``host.containers.internal``.
Note: cloud metadata endpoints in ``ALWAYS_BLOCKED_METADATA_IPS``
(AWS / Azure / GCP / OCI / Scaleway / Akamai / OpenStack /
- Hetzner / DigitalOcean / AlibabaCloud / Tencent) are ALWAYS
- blocked regardless of these flags.
+ Hetzner / DigitalOcean / AlibabaCloud / Tencent / Oracle
+ Compute Classic) and the hostnames in
+ ``ALWAYS_BLOCKED_METADATA_HOSTNAMES`` (GCP, IBM Cloud classic)
+ are ALWAYS blocked regardless of these flags.
Returns:
True if URL is safe, False otherwise
@@ -540,6 +686,16 @@ def validate_url(
)
return False
+ # Cloud-metadata endpoints reachable by NAME (GCP, IBM Cloud classic).
+ # Checked before the IP / DNS checks and independent of every flag:
+ # IBM's api.service.softlayer.com resolves to RFC1918, so the
+ # resolved-IP check below would pass it under allow_private_ips=True.
+ if is_metadata_hostname(hostname):
+ logger.warning(
+ f"Blocked URL to cloud-metadata hostname: {redact_url_for_log(url)}"
+ )
+ return False
+
# Check if hostname is an IP address
try:
ip = ipaddress.ip_address(hostname)
@@ -642,12 +798,15 @@ def assert_base_url_safe(base_url: str, *, setting_key: str) -> str:
``host-gateway`` and Docker Desktop's ``host.docker.internal`` are not
link-local.) Excluding link-local would break host-side Ollama /
LM Studio backends under Podman. The
- ``ALWAYS_BLOCKED_METADATA_IPS`` set still fires under those flags and
- prevents the auth-gated SSRF that would otherwise reach cloud-credential
- endpoints (AWS IMDS / ECS / EKS Pod Identity, Azure IMDS and WireServer,
- GCP, OCI, Scaleway, Akamai / Linode, OpenStack, DigitalOcean,
- AlibabaCloud, Tencent). See SECURITY.md ("LLM Provider URL
- Validation").
+ ``ALWAYS_BLOCKED_METADATA_IPS`` set and the
+ ``ALWAYS_BLOCKED_METADATA_HOSTNAMES`` name block (via ``validate_url``)
+ still fire under those flags and prevent the auth-gated SSRF that would
+ otherwise reach cloud-credential endpoints (AWS IMDS / ECS / EKS Pod
+ Identity, Azure IMDS and WireServer, GCP, OCI, Scaleway, Akamai /
+ Linode, OpenStack, DigitalOcean, AlibabaCloud, Tencent, Oracle Compute
+ Classic, and by name GCP and IBM Cloud classic). The IBM name block does
+ not cover another DNS name, or the literal RFC1918 address, that reaches
+ the same service. See SECURITY.md ("LLM Provider URL Validation").
Residual risk (documented, not closed here): this guard validates once
at provider construction, and the LLM SDK re-resolves the hostname on
diff --git a/src/local_deep_research/settings/env_definitions/security.py b/src/local_deep_research/settings/env_definitions/security.py
index 7f23e831a6..6de2a945d2 100644
--- a/src/local_deep_research/settings/env_definitions/security.py
+++ b/src/local_deep_research/settings/env_definitions/security.py
@@ -107,8 +107,9 @@ SECURITY_SETTINGS = [
"Docker deployments instead pin the URL "
"directly via LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_"
"INSTANCE_URL, which is trusted as operator-provisioned. "
- "Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS) stay "
- "blocked regardless of this flag."
+ "Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS and "
+ "ALWAYS_BLOCKED_METADATA_HOSTNAMES) stay blocked regardless of "
+ "this flag."
),
default=False,
),
@@ -130,8 +131,9 @@ SECURITY_SETTINGS = [
"path/query ignored); no wildcards or CIDR ranges. Entries that "
"fail to parse are ignored. Environment-only operator setting "
"so it cannot be edited through the user-writable settings API. "
- "Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS) and "
- "non-http(s) schemes stay blocked even if listed."
+ "Cloud-metadata endpoints (ALWAYS_BLOCKED_METADATA_IPS and "
+ "ALWAYS_BLOCKED_METADATA_HOSTNAMES) and non-http(s) schemes stay "
+ "blocked even if listed."
),
default=None,
),
@@ -155,8 +157,9 @@ SECURITY_SETTINGS = [
"Disabled by default and not set by the bundled "
"docker-compose.yml. Link-local addresses (169.254.0.0/16, "
"fe80::/10) and the cloud-metadata endpoints in "
- "ALWAYS_BLOCKED_METADATA_IPS stay blocked regardless of this "
- "flag. Takes effect only under the unprotected egress scope: "
+ "ALWAYS_BLOCKED_METADATA_IPS and "
+ "ALWAYS_BLOCKED_METADATA_HOSTNAMES stay blocked regardless of "
+ "this flag. Takes effect only under the unprotected egress scope: "
"under every other scope, the default adaptive scope included, "
"private result URLs stay blocked even with this flag set."
),
diff --git a/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py b/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py
index 6c36fdd965..7d0d898e6c 100644
--- a/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py
+++ b/src/local_deep_research/web_search_engines/engines/search_engine_elasticsearch.py
@@ -1,4 +1,5 @@
import json
+import re
import threading
import time as _time
from typing import Any, Dict, List, Optional
@@ -11,6 +12,12 @@ from ...security.secure_logging import logger
from ...settings.exceptions import SettingsValueError
from ..search_engine_base import BaseSearchEngine, Exposure, Sensitivity
+# A host name as the Elasticsearch SSRF check accepts it: dot-separated runs
+# of ASCII letters, digits, hyphens and underscores (compose service names
+# such as ``es01`` or ``elastic_search`` included). IP addresses are checked
+# separately.
+_HOST_NAME_RE = re.compile(r"[A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+)*")
+
class ElasticsearchSearchEngine(BaseSearchEngine):
"""Elasticsearch search engine implementation with two-phase approach"""
@@ -180,6 +187,165 @@ class ElasticsearchSearchEngine(BaseSearchEngine):
)
return True
+ @staticmethod
+ def _host_probe_target(host: Any) -> Optional[tuple[str, str, int]]:
+ """``(scheme, connection_host, port)`` for one ``hosts`` entry: the
+ destination the client connects to for it, which the SSRF check
+ validates and the availability probe connects to.
+
+ Read the way ``elasticsearch`` builds its node from the entry:
+
+ * a dictionary: its ``scheme``, ``host`` and ``port`` values, as
+ given. The client hands ``host`` to its connection pool unparsed
+ and uses an HTTPS pool only for the scheme ``"https"``, plain
+ HTTP for any other value. (It reads no ``hostname`` key; a
+ dictionary carrying one is refused by the client.)
+ * a URL string: parsed with urllib3's ``parse_url``, the parser the
+ client uses (``url_to_node_config``), with IPv6 brackets removed
+ as the client removes them. A string without ``://`` is read as
+ ``http://``.
+
+ A missing scheme reads as ``http``; a missing port as 443 for https
+ and 9200 otherwise (the client itself requires both in a
+ dictionary). Returns None -- the entry is refused -- when the scheme
+ is not ``http`` or ``https`` (letter case ignored), when a given
+ port is not an integer from 1 to 65535, or when there is no host
+ string (a bracketed dictionary host, which the client refuses,
+ included).
+ """
+ if isinstance(host, dict):
+ scheme = host.get("scheme", "http")
+ connection_host = host.get("host")
+ port = host.get("port")
+ if not isinstance(connection_host, str) or (
+ "[" in connection_host or "]" in connection_host
+ ):
+ return None
+ elif isinstance(host, str):
+ from urllib3.exceptions import LocationParseError
+ from urllib3.util import parse_url
+
+ url_str = host if "://" in host else f"http://{host}"
+ try:
+ parsed = parse_url(url_str)
+ except (LocationParseError, ValueError):
+ return None
+ scheme = parsed.scheme or "http"
+ connection_host = (parsed.host or "").strip("[]")
+ port = parsed.port
+ else:
+ return None
+
+ if not isinstance(scheme, str) or scheme.lower() not in (
+ "http",
+ "https",
+ ):
+ return None
+ scheme = scheme.lower()
+ if port is None:
+ port = 443 if scheme == "https" else 9200
+ if (
+ not isinstance(port, int)
+ or isinstance(port, bool)
+ or not 1 <= port <= 65535
+ ):
+ return None
+ if not connection_host:
+ return None
+ return scheme, connection_host, port
+
+ @staticmethod
+ def _target_passes_ssrf_check(
+ host: Any, target: tuple[str, str, int]
+ ) -> bool:
+ """Whether the destination ``target`` (``_host_probe_target`` of the
+ ``hosts`` entry ``host``) passes the SSRF check.
+
+ The check judges the connection host itself, never a URL built
+ from it: it must be ASCII and contain no ``%`` (the client's
+ connection pool percent-decodes and IDNA-encodes a host, so any
+ other text is not the name it resolves; this also refuses an IPv6
+ zone ID), whitespace or control characters. An IP address is
+ checked as such, and refused with a trailing dot (a resolver may
+ read that as a name instead). Anything else must be a plain host
+ name (ASCII letters, digits, hyphens, underscores and dots), not a
+ cloud-metadata name, not an IPv4 address in an integer / hex /
+ octal / short notation, and must resolve, with every address it
+ resolves to passing too. Localhost and private addresses are
+ allowed (a self-hosted cluster lives there, link-local included),
+ cloud-metadata names and addresses never are
+ (``is_ip_blocked(allow_localhost=True, allow_private_ips=True)``).
+ A URL string entry is also run through ``validate_url`` as written.
+ """
+ import ipaddress
+ import socket
+
+ from ...security.legacy_ipv4 import is_ambiguous_numeric_ipv4_host
+ from ...security.ssrf_validator import (
+ RFC_FORBIDDEN_URL_CHARS_RE,
+ is_ip_blocked,
+ is_metadata_hostname,
+ validate_url,
+ )
+
+ connection_host = target[1]
+ if (
+ not connection_host.isascii()
+ or "%" in connection_host
+ or RFC_FORBIDDEN_URL_CHARS_RE.search(connection_host)
+ ):
+ return False
+ bare = connection_host.rstrip(".")
+ if not bare or len(bare) > 253:
+ return False
+ try:
+ ipaddress.ip_address(bare)
+ except ValueError:
+ if (
+ not _HOST_NAME_RE.fullmatch(bare)
+ or is_metadata_hostname(bare)
+ or is_ambiguous_numeric_ipv4_host(bare)
+ ):
+ return False
+ try:
+ answers = socket.getaddrinfo(
+ connection_host,
+ None,
+ socket.AF_UNSPEC,
+ socket.SOCK_STREAM,
+ )
+ except (OSError, UnicodeError, ValueError):
+ return False
+ addresses = [str(answer[4][0]) for answer in answers]
+ else:
+ if bare != connection_host:
+ # An IP address with a trailing dot: the platform resolver
+ # may read it as the address or look it up as a name.
+ return False
+ addresses = [bare]
+ if not addresses or any(
+ is_ip_blocked(address, allow_localhost=True, allow_private_ips=True)
+ for address in addresses
+ ):
+ return False
+ if isinstance(host, str):
+ url_str = host if "://" in host else f"http://{host}"
+ if not validate_url(
+ url_str, allow_localhost=True, allow_private_ips=True
+ ):
+ return False
+ return True
+
+ @classmethod
+ def _host_passes_ssrf_check(cls, host: Any) -> bool:
+ """Whether one ``hosts`` entry passes the SSRF check the probe uses:
+ ``_target_passes_ssrf_check`` on its ``_host_probe_target``. An
+ entry that cannot be parsed fails."""
+ target = cls._host_probe_target(host)
+ return target is not None and cls._target_passes_ssrf_check(
+ host, target
+ )
+
@staticmethod
def _probe_hosts_available(hosts: List[Any]) -> bool:
"""TCP-connect to the first reachable host in ``hosts``.
@@ -189,11 +355,12 @@ class ElasticsearchSearchEngine(BaseSearchEngine):
Cheap (no HTTP) and short-timeout (1s per host, capped at a 2.0s
connection budget across all hosts). SSRF validation runs before each
connection and can perform DNS resolution that cannot be interrupted
- by this connection-only budget.
+ by this connection-only budget. Each entry is parsed once
+ (``_host_probe_target``); that one ``(scheme, host, port)`` is what
+ the SSRF check validates and what the probe connects to.
"""
import socket
- from urllib.parse import urlparse
- from ...security.ssrf_validator import validate_url
+ from ...security.ssrf_validator import redact_url_for_log
start_time = _time.monotonic()
total_budget = 2.0 # Cap aggregate probe duration across all hosts
@@ -204,40 +371,25 @@ class ElasticsearchSearchEngine(BaseSearchEngine):
if remaining <= 0:
break
- if isinstance(host, dict):
- hostname = host.get("host") or host.get("hostname")
- if not hostname:
- continue
- scheme = host.get("scheme", "http")
- port = host.get("port")
- if port is None:
- port = 443 if scheme == "https" else 9200
- elif isinstance(host, str):
- url_str = host if "://" in host else f"http://{host}"
- try:
- parsed = urlparse(url_str)
- hostname = parsed.hostname or host
- scheme = parsed.scheme or "http"
- port = parsed.port
- if port is None:
- port = 443 if scheme == "https" else 9200
- except ValueError:
- continue
- else:
+ target = ElasticsearchSearchEngine._host_probe_target(host)
+ if target is None:
+ logger.warning(
+ "Elasticsearch host entry has no usable http(s) "
+ "destination; skipping it"
+ )
continue
-
- connection_host = str(hostname).strip("[]")
- url_host = (
- f"[{connection_host}]"
- if ":" in connection_host
- else connection_host
- )
- probe_url = f"{scheme}://{url_host}:{port}"
- if not validate_url(
- probe_url, allow_localhost=True, allow_private_ips=True
+ scheme, connection_host, port = target
+ if not ElasticsearchSearchEngine._target_passes_ssrf_check(
+ host, target
):
+ url_host = (
+ f"[{connection_host}]"
+ if ":" in connection_host
+ else connection_host
+ )
logger.warning(
- "Elasticsearch host failed SSRF validation: {}", probe_url
+ "Elasticsearch host failed SSRF validation: {}",
+ redact_url_for_log(f"{scheme}://{url_host}:{port}"),
)
continue
@@ -308,6 +460,25 @@ class ElasticsearchSearchEngine(BaseSearchEngine):
# Normalize hosts – may arrive as a JSON-encoded string from settings
hosts = self._ensure_list(hosts, default=["http://localhost:9200"])
+ # SSRF check on every node before the client is built: the client
+ # connects in client.info() below without any other validation, and
+ # the egress policy's engine gate is lifted under the unprotected
+ # scope. Each entry is judged as the destination the client connects
+ # to (_host_probe_target: its scheme, host and port, never a URL
+ # rebuilt from them). Localhost and private addresses pass; an
+ # entry whose scheme is not http(s), whose port is not a valid
+ # integer, whose host does not resolve or is not a plain name or
+ # address, and every cloud-metadata name or address are refused.
+ for host in hosts:
+ if not self._host_passes_ssrf_check(host):
+ logger.bind(policy_audit=True).warning(
+ "refusing Elasticsearch host that failed SSRF validation"
+ )
+ raise ValueError(
+ "Elasticsearch host failed SSRF validation; refusing to "
+ "connect. Check the configured Elasticsearch hosts."
+ )
+
# Initialize the Elasticsearch client
es_args: Dict[str, Any] = {}
diff --git a/tests/security/test_egress_elasticsearch_host_shapes.py b/tests/security/test_egress_elasticsearch_host_shapes.py
index eb801eb340..867668a92f 100644
--- a/tests/security/test_egress_elasticsearch_host_shapes.py
+++ b/tests/security/test_egress_elasticsearch_host_shapes.py
@@ -463,9 +463,6 @@ HOST_IN_URL_AND_DICTIONARY = [
pytest.param("Kibana.Corp.Example", LOCAL, id="mixed_case"),
# Punycode is ASCII: classified as the client connects to it.
pytest.param("xn--fa-hia.corp.example", LOCAL, id="punycode"),
- # The Kelvin sign lower-cases to an ASCII "k" for the resolver and the SDK
- # alike.
- pytest.param("Kibana.corp.example", LOCAL, id="kelvin_sign"),
pytest.param("faß.corp.example", UNKNOWN, id="sharp_s"),
pytest.param("straße.example", UNKNOWN, id="strasse"),
pytest.param("münchen.example", UNKNOWN, id="encodings_agree"),
@@ -490,6 +487,33 @@ def test_dictionary_host_is_classified_like_the_same_host_in_a_url(
) == expected, entry
+def test_kelvin_sign_host_is_classified_by_its_shape():
+ """U+212A KELVIN SIGN lower-cases to an ASCII "k". A URL host is
+ lower-cased by the parser before classification (``urlsplit`` and
+ urllib3's ``parse_url`` both read ``kibana.corp.example``, the name the
+ client connects to), so the URL is classified as that ASCII name. A
+ dictionary host is classified as written, and the canonical-host rule
+ checks for ASCII before case folding, so the raw non-ASCII spelling is
+ unclassified. This is the one host on which the two shapes differ, and
+ the dictionary shape errs closed: unknown is refused under every
+ protected scope, and the engine's own connection check refuses the
+ same non-ASCII dictionary host."""
+ host = "\u212aibana.corp.example"
+ assert sdk_connects_to(host) == "kibana.corp.example"
+
+ for entry, expected in (
+ (dictionary_host(host), UNKNOWN),
+ (f"http://{host}:9200", LOCAL),
+ ):
+ classification, _, _ = classify_private_names(
+ [entry], EgressScope.PRIVATE_ONLY
+ )
+ assert (
+ classification.is_public,
+ classification.is_local,
+ ) == expected, entry
+
+
def test_punycode_dictionary_host_is_classified_as_the_client_connects():
"""Writing the host in its ``xn--`` form is the way to keep an
internationalized name classified."""
diff --git a/tests/security/test_egress_host_canonical_classification.py b/tests/security/test_egress_host_canonical_classification.py
new file mode 100644
index 0000000000..c9acd812ab
--- /dev/null
+++ b/tests/security/test_egress_host_canonical_classification.py
@@ -0,0 +1,1014 @@
+"""Host classification accepts only canonical host spellings (#6869 follow-up).
+
+Host classification -- the require-local LLM / embeddings checks, the
+Elasticsearch / Paperless endpoint fail-up, the ``llm.allowed_local_hostnames``
+override and that setting's save-time validator -- reads a host through ONE
+canonicaliser, ``policy._canonical_host``: a host containing both ``:`` and
+``%`` as written (an IPv6 zone ID, encoded or not) is refused before any
+decoding; any other host is percent-decoded exactly once, case folded, one
+trailing root dot dropped. The result is classified only if it is a plain DNS
+name or an IP literal ``ipaddress`` parses as written. Every other spelling,
+and every cloud-metadata name or address (wrapped in an IPv6 form or not), is
+unclassified: never local, never eligible as a public endpoint, and ignored
+when listed in ``llm.allowed_local_hostnames``.
+
+The earlier design enumerated metadata spellings instead, and each review
+round found one more that slipped through (most recently a zone ID encoded
+twice, which callers decoded once and the metadata check decoded again). The
+property tests below run a generated corpus of spellings through every
+classification entry point instead of listing cases one by one:
+
+* no non-canonical spelling is ever classified local or eligible as public;
+* no canonical metadata name or address (or an IPv6 wrap of one) is;
+* canonical private / public names and addresses keep their classification
+ (regression control), whichever canonical-preserving spelling is used.
+
+``evaluate_url`` (the fetch-time check) keeps its broader normalisation; the
+corpus also pins that it refuses every spelling of a metadata address a
+single-decoding resolver or HTTP client would connect to.
+"""
+
+import ipaddress
+import socket
+from collections import namedtuple
+from unittest.mock import patch
+from urllib.parse import urlsplit
+
+import pytest
+
+from local_deep_research.security.egress import validators
+from local_deep_research.security.egress.policy import (
+ EgressContext,
+ EgressScope,
+ _canonical_host,
+ _classify_host,
+ evaluate_embeddings,
+ evaluate_engine,
+ evaluate_llm_endpoint,
+ evaluate_url,
+ host_not_classified_public,
+ url_host_is_declared_local,
+)
+from local_deep_research.security.ssrf_validator import (
+ ALWAYS_BLOCKED_METADATA_HOSTNAMES,
+ ALWAYS_BLOCKED_METADATA_IPS,
+)
+
+# ---------------------------------------------------------------------------
+# Corpus generator (pure: standard library only)
+# ---------------------------------------------------------------------------
+
+# host: the spelling as urlsplit / a configured value would carry it.
+# canonical: what _canonical_host must return for it (None: unclassified
+# spelling). An IPv6 wrap of an IPv4 address is a canonical literal.
+# metadata: the destination is a cloud-metadata endpoint.
+# fetch_metadata: evaluate_url must refuse it as blocked_metadata_ip, because
+# a single-decoding resolver or HTTP client reaches a metadata endpoint.
+Spelling = namedtuple(
+ "Spelling", "host base canonical metadata fetch_metadata transform"
+)
+
+# Non-metadata bases. Their canonical spellings are the regression controls.
+CONTROL_IPV4 = ["10.0.0.42", "192.168.1.5", "127.0.0.1", "93.184.216.34"]
+CONTROL_IPV6 = ["fd12::1", "::1", "2606:4700:4700::1111"]
+CONTROL_NAMES = ["gpu-box.lan", "search.example.com", "ollama_service"]
+
+_FULLWIDTH = {
+ **{c: chr(ord(c) - ord("0") + 0xFF10) for c in "0123456789"},
+ **{
+ c: chr(ord(c) - ord("a") + 0xFF41) for c in "abcdefghijklmnopqrstuvwxyz"
+ },
+ **{
+ c: chr(ord(c) - ord("A") + 0xFF21) for c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+ },
+ ":": ":",
+}
+
+
+def _fullwidth(text):
+ return "".join(_FULLWIDTH.get(c, c) for c in text)
+
+
+def _percent_all(text):
+ return "".join(f"%{b:02x}" for b in text.encode())
+
+
+# Control and whitespace characters a configured value or a parser
+# differential could carry into a host.
+_CONTROL_SUFFIXES = ["\x00", "\t", "\n", "\x7f", "\x1f"]
+
+# IPv6 zone identifiers, as written in the host: numeric, hex-looking,
+# alphabetic, the RFC 6874 "%25" form, and twice-encoded forms whose single
+# decode leaves a residual "%" (a second decode would turn "%10" into a
+# control character, the round-3 case). Also zone IDs that are two hex digits
+# or contain an encoded colon, whose single decode splices them into a
+# different, valid address ("fd00:ec2::254%31" decodes to fd00:ec2::2541,
+# "%41" to a trailing "A", "%3a1" to a further ":1" group), while an HTTP
+# client reads them as the zone of the address written before the "%".
+_ZONES = [
+ "%1",
+ "%10",
+ "%a",
+ "%eth0",
+ "%25eth0",
+ "%2510",
+ "%251",
+ "%252510",
+ "%31",
+ "%41",
+ "%3a1",
+ "%3A1",
+]
+
+
+def _ipv6_from_bytes(data):
+ return str(ipaddress.IPv6Address(bytes(data)))
+
+
+def _ipv6_wraps(v4):
+ """Canonical IPv6 literals that carry ``v4`` where an IPv4-embedding
+ mechanism puts the address it reaches."""
+ o = list(ipaddress.IPv4Address(v4).packed)
+ hi, lo = (o[0] << 8) | o[1], (o[2] << 8) | o[3]
+ inverted = [b ^ 0xFF for b in o]
+ return {
+ "mapped": f"::ffff:{v4}",
+ "mapped_hex": f"::ffff:{hi:x}:{lo:x}",
+ "nat64_wkp": f"64:ff9b::{v4}",
+ # RFC 6052 /48 layout: octets 6-7, then 9-10 around the zero octet 8.
+ "nat64_local_use": _ipv6_from_bytes(
+ [0, 0x64, 0xFF, 0x9B, 0, 1, o[0], o[1], 0, o[2], o[3]] + [0] * 5
+ ),
+ # A network-specific NAT64 prefix (RFC 6052 section 2.2) in every
+ # layout: /96 and /64, and /56, /40 and /32, where the address is
+ # split around the zero octet 8 or sits in octets 4-7.
+ "nat64_nsp_96": f"2001:db8:64::{v4}",
+ "nat64_nsp_64": _ipv6_from_bytes(
+ [0x20, 0x01, 0x0D, 0xB8, 0, 0x64, 0, 0, 0] + o + [0, 0, 0]
+ ),
+ "nat64_nsp_56": _ipv6_from_bytes(
+ [0x20, 0x01, 0x0D, 0xB8, 0, 0x64, 0x01, o[0], 0] + o[1:] + [0] * 4
+ ),
+ "nat64_nsp_40": _ipv6_from_bytes(
+ [0x20, 0x01, 0x0D, 0xB8, 0x64] + o[:3] + [0, o[3]] + [0] * 6
+ ),
+ "nat64_nsp_32": _ipv6_from_bytes(
+ [0x20, 0x01, 0x0D, 0xB8] + o + [0] * 8
+ ),
+ "compatible": f"::{v4}",
+ "translated": f"::ffff:0:{v4}",
+ "6to4": _ipv6_from_bytes([0x20, 0x02] + o + [0] * 9 + [1]),
+ # Teredo: server IPv4 in octets 4-7, client IPv4 inverted in 12-15.
+ "teredo_client": _ipv6_from_bytes(
+ [0x20, 0x01, 0, 0, 203, 0, 113, 1, 0x80, 0, 0xFF, 0xFF] + inverted
+ ),
+ "teredo_server": _ipv6_from_bytes(
+ [0x20, 0x01, 0, 0]
+ + o
+ + [0x80, 0, 0xFF, 0xFF, 0x3F, 0xFF, 0xFF, 0xFE]
+ ),
+ "isatap": f"fe80::5efe:{v4}",
+ }
+
+
+def _ipv4_spellings(v4, metadata):
+ o = list(ipaddress.IPv4Address(v4).packed)
+ n = int(ipaddress.IPv4Address(v4))
+ preserving = {
+ "as_is": v4,
+ "upper": v4.upper(),
+ "trailing_dot": v4 + ".",
+ "pct_dots": v4.replace(".", "%2e"),
+ "pct_dots_upper": v4.replace(".", "%2E"),
+ "pct_all": _percent_all(v4),
+ }
+ # Spellings a resolver may still read as ``v4`` (evaluate_url refuses a
+ # metadata one), but which classification refuses to interpret.
+ resolver_forms = {
+ "integer": str(n),
+ "integer_trailing_dot": f"{n}.",
+ "hex": f"0x{n:08x}",
+ "hex_upper": f"0X{n:08X}",
+ "octal_dotted": ".".join(f"0{b:o}" for b in o),
+ "hex_dotted": ".".join(f"0x{b:x}" for b in o),
+ "short_3": f"{o[0]}.{o[1]}.{(o[2] << 8) | o[3]}",
+ "short_2": f"{o[0]}.{(o[1] << 16) | (o[2] << 8) | o[3]}",
+ "double_trailing_dot": v4 + "..",
+ "fullwidth": _fullwidth(v4),
+ "ideographic_dots": v4.replace(".", "。"),
+ "fullwidth_dots": v4.replace(".", "."),
+ "trailing_ideographic_dot": v4 + "。",
+ "pct_hex": _percent_all(f"0x{n:08x}"),
+ }
+ # Spellings no resolver reads as ``v4``.
+ other_forms = {
+ # Zero-padded octets read as octal, so not as ``v4``.
+ "padded": ".".join(f"{b:04d}" for b in o),
+ "double_pct": v4.replace(".", "%252e"),
+ "bracketed": f"[{v4}]",
+ "zone_rfc6874": v4 + "%25eth0",
+ "zone_raw": v4 + "%eth0",
+ "pct_nul": v4 + "%00",
+ "leading_space": " " + v4,
+ }
+ for i, suffix in enumerate(_CONTROL_SUFFIXES):
+ other_forms[f"control_{i}"] = v4 + suffix
+ out = [
+ Spelling(host, v4, v4, metadata, metadata, f"v4:{name}")
+ for name, host in preserving.items()
+ ]
+ out += [
+ Spelling(host, v4, None, metadata, metadata, f"v4:{name}")
+ for name, host in resolver_forms.items()
+ ]
+ out += [
+ Spelling(host, v4, None, metadata, False, f"v4:{name}")
+ for name, host in other_forms.items()
+ ]
+ if metadata:
+ for name, wrap in _ipv6_wraps(v4).items():
+ canonical = str(ipaddress.IPv6Address(wrap))
+ for host in (wrap, f"[{wrap.upper()}]"):
+ out.append(
+ Spelling(host, v4, canonical, True, True, f"wrap:{name}")
+ )
+ return out
+
+
+def _ipv6_spellings(v6, metadata):
+ ip = ipaddress.IPv6Address(v6)
+ canonical = str(ip)
+ preserving = {
+ "as_is": v6,
+ "bracketed": f"[{v6}]",
+ "upper": v6.upper(),
+ "exploded": ip.exploded,
+ "bracketed_exploded_upper": f"[{ip.exploded.upper()}]",
+ "pct_colons": v6.replace(":", "%3a"),
+ }
+ zoned = {f"zone{zone}": v6 + zone for zone in _ZONES}
+ zoned["bracketed_zone"] = f"[{v6}%25eth0]"
+ zoned["bracketed_raw_zone"] = f"[{v6}%10]"
+ zoned["exploded_zone"] = ip.exploded.upper() + "%1"
+ other_forms = {
+ "trailing_dot": v6 + ".",
+ "bracketed_trailing_dot": f"[{v6}].",
+ "double_pct_colons": v6.replace(":", "%253a"),
+ "fullwidth": _fullwidth(v6),
+ "leading_space": " " + v6,
+ }
+ for i, suffix in enumerate(_CONTROL_SUFFIXES):
+ other_forms[f"control_{i}"] = v6 + suffix
+ out = [
+ Spelling(host, v6, canonical, metadata, metadata, f"v6:{name}")
+ for name, host in preserving.items()
+ ]
+ # A zone-scoped literal reaches the address on that interface; the
+ # fetch check refuses a metadata one, classification refuses them all.
+ out += [
+ Spelling(host, v6, None, metadata, metadata, f"v6:{name}")
+ for name, host in zoned.items()
+ ]
+ out += [
+ Spelling(host, v6, None, metadata, False, f"v6:{name}")
+ for name, host in other_forms.items()
+ ]
+ return out
+
+
+def _name_spellings(name, metadata):
+ preserving = {
+ "as_is": name,
+ "upper": name.upper(),
+ "title": name.title(),
+ "trailing_dot": name + ".",
+ "pct_dots": name.replace(".", "%2e"),
+ "pct_first": f"%{ord(name[0]):02x}" + name[1:],
+ }
+ # Spellings the resolver's IDNA encoding folds back to ``name``.
+ idna_forms = {
+ "fullwidth": _fullwidth(name),
+ "ideographic_dots": name.replace(".", "。"),
+ "fullwidth_dots": name.replace(".", "."),
+ "halfwidth_dots": name.replace(".", "。"),
+ "trailing_ideographic_dot": name + "。",
+ "soft_hyphen": name[:2] + "­" + name[2:],
+ "double_trailing_dot": name + "..",
+ }
+ other_forms = {
+ "double_pct": name.replace(".", "%252e"),
+ "pct_nul": name + "%00",
+ "zone": name + "%25eth0",
+ "bracketed": f"[{name}]",
+ "leading_space": " " + name,
+ "non_ascii_suffix": name + "é",
+ }
+ for i, suffix in enumerate(_CONTROL_SUFFIXES):
+ other_forms[f"control_{i}"] = name + suffix
+ # A dot-replacing transform is a no-op on a single-label name.
+ idna_forms = {k: v for k, v in idna_forms.items() if v != name}
+ other_forms = {k: v for k, v in other_forms.items() if v != name}
+ out = [
+ Spelling(host, name, name, metadata, metadata, f"name:{label}")
+ for label, host in preserving.items()
+ ]
+ out += [
+ Spelling(host, name, None, metadata, metadata, f"name:{label}")
+ for label, host in idna_forms.items()
+ ]
+ out += [
+ Spelling(host, name, None, metadata, False, f"name:{label}")
+ for label, host in other_forms.items()
+ ]
+ return out
+
+
+def spelling_corpus(metadata_ips, metadata_names):
+ """Every spelling of every metadata and control base."""
+ corpus = []
+ for address in sorted(metadata_ips):
+ if ipaddress.ip_address(address).version == 4:
+ corpus += _ipv4_spellings(address, True)
+ else:
+ corpus += _ipv6_spellings(address, True)
+ for name in sorted(metadata_names):
+ corpus += _name_spellings(name, True)
+ for address in CONTROL_IPV4:
+ corpus += _ipv4_spellings(address, False)
+ for address in CONTROL_IPV6:
+ corpus += _ipv6_spellings(address, False)
+ for name in CONTROL_NAMES:
+ corpus += _name_spellings(name, False)
+ return corpus
+
+
+def url_for(host):
+ """An http URL whose urlsplit hostname is ``host`` (case folded), or
+ None when the spelling cannot round-trip through a URL (urlsplit drops
+ tabs and newlines, and rejects some bracketed forms)."""
+ authority = host
+ if ":" in host and not host.startswith("["):
+ authority = f"[{host}]"
+ url = f"http://{authority}:8000/v1"
+ try:
+ parsed = urlsplit(url).hostname
+ except ValueError:
+ return None
+ expected = authority.lower()
+ if expected.startswith("[") and expected.endswith("]"):
+ expected = expected[1:-1]
+ return url if parsed == expected else None
+
+
+CORPUS = spelling_corpus(
+ ALWAYS_BLOCKED_METADATA_IPS, ALWAYS_BLOCKED_METADATA_HOSTNAMES
+)
+UNCLASSIFIED = [s for s in CORPUS if s.canonical is None or s.metadata]
+CANONICAL_CONTROLS = [s for s in CORPUS if s.canonical and not s.metadata]
+
+# ---------------------------------------------------------------------------
+# Fixtures and helpers
+# ---------------------------------------------------------------------------
+
+PRIVATE_ANSWER = [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("10.0.0.42", 0))]
+PUBLIC_ANSWER = [
+ (socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))
+]
+PROTECTED_SCOPES = [
+ EgressScope.PUBLIC_ONLY,
+ EgressScope.BOTH,
+ EgressScope.PRIVATE_ONLY,
+ EgressScope.STRICT,
+]
+ENGINE_SETTINGS = {
+ "paperless": "search.engine.web.paperless.default_params.api_url",
+ "elasticsearch": "search.engine.web.elasticsearch.default_params.hosts",
+}
+LOCAL_HOSTNAMES_KEY = "llm.allowed_local_hostnames"
+
+
[email protected](autouse=True)
+def _canonical_corpus_without_active_context():
+ """The fetch-check loops below make thousands of evaluate_url calls.
+ Its denial quota follows the thread's active run context when one is
+ armed, so make sure none leaks in from elsewhere; the loops also pass
+ record_denial=False and a fresh context per call."""
+ from local_deep_research.security.egress.audit_hook import (
+ clear_active_context,
+ )
+
+ clear_active_context()
+ yield
+ clear_active_context()
+
+
+def _resolver(answer):
+ """A resolver stub that answers ``answer`` for any name and, as
+ getaddrinfo does, an IP literal with itself."""
+
+ def resolve(host):
+ try:
+ address = ipaddress.ip_address(host)
+ except ValueError:
+ return answer
+ family = socket.AF_INET6 if address.version == 6 else socket.AF_INET
+ return [(family, socket.SOCK_STREAM, 6, "", (str(address), 0))]
+
+ return resolve
+
+
+def _dns(answer):
+ return patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ side_effect=_resolver(answer),
+ )
+
+
+def _ctx(scope=EgressScope.PRIVATE_ONLY, *, local=(), engine="paperless"):
+ return EgressContext(
+ scope=scope,
+ primary_engine=engine,
+ require_local_llm=True,
+ require_local_embeddings=True,
+ local_hostnames=tuple(local),
+ )
+
+
+def _engine_values(engine, spelling):
+ """The setting values that configure ``engine`` with this host."""
+ values = []
+ url = url_for(spelling.host)
+ if url is not None:
+ values.append(url if engine == "paperless" else [url])
+ if engine == "elasticsearch":
+ values.append([{"host": spelling.host, "port": 9200, "scheme": "http"}])
+ return values
+
+
+def _expected_control(base, answer):
+ """Main's classification of a canonical control base."""
+ try:
+ address = ipaddress.ip_address(base)
+ except ValueError:
+ return answer is PRIVATE_ANSWER # a name: its DNS answer decides
+ return address.is_private or address.is_loopback
+
+
+# ---------------------------------------------------------------------------
+# Corpus properties
+# ---------------------------------------------------------------------------
+
+
+def test_corpus_covers_every_kind_of_spelling():
+ """Sanity: the generator produced each family the properties rely on."""
+ kinds = {s.transform.split(":")[1] for s in CORPUS}
+ for family in (
+ "double_pct",
+ "integer",
+ "hex",
+ "octal_dotted",
+ "short_3",
+ "fullwidth",
+ "ideographic_dots",
+ "zone%2510",
+ "zone%eth0",
+ "zone%31",
+ "zone%3a1",
+ "trailing_dot",
+ "bracketed",
+ "mapped",
+ "nat64_wkp",
+ "nat64_nsp_96",
+ "nat64_nsp_64",
+ "nat64_nsp_56",
+ "nat64_nsp_40",
+ "nat64_nsp_32",
+ "compatible",
+ "6to4",
+ "teredo_client",
+ "control_0",
+ ):
+ assert family in kinds, family
+ assert len(UNCLASSIFIED) > 400
+ assert len(CANONICAL_CONTROLS) > 50
+
+
+def test_corpus_canonical_form_is_exactly_the_expected_one():
+ """_canonical_host accepts only plain DNS names and IP literals that
+ ipaddress parses as written, after one percent-decode, case folding and
+ one trailing dot; it returns None for every other spelling."""
+ wrong = [
+ (s.transform, s.host, _canonical_host(s.host), s.canonical)
+ for s in CORPUS
+ if _canonical_host(s.host) != s.canonical
+ ]
+ assert not wrong
+
+
[email protected]("answer", [PRIVATE_ANSWER, PUBLIC_ANSWER])
+def test_corpus_unclassified_host_is_never_local_even_when_listed(answer):
+ """A non-canonical spelling or a metadata host is not local -- also when
+ it is itself listed in llm.allowed_local_hostnames, and whatever DNS
+ answers -- and is never declared local at the DNS-pinning step."""
+ local = []
+ with _dns(answer):
+ for s in UNCLASSIFIED:
+ ctx = _ctx(local=[s.host])
+ if _classify_host(s.host, ctx) is not False:
+ local.append((s.transform, s.host, "classify_host"))
+ url = url_for(s.host)
+ if url is not None and url_host_is_declared_local(url, ctx):
+ local.append((s.transform, s.host, "declared_local"))
+ assert not local
+
+
[email protected]("answer", [PRIVATE_ANSWER, PUBLIC_ANSWER])
[email protected]("engine", list(ENGINE_SETTINGS))
+def test_corpus_unclassified_engine_endpoint_is_refused_in_every_scope(
+ engine, answer
+):
+ """An Elasticsearch or Paperless endpoint with such a host is neither
+ local nor failed up to public: evaluate_engine refuses it as
+ unclassified under every protected scope. The private answer is what
+ would make it local; the public answer is what would fail it up."""
+ admitted = []
+ with _dns(answer):
+ for s in UNCLASSIFIED:
+ for value in _engine_values(engine, s):
+ for scope in PROTECTED_SCOPES:
+ decision = evaluate_engine(
+ engine,
+ _ctx(scope, local=[s.host], engine=engine),
+ settings_snapshot={ENGINE_SETTINGS[engine]: value},
+ )
+ if decision.allowed or decision.reason != "unclassified":
+ admitted.append(
+ (s.transform, s.host, scope.value, decision.reason)
+ )
+ assert not admitted
+
+
+def test_corpus_require_local_inference_refuses_unclassified_host():
+ """require-local LLM and embeddings checks never accept such a host as
+ local, even listed in llm.allowed_local_hostnames with a private DNS
+ answer."""
+ accepted = []
+ with _dns(PRIVATE_ANSWER):
+ for s in UNCLASSIFIED:
+ url = url_for(s.host)
+ if url is None:
+ continue
+ ctx = _ctx(local=[s.host])
+ llm = evaluate_llm_endpoint(
+ "ollama", ctx, settings_snapshot={"llm.ollama.url": url}
+ )
+ embeddings = evaluate_embeddings(
+ "openai",
+ ctx,
+ settings_snapshot={"embeddings.openai.base_url": url},
+ )
+ ollama_embeddings = evaluate_embeddings(
+ "ollama",
+ ctx,
+ settings_snapshot={"embeddings.ollama.url": url},
+ )
+ for check, decision in (
+ ("llm", llm),
+ ("embeddings_openai", embeddings),
+ ("embeddings_ollama", ollama_embeddings),
+ ):
+ if decision.allowed:
+ accepted.append((s.transform, s.host, check))
+ assert not accepted
+
+
+def test_corpus_local_hostnames_validator_refuses_unclassified_entry():
+ """The save-time validator refuses every such entry, never raises, and
+ calls an entry a cloud-metadata endpoint only when it is written in a
+ canonical spelling without ``%``; a non-canonical or percent-encoded
+ spelling is reported as not a valid hostname for the list."""
+ wrong = []
+ with (
+ _dns(PRIVATE_ANSWER),
+ patch.object(
+ validators,
+ "_resolve_with_timeout",
+ side_effect=_resolver(PRIVATE_ANSWER),
+ ),
+ ):
+ for s in UNCLASSIFIED:
+ result = validators.validate_allowed_local_hostnames(
+ {LOCAL_HOSTNAMES_KEY: [s.host]}, {}
+ )
+ error = (result or {}).get("error", "")
+ if s.canonical is None or "%" in s.host:
+ ok = (
+ "not valid hostnames" in error
+ and "cloud-metadata" not in error
+ )
+ else:
+ ok = "cloud-metadata" in error and "not valid" not in error
+ if not ok:
+ wrong.append((s.transform, s.host, error))
+ assert not wrong
+
+
[email protected]("answer", [PRIVATE_ANSWER, PUBLIC_ANSWER])
+def test_corpus_canonical_controls_keep_their_classification(answer):
+ """Regression control: a canonical private / public name or address, in
+ any canonical-preserving spelling (case, one trailing dot, single
+ percent-encoding, IPv6 brackets or exploded form), classifies as before
+ -- local iff private (a name: iff its DNS answer is), and a Paperless
+ endpoint on it is local or fails up to public accordingly."""
+ wrong = []
+ with _dns(answer):
+ for s in CANONICAL_CONTROLS:
+ expected = _expected_control(s.base, answer)
+ got = _classify_host(s.host, _ctx())
+ if got is not expected:
+ wrong.append((s.transform, s.host, "classify_host", got))
+ url = url_for(s.host)
+ if url is None:
+ continue
+ setting = {ENGINE_SETTINGS["paperless"]: url}
+ private = evaluate_engine(
+ "paperless",
+ _ctx(EgressScope.PRIVATE_ONLY),
+ settings_snapshot=setting,
+ )
+ public = evaluate_engine(
+ "paperless",
+ _ctx(EgressScope.PUBLIC_ONLY),
+ settings_snapshot=setting,
+ )
+ if (private.allowed, public.allowed) != (expected, not expected):
+ wrong.append((s.transform, s.host, "paperless", expected))
+ llm = evaluate_llm_endpoint(
+ "ollama", _ctx(), settings_snapshot={"llm.ollama.url": url}
+ )
+ if llm.allowed is not expected:
+ wrong.append((s.transform, s.host, "llm", llm.reason))
+ assert not wrong
+
+
+def test_corpus_fetch_check_refuses_every_reachable_metadata_spelling():
+ """evaluate_url keeps its broader normalisation: every spelling a
+ single-decoding resolver or client connects to a metadata endpoint with
+ -- canonical, alternate IPv4 notation, IDNA-folded, zone-scoped, IPv6
+ wrapped -- is refused as blocked_metadata_ip in every scope, before any
+ scope or local_hostnames logic. The resolver answers public, so no other
+ denial reason can stand in for it."""
+ allowed = []
+ with _dns(PUBLIC_ANSWER):
+ for s in CORPUS:
+ url = url_for(s.host)
+ if not s.fetch_metadata or url is None:
+ continue
+ for scope in EgressScope:
+ decision = evaluate_url(
+ url, _ctx(scope, local=[s.host]), record_denial=False
+ )
+ if decision.reason != "blocked_metadata_ip":
+ allowed.append((s.transform, s.host, scope.value))
+ assert not allowed
+
+
[email protected]("answer", [PRIVATE_ANSWER, PUBLIC_ANSWER])
+def test_corpus_uncounted_prefilter_agrees_with_the_fetch_check(answer):
+ """host_not_classified_public (full_search's uncounted pre-filter) is
+ True exactly when evaluate_url under public_only refuses the URL as
+ scope_mismatch_public_only, for every spelling in the corpus, listed in
+ local_hostnames or not; anything evaluate_url refuses for another
+ reason first (metadata, malformed host) is left to it."""
+ disagree = []
+ with _dns(answer):
+ for s in CORPUS:
+ url = url_for(s.host)
+ if url is None:
+ continue
+ for local in ((), (s.host,)):
+ refused = host_not_classified_public(url, _ctx(local=local))
+ decision = evaluate_url(
+ url,
+ _ctx(EgressScope.PUBLIC_ONLY, local=local),
+ record_denial=False,
+ )
+ expected = decision.reason == "scope_mismatch_public_only"
+ if refused is not expected:
+ disagree.append((s.transform, s.host, decision.reason))
+ assert not disagree
+
+
+# ---------------------------------------------------------------------------
+# Targeted: the round-3 zone case
+# ---------------------------------------------------------------------------
+#
+# ``fd00:ec2::254%2510`` is the AWS IPv6 metadata address with its zone ID
+# encoded twice. Callers used to percent-decode the host and the metadata
+# check decoded it again, so it saw "fd00:ec2::254" followed by a control
+# character, did not recognise it, and the host fell through to the
+# address checks (which read the zone after ONE decode and call the address
+# not local). The engine fail-up then made Paperless eligible as public, and
+# a listed local_hostnames entry made it local.
+
+ROUND3_URLS = [
+ "http://[fd00:ec2::254%2510]:8000",
+ "http://[fe80::a9fe:a9fe%2510]:8000",
+]
+
+
[email protected]("scope", PROTECTED_SCOPES)
[email protected]("url", ROUND3_URLS)
+def test_round3_double_encoded_zone_paperless_is_refused(url, scope):
+ with _dns(PUBLIC_ANSWER):
+ decision = evaluate_engine(
+ "paperless",
+ _ctx(scope),
+ settings_snapshot={ENGINE_SETTINGS["paperless"]: url},
+ )
+ assert not decision.allowed
+ assert decision.reason == "unclassified"
+
+
[email protected]("scope", PROTECTED_SCOPES)
+ "hosts",
+ [
+ [f"{ROUND3_URLS[0]}"],
+ f"http://10.0.0.42:9200,{ROUND3_URLS[0]}",
+ [{"host": "fd00:ec2::254%10", "port": 9200, "scheme": "http"}],
+ [{"host": "fd00:ec2::254%2510", "port": 9200, "scheme": "http"}],
+ ],
+)
+def test_round3_double_encoded_zone_elasticsearch_is_refused(hosts, scope):
+ with _dns(PUBLIC_ANSWER):
+ decision = evaluate_engine(
+ "elasticsearch",
+ _ctx(scope, engine="elasticsearch"),
+ settings_snapshot={ENGINE_SETTINGS["elasticsearch"]: hosts},
+ )
+ assert not decision.allowed
+ assert decision.reason == "unclassified"
+
+
+ "entry", ["fd00:ec2::254%10", "fe80::a9fe:a9fe%10", "fd12::1%10"]
+)
+def test_round3_zone_scoped_local_hostnames_entry_is_ignored(entry):
+ """A zone-scoped entry (metadata or not) is not a canonical spelling, so
+ it is ignored: the host it names is not local, and a require-local LLM
+ endpoint on it is refused even with a private DNS answer."""
+ url = f"http://[{entry.replace('%', '%25')}]:11434"
+ ctx = _ctx(local=[entry])
+ with _dns(PRIVATE_ANSWER):
+ assert _classify_host(entry, ctx) is False
+ assert _classify_host(urlsplit(url).hostname, ctx) is False
+ decision = evaluate_llm_endpoint(
+ "ollama", ctx, settings_snapshot={"llm.ollama.url": url}
+ )
+ assert not decision.allowed
+ assert decision.reason == "provider_remote"
+
+
+ "host",
+ ["fd00:ec2::254%31", "fd12::1%31", "fd12::1%41", "[fd12::1%3a1]"],
+)
+def test_encoded_zone_id_is_not_read_as_part_of_the_address(host):
+ """``%`` is both the IPv6 zone delimiter and the escape character. An
+ HTTP client reads ``fd12::1%31`` as fd12::1 on interface "31", while
+ one decode reads the different address fd12::11 (and
+ ``fd00:ec2::254%31``, the AWS metadata address on an interface, as
+ fd00:ec2::2541). Such a host has no canonical spelling: it is not
+ local, and a require-local LLM endpoint on it is refused even with a
+ private DNS answer."""
+ assert _canonical_host(host) is None
+ url = f"http://[{host.strip('[]')}]:11434"
+ with _dns(PRIVATE_ANSWER):
+ assert _classify_host(host, _ctx()) is False
+ decision = evaluate_llm_endpoint(
+ "ollama", _ctx(), settings_snapshot={"llm.ollama.url": url}
+ )
+ assert not decision.allowed
+ assert decision.reason == "provider_remote"
+
+
+def test_round3_zone_scoped_entry_is_rejected_at_save_without_metadata_claim():
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=PRIVATE_ANSWER
+ ):
+ result = validators.validate_allowed_local_hostnames(
+ {LOCAL_HOSTNAMES_KEY: ["fd12::1%10"]}, {}
+ )
+ assert result is not None
+ assert "not valid hostnames" in result["error"]
+ assert "metadata" not in result["error"]
+
+
+# ---------------------------------------------------------------------------
+# The fetch check (evaluate_url) is never looser than before
+# ---------------------------------------------------------------------------
+
+
[email protected]("scope", list(EgressScope))
+ "host", ["169%252e254%252e169%252e254", "fd00%253aec2%253a%253a254"]
+)
+def test_fetch_check_refuses_a_metadata_address_encoded_twice(host, scope):
+ """No HTTP client decodes twice, but a host that only reaches a metadata
+ address after a second decode is never legitimate; the broad fetch check
+ refuses it as one in every scope."""
+ url = f"http://{host}/latest"
+ with _dns(PRIVATE_ANSWER):
+ decision = evaluate_url(url, _ctx(scope, local=[host]))
+ assert decision.reason == "blocked_metadata_ip"
+
+
[email protected]("scope", list(EgressScope))
+ "host",
+ [
+ "api%25252eservice%25252esoftlayer%25252ecom",
+ "metadata%25252egoogle%25252einternal",
+ ],
+)
+def test_fetch_check_refuses_a_metadata_name_encoded_three_times(host, scope):
+ """Every spelling the fetch check derives, the host decoded until stable
+ included, is matched against the metadata hostnames, not only the
+ once-decoded one. The private DNS answer would otherwise admit the host
+ under PRIVATE_ONLY and STRICT."""
+ url = f"http://{host}/latest"
+ with _dns(PRIVATE_ANSWER):
+ decision = evaluate_url(url, _ctx(scope, local=[host]))
+ assert decision.reason == "blocked_metadata_ip"
+
+
[email protected]("scope", list(EgressScope))
+ "host",
+ [
+ "0.0.0.0",
+ "0%2e0%2e0%2e0",
+ "[::]",
+ # 6to4 and Teredo addresses carrying 127.0.0.1.
+ "[2002:7f00:1::]",
+ "[2001:0:4136:e378:8000:63bf:80ff:fffe]",
+ ],
+)
+def test_fetch_check_refuses_other_always_blocked_addresses(host, scope):
+ """Not metadata, but in the SSRF validator's blocked ranges and not one
+ of the private ranges it can allow: refused by the is_ip_blocked check
+ over every spelling, under the same reason, before classification (which
+ would call them public)."""
+ with _dns(PUBLIC_ANSWER):
+ decision = evaluate_url(f"http://{host}:8000/", _ctx(scope))
+ assert decision.reason == "blocked_metadata_ip"
+
+
[email protected]("scope", list(EgressScope))
+ "host", ["gpu-box.lan%00", "gpu-box.lan%09", "gpu-box.lan%0a"]
+)
+def test_fetch_check_refuses_a_host_with_control_characters(host, scope):
+ """A host that decodes to a control character reaches nothing (the
+ resolver refuses an embedded NUL); it is refused as malformed in every
+ scope instead of being classified, and is not counted as a policy
+ denial."""
+ with _dns(PRIVATE_ANSWER):
+ decision = evaluate_url(f"http://{host}/", _ctx(scope))
+ assert not decision.allowed
+ assert decision.reason == "url_malformed"
+
+
+ "scope", [EgressScope.PUBLIC_ONLY, EgressScope.BOTH, EgressScope.STRICT]
+)
[email protected]("host", ["fd12::1%2d", "fd12::1%10"])
+def test_fetch_check_zone_id_read_two_ways_is_not_admitted(host, scope):
+ """An HTTP client reads ``fd12::1%2d`` as the private fd12::1 on
+ interface "2d", while the once-decoded text ``fd12::1-`` is no address
+ and classifies public after a lookup. The two readings disagree, so the
+ fetch check leaves the host unclassified rather than admit it under
+ PUBLIC_ONLY. (``fd12::1%10`` decodes to a control character and is
+ refused as malformed.)"""
+ with _dns(PUBLIC_ANSWER):
+ decision = evaluate_url(f"http://[{host}]/", _ctx(scope))
+ assert not decision.allowed
+
+
+ "entry", ["0x0a00002a", "b\u00fccherei.lan", "fd12::1%eth0"]
+)
+ "scope",
+ [EgressScope.PUBLIC_ONLY, EgressScope.PRIVATE_ONLY, EgressScope.STRICT],
+)
+def test_fetch_check_ignored_local_hostnames_entry_cannot_admit_its_host(
+ entry, scope
+):
+ """A non-canonical llm.allowed_local_hostnames entry is ignored, but a
+ URL host matching it exactly must not become admissible because of
+ that: it is unclassified at fetch time (it used to be local), so with a
+ public DNS answer it is still refused under PUBLIC_ONLY, and it is not
+ admitted as local under PRIVATE_ONLY / STRICT."""
+ url = url_for(entry)
+ assert url is not None
+ with _dns(PUBLIC_ANSWER):
+ decision = evaluate_url(url, _ctx(scope, local=[entry]))
+ assert not decision.allowed
+
+
+# ---------------------------------------------------------------------------
+# Declared local hostnames are compared in canonical form
+# ---------------------------------------------------------------------------
+
+
+ "entry", ["GPU-Box.LAN", "gpu-box.lan.", "gpu%2dbox.lan"]
+)
+ "url", ["http://gpu-box.lan:11434", "http://GPU-BOX.lan.:11434"]
+)
+def test_declared_local_hostname_matches_in_canonical_form(entry, url):
+ """An entry and a URL host that are the same canonical name match
+ whatever their case or trailing dot, so the declaration takes effect
+ (and holds at the DNS-pinning step) even though DNS answers public."""
+ ctx = _ctx(local=[entry])
+ with _dns(PUBLIC_ANSWER):
+ decision = evaluate_llm_endpoint(
+ "ollama", ctx, settings_snapshot={"llm.ollama.url": url}
+ )
+ assert decision.allowed, decision.reason
+ assert url_host_is_declared_local(url, ctx)
+
+
+ "host", ["\u212a8s.lan", "%E2%84%AA8s.lan", "%e2%84%aa8s.lan"]
+)
+def test_non_ascii_letter_that_lowercases_to_ascii_is_unclassified(host):
+ """U+212A KELVIN SIGN lower-cases to an ASCII "k". The ASCII test runs on
+ the decoded host before case folding, so this is not read as the
+ declared ``k8s.lan``, and such an entry is refused at save."""
+ assert _canonical_host(host) is None
+ with _dns(PRIVATE_ANSWER):
+ assert _classify_host(host, _ctx(local=["k8s.lan"])) is False
+ assert _classify_host(host, _ctx(local=[host])) is False
+ with patch.object(
+ validators,
+ "_resolve_with_timeout",
+ side_effect=_resolver(PRIVATE_ANSWER),
+ ):
+ result = validators.validate_allowed_local_hostnames(
+ {LOCAL_HOSTNAMES_KEY: [host]}, {}
+ )
+ assert result is not None
+ assert "not valid hostnames" in result["error"]
+
+
+ "entry", ["gpu%2dbox.lan", "10%2e0%2e0%2e42", "%67pu-box.lan", "fd12::1%25"]
+)
+def test_local_hostnames_validator_refuses_a_percent_encoded_entry(entry):
+ """A new entry with ``%`` is refused at save even when one decode would
+ make it a plain private name: no hostname needs percent-encoding, and
+ the saved text then always is the name it matches."""
+ with patch.object(
+ validators,
+ "_resolve_with_timeout",
+ side_effect=_resolver(PRIVATE_ANSWER),
+ ):
+ result = validators.validate_allowed_local_hostnames(
+ {LOCAL_HOSTNAMES_KEY: [entry]}, {}
+ )
+ assert result is not None
+ assert "not valid hostnames" in result["error"]
+
+
+# ---------------------------------------------------------------------------
+# The settings validator never raises
+# ---------------------------------------------------------------------------
+
+
+ "value",
+ [
+ ["a\x00b"],
+ ["\x00"],
+ ["%00"],
+ ["169.254.169.254\x00"],
+ ["0x" + "f" * 4096],
+ ["a" * 10000],
+ ["[" * 50],
+ ["%"],
+ [12, None, {"host": "x"}, ["nested"]],
+ '["\\u0000"]',
+ ],
+)
+def test_local_hostnames_validator_never_raises(value):
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=PRIVATE_ANSWER
+ ):
+ result = validators.validate_allowed_local_hostnames(
+ {LOCAL_HOSTNAMES_KEY: value}, {}
+ )
+ assert result is None or set(result) == {"key", "error"}
diff --git a/tests/security/test_egress_settings_save_validators.py b/tests/security/test_egress_settings_save_validators.py
index ea26729407..8f4473b89f 100644
--- a/tests/security/test_egress_settings_save_validators.py
+++ b/tests/security/test_egress_settings_save_validators.py
@@ -145,6 +145,70 @@ def test_hosts_key_absent_returns_none():
assert validate_allowed_local_hostnames({"other.key": "x"}, {}) is None
[email protected]("all_db_settings", [{}, None])
+def test_missing_form_data_returns_none(all_db_settings):
+ """No form data means nothing to validate; the validator must not raise
+ (it promises never to)."""
+ assert validate_allowed_local_hostnames(None, all_db_settings) is None
+
+
+# An entry list that fails validation in every way it can: a zone-scoped
+# address, a percent-encoded name and a public address.
+_STALE_ENTRIES = ["fd12::1%10", "gpu%2dbox.lan", "8.8.8.8"]
+
+
+ "stored",
+ [
+ {"value": list(_STALE_ENTRIES)},
+ MagicMock(value=list(_STALE_ENTRIES)),
+ {"value": '["fd12::1%10", "gpu%2dbox.lan", "8.8.8.8"]'},
+ ],
+)
+ "submitted",
+ [list(_STALE_ENTRIES), '["fd12::1%10", "gpu%2dbox.lan", "8.8.8.8"]'],
+)
+def test_unchanged_list_is_not_revalidated(stored, submitted):
+ """Validate-on-change, as for the engine URLs: the All-Settings tab
+ resubmits every setting, so a stored list holding entries a stricter
+ validator now refuses must not block an unrelated save. A list and its
+ JSON text are the same value."""
+ assert (
+ validate_allowed_local_hostnames(
+ {HOSTS_KEY: submitted}, {HOSTS_KEY: stored}
+ )
+ is None
+ )
+
+
+ "submitted",
+ [
+ _STALE_ENTRIES + ["10.0.0.1"],
+ _STALE_ENTRIES[:-1],
+ list(reversed(_STALE_ENTRIES)),
+ ],
+)
+def test_changed_list_is_validated_in_full(submitted):
+ """Deny counterpart: any change to the stored list (an added, removed
+ or reordered entry) is validated in full, so the unchanged stale entries
+ are reported too."""
+ err = validate_allowed_local_hostnames(
+ {HOSTS_KEY: submitted}, {HOSTS_KEY: {"value": list(_STALE_ENTRIES)}}
+ )
+ assert err is not None
+ assert "not valid hostnames" in err["error"]
+
+
+def test_list_without_stored_value_is_validated():
+ """A key with no stored value counts as changed."""
+ err = validate_allowed_local_hostnames(
+ {HOSTS_KEY: list(_STALE_ENTRIES)}, {"other.key": {"value": "x"}}
+ )
+ assert err is not None
+
+
# ---------------------------------------------------------------------------
# validate_engine_instance_urls
# (a PUBLIC engine's user-editable URL may not point at a private host)
diff --git a/tests/security/test_metadata_hostname_block.py b/tests/security/test_metadata_hostname_block.py
new file mode 100644
index 0000000000..2cef032a8d
--- /dev/null
+++ b/tests/security/test_metadata_hostname_block.py
@@ -0,0 +1,1199 @@
+"""Cloud-metadata HOSTNAME block (issue #6869).
+
+IBM Cloud classic infrastructure serves its instance metadata
+(``SoftLayer_Resource_Metadata``) from ``api.service.softlayer.com``, which
+resolves to ``10.0.80.88`` -- an RFC1918 address. Every IP-based check
+therefore passes it whenever private addresses are allowed
+(``allow_private_ips=True``, the LLM base-URL path, the notification
+plugin partition, STRICT / PRIVATE_ONLY egress). The name is blocked instead,
+from the single list ``ssrf_validator.ALWAYS_BLOCKED_METADATA_HOSTNAMES``,
+at every entry point that classifies a destination host.
+
+Each test resolves the name to the REAL IBM address (10.0.80.88) and uses the
+most permissive flags, so only the name block can refuse it: removing the
+IBM entry from the list, or dropping the ``is_metadata_hostname`` call at the
+entry point under test, makes the test fail.
+
+The controls pin the deliberate trade-off: 10.0.80.88 itself stays allowed
+under ``allow_private_ips=True`` (a LAN service may legitimately live there),
+and neighbouring SoftLayer names are not over-blocked. The name block does
+NOT stop a different DNS name that resolves to 10.0.80.88; that limit is
+documented next to the list.
+"""
+
+import ipaddress
+import socket
+from unittest.mock import patch
+
+import pytest
+
+from local_deep_research.security.egress import validators
+from local_deep_research.security.egress.policy import (
+ EgressContext,
+ EgressScope,
+ _classify_host,
+ _is_metadata_destination,
+ evaluate_engine,
+ evaluate_llm_endpoint,
+ evaluate_url,
+)
+from local_deep_research.security.notification_validator import (
+ NotificationURLValidator,
+)
+from local_deep_research.security.ssrf_validator import (
+ ALWAYS_BLOCKED_METADATA_HOSTNAMES,
+ ALWAYS_BLOCKED_METADATA_IPS,
+ assert_base_url_safe,
+ is_metadata_hostname,
+ validate_url,
+)
+from local_deep_research.web_search_engines.engines import (
+ search_engine_elasticsearch as es_engine,
+)
+
+IBM_HOST = "api.service.softlayer.com"
+IBM_IP = "10.0.80.88"
+IBM_ADDRINFO = [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (IBM_IP, 0))]
+IBM_RESOLVED = [ipaddress.IPv4Address(IBM_IP)]
+
+# Spellings the resolver / HTTP client treat as the same host.
+IBM_SPELLINGS = [
+ IBM_HOST,
+ "API.SERVICE.SOFTLAYER.COM",
+ "Api.Service.SoftLayer.Com",
+ f"{IBM_HOST}.",
+ f"{IBM_HOST}..",
+ "API.Service.SoftLayer.com.",
+ "api%2eservice%2esoftlayer%2ecom",
+]
+
+# Nearby names that are NOT the metadata endpoint. Resolved to the same
+# RFC1918 address in the controls, so only the exact-name match separates
+# them from IBM_HOST.
+NEARBY_NON_METADATA = [
+ "api.softlayer.com",
+ "service.softlayer.com",
+ "softlayer.com",
+ "xapi.service.softlayer.com",
+ f"{IBM_HOST}.attacker.example",
+ "api-service.softlayer.com",
+]
+
+
+def _ssrf_dns(addrinfo):
+ return patch(
+ "local_deep_research.security.ssrf_validator.socket.getaddrinfo",
+ return_value=addrinfo,
+ )
+
+
+# ---------------------------------------------------------------------------
+# Single source of truth
+# ---------------------------------------------------------------------------
+
+
+def test_ibm_classic_metadata_hostname_is_listed():
+ assert IBM_HOST in ALWAYS_BLOCKED_METADATA_HOSTNAMES
+
+
+def test_ibm_rfc1918_address_is_not_always_blocked():
+ """The trade-off: 10.0.80.88 is RFC1918 and must NOT be an absolute
+ block, or a user's own LAN service at that address would break.
+ (161.26.0.6 is IBM's package mirror / NTP, not metadata, and must not be
+ listed either.)"""
+ assert IBM_IP not in ALWAYS_BLOCKED_METADATA_IPS
+ assert "161.26.0.6" not in ALWAYS_BLOCKED_METADATA_IPS
+
+
[email protected]("host", IBM_SPELLINGS)
+def test_is_metadata_hostname_normalizes_case_and_trailing_dot(host):
+ assert is_metadata_hostname(host) is True
+
+
[email protected]("host", NEARBY_NON_METADATA)
+def test_is_metadata_hostname_is_exact_match(host):
+ assert is_metadata_hostname(host) is False
+
+
+# ---------------------------------------------------------------------------
+# validate_url / assert_base_url_safe
+# ---------------------------------------------------------------------------
+
+
[email protected]("host", IBM_SPELLINGS)
+ "flags",
+ [
+ {},
+ {"allow_localhost": True},
+ {"allow_private_ips": True},
+ {"allow_localhost": True, "allow_private_ips": True},
+ ],
+)
+def test_validate_url_blocks_ibm_metadata_hostname_under_all_flags(host, flags):
+ with _ssrf_dns(IBM_ADDRINFO):
+ assert (
+ validate_url(
+ f"https://{host}/rest/v3.1/SoftLayer_Resource_Metadata/getId",
+ **flags,
+ )
+ is False
+ )
+
+
+def test_assert_base_url_safe_refuses_ibm_metadata_hostname():
+ """The LLM base-URL path uses allow_private_ips=True, so the RFC1918
+ answer alone would pass; only the name block refuses it."""
+ with _ssrf_dns(IBM_ADDRINFO):
+ with pytest.raises(ValueError):
+ assert_base_url_safe(
+ f"https://{IBM_HOST}./v1", setting_key="llm.openai.url"
+ )
+
+
[email protected]("host", NEARBY_NON_METADATA)
+def test_validate_url_allows_nearby_hostname_on_same_address(host):
+ """CONTROL: same RFC1918 answer, different name -> allowed under
+ allow_private_ips=True. Fails if the block is widened to a suffix /
+ substring match or to the IP itself."""
+ with _ssrf_dns(IBM_ADDRINFO):
+ assert validate_url(f"https://{host}/", allow_private_ips=True) is True
+
+
+def test_validate_url_allows_lan_service_at_ibm_address():
+ """CONTROL: a LAN service on 10.0.80.88 keeps working under
+ allow_private_ips=True (the documented reason the IP is not blocked)."""
+ assert validate_url(f"http://{IBM_IP}:11434/", allow_private_ips=True)
+
+
+# ---------------------------------------------------------------------------
+# Egress policy
+# ---------------------------------------------------------------------------
+
+
+def _ctx(scope, local_hostnames=()):
+ return EgressContext(
+ scope=scope,
+ primary_engine="test",
+ require_local_llm=True,
+ require_local_embeddings=True,
+ local_hostnames=tuple(local_hostnames),
+ )
+
+
[email protected]("scope", list(EgressScope))
[email protected]("host", IBM_SPELLINGS)
+def test_evaluate_url_blocks_ibm_metadata_hostname_in_every_scope(scope, host):
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ d = evaluate_url(f"https://{host}/", _ctx(scope))
+ assert not d.allowed
+ assert d.reason == "blocked_metadata_ip"
+
+
+def test_evaluate_url_allows_nearby_hostname_under_private_only():
+ """CONTROL: a nearby name on the same RFC1918 answer is local and allowed
+ under PRIVATE_ONLY."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ d = evaluate_url(
+ "https://api.softlayer.com/", _ctx(EgressScope.PRIVATE_ONLY)
+ )
+ assert d.allowed
+
+
+def test_classify_host_never_calls_ibm_metadata_hostname_local():
+ """Reached by evaluate_llm_endpoint / evaluate_embeddings / engines,
+ which do not go through evaluate_url. The RFC1918 answer would classify
+ it local; it must not -- even if the user listed it as a local host."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ ctx = _ctx(EgressScope.PRIVATE_ONLY, local_hostnames=[IBM_HOST])
+ assert _classify_host(IBM_HOST, ctx) is False
+ assert _classify_host("API.Service.SoftLayer.com.", ctx) is False
+ # CONTROL: a nearby name on the same answer is still local.
+ assert _classify_host("api.softlayer.com", ctx) is True
+
+
+# ---------------------------------------------------------------------------
+# Notification validator
+# ---------------------------------------------------------------------------
+
+
+ "url",
+ [
+ f"https://{IBM_HOST}/hook",
+ "https://API.Service.SoftLayer.com./hook",
+ f"json://{IBM_HOST}/hook",
+ f"ntfy://{IBM_HOST}./topic",
+ ],
+)
+def test_notification_validator_blocks_ibm_metadata_hostname(url):
+ with patch.object(
+ NotificationURLValidator,
+ "_resolve_hostname_ips",
+ return_value=IBM_RESOLVED,
+ ):
+ ok, err = NotificationURLValidator.validate_service_url(
+ url, allow_private_ips=True
+ )
+ assert ok is False
+ assert err
+
+
+ "url", ["https://api.softlayer.com/hook", "json://api.softlayer.com/hook"]
+)
+def test_notification_validator_allows_nearby_hostname(url):
+ """CONTROL: same RFC1918 answer, different name, private IPs allowed."""
+ with patch.object(
+ NotificationURLValidator,
+ "_resolve_hostname_ips",
+ return_value=IBM_RESOLVED,
+ ):
+ ok, err = NotificationURLValidator.validate_service_url(
+ url, allow_private_ips=True
+ )
+ assert ok is True, err
+
+
+# ---------------------------------------------------------------------------
+# Search-engine URL settings validator
+# ---------------------------------------------------------------------------
+
+
+def test_engine_url_validator_refuses_ibm_metadata_hostname_with_opt_in():
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=IBM_ADDRINFO
+ ):
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ f"https://{IBM_HOST}./",
+ allow_private=True,
+ )
+ assert err is not None
+ assert "cloud-metadata" in err["error"]
+
+
+def test_engine_url_validator_allows_nearby_hostname_with_opt_in():
+ """CONTROL."""
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=IBM_ADDRINFO
+ ):
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ "https://api.softlayer.com/",
+ allow_private=True,
+ )
+ assert err is None
+
+
+def test_policy_keeps_no_private_copy_of_the_hostname_list():
+ """Single source of truth: the egress policy must consult
+ ssrf_validator.is_metadata_hostname, not a private list that can drift
+ (the pre-#6869 ``_METADATA_HOSTNAMES`` held only the GCP names)."""
+ from local_deep_research.security.egress import policy
+
+ assert not hasattr(policy, "_METADATA_HOSTNAMES")
+
+
+# ---------------------------------------------------------------------------
+# IDNA look-alike spellings
+#
+# getaddrinfo IDNA-encodes a non-ASCII str host (stdlib nameprep) and httpx /
+# requests use the idna package (UTS #46). Both fold the ideographic /
+# fullwidth / halfwidth full stops to "." and fullwidth letters to ASCII, so
+# these spellings connect to the IBM metadata host.
+# ---------------------------------------------------------------------------
+
+IDNA_SPELLINGS = [
+ "api。service。softlayer。com", # U+3002 ideographic stop
+ "api.service.softlayer.com", # U+FF0E fullwidth stop
+ "api。service。softlayer。com", # U+FF61 halfwidth stop
+ "api.service.softlayer.com", # fullwidth "api"
+ # fullwidth upper-case letters and stops throughout
+ "API.SERVICE.SOFTLAYER.COM",
+ "api.service.softlayer.com。", # trailing ideographic stop
+ "api.servi­ce.softlayer.com", # soft hyphen is dropped
+]
+
+# Non-ASCII names that must stay allowed: a genuine IDN, and a fullwidth
+# spelling that folds to a NEARBY (non-metadata) name.
+NON_ASCII_CONTROLS = [
+ "bücher.example",
+ "api.softlayer.com", # folds to api.softlayer.com
+ "api。softlayer。com",
+]
+
+
[email protected]("host", IDNA_SPELLINGS)
+def test_is_metadata_hostname_folds_idna_spellings(host):
+ assert is_metadata_hostname(host) is True
+
+
[email protected]("host", NON_ASCII_CONTROLS)
+def test_is_metadata_hostname_allows_non_ascii_controls(host):
+ """CONTROL: IDNA folding must not turn into a blanket non-ASCII block."""
+ assert is_metadata_hostname(host) is False
+
+
+def test_is_metadata_hostname_fails_closed_on_unencodable_name():
+ """A non-ASCII name the stdlib IDNA codec cannot encode (label > 63
+ octets) is treated as a match: the check must not fail open."""
+ assert is_metadata_hostname("a" * 64 + "ü.example") is True
+
+
[email protected]("scope", list(EgressScope))
[email protected]("host", IDNA_SPELLINGS)
+def test_evaluate_url_blocks_idna_spellings_in_every_scope(scope, host):
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ d = evaluate_url(f"https://{host}/", _ctx(scope))
+ assert not d.allowed
+ assert d.reason == "blocked_metadata_ip"
+
+
[email protected]("host", NON_ASCII_CONTROLS)
+def test_evaluate_url_allows_non_ascii_controls_under_private_only(host):
+ """CONTROL: same RFC1918 answer, non-metadata non-ASCII name."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ d = evaluate_url(f"https://{host}/", _ctx(EgressScope.PRIVATE_ONLY))
+ assert d.allowed, d.reason
+
+
[email protected]("host", IDNA_SPELLINGS)
+def test_classify_host_never_calls_idna_spelling_local(host):
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ assert _classify_host(host, _ctx(EgressScope.PRIVATE_ONLY)) is False
+
+
+def test_classify_host_does_not_call_a_non_ascii_name_local():
+ """Changed by the canonical-host rule: host classification accepts only
+ plain ASCII DNS names and canonical IP literals, so a non-ASCII name is
+ not local even when its IDNA folding is an ordinary name on a private
+ address (it used to be classified local here). Writing the name in its
+ ASCII / ``xn--`` form keeps it classifiable, as the control shows. The
+ fetch-time check keeps classifying such a name by DNS
+ (``test_evaluate_url_allows_non_ascii_controls_under_private_only``)."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ ctx = _ctx(EgressScope.PRIVATE_ONLY)
+ assert _classify_host("api.softlayer.com", ctx) is False
+ # CONTROL: the same name written in ASCII is still local.
+ assert _classify_host("api.softlayer.com", ctx) is True
+
+
[email protected]("host", IDNA_SPELLINGS)
+def test_engine_url_validator_refuses_idna_spellings_with_opt_in(host):
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=IBM_ADDRINFO
+ ):
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ f"https://{host}/",
+ allow_private=True,
+ )
+ assert err is not None
+ assert "cloud-metadata" in err["error"]
+
+
+def test_engine_url_validator_allows_non_ascii_control_with_opt_in():
+ """CONTROL."""
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=IBM_ADDRINFO
+ ):
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ "https://api.softlayer.com/",
+ allow_private=True,
+ )
+ assert err is None
+
+
+UNENCODABLE_IDN = "a" * 64 + "ü.example"
+
+
+def test_engine_url_validator_reports_unencodable_idn_as_invalid():
+ """A non-ASCII name the IDNA codec cannot encode is refused (the
+ metadata-name check fails closed on it) but reported as an invalid
+ hostname, not as a cloud-metadata address."""
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ f"https://{UNENCODABLE_IDN}/",
+ allow_private=True,
+ )
+ assert err is not None
+ assert "invalid or unsupported hostname" in err["error"]
+ assert "cloud-metadata" not in err["error"]
+
+
+def test_local_hostnames_validator_reports_unencodable_idn_as_invalid():
+ """Any non-ASCII entry is refused as an invalid spelling for the list
+ (classification would ignore it), not as a cloud-metadata endpoint."""
+ result = validators.validate_allowed_local_hostnames(
+ {"llm.allowed_local_hostnames": [UNENCODABLE_IDN]}, {}
+ )
+ assert result is not None
+ assert "not valid hostnames" in result["error"]
+ assert "cloud-metadata" not in result["error"]
+
+
+# A public name the stdlib IDNA 2003 codec rejects but IDNA 2008 accepts: a
+# right-to-left label ending in a digit. urllib3 and httpx encode it with the
+# idna package (IDNA 2008) and connect to ``xn--1-9hcuf1d.co.il``, so it is
+# judged by that form instead of being refused as an unencodable name.
+IDNA2008_ONLY_NAME = "\u05e9\u05dc\u05d5\u05dd1.co.il"
+
+# A metadata name with a character UTS #46 ignores but the IDNA 2003 codec
+# rejects (U+2063 INVISIBLE SEPARATOR): only the UTS #46 form is the listed
+# name, so it is matched there rather than through the fail-closed path.
+UTS46_ONLY_METADATA_SPELLING = "metadata.goo\u2063g"
+
+
+def test_idna2008_only_name_is_not_a_metadata_name():
+ assert is_metadata_hostname(IDNA2008_ONLY_NAME) is False
+
+
+ "scope",
+ [EgressScope.PUBLIC_ONLY, EgressScope.BOTH, EgressScope.UNPROTECTED],
+)
+def test_evaluate_url_admits_idna2008_only_public_name(scope):
+ """evaluate_url used to refuse such a name as blocked_metadata_ip in
+ every scope while validate_url (which sees urllib3's IDNA 2008 form)
+ admitted it. With a public DNS answer it is now an ordinary public
+ host."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=PUBLIC_ADDRINFO,
+ ):
+ d = evaluate_url(f"https://{IDNA2008_ONLY_NAME}/", _ctx(scope))
+ assert d.allowed, d.reason
+ with _ssrf_dns(PUBLIC_ADDRINFO):
+ assert validate_url(f"https://{IDNA2008_ONLY_NAME}/") is True
+
+
+def test_engine_url_validator_accepts_idna2008_only_public_name():
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=PUBLIC_ADDRINFO
+ ):
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ f"https://{IDNA2008_ONLY_NAME}/",
+ allow_private=False,
+ )
+ assert err is None
+
+
[email protected]("scope", list(EgressScope))
+def test_metadata_spelling_only_uts46_folds_is_still_refused(scope):
+ """A name is still refused when only one encoder's form is a listed
+ metadata name, and the engine URL validator reports it as a metadata
+ endpoint (an encoder does accept it), not as an unencodable name."""
+ assert is_metadata_hostname(UTS46_ONLY_METADATA_SPELLING) is True
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=PUBLIC_ADDRINFO,
+ ):
+ d = evaluate_url(
+ f"https://{UTS46_ONLY_METADATA_SPELLING}/", _ctx(scope)
+ )
+ assert d.reason == "blocked_metadata_ip"
+ err = validators._engine_url_ssrf_error(
+ "search.engine.web.searxng.default_params.instance_url",
+ f"https://{UTS46_ONLY_METADATA_SPELLING}/",
+ allow_private=True,
+ )
+ assert err is not None
+ assert "cloud-metadata" in err["error"]
+
+
[email protected]("host", IDNA_SPELLINGS)
+def test_validate_url_refuses_idna_spellings(host):
+ """validate_url / assert_base_url_safe refuse these through urllib3's
+ host parsing and the ASCII-only authority guard; pinned here so a
+ loosening of either is caught at this entry point too."""
+ with _ssrf_dns(IBM_ADDRINFO):
+ assert validate_url(f"https://{host}/", allow_private_ips=True) is False
+
+
[email protected]("host", IDNA_SPELLINGS)
+def test_notification_validator_refuses_idna_spellings(host):
+ """Held by the ASCII-only authority guard; pinned for the same reason."""
+ for url in (f"https://{host}/hook", f"json://{host}/hook"):
+ with patch.object(
+ NotificationURLValidator,
+ "_resolve_hostname_ips",
+ return_value=IBM_RESOLVED,
+ ):
+ ok, _ = NotificationURLValidator.validate_service_url(
+ url, allow_private_ips=True
+ )
+ assert ok is False, url
+
+
+def test_evaluate_url_name_denial_survives_ip_helper_exception():
+ """The metadata-NAME denial sits outside the try/except that guards the
+ IP helpers, so an exception there cannot swallow it. Under PUBLIC_ONLY
+ the name otherwise classifies public and would be ALLOWED."""
+ with (
+ patch(
+ "local_deep_research.security.ssrf_validator.is_ip_blocked",
+ side_effect=RuntimeError("boom"),
+ ),
+ patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ),
+ ):
+ d = evaluate_url(f"https://{IBM_HOST}/", _ctx(EgressScope.PUBLIC_ONLY))
+ assert not d.allowed
+ assert d.reason == "blocked_metadata_ip"
+
+
+# ---------------------------------------------------------------------------
+# Elasticsearch engine: scope gate and constructor
+# ---------------------------------------------------------------------------
+#
+# _classify_host reports a metadata host as NOT local (False). The engine URL
+# fail-up used to read that False as "public" and turn the local-nature
+# Elasticsearch engine public, so PUBLIC_ONLY (and BOTH) admitted it. The
+# fail-up now leaves unclassified every host _classifiable_host refuses (a
+# metadata name or address, or any non-canonical spelling), and the
+# constructor validates every node before the client connects.
+
+ES_HOSTS_SETTING = "search.engine.web.elasticsearch.default_params.hosts"
+PUBLIC_IP = "93.184.216.34"
+PUBLIC_ADDRINFO = [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (PUBLIC_IP, 0))]
+
+# Every shape the hosts setting takes: a list of URLs, a comma-separated
+# string (split into nodes by the engine's _ensure_list), and a host
+# dictionary. A private node sits next to the metadata one in the string
+# form so the metadata node is not the only entry.
+IBM_ES_HOSTS = [
+ [f"http://{IBM_HOST}:9200"],
+ f"http://10.0.0.42:9200,http://{IBM_HOST}:9200",
+ [{"host": IBM_HOST, "port": 9200, "scheme": "http"}],
+]
+
+METADATA_IP_ES_HOSTS = [
+ ["http://169.254.169.254:9200"],
+ "http://10.0.0.42:9200,http://169.254.169.254:9200",
+ ["http://[fd00:ec2::254]:9200"],
+ [{"host": "169.254.169.254", "port": 9200, "scheme": "http"}],
+ [{"host": "::ffff:169.254.169.254", "port": 9200, "scheme": "http"}],
+]
+
+# PUBLIC_ONLY and BOTH admitted the engine before the fix (it failed up to
+# public); PRIVATE_ONLY and STRICT would admit it if the name block in
+# _classify_host were lost (the IBM name resolves to RFC1918).
+ES_PROTECTED_SCOPES = [
+ EgressScope.PUBLIC_ONLY,
+ EgressScope.BOTH,
+ EgressScope.PRIVATE_ONLY,
+ EgressScope.STRICT,
+]
+
+
+def _es_ctx(scope):
+ return EgressContext(
+ scope=scope,
+ primary_engine="elasticsearch",
+ require_local_llm=False,
+ require_local_embeddings=False,
+ )
+
+
+def _evaluate_es(hosts, scope, addrinfo):
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=addrinfo,
+ ):
+ return evaluate_engine(
+ "elasticsearch",
+ _es_ctx(scope),
+ settings_snapshot={ES_HOSTS_SETTING: hosts},
+ )
+
+
[email protected]("scope", ES_PROTECTED_SCOPES)
[email protected]("hosts", IBM_ES_HOSTS)
+def test_evaluate_engine_denies_elasticsearch_on_ibm_metadata_hostname(
+ scope, hosts
+):
+ """The IBM name resolves to RFC1918; neither the local nor the public
+ classification may admit it."""
+ d = _evaluate_es(hosts, scope, IBM_ADDRINFO)
+ assert not d.allowed
+ assert d.reason == "unclassified"
+
+
[email protected]("scope", ES_PROTECTED_SCOPES)
[email protected]("hosts", METADATA_IP_ES_HOSTS)
+def test_evaluate_engine_denies_elasticsearch_on_literal_metadata_ip(
+ scope, hosts
+):
+ """A literal metadata address is classified without DNS. The resolver
+ is stubbed with a PUBLIC answer so that, should a literal ever reach it,
+ the test still fails rather than passing on a lookup error."""
+ d = _evaluate_es(hosts, scope, PUBLIC_ADDRINFO)
+ assert not d.allowed
+ assert d.reason == "unclassified"
+
+
+ "hosts",
+ [
+ [f"http://{PUBLIC_IP}:9200"],
+ ["https://search.example.com:9243"],
+ [{"host": "search.example.com", "port": 9243, "scheme": "https"}],
+ ],
+)
+def test_evaluate_engine_still_allows_public_elasticsearch_under_public_only(
+ hosts,
+):
+ """CONTROL: an ordinary public cluster still fails up to public and is
+ allowed under PUBLIC_ONLY."""
+ d = _evaluate_es(hosts, EgressScope.PUBLIC_ONLY, PUBLIC_ADDRINFO)
+ assert d.allowed, d.reason
+
+
+def test_evaluate_engine_still_allows_nearby_hostname_under_private_only():
+ """CONTROL: a non-metadata name on the same RFC1918 answer stays local."""
+ d = _evaluate_es(
+ ["http://api.softlayer.com:9200"],
+ EgressScope.PRIVATE_ONLY,
+ IBM_ADDRINFO,
+ )
+ assert d.allowed, d.reason
+
+
+# A backslash before "@" is a parser differential: urlparse (and so the
+# probe-target check) reads the host after "@" (localhost), while urllib3 and
+# elastic_transport, which the client uses, read the metadata host before the
+# backslash. Only the constructor's second validate_url, on the entry as
+# written, refuses these; with it removed the client is built for them.
+BACKSLASH_USERINFO_ES_HOSTS = [
+ ["http://169.254.169.254:80\\@localhost:9200"],
+ ["http://metadata.google.internal:80\\@localhost:9200"],
+ [f"http://{IBM_HOST}:80\\@localhost:9200"],
+]
+
+
+ "hosts,addrinfo",
+ [(h, IBM_ADDRINFO) for h in IBM_ES_HOSTS]
+ + [(h, PUBLIC_ADDRINFO) for h in METADATA_IP_ES_HOSTS]
+ + [(h, PUBLIC_ADDRINFO) for h in BACKSLASH_USERINFO_ES_HOSTS],
+)
+def test_elasticsearch_constructor_refuses_metadata_host(hosts, addrinfo):
+ """Defence in depth: the client is never built for a metadata node, even
+ when no scope gate ran (unprotected scope, direct construction). The
+ resolver answers PUBLIC for every name, so a backslash-userinfo entry
+ passes the probe-target check (host "localhost") and is refused only by
+ validating the entry as written."""
+ with (
+ patch.object(es_engine, "Elasticsearch") as client_cls,
+ _ssrf_dns(addrinfo),
+ ):
+ with pytest.raises(ValueError, match="SSRF validation"):
+ es_engine.ElasticsearchSearchEngine(hosts=hosts)
+ client_cls.assert_not_called()
+
+
+ "hosts,addrinfo",
+ [
+ ([f"http://{PUBLIC_IP}:9200"], PUBLIC_ADDRINFO),
+ ("https://search.example.com:9243", PUBLIC_ADDRINFO),
+ (
+ [{"host": "search.example.com", "port": 9243, "scheme": "https"}],
+ PUBLIC_ADDRINFO,
+ ),
+ (["http://api.softlayer.com:9200"], IBM_ADDRINFO),
+ ],
+)
+def test_elasticsearch_constructor_allows_ordinary_hosts(hosts, addrinfo):
+ """CONTROL: a public cluster, and a LAN name on the IBM address, still
+ connect."""
+ with (
+ patch.object(es_engine, "Elasticsearch") as client_cls,
+ _ssrf_dns(addrinfo),
+ ):
+ client_cls.return_value.info.return_value = {"cluster_name": "c"}
+ es_engine.ElasticsearchSearchEngine(hosts=hosts)
+ client_cls.assert_called_once()
+
+
+# A host dictionary is not a URL. The client hands its ``host`` value to the
+# connection pool unparsed and picks the pool by ``scheme`` alone: an HTTPS
+# pool for "https", plain HTTP for any other value. The check therefore
+# judges that (scheme, host, port) itself: a scheme other than http(s) is
+# refused whatever it contains, and the host is never re-read out of a URL
+# rebuilt from the dictionary. Every name resolves to the IBM address
+# (RFC1918, allowed) here, so only that rule can refuse these entries.
+NON_HTTP_SCHEME_ES_HOSTS = [
+ {"host": "169.254.169.254", "port": 80, "scheme": "http://localhost/#"},
+ {"host": "169.254.169.254", "port": 80, "scheme": "http://localhost?"},
+ {"host": IBM_HOST, "port": 80, "scheme": "http://127.0.0.1/#"},
+ {
+ "host": "metadata.google.internal",
+ "port": 80,
+ "scheme": "http://127.0.0.1#",
+ },
+ {"host": "fd00:ec2::254", "port": 80, "scheme": "http://localhost/#"},
+ {"host": "10.0.0.42", "port": 9200, "scheme": "ftp"},
+]
+
+
[email protected]("entry", NON_HTTP_SCHEME_ES_HOSTS)
+def test_elasticsearch_constructor_refuses_dict_host_without_http_scheme(
+ entry,
+):
+ with (
+ patch.object(es_engine, "Elasticsearch") as client_cls,
+ _ssrf_dns(IBM_ADDRINFO),
+ ):
+ with pytest.raises(ValueError, match="SSRF validation"):
+ es_engine.ElasticsearchSearchEngine(hosts=[entry])
+ client_cls.assert_not_called()
+
+
[email protected]("entry", NON_HTTP_SCHEME_ES_HOSTS)
+def test_elasticsearch_probe_skips_dict_host_without_http_scheme(entry):
+ with (
+ patch("socket.create_connection") as connect,
+ _ssrf_dns(IBM_ADDRINFO),
+ ):
+ assert (
+ es_engine.ElasticsearchSearchEngine._probe_hosts_available([entry])
+ is False
+ )
+ connect.assert_not_called()
+
+
+ "entry",
+ [
+ # A port the client would not connect to as given.
+ {"host": "10.0.0.42", "port": "9200", "scheme": "http"},
+ {"host": "10.0.0.42", "port": True, "scheme": "http"},
+ {"host": "10.0.0.42", "port": 9200.0, "scheme": "http"},
+ {"host": "10.0.0.42", "port": 0, "scheme": "http"},
+ {"host": "10.0.0.42", "port": 70000, "scheme": "http"},
+ # A host the connection pool would rewrite (it percent-decodes and
+ # IDNA-encodes) or the client refuses (brackets).
+ {"host": "10%2e0%2e0%2e42", "port": 9200, "scheme": "http"},
+ {"host": "metadata%2egoogle%2einternal", "port": 80, "scheme": "http"},
+ {"host": "fd12::1%eth0", "port": 9200, "scheme": "http"},
+ {"host": "[fd12::1]", "port": 9200, "scheme": "http"},
+ {
+ "host": "\uff41pi.service.softlayer.com",
+ "port": 80,
+ "scheme": "http",
+ },
+ # Not a host name at all.
+ {"host": "localhost#@169.254.169.254", "port": 80, "scheme": "http"},
+ {"host": "2852039166", "port": 80, "scheme": "http"},
+ # An address with a trailing dot, which a resolver may look up as a
+ # name.
+ {"host": "10.0.0.42.", "port": 9200, "scheme": "http"},
+ {"host": None, "port": 9200, "scheme": "http"},
+ ],
+)
+def test_elasticsearch_check_refuses_unusable_dict_entry(entry):
+ with _ssrf_dns(IBM_ADDRINFO):
+ assert (
+ es_engine.ElasticsearchSearchEngine._host_passes_ssrf_check(entry)
+ is False
+ )
+
+
+ "entry,target",
+ [
+ (
+ {"host": "es.internal", "port": 9201, "scheme": "https"},
+ ("https", "es.internal", 9201),
+ ),
+ (
+ {"host": "es.internal", "port": 9201, "scheme": "HTTPS"},
+ ("https", "es.internal", 9201),
+ ),
+ ({"host": "es.internal"}, ("http", "es.internal", 9200)),
+ (
+ {"host": "fd12::1", "port": 9200, "scheme": "http"},
+ ("http", "fd12::1", 9200),
+ ),
+ ("https://es.internal", ("https", "es.internal", 443)),
+ ("es.internal:9201", ("http", "es.internal", 9201)),
+ ("http://[fd12::1]:9200", ("http", "fd12::1", 9200)),
+ # Parsed as the client parses it (urllib3), not as urlsplit does.
+ (
+ "http://169.254.169.254:80\\@localhost:9200",
+ ("http", "169.254.169.254", 80),
+ ),
+ ({"host": "es.internal", "port": 9200, "scheme": "gopher"}, None),
+ ({"host": "es.internal", "port": "9200", "scheme": "http"}, None),
+ ],
+)
+def test_elasticsearch_probe_target_is_the_client_destination(entry, target):
+ assert es_engine.ElasticsearchSearchEngine._host_probe_target(entry) == (
+ target
+ )
+
+
+ "entry,destination",
+ [
+ (
+ {"host": "es.internal", "port": 9201, "scheme": "https"},
+ ("es.internal", 9201),
+ ),
+ ("http://es.internal:9202", ("es.internal", 9202)),
+ ("http://[fd12::1]:9200", ("fd12::1", 9200)),
+ ],
+)
+def test_elasticsearch_probe_connects_to_the_checked_destination(
+ entry, destination
+):
+ """The probe parses each entry once: the destination the SSRF check
+ judged (and the only name it resolved) is the one it connects to."""
+ engine_cls = es_engine.ElasticsearchSearchEngine
+ check = engine_cls._target_passes_ssrf_check
+ checked = []
+ resolved = []
+
+ def record_check(host, target):
+ checked.append(target)
+ return check(host, target)
+
+ def resolve(host, *args, **kwargs):
+ resolved.append(host)
+ return IBM_ADDRINFO
+
+ with (
+ patch.object(
+ engine_cls, "_target_passes_ssrf_check", side_effect=record_check
+ ),
+ patch(
+ "local_deep_research.security.ssrf_validator.socket.getaddrinfo",
+ side_effect=resolve,
+ ),
+ patch("socket.create_connection") as connect,
+ ):
+ assert engine_cls._probe_hosts_available([entry]) is True
+ connect.assert_called_once_with(destination, timeout=1.0)
+ assert [(t[1], t[2]) for t in checked] == [destination]
+ assert set(resolved) == {destination[0]}
+
+
+# ---------------------------------------------------------------------------
+# Non-canonical spellings of a metadata address
+# ---------------------------------------------------------------------------
+#
+# Each entry reaches a listed metadata address: the resolver ignores trailing
+# dots and accepts the inet_aton notations (integer, hex, octal, short
+# forms), and an IPv4-mapped or NAT64 address carries the metadata IPv4 as
+# its payload. The engine checks no longer need to recognise these: host
+# classification accepts only canonical spellings (_canonical_host), so an
+# alternate notation is unclassified whatever address it names, and a
+# canonical metadata literal or wrap is refused by _is_metadata_host. The
+# fetch-time check is broader on purpose: evaluate_url refuses every one of
+# these as a metadata destination (_is_metadata_destination, over the forms
+# _policy_host_forms produces). The resolver is stubbed with a PUBLIC answer:
+# a spelling that slipped through would classify public and fail the local
+# engine UP, which PUBLIC_ONLY and BOTH admit (and STRICT admits for the
+# primary engine). The generated corpus in
+# test_egress_host_canonical_classification.py covers far more spellings.
+
+METADATA_IP_SPELLINGS = [
+ # trailing dots
+ "169.254.169.254.",
+ "169.254.169.254..",
+ "168.63.129.16.",
+ # inet_aton notations, bare and with a trailing dot
+ "2852039166",
+ "2852039166.",
+ "0xa9fea9fe",
+ "0XA9FEA9FE.",
+ "0251.0376.0251.0376",
+ "0251.0376.0251.0376.",
+ "0xa9.0xfe.0xa9.0xfe",
+ "169.254.43518",
+ "169.254.43518.",
+ "169.16689662",
+ "0xa83f8110", # 168.63.129.16 (Azure WireServer)
+ # non-ASCII spellings the resolver's IDNA encoding folds to the address
+ "\uff11\uff16\uff19.\uff12\uff15\uff14.\uff11\uff16\uff19.\uff12\uff15\uff14",
+ "169\u3002254\u3002169\u3002254",
+ "\uff12\uff18\uff15\uff12\uff10\uff13\uff19\uff11\uff16\uff16",
+ # IPv4-mapped, NAT64 (both well-known prefixes) and non-canonical IPv6
+ "[::ffff:169.254.169.254]",
+ "[::ffff:a9fe:a9fe]",
+ "[64:ff9b::169.254.169.254]",
+ "[64:ff9b::a9fe:a9fe]",
+ "[64:ff9b:1:a9fe:a9:fe00::]",
+ "[FD00:EC2:0:0:0:0:0:254]",
+]
+
+# Same notations, but not a metadata address.
+NON_METADATA_SPELLINGS = [
+ "10.0.0.42.",
+ "0x0a00002a", # 10.0.0.42
+ "2852039234", # 169.254.170.66
+ "169.254.1.2",
+ "search.example.com.",
+ "\uff11\uff10.\uff10.\uff10.\uff14\uff12", # fullwidth 10.0.0.42
+]
+
+URL_ENGINE_SETTINGS = {
+ "elasticsearch": ES_HOSTS_SETTING,
+ "paperless": "search.engine.web.paperless.default_params.api_url",
+}
+
+
+def _evaluate_url_engine(engine, value, scope):
+ ctx = EgressContext(
+ scope=scope,
+ primary_engine=engine,
+ require_local_llm=False,
+ require_local_embeddings=False,
+ )
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=PUBLIC_ADDRINFO,
+ ):
+ return evaluate_engine(
+ engine, ctx, settings_snapshot={URL_ENGINE_SETTINGS[engine]: value}
+ )
+
+
[email protected]("scope", ES_PROTECTED_SCOPES)
[email protected]("engine", list(URL_ENGINE_SETTINGS))
[email protected]("host", METADATA_IP_SPELLINGS)
+def test_evaluate_engine_denies_non_canonical_metadata_spelling(
+ scope, engine, host
+):
+ d = _evaluate_url_engine(engine, f"http://{host}:9200", scope)
+ assert not d.allowed
+ assert d.reason == "unclassified"
+
+
[email protected]("scope", ES_PROTECTED_SCOPES)
[email protected]("host", METADATA_IP_SPELLINGS)
+def test_evaluate_engine_denies_metadata_spelling_in_es_host_list(scope, host):
+ """A metadata spelling next to a public node in a comma-separated hosts
+ string still withholds the engine."""
+ d = _evaluate_url_engine(
+ "elasticsearch",
+ f"http://{PUBLIC_IP}:9200,http://{host}:9200",
+ scope,
+ )
+ assert not d.allowed
+ assert d.reason == "unclassified"
+
+
[email protected]("engine", list(URL_ENGINE_SETTINGS))
+def test_evaluate_engine_trailing_dot_public_host_still_fails_up(engine):
+ """CONTROL: stripping trailing dots does not withhold an ordinary host;
+ a public name with a trailing dot still fails up to public."""
+ d = _evaluate_url_engine(
+ engine, "https://search.example.com.:9243", EgressScope.PUBLIC_ONLY
+ )
+ assert d.allowed, d.reason
+
+
[email protected]("engine", list(URL_ENGINE_SETTINGS))
+def test_evaluate_engine_trailing_dot_private_literal_stays_local(engine):
+ """CONTROL for the string branch's trailing-dot strip: a private literal
+ written with a trailing dot is classified as that literal (local), not
+ sent to the resolver, whose stub here answers PUBLIC."""
+ d = _evaluate_url_engine(
+ engine, "http://10.0.0.42.:9200", EgressScope.PRIVATE_ONLY
+ )
+ assert d.allowed, d.reason
+
+
[email protected]("scope", list(EgressScope))
[email protected]("host", METADATA_IP_SPELLINGS)
+def test_evaluate_url_blocks_non_canonical_metadata_spelling(scope, host):
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=PUBLIC_ADDRINFO,
+ ):
+ d = evaluate_url(f"http://{host}/", _ctx(scope))
+ assert not d.allowed
+ assert d.reason == "blocked_metadata_ip"
+
+
[email protected]("host", METADATA_IP_SPELLINGS)
+def test_is_metadata_destination_recognises_spelling(host):
+ assert _is_metadata_destination(host) is True
+ assert _is_metadata_destination(host.strip("[]")) is True
+
+
[email protected]("host", NON_METADATA_SPELLINGS)
+def test_is_metadata_destination_control(host):
+ """CONTROL: the normaliser does not turn other addresses into
+ metadata."""
+ assert _is_metadata_destination(host) is False
+
+
[email protected]("host", METADATA_IP_SPELLINGS + NON_METADATA_SPELLINGS)
+def test_evaluate_url_and_is_metadata_destination_agree(host):
+ """Both decide from the same normalised host, so they agree on every
+ spelling in the corpus."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=PUBLIC_ADDRINFO,
+ ):
+ d = evaluate_url(f"http://{host}/", _ctx(EgressScope.PUBLIC_ONLY))
+ assert (d.reason == "blocked_metadata_ip") is _is_metadata_destination(host)
+
+
+# NAT64 and zone-scope branches of _is_metadata_destination.
+NAT64_METADATA = [
+ "64:ff9b::a9fe:a9fe",
+ "64:ff9b::169.254.169.254",
+ "64:ff9b::a9fe:aa02", # 169.254.170.2 (ECS)
+ "64:ff9b:1:a9fe:a9:fe00::", # /48 local-use prefix, RFC 6052 layout
+]
+ZONE_SCOPED_METADATA = [
+ "fd00:ec2::254%eth0",
+ "fe80::a9fe:a9fe%eth0",
+ "[fe80::a9fe:a9fe%25eth0]", # as urlsplit returns a zone in a URL
+ "::ffff:169.254.169.254%1",
+]
+
+
[email protected]("host", NAT64_METADATA)
+def test_is_metadata_destination_nat64_branch(host):
+ assert _is_metadata_destination(host) is True
+
+
[email protected]("host", ZONE_SCOPED_METADATA)
+def test_is_metadata_destination_zone_scope_branch(host):
+ assert _is_metadata_destination(host) is True
+
+
+ "host",
+ ["64:ff9b::808:808", "64:ff9b::a00:2a", "fe80::1%eth0", "fd00:ec2::253%1"],
+)
+def test_is_metadata_destination_nat64_and_zone_controls(host):
+ """CONTROL: a NAT64 wrap of a non-metadata IPv4, and a zone-scoped
+ non-metadata address."""
+ assert _is_metadata_destination(host) is False
+
+
+# ---------------------------------------------------------------------------
+# A metadata address listed in llm.allowed_local_hostnames is still not local
+# ---------------------------------------------------------------------------
+
+LOCAL_HOSTNAME_METADATA_ENTRIES = [
+ "169.254.169.254",
+ "169\u3002254\u3002169\u3002254",
+ "fd00:ec2::254",
+ "169.254.169.254.",
+ "2852039166",
+ "0xa9fea9fe",
+ "::ffff:169.254.169.254",
+ "64:ff9b::a9fe:a9fe",
+]
+
+
[email protected]("entry", LOCAL_HOSTNAME_METADATA_ENTRIES)
+def test_classify_host_ignores_local_hostnames_for_metadata_address(entry):
+ """An entry that names a metadata endpoint (a canonical literal or wrap)
+ or has no canonical spelling (the other entries) is ignored by the
+ user-declared local_hostnames override, and the host itself is never
+ local. The resolver answers RFC1918, so only that rule keeps the entry
+ from classifying local."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ ctx = _ctx(EgressScope.PRIVATE_ONLY, local_hostnames=[entry])
+ assert _classify_host(entry, ctx) is False
+
+
[email protected]("entry", LOCAL_HOSTNAME_METADATA_ENTRIES)
+def test_require_local_llm_refuses_metadata_address_in_local_hostnames(entry):
+ host = f"[{entry}]" if ":" in entry else entry
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=IBM_ADDRINFO,
+ ):
+ d = evaluate_llm_endpoint(
+ "ollama",
+ _ctx(EgressScope.PRIVATE_ONLY, local_hostnames=[entry]),
+ settings_snapshot={"llm.ollama.url": f"http://{host}:11434"},
+ )
+ assert not d.allowed
+ assert d.reason == "provider_remote"
+
+
+def test_require_local_llm_still_honours_local_hostnames_control():
+ """CONTROL: an ordinary declared local name is still local."""
+ with patch(
+ "local_deep_research.security.egress.policy._resolve_with_timeout",
+ return_value=PUBLIC_ADDRINFO,
+ ):
+ d = evaluate_llm_endpoint(
+ "ollama",
+ _ctx(EgressScope.PRIVATE_ONLY, local_hostnames=["gpu-box.lan"]),
+ settings_snapshot={"llm.ollama.url": "http://gpu-box.lan:11434"},
+ )
+ assert d.allowed, d.reason
+
+
+# ---------------------------------------------------------------------------
+# Error wording names a metadata endpoint as such
+# ---------------------------------------------------------------------------
+
+
[email protected]("entry", [IBM_HOST, "169.254.169.254"])
+def test_local_hostnames_validator_names_metadata_entry(entry):
+ """The IBM name resolves to RFC1918, so "resolves to PUBLIC addresses"
+ would be wrong; the entry is refused as a metadata endpoint."""
+ with patch.object(
+ validators, "_resolve_with_timeout", return_value=IBM_ADDRINFO
+ ):
+ result = validators.validate_allowed_local_hostnames(
+ {"llm.allowed_local_hostnames": [entry]}, {}
+ )
+ assert result is not None
+ assert "cloud-metadata" in result["error"]
+ assert "PUBLIC" not in result["error"]
+
+
+def test_notification_validator_names_metadata_hostname():
+ """A name block must not be reported as an IP-address block."""
+ with patch.object(
+ NotificationURLValidator,
+ "_resolve_hostname_ips",
+ return_value=IBM_RESOLVED,
+ ):
+ ok, err = NotificationURLValidator.validate_service_url(
+ f"https://{IBM_HOST}/hook", allow_private_ips=True
+ )
+ assert ok is False
+ assert err.startswith("Blocked cloud-metadata hostname:")