~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(security): follow-up fixes for #5777 egress vector-store containment (#6197)
LearningCircuit2a594e0193d2
diff --git a/changelog.d/5763.security.md b/changelog.d/5763.security.mdindex ca41394108..3f60e62f6f 100644--- a/changelog.d/5763.security.md+++ b/changelog.d/5763.security.md@@ -17,11 +17,30 @@ reclassified fully *public* on a public endpoint, so `private_only` aloneblocks them while `public_only`/adaptive-public still allow them; anon-contained private collection is blocked everywhere instead, becauselosing containment must only ever add restriction, never grant new-eligibility. The two-axis run-start audit (ADR-0007) independently reaches-the same result for a private collection: it labels a non-contained-collection `EXPOSING` regardless of `is_public`, and a `SENSITIVE` (private)-source may not reach an `EXPOSING` sink under that rule, so the private,-non-contained collection is blocked by both mechanisms.+eligibility.++For the common single-engine run, the two-axis run-start audit (ADR-0007) is+NOT an independent second line of defence here: `audit_run_from_snapshot`+(the only production entry point, called from `web/routers/research.py` and+`web/services/research_service.py`) always classifies just the run's primary+engine. When that primary is a private, non-contained collection, it is the+SOLE sensitive source in the audited set, so the quadrant-4 self-exclusion in+`classification.py` applies (a component may reach a sink that is its own+data) and the audit returns an ALLOW (`"sensitive_contained"`), not a denial —+provided the run's inference sinks are themselves contained; an exposing+LLM/embeddings sink denies the run first+(`"sensitive_to_exposing_inference"`, checked in `classification.py` before+the self-exclusion is considered).+For a single-engine run the audit therefore adds nothing: it evaluates the+primary engine only (its LLM/embeddings entries are inference sinks, not+sensitive sources), so a second sensitive source can never appear in the+audited set. Two independent guards still cover the case. The+`unclassified` block above governs ENGINE ELIGIBILITY — the collection is+denied under every scope. Separately, on the SCOPE side, a primary engine+that classifies as neither public nor local resolves `adaptive` to the+restrictive `private_only` rather than the permissive `both`, which keeps the+rest of the run (LLM and embeddings included) local as well. Neither guard+depends on the other.A non-local-file store whose endpoint cannot be resolved is treated asremote (not contained).@@ -35,3 +54,67 @@ extending `trusted_search_engines` to cover vector-store endpoints, or aper-collection "trusted remote store" flag) is a maintainer decision thatshould land before server-backed vector stores (tracked in #5331) areexpected to be usable under anything other than `unprotected`.++A rejected network-share path in a file URI remains non-contained even+when its nominal authority resolves locally, and that rejection is now+applied to every entry of the endpoint setting rather than only to a+plain-string value — a list-shaped setting (the shape an Elasticsearch-style+endpoint list uses, and the shape a `{"value": [...]}` snapshot unwraps to)+no longer reaches hostname classification with a rejected path still in it.+A single rejected entry makes the whole setting non-contained, matching the+existing "any public entry wins" fail-up. Current FAISS configurations are+unaffected.++Windows drive-absolute store paths written with a doubled or mixed separator+after the drive letter (`C:\\data\store.db`, `C:/\data`, `C:\/data`) are+recognised as local again: a drive letter always wins over a UNC prefix on+Windows and none of these forms can name a remote host. **Known limitation:**+the forward-slash form `C://data/store.db` stays non-contained, because it is+byte-for-byte a single-letter-scheme URL (scheme `c`, hostname `data`) and+cannot be told apart from the `x://host` authority form the classifier must+reject. Write such a path as `C:/data/store.db` or `C:\data\store.db`.+**Second known limitation:** because the decoded path is what decides whether+a `file://` URI names a network share, a `file://` URI whose FIRST path+segment literally begins with an unescaped `%2f` or `%5c`+(`file:///%2fdir/store.db`) is treated as non-contained. `%252f` (the correct+RFC 3986 encoding of a literal `%2f`) and `%2f` in any later segment stay+contained. This only affects URIs built by concatenating `"file://"` with a+raw path; `pathlib.Path.as_uri()` never produces one.++Whether a store URI is a local filesystem path, and whether it is a `file://`+URI naming a network share, are now decided the way `urllib.parse.urlsplit`+itself reads the value rather than from the raw setting string. `urlsplit`+follows the WHATWG URL parser: it deletes every tab, CR and LF from *anywhere*+in the URL and strips leading C0 control characters. `str.strip()` does+neither of the two things that matter here — it never touches a tab, CR or LF+in the *middle* of a value, and it leaves a leading non-whitespace C0+character such as `\x00` or `\x01` in place. Deciding the path question on the+raw string while the host was decided on the parsed one let a control+character split the two readings apart: `x:/<TAB>/host:19530` and+`/<TAB>/host/share/db` were "a local drive path" / "an ordinary absolute path"+to the raw tests but a `host:19530` / `host` authority to the parser, so the+endpoint short-circuited to contained without any host classification. The+path tests now run on the same string the parser sees, and the "is this a+`file://` URI?" question is answered by the parsed scheme, so+`fi<TAB>le://localhost//host/share` — not a `file://` URI to a raw+`startswith`, but one with a local-looking authority to every real parser —+no longer slips past the network-share rejection. For the characters named+above — a tab, CR or LF anywhere, and a leading C0 control — such values are+now classified exactly as their control-character-free equivalents are. That+scope is the claim: an *interior* `\x00` is not covered, because `urlsplit`+does not delete it either, so it stays in the path and such a value can still+classify differently from its NUL-free equivalent. That difference is not a+route off the box — `ntpath.normpath` resolves `/\x00//host/share` to a+single leading separator, not to a `\\host\share` UNC prefix.++A Windows drive-absolute store path must now begin with an *ASCII* drive+letter. `str.isalpha()` is Unicode-wide, so `с:/data/store.db` (Cyrillic+"es"), `C:/data/store.db` (fullwidth) and `é:/data/store.db` previously+short-circuited to "local file" and skipped host classification entirely.+None of them is a drive path to any resolver. This matches the ASCII+requirement already applied to the `file://<letter>:/path` authority form;+real `A:`–`Z:` paths in either case are unaffected.++This release does not change how `.local` and RFC1918 authorities are+classified: they continue to count as contained, exactly as they did before+this fix, and revisiting that is out of scope here.diff --git a/src/local_deep_research/security/egress/policy.py b/src/local_deep_research/security/egress/policy.pyindex ed956c7931..3483884c8b 100644--- a/src/local_deep_research/security/egress/policy.py+++ b/src/local_deep_research/security/egress/policy.py@@ -638,11 +638,81 @@ def _starts_with_double_separator(s: str) -> bool:return len(s) >= 2 and s[0] in "/\\" and s[1] in "/\\"+# ``urllib.parse.urlsplit`` does NOT parse the string it is handed verbatim.+# Following the WHATWG URL parser it first lstrips every C0 control character+# and space, then DELETES every tab, CR and LF from ANYWHERE in the URL+# (CPython's ``_WHATWG_C0_CONTROL_OR_SPACE`` and+# ``_UNSAFE_URL_BYTES_TO_REMOVE``). ``str.strip()`` does neither: it leaves+# "\x00" and the other non-whitespace C0 controls in place, and never touches+# a tab in the MIDDLE of a string.+_URL_PARSER_LEADING_STRIP = "".join(chr(code) for code in range(0x20)) + " "+_URL_PARSER_REMOVED_CHARS = ("\t", "\r", "\n")+++def _normalize_uri_like_url_parser(value: str) -> str:+ """``value`` reduced to the string ``urlsplit`` would actually parse.++ Exists for the RAW string tests in ``_is_local_filesystem_uri``+ (``startswith``, ``_starts_with_double_separator``, a drive-letter index),+ whose verdict is consumed side by side with ``_classify_engine_url``,+ which reads the host out of ``urlsplit``. Judging the raw string opens a+ parser differential in which one value is simultaneously "a local+ filesystem path" to ``str.startswith`` and "an authority" to every real+ URL parser:++ * ``"x:/\\t/attacker.example.com:19530"`` looks drive-absolute to a+ literal ``"//"`` test (``uri[2:]`` is ``"/\\t/..."``), while+ ``urlsplit`` deletes the tab and reports netloc+ ``attacker.example.com:19530``.+ * ``"/\\t/attacker.example.com/share/db"`` looks like an ordinary+ single-slash absolute path to ``_starts_with_double_separator``, while+ ``urlsplit`` deletes the tab and reports the same authority.+ * A leading ``"\\x00"`` (or any other non-whitespace C0 control) survives+ ``str.strip()`` and defeats every raw prefix test, while ``urlsplit``+ lstrips it away.++ Normalising once, here, is what keeps those predicates and ``urlsplit``+ from drifting apart again.++ A raw-string test is not the only caller that needs this. The final+ ``strip()`` is BIDIRECTIONAL, and its LEADING half removes 19+ Unicode-whitespace codepoints that ``urlsplit``'s WHATWG lstrip+ (``\\x00``-``\\x20`` only) leaves in place: U+0085, U+00A0, U+1680,+ U+2000-U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. A leading one+ of those makes ``urlsplit`` report an EMPTY scheme, so a function that+ asks the parser for the scheme does NOT already get this preprocessing+ for free. The call is therefore load-bearing in+ ``_file_uri_names_a_remote_share``: without it,+ "\\xa0file://localhost//attacker.example.com/share/db" has scheme "" and+ is not a ``file://`` URI to that function, the network-share rejection+ never fires, and the entry falls through to hostname classification —+ where a sibling entry such as "127.0.0.1" answers LOCAL and the whole+ setting classifies contained. Never drop this call on the ground that+ ``urlsplit`` repeats it; for the leading-whitespace half it does not.++ Only the TRAILING half of that ``strip()`` is a pure caller-compatibility+ trim: ``urlsplit`` keeps trailing whitespace where this does not, but+ trailing whitespace can never turn a local path into an authority, so the+ extra trim cannot make a remote endpoint look local.+ """+ normalized = value.lstrip(_URL_PARSER_LEADING_STRIP)+ for char in _URL_PARSER_REMOVED_CHARS:+ normalized = normalized.replace(char, "")+ return normalized.strip()++def _is_local_filesystem_uri(value) -> bool:"""True when a vector-store URI is unambiguously a local file path."""if not isinstance(value, str):return False- uri = value.strip()+ # Judge the string ``urlsplit`` would parse, never the raw one — see+ # ``_normalize_uri_like_url_parser``. Every prefix, separator and+ # drive-letter test below is a RAW string test, so without this the+ # "local drive path" / "single-slash absolute path" verdicts diverge from+ # the host ``_classify_engine_url`` goes on to classify. This call is what+ # stops "x:/\t/attacker.example.com:19530" and+ # "/\t/attacker.example.com/share/db" short-circuiting LOCAL.+ uri = _normalize_uri_like_url_parser(value)if not uri:return Falseif uri.lower().startswith("file://"):@@ -654,10 +724,34 @@ def _is_local_filesystem_uri(value) -> bool:# is a remote-share reference some resolvers treat as a host (e.g.# "file://attacker.example.com/share/store.db") and must NOT be# classified as on-box — fall through to the remote/DNS path.- parsed = urlsplit(uri)+ #+ # ``urlsplit`` is NOT exception-safe: a malformed authority (e.g. an+ # unbalanced IPv6-literal bracket, "file://[attacker.example.com/x"+ # or "file://[::1/x") raises ``ValueError: Invalid IPv6 URL`` instead+ # of returning a parse. Unlike its sibling in ``_classify_engine_url``+ # (which already wraps this identical call), this call was bare, so+ # the exception used to propagate all the way up through+ # ``_vector_store_endpoint_is_local`` -> ``vector_store_is_contained``+ # -> ``classify_engine`` into ``_resolve_adaptive_scope``'s catch-all,+ # which resolves the PERMISSIVE ``EgressScope.BOTH`` — exactly the+ # cloud-capable scope this module's fail-up rules exist to avoid.+ # Fail toward not-local instead, matching every other unknown here.+ try:+ parsed = urlsplit(uri)+ except Exception:+ return Falsenetloc = parsed.netloc+ # Windows drive letters are ASCII A-Z only. ``str.isalpha()`` is+ # Unicode-wide (Cyrillic "с", fullwidth "C", accented "é" all+ # match), so a Unicode lookalike authority (e.g.+ # "file://с:/share/db") must NOT pass as an on-box drive path —+ # the same hazard class as the single-letter-scheme lookalikes at+ # the bottom of this function.is_drive_letter_netloc = (- len(netloc) == 2 and netloc[0].isalpha() and netloc[1] == ":"+ len(netloc) == 2+ and netloc[0].isascii()+ and netloc[0].isalpha()+ and netloc[1] == ":")if not (netloc == ""@@ -676,8 +770,27 @@ def _is_local_filesystem_uri(value) -> bool:# smuggles the same UNC authority past a forward-slash-only check:# ``ntpath.normpath`` treats ANY pair of leading separators — "//",# "/\\", "\\/", or "\\\\" — as a UNC prefix on Windows, so reject all- # four combinations, not just the literal "//" one.- return not _starts_with_double_separator(parsed.path)+ # four combinations, not just the literal "//" one. The path may be+ # percent-encoded (RFC 8089), so DECODE it before the separator+ # check: "file:///%2fattacker.example.com/share/db" and the "%5c"+ # twin read as single-separator paths while encoded, but unquote+ # to exactly the "//host" / "/\\host" UNC prefixes the check exists+ # to catch (``ntpath.normpath`` resolves both to a remote share).+ # Decoding cannot smuggle an authority in: netloc was already+ # validated above, and plain local paths ("/var/lib/x",+ # "/C:/data") keep their single leading separator through unquote.+ #+ # DEFENCE IN DEPTH: this decode-and-check is a deliberate duplicate+ # of ``_file_uri_names_a_remote_share``, which+ # ``_vector_store_endpoint_is_local`` — the ONLY production caller of+ # this function — already ran one frame earlier over every entry of+ # the setting. At every production entry point the earlier check has+ # therefore already returned "not contained", so reverting the+ # ``unquote`` here changes no end-to-end outcome; it stays because+ # this function is a general-purpose predicate that a future caller+ # may reach directly, and it is pinned by its own unit test+ # (``test_file_uri_percent_encoded_unc_in_path_is_not_a_local_file``).+ return not _starts_with_double_separator(unquote(parsed.path))if uri.startswith(_LOCAL_PATH_URI_PREFIXES):return True# A single leading "/" is a local absolute path ("/var/lib/x.db") ONLY@@ -689,12 +802,106 @@ def _is_local_filesystem_uri(value) -> bool:# instead of being silently treated as on-box.if uri.startswith("/") and not _starts_with_double_separator(uri):return True- # Windows drive-absolute path ("C:\\data\\store.db").+ # Windows drive-absolute path ("C:\\data\\store.db"). What follows the+ # drive letter must not be a literal "//" — "<letter>://" is a+ # "scheme://host" URL authority marker, not a path, so e.g.+ # "x://attacker.example.com:19530" must NOT be misread as a local drive+ # path and skip the DNS classifier. Note ``str.isalpha()`` is+ # Unicode-wide (Cyrillic/fullwidth letters match too, e.g. "с://host" or+ # "C://host"), so the drive letter must be ASCII here for the same+ # reason the ``file://`` netloc test above requires it: "с:/x", "C:/x"+ # and "é:/x" are not Windows drive-absolute paths, and short-circuiting+ # them to on-box would skip the DNS classifier on a value no resolver+ # reads as a local drive. The literal-"//" authority check below is a+ # separate, independent guard covering the "<letter>://host" form.+ #+ # ONLY the literal "//" form is rejected, not every doubled separator:+ # a drive letter always wins over a UNC prefix on Windows+ # (``ntpath.normpath("C:\\\\data") == "C:\\data"``, never a network+ # share), and — ON THE NORMALISED STRING this function tests —+ # ``urlsplit`` gives an authority only to a literal "//", so+ # "C:\\\\data\\store.db", "C:/\\data" and "C:\\/data" cannot name a+ # remote host under either reading and are genuine local paths. That+ # equivalence holds ONLY because ``_normalize_uri_like_url_parser`` has+ # already deleted the tab/CR/LF and leading C0 controls that ``urlsplit``+ # itself deletes: on the RAW string "x:/\t/attacker.example.com:19530"+ # this literal-"//" test answers "no authority, local drive path" while+ # ``urlsplit`` answers "netloc=attacker.example.com:19530". Never move+ # this test back onto an un-normalised string. The+ # forward-slash form "C://data" is irreducibly ambiguous — it parses as+ # scheme "c" with hostname "data" — so it stays rejected and falls+ # through to the DNS classifier; see changelog.d/5763.security.md.return (- len(uri) > 2 and uri[0].isalpha() and uri[1] == ":" and uri[2] in "\\/"+ len(uri) > 2+ and uri[0].isascii()+ and uri[0].isalpha()+ and uri[1] == ":"+ and uri[2] in "\\/"+ and not uri[2:].startswith("//"))+def _file_uri_names_a_remote_share(value) -> bool:+ """True when ``value`` is a ``file://`` URI that does not stay on the box.++ The DECODED path of a ``file://`` URI can name a UNC network share+ (``//host/share``, or any of the mixed "/"/"\\" separator pairs+ ``ntpath.normpath`` also resolves to one) regardless of what the URI's+ nominal authority says — "file://localhost//attacker.example.com/share"+ reads as localhost but resolves off-box. Such a URI must stay+ non-contained even after ``_is_local_filesystem_uri`` has declined it,+ so hostname classification cannot rescue it via the local-looking+ authority.++ The "is this a ``file://`` URI?" question is answered by the PARSED+ scheme, not by a raw ``startswith`` on the string: ``urlsplit`` deletes+ tab/CR/LF from anywhere in a URL and lstrips C0 controls, so+ "fi\\tle://localhost//attacker.example.com/share/db" is not a ``file://``+ URI to ``startswith`` but IS one to every real URL parser — and+ ``_classify_engine_url``, the frame that decides the host, is a real URL+ parser, which for that value reports the local-looking authority+ ``localhost`` and would classify the endpoint contained. Asking+ ``urlsplit`` for the scheme is what keeps this function from being+ stepped around by such a value.++ ``value`` is normalised by ``_normalize_uri_like_url_parser`` BEFORE it+ reaches ``urlsplit``, and that call is load-bearing here, not a duplicate+ of the parser's own preprocessing. ``urlsplit`` lstrips only+ ``\\x00``-``\\x20``, while the normaliser's final ``strip()`` is+ bidirectional and additionally removes the 19 leading Unicode-whitespace+ codepoints ``str.strip()`` knows and the parser does not (U+0085, U+00A0,+ U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000). Handed+ the RAW value, "\\xa0file://localhost//attacker.example.com/share/db"+ parses with an EMPTY scheme, this function answers False, the+ network-share rejection never fires, and — inside a list-shaped endpoint+ setting whose sibling entry is "127.0.0.1" — the whole setting classifies+ CONTAINED. Normalise first; see+ ``test_leading_unicode_whitespace_cannot_hide_a_network_share``.++ Fails toward "remote" for a malformed authority: ``urlsplit`` raises+ (``ValueError``: "Invalid IPv6 URL") on an unbalanced IPv6 bracket rather+ than parsing, and an endpoint we cannot parse is one we cannot prove+ local. The catch is broad, matching its sibling in+ ``_is_local_filesystem_uri``, so a future parser error class cannot turn+ a fail-closed answer into an exception that escapes into+ ``_resolve_adaptive_scope``'s permissive catch-all.++ Non-strings answer False here — they are not ``file://`` URIs, and every+ other non-string shape already fails closed downstream by never proving+ itself local.+ """+ if not isinstance(value, str):+ return False+ uri = _normalize_uri_like_url_parser(value)+ try:+ parsed = urlsplit(uri)+ if parsed.scheme.lower() != "file":+ return False+ return _starts_with_double_separator(unquote(parsed.path))+ except Exception:+ return True++def _vector_store_endpoint_is_local(provider: str,settings_snapshot: Optional[dict],@@ -714,9 +921,31 @@ def _vector_store_endpoint_is_local(uri_setting = getattr(cls, "uri_setting", None) if cls else Noneif not uri_setting:return False- if _is_local_filesystem_uri(- _get_setting_value(settings_snapshot or {}, uri_setting, None)- ):+ uri = _get_setting_value(settings_snapshot or {}, uri_setting, None)+ # Reject a network-share file path PER ENTRY. A setting value is not+ # necessarily a string: ``unwrap_setting`` peels the ``{"value": ...}``+ # snapshot wrapper and returns whatever it wrapped, uncoerced, and+ # ``_classify_engine_url`` below explicitly supports the LIST shape+ # (Elasticsearch-style endpoint lists). A string-only guard here would+ # leave a one-element list to fall through to hostname classification,+ # where a nominally local authority ("localhost", "127.0.0.1") answers+ # True and overrules the path rejection. Normalising to entries first+ # keeps a single rejection site that covers every shape the setting can+ # hold, and fails the WHOLE setting closed when ANY entry is a rejected+ # path — matching the "any public entry wins" fail-up+ # ``_classify_engine_url`` already applies to list-typed endpoints.+ #+ # COUPLED WITH ``_classify_engine_url``: this loop and that function must+ # keep the SAME container test (``isinstance(..., list)``, not+ # ``(list, tuple)``) and the same "skip non-``str`` entries" rule. If one+ # side alone grows tuple support, a tuple-shaped setting reaches hostname+ # classification with a rejected network-share path still in it — the+ # exact bypass this per-entry loop exists to close. Change both or+ # neither.+ for entry in uri if isinstance(uri, list) else (uri,):+ if _file_uri_names_a_remote_share(entry):+ return False+ if _is_local_filesystem_uri(uri):return Truereturn (_classify_engine_url(@@ -1041,6 +1270,12 @@ def _classify_engine_url(if not value:return None+ # COUPLED WITH the per-entry network-share rejection in+ # ``_vector_store_endpoint_is_local``: both must keep the SAME container+ # test (``isinstance(..., list)``, not ``(list, tuple)``) and the same+ # "skip non-``str`` entries" rule. Widening only one side lets a+ # tuple-shaped setting reach this classifier with an entry the other side+ # never examined. Change both or neither.entries = value if isinstance(value, list) else [value]any_public = Falseany_local = False@@ -1799,12 +2034,18 @@ def _resolve_adaptive_scope(# regardless of the scope resolved here; PRIVATE_ONLY's role is to keep# every OTHER engine/inference path in the run local too, rather than# letting the run silently widen to cloud-capable BOTH. This branch is- # reached via the private-non-contained-collection fail-up above (or a- # static engine that explicitly declares BOTH flags False) — NOT by a- # static engine that simply forgot to declare its flags: ``_engine_flags``- # reads missing attributes as ``None`` (not ``False``), so an undeclared- # engine classifies as ``(None, None)`` and falls to the retriever-lookup- # branch below instead. Dedup/log via the same per-(user, primary) mechanism+ # reached via the private-non-contained-collection fail-up above, or by a+ # static engine whose class overrides NEITHER flag: ``BaseSearchEngine``+ # declares ``is_public = False`` and ``is_local = False`` as CLASS+ # DEFAULTS (``search_engine_base.py``), so such a subclass inherits+ # ``(False, False)`` from the base class — it need not explicitly write+ # either assignment itself — and lands HERE, not in the ``(None, None)``+ # branch below. ``_engine_flags`` only returns ``None`` for a flag that is+ # genuinely absent from the class's MRO, which no attribute on+ # ``BaseSearchEngine`` ever is; all 29 current registry engines override+ # exactly one flag to ``True`` and so don't reach this branch today, but+ # the next engine author who forgets to override either flag will, not+ # the retriever-lookup fallback. Dedup/log via the same per-(user, primary) mechanism# as the fail-open warning below — this fires on the identical hot path# (once per research-run start AND once per fetch-gate check).if classification.is_public is False and classification.is_local is False:diff --git a/tests/security/test_egress_vector_store_exposure.py b/tests/security/test_egress_vector_store_exposure.pyindex 44960609b3..59323fc80e 100644--- a/tests/security/test_egress_vector_store_exposure.py+++ b/tests/security/test_egress_vector_store_exposure.py@@ -19,6 +19,8 @@ from __future__ import annotationsfrom contextlib import ExitStackfrom unittest.mock import patch+import pytest+from local_deep_research.security.egress import policy as Pfrom local_deep_research.security.egress import run_classification as rcfrom local_deep_research.security.egress.classification import (@@ -335,13 +337,31 @@ def test_recorded_index_provider_tightens_the_settings_default():def test_index_provider_lookup_failure_is_soft():"""An unavailable DB must not break classification: the settings-resolved- provider (FAISS today) remains the primary signal."""- with patch(- "local_deep_research.database.session_context.get_user_db_session",- side_effect=RuntimeError("db unavailable"),- ):- assert P._resolve_index_vector_store_providers("library", None) == set()- assert P.vector_store_is_contained("library", ctx(), {}) is True+ provider (FAISS today) remains the primary signal.++ ``RAGIndex.vector_store_provider`` does not exist on the CURRENT schema,+ so the ``column is None`` guard in+ ``_resolve_index_vector_store_providers`` returns ``set()`` before+ ``get_user_db_session`` is ever called — patching only the session opener+ (as this test used to) never reaches the DB call or its ``except``, so it+ passed unconditionally regardless of what was patched. Patch the column+ onto the model too (``create=True`` — the attribute is genuinely absent+ today, this is simulating the future schema) so the test actually drives+ execution into the ``with get_user_db_session(...)`` block and exercises+ the failure path this test is named for.+ """+ from local_deep_research.database.models.library import RAGIndex++ with patch.object(RAGIndex, "vector_store_provider", object(), create=True):+ with patch(+ "local_deep_research.database.session_context.get_user_db_session",+ side_effect=RuntimeError("db unavailable"),+ ):+ assert (+ P._resolve_index_vector_store_providers("library", None)+ == set()+ )+ assert P.vector_store_is_contained("library", ctx(), {}) is True# ---------------------------------------------------------------------------@@ -447,13 +467,667 @@ def test_file_uri_windows_drive_authority_is_a_local_file():assert P._is_local_filesystem_uri("FILE://c:/data/store.db") is True+def test_malformed_file_uri_authority_does_not_raise():+ """A ``file://`` URI whose authority is an unbalanced IPv6-literal+ bracket makes ``urlsplit`` raise ``ValueError: Invalid IPv6 URL`` instead+ of returning a parse. Unlike its sibling in ``_classify_engine_url``+ (which already wraps this identical call), the ``urlsplit`` call here+ used to be bare, so the exception propagated out of+ ``_is_local_filesystem_uri`` and up through+ ``_vector_store_endpoint_is_local`` -> ``vector_store_is_contained`` ->+ ``classify_engine`` into ``_resolve_adaptive_scope``'s catch-all, which+ resolves the PERMISSIVE ``EgressScope.BOTH`` — exactly the cloud-capable+ scope this module's fail-up rules exist to avoid admitting a remote+ endpoint into. Must instead fail toward not-local, like every other+ unrecognised form in this function."""+ for value in (+ "file://[attacker.example.com/share/db",+ "file://[::1/db",+ ):+ assert P._is_local_filesystem_uri(value) is False+++def test_malformed_file_uri_does_not_widen_adaptive_scope_to_both():+ """End-to-end: a private, non-contained collection whose store endpoint+ is a malformed ``file://[...`` URI must resolve ADAPTIVE to the+ restrictive PRIVATE_ONLY, not the permissive BOTH the pre-fix+ ``ValueError`` used to force via the catch-all in+ ``_resolve_adaptive_scope``.++ What this pins is the END-TO-END outcome, not any one guard: today the+ ``ValueError`` is absorbed by ``_file_uri_names_a_remote_share`` in+ ``_vector_store_endpoint_is_local``, one frame BEFORE+ ``_is_local_filesystem_uri`` is reached, so reverting the ``try``/+ ``except`` inside ``_is_local_filesystem_uri`` would NOT flip this+ assertion. That revert is caught by+ ``test_malformed_file_uri_authority_does_not_raise`` (the unit test+ above); this test catches the loss of the fail-closed *result* through+ whichever frame absorbs it."""+ malformed_uri = {+ URI_SETTING: "file://[attacker.example.com/share/db",+ }+ with _store(_RemoteStore):+ resolved = P._resolve_adaptive_scope(+ "collection_x",+ malformed_uri,+ username=None,+ local_hostnames=(),+ )+ assert resolved == P.EgressScope.PRIVATE_ONLY+ assert resolved != P.EgressScope.BOTH+++def test_single_letter_scheme_authority_is_not_a_drive_path():+ """``x://host:port`` must NOT be classified as a Windows drive-absolute+ path just because it matches ``<letter>:<sep>``: what follows the+ "drive letter" is a DOUBLE separator ("//"), which is a URL authority+ marker, not a path — so a vector-store endpoint written this way must+ fall through to the DNS classifier instead of short-circuiting local.+ ``str.isalpha()`` is Unicode-wide, so a Cyrillic or fullwidth lookalike+ scheme letter must be rejected the same way."""+ for value in (+ "x://attacker.example.com:19530",+ "с://attacker.example.com:19530", # Cyrillic "es", not ASCII "c"+ "\uff23://attacker.example.com:19530", # fullwidth "C"+ ):+ assert P._is_local_filesystem_uri(value) is False+ # Real Windows drive-absolute paths (single separator after the drive+ # letter, no authority) are unaffected.+ assert P._is_local_filesystem_uri("C:\\data\\store.db") is True+ assert P._is_local_filesystem_uri("C:/data/store.db") is True+ assert P._is_local_filesystem_uri("file://C:/data/store.db") is True+++def test_single_letter_scheme_endpoint_is_not_contained():+ """End-to-end: a store whose ``uri_setting`` uses a single-letter-scheme+ authority (rejected as a local FILE path by the test above) must be+ classified REMOTE by the DNS classifier it falls through to, not+ silently treated as an on-box drive path."""+ drive_lookalike = {URI_SETTING: "x://93.184.216.34:19530"}+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained("library", ctx(), drive_lookalike)+ is False+ )+++def test_unicode_drive_letter_path_is_not_a_local_file():+ """``<letter>:/path`` is a Windows drive-absolute path ONLY for an ASCII+ drive letter. ``str.isalpha()`` is Unicode-wide, so "\u0441:/data/store.db"+ (Cyrillic "es"), "\uff23:/data/store.db" (fullwidth "C") and+ "\u00e9:/data/store.db" matched the drive-path branch and short-circuited+ LOCAL — skipping host classification entirely on a value no resolver+ reads as a drive path. This is the bare-path twin of the ``file://``+ netloc check pinned by+ ``test_unicode_drive_letter_authority_is_not_a_local_file``.++ WHICH REVERT EACH CASE CATCHES: dropping the ``isascii()`` guard from the+ drive-letter branch of ``_is_local_filesystem_uri`` flips every attack row+ here to True (and the end-to-end row below to contained). The literal-"//"+ authority check does NOT catch them — these have a SINGLE separator after+ the colon, so that guard never fires."""+ for value in (+ "\u0441:/data/store.db", # Cyrillic "es", not ASCII "c"+ "\uff23:/data/store.db", # fullwidth "C"+ "\u00e9:/data/store.db", # accented "e"+ "\u0441:\\data\\store.db", # backslash separator twin+ ):+ assert P._is_local_filesystem_uri(value) is False+ # Real ASCII drive-absolute paths (either case, either separator) are+ # unaffected.+ for value in ("C:/data/store.db", "c:\\data\\store.db", "Z:/x"):+ assert P._is_local_filesystem_uri(value) is True+++def test_unicode_drive_letter_path_endpoint_is_not_contained():+ """End-to-end twin of the test above: a store whose ``uri_setting`` is a+ Unicode drive-letter lookalike PATH must not be silently treated as+ on-box. ``allow_dns=False`` keeps this off the network — the lookalike is+ not a literal IP, so the classifier it falls through to answers+ "undetermined", which is not contained. Restoring the Unicode-wide+ ``isalpha()`` makes the file-path check short-circuit LOCAL instead and+ fails this."""+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library",+ ctx(),+ {URI_SETTING: "\u0441:/data/store.db"},+ allow_dns=False,+ )+ is False+ )+++def test_unicode_drive_letter_authority_is_not_a_local_file():+ """``file://<letter>:/path`` is the Windows drive-authority form ONLY+ for an ASCII drive letter. ``str.isalpha()`` is Unicode-wide, so a+ Cyrillic, fullwidth, or accented lookalike authority names a HOST to+ any resolver that reads the netloc, not an on-box drive. Restoring the+ Unicode-wide check (dropping the ``isascii()`` guard) re-classifies all+ three attack forms LOCAL and fails these assertions."""+ for value in (+ "file://\u0441:/share/db", # Cyrillic "es", not ASCII "c"+ "file://\uff23:/share/db", # fullwidth "C"+ "file://\u00e9:/share/db", # accented "e"+ ):+ assert P._is_local_filesystem_uri(value) is False+ # Real Windows drive-letter authorities (ASCII, either case) are+ # unaffected.+ assert P._is_local_filesystem_uri("file://C:/data/store.db") is True+ assert P._is_local_filesystem_uri("file://c:/data/store.db") is True+++def test_unicode_drive_letter_authority_endpoint_is_not_contained():+ """End-to-end: a store whose ``uri_setting`` uses a Unicode drive-letter+ lookalike authority (rejected as a local FILE path by the test above)+ must NOT be silently treated as an on-box drive path. ``allow_dns=False``+ keeps this test off the network: the lookalike authority is not a literal+ IP, so the DNS classifier it falls through to answers "undetermined",+ which is not contained — restoring the Unicode-wide ``isalpha()`` makes+ the file-path check short-circuit LOCAL instead and fails this."""+ unicode_drive = {URI_SETTING: "file://\u0441:/share/db"}+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), unicode_drive, allow_dns=False+ )+ is False+ )+ # The ASCII drive-authority form (either case) stays contained.+ for local_uri in ("file://C:/data/store.db", "file://c:/data/store.db"):+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: local_uri}+ )+ is True+ )+++# ---------------------------------------------------------------------------+# Parser differential: the guards and ``urlsplit`` must read the same string+# ---------------------------------------------------------------------------++# Each entry is (id, uri). Every one names ``attacker.example.com`` off-box+# while carrying a control character that ``urllib.parse.urlsplit`` deletes+# (tab/CR/LF anywhere, leading C0 controls) but ``str.strip()`` does not.+_PARSER_DIFFERENTIAL_URIS = (+ ("tab_in_file_scheme", "fi\tle://localhost//attacker.example.com/share/db"),+ ("cr_in_file_scheme", "fi\rle://localhost//attacker.example.com/share/db"),+ ("lf_in_file_scheme", "fi\nle://localhost//attacker.example.com/share/db"),+ ("nul_before_file_scheme", "\x00file://localhost//attacker.example.com/db"),+ ("c0_before_file_scheme", "\x01file://localhost//attacker.example.com/db"),+ ("tab_in_drive_separator", "x:/\t/attacker.example.com:19530"),+ ("cr_in_drive_separator", "x:/\r/attacker.example.com:19530"),+ ("tab_in_bare_double_separator", "/\t/attacker.example.com/share/db"),+)+++ "uri",+ [uri for _, uri in _PARSER_DIFFERENTIAL_URIS],+ ids=[name for name, _ in _PARSER_DIFFERENTIAL_URIS],+)+def test_control_characters_cannot_smuggle_a_remote_store_past_the_guards(uri):+ """A control character must not split the containment guards away from+ the URL parser that decides the host.++ ``urlsplit`` follows the WHATWG parser: it DELETES every tab, CR and LF+ from anywhere in a URL and lstrips C0 controls. ``str.strip()`` removes+ neither ``"\\x00"`` nor a tab in the middle of a string. So while+ ``_file_uri_names_a_remote_share`` and ``_is_local_filesystem_uri``+ judged the RAW value, each URI here was simultaneously "not a+ ``file://`` URI" / "a local drive path" to them and a local-looking+ ``file://`` URI / an ``attacker.example.com:19530`` authority to+ ``_classify_engine_url`` — every one of them classified CONTAINED.++ WHICH REVERT EACH CASE CATCHES (measured, one independent cause per+ row):++ * ``*_drive_separator`` / ``*_double_separator`` — held ONLY by the+ ``_normalize_uri_like_url_parser`` call in ``_is_local_filesystem_uri``.+ Revert that to ``value.strip()`` and all three short-circuit LOCAL with+ no host classification at all. The parsed-scheme gate is irrelevant to+ them (their scheme is ``x``/empty, never ``file``).+ * ``*_file_scheme`` — held ONLY by the parsed-scheme gate in+ ``_file_uri_names_a_remote_share``. Revert that to+ ``value.lower().startswith("file://")`` and all five classify+ CONTAINED. The ``_is_local_filesystem_uri`` normalisation is irrelevant+ to them: it answers "not a local file" for these either way, and the+ containment then comes from the classifier, not from it.++ ``allow_dns=False`` keeps this off the network, but it does NOT by itself+ make these values non-contained. The two ``*_drive_separator`` rows reach+ ``_classify_engine_url`` as+ ``"http://x:/<CTL>/attacker.example.com:19530"``, whose hostname is the+ bare ``"x"`` — not a literal IP, so the classifier answers+ "undetermined". ``tab_in_bare_double_separator`` reaches it as+ ``"http:///<TAB>/attacker.example.com/share/db"``, which after the+ parser's tab deletion has netloc ``""`` and ``hostname is None``, so the+ entry is skipped for having no host at all. The ``*_file_scheme`` values+ parse to authority ``localhost``, which classifies LOCAL with no DNS at+ all — that is exactly why the network-share rejection has to see them,+ and why reverting the gate flips those five rows to contained."""+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: uri}, allow_dns=False+ )+ is False+ )+++def test_file_uri_scheme_is_decided_by_the_parser_not_the_prefix():+ """``_file_uri_names_a_remote_share`` must ask ``urlsplit`` what the+ scheme is, not ``str.startswith("file://")``.++ ``file:%2f%2fhost/share`` has scheme ``file`` and a path that decodes to+ the UNC prefix ``//host/share``, but it never contains the literal+ ``"file://"``, so a raw-prefix gate declines to look at it at all.+ Reverting the gate to that prefix test flips these three assertions to+ False. For THESE percent-encoded forms the gate is defence in depth —+ end-to-end they still fail closed via the DNS classifier, which finds no+ local host in them — but the same gate is load-bearing for the+ control-character forms above, whose parsed authority is ``localhost``:+ see+ ``test_control_characters_cannot_smuggle_a_remote_store_past_the_guards``,+ where reverting it flips all five ``*_file_scheme`` rows to CONTAINED.+ Note this function DOES pre-normalise its input, and that call is a+ second guard rather than a no-op: ``urlsplit`` lstrips only+ ``\\x00``-``\\x20``, while the normaliser's final ``strip()`` is+ bidirectional and also removes the 19 leading Unicode-whitespace+ codepoints (U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029,+ U+202F, U+205F, U+3000) the parser leaves in place. Drop it and+ "\\xa0file://localhost//attacker.example.com/share/db" parses with an+ EMPTY scheme, so the network-share rejection never sees it — see+ ``test_leading_unicode_whitespace_cannot_hide_a_network_share``."""+ for attack_uri in (+ "file:%2f%2fattacker.example.com/share/db",+ "file:/%2f%2fattacker.example.com/share/db",+ "FILE:%5C%5Cattacker.example.com/share/db",+ ):+ assert P._file_uri_names_a_remote_share(attack_uri) is True+ # A scheme-gated check must not start rejecting the ordinary forms.+ for local_uri in (+ "file:///var/lib/store.db",+ "file://localhost/var/lib/store.db",+ "file://C:/data/store.db",+ "file:relative/store.db",+ ):+ assert P._file_uri_names_a_remote_share(local_uri) is False+++# Leading whitespace ``str.strip()`` removes and ``urlsplit`` does NOT.+# ``urlsplit`` follows the WHATWG parser and lstrips only "\x00"-"\x20";+# ``str.strip()`` additionally removes 19 Unicode-whitespace codepoints, and+# a leading one of those makes ``urlsplit`` report an EMPTY scheme.+_LEADING_UNICODE_WHITESPACE = (+ ("nbsp", "\xa0"),+ ("nel", "\x85"),+ ("ideographic_space", "\u3000"),+ ("thin_space", "\u2009"),+)+_SHARE_URI = "file://localhost//attacker.example.com/share/db"+++ "prefix",+ [prefix for _, prefix in _LEADING_UNICODE_WHITESPACE],+ ids=[name for name, _ in _LEADING_UNICODE_WHITESPACE],+)+def test_leading_unicode_whitespace_cannot_hide_a_network_share(prefix):+ """Leading Unicode whitespace must not hide a UNC share from the parser.++ ``_file_uri_names_a_remote_share`` asks ``urlsplit`` for the scheme, and+ ``urlsplit``'s lstrip covers only "\\x00"-"\\x20". Every prefix here is+ OUTSIDE that range, so on the RAW value the scheme parses as "" — not+ "file" — and the network-share rejection declines to look at the URI at+ all. The ``_normalize_uri_like_url_parser`` call in that function is what+ closes this: its final ``strip()`` is bidirectional and removes exactly+ these codepoints before the parse.++ The LIST shape is where it bites end to end. On the raw value the entry+ also parses to ``hostname is None``, so ``_classify_engine_url`` skips it+ entirely and the sibling "127.0.0.1" — a loopback IP literal, classified+ LOCAL with NO DNS — makes the whole setting answer CONTAINED.++ Measured on every prefix here: drop the ``_normalize_uri_like_url_parser``+ call and hand ``value`` to ``urlsplit`` raw, and BOTH assertions below+ flip — the direct one to False, the containment one to True.+ """+ attack = prefix + _SHARE_URI+ assert P._file_uri_names_a_remote_share(attack) is True+ with _store(_RemoteStore):+ for value in ([attack, "127.0.0.1"], {"value": [attack, "127.0.0.1"]}):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: value}, allow_dns=False+ )+ is False+ ), value+++# Each entry is (id, raw, expected). The expectation is an EXPLICIT string,+# not "whatever ``urlsplit`` happens to agree with": an+# ``urlsplit(f(raw)) == urlsplit(raw)`` assertion holds for the identity+# function and for ``str.strip()`` on every row here EXCEPT+# ``trailing_whitespace_trimmed`` (``urlsplit`` keeps the trailing spaces in+# ``path``, so ``str.strip()`` changes the parse there), so it can only ever+# catch OVER-normalisation, never the under-normalisation that is the+# security direction.+_NORMALISER_CASES = (+ # Interior tab/CR/LF deletion — ``str.strip()`` never touches these, so+ # each of these rows fails under BOTH ``str.strip()`` and the identity.+ (+ "interior_tab_deleted",+ "fi\tle://localhost//attacker.example.com/db",+ "file://localhost//attacker.example.com/db",+ ),+ (+ "interior_cr_lf_deleted",+ "http://loc\r\nalhost:19530",+ "http://localhost:19530",+ ),+ (+ "tab_in_drive_separator_deleted",+ "x:/\t/attacker.example.com:19530",+ "x://attacker.example.com:19530",+ ),+ # Leading NON-whitespace C0 controls — ``str.strip()`` stops at the first+ # of them, so these rows fail under ``str.strip()`` and the identity too.+ (+ "leading_c0_stripped",+ "\x01\x1bfile://localhost/db",+ "file://localhost/db",+ ),+ (+ "leading_nul_after_whitespace_stripped",+ "\t\x00 file:///var/lib/store.db",+ "file:///var/lib/store.db",+ ),+ # Trailing whitespace: the deliberate delta from ``urlsplit`` (which+ # lstrips only). Fails under the identity, passes under ``str.strip()``.+ (+ "trailing_whitespace_trimmed",+ "file:///var/lib/store.db ",+ "file:///var/lib/store.db",+ ),+ # Over-normalisation controls: an INTERIOR space and a percent-encoded+ # tab are part of the path and must survive untouched. These are the rows+ # that fail if the normaliser starts removing more than the parser does.+ (+ "interior_space_kept",+ "file:///home/u/my files/store.db",+ "file:///home/u/my files/store.db",+ ),+ (+ "percent_encoded_tab_kept",+ "file:///var/lib/%09store.db",+ "file:///var/lib/%09store.db",+ ),+)+++ ("raw", "expected"),+ [(raw, expected) for _, raw, expected in _NORMALISER_CASES],+ ids=[name for name, _, _ in _NORMALISER_CASES],+)+def test_normaliser_output_is_the_string_urlsplit_would_parse(raw, expected):+ """The normaliser's OUTPUT is pinned to an explicit string per row.++ ``urlsplit`` follows the WHATWG parser: it lstrips C0 controls and space,+ then deletes every tab, CR and LF from anywhere in the URL. The+ normaliser must reproduce that (plus a trailing trim the callers relied+ on before it existed).++ WHICH REPLACEMENT EACH ROW CATCHES: the three deletion rows and the two+ leading-C0 rows fail under BOTH ``str.strip()`` and the identity+ function — ``str.strip()`` never removes an interior tab/CR/LF and stops+ at the first non-whitespace C0 character. ``trailing_whitespace_trimmed``+ fails under the identity only (``str.strip()`` already trims it). The two+ ``*_kept`` rows fail only under an OVER-normaliser: an interior space and+ a ``%09`` are path content the parser preserves, and stripping them would+ be the opposite mistake.++ Deliberately NOT written as ``urlsplit(f(raw)) == urlsplit(raw)``: that+ assertion holds for the identity function and for ``str.strip()`` on+ every row here except ``trailing_whitespace_trimmed`` — for that one row+ ``urlsplit`` keeps the trailing spaces in ``path``+ (``urlsplit("file:///var/lib/store.db ").path`` is+ ``"/var/lib/store.db "``), so ``str.strip()`` does change the parse. On+ every other row the assertion can only catch over-normalisation, never+ the under-normalisation this test exists to catch."""+ assert P._normalize_uri_like_url_parser(raw) == expected+++# Positive control for the tests above: the ordinary, control-character-+# free forms must keep the classification they had before the normalisation+# was introduced. A normaliser that over-strips would show up here.+_BENIGN_LOCAL_URIS = (+ "/var/lib/ldr/store.db",+ "file:///var/lib/ldr/store.db",+ "file://localhost/var/lib/ldr/store.db",+ "file://C:/data/store.db",+ "C:\\data\\store.db",+ "C:/data/store.db",+ "C:\\\\data\\store.db",+ "C:/\\data\\store.db",+ "C:\\/data/store.db",+ "./data/store.db",+ "~/data/store.db",+ "file:///var/lib/%2fdir/store.db",+ "file:///%252fdir/store.db",+)++[email protected]("uri", _BENIGN_LOCAL_URIS)+def test_benign_local_uris_stay_contained_after_normalisation(uri):+ """Positive control: normalising the string the guards read must not+ reclassify any ordinary local path or ``file://`` URI. Covers all four+ drive-separator truth-table rows (``C:\\\\``, ``C:/\\``, ``C:\\/``,+ ``C:/``) plus the RFC 8089 and relative-path forms. ``allow_dns=False``+ keeps it off the network — every URI here must be answered by the+ filesystem-path check alone, so a regression that pushes one into the+ DNS classifier fails this too."""+ assert P._is_local_filesystem_uri(uri) is True+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: uri}, allow_dns=False+ )+ is True+ )+++def test_file_uri_percent_encoded_unc_in_path_is_not_a_local_file():+ """A ``file://`` URI may percent-encode its path (RFC 8089), so the+ leading-separator check must run on the DECODED path:+ "file:///%2fattacker.example.com/share/db" and the "%5c" twin read+ as single-separator paths while encoded, but ``unquote`` yields+ exactly the "//host" / "/\\host" UNC prefixes that+ ``ntpath.normpath`` resolves to a remote share on Windows. Reverting+ the decode (checking the literal ``parsed.path`` again, as before+ the fix) makes both attack forms classify LOCAL again — these+ assertions are what catch that revert."""+ assert (+ P._is_local_filesystem_uri("file:///%2fattacker.example.com/share/db")+ is False+ )+ assert (+ P._is_local_filesystem_uri("file:///%5cattacker.example.com/share/db")+ is False+ )+ # The mixed-separator combination of the same encoding, and the+ # localhost-authority form: the netloc check passes for both, so only+ # the decoded-path separator check stands between them and a LOCAL+ # classification.+ assert (+ P._is_local_filesystem_uri("file:///%5c/attacker.example.com/share/db")+ is False+ )+ assert (+ P._is_local_filesystem_uri(+ "file://localhost/%2fattacker.example.com/share/db"+ )+ is False+ )+ # The RFC 8089 local forms stay local after the decode — including a+ # path whose percent-encoding sits AFTER the leading separator, where+ # decoding changes nothing about locality.+ assert P._is_local_filesystem_uri("file:///var/lib/store.db") is True+ assert (+ P._is_local_filesystem_uri("file://localhost/var/lib/store.db") is True+ )+ assert P._is_local_filesystem_uri("file://C:/data/store.db") is True+ assert P._is_local_filesystem_uri("file:///C:/data/store.db") is True+ assert (+ P._is_local_filesystem_uri("file:///home/u/my%20files/store.db") is True+ )+++def test_file_uri_percent_encoded_unc_endpoint_is_not_contained():+ """End-to-end: a store whose ``uri_setting`` percent-encodes the UNC+ prefix of its path must be classified REMOTE, exactly like the literal+ four-slash / mixed-slash forms.++ What this pins is the END-TO-END outcome, not the ``unquote`` inside+ ``_is_local_filesystem_uri``: ``_vector_store_endpoint_is_local`` runs+ the same decode-and-check via ``_file_uri_names_a_remote_share`` one+ frame earlier, so reverting that ``unquote`` alone would NOT flip these+ assertions. That revert is caught by+ ``test_file_uri_percent_encoded_unc_in_path_is_not_a_local_file`` (the+ unit test above); this test catches the loss of the fail-closed+ *result* through whichever frame decodes."""+ for attack_uri in (+ "file:///%2fattacker.example.com/share/db",+ "file:///%5cattacker.example.com/share/db",+ ):+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: attack_uri}+ )+ is False+ )+ # The plain RFC 8089 local forms remain contained.+ for local_uri in (+ "file:///var/lib/store.db",+ "file://localhost/var/lib/store.db",+ "file://C:/data/store.db",+ ):+ with _store(_RemoteStore):+ assert (+ P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: local_uri}+ )+ is True+ )+++def test_rejected_file_paths_cannot_be_allowed_by_hostname_fallback():+ """A nominally local authority cannot make a decoded share path local."""+ rejected_uris = (+ "file://localhost/%2freview.invalid/share/db",+ "file://localhost/%5creview.invalid/share/db",+ "file://localhost//review.invalid/share/db",+ "FILE://LOCALHOST/%5c/review.invalid/share/db",+ "file://127.0.0.1/%2freview.invalid/share/db",+ "file://C:/%2freview.invalid/share/db",+ "file://[::1/db",+ )+ with (+ _store(_RemoteStore),+ patch(+ f"{POLICY}._classify_engine_url", return_value=True+ ) as classify_url,+ ):+ for uri in rejected_uris:+ assert not P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: uri}+ ), uri+ # A setting value need not be a STRING. ``unwrap_setting``+ # returns whatever the snapshot held, uncoerced, and+ # ``_classify_engine_url`` explicitly supports the list shape,+ # so the rejection must run per ENTRY. Reverting it to the+ # string-only guard lets the one-element list skip the+ # rejection and be rescued by the local-looking authority+ # ("localhost" / "127.0.0.1" / drive letter), which the patched+ # classifier above stands in for. Because that patch makes+ # ``_classify_engine_url`` return True unconditionally, ALL SEVEN+ # flip to contained under the revert (unpatched, only the five+ # with a resolvably-local authority would — the "C:" and the+ # unbalanced-bracket forms fail hostname classification on their+ # own), and ``classify_url.assert_not_called()`` fires too.+ assert not P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: [uri]}+ ), uri+ # ...including through the ``{"value": ...}`` snapshot wrapper+ # ``_get_setting_value`` peels: unwrapping yields the same list,+ # never a string, so this shape needs the same per-entry pass.+ assert not P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: {"value": [uri]}}+ ), uri+ classify_url.assert_not_called()++ # ANY rejected entry sinks the whole setting, even alongside a+ # legitimately local endpoint: containment is a property of the+ # whole configured endpoint set, mirroring the "any public entry+ # wins" fail-up. Reverting to the string-only guard (or to a+ # first-entry-only / all-entries-must-be-rejected check) makes this+ # mixed list pass as contained via the local entry.+ assert not P.vector_store_is_contained(+ "library",+ ctx(),+ {URI_SETTING: ["http://127.0.0.1:19530", rejected_uris[0]]},+ )+ assert not P.vector_store_is_contained(+ "library",+ ctx(),+ {+ URI_SETTING: {+ "value": [LOCAL_URI[URI_SETTING], rejected_uris[2]]+ }+ },+ )+ classify_url.assert_not_called()++ # Local paths still pass without DNS, including encoded spaces.+ for uri in (+ "file:///var/lib/store.db",+ "file://localhost/var/lib/store.db",+ "file://C:/data/store.db",+ "file:///home/u/my%20files/store.db",+ ):+ assert P.vector_store_is_contained(+ "library", ctx(), {URI_SETTING: uri}+ ), uri+ classify_url.assert_not_called()++ # Network endpoints still consult the configured host classifier.+ assert P.vector_store_is_contained("library", ctx(), LOCAL_URI)+ classify_url.assert_called_once()++def test_file_uri_truth_table_full_regression():"""Full before/after truth table for ``_is_local_filesystem_uri``,pinning the RFC 8089 local forms, the round-6 fixes (4-slashUNC-in-path bypass rejected; Windows drive-letter authority accepted),the round-7 mixed-slash UNC fix (``ntpath.normpath`` treats ANY pair ofleading "/"/"\\" separators as a UNC prefix on Windows, not just "//"),- and every previously-covered row unchanged."""+ the malformed-IPv6-authority fix (a bare ``urlsplit`` used to raise+ instead of returning False), the single-letter-scheme-authority fix+ (``x://host`` used to be misread+ as a Windows drive path), the percent-encoded-UNC fix (the separator+ check now runs on the unquoted path, so a "%2f"/"%5c"-encoded UNC+ prefix can no longer classify local), the Unicode-drive-letter fix (the+ drive-authority netloc check AND the bare drive-path branch now require an+ ASCII letter, so Cyrillic/fullwidth/accented lookalikes no longer classify+ local in either position), the drive-path narrowing (only a literal "//"+ after the drive letter is a scheme authority, so doubled-/mixed-backslash+ drive paths are local again), and every previously-covered row+ unchanged."""cases = [# (uri, expected, label)(@@ -461,6 +1135,64 @@ def test_file_uri_truth_table_full_regression():False,"4-slash UNC bypass",),+ (+ "file://[attacker.example.com/share/db",+ False,+ "malformed IPv6-literal authority (unbalanced bracket)",+ ),+ (+ "file://[::1/db",+ False,+ "malformed IPv6-literal authority (truncated)",+ ),+ (+ "x://attacker.example.com:19530",+ False,+ "single-letter-scheme authority, not a drive path",+ ),+ (+ "\u0441://attacker.example.com:19530",+ False,+ "Cyrillic-letter-scheme authority",+ ),+ (+ "\uff23://attacker.example.com:19530",+ False,+ "fullwidth-letter-scheme authority",+ ),+ (+ "file://\u0441:/share/db",+ False,+ "Cyrillic drive-letter authority",+ ),+ (+ "file://\uff23:/share/db",+ False,+ "fullwidth drive-letter authority",+ ),+ (+ "file://\u00e9:/share/db",+ False,+ "accented drive-letter authority",+ ),+ # The bare-path twins: a drive-absolute path needs an ASCII drive+ # letter too, or the drive-path branch short-circuits LOCAL on a+ # value no resolver reads as a drive path.+ (+ "\u0441:/data/store.db",+ False,+ "Cyrillic drive-letter path",+ ),+ (+ "\uff23:/data/store.db",+ False,+ "fullwidth drive-letter path",+ ),+ (+ "\u00e9:\\data\\store.db",+ False,+ "accented drive-letter path (backslash)",+ ),("/\\attacker.example.com/share/store.db",False,@@ -476,13 +1208,33 @@ def test_file_uri_truth_table_full_regression():False,"mixed-slash UNC bypass (backslash-first bare path)",),+ (+ "file:///%2fattacker.example.com/share/db",+ False,+ "percent-encoded UNC bypass (%2f)",+ ),+ (+ "file:///%5cattacker.example.com/share/db",+ False,+ "percent-encoded UNC bypass (%5c)",+ ),("file://C:/data/store.db", True, "windows drive-letter authority"),+ (+ "file://c:/data/store.db",+ True,+ "windows drive-letter authority (lowercase)",+ ),("file:///var/lib/store.db", True, "RFC8089 empty authority"),("file://localhost/var/lib/store.db",True,"RFC8089 localhost authority",),+ (+ "file:///home/u/my%20files/store.db",+ True,+ "percent-encoded but plainly local path",+ ),("FILE://LOCALHOST/var/lib/x", True, "case-insensitive localhost"),("file://attacker.example.com/share/db",@@ -495,6 +1247,33 @@ def test_file_uri_truth_table_full_regression():("~/store.db", True, "home-relative"),("C:\\data\\store.db", True, "windows backslash drive path"),("C:/data/store.db", True, "windows forward-slash drive path"),+ # A drive letter always wins over a UNC prefix on Windows+ # (``ntpath.normpath("C:\\\\data") == "C:\\data"``) and ``urlsplit``+ # gives an authority only to a literal "//", so these doubled- and+ # mixed-separator drive paths cannot name a remote host under+ # either reading. Re-widening the guard to reject every doubled+ # separator makes them non-contained and locks Windows users out+ # of a legitimately local store.+ (+ "C:\\\\data\\store.db",+ True,+ "windows doubled-backslash drive path",+ ),+ ("C:/\\data/store.db", True, "windows mixed-separator drive path"),+ (+ "C:\\/data/store.db",+ True,+ "windows mixed-separator drive path (backslash first)",+ ),+ # The forward-slash twin stays rejected: "C://data/store.db" is+ # byte-for-byte a single-letter-scheme URL (scheme "c", hostname+ # "data"), indistinguishable from the "x://host" authority the row+ # above pins as remote. Documented in changelog.d/5763.security.md.+ (+ "C://data/store.db",+ False,+ "drive letter + literal // is a scheme authority",+ ),("//attacker.example.com:19530", False, "scheme-less authority"),("//host/path", False, "scheme-less authority 2"),("\\\\server\\share\\store.db", False, "UNC path"),