Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(security): follow-up fixes for #5777 egress vector-store containment (#6197)

LearningCircuit2a594e0193d2
diff --git a/changelog.d/5763.security.md b/changelog.d/5763.security.md
index ca41394108..3f60e62f6f 100644
--- a/changelog.d/5763.security.md
+++ b/changelog.d/5763.security.md
@@ -17,11 +17,30 @@ reclassified fully *public* on a public endpoint, so `private_only` alone
blocks them while `public_only`/adaptive-public still allow them; a
non-contained private collection is blocked everywhere instead, because
losing containment must only ever add restriction, never grant new
-eligibility. The two-axis run-start audit (ADR-0007) independently reaches
-the same result for a private collection: it labels a non-contained
-collection `EXPOSING` regardless of `is_public`, and a `SENSITIVE` (private)
-source may not reach an `EXPOSING` sink under that rule, so the private,
-non-contained collection is blocked by both mechanisms.
+eligibility.
+
+For the common single-engine run, the two-axis run-start audit (ADR-0007) is
+NOT an independent second line of defence here: `audit_run_from_snapshot`
+(the only production entry point, called from `web/routers/research.py` and
+`web/services/research_service.py`) always classifies just the run's primary
+engine. When that primary is a private, non-contained collection, it is the
+SOLE sensitive source in the audited set, so the quadrant-4 self-exclusion in
+`classification.py` applies (a component may reach a sink that is its own
+data) and the audit returns an ALLOW (`"sensitive_contained"`), not a denial —
+provided the run's inference sinks are themselves contained; an exposing
+LLM/embeddings sink denies the run first
+(`"sensitive_to_exposing_inference"`, checked in `classification.py` before
+the self-exclusion is considered).
+For a single-engine run the audit therefore adds nothing: it evaluates the
+primary engine only (its LLM/embeddings entries are inference sinks, not
+sensitive sources), so a second sensitive source can never appear in the
+audited set. Two independent guards still cover the case. The
+`unclassified` block above governs ENGINE ELIGIBILITY — the collection is
+denied under every scope. Separately, on the SCOPE side, a primary engine
+that classifies as neither public nor local resolves `adaptive` to the
+restrictive `private_only` rather than the permissive `both`, which keeps the
+rest of the run (LLM and embeddings included) local as well. Neither guard
+depends on the other.
A non-local-file store whose endpoint cannot be resolved is treated as
remote (not contained).
@@ -35,3 +54,67 @@ extending `trusted_search_engines` to cover vector-store endpoints, or a
per-collection "trusted remote store" flag) is a maintainer decision that
should land before server-backed vector stores (tracked in #5331) are
expected to be usable under anything other than `unprotected`.
+
+A rejected network-share path in a file URI remains non-contained even
+when its nominal authority resolves locally, and that rejection is now
+applied to every entry of the endpoint setting rather than only to a
+plain-string value — a list-shaped setting (the shape an Elasticsearch-style
+endpoint list uses, and the shape a `{"value": [...]}` snapshot unwraps to)
+no longer reaches hostname classification with a rejected path still in it.
+A single rejected entry makes the whole setting non-contained, matching the
+existing "any public entry wins" fail-up. Current FAISS configurations are
+unaffected.
+
+Windows drive-absolute store paths written with a doubled or mixed separator
+after the drive letter (`C:\\data\store.db`, `C:/\data`, `C:\/data`) are
+recognised as local again: a drive letter always wins over a UNC prefix on
+Windows and none of these forms can name a remote host. **Known limitation:**
+the forward-slash form `C://data/store.db` stays non-contained, because it is
+byte-for-byte a single-letter-scheme URL (scheme `c`, hostname `data`) and
+cannot be told apart from the `x://host` authority form the classifier must
+reject. Write such a path as `C:/data/store.db` or `C:\data\store.db`.
+**Second known limitation:** because the decoded path is what decides whether
+a `file://` URI names a network share, a `file://` URI whose FIRST path
+segment literally begins with an unescaped `%2f` or `%5c`
+(`file:///%2fdir/store.db`) is treated as non-contained. `%252f` (the correct
+RFC 3986 encoding of a literal `%2f`) and `%2f` in any later segment stay
+contained. This only affects URIs built by concatenating `"file://"` with a
+raw path; `pathlib.Path.as_uri()` never produces one.
+
+Whether a store URI is a local filesystem path, and whether it is a `file://`
+URI naming a network share, are now decided the way `urllib.parse.urlsplit`
+itself reads the value rather than from the raw setting string. `urlsplit`
+follows the WHATWG URL parser: it deletes every tab, CR and LF from *anywhere*
+in the URL and strips leading C0 control characters. `str.strip()` does
+neither of the two things that matter here — it never touches a tab, CR or LF
+in the *middle* of a value, and it leaves a leading non-whitespace C0
+character such as `\x00` or `\x01` in place. Deciding the path question on the
+raw string while the host was decided on the parsed one let a control
+character split the two readings apart: `x:/<TAB>/host:19530` and
+`/<TAB>/host/share/db` were "a local drive path" / "an ordinary absolute path"
+to the raw tests but a `host:19530` / `host` authority to the parser, so the
+endpoint short-circuited to contained without any host classification. The
+path tests now run on the same string the parser sees, and the "is this a
+`file://` URI?" question is answered by the parsed scheme, so
+`fi<TAB>le://localhost//host/share` — not a `file://` URI to a raw
+`startswith`, but one with a local-looking authority to every real parser —
+no longer slips past the network-share rejection. For the characters named
+above — a tab, CR or LF anywhere, and a leading C0 control — such values are
+now classified exactly as their control-character-free equivalents are. That
+scope is the claim: an *interior* `\x00` is not covered, because `urlsplit`
+does not delete it either, so it stays in the path and such a value can still
+classify differently from its NUL-free equivalent. That difference is not a
+route off the box — `ntpath.normpath` resolves `/\x00//host/share` to a
+single leading separator, not to a `\\host\share` UNC prefix.
+
+A Windows drive-absolute store path must now begin with an *ASCII* drive
+letter. `str.isalpha()` is Unicode-wide, so `с:/data/store.db` (Cyrillic
+"es"), `C:/data/store.db` (fullwidth) and `é:/data/store.db` previously
+short-circuited to "local file" and skipped host classification entirely.
+None of them is a drive path to any resolver. This matches the ASCII
+requirement already applied to the `file://<letter>:/path` authority form;
+real `A:`–`Z:` paths in either case are unaffected.
+
+This release does not change how `.local` and RFC1918 authorities are
+classified: they continue to count as contained, exactly as they did before
+this fix, and revisiting that is out of scope here.
diff --git a/src/local_deep_research/security/egress/policy.py b/src/local_deep_research/security/egress/policy.py
index ed956c7931..3483884c8b 100644
--- a/src/local_deep_research/security/egress/policy.py
+++ b/src/local_deep_research/security/egress/policy.py
@@ -638,11 +638,81 @@ def _starts_with_double_separator(s: str) -> bool:
return len(s) >= 2 and s[0] in "/\\" and s[1] in "/\\"
+# ``urllib.parse.urlsplit`` does NOT parse the string it is handed verbatim.
+# Following the WHATWG URL parser it first lstrips every C0 control character
+# and space, then DELETES every tab, CR and LF from ANYWHERE in the URL
+# (CPython's ``_WHATWG_C0_CONTROL_OR_SPACE`` and
+# ``_UNSAFE_URL_BYTES_TO_REMOVE``). ``str.strip()`` does neither: it leaves
+# "\x00" and the other non-whitespace C0 controls in place, and never touches
+# a tab in the MIDDLE of a string.
+_URL_PARSER_LEADING_STRIP = "".join(chr(code) for code in range(0x20)) + " "
+_URL_PARSER_REMOVED_CHARS = ("\t", "\r", "\n")
+
+
+def _normalize_uri_like_url_parser(value: str) -> str:
+ """``value`` reduced to the string ``urlsplit`` would actually parse.
+
+ Exists for the RAW string tests in ``_is_local_filesystem_uri``
+ (``startswith``, ``_starts_with_double_separator``, a drive-letter index),
+ whose verdict is consumed side by side with ``_classify_engine_url``,
+ which reads the host out of ``urlsplit``. Judging the raw string opens a
+ parser differential in which one value is simultaneously "a local
+ filesystem path" to ``str.startswith`` and "an authority" to every real
+ URL parser:
+
+ * ``"x:/\\t/attacker.example.com:19530"`` looks drive-absolute to a
+ literal ``"//"`` test (``uri[2:]`` is ``"/\\t/..."``), while
+ ``urlsplit`` deletes the tab and reports netloc
+ ``attacker.example.com:19530``.
+ * ``"/\\t/attacker.example.com/share/db"`` looks like an ordinary
+ single-slash absolute path to ``_starts_with_double_separator``, while
+ ``urlsplit`` deletes the tab and reports the same authority.
+ * A leading ``"\\x00"`` (or any other non-whitespace C0 control) survives
+ ``str.strip()`` and defeats every raw prefix test, while ``urlsplit``
+ lstrips it away.
+
+ Normalising once, here, is what keeps those predicates and ``urlsplit``
+ from drifting apart again.
+
+ A raw-string test is not the only caller that needs this. The final
+ ``strip()`` is BIDIRECTIONAL, and its LEADING half removes 19
+ Unicode-whitespace codepoints that ``urlsplit``'s WHATWG lstrip
+ (``\\x00``-``\\x20`` only) leaves in place: U+0085, U+00A0, U+1680,
+ U+2000-U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. A leading one
+ of those makes ``urlsplit`` report an EMPTY scheme, so a function that
+ asks the parser for the scheme does NOT already get this preprocessing
+ for free. The call is therefore load-bearing in
+ ``_file_uri_names_a_remote_share``: without it,
+ "\\xa0file://localhost//attacker.example.com/share/db" has scheme "" and
+ is not a ``file://`` URI to that function, the network-share rejection
+ never fires, and the entry falls through to hostname classification —
+ where a sibling entry such as "127.0.0.1" answers LOCAL and the whole
+ setting classifies contained. Never drop this call on the ground that
+ ``urlsplit`` repeats it; for the leading-whitespace half it does not.
+
+ Only the TRAILING half of that ``strip()`` is a pure caller-compatibility
+ trim: ``urlsplit`` keeps trailing whitespace where this does not, but
+ trailing whitespace can never turn a local path into an authority, so the
+ extra trim cannot make a remote endpoint look local.
+ """
+ normalized = value.lstrip(_URL_PARSER_LEADING_STRIP)
+ for char in _URL_PARSER_REMOVED_CHARS:
+ normalized = normalized.replace(char, "")
+ return normalized.strip()
+
+
def _is_local_filesystem_uri(value) -> bool:
"""True when a vector-store URI is unambiguously a local file path."""
if not isinstance(value, str):
return False
- uri = value.strip()
+ # Judge the string ``urlsplit`` would parse, never the raw one — see
+ # ``_normalize_uri_like_url_parser``. Every prefix, separator and
+ # drive-letter test below is a RAW string test, so without this the
+ # "local drive path" / "single-slash absolute path" verdicts diverge from
+ # the host ``_classify_engine_url`` goes on to classify. This call is what
+ # stops "x:/\t/attacker.example.com:19530" and
+ # "/\t/attacker.example.com/share/db" short-circuiting LOCAL.
+ uri = _normalize_uri_like_url_parser(value)
if not uri:
return False
if uri.lower().startswith("file://"):
@@ -654,10 +724,34 @@ def _is_local_filesystem_uri(value) -> bool:
# is a remote-share reference some resolvers treat as a host (e.g.
# "file://attacker.example.com/share/store.db") and must NOT be
# classified as on-box — fall through to the remote/DNS path.
- parsed = urlsplit(uri)
+ #
+ # ``urlsplit`` is NOT exception-safe: a malformed authority (e.g. an
+ # unbalanced IPv6-literal bracket, "file://[attacker.example.com/x"
+ # or "file://[::1/x") raises ``ValueError: Invalid IPv6 URL`` instead
+ # of returning a parse. Unlike its sibling in ``_classify_engine_url``
+ # (which already wraps this identical call), this call was bare, so
+ # the exception used to propagate all the way up through
+ # ``_vector_store_endpoint_is_local`` -> ``vector_store_is_contained``
+ # -> ``classify_engine`` into ``_resolve_adaptive_scope``'s catch-all,
+ # which resolves the PERMISSIVE ``EgressScope.BOTH`` — exactly the
+ # cloud-capable scope this module's fail-up rules exist to avoid.
+ # Fail toward not-local instead, matching every other unknown here.
+ try:
+ parsed = urlsplit(uri)
+ except Exception:
+ return False
netloc = parsed.netloc
+ # Windows drive letters are ASCII A-Z only. ``str.isalpha()`` is
+ # Unicode-wide (Cyrillic "с", fullwidth "C", accented "é" all
+ # match), so a Unicode lookalike authority (e.g.
+ # "file://с:/share/db") must NOT pass as an on-box drive path —
+ # the same hazard class as the single-letter-scheme lookalikes at
+ # the bottom of this function.
is_drive_letter_netloc = (
- len(netloc) == 2 and netloc[0].isalpha() and netloc[1] == ":"
+ len(netloc) == 2
+ and netloc[0].isascii()
+ and netloc[0].isalpha()
+ and netloc[1] == ":"
)
if not (
netloc == ""
@@ -676,8 +770,27 @@ def _is_local_filesystem_uri(value) -> bool:
# smuggles the same UNC authority past a forward-slash-only check:
# ``ntpath.normpath`` treats ANY pair of leading separators — "//",
# "/\\", "\\/", or "\\\\" — as a UNC prefix on Windows, so reject all
- # four combinations, not just the literal "//" one.
- return not _starts_with_double_separator(parsed.path)
+ # four combinations, not just the literal "//" one. The path may be
+ # percent-encoded (RFC 8089), so DECODE it before the separator
+ # check: "file:///%2fattacker.example.com/share/db" and the "%5c"
+ # twin read as single-separator paths while encoded, but unquote
+ # to exactly the "//host" / "/\\host" UNC prefixes the check exists
+ # to catch (``ntpath.normpath`` resolves both to a remote share).
+ # Decoding cannot smuggle an authority in: netloc was already
+ # validated above, and plain local paths ("/var/lib/x",
+ # "/C:/data") keep their single leading separator through unquote.
+ #
+ # DEFENCE IN DEPTH: this decode-and-check is a deliberate duplicate
+ # of ``_file_uri_names_a_remote_share``, which
+ # ``_vector_store_endpoint_is_local`` — the ONLY production caller of
+ # this function — already ran one frame earlier over every entry of
+ # the setting. At every production entry point the earlier check has
+ # therefore already returned "not contained", so reverting the
+ # ``unquote`` here changes no end-to-end outcome; it stays because
+ # this function is a general-purpose predicate that a future caller
+ # may reach directly, and it is pinned by its own unit test
+ # (``test_file_uri_percent_encoded_unc_in_path_is_not_a_local_file``).
+ return not _starts_with_double_separator(unquote(parsed.path))
if uri.startswith(_LOCAL_PATH_URI_PREFIXES):
return True
# A single leading "/" is a local absolute path ("/var/lib/x.db") ONLY
@@ -689,12 +802,106 @@ def _is_local_filesystem_uri(value) -> bool:
# instead of being silently treated as on-box.
if uri.startswith("/") and not _starts_with_double_separator(uri):
return True
- # Windows drive-absolute path ("C:\\data\\store.db").
+ # Windows drive-absolute path ("C:\\data\\store.db"). What follows the
+ # drive letter must not be a literal "//" — "<letter>://" is a
+ # "scheme://host" URL authority marker, not a path, so e.g.
+ # "x://attacker.example.com:19530" must NOT be misread as a local drive
+ # path and skip the DNS classifier. Note ``str.isalpha()`` is
+ # Unicode-wide (Cyrillic/fullwidth letters match too, e.g. "с://host" or
+ # "C://host"), so the drive letter must be ASCII here for the same
+ # reason the ``file://`` netloc test above requires it: "с:/x", "C:/x"
+ # and "é:/x" are not Windows drive-absolute paths, and short-circuiting
+ # them to on-box would skip the DNS classifier on a value no resolver
+ # reads as a local drive. The literal-"//" authority check below is a
+ # separate, independent guard covering the "<letter>://host" form.
+ #
+ # ONLY the literal "//" form is rejected, not every doubled separator:
+ # a drive letter always wins over a UNC prefix on Windows
+ # (``ntpath.normpath("C:\\\\data") == "C:\\data"``, never a network
+ # share), and — ON THE NORMALISED STRING this function tests —
+ # ``urlsplit`` gives an authority only to a literal "//", so
+ # "C:\\\\data\\store.db", "C:/\\data" and "C:\\/data" cannot name a
+ # remote host under either reading and are genuine local paths. That
+ # equivalence holds ONLY because ``_normalize_uri_like_url_parser`` has
+ # already deleted the tab/CR/LF and leading C0 controls that ``urlsplit``
+ # itself deletes: on the RAW string "x:/\t/attacker.example.com:19530"
+ # this literal-"//" test answers "no authority, local drive path" while
+ # ``urlsplit`` answers "netloc=attacker.example.com:19530". Never move
+ # this test back onto an un-normalised string. The
+ # forward-slash form "C://data" is irreducibly ambiguous — it parses as
+ # scheme "c" with hostname "data" — so it stays rejected and falls
+ # through to the DNS classifier; see changelog.d/5763.security.md.
return (
- len(uri) > 2 and uri[0].isalpha() and uri[1] == ":" and uri[2] in "\\/"
+ len(uri) > 2
+ and uri[0].isascii()
+ and uri[0].isalpha()
+ and uri[1] == ":"
+ and uri[2] in "\\/"
+ and not uri[2:].startswith("//")
)
+def _file_uri_names_a_remote_share(value) -> bool:
+ """True when ``value`` is a ``file://`` URI that does not stay on the box.
+
+ The DECODED path of a ``file://`` URI can name a UNC network share
+ (``//host/share``, or any of the mixed "/"/"\\" separator pairs
+ ``ntpath.normpath`` also resolves to one) regardless of what the URI's
+ nominal authority says — "file://localhost//attacker.example.com/share"
+ reads as localhost but resolves off-box. Such a URI must stay
+ non-contained even after ``_is_local_filesystem_uri`` has declined it,
+ so hostname classification cannot rescue it via the local-looking
+ authority.
+
+ The "is this a ``file://`` URI?" question is answered by the PARSED
+ scheme, not by a raw ``startswith`` on the string: ``urlsplit`` deletes
+ tab/CR/LF from anywhere in a URL and lstrips C0 controls, so
+ "fi\\tle://localhost//attacker.example.com/share/db" is not a ``file://``
+ URI to ``startswith`` but IS one to every real URL parser — and
+ ``_classify_engine_url``, the frame that decides the host, is a real URL
+ parser, which for that value reports the local-looking authority
+ ``localhost`` and would classify the endpoint contained. Asking
+ ``urlsplit`` for the scheme is what keeps this function from being
+ stepped around by such a value.
+
+ ``value`` is normalised by ``_normalize_uri_like_url_parser`` BEFORE it
+ reaches ``urlsplit``, and that call is load-bearing here, not a duplicate
+ of the parser's own preprocessing. ``urlsplit`` lstrips only
+ ``\\x00``-``\\x20``, while the normaliser's final ``strip()`` is
+ bidirectional and additionally removes the 19 leading Unicode-whitespace
+ codepoints ``str.strip()`` knows and the parser does not (U+0085, U+00A0,
+ U+1680, U+2000-U+200A, U+2028, U+2029, U+202F, U+205F, U+3000). Handed
+ the RAW value, "\\xa0file://localhost//attacker.example.com/share/db"
+ parses with an EMPTY scheme, this function answers False, the
+ network-share rejection never fires, and — inside a list-shaped endpoint
+ setting whose sibling entry is "127.0.0.1" — the whole setting classifies
+ CONTAINED. Normalise first; see
+ ``test_leading_unicode_whitespace_cannot_hide_a_network_share``.
+
+ Fails toward "remote" for a malformed authority: ``urlsplit`` raises
+ (``ValueError``: "Invalid IPv6 URL") on an unbalanced IPv6 bracket rather
+ than parsing, and an endpoint we cannot parse is one we cannot prove
+ local. The catch is broad, matching its sibling in
+ ``_is_local_filesystem_uri``, so a future parser error class cannot turn
+ a fail-closed answer into an exception that escapes into
+ ``_resolve_adaptive_scope``'s permissive catch-all.
+
+ Non-strings answer False here — they are not ``file://`` URIs, and every
+ other non-string shape already fails closed downstream by never proving
+ itself local.
+ """
+ if not isinstance(value, str):
+ return False
+ uri = _normalize_uri_like_url_parser(value)
+ try:
+ parsed = urlsplit(uri)
+ if parsed.scheme.lower() != "file":
+ return False
+ return _starts_with_double_separator(unquote(parsed.path))
+ except Exception:
+ return True
+
+
def _vector_store_endpoint_is_local(
provider: str,
settings_snapshot: Optional[dict],
@@ -714,9 +921,31 @@ def _vector_store_endpoint_is_local(
uri_setting = getattr(cls, "uri_setting", None) if cls else None
if not uri_setting:
return False
- if _is_local_filesystem_uri(
- _get_setting_value(settings_snapshot or {}, uri_setting, None)
- ):
+ uri = _get_setting_value(settings_snapshot or {}, uri_setting, None)
+ # Reject a network-share file path PER ENTRY. A setting value is not
+ # necessarily a string: ``unwrap_setting`` peels the ``{"value": ...}``
+ # snapshot wrapper and returns whatever it wrapped, uncoerced, and
+ # ``_classify_engine_url`` below explicitly supports the LIST shape
+ # (Elasticsearch-style endpoint lists). A string-only guard here would
+ # leave a one-element list to fall through to hostname classification,
+ # where a nominally local authority ("localhost", "127.0.0.1") answers
+ # True and overrules the path rejection. Normalising to entries first
+ # keeps a single rejection site that covers every shape the setting can
+ # hold, and fails the WHOLE setting closed when ANY entry is a rejected
+ # path — matching the "any public entry wins" fail-up
+ # ``_classify_engine_url`` already applies to list-typed endpoints.
+ #
+ # COUPLED WITH ``_classify_engine_url``: this loop and that function must
+ # keep the SAME container test (``isinstance(..., list)``, not
+ # ``(list, tuple)``) and the same "skip non-``str`` entries" rule. If one
+ # side alone grows tuple support, a tuple-shaped setting reaches hostname
+ # classification with a rejected network-share path still in it — the
+ # exact bypass this per-entry loop exists to close. Change both or
+ # neither.
+ for entry in uri if isinstance(uri, list) else (uri,):
+ if _file_uri_names_a_remote_share(entry):
+ return False
+ if _is_local_filesystem_uri(uri):
return True
return (
_classify_engine_url(
@@ -1041,6 +1270,12 @@ def _classify_engine_url(
if not value:
return None
+ # COUPLED WITH the per-entry network-share rejection in
+ # ``_vector_store_endpoint_is_local``: both must keep the SAME container
+ # test (``isinstance(..., list)``, not ``(list, tuple)``) and the same
+ # "skip non-``str`` entries" rule. Widening only one side lets a
+ # tuple-shaped setting reach this classifier with an entry the other side
+ # never examined. Change both or neither.
entries = value if isinstance(value, list) else [value]
any_public = False
any_local = False
@@ -1799,12 +2034,18 @@ def _resolve_adaptive_scope(
# regardless of the scope resolved here; PRIVATE_ONLY's role is to keep
# every OTHER engine/inference path in the run local too, rather than
# letting the run silently widen to cloud-capable BOTH. This branch is
- # reached via the private-non-contained-collection fail-up above (or a
- # static engine that explicitly declares BOTH flags False) — NOT by a
- # static engine that simply forgot to declare its flags: ``_engine_flags``
- # reads missing attributes as ``None`` (not ``False``), so an undeclared
- # engine classifies as ``(None, None)`` and falls to the retriever-lookup
- # branch below instead. Dedup/log via the same per-(user, primary) mechanism
+ # reached via the private-non-contained-collection fail-up above, or by a
+ # static engine whose class overrides NEITHER flag: ``BaseSearchEngine``
+ # declares ``is_public = False`` and ``is_local = False`` as CLASS
+ # DEFAULTS (``search_engine_base.py``), so such a subclass inherits
+ # ``(False, False)`` from the base class — it need not explicitly write
+ # either assignment itself — and lands HERE, not in the ``(None, None)``
+ # branch below. ``_engine_flags`` only returns ``None`` for a flag that is
+ # genuinely absent from the class's MRO, which no attribute on
+ # ``BaseSearchEngine`` ever is; all 29 current registry engines override
+ # exactly one flag to ``True`` and so don't reach this branch today, but
+ # the next engine author who forgets to override either flag will, not
+ # the retriever-lookup fallback. Dedup/log via the same per-(user, primary) mechanism
# as the fail-open warning below — this fires on the identical hot path
# (once per research-run start AND once per fetch-gate check).
if classification.is_public is False and classification.is_local is False:
diff --git a/tests/security/test_egress_vector_store_exposure.py b/tests/security/test_egress_vector_store_exposure.py
index 44960609b3..59323fc80e 100644
--- a/tests/security/test_egress_vector_store_exposure.py
+++ b/tests/security/test_egress_vector_store_exposure.py
@@ -19,6 +19,8 @@ from __future__ import annotations
from contextlib import ExitStack
from unittest.mock import patch
+import pytest
+
from local_deep_research.security.egress import policy as P
from local_deep_research.security.egress import run_classification as rc
from local_deep_research.security.egress.classification import (
@@ -335,13 +337,31 @@ def test_recorded_index_provider_tightens_the_settings_default():
def test_index_provider_lookup_failure_is_soft():
"""An unavailable DB must not break classification: the settings-resolved
- provider (FAISS today) remains the primary signal."""
- with patch(
- "local_deep_research.database.session_context.get_user_db_session",
- side_effect=RuntimeError("db unavailable"),
- ):
- assert P._resolve_index_vector_store_providers("library", None) == set()
- assert P.vector_store_is_contained("library", ctx(), {}) is True
+ provider (FAISS today) remains the primary signal.
+
+ ``RAGIndex.vector_store_provider`` does not exist on the CURRENT schema,
+ so the ``column is None`` guard in
+ ``_resolve_index_vector_store_providers`` returns ``set()`` before
+ ``get_user_db_session`` is ever called — patching only the session opener
+ (as this test used to) never reaches the DB call or its ``except``, so it
+ passed unconditionally regardless of what was patched. Patch the column
+ onto the model too (``create=True`` — the attribute is genuinely absent
+ today, this is simulating the future schema) so the test actually drives
+ execution into the ``with get_user_db_session(...)`` block and exercises
+ the failure path this test is named for.
+ """
+ from local_deep_research.database.models.library import RAGIndex
+
+ with patch.object(RAGIndex, "vector_store_provider", object(), create=True):
+ with patch(
+ "local_deep_research.database.session_context.get_user_db_session",
+ side_effect=RuntimeError("db unavailable"),
+ ):
+ assert (
+ P._resolve_index_vector_store_providers("library", None)
+ == set()
+ )
+ assert P.vector_store_is_contained("library", ctx(), {}) is True
# ---------------------------------------------------------------------------
@@ -447,13 +467,667 @@ def test_file_uri_windows_drive_authority_is_a_local_file():
assert P._is_local_filesystem_uri("FILE://c:/data/store.db") is True
+def test_malformed_file_uri_authority_does_not_raise():
+ """A ``file://`` URI whose authority is an unbalanced IPv6-literal
+ bracket makes ``urlsplit`` raise ``ValueError: Invalid IPv6 URL`` instead
+ of returning a parse. Unlike its sibling in ``_classify_engine_url``
+ (which already wraps this identical call), the ``urlsplit`` call here
+ used to be bare, so the exception propagated out of
+ ``_is_local_filesystem_uri`` and up through
+ ``_vector_store_endpoint_is_local`` -> ``vector_store_is_contained`` ->
+ ``classify_engine`` into ``_resolve_adaptive_scope``'s catch-all, which
+ resolves the PERMISSIVE ``EgressScope.BOTH`` — exactly the cloud-capable
+ scope this module's fail-up rules exist to avoid admitting a remote
+ endpoint into. Must instead fail toward not-local, like every other
+ unrecognised form in this function."""
+ for value in (
+ "file://[attacker.example.com/share/db",
+ "file://[::1/db",
+ ):
+ assert P._is_local_filesystem_uri(value) is False
+
+
+def test_malformed_file_uri_does_not_widen_adaptive_scope_to_both():
+ """End-to-end: a private, non-contained collection whose store endpoint
+ is a malformed ``file://[...`` URI must resolve ADAPTIVE to the
+ restrictive PRIVATE_ONLY, not the permissive BOTH the pre-fix
+ ``ValueError`` used to force via the catch-all in
+ ``_resolve_adaptive_scope``.
+
+ What this pins is the END-TO-END outcome, not any one guard: today the
+ ``ValueError`` is absorbed by ``_file_uri_names_a_remote_share`` in
+ ``_vector_store_endpoint_is_local``, one frame BEFORE
+ ``_is_local_filesystem_uri`` is reached, so reverting the ``try``/
+ ``except`` inside ``_is_local_filesystem_uri`` would NOT flip this
+ assertion. That revert is caught by
+ ``test_malformed_file_uri_authority_does_not_raise`` (the unit test
+ above); this test catches the loss of the fail-closed *result* through
+ whichever frame absorbs it."""
+ malformed_uri = {
+ URI_SETTING: "file://[attacker.example.com/share/db",
+ }
+ with _store(_RemoteStore):
+ resolved = P._resolve_adaptive_scope(
+ "collection_x",
+ malformed_uri,
+ username=None,
+ local_hostnames=(),
+ )
+ assert resolved == P.EgressScope.PRIVATE_ONLY
+ assert resolved != P.EgressScope.BOTH
+
+
+def test_single_letter_scheme_authority_is_not_a_drive_path():
+ """``x://host:port`` must NOT be classified as a Windows drive-absolute
+ path just because it matches ``<letter>:<sep>``: what follows the
+ "drive letter" is a DOUBLE separator ("//"), which is a URL authority
+ marker, not a path — so a vector-store endpoint written this way must
+ fall through to the DNS classifier instead of short-circuiting local.
+ ``str.isalpha()`` is Unicode-wide, so a Cyrillic or fullwidth lookalike
+ scheme letter must be rejected the same way."""
+ for value in (
+ "x://attacker.example.com:19530",
+ "с://attacker.example.com:19530", # Cyrillic "es", not ASCII "c"
+ "\uff23://attacker.example.com:19530", # fullwidth "C"
+ ):
+ assert P._is_local_filesystem_uri(value) is False
+ # Real Windows drive-absolute paths (single separator after the drive
+ # letter, no authority) are unaffected.
+ assert P._is_local_filesystem_uri("C:\\data\\store.db") is True
+ assert P._is_local_filesystem_uri("C:/data/store.db") is True
+ assert P._is_local_filesystem_uri("file://C:/data/store.db") is True
+
+
+def test_single_letter_scheme_endpoint_is_not_contained():
+ """End-to-end: a store whose ``uri_setting`` uses a single-letter-scheme
+ authority (rejected as a local FILE path by the test above) must be
+ classified REMOTE by the DNS classifier it falls through to, not
+ silently treated as an on-box drive path."""
+ drive_lookalike = {URI_SETTING: "x://93.184.216.34:19530"}
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained("library", ctx(), drive_lookalike)
+ is False
+ )
+
+
+def test_unicode_drive_letter_path_is_not_a_local_file():
+ """``<letter>:/path`` is a Windows drive-absolute path ONLY for an ASCII
+ drive letter. ``str.isalpha()`` is Unicode-wide, so "\u0441:/data/store.db"
+ (Cyrillic "es"), "\uff23:/data/store.db" (fullwidth "C") and
+ "\u00e9:/data/store.db" matched the drive-path branch and short-circuited
+ LOCAL — skipping host classification entirely on a value no resolver
+ reads as a drive path. This is the bare-path twin of the ``file://``
+ netloc check pinned by
+ ``test_unicode_drive_letter_authority_is_not_a_local_file``.
+
+ WHICH REVERT EACH CASE CATCHES: dropping the ``isascii()`` guard from the
+ drive-letter branch of ``_is_local_filesystem_uri`` flips every attack row
+ here to True (and the end-to-end row below to contained). The literal-"//"
+ authority check does NOT catch them — these have a SINGLE separator after
+ the colon, so that guard never fires."""
+ for value in (
+ "\u0441:/data/store.db", # Cyrillic "es", not ASCII "c"
+ "\uff23:/data/store.db", # fullwidth "C"
+ "\u00e9:/data/store.db", # accented "e"
+ "\u0441:\\data\\store.db", # backslash separator twin
+ ):
+ assert P._is_local_filesystem_uri(value) is False
+ # Real ASCII drive-absolute paths (either case, either separator) are
+ # unaffected.
+ for value in ("C:/data/store.db", "c:\\data\\store.db", "Z:/x"):
+ assert P._is_local_filesystem_uri(value) is True
+
+
+def test_unicode_drive_letter_path_endpoint_is_not_contained():
+ """End-to-end twin of the test above: a store whose ``uri_setting`` is a
+ Unicode drive-letter lookalike PATH must not be silently treated as
+ on-box. ``allow_dns=False`` keeps this off the network — the lookalike is
+ not a literal IP, so the classifier it falls through to answers
+ "undetermined", which is not contained. Restoring the Unicode-wide
+ ``isalpha()`` makes the file-path check short-circuit LOCAL instead and
+ fails this."""
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library",
+ ctx(),
+ {URI_SETTING: "\u0441:/data/store.db"},
+ allow_dns=False,
+ )
+ is False
+ )
+
+
+def test_unicode_drive_letter_authority_is_not_a_local_file():
+ """``file://<letter>:/path`` is the Windows drive-authority form ONLY
+ for an ASCII drive letter. ``str.isalpha()`` is Unicode-wide, so a
+ Cyrillic, fullwidth, or accented lookalike authority names a HOST to
+ any resolver that reads the netloc, not an on-box drive. Restoring the
+ Unicode-wide check (dropping the ``isascii()`` guard) re-classifies all
+ three attack forms LOCAL and fails these assertions."""
+ for value in (
+ "file://\u0441:/share/db", # Cyrillic "es", not ASCII "c"
+ "file://\uff23:/share/db", # fullwidth "C"
+ "file://\u00e9:/share/db", # accented "e"
+ ):
+ assert P._is_local_filesystem_uri(value) is False
+ # Real Windows drive-letter authorities (ASCII, either case) are
+ # unaffected.
+ assert P._is_local_filesystem_uri("file://C:/data/store.db") is True
+ assert P._is_local_filesystem_uri("file://c:/data/store.db") is True
+
+
+def test_unicode_drive_letter_authority_endpoint_is_not_contained():
+ """End-to-end: a store whose ``uri_setting`` uses a Unicode drive-letter
+ lookalike authority (rejected as a local FILE path by the test above)
+ must NOT be silently treated as an on-box drive path. ``allow_dns=False``
+ keeps this test off the network: the lookalike authority is not a literal
+ IP, so the DNS classifier it falls through to answers "undetermined",
+ which is not contained — restoring the Unicode-wide ``isalpha()`` makes
+ the file-path check short-circuit LOCAL instead and fails this."""
+ unicode_drive = {URI_SETTING: "file://\u0441:/share/db"}
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), unicode_drive, allow_dns=False
+ )
+ is False
+ )
+ # The ASCII drive-authority form (either case) stays contained.
+ for local_uri in ("file://C:/data/store.db", "file://c:/data/store.db"):
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: local_uri}
+ )
+ is True
+ )
+
+
+# ---------------------------------------------------------------------------
+# Parser differential: the guards and ``urlsplit`` must read the same string
+# ---------------------------------------------------------------------------
+
+# Each entry is (id, uri). Every one names ``attacker.example.com`` off-box
+# while carrying a control character that ``urllib.parse.urlsplit`` deletes
+# (tab/CR/LF anywhere, leading C0 controls) but ``str.strip()`` does not.
+_PARSER_DIFFERENTIAL_URIS = (
+ ("tab_in_file_scheme", "fi\tle://localhost//attacker.example.com/share/db"),
+ ("cr_in_file_scheme", "fi\rle://localhost//attacker.example.com/share/db"),
+ ("lf_in_file_scheme", "fi\nle://localhost//attacker.example.com/share/db"),
+ ("nul_before_file_scheme", "\x00file://localhost//attacker.example.com/db"),
+ ("c0_before_file_scheme", "\x01file://localhost//attacker.example.com/db"),
+ ("tab_in_drive_separator", "x:/\t/attacker.example.com:19530"),
+ ("cr_in_drive_separator", "x:/\r/attacker.example.com:19530"),
+ ("tab_in_bare_double_separator", "/\t/attacker.example.com/share/db"),
+)
+
+
+ "uri",
+ [uri for _, uri in _PARSER_DIFFERENTIAL_URIS],
+ ids=[name for name, _ in _PARSER_DIFFERENTIAL_URIS],
+)
+def test_control_characters_cannot_smuggle_a_remote_store_past_the_guards(uri):
+ """A control character must not split the containment guards away from
+ the URL parser that decides the host.
+
+ ``urlsplit`` follows the WHATWG parser: it DELETES every tab, CR and LF
+ from anywhere in a URL and lstrips C0 controls. ``str.strip()`` removes
+ neither ``"\\x00"`` nor a tab in the middle of a string. So while
+ ``_file_uri_names_a_remote_share`` and ``_is_local_filesystem_uri``
+ judged the RAW value, each URI here was simultaneously "not a
+ ``file://`` URI" / "a local drive path" to them and a local-looking
+ ``file://`` URI / an ``attacker.example.com:19530`` authority to
+ ``_classify_engine_url`` — every one of them classified CONTAINED.
+
+ WHICH REVERT EACH CASE CATCHES (measured, one independent cause per
+ row):
+
+ * ``*_drive_separator`` / ``*_double_separator`` — held ONLY by the
+ ``_normalize_uri_like_url_parser`` call in ``_is_local_filesystem_uri``.
+ Revert that to ``value.strip()`` and all three short-circuit LOCAL with
+ no host classification at all. The parsed-scheme gate is irrelevant to
+ them (their scheme is ``x``/empty, never ``file``).
+ * ``*_file_scheme`` — held ONLY by the parsed-scheme gate in
+ ``_file_uri_names_a_remote_share``. Revert that to
+ ``value.lower().startswith("file://")`` and all five classify
+ CONTAINED. The ``_is_local_filesystem_uri`` normalisation is irrelevant
+ to them: it answers "not a local file" for these either way, and the
+ containment then comes from the classifier, not from it.
+
+ ``allow_dns=False`` keeps this off the network, but it does NOT by itself
+ make these values non-contained. The two ``*_drive_separator`` rows reach
+ ``_classify_engine_url`` as
+ ``"http://x:/<CTL>/attacker.example.com:19530"``, whose hostname is the
+ bare ``"x"`` — not a literal IP, so the classifier answers
+ "undetermined". ``tab_in_bare_double_separator`` reaches it as
+ ``"http:///<TAB>/attacker.example.com/share/db"``, which after the
+ parser's tab deletion has netloc ``""`` and ``hostname is None``, so the
+ entry is skipped for having no host at all. The ``*_file_scheme`` values
+ parse to authority ``localhost``, which classifies LOCAL with no DNS at
+ all — that is exactly why the network-share rejection has to see them,
+ and why reverting the gate flips those five rows to contained."""
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: uri}, allow_dns=False
+ )
+ is False
+ )
+
+
+def test_file_uri_scheme_is_decided_by_the_parser_not_the_prefix():
+ """``_file_uri_names_a_remote_share`` must ask ``urlsplit`` what the
+ scheme is, not ``str.startswith("file://")``.
+
+ ``file:%2f%2fhost/share`` has scheme ``file`` and a path that decodes to
+ the UNC prefix ``//host/share``, but it never contains the literal
+ ``"file://"``, so a raw-prefix gate declines to look at it at all.
+ Reverting the gate to that prefix test flips these three assertions to
+ False. For THESE percent-encoded forms the gate is defence in depth —
+ end-to-end they still fail closed via the DNS classifier, which finds no
+ local host in them — but the same gate is load-bearing for the
+ control-character forms above, whose parsed authority is ``localhost``:
+ see
+ ``test_control_characters_cannot_smuggle_a_remote_store_past_the_guards``,
+ where reverting it flips all five ``*_file_scheme`` rows to CONTAINED.
+ Note this function DOES pre-normalise its input, and that call is a
+ second guard rather than a no-op: ``urlsplit`` lstrips only
+ ``\\x00``-``\\x20``, while the normaliser's final ``strip()`` is
+ bidirectional and also removes the 19 leading Unicode-whitespace
+ codepoints (U+0085, U+00A0, U+1680, U+2000-U+200A, U+2028, U+2029,
+ U+202F, U+205F, U+3000) the parser leaves in place. Drop it and
+ "\\xa0file://localhost//attacker.example.com/share/db" parses with an
+ EMPTY scheme, so the network-share rejection never sees it — see
+ ``test_leading_unicode_whitespace_cannot_hide_a_network_share``."""
+ for attack_uri in (
+ "file:%2f%2fattacker.example.com/share/db",
+ "file:/%2f%2fattacker.example.com/share/db",
+ "FILE:%5C%5Cattacker.example.com/share/db",
+ ):
+ assert P._file_uri_names_a_remote_share(attack_uri) is True
+ # A scheme-gated check must not start rejecting the ordinary forms.
+ for local_uri in (
+ "file:///var/lib/store.db",
+ "file://localhost/var/lib/store.db",
+ "file://C:/data/store.db",
+ "file:relative/store.db",
+ ):
+ assert P._file_uri_names_a_remote_share(local_uri) is False
+
+
+# Leading whitespace ``str.strip()`` removes and ``urlsplit`` does NOT.
+# ``urlsplit`` follows the WHATWG parser and lstrips only "\x00"-"\x20";
+# ``str.strip()`` additionally removes 19 Unicode-whitespace codepoints, and
+# a leading one of those makes ``urlsplit`` report an EMPTY scheme.
+_LEADING_UNICODE_WHITESPACE = (
+ ("nbsp", "\xa0"),
+ ("nel", "\x85"),
+ ("ideographic_space", "\u3000"),
+ ("thin_space", "\u2009"),
+)
+_SHARE_URI = "file://localhost//attacker.example.com/share/db"
+
+
+ "prefix",
+ [prefix for _, prefix in _LEADING_UNICODE_WHITESPACE],
+ ids=[name for name, _ in _LEADING_UNICODE_WHITESPACE],
+)
+def test_leading_unicode_whitespace_cannot_hide_a_network_share(prefix):
+ """Leading Unicode whitespace must not hide a UNC share from the parser.
+
+ ``_file_uri_names_a_remote_share`` asks ``urlsplit`` for the scheme, and
+ ``urlsplit``'s lstrip covers only "\\x00"-"\\x20". Every prefix here is
+ OUTSIDE that range, so on the RAW value the scheme parses as "" — not
+ "file" — and the network-share rejection declines to look at the URI at
+ all. The ``_normalize_uri_like_url_parser`` call in that function is what
+ closes this: its final ``strip()`` is bidirectional and removes exactly
+ these codepoints before the parse.
+
+ The LIST shape is where it bites end to end. On the raw value the entry
+ also parses to ``hostname is None``, so ``_classify_engine_url`` skips it
+ entirely and the sibling "127.0.0.1" — a loopback IP literal, classified
+ LOCAL with NO DNS — makes the whole setting answer CONTAINED.
+
+ Measured on every prefix here: drop the ``_normalize_uri_like_url_parser``
+ call and hand ``value`` to ``urlsplit`` raw, and BOTH assertions below
+ flip — the direct one to False, the containment one to True.
+ """
+ attack = prefix + _SHARE_URI
+ assert P._file_uri_names_a_remote_share(attack) is True
+ with _store(_RemoteStore):
+ for value in ([attack, "127.0.0.1"], {"value": [attack, "127.0.0.1"]}):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: value}, allow_dns=False
+ )
+ is False
+ ), value
+
+
+# Each entry is (id, raw, expected). The expectation is an EXPLICIT string,
+# not "whatever ``urlsplit`` happens to agree with": an
+# ``urlsplit(f(raw)) == urlsplit(raw)`` assertion holds for the identity
+# function and for ``str.strip()`` on every row here EXCEPT
+# ``trailing_whitespace_trimmed`` (``urlsplit`` keeps the trailing spaces in
+# ``path``, so ``str.strip()`` changes the parse there), so it can only ever
+# catch OVER-normalisation, never the under-normalisation that is the
+# security direction.
+_NORMALISER_CASES = (
+ # Interior tab/CR/LF deletion — ``str.strip()`` never touches these, so
+ # each of these rows fails under BOTH ``str.strip()`` and the identity.
+ (
+ "interior_tab_deleted",
+ "fi\tle://localhost//attacker.example.com/db",
+ "file://localhost//attacker.example.com/db",
+ ),
+ (
+ "interior_cr_lf_deleted",
+ "http://loc\r\nalhost:19530",
+ "http://localhost:19530",
+ ),
+ (
+ "tab_in_drive_separator_deleted",
+ "x:/\t/attacker.example.com:19530",
+ "x://attacker.example.com:19530",
+ ),
+ # Leading NON-whitespace C0 controls — ``str.strip()`` stops at the first
+ # of them, so these rows fail under ``str.strip()`` and the identity too.
+ (
+ "leading_c0_stripped",
+ "\x01\x1bfile://localhost/db",
+ "file://localhost/db",
+ ),
+ (
+ "leading_nul_after_whitespace_stripped",
+ "\t\x00 file:///var/lib/store.db",
+ "file:///var/lib/store.db",
+ ),
+ # Trailing whitespace: the deliberate delta from ``urlsplit`` (which
+ # lstrips only). Fails under the identity, passes under ``str.strip()``.
+ (
+ "trailing_whitespace_trimmed",
+ "file:///var/lib/store.db ",
+ "file:///var/lib/store.db",
+ ),
+ # Over-normalisation controls: an INTERIOR space and a percent-encoded
+ # tab are part of the path and must survive untouched. These are the rows
+ # that fail if the normaliser starts removing more than the parser does.
+ (
+ "interior_space_kept",
+ "file:///home/u/my files/store.db",
+ "file:///home/u/my files/store.db",
+ ),
+ (
+ "percent_encoded_tab_kept",
+ "file:///var/lib/%09store.db",
+ "file:///var/lib/%09store.db",
+ ),
+)
+
+
+ ("raw", "expected"),
+ [(raw, expected) for _, raw, expected in _NORMALISER_CASES],
+ ids=[name for name, _, _ in _NORMALISER_CASES],
+)
+def test_normaliser_output_is_the_string_urlsplit_would_parse(raw, expected):
+ """The normaliser's OUTPUT is pinned to an explicit string per row.
+
+ ``urlsplit`` follows the WHATWG parser: it lstrips C0 controls and space,
+ then deletes every tab, CR and LF from anywhere in the URL. The
+ normaliser must reproduce that (plus a trailing trim the callers relied
+ on before it existed).
+
+ WHICH REPLACEMENT EACH ROW CATCHES: the three deletion rows and the two
+ leading-C0 rows fail under BOTH ``str.strip()`` and the identity
+ function — ``str.strip()`` never removes an interior tab/CR/LF and stops
+ at the first non-whitespace C0 character. ``trailing_whitespace_trimmed``
+ fails under the identity only (``str.strip()`` already trims it). The two
+ ``*_kept`` rows fail only under an OVER-normaliser: an interior space and
+ a ``%09`` are path content the parser preserves, and stripping them would
+ be the opposite mistake.
+
+ Deliberately NOT written as ``urlsplit(f(raw)) == urlsplit(raw)``: that
+ assertion holds for the identity function and for ``str.strip()`` on
+ every row here except ``trailing_whitespace_trimmed`` — for that one row
+ ``urlsplit`` keeps the trailing spaces in ``path``
+ (``urlsplit("file:///var/lib/store.db ").path`` is
+ ``"/var/lib/store.db "``), so ``str.strip()`` does change the parse. On
+ every other row the assertion can only catch over-normalisation, never
+ the under-normalisation this test exists to catch."""
+ assert P._normalize_uri_like_url_parser(raw) == expected
+
+
+# Positive control for the tests above: the ordinary, control-character-
+# free forms must keep the classification they had before the normalisation
+# was introduced. A normaliser that over-strips would show up here.
+_BENIGN_LOCAL_URIS = (
+ "/var/lib/ldr/store.db",
+ "file:///var/lib/ldr/store.db",
+ "file://localhost/var/lib/ldr/store.db",
+ "file://C:/data/store.db",
+ "C:\\data\\store.db",
+ "C:/data/store.db",
+ "C:\\\\data\\store.db",
+ "C:/\\data\\store.db",
+ "C:\\/data/store.db",
+ "./data/store.db",
+ "~/data/store.db",
+ "file:///var/lib/%2fdir/store.db",
+ "file:///%252fdir/store.db",
+)
+
+
[email protected]("uri", _BENIGN_LOCAL_URIS)
+def test_benign_local_uris_stay_contained_after_normalisation(uri):
+ """Positive control: normalising the string the guards read must not
+ reclassify any ordinary local path or ``file://`` URI. Covers all four
+ drive-separator truth-table rows (``C:\\\\``, ``C:/\\``, ``C:\\/``,
+ ``C:/``) plus the RFC 8089 and relative-path forms. ``allow_dns=False``
+ keeps it off the network — every URI here must be answered by the
+ filesystem-path check alone, so a regression that pushes one into the
+ DNS classifier fails this too."""
+ assert P._is_local_filesystem_uri(uri) is True
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: uri}, allow_dns=False
+ )
+ is True
+ )
+
+
+def test_file_uri_percent_encoded_unc_in_path_is_not_a_local_file():
+ """A ``file://`` URI may percent-encode its path (RFC 8089), so the
+ leading-separator check must run on the DECODED path:
+ "file:///%2fattacker.example.com/share/db" and the "%5c" twin read
+ as single-separator paths while encoded, but ``unquote`` yields
+ exactly the "//host" / "/\\host" UNC prefixes that
+ ``ntpath.normpath`` resolves to a remote share on Windows. Reverting
+ the decode (checking the literal ``parsed.path`` again, as before
+ the fix) makes both attack forms classify LOCAL again — these
+ assertions are what catch that revert."""
+ assert (
+ P._is_local_filesystem_uri("file:///%2fattacker.example.com/share/db")
+ is False
+ )
+ assert (
+ P._is_local_filesystem_uri("file:///%5cattacker.example.com/share/db")
+ is False
+ )
+ # The mixed-separator combination of the same encoding, and the
+ # localhost-authority form: the netloc check passes for both, so only
+ # the decoded-path separator check stands between them and a LOCAL
+ # classification.
+ assert (
+ P._is_local_filesystem_uri("file:///%5c/attacker.example.com/share/db")
+ is False
+ )
+ assert (
+ P._is_local_filesystem_uri(
+ "file://localhost/%2fattacker.example.com/share/db"
+ )
+ is False
+ )
+ # The RFC 8089 local forms stay local after the decode — including a
+ # path whose percent-encoding sits AFTER the leading separator, where
+ # decoding changes nothing about locality.
+ assert P._is_local_filesystem_uri("file:///var/lib/store.db") is True
+ assert (
+ P._is_local_filesystem_uri("file://localhost/var/lib/store.db") is True
+ )
+ assert P._is_local_filesystem_uri("file://C:/data/store.db") is True
+ assert P._is_local_filesystem_uri("file:///C:/data/store.db") is True
+ assert (
+ P._is_local_filesystem_uri("file:///home/u/my%20files/store.db") is True
+ )
+
+
+def test_file_uri_percent_encoded_unc_endpoint_is_not_contained():
+ """End-to-end: a store whose ``uri_setting`` percent-encodes the UNC
+ prefix of its path must be classified REMOTE, exactly like the literal
+ four-slash / mixed-slash forms.
+
+ What this pins is the END-TO-END outcome, not the ``unquote`` inside
+ ``_is_local_filesystem_uri``: ``_vector_store_endpoint_is_local`` runs
+ the same decode-and-check via ``_file_uri_names_a_remote_share`` one
+ frame earlier, so reverting that ``unquote`` alone would NOT flip these
+ assertions. That revert is caught by
+ ``test_file_uri_percent_encoded_unc_in_path_is_not_a_local_file`` (the
+ unit test above); this test catches the loss of the fail-closed
+ *result* through whichever frame decodes."""
+ for attack_uri in (
+ "file:///%2fattacker.example.com/share/db",
+ "file:///%5cattacker.example.com/share/db",
+ ):
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: attack_uri}
+ )
+ is False
+ )
+ # The plain RFC 8089 local forms remain contained.
+ for local_uri in (
+ "file:///var/lib/store.db",
+ "file://localhost/var/lib/store.db",
+ "file://C:/data/store.db",
+ ):
+ with _store(_RemoteStore):
+ assert (
+ P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: local_uri}
+ )
+ is True
+ )
+
+
+def test_rejected_file_paths_cannot_be_allowed_by_hostname_fallback():
+ """A nominally local authority cannot make a decoded share path local."""
+ rejected_uris = (
+ "file://localhost/%2freview.invalid/share/db",
+ "file://localhost/%5creview.invalid/share/db",
+ "file://localhost//review.invalid/share/db",
+ "FILE://LOCALHOST/%5c/review.invalid/share/db",
+ "file://127.0.0.1/%2freview.invalid/share/db",
+ "file://C:/%2freview.invalid/share/db",
+ "file://[::1/db",
+ )
+ with (
+ _store(_RemoteStore),
+ patch(
+ f"{POLICY}._classify_engine_url", return_value=True
+ ) as classify_url,
+ ):
+ for uri in rejected_uris:
+ assert not P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: uri}
+ ), uri
+ # A setting value need not be a STRING. ``unwrap_setting``
+ # returns whatever the snapshot held, uncoerced, and
+ # ``_classify_engine_url`` explicitly supports the list shape,
+ # so the rejection must run per ENTRY. Reverting it to the
+ # string-only guard lets the one-element list skip the
+ # rejection and be rescued by the local-looking authority
+ # ("localhost" / "127.0.0.1" / drive letter), which the patched
+ # classifier above stands in for. Because that patch makes
+ # ``_classify_engine_url`` return True unconditionally, ALL SEVEN
+ # flip to contained under the revert (unpatched, only the five
+ # with a resolvably-local authority would — the "C:" and the
+ # unbalanced-bracket forms fail hostname classification on their
+ # own), and ``classify_url.assert_not_called()`` fires too.
+ assert not P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: [uri]}
+ ), uri
+ # ...including through the ``{"value": ...}`` snapshot wrapper
+ # ``_get_setting_value`` peels: unwrapping yields the same list,
+ # never a string, so this shape needs the same per-entry pass.
+ assert not P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: {"value": [uri]}}
+ ), uri
+ classify_url.assert_not_called()
+
+ # ANY rejected entry sinks the whole setting, even alongside a
+ # legitimately local endpoint: containment is a property of the
+ # whole configured endpoint set, mirroring the "any public entry
+ # wins" fail-up. Reverting to the string-only guard (or to a
+ # first-entry-only / all-entries-must-be-rejected check) makes this
+ # mixed list pass as contained via the local entry.
+ assert not P.vector_store_is_contained(
+ "library",
+ ctx(),
+ {URI_SETTING: ["http://127.0.0.1:19530", rejected_uris[0]]},
+ )
+ assert not P.vector_store_is_contained(
+ "library",
+ ctx(),
+ {
+ URI_SETTING: {
+ "value": [LOCAL_URI[URI_SETTING], rejected_uris[2]]
+ }
+ },
+ )
+ classify_url.assert_not_called()
+
+ # Local paths still pass without DNS, including encoded spaces.
+ for uri in (
+ "file:///var/lib/store.db",
+ "file://localhost/var/lib/store.db",
+ "file://C:/data/store.db",
+ "file:///home/u/my%20files/store.db",
+ ):
+ assert P.vector_store_is_contained(
+ "library", ctx(), {URI_SETTING: uri}
+ ), uri
+ classify_url.assert_not_called()
+
+ # Network endpoints still consult the configured host classifier.
+ assert P.vector_store_is_contained("library", ctx(), LOCAL_URI)
+ classify_url.assert_called_once()
+
+
def test_file_uri_truth_table_full_regression():
"""Full before/after truth table for ``_is_local_filesystem_uri``,
pinning the RFC 8089 local forms, the round-6 fixes (4-slash
UNC-in-path bypass rejected; Windows drive-letter authority accepted),
the round-7 mixed-slash UNC fix (``ntpath.normpath`` treats ANY pair of
leading "/"/"\\" separators as a UNC prefix on Windows, not just "//"),
- and every previously-covered row unchanged."""
+ the malformed-IPv6-authority fix (a bare ``urlsplit`` used to raise
+ instead of returning False), the single-letter-scheme-authority fix
+ (``x://host`` used to be misread
+ as a Windows drive path), the percent-encoded-UNC fix (the separator
+ check now runs on the unquoted path, so a "%2f"/"%5c"-encoded UNC
+ prefix can no longer classify local), the Unicode-drive-letter fix (the
+ drive-authority netloc check AND the bare drive-path branch now require an
+ ASCII letter, so Cyrillic/fullwidth/accented lookalikes no longer classify
+ local in either position), the drive-path narrowing (only a literal "//"
+ after the drive letter is a scheme authority, so doubled-/mixed-backslash
+ drive paths are local again), and every previously-covered row
+ unchanged."""
cases = [
# (uri, expected, label)
(
@@ -461,6 +1135,64 @@ def test_file_uri_truth_table_full_regression():
False,
"4-slash UNC bypass",
),
+ (
+ "file://[attacker.example.com/share/db",
+ False,
+ "malformed IPv6-literal authority (unbalanced bracket)",
+ ),
+ (
+ "file://[::1/db",
+ False,
+ "malformed IPv6-literal authority (truncated)",
+ ),
+ (
+ "x://attacker.example.com:19530",
+ False,
+ "single-letter-scheme authority, not a drive path",
+ ),
+ (
+ "\u0441://attacker.example.com:19530",
+ False,
+ "Cyrillic-letter-scheme authority",
+ ),
+ (
+ "\uff23://attacker.example.com:19530",
+ False,
+ "fullwidth-letter-scheme authority",
+ ),
+ (
+ "file://\u0441:/share/db",
+ False,
+ "Cyrillic drive-letter authority",
+ ),
+ (
+ "file://\uff23:/share/db",
+ False,
+ "fullwidth drive-letter authority",
+ ),
+ (
+ "file://\u00e9:/share/db",
+ False,
+ "accented drive-letter authority",
+ ),
+ # The bare-path twins: a drive-absolute path needs an ASCII drive
+ # letter too, or the drive-path branch short-circuits LOCAL on a
+ # value no resolver reads as a drive path.
+ (
+ "\u0441:/data/store.db",
+ False,
+ "Cyrillic drive-letter path",
+ ),
+ (
+ "\uff23:/data/store.db",
+ False,
+ "fullwidth drive-letter path",
+ ),
+ (
+ "\u00e9:\\data\\store.db",
+ False,
+ "accented drive-letter path (backslash)",
+ ),
(
"/\\attacker.example.com/share/store.db",
False,
@@ -476,13 +1208,33 @@ def test_file_uri_truth_table_full_regression():
False,
"mixed-slash UNC bypass (backslash-first bare path)",
),
+ (
+ "file:///%2fattacker.example.com/share/db",
+ False,
+ "percent-encoded UNC bypass (%2f)",
+ ),
+ (
+ "file:///%5cattacker.example.com/share/db",
+ False,
+ "percent-encoded UNC bypass (%5c)",
+ ),
("file://C:/data/store.db", True, "windows drive-letter authority"),
+ (
+ "file://c:/data/store.db",
+ True,
+ "windows drive-letter authority (lowercase)",
+ ),
("file:///var/lib/store.db", True, "RFC8089 empty authority"),
(
"file://localhost/var/lib/store.db",
True,
"RFC8089 localhost authority",
),
+ (
+ "file:///home/u/my%20files/store.db",
+ True,
+ "percent-encoded but plainly local path",
+ ),
("FILE://LOCALHOST/var/lib/x", True, "case-insensitive localhost"),
(
"file://attacker.example.com/share/db",
@@ -495,6 +1247,33 @@ def test_file_uri_truth_table_full_regression():
("~/store.db", True, "home-relative"),
("C:\\data\\store.db", True, "windows backslash drive path"),
("C:/data/store.db", True, "windows forward-slash drive path"),
+ # A drive letter always wins over a UNC prefix on Windows
+ # (``ntpath.normpath("C:\\\\data") == "C:\\data"``) and ``urlsplit``
+ # gives an authority only to a literal "//", so these doubled- and
+ # mixed-separator drive paths cannot name a remote host under
+ # either reading. Re-widening the guard to reject every doubled
+ # separator makes them non-contained and locks Windows users out
+ # of a legitimately local store.
+ (
+ "C:\\\\data\\store.db",
+ True,
+ "windows doubled-backslash drive path",
+ ),
+ ("C:/\\data/store.db", True, "windows mixed-separator drive path"),
+ (
+ "C:\\/data/store.db",
+ True,
+ "windows mixed-separator drive path (backslash first)",
+ ),
+ # The forward-slash twin stays rejected: "C://data/store.db" is
+ # byte-for-byte a single-letter-scheme URL (scheme "c", hostname
+ # "data"), indistinguishable from the "x://host" authority the row
+ # above pins as remote. Documented in changelog.d/5763.security.md.
+ (
+ "C://data/store.db",
+ False,
+ "drive letter + literal // is a scheme authority",
+ ),
("//attacker.example.com:19530", False, "scheme-less authority"),
("//host/path", False, "scheme-less authority 2"),
("\\\\server\\share\\store.db", False, "UNC path"),