~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Security coverage for the library / RAG surface lost in the Flask ->
FastAPI migration (PR #3299).
The migration deleted ``tests/research_library/`` (21 paths, 682 tests) while
porting all four blueprints 1:1 into ``web/routers/{library,rag,
library_delete,library_search}.py``. A provenance audit of those deletions
found five security-flagged behaviours that survived in ``src/`` but lost
every assertion that pinned them. This file re-pins exactly those five, at
the layer where the contract actually lives.
COVERAGE AREA 1 -- note-protection 403 on ``DELETE /library/api/document/{id}``.
Guard: ``research_library/deletion/services/document_deletion.py``
``_delete_document_locked`` -> ``_is_note_document`` -> ``is_note: True``,
mapped to 403 at ``web/routers/library_delete.py::delete_document``.
At the ADR-0010 snapshot, no direct assertion covered this route: the
branch's service test
(``tests/research_library/deletion/services/test_document_deletion.py::
TestDocumentDeletionServiceNoteRefusal``) covers only ``delete_blob_only``
and ``remove_from_collection``, and its own docstring defers the
``delete_document`` case to the route suite this PR deleted. A regression
lets a note Document be hard-deleted (and bulk-amplified) through the
generic document API, bypassing NoteService's version/link cleanup.
COVERAGE AREA 2 -- the 403-vs-404 mapping on the two sibling delete routes,
``DELETE /library/api/document/{id}/blob`` and
``DELETE /library/api/collection/{cid}/document/{did}``. The service-level
refusals were covered by the branch test named above; the status mapping
was not. The tests below now pin that mapping because a refusal surfaced as
404 instead of 403 is a different contract (the frontend uses the
distinction to route the user to the notes API).
COVERAGE AREA 3 -- ``web/routers/rag.py::_format_test_embedding_error``.
At the review snapshot it had no direct assertions (11 predecessor tests
were deleted); this file now pins its branches. This is the CWE-209 error
boundary on
``POST /library/api/rag/test-embedding``: the function's output goes
straight into the response body. Three branches matter -- internal LDR,
upstream provider, and unexpected failures. Every branch returns a fixed
category message because exception text can carry filesystem paths, SQL,
response bodies, or credentials.
``tests/web/routers/test_rag_hostile_input.py::
test_test_embedding_does_not_leak_decoder_message`` covers only the
malformed-body 400, which returns before this function is ever reached.
COVERAGE AREA 4 -- ``web/routers/rag.py::_sanitized_indexing_errors``. Its
output is
stored in ``TaskMetadata.result_metadata`` by both terminal paths of
``_background_index_worker`` and read back by the index-status / SSE
endpoints, i.e. it reaches the browser. Unscrubbed per-document errors are
a disclosure; an unbounded list is a response-size problem.
``TestIndexingErrorAllowlistMatchesService`` checks the allowlist behind it
against the error text ``LibraryRAGService`` returns, so a fixed message
the service adds cannot be withheld unnoticed and an allowlisted one
cannot outlive the code that returns it.
``TestIndexingTaskErrorPassThroughs`` does the same for the stored task
error the index-status endpoint serves, against the text
``web/routers/rag.py`` writes to indexing task rows.
COVERAGE AREA 5 -- ``research_library/utils/is_downloadable_domain``. It had no
direct coverage at the review snapshot; the tests below and
``tests/web/routers/test_library_download_outcomes.py`` now cover it. It is
the
allowlist gate deciding which remote URLs
the server will fetch (``web/routers/library.py:879``, ``:1539`` and
``:1766``), so it is SSRF-adjacent. The tests below pin the policy the implementation and
its docstring actually describe -- "academic domain OR direct PDF link" --
and deliberately pin the remaining CURRENT-BEHAVIOUR weaknesses (plus the
regressions closed by ``harden: reject private hosts in
is_downloadable_domain, anchor the PubMed rule``) rather than inventing a
stricter policy (see ``TestIsDownloadableDomainKnownWeaknesses``).
``TestIsDownloadableDomainPrivateHostNormalization`` separately pins the
call-site-only IP-literal normalization (trailing dot, decimal/hex,
shorthand-dotted, per-octet octal, RFC 6598 CGNAT) added on top of that
hardening -- see that class's docstring for why the shared
``is_private_ip()`` helper itself is untouched.
Harness: the ``auth_client`` idiom from
``tests/web/routers/test_library_delete_hostile_input.py`` -- a real,
in-process ``TestClient(app, raise_server_exceptions=False)`` against the live
FastAPI app, a freshly registered and auto-authenticated throwaway user, a real
CSRF token (CSRF is ASGI-middleware-enforced, not a config flag), a per-test
``LDR_DATA_DIR``, and a distinct ``X-Forwarded-For`` so the per-IP limiter
cannot bucket these clients together. No network, no LLM.
Non-vacuity: every "the forbidden thing is absent" assertion is paired with a
positive assertion that the *expected* content is present. An empty library or
a 500 would otherwise satisfy "no filesystem path in the response" trivially.
"""
import os
import uuid
from contextlib import suppress
from unittest.mock import patch
import pytest
from fastapi.testclient import TestClient
# The app's docs_url toggle reads PYTEST_CURRENT_TEST at app-import time;
# pytest only sets that per-test, not at collection.
os.environ.setdefault("TESTING", "1")
# ---------------------------------------------------------------------------
# Harness
# ---------------------------------------------------------------------------
def _cleanup_client(client, username):
try:
with suppress(Exception):
client.post("/auth/logout", follow_redirects=False)
with suppress(Exception):
from local_deep_research.web.auth.session_manager import (
session_manager,
)
session_manager.destroy_all_user_sessions(username)
with suppress(Exception):
from local_deep_research.database.session_passwords import (
session_password_store,
)
session_password_store.clear_all_for_user(username)
with suppress(Exception):
from local_deep_research.database.thread_local_session import (
clear_user_credentials,
)
clear_user_credentials(username)
finally:
client.close()
@pytest.fixture
def library_client(temp_data_dir, monkeypatch, request):
"""Authenticated TestClient for a throwaway user against a per-test
``LDR_DATA_DIR``. Returns ``(client, username, password)`` so tests can
open a direct DB session for the same user to seed rows and to verify
post-conditions in the database rather than only in the response body.
"""
monkeypatch.setenv("LDR_DATA_DIR", str(temp_data_dir))
# Production PBKDF2 iteration count would dominate wall-clock in a
# per-test fixture (same reason test_library_delete_hostile_input.py
# lowers it).
monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")
from local_deep_research.web.fastapi_app import app
client = TestClient(app, raise_server_exceptions=False)
username = f"test_lib_sec_{uuid.uuid4().hex[:8]}"
password = "TestPassword123!" # noqa: S105
request.addfinalizer(lambda: _cleanup_client(client, username))
# Distinct peer per client: the auth/registration routes are per-IP rate
# limited, and several of these fixtures can run inside one session.
client.headers.update(
{"X-Forwarded-For": f"10.{uuid.uuid4().int % 254 + 1}.13.7"}
)
def _csrf():
client.get("/auth/login")
resp = client.get("/auth/csrf-token")
return (
resp.json().get("csrf_token", "") if resp.status_code == 200 else ""
)
reg = client.post(
"/auth/register",
data={
"username": username,
"password": password,
"confirm_password": password,
"acknowledge": "true",
"csrf_token": _csrf(),
},
follow_redirects=False,
)
if reg.status_code != 302:
pytest.fail(
f"Auth bootstrap broken: registration returned {reg.status_code} "
f"(expected 302): {reg.text[:300]}"
)
who = client.get("/auth/check")
if (
who.status_code != 200
or who.json().get("authenticated") is not True
or who.json().get("username") != username
):
pytest.fail(
"Auth bootstrap broken: registration did not establish the "
f"expected session: {who.status_code} {who.text[:300]}"
)
csrf_resp = client.get("/auth/csrf-token")
if csrf_resp.status_code == 200:
token = csrf_resp.json().get("csrf_token")
if token:
client.headers.update({"X-CSRFToken": token})
return client, username, password
def _seed_document(
username, password, *, source_type_name, title, storage_mode
):
"""Insert one real ``Document`` row owned by *username*.
``source_type_name`` selects the ``SourceType`` seeded by
``initialize_library_for_user`` on register/login -- "note" produces a
document that ``_is_note_document`` recognises, "user_upload" an ordinary
one. Returns the new document id.
"""
from local_deep_research.database.models.library import Document, SourceType
from local_deep_research.database.session_context import get_user_db_session
doc_id = str(uuid.uuid4())
with get_user_db_session(username, password) as session:
source_type = (
session.query(SourceType).filter_by(name=source_type_name).first()
)
assert source_type is not None, (
f"{source_type_name!r} SourceType should have been seeded by "
"initialize_library_for_user on register/login"
)
session.add(
Document(
id=doc_id,
source_type_id=source_type.id,
document_hash=uuid.uuid4().hex,
filename=f"{source_type_name}-fixture.md",
file_size=11,
file_type="md",
text_content="hello world",
title=title,
storage_mode=storage_mode,
)
)
session.commit()
return doc_id
def _seed_collection(username, password, *, collection_type, name):
"""Insert one real ``Collection`` row and return its id."""
from local_deep_research.database.models.library import Collection
from local_deep_research.database.session_context import get_user_db_session
collection_id = str(uuid.uuid4())
with get_user_db_session(username, password) as session:
session.add(
Collection(
id=collection_id, name=name, collection_type=collection_type
)
)
session.commit()
return collection_id
def _link(username, password, document_id, collection_id):
"""Link a document into a collection (the ``document_collections`` join)."""
from local_deep_research.database.models.library import DocumentCollection
from local_deep_research.database.session_context import get_user_db_session
with get_user_db_session(username, password) as session:
session.add(
DocumentCollection(
document_id=document_id, collection_id=collection_id
)
)
session.commit()
def _document_exists(username, password, document_id):
from local_deep_research.database.models.library import Document
from local_deep_research.database.session_context import get_user_db_session
with get_user_db_session(username, password) as session:
return session.get(Document, document_id) is not None
def _link_exists(username, password, document_id, collection_id):
from local_deep_research.database.models.library import DocumentCollection
from local_deep_research.database.session_context import get_user_db_session
with get_user_db_session(username, password) as session:
return (
session.query(DocumentCollection)
.filter_by(document_id=document_id, collection_id=collection_id)
.first()
is not None
)
# ===========================================================================
# COVERAGE AREA 1 -- DELETE /library/api/document/{id} note protection
# and the note must survive.
# ===========================================================================
class TestDeleteDocumentNoteRefusal:
"""``DELETE /library/api/document/{id}`` on a note Document.
This is the direct route-level regression evidence. The guard lives in
``document_deletion.py::_delete_document_locked`` (``_is_note_document``
-> ``{"is_note": True}``) and is mapped to 403 in
``library_delete.py::delete_document``. Deleting the guard, or downgrading
the 403 to the generic 404 branch below it, flips these assertions.
"""
def test_deleting_a_note_is_refused_with_403_and_the_note_survives(
self, library_client
):
client, username, password = library_client
note_id = _seed_document(
username,
password,
source_type_name="note",
title="Protected note",
storage_mode="database",
)
resp = client.delete(f"/library/api/document/{note_id}")
assert resp.status_code == 403, (
"note deletion via the generic document API must be refused with "
f"403 (not 404, not 200): got {resp.status_code}: {resp.text[:300]}"
)
body = resp.json()
assert body["success"] is False
assert body["deleted"] is False
assert body["is_note"] is True, (
f"response must carry the is_note discriminator: {body!r}"
)
assert "/api/notes/" in body["error"], (
f"refusal must route the caller to the notes API: {body!r}"
)
# The point of the guard: the row is still there.
assert _document_exists(username, password, note_id), (
"the note Document must still exist after the refused delete"
)
def test_bulk_delete_cannot_amplify_past_the_note_guard(
self, library_client
):
"""The bulk endpoint loops over the same service method, so the guard
(not the route's 403 mapping) is what protects it. Bulk returns 200
with a per-item failure -- the note must still be in the database.
"""
client, username, password = library_client
note_id = _seed_document(
username,
password,
source_type_name="note",
title="Protected note",
storage_mode="database",
)
resp = client.request(
"DELETE",
"/library/api/documents/bulk",
json={"document_ids": [note_id]},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["deleted"] == 0, (
f"the note must not be counted as deleted: {body!r}"
)
assert _document_exists(username, password, note_id), (
"the note Document must survive the bulk delete path too"
)
def test_an_ordinary_document_is_still_deleted_with_200(
self, library_client
):
"""Positive control (non-vacuity). Proves the 403 above is the note
guard firing and not a blanket refusal, a broken harness, or an empty
library: the same route, same client, same request shape deletes a
non-note document and the row really goes away.
"""
client, username, password = library_client
doc_id = _seed_document(
username,
password,
source_type_name="user_upload",
title="Ordinary document",
storage_mode="database",
)
resp = client.delete(f"/library/api/document/{doc_id}")
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["success"] is True
assert body["deleted"] is True
assert "is_note" not in body
assert not _document_exists(username, password, doc_id), (
"an ordinary document must actually be gone from the DB"
)
# ===========================================================================
# COVERAGE AREA 2 -- 403-vs-404 mapping on sibling delete routes
# ===========================================================================
class TestSiblingDeleteRoutes403VsNotFound:
"""The refusals themselves are covered at the service layer; what is
untested is that ``library_delete.py`` maps them to 403 and NOT to the
404/400 branch sitting immediately below in the same handler. Each test
pairs the 403 case with the 404 case on the same route so a mapping that
collapsed the two would fail.
"""
def test_blob_delete_of_a_note_returns_403_not_404(self, library_client):
client, username, password = library_client
note_id = _seed_document(
username,
password,
source_type_name="note",
title="Protected note",
storage_mode="database",
)
resp = client.delete(f"/library/api/document/{note_id}/blob")
assert resp.status_code == 403, (
"a note blob-delete refusal must map to 403, not the sibling "
f"404/400 branch: got {resp.status_code}: {resp.text[:300]}"
)
body = resp.json()
assert body["success"] is False
assert body["is_note"] is True
assert body["bytes_freed"] == 0
assert _document_exists(username, password, note_id)
def test_blob_delete_of_an_unknown_document_returns_404(
self, library_client
):
"""Discriminator for the test above: the same route's not-found case
must stay 404, so "everything is 403" would not pass.
"""
client, _, _ = library_client
resp = client.delete("/library/api/document/no-such-document/blob")
assert resp.status_code == 404, resp.text
body = resp.json()
assert body["success"] is False
assert "not found" in body["error"].lower()
assert "is_note" not in body
def test_blob_delete_of_an_ordinary_document_returns_200(
self, library_client
):
"""Positive control: the 403 and 404 above are real branch outcomes,
not a route that refuses everything.
"""
client, username, password = library_client
doc_id = _seed_document(
username,
password,
source_type_name="user_upload",
title="Ordinary document",
storage_mode="database",
)
resp = client.delete(f"/library/api/document/{doc_id}/blob")
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["success"] is True
assert body["deleted"] is True
def test_remove_note_from_its_notes_collection_returns_403_not_404(
self, library_client
):
client, username, password = library_client
note_id = _seed_document(
username,
password,
source_type_name="note",
title="Protected note",
storage_mode="database",
)
notes_collection_id = _seed_collection(
username, password, collection_type="notes", name="Notes"
)
_link(username, password, note_id, notes_collection_id)
resp = client.delete(
f"/library/api/collection/{notes_collection_id}/document/{note_id}"
)
assert resp.status_code == 403, (
"unlinking a note from its permanent notes-collection home must "
f"map to 403, not 404: got {resp.status_code}: {resp.text[:300]}"
)
body = resp.json()
assert body["success"] is False
assert body["unlinked"] is False
assert body["protected"] is True, (
f"response must carry the protected discriminator: {body!r}"
)
# Both the link and the note itself must survive.
assert _link_exists(username, password, note_id, notes_collection_id)
assert _document_exists(username, password, note_id)
def test_remove_unknown_document_from_collection_returns_404(
self, library_client
):
"""Discriminator for the test above."""
client, username, password = library_client
notes_collection_id = _seed_collection(
username, password, collection_type="notes", name="Notes"
)
resp = client.delete(
f"/library/api/collection/{notes_collection_id}/document/nope"
)
assert resp.status_code == 404, resp.text
body = resp.json()
assert body["success"] is False
assert "not found" in body["error"].lower()
assert "protected" not in body
def test_remove_ordinary_document_from_collection_returns_200(
self, library_client
):
"""Positive control for the two above: the route can and does return
200 for a permitted unlink.
"""
client, username, password = library_client
doc_id = _seed_document(
username,
password,
source_type_name="user_upload",
title="Ordinary document",
storage_mode="database",
)
collection_id = _seed_collection(
username,
password,
collection_type="user_collection",
name="Ordinary collection",
)
_link(username, password, doc_id, collection_id)
resp = client.delete(
f"/library/api/collection/{collection_id}/document/{doc_id}"
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["success"] is True
assert body["unlinked"] is True
assert not _link_exists(username, password, doc_id, collection_id)
# ===========================================================================
# COVERAGE AREA 3 -- _format_test_embedding_error (CWE-209)
# ===========================================================================
# A realistic OpenAI-style key. Written split so the literal is not itself a
# credential-looking blob in the repo, and long enough to trip the
# ``sk-[A-Za-z0-9-]{20,}`` pattern in security/log_sanitizer.py.
_FAKE_API_KEY = "sk-proj-" + "A1b2C3d4E5f6G7h8" * 3
# A server-side filesystem path of the shape internal LDR exceptions really
# carry (encrypted per-user database files).
_SERVER_PATH = "/srv/ldr/data/encrypted_databases/victim_user.db"
def _internal_exception(message):
"""A genuine LDR-internal exception (``type(exc).__module__`` starts with
``local_deep_research``), which is what the suppression branch keys on.
"""
from local_deep_research.config.thread_settings import (
NoSettingsContextError,
)
return NoSettingsContextError(message)
def _upstream_exception(message):
"""A genuine ``openai``-module exception, matching
``_UPSTREAM_MODULE_PREFIXES``.
"""
import httpx
import openai
return openai.AuthenticationError(
message,
response=httpx.Response(
401, request=httpx.Request("GET", "http://provider.invalid/v1")
),
body=None,
)
class TestFormatTestEmbeddingErrorUnit:
"""Direct coverage of ``rag.py::_format_test_embedding_error``. Nothing on
the branch referenced this function at all.
"""
def test_internal_exception_detail_is_suppressed_entirely(self):
"""The rag.py:334-338 branch. Its comment states the reason outright:
internal exceptions "can carry filesystem paths / SQL fragments that
sanitize_error_message() does not pattern-match", so their detail is
withheld from the browser. The exception below carries exactly such a
path.
"""
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(
_internal_exception(f"could not open {_SERVER_PATH}"),
"nomic-embed-text",
)
# Positive first: the caller still gets a useful, correctly
# categorised message (so the absence checks below are not vacuous).
assert "internal LDR error" in message
assert "NoSettingsContextError" not in message
assert "nomic-embed-text" in message
assert "report it on GitHub" in message
# And the detail -- including the server filesystem path -- is gone.
assert _SERVER_PATH not in message
assert "encrypted_databases" not in message
assert "could not open" not in message
def test_internal_exception_does_not_leak_a_key_either(self):
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(
_internal_exception(f"auth header was {_FAKE_API_KEY}"), "m"
)
assert "internal LDR error" in message
assert _FAKE_API_KEY not in message
def test_upstream_provider_error_withholds_exception_text(self):
"""Provider failures retain guidance without reflecting detail."""
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(
_upstream_exception(f"Incorrect API key provided: {_FAKE_API_KEY}"),
"text-embedding-3-small",
)
assert "The request to the provider failed" in message
assert "text-embedding-3-small" in message
assert "Incorrect API key provided" not in message
assert _FAKE_API_KEY not in message
assert "[REDACTED_KEY]" not in message
def test_stdlib_exception_no_longer_echoes_verbatim_detail(self):
"""The final fallback. An exception from neither the internal nor an
upstream module prefix now gets a fixed category message -- neither
``str(exc)`` nor exception type metadata is echoed.
HARDENED for CodeQL alert 8001 (CWE-209). This branch previously
returned ``f"Embedding test failed for model '{model}': {detail}"``
where ``detail = sanitize_error_message(str(exc))`` -- credential
shapes only, no path/SQL suppression and no length cap. Full leak-path
coverage now lives in
``tests/web/routers/test_rag_embedding_error_sanitisation.py``.
"""
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(
RuntimeError("Connection refused by embedding backend"), "m"
)
# Positive: still correctly categorised and names the model.
assert "Embedding test failed for model 'm'" in message
assert "unexpected error" in message
assert "RuntimeError" not in message
# Negative: the exception text itself no longer reaches the caller.
assert "Connection refused by embedding backend" not in message
# Not miscategorised as an LDR bug -- the #4208 regression.
assert "bug in LDR" not in message
assert "The request to the provider failed" not in message
def test_stdlib_exception_branch_withholds_the_detail_entirely(self):
"""Credential redaction used to be the only mitigation on this branch.
Now the detail is withheld outright, so a credential in a stdlib
exception cannot reach the browser even if a scrubber gap let its
shape through.
"""
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(
RuntimeError(f"auth failed for {_FAKE_API_KEY}"), "m"
)
assert "unexpected error" in message
assert "RuntimeError" not in message
assert "auth failed for" not in message
assert _FAKE_API_KEY not in message
def test_stdlib_exception_branch_no_longer_carries_a_server_path(self):
"""The hardening the previous version of this test anticipated.
``sanitize_error_message`` is a credential-shape scrubber and does not
touch filesystem paths, so a stdlib ``OSError``/``FileNotFoundError``
raised anywhere under ``get_embedding_function`` used to put a server
path in the HTTP 500 body -- exactly the disclosure the
internal-module branch exists to prevent, one branch further down.
The fallback now returns a fixed category message.
"""
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(
FileNotFoundError(f"No such file or directory: {_SERVER_PATH}"),
"m",
)
# Positive control first.
assert "Embedding test failed for model 'm'" in message
assert "unexpected error" in message
assert "FileNotFoundError" not in message
# The disclosure.
assert _SERVER_PATH not in message
assert "encrypted_databases" not in message
assert "No such file or directory" not in message
def test_empty_exception_message_still_returns_a_category(self):
"""An exception with no message still produces useful guidance."""
from local_deep_research.web.routers.rag import (
_format_test_embedding_error,
)
message = _format_test_embedding_error(ValueError(""), "m")
assert "Embedding test failed for model 'm'" in message
assert "unexpected error" in message
assert "ValueError" not in message
def test_module_prefix_match_is_boundary_anchored(self):
"""``_module_matches`` must not treat ``local_deep_research_evil`` (or
``openai_shim``) as the real prefix -- a substring match here would
misroute an unrelated third-party exception into either the
suppression branch or the "provider error" wording.
"""
from local_deep_research.web.routers.rag import _module_matches
assert _module_matches("local_deep_research", "local_deep_research")
assert _module_matches(
"local_deep_research.config.thread_settings", "local_deep_research"
)
assert not _module_matches(
"local_deep_research_evil", "local_deep_research"
)
assert not _module_matches("openai_shim", "openai")
assert not _module_matches("", "local_deep_research")
class TestTestEmbeddingEndpointDoesNotLeak:
"""End-to-end proof over real HTTP that the categorisation above is what
the browser actually receives from
``POST /library/api/rag/test-embedding``. The deleted suite's
``TestTestEmbeddingErrorCategorization`` had no HTTP half at the historical
snapshot; the branch's ``test_rag_hostile_input.py`` covered only the
malformed-body 400, which returns before ``_format_test_embedding_error``
is reached. This class now pins the HTTP wiring.
``get_embeddings`` (not ``get_embedding_function``) is patched at its
definition site (``embeddings.embeddings_config``) because the route
imports it inside the handler body, so the patch is picked up per
request. The route calls ``get_embeddings`` directly -- rather than the
``get_embedding_function`` wrapper -- so the returned instance is
reachable for ``_close_base_llm`` cleanup after the test embed; see
``local_deep_research.web.routers.rag.test_embedding``. Nothing else is
mocked: real auth, real CSRF, real settings load, real response.
"""
_PATCH_TARGET = (
"local_deep_research.embeddings.embeddings_config.get_embeddings"
)
def test_internal_error_does_not_put_a_server_path_in_the_response(
self, library_client
):
client, _, _ = library_client
with patch(
self._PATCH_TARGET,
side_effect=_internal_exception(f"could not open {_SERVER_PATH}"),
):
resp = client.post(
"/library/api/rag/test-embedding",
json={"provider": "ollama", "model": "nomic-embed-text"},
)
assert resp.status_code == 500, resp.text
body = resp.json()
assert body["success"] is False
# Positive control FIRST: the response really is the formatted
# internal-error message, so the absence assertions below cannot be
# satisfied by an empty body, a 401, or an unrelated error page.
assert "internal LDR error" in body["error"], (
f"expected the internal-error wording, got: {body!r}"
)
assert "NoSettingsContextError" not in body["error"]
# Now the disclosure check.
assert _SERVER_PATH not in resp.text
assert "encrypted_databases" not in resp.text
assert "could not open" not in resp.text
def test_provider_error_reaches_the_browser_without_exception_text(
self, library_client
):
client, _, _ = library_client
with patch(
self._PATCH_TARGET,
side_effect=_upstream_exception(
f"Incorrect API key provided: {_FAKE_API_KEY}"
),
):
resp = client.post(
"/library/api/rag/test-embedding",
json={
"provider": "openai",
"model": "text-embedding-3-small",
},
)
assert resp.status_code == 500, resp.text
body = resp.json()
assert "The request to the provider failed" in body["error"]
assert "Incorrect API key provided" not in body["error"]
assert _FAKE_API_KEY not in resp.text
assert "[REDACTED_KEY]" not in body["error"]
def test_successful_embedding_still_returns_200(self, library_client):
"""Non-vacuity control for the two tests above: with the same patch
seam returning a working embedding function, the route reaches its
success path. This proves the 500s above come from the exception
under test and not from the route failing earlier (settings load,
auth, threadpool) for unrelated reasons.
"""
client, _, _ = library_client
class _FakeEmbeddings:
def embed_documents(self, texts):
return [[0.1, 0.2, 0.3]]
with patch(self._PATCH_TARGET, return_value=_FakeEmbeddings()):
resp = client.post(
"/library/api/rag/test-embedding",
json={"provider": "ollama", "model": "nomic-embed-text"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["success"] is True
assert body["dimension"] == 3
# ===========================================================================
# COVERAGE AREA 4 -- _sanitized_indexing_errors and the 50-item bound
# ===========================================================================
class TestSanitizedIndexingErrors:
"""``rag.py::_sanitized_indexing_errors`` is the client-message gate
for the per-document error list that both terminal paths of
``_background_index_worker`` write into ``TaskMetadata.result_metadata``,
which the index-status / SSE endpoints hand to the browser. The tests below
pin its redaction and length-bound contracts.
"""
def test_exception_detail_is_replaced_without_losing_document_identity(
self,
):
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
out = _sanitized_indexing_errors(
{
"errors": [
{
"doc_id": "doc-1",
"title": "Quarterly report",
"error": f"embedding call rejected key {_FAKE_API_KEY}",
}
]
}
)
assert len(out) == 1
entry = out[0]
# The UI still identifies the failed document.
assert entry["doc_id"] == "doc-1"
assert entry["title"] == "Quarterly report"
assert (
entry["error"]
== "Indexing failed. Check the server logs for details."
)
assert _FAKE_API_KEY not in entry["error"]
def test_path_sql_and_provider_detail_are_not_returned(self):
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
out = _sanitized_indexing_errors(
{
"errors": [
{
"doc_id": "doc-1",
"error": (
"DB at /srv/private/user.db failed [SQL: SELECT * "
"FROM secrets] via http://internal.example:11434"
),
}
]
}
)
assert (
out[0]["error"]
== "Indexing failed. Check the server logs for details."
)
def test_output_is_bounded_to_fifty_entries_by_default(self):
"""A failed index of a large collection can produce one error per
document. The list is embedded in task metadata that is serialised
into an index-status response, so the bound is what keeps a failed
1000-document job from becoming a multi-megabyte response.
"""
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
results = {
"errors": [
{"doc_id": f"doc-{i}", "title": f"T{i}", "error": f"boom {i}"}
for i in range(500)
]
}
out = _sanitized_indexing_errors(results)
assert len(out) == 50, (
f"the per-document error list must be bounded, got {len(out)}"
)
# It is the FIRST 50, in order -- not a random or trailing slice.
assert out[0]["doc_id"] == "doc-0"
assert out[-1]["doc_id"] == "doc-49"
assert all(
entry["error"]
== "Indexing failed. Check the server logs for details."
for entry in out
)
def test_limit_is_overridable_and_still_applied(self):
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
results = {"errors": [{"doc_id": str(i)} for i in range(20)]}
assert len(_sanitized_indexing_errors(results, limit=3)) == 3
assert len(_sanitized_indexing_errors(results, limit=0)) == 0
def test_shorter_lists_are_passed_through_unbounded(self):
"""Non-vacuity control for the bound: a list under the limit must not
be truncated, so "always returns <= 50" is not achieved by returning
nothing.
"""
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
results = {
"errors": [
{"doc_id": f"doc-{i}", "title": None, "error": "boom"}
for i in range(7)
]
}
assert len(_sanitized_indexing_errors(results)) == 7
def test_missing_or_empty_error_fields_get_a_neutral_placeholder(self):
"""``str(item.get("error") or "Indexing failed")`` -- a missing/None/
empty error must not serialise as the literal string "None".
"""
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
out = _sanitized_indexing_errors(
{
"errors": [
{"doc_id": "a"},
{"doc_id": "b", "error": None},
{"doc_id": "c", "error": ""},
]
}
)
assert [entry["error"] for entry in out] == ["Indexing failed"] * 3
assert [entry["doc_id"] for entry in out] == ["a", "b", "c"]
# doc_id/title are always present keys, even when the input omitted
# them -- the index-status consumer indexes them directly.
assert all(entry["title"] is None for entry in out)
def test_no_errors_key_yields_an_empty_list(self):
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
assert _sanitized_indexing_errors({}) == []
assert _sanitized_indexing_errors({"errors": []}) == []
def test_non_string_error_values_use_the_generic_message(self):
"""An exception object must never be rendered for the client."""
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
out = _sanitized_indexing_errors(
{
"errors": [
{
"doc_id": "x",
"error": RuntimeError(f"bad key {_FAKE_API_KEY}"),
}
]
}
)
assert (
out[0]["error"]
== "Indexing failed. Check the server logs for details."
)
assert _FAKE_API_KEY not in out[0]["error"]
def test_only_known_service_messages_are_preserved(self):
from local_deep_research.web.routers.rag import (
_sanitized_indexing_errors,
)
out = _sanitized_indexing_errors(
{
"errors": [
{"error": "Document deleted during indexing"},
{"error": "Document deleted during indexing /srv/private"},
]
}
)
assert out[0]["error"] == "Document deleted during indexing"
assert (
out[1]["error"]
== "Indexing failed. Check the server logs for details."
)
#: Text ``LibraryRAGService`` returns as a result's ``"error"`` that clients
#: must not see verbatim: ``_client_indexing_error`` gives them the generic
#: message instead. ``{}`` marks each f-string placeholder.
_WITHHELD_SERVICE_ERRORS = frozenset(
{
"Operation failed: {}",
"Operation failed: VectorStoreIntegrityError",
"Indexing failed: {}",
"Worker crashed: {}",
"Indexing must run outside an existing database session",
}
)
#: ``"error"`` values that copy the error of a result the service's own
#: methods returned, as ``(enclosing function, source)``. The text they copy
#: is classified by ``_SAFE_INDEXING_ERRORS`` and the set above.
_FORWARDED_SERVICE_ERRORS = frozenset(
{
("index_all_documents", "result.get('error')"),
("index_documents_parallel", "result.get('error')"),
("index_documents_parallel", "scope_error['error']"),
}
)
#: Reconciliation reasons the service returns that clients must not see
#: verbatim. The status endpoint shows only ``_EMPTY_VECTOR_STORE_REASON``.
_WITHHELD_SERVICE_REASONS = frozenset()
def _library_rag_service_values(key):
"""Every value the ``LibraryRAGService`` module builds under ``key``.
A value counts when it is a dict-literal entry, a ``dict(...)`` or
``.update(...)`` keyword, or an ``x[key] = ...`` assignment; a
conditional expression counts as both of its branches. Returns
``(texts, others)``: ``texts`` holds each string and f-string, with
``{}`` for each placeholder, and ``others`` holds every other value as
``(enclosing function, source)``.
"""
import ast
import inspect
from local_deep_research.research_library.services import (
library_rag_service,
)
texts, others = set(), set()
def record(value, function):
if isinstance(value, ast.IfExp):
record(value.body, function)
record(value.orelse, function)
elif isinstance(value, ast.Constant) and isinstance(value.value, str):
texts.add(value.value)
elif isinstance(value, ast.JoinedStr):
texts.add(
"".join(
part.value if isinstance(part, ast.Constant) else "{}"
for part in value.values
)
)
else:
others.add((function, ast.unparse(value)))
def visit(node, function):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
function = node.name
if isinstance(node, ast.Dict):
for dict_key, value in zip(node.keys, node.values):
if isinstance(dict_key, ast.Constant) and dict_key.value == key:
record(value, function)
elif isinstance(node, ast.Call) and (
getattr(node.func, "id", None) == "dict"
or getattr(node.func, "attr", None) == "update"
):
for keyword in node.keywords:
if keyword.arg == key:
record(keyword.value, function)
elif (
isinstance(node, (ast.Assign, ast.AnnAssign))
and node.value is not None
):
targets = (
node.targets if isinstance(node, ast.Assign) else [node.target]
)
for target in targets:
if (
isinstance(target, ast.Subscript)
and isinstance(target.slice, ast.Constant)
and target.slice.value == key
):
record(node.value, function)
for child in ast.iter_child_nodes(node):
visit(child, function)
visit(ast.parse(inspect.getsource(library_rag_service)), "<module>")
return texts, others
class TestIndexingErrorAllowlistMatchesService:
"""``_SAFE_INDEXING_ERRORS`` must match the error text that
``LibraryRAGService`` returns.
The RAG document index and remove routes, the note index route, both
indexing SSE streams and the index-status endpoint pass each
per-document ``"error"`` through ``_client_indexing_error``. An exact
match in ``_SAFE_INDEXING_ERRORS`` reaches the client, and any other text
becomes the generic message. Every such value comes from
``LibraryRAGService``, so these tests collect the values its module
builds under ``"error"``, in the forms ``_library_rag_service_values``
lists. Each string or f-string must be allowlisted or
listed in ``_WITHHELD_SERVICE_ERRORS``, and each allowlisted string must
still be returned. Reconciliation ``"reason"`` values get the same check
against ``_EMPTY_VECTOR_STORE_REASON``, the one reason the status
endpoint shows. New or reworded text fails here until someone decides
whether clients may see it.
"""
def test_every_error_the_service_returns_is_classified(self):
from local_deep_research.web.routers.rag import _SAFE_INDEXING_ERRORS
texts, others = _library_rag_service_values("error")
unclassified = sorted(
texts - _SAFE_INDEXING_ERRORS - _WITHHELD_SERVICE_ERRORS
)
assert unclassified == [], (
f"LibraryRAGService returns unclassified error text {unclassified}."
" Add fixed guidance a user can act on to _SAFE_INDEXING_ERRORS"
" in web/routers/rag.py, and anything else to"
" _WITHHELD_SERVICE_ERRORS in this file."
)
assert others == _FORWARDED_SERVICE_ERRORS, (
"LibraryRAGService builds an error value that is neither text nor"
f" a known copy of another result's error: {sorted(others)}"
)
def test_every_classified_error_is_still_returned(self):
"""An allowlisted string that nothing returns hides drift: a test
that pins it passes while the service's real message is withheld."""
from local_deep_research.web.routers.rag import _SAFE_INDEXING_ERRORS
texts, _ = _library_rag_service_values("error")
assert not _SAFE_INDEXING_ERRORS & _WITHHELD_SERVICE_ERRORS
assert sorted(_SAFE_INDEXING_ERRORS - texts) == [], (
"allowlisted error text that LibraryRAGService no longer returns"
)
assert sorted(_WITHHELD_SERVICE_ERRORS - texts) == [], (
"withheld error text that LibraryRAGService no longer returns"
)
def test_every_reconciliation_reason_is_classified(self):
from local_deep_research.web.routers.rag import (
_EMPTY_VECTOR_STORE_REASON,
)
texts, others = _library_rag_service_values("reason")
unclassified = sorted(
texts - {_EMPTY_VECTOR_STORE_REASON} - _WITHHELD_SERVICE_REASONS
)
assert unclassified == [], (
f"LibraryRAGService returns unclassified reason {unclassified}."
" Make it _EMPTY_VECTOR_STORE_REASON in web/routers/rag.py if"
" clients should see it, or add it to _WITHHELD_SERVICE_REASONS"
" in this file."
)
assert others == set()
assert _EMPTY_VECTOR_STORE_REASON in texts
assert _WITHHELD_SERVICE_REASONS <= texts
def test_clients_receive_each_text_as_classified(self):
from local_deep_research.web.routers.rag import (
_EMPTY_VECTOR_STORE_REASON,
_SAFE_INDEXING_ERRORS,
_client_indexing_error,
_client_reconciliation_reason,
)
for text in _SAFE_INDEXING_ERRORS:
assert _client_indexing_error(text) == text
for detail in (
f"{text}: /srv/private.db",
f"/srv/private.db: {text}",
):
assert _client_indexing_error(detail) == (
"Indexing failed. Check the server logs for details."
)
for template in _WITHHELD_SERVICE_ERRORS:
assert _client_indexing_error(
template.replace("{}", "OperationalError")
) == ("Indexing failed. Check the server logs for details.")
assert (
_client_reconciliation_reason(_EMPTY_VECTOR_STORE_REASON)
== _EMPTY_VECTOR_STORE_REASON
)
#: Fixed text that ``web/routers/rag.py`` writes to an indexing task's
#: ``error_message`` without a failure record, and that the index-status
#: endpoint must not serve as stored.
_WITHHELD_TASK_ERRORS = frozenset()
def _indexing_task_error_writes():
"""The fixed texts ``web/routers/rag.py`` writes to an indexing task's
``error_message``, as ``(with_failure, without_failure)``.
That module creates every indexing ``TaskMetadata`` row and is the only
one that writes them. A write is an ``error_message=`` keyword of any
call, such as ``_update_task_status(...)`` or ``TaskMetadata(...)``, or
an ``x.error_message = ...`` assignment. Its text counts when it is a
string, an f-string (``{}`` for each placeholder) or a module-level
string constant; a conditional expression counts as both of its
branches. A write whose call also passes a ``failure`` record goes in
``with_failure``: for a record it recognises, the status endpoint
rebuilds the text instead of matching it.
"""
import ast
import inspect
from local_deep_research.web.routers import rag
namespace = vars(rag)
with_failure, without_failure = set(), set()
def texts(value):
if isinstance(value, ast.IfExp):
return texts(value.body) | texts(value.orelse)
if isinstance(value, ast.Constant) and isinstance(value.value, str):
return {value.value}
if isinstance(value, ast.JoinedStr):
return {
"".join(
part.value if isinstance(part, ast.Constant) else "{}"
for part in value.values
)
}
if isinstance(value, ast.Name) and isinstance(
namespace.get(value.id), str
):
return {namespace[value.id]}
return set()
for node in ast.walk(ast.parse(inspect.getsource(rag))):
if isinstance(node, ast.Call):
keywords = {keyword.arg: keyword.value for keyword in node.keywords}
if "error_message" in keywords:
failure = keywords.get("failure")
records = failure is not None and not (
isinstance(failure, ast.Constant) and failure.value is None
)
(with_failure if records else without_failure).update(
texts(keywords["error_message"])
)
elif isinstance(node, ast.Assign) and any(
isinstance(target, ast.Attribute) and target.attr == "error_message"
for target in node.targets
):
without_failure.update(texts(node.value))
return with_failure, without_failure
def _stored_message_pass_throughs(source, namespace):
"""The stored texts the task-error function in ``source`` serves
unchanged.
The function's first parameter is the stored ``error_message``. Its
pass-throughs are the texts it compares that parameter with exactly:
``==`` with a string, or ``in`` with a collection of strings, each
written out or a name bound in ``namespace``. Returns
``(texts, others)``. ``others`` holds the source of every other use of
the parameter, so a new shape fails the guard instead of passing
unseen. Three uses are allowed: an ``is None`` or ``is not None`` check,
an ``isinstance`` check, and a ``return`` of the parameter directly in
the body of an ``if`` whose test is such a comparison, alone or in an
``and``.
"""
import ast
import textwrap
function = ast.parse(textwrap.dedent(source)).body[0]
param = function.args.args[0].arg
parents = {
child: node
for node in ast.walk(function)
for child in ast.iter_child_nodes(node)
}
def is_param(node):
return isinstance(node, ast.Name) and node.id == param
def text_of(node):
if isinstance(node, ast.Constant) and isinstance(node.value, str):
return node.value
if isinstance(node, ast.Name) and isinstance(
namespace.get(node.id), str
):
return namespace[node.id]
return None
def texts_in(node):
if isinstance(node, (ast.Set, ast.Tuple, ast.List)):
items = [text_of(element) for element in node.elts]
elif isinstance(node, ast.Name) and isinstance(
namespace.get(node.id), (frozenset, set, tuple, list)
):
items = list(namespace[node.id])
else:
return None
if all(isinstance(item, str) for item in items):
return set(items)
return None
def exact_match(node):
if not (isinstance(node, ast.Compare) and len(node.ops) == 1):
return None
left, op, right = node.left, node.ops[0], node.comparators[0]
if isinstance(op, ast.Eq):
if is_param(left) and text_of(right) is not None:
return {text_of(right)}
if is_param(right) and text_of(left) is not None:
return {text_of(left)}
if isinstance(op, ast.In) and is_param(left):
return texts_in(right)
return None
def guarded(statement):
branch = parents.get(statement)
if not (isinstance(branch, ast.If) and statement in branch.body):
return False
test = branch.test
if isinstance(test, ast.BoolOp) and isinstance(test.op, ast.And):
tests = test.values
else:
tests = [test]
return any(exact_match(each) is not None for each in tests)
def allowed(use, parent):
if isinstance(parent, ast.Compare):
return (
parent.left is use
and len(parent.ops) == 1
and isinstance(parent.ops[0], (ast.Is, ast.IsNot))
and isinstance(parent.comparators[0], ast.Constant)
and parent.comparators[0].value is None
)
if isinstance(parent, ast.Call):
return (
getattr(parent.func, "id", None) == "isinstance"
and bool(parent.args)
and parent.args[0] is use
)
return isinstance(parent, ast.Return) and guarded(parent)
texts, others = set(), set()
for node in ast.walk(function):
if not (is_param(node) and isinstance(node.ctx, ast.Load)):
continue
parent = parents[node]
matched = exact_match(parent)
if matched is not None:
texts |= matched
elif not allowed(node, parent):
others.add(ast.unparse(parent))
return texts, others
def _task_error_pass_throughs():
"""``_stored_message_pass_throughs`` for ``_client_indexing_task_error``."""
import inspect
from local_deep_research.web.routers import rag
return _stored_message_pass_throughs(
inspect.getsource(rag._client_indexing_task_error), vars(rag)
)
class TestIndexingTaskErrorPassThroughs:
"""Stored task errors reach clients unchanged only for text the indexing
code writes.
``get_index_status`` passes each task's stored ``error_message`` through
``_client_indexing_task_error``. That function rebuilds the client text
from the failure record or the result counts, and serves the stored text
itself only on an exact match, a pass-through. These tests check the
pass-throughs against the text ``web/routers/rag.py`` writes to indexing
task rows. A pass-through that nothing writes is dead: a test that seeds
its text into a row passes while no code path produces it. Fixed text
written without a failure record must be passed through or listed in
``_WITHHELD_TASK_ERRORS``, so it cannot be withheld unnoticed.
"""
def test_every_pass_through_is_written_to_a_task_row(self):
from local_deep_research.web.routers.rag import (
_client_indexing_task_error,
)
with_failure, without_failure = _indexing_task_error_writes()
pass_throughs, _ = _task_error_pass_throughs()
dead = sorted(pass_throughs - with_failure - without_failure)
assert dead == [], (
f"_client_indexing_task_error serves stored text {dead} unchanged,"
" but web/routers/rag.py never writes it to an indexing task row."
" Remove the pass-through, or add the code that writes the text."
)
for text in pass_throughs:
assert _client_indexing_task_error(text, None) == text
def test_every_fixed_text_written_without_a_record_is_classified(self):
from local_deep_research.web.routers.rag import (
_COLLECTION_NOT_FOUND,
_client_indexing_task_error,
)
with_failure, without_failure = _indexing_task_error_writes()
pass_throughs, _ = _task_error_pass_throughs()
unclassified = sorted(
without_failure - pass_throughs - _WITHHELD_TASK_ERRORS
)
assert unclassified == [], (
f"web/routers/rag.py writes task error text {unclassified} with no"
" failure record, and the index-status endpoint does not serve it."
" Add an exact-match pass-through to _client_indexing_task_error"
" if clients should see it, or add it to _WITHHELD_TASK_ERRORS in"
" this file."
)
assert not pass_throughs & _WITHHELD_TASK_ERRORS
assert sorted(_WITHHELD_TASK_ERRORS - without_failure) == [], (
"withheld task error text that web/routers/rag.py no longer writes"
)
for text in _WITHHELD_TASK_ERRORS:
assert _client_indexing_task_error(text, None) == (
"Indexing failed. Check the server logs for details."
)
# Control: the scan sees the worker's writes.
assert _COLLECTION_NOT_FOUND in with_failure
def test_stored_text_is_served_only_on_an_exact_match(self):
_, others = _task_error_pass_throughs()
assert sorted(others) == [], (
"_client_indexing_task_error uses the stored message in a way"
f" these checks do not classify: {sorted(others)}"
)
def test_the_scan_recognises_each_pass_through_shape(self):
"""Control for the checks above, which find no pass-through in the
current code."""
source = """
def task_error(message, result, failure=None):
if message is None:
return None
if message == "Literal text":
return message
if isinstance(message, str) and message in NAMED:
return message
if message in {"Set text"}:
return message
if message.startswith("Prefix"):
return message
return "generic"
"""
texts, others = _stored_message_pass_throughs(
source, {"NAMED": frozenset({"Named text"})}
)
assert texts == {"Literal text", "Named text", "Set text"}
assert others == {"message.startswith", "return message"}
# ===========================================================================
# COVERAGE AREA 5 -- is_downloadable_domain URL allowlist
# ===========================================================================
class TestIsDownloadableDomainAllowlist:
"""``research_library/utils/is_downloadable_domain`` decides which remote
URLs the server will fetch (``library.py:879`` and ``:1539``, and the
``POST /library/api/download-source`` gate at ``:1766``). This class is the
direct policy regression evidence that was absent at the review snapshot.
The policy these tests pin is the one the implementation and the
``is_downloadable_url`` docstring describe: "from a downloadable academic
domain OR a direct PDF link". Host matching is exact-or-dot-suffix against
a fixed list, after ``urlparse(url.lower()).hostname``.
"""
def test_allowlisted_hosts_pass(self):
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://arxiv.org/abs/2401.12345",
"https://www.nature.com/articles/s41586-024-00000-0",
"https://pubmed.ncbi.nlm.nih.gov/12345678/",
"https://doi.org/10.1000/xyz123",
"https://openreview.net/forum?id=abc",
]:
assert is_downloadable_domain(url) is True, url
def test_subdomains_of_an_allowlisted_host_pass(self):
"""``hostname.endswith("." + domain)`` -- a real subdomain is in."""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("https://export.arxiv.org/abs/1") is True
def test_matching_is_case_insensitive(self):
"""``urlparse(url.lower())`` -- an upper/mixed-case host must not be
able to slip past the allowlist, and must not be wrongly rejected
either.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("https://ARXIV.ORG/abs/1") is True
assert is_downloadable_domain("https://ArXiv.Org/abs/1") is True
assert is_downloadable_domain("https://EVIL.TEST/index.html") is False
def test_non_allowlisted_hosts_are_refused(self):
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://evil.test/index.html",
"https://example.com/paper",
# SSRF classics -- no allowlisted host, no PDF-shaped path.
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:8080/admin",
"http://[::1]/admin",
"http://localhost/admin",
]:
assert is_downloadable_domain(url) is False, url
def test_userinfo_cannot_forge_an_allowlisted_host(self):
"""``https://[email protected]/`` -- the allowlisted name appears in
the userinfo, not the host. The check reads ``parsed.hostname`` (which
strips userinfo), never the raw netloc, so this is refused. A rewrite
to string matching on the URL or the netloc would flip this.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://[email protected]/",
"https://arxiv.org:[email protected]/",
"https://[email protected]:[email protected]/paper",
]:
assert is_downloadable_domain(url) is False, url
def test_suffix_and_prefix_lookalike_hosts_are_refused(self):
"""The dot-anchored suffix check must not admit
``evil-arxiv.org``/``arxivorg.test``, and the exact check must not
admit ``arxiv.org.evil.test``.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://evil-arxiv.org/abs/1",
"https://arxivorg.test/abs/1",
"https://arxiv.org.evil.test/abs/1",
"https://notarxiv.org/abs/1",
"https://nature.com.evil.test/paper",
]:
assert is_downloadable_domain(url) is False, url
def test_trailing_dot_fqdn_is_refused(self):
"""CURRENT BEHAVIOUR: ``arxiv.org.`` (root-anchored FQDN) is not
normalised, so it does not match the allowlist and the URL is
REFUSED. That is the fail-closed direction, so it is pinned as-is --
but it is the one place where an allowlisted resource is turned away.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("https://arxiv.org./abs/1") is False
def test_falsy_and_unparseable_input_is_refused(self):
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in ["", None, "not a url", "://", "https://"]:
assert is_downloadable_domain(url) is False, repr(url)
def test_non_http_schemes_without_a_pdf_shape_are_refused(self):
"""The deleted suite's ``file://``/``ftp://`` tests were vacuous
(``assert result is True or result is False``). These assert the
outcome.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("file:///etc/passwd") is False
assert is_downloadable_domain("ftp://evil.test/x") is False
def test_is_downloadable_url_delegates_unchanged(self):
"""``is_downloadable_url`` is the documented "single source of truth"
wrapper used elsewhere; it must not diverge from the gate.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
is_downloadable_url,
)
for url in [
"https://arxiv.org/abs/1",
"https://evil.test/index.html",
"https://[email protected]/",
]:
assert is_downloadable_url(url) is is_downloadable_domain(url), url
class TestIsDownloadableDomainKnownWeaknesses:
"""CURRENT BEHAVIOUR, pinned deliberately -- these are NOT assertions that
the remaining behaviour is correct.
Private-host PDF candidates and PubMed hostname lookalikes are rejected.
PDF-shaped URLs on public hosts and the host-independent PubMed-path
heuristic remain accepted for compatibility.
"""
def test_pdf_shaped_paths_still_bypass_the_allowlist_for_public_hosts(
self,
):
"""The ``.pdf`` / ``/pdf/`` / ``type=pdf`` / ``format=pdf`` checks run
independently of the host allowlist: the direct-PDF check precedes
it, while the path/query checks follow it. They accept hosts that
pass the private/local exclusion. This remains the documented
"or is a direct PDF link" half of the policy (``is_downloadable_url``'s
docstring): the allowlist still does not constrain the fetch target
when the attacker controls the path or query on a public host --
e.g. ``https://evil.test/x?type=pdf``.
(Internal/metadata hosts are covered separately below -- those are
now rejected regardless of PDF shape.)
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://evil.test/payload.pdf",
"https://evil.test/pdf/anything",
"https://evil.test/x?type=pdf",
"https://evil.test/x?format=pdf",
]:
assert is_downloadable_domain(url) is True, url
def test_private_host_pdf_shaped_paths_are_now_rejected(self):
"""Regression pin for ``harden: reject private hosts in
is_downloadable_domain``. A private/loopback/link-local hostname is
now rejected FIRST, before any PDF-shape heuristic can short-circuit
past it -- closing the misleading appearance that the PDF checks
bypass the allowlist for internal or cloud-metadata targets such as
``http://127.0.0.1:8080/x?type=pdf`` or
``http://169.254.169.254/...?type=pdf``.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"http://127.0.0.1:8080/internal?type=pdf",
"http://169.254.169.254/latest/meta-data/x?type=pdf",
]:
assert is_downloadable_domain(url) is False, url
def test_the_pubmed_hostname_check_is_now_anchored(self):
"""Regression pin for ``harden: ... anchor the PubMed rule``. The
unanchored ``"pubmed" in hostname`` substring arm was removed, unlike
the exact/dot-suffix matching used for every entry in
``downloadable_domains``. An attacker-registered host that merely
contains "pubmed" as a substring -- and is not itself allowlisted or
a dot-suffix match of an allowlisted domain -- is no longer admitted.
(The real PubMed domains are unaffected: they are already in the
allowlist above and matched there by exact/suffix comparison.)
The same anchoring applies to the ``pubmed.gov`` /
``pubmedcentral.nih.gov`` entries added alongside this test: a host
that merely starts or ends with those strings -- without being an
exact match or dot-suffix -- does not qualify either.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://pubmed.evil.test/anything",
"https://evil-pubmed-mirror.test/x",
"https://pubmed.gov.evil.test/anything",
"https://evilpubmed.gov/x",
"https://pubmedcentral.nih.gov.attacker.com/x",
]:
assert is_downloadable_domain(url) is False, url
def test_the_pubmed_path_check_remains_an_unanchored_substring_match(
self,
):
"""CURRENT BEHAVIOUR, pinned deliberately -- the hardening commit
intentionally kept the PATH arm of the PubMed special case
(``"/pubmed/" in path``), and it is still an unanchored substring
match: any public host serving a ``/pubmed/`` path is admitted. This
is narrower than the removed hostname arm (an attacker needs a path
segment, not just a hostname substring) but is still not
allowlist-anchored.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("https://evil.test/pubmed/x") is True
def test_pubmed_and_pmc_real_host_forms_still_pass(self):
"""The hardening commit removed only the unanchored ``"pubmed" in
hostname`` substring arm; the exact/dot-suffix allowlist match
above it (untouched by that removal) is what actually admits every
real PubMed/PMC/NCBI host, so none of them regressed.
``pubmed.gov`` and ``pubmedcentral.nih.gov`` are the official
short-form PubMed/PMC hosts and are separate entries from
``pubmed.ncbi.nlm.nih.gov`` / ``ncbi.nlm.nih.gov`` -- removing the
unanchored hostname substring arm dropped them (they are not a
dot-suffix of any allowlisted domain), so they are pinned here as
their own explicit allowlist entries rather than folded into the
broader ``nih.gov``.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"https://pubmed.ncbi.nlm.nih.gov/12345678/",
"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234567/",
"https://europepmc.org/article/MED/12345678",
"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/",
"https://pubmed.gov/12345678",
"https://www.pubmed.gov/12345678",
"https://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=123456",
]:
assert is_downloadable_domain(url) is True, url
class TestIsDownloadableDomainPrivateHostNormalization:
"""Regression pins for the call-site-only normalization layered on top
of ``is_private_ip()`` inside ``is_downloadable_domain`` (trailing dot,
single-token decimal/hex IPv4, RFC 6598 CGNAT).
``is_private_ip()`` itself is unmodified: it is a SHARED helper (also
used by ``utilities/url_utils.py`` for outbound-request scheme
selection and ``web/dependencies/rate_limit.py`` for the local-network
rate-limit exemption), so changing what it considers private is out of
scope for this filter and could change behaviour for those unrelated
callers. The normalization instead lives in
``research_library/utils/__init__.py`` (``_decode_ipv4_literal``,
``_is_blocked_private_host``), applied only at this call site.
"""
def test_alternate_ip_encodings_of_a_private_host_are_now_rejected(self):
"""``is_private_ip()`` only parses a hostname already written as a
dotted-quad / colon-hex literal, so a private-host URL that is
PDF-shaped used to sail straight past the guard (the PDF check
returns True before the allowlist, and the private-host check
could not recognise the host). These four textual variants of
``127.0.0.1`` / ``192.168.1.10`` / ``printer.local`` -- all
resolved by glibc/requests to their private target -- are now
normalized and rejected.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"http://2130706433/x.pdf", # decimal for 127.0.0.1
"http://0x7f000001/x.pdf", # hex for 127.0.0.1
"http://192.168.1.10./x.pdf", # trailing-dot RFC1918
"http://printer.local./x.pdf", # trailing-dot mDNS
]:
assert is_downloadable_domain(url) is False, url
def test_idna_private_host_spellings_are_rejected(self):
"""Normalize Unicode numeric labels and separators before filtering."""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for host in [
"127.0.0.1",
"127。0。0。1",
"127.0.0.1",
"127。0。0。1",
"2130706433",
"0x7f000001",
"printer。local。",
"100.64.0.1",
]:
for suffix in ["/x.pdf", "/pdf/x", "/x?type=pdf", "/pubmed/x"]:
url = f"http://{host}{suffix}"
assert is_downloadable_domain(url) is False, url
for host in [
"8.8.8.8",
"100.63.255.255",
"bücher.example",
]:
url = f"http://{host}/x.pdf"
assert is_downloadable_domain(url) is True, url
def test_mapped_ipv6_cgnat_hosts_are_rejected(self):
"""Apply the CGNAT exclusion to the embedded IPv4 address."""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for host in [
"::ffff:100.64.0.0",
"::ffff:100.64.0.1",
"::ffff:100.127.255.255",
"::ffff:6440:1",
]:
for suffix in ["/x.pdf", "/x?format=pdf", "/pubmed/x"]:
url = f"http://[{host}]{suffix}"
assert is_downloadable_domain(url) is False, url
for host in [
"::ffff:100.63.255.255",
"::ffff:100.128.0.0",
"::ffff:8.8.8.8",
]:
url = f"http://[{host}]/x.pdf"
assert is_downloadable_domain(url) is True, url
def test_wrapped_ipv6_private_spellings_are_rejected(self):
"""The candidate loop's ``ipv4_mapped`` unwrap used to feed ONLY the
CGNAT check, so IPv6-wrapped spellings of exactly the private
targets this filter claims to reject were admitted: the
IPv4-compatible (``::169.254.169.254``), IPv4-translated
(``::ffff:0:169.254.169.254``) and NAT64 (``64:ff9b::a9fe:a9fe``)
forms all parse to IPv6 addresses whose ``ipv4_mapped`` is None,
and ``is_private_ip`` does not classify the transition prefixes as
private. The shared recognition table
``security.ip_ranges.PRIVATE_IP_RANGES`` -- already the source of
truth for the fetch-time chokepoint -- recognizes every one of
these as private, so the candidate loop now checks each parsed
candidate (and its mapped unwrap) against that table.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for host in [
"::169.254.169.254", # IPv4-compatible wrap of IMDS
"::ffff:0:169.254.169.254", # IPv4-translated wrap of IMDS
"64:ff9b::a9fe:a9fe", # NAT64 well-known prefix wrap of IMDS
"::127.0.0.1", # IPv4-compatible wrap of loopback
"::ffff:0:100.64.0.1", # translated wrap of the pinned CGNAT vector
]:
for suffix in ["/x.pdf", "/latest/meta-data/x?type=pdf"]:
url = f"http://[{host}]{suffix}"
assert is_downloadable_domain(url) is False, url
# WHAT-HELD controls: public IPv6 literals (native and mapped) and
# the CGNAT boundary publics keep their exact head behaviour.
for url in [
"http://[2600::]/x.pdf",
"http://[::ffff:8.8.8.8]/x.pdf",
"http://100.63.255.255/x.pdf",
"http://100.128.0.1/x.pdf",
]:
assert is_downloadable_domain(url) is True, url
def test_rfc6598_cgnat_hosts_are_now_rejected(self):
"""``is_private_ip()`` does not classify ``100.64.0.0/10`` (RFC 6598
shared/CGNAT address space) as private; this module checks the
range itself at the call site, without touching the shared helper.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("http://100.64.0.5/x.pdf") is False
# Sanity: just outside the /10 range is unaffected either way.
assert is_downloadable_domain("http://100.63.255.255/x.pdf") is True
assert is_downloadable_domain("http://100.128.0.1/x.pdf") is True
def test_dotted_ipv4_literal_shorthand_and_octal_are_now_rejected(self):
"""Regression pin: the shorthand-dotted (``127.1``) and per-octet
octal (``0177.0.0.1``) gaps that ``test_is_downloadable_domain_known_
private_ip_gaps`` used to pin as open are now closed.
``_decode_ipv4_literal`` decodes via ``socket.inet_aton`` (glibc's
own IPv4-literal grammar), not a hand-rolled single-token parser, so
it recognizes every dotted/shorthand/octal/hex form a resolver
does -- not just a bare integer with no dots.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
"http://127.1/x.pdf", # shorthand for 127.0.0.1
"http://0177.0.0.1/x.pdf", # per-octet octal for 127.0.0.1
]:
assert is_downloadable_domain(url) is False, url
def test_single_token_leading_zero_octal_is_now_rejected(self):
"""CONFIRMED bypass, now fixed: the previous decoder used
``int(hostname, 0)`` -- Python integer-literal syntax -- which
raises ``ValueError`` on a single-token leading-zero octal string
like ``017700000001`` (Python requires an explicit ``0o`` prefix
for octal; a bare leading zero is a ``SyntaxError``/``ValueError``
in modern Python). glibc's ``inet_aton`` has no such requirement:
``017700000001`` and ``0177`` are valid octal literals to it and
resolve to ``127.0.0.1`` / ``0.0.0.127`` respectively, so a URL
built from either sailed straight past the old guard. Verified via
``socket.getaddrinfo`` that this is exactly what a real resolver
does with these strings too.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
assert is_downloadable_domain("http://017700000001/x.pdf") is False
assert is_downloadable_domain("http://0177/x.pdf") is False
def test_python_literal_only_forms_are_no_longer_misdecoded(self):
"""The old ``int(hostname, 0)`` decoder also over-decoded: Python
accepts ``0b``-binary and underscore-grouped digits as integer
literals, but no resolver treats either as an IP address --
``socket.inet_aton`` (and ``socket.getaddrinfo``) reject both
outright. These hostnames both encode ``127.0.0.1`` under Python's
literal grammar, but since a resolver never decodes them that way,
they must NOT be treated as the private loopback address by this
filter either -- confirmed here via a PDF-shaped path, which would
be forced to False if (wrongly) decoded as private.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for url in [
# 0b01111111... is 127.0.0.1 in binary -- a Python int literal,
# not anything a resolver treats as an IPv4 address.
"http://0b01111111000000000000000000000001/x.pdf",
# Underscore-grouped decimal for 2130706433 (127.0.0.1) -- valid
# to Python's int(x, 0), meaningless to a resolver.
"http://2_130_706_433/x.pdf",
]:
assert is_downloadable_domain(url) is True, url
def test_percent_encoded_private_hosts_are_now_rejected(self):
"""CONFIRMED bypass, now fixed: the private-host check ran against
the raw, still-percent-encoded ``urlparse().hostname`` string, so an
encoded loopback/local host never matched any recognized literal
form and sailed straight past the guard as an opaque non-matching
hostname -- exactly the same way an unrecognized public hostname
would. ``%31%32%37.0.0.1`` percent-decodes to ``127.0.0.1``;
``%6c%6f%63%61%6c%68%6f%73%74`` decodes to ``localhost``;
``%31%30%2e%30%2e%30%2e%31`` decodes to ``10.0.0.1``. The hostname
is now percent-decoded (one pass, via ``urllib.parse.unquote``)
before IDNA normalization and the legacy-IPv4 decode, so these are
recognized the same as their plain-text spellings.
"""
from local_deep_research.research_library.utils import (
is_downloadable_domain,
)
for host in [
"%31%32%37.0.0.1", # percent-encoded 127.0.0.1
"127.0.0.1%00", # trailing percent-encoded NUL after loopback
"127.0.0.1%2e", # percent-encoded trailing dot
"%6c%6f%63%61%6c%68%6f%73%74", # percent-encoded "localhost"
"%31%30%2e%30%2e%30%2e%31", # percent-encoded 10.0.0.1
]:
for suffix in ["/x.pdf", "/pubmed/x", "/x?type=pdf"]:
url = f"http://{host}{suffix}"
assert is_downloadable_domain(url) is False, url
# WHAT-HELD control: an ordinary allowlisted host is unaffected by
# percent-decoding (there is nothing to decode).
assert is_downloadable_domain("https://arxiv.org/abs/1") is True
class TestDownloadSourceEnforcesTheAllowlist:
"""The HTTP enforcement point (``library.py:1766``). Pairs the refusal
with a positive control that an allowlisted URL gets PAST the gate --
otherwise a route that 400s on everything would satisfy the refusal test.
Neither case performs any network IO: the refusal returns before the
threadpool call, and the allowlisted case stops at "Resource not found"
because no matching ``ResearchResource`` row exists.
"""
def test_non_allowlisted_url_is_refused_with_400(self, library_client):
client, _, _ = library_client
resp = client.post(
"/library/api/download-source",
json={"research_id": 1, "url": "https://evil.test/index.html"},
)
assert resp.status_code == 400, resp.text
assert resp.json()["error"] == "URL is not from a downloadable domain"
def test_allowlisted_url_passes_the_gate(self, library_client):
"""Positive control: the same request shape with an allowlisted host
reaches the resource lookup (404), proving the 400 above is the
allowlist gate and not generic request rejection.
"""
client, _, _ = library_client
resp = client.post(
"/library/api/download-source",
json={"research_id": 1, "url": "https://arxiv.org/abs/2401.12345"},
)
assert resp.status_code == 404, resp.text
assert resp.json()["error"] == "Resource not found"