~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(security): green main (TestQuickSummary) + clear egress block messages + quick_summary settings opt-in (#4456)
LearningCircuit8c0b0028fff7
diff --git a/changelog.d/+egress-clear-block-messages.feature.md b/changelog.d/+egress-clear-block-messages.feature.mdnew file mode 100644index 0000000000..24285b9edc--- /dev/null+++ b/changelog.d/+egress-clear-block-messages.feature.md@@ -0,0 +1,3 @@+When the egress policy blocks something, the user now gets a **clear, actionable message** — what was blocked, why, and exactly which setting to change to allow it (e.g. "blocked because your Egress Scope is Private only → change it in Settings → Privacy & Egress, or turn off 'Require local LLM endpoint'") — instead of a terse machine code. A new `security.egress.guidance.denial_guidance()` helper centralises these messages; it's wired into the research start-up engine check, and denials surface the raw reason code alongside for support.++The `quick_summary` API also gains an opt-in: pass `"allow_default_settings": true` to deliberately run with default settings (no egress policy) when your settings can't be loaded, instead of the default fail-closed `503`. The 503 itself now explains the cause and the fix (retry / re-authenticate / opt in).diff --git a/changelog.d/+fix-get-settings-manager-import-shadow.misc.md b/changelog.d/+fix-get-settings-manager-import-shadow.misc.mdnew file mode 100644index 0000000000..d90d2c5834--- /dev/null+++ b/changelog.d/+fix-get-settings-manager-import-shadow.misc.md@@ -0,0 +1 @@+Removed the redundant local re-import of `get_settings_manager` inside the `quick_summary` API endpoint (it was already bound at module level), eliminating the same import-shadowing anti-pattern fixed for `get_user_db_session` — so `patch("...web.api.get_settings_manager")` now reliably intercepts. Both helpers are now module-level for one consistent patch surface; only `quick_summary` itself remains a local import (it pulls in the research stack, which would cycle).diff --git a/changelog.d/+fix-quick-summary-settings-mock.bugfix.md b/changelog.d/+fix-quick-summary-settings-mock.bugfix.mdnew file mode 100644index 0000000000..0c003207fe--- /dev/null+++ b/changelog.d/+fix-quick-summary-settings-mock.bugfix.md@@ -0,0 +1 @@+Fixed 5 `TestQuickSummary` API tests that broke after the egress fail-closed change (#4300): the `quick_summary` endpoint re-imported `get_user_db_session` locally, shadowing the module-level binding so `patch("...web.api.get_user_db_session")` never intercepted — the tests passed only because the old code swallowed the resulting DB error. Removed the redundant local import (one consistent patch target) and updated the settings-failure test to assert the new fail-closed `503` behaviour instead of the obsolete empty-snapshot fallback.diff --git a/src/local_deep_research/security/egress/guidance.py b/src/local_deep_research/security/egress/guidance.pynew file mode 100644index 0000000000..c94a546a1e--- /dev/null+++ b/src/local_deep_research/security/egress/guidance.py@@ -0,0 +1,183 @@+"""Human-facing guidance for egress-policy denials.++The PDP returns terse machine reason codes (``scope_mismatch_private_only``,+``provider_cloud_only``, …) — great for logs, useless for a user staring at a+blocked action. This module maps each reason to a CLEAR sentence that says+WHAT was blocked, WHY, and — crucially — HOW TO ALLOW IT (the exact setting to+change), so a block is never a dead end.++Use :func:`denial_guidance` at any USER-FACING surface (HTTP error responses,+the research form, tool errors). Keep the raw ``reason`` code in audit logs.++Implementation note: messages are plain strings with ``{target}`` and+``{scope_setting}`` placeholders, resolved by a single ``str.format`` in+:func:`denial_guidance` — NOT f-strings — so every entry uses the same+``{target}`` syntax (no ``{{target}}`` escaping traps). Don't put a literal+``{``/``}`` in a message, or ``.format`` will choke.+"""++from __future__ import annotations++from typing import Optional++# Where the user changes the egress scope, named once so the wording stays+# consistent everywhere. Injected as the ``{scope_setting}`` placeholder.+_SCOPE_SETTING = (+ "Settings → Privacy & Egress → Egress Scope (or the Privacy & Egress panel "+ "on the research form, for a one-off run)"+)++# reason code -> (what happened, how to allow it). ``{target}`` is filled with+# the blocked engine/provider/host; ``{scope_setting}`` with _SCOPE_SETTING.+_GUIDANCE: dict[str, tuple[str, str]] = {+ "scope_mismatch_private_only": (+ "{target} was blocked because your Egress Scope is set to "+ "Private only — only local sources (your collections, local engines) "+ "may run, and nothing leaves the machine.",+ "To use it, change your Egress Scope to Both or Public only in "+ "{scope_setting}.",+ ),+ "scope_mismatch_public_only": (+ "{target} was blocked because your Egress Scope is set to "+ "Public only — local/private sources are excluded from this run.",+ "To use it, change your Egress Scope to Both or Private only in "+ "{scope_setting}; or, if this is a collection whose contents are "+ "non-sensitive, mark it Public on the collection page.",+ ),+ "strict_public_host": (+ "{target} was blocked because your Egress Scope is Strict — only your "+ "single primary engine may run, with no expansion to other hosts.",+ "To allow it, change your Egress Scope away from Strict in "+ "{scope_setting}.",+ ),+ "strict_not_primary": (+ "{target} was blocked because your Egress Scope is Strict — only your "+ "chosen primary search engine may run.",+ "To use {target}, either make it your primary engine, or change "+ "your Egress Scope away from Strict in {scope_setting}.",+ ),+ "strict_with_meta_picker": (+ "Strict scope can't be combined with a meta-picker primary engine "+ "(auto / meta / parallel), which has no single concrete engine to "+ "restrict to.",+ "Pick a specific primary engine (e.g. arxiv, searxng), or change your "+ "Egress Scope away from Strict in {scope_setting}.",+ ),+ "blocked_metadata_ip": (+ "{target} was blocked because it targets a cloud-metadata endpoint "+ "(e.g. 169.254.169.254). These are NEVER permitted under any scope — "+ "they are a common credential-theft (SSRF) target.",+ "This is a hard safety rule and cannot be overridden in settings. If "+ "you believe this is a mistake, the host genuinely resolves to a "+ "cloud-metadata address.",+ ),+ "provider_cloud": (+ "Cloud embeddings ({target}) were blocked because local embeddings are "+ "required for this run (Private only scope, or the “Require local "+ "embeddings” toggle).",+ "To use cloud embeddings, turn off “Require local embeddings”, or "+ "change your Egress Scope away from Private only in {scope_setting}; "+ "or configure a local embeddings endpoint (e.g. sentence_transformers "+ "or a local Ollama URL).",+ ),+ "provider_cloud_only": (+ "The cloud LLM provider “{target}” was blocked because a local LLM is "+ "required for this run (Private only scope, or the “Require local LLM "+ "endpoint” toggle).",+ "To use this provider, turn off “Require local LLM endpoint”, or change "+ "your Egress Scope away from Private only in {scope_setting}; or "+ "switch to a local provider (Ollama, LM Studio, LlamaCpp).",+ ),+ "provider_remote": (+ "“{target}” was blocked because its configured endpoint resolves to a "+ "remote (non-local) host, and a local endpoint is required for this "+ "run.",+ "To use it, point its URL at a local address (localhost / your LAN), "+ "turn off the “Require local” toggle, or change your Egress Scope in "+ "{scope_setting}.",+ ),+ "provider_url_unset": (+ "“{target}” was blocked because no local endpoint URL is configured for "+ "it, so it can't be certified as local while a local endpoint is "+ "required.",+ "Configure a local URL for the provider, or use a local-default "+ "provider (Ollama, LM Studio, LlamaCpp), or turn off the “Require "+ "local” toggle.",+ ),+ "elasticsearch_cloud_id_public_egress": (+ "The Elasticsearch Cloud ID was blocked because your Egress Scope is "+ "Private only / Strict — a Cloud ID points at hosted Elastic Cloud "+ "(public egress).",+ "To use Elasticsearch locally, configure a local hosts= URL instead of "+ "a Cloud ID; or change your Egress Scope away from Private only / "+ "Strict in {scope_setting}.",+ ),+ "denial_quota_exceeded": (+ "This run hit its limit of blocked URL fetches — typically a document "+ "that loops the agent through many forbidden links.",+ "The run continues, but further blocked fetches are refused to protect "+ "performance. Widen your Egress Scope if these URLs should be allowed, "+ "or ignore this if the blocked links are junk.",+ ),+ "unknown_egress_scope": (+ "Your saved Egress Scope value is unrecognised (corrupted or set to an "+ "invalid value), so the run was refused rather than guessing.",+ "Re-select a valid Egress Scope in {scope_setting}.",+ ),+ "engine_unknown": (+ "The search engine “{target}” isn't recognised, so it was refused "+ "(fail-closed).",+ "Check the engine name, or pick a different search engine.",+ ),+ "unclassified": (+ "“{target}” couldn't be classified as public or local, so it was "+ "refused (fail-closed).",+ "Pick a recognised engine, or check the engine's configuration.",+ ),+}++# Reasons that are NOT really a policy block the user can act on — they are+# parse/format failures. Given a short, honest explanation instead of "change+# a setting".+_NON_POLICY = {+ "url_malformed": "The URL is malformed and could not be fetched.",+ "no_hostname": "The URL has no host and could not be fetched.",+ "unsupported_scheme": (+ "The link uses a scheme that can't be fetched (only http/https URLs "+ "are retrieved)."+ ),+ "dangerous_scheme": (+ "The link uses a non-web scheme (javascript:/data:/file:/…) and was "+ "skipped — it isn't fetchable content."+ ),+ "host_unclassified": "The host could not be resolved or classified.",+ "internal_error": (+ "An internal error occurred while evaluating the egress policy."+ ),+}+++def denial_guidance(reason: str, *, target: Optional[str] = None) -> str:+ """Return a clear, user-facing explanation + instructions for a denial.++ ``reason`` is the PDP machine code; ``target`` is the blocked thing+ (engine / provider / host), inserted into the message. Always returns a+ non-empty string, even for unknown reasons.+ """+ label = target or "This action"+ if reason in _GUIDANCE:+ what, how = _GUIDANCE[reason]+ # Two-step: the f-string only CONCATENATES the two plain templates+ # (no placeholder interpolation — {target}/{scope_setting} are literal+ # here), then .format() fills them. Don't collapse this into a single+ # f-string or the placeholders would need escaping again.+ return f"{what} {how}".format(+ target=label, scope_setting=_SCOPE_SETTING+ )+ if reason in _NON_POLICY:+ return _NON_POLICY[reason]+ # Unknown reason — be honest, don't invent an instruction.+ return (+ f"{label} was blocked by the egress policy (reason: {reason}). Check "+ f"your Egress Scope and the “Require local” toggles in {_SCOPE_SETTING}."+ )diff --git a/src/local_deep_research/web/api.py b/src/local_deep_research/web/api.pyindex 85e50766e4..82a1467861 100644--- a/src/local_deep_research/web/api.py+++ b/src/local_deep_research/web/api.py@@ -212,7 +212,20 @@ def api_quick_summary():query = data.get("query")if not isinstance(query, str):return jsonify({"error": "Query must be a string"}), 400- params = {k: v for k, v in data.items() if k != "query"}+ # Opt-in escape hatch for programmatic callers: when settings can't be+ # loaded, proceed with defaults (empty snapshot → permissive scope) instead+ # of failing closed (503). Default False (fail closed) so a configured+ # PRIVATE_ONLY user is never silently downgraded; setting it true is a+ # CONSCIOUS "I'm fine running without my settings/egress policy" choice.+ # Excluded from ``params`` so it isn't forwarded to quick_summary().+ # Strict ``is True`` (not bool()): for a security-boundary flag we only opt+ # in on a real JSON ``true`` — not on a truthy string like "false"/"0".+ allow_default_settings = data.get("allow_default_settings") is True+ params = {+ k: v+ for k, v in data.items()+ if k not in ("query", "allow_default_settings")+ }logger.debug(f"Query length: {len(query) if query else 0}, params keys: {list(params.keys()) if params else 'None'}")@@ -222,10 +235,15 @@ def api_quick_summary():params["username"] = usernametry:- # Import here to avoid circular imports+ # Import here to avoid circular imports. NOTE: get_user_db_session and+ # get_settings_manager are NOT re-imported here — both are bound at+ # module level (top of file). A local re-import would shadow the+ # module-level name and silently defeat ``patch("...web.api.<name>")``+ # in tests (the function would fetch the real, encrypted-DB-requiring+ # implementation instead of the mock), so keep them module-level for one+ # consistent patch target. Only quick_summary genuinely needs the local+ # import (it pulls in the research stack, which would cycle).from ..api.research_functions import quick_summary- from ..database.session_context import get_user_db_session- from ..utilities.db_utils import get_settings_managerlogger.info(f"Processing quick_summary request: query='{query}' for user='{username}'"@@ -272,25 +290,52 @@ def api_quick_summary():f"No database session for user: {username}")except Exception:- # Fail CLOSED: if the user's settings can't be loaded we can't- # read their egress policy (PRIVATE_ONLY, require-local, …), and- # continuing with an empty snapshot would silently downgrade- # them to the permissive BOTH scope — bypassing the very- # boundary they configured. Refuse the run instead of leaking.- logger.exception(- "Failed to load settings snapshot; refusing research to "- "avoid running without the user's egress policy"- )- return (- jsonify(- {- "error": "Could not load your settings, so the "- "research was refused to avoid bypassing your "- "privacy/egress policy. Please try again."- }- ),- 503,- )+ if allow_default_settings:+ # Caller explicitly opted in to run without their settings.+ # Proceed with defaults (empty snapshot → permissive scope).+ # Logged loudly so it's never a silent downgrade.+ logger.bind(policy_audit=True).warning(+ "Settings snapshot failed to load; proceeding with "+ "DEFAULT settings because allow_default_settings=true "+ "— this run is NOT bound by the user's egress policy",+ user=username,+ )+ params["settings_snapshot"] = {}+ else:+ # Fail CLOSED: if the user's settings can't be loaded we+ # can't read their egress policy (PRIVATE_ONLY, require-+ # local, …), and continuing with an empty snapshot would+ # silently downgrade them to the permissive BOTH scope —+ # bypassing the very boundary they configured. Refuse the+ # run with a clear, actionable message.+ logger.exception(+ "Failed to load settings snapshot; refusing research to "+ "avoid running without the user's egress policy"+ )+ return (+ jsonify(+ {+ "error": (+ "Your settings could not be loaded, so the "+ "research was REFUSED to avoid silently "+ "running without your privacy/egress policy "+ "(which could send your data to the cloud "+ "when you meant to keep it local)."+ ),+ "how_to_fix": (+ "This is usually transient — try again. If "+ "it persists, your encrypted settings "+ "database may be unavailable (e.g. a session "+ "/ password issue), so re-authenticate. To "+ "deliberately run with default settings and "+ "NO egress policy, resend the request with "+ '"allow_default_settings": true.'+ ),+ "reason": "settings_unavailable",+ }+ ),+ 503,+ )else:logger.debug("No username in session, skipping settings snapshot")params["settings_snapshot"] = {}diff --git a/src/local_deep_research/web/routes/research_routes.py b/src/local_deep_research/web/routes/research_routes.pyindex ca65b74938..4d5474c812 100644--- a/src/local_deep_research/web/routes/research_routes.py+++ b/src/local_deep_research/web/routes/research_routes.py@@ -289,15 +289,22 @@ def _precheck_engine_policy(settings_manager, params, search_engine, username):engine=search_engine,reason=decision.reason,)+ # Local import: this module imports the egress policy lazily inside+ # this function to avoid a circular import at load time; keep the+ # guidance import with it.+ from ...security.egress.guidance import denial_guidance+return (jsonify({"status": "error",- "message": (- f"Search engine '{search_engine}' is not "- f"permitted under the current egress "- f"policy: {decision.reason}"+ # Clear, actionable message (what + why + how to allow),+ # plus the raw reason code for support/logs.+ "message": denial_guidance(+ decision.reason,+ target=f"Search engine '{search_engine}'",),+ "reason": decision.reason,}),400,diff --git a/tests/security/test_egress_guidance.py b/tests/security/test_egress_guidance.pynew file mode 100644index 0000000000..f9cf0ecc6b--- /dev/null+++ b/tests/security/test_egress_guidance.py@@ -0,0 +1,83 @@+"""The egress denial-guidance helper must turn machine reason codes into clear,+actionable user messages (what was blocked + how to allow it)."""++from __future__ import annotations++import pytest++from local_deep_research.security.egress.guidance import denial_guidance+++ "reason,target,must_contain",+ [+ # Scope blocks name the scope AND the setting to change.+ (+ "scope_mismatch_private_only",+ "arxiv",+ ["Private only", "Egress Scope"],+ ),+ (+ "scope_mismatch_public_only",+ "library",+ ["Public only", "Egress Scope"],+ ),+ ("strict_not_primary", "wikipedia", ["Strict", "primary"]),+ ("strict_with_meta_picker", None, ["Strict", "meta-picker"]),+ # Inference blocks name the require-local toggle.+ ("provider_cloud_only", "openai", ["Require local LLM", "openai"]),+ ("provider_cloud", "openai", ["Require local embeddings"]),+ ("provider_remote", "ollama", ["local"]),+ # Hard rule: metadata cannot be overridden.+ ("blocked_metadata_ip", None, ["cannot be overridden", "metadata"]),+ ("elasticsearch_cloud_id_public_egress", None, ["Cloud ID", "hosts"]),+ ("unknown_egress_scope", None, ["unrecognised", "Egress Scope"]),+ ],+)+def test_guidance_is_clear_and_actionable(reason, target, must_contain):+ msg = denial_guidance(reason, target=target)+ assert msg and len(msg) > 20+ for needle in must_contain:+ assert needle in msg, f"{reason}: expected '{needle}' in: {msg}"+++def test_guidance_inserts_target():+ msg = denial_guidance("scope_mismatch_private_only", target="arxiv")+ assert "arxiv" in msg+++def test_non_policy_reasons_get_honest_explanation_not_setting_advice():+ # A malformed URL isn't a policy block the user fixes via settings.+ msg = denial_guidance("url_malformed")+ assert "malformed" in msg+ assert "Egress Scope" not in msg+++def test_unknown_reason_is_safe_and_nonempty():+ msg = denial_guidance("brand_new_code_42", target="X")+ assert "X" in msg+ assert "brand_new_code_42" in msg # surfaced for support+ assert msg # never empty+++def test_target_none_falls_back_to_this_action():+ msg = denial_guidance("scope_mismatch_private_only") # no target+ assert msg.startswith("This action")+ assert "Egress Scope" in msg+++ "reason,needle",+ [+ ("no_hostname", "no host"),+ ("unsupported_scheme", "http/https"),+ ("dangerous_scheme", "non-web scheme"),+ ("host_unclassified", "resolved or classified"),+ ("internal_error", "internal error"),+ ],+)+def test_all_non_policy_reasons_explain_without_setting_advice(reason, needle):+ msg = denial_guidance(reason)+ assert needle in msg+ # These are parse/format failures, not a policy the user toggles.+ assert "Egress Scope" not in msgdiff --git a/tests/web/test_api_coverage.py b/tests/web/test_api_coverage.pyindex 089a06fd02..63a4220cdf 100644--- a/tests/web/test_api_coverage.py+++ b/tests/web/test_api_coverage.py@@ -576,34 +576,112 @@ class TestQuickSummary:assert snapshot["some.key"] == 42assert snapshot["another.key"] == "raw_value"- def test_settings_snapshot_failure_uses_empty(self, authed_client):- """If loading settings snapshot fails, an empty dict is used- and the request still succeeds."""- # The function body re-imports get_user_db_session from- # database.session_context, so patch that to raise.- failing_ctx = MagicMock()- failing_ctx.return_value.__enter__ = MagicMock(- side_effect=RuntimeError("db fail")- )- failing_ctx.return_value.__exit__ = MagicMock(return_value=None)-- with _mock_access_control():+ def test_settings_load_failure_fails_closed(self, authed_client):+ """If loading the settings snapshot fails, the request is REFUSED+ (HTTP 503) rather than silently continuing with an empty snapshot.++ Continuing with ``{}`` would resolve to the permissive BOTH scope,+ downgrading a configured PRIVATE_ONLY / require-local user — so the+ endpoint fails closed and the research function is never called.+ We fail the endpoint's full-snapshot load (``get_all_settings``) on the+ shared settings-manager mock, so the auth decorator's ``get_setting``+ calls keep working — only the snapshot build inside the endpoint's try+ is broken. (Both get_user_db_session and get_settings_manager are+ bound at module level, so they share one patch surface.)+ """+ with _mock_access_control() as (_ctx, mock_sm):+ mock_sm.return_value.get_all_settings.side_effect = RuntimeError(+ "settings fail"+ )+ with patch(+ "local_deep_research.api.research_functions.quick_summary",+ return_value={"findings": []},+ ) as mock_qs:+ resp = authed_client.post(+ "/api/v1/quick_summary",+ json={"query": "test"},+ )+ assert resp.status_code == 503+ mock_qs.assert_not_called()+ # The 503 carries actionable guidance, not just a code.+ body = resp.get_json()+ assert "how_to_fix" in body+ assert "allow_default_settings" in body["how_to_fix"]++ def test_settings_load_failure_opt_in_continues_with_defaults(+ self, authed_client+ ):+ """With ``allow_default_settings=true`` the caller consciously opts in+ to run with defaults (empty snapshot) when settings can't load — the+ request proceeds (200) instead of failing closed, and quick_summary is+ called with an empty settings_snapshot."""+ with _mock_access_control() as (_ctx, mock_sm):+ mock_sm.return_value.get_all_settings.side_effect = RuntimeError(+ "settings fail"+ )+ with patch(+ "local_deep_research.api.research_functions.quick_summary",+ return_value={"findings": []},+ ) as mock_qs:+ resp = authed_client.post(+ "/api/v1/quick_summary",+ json={"query": "test", "allow_default_settings": True},+ )+ assert resp.status_code == 200+ mock_qs.assert_called_once()+ assert mock_qs.call_args.kwargs["settings_snapshot"] == {}+ # The opt-in flag must NOT be forwarded to quick_summary.+ assert "allow_default_settings" not in mock_qs.call_args.kwargs++ def test_opt_in_requires_real_true_not_truthy_string(self, authed_client):+ """Security-boundary flag: a truthy STRING like "false" must NOT opt in+ — only a real JSON ``true`` does. Otherwise it still fails closed."""+ with _mock_access_control() as (_ctx, mock_sm):+ mock_sm.return_value.get_all_settings.side_effect = RuntimeError(+ "settings fail"+ )+ with patch(+ "local_deep_research.api.research_functions.quick_summary",+ return_value={"findings": []},+ ) as mock_qs:+ resp = authed_client.post(+ "/api/v1/quick_summary",+ # JSON string "false" is truthy but is not boolean true.+ json={"query": "t", "allow_default_settings": "false"},+ )+ assert resp.status_code == 503+ mock_qs.assert_not_called()++ def test_opt_in_path_emits_policy_audit_warning(self, authed_client):+ """The opt-in (run-without-settings) path must log a loud policy_audit+ warning — the security claim is that it is never silent."""+ with _mock_access_control() as (_ctx, mock_sm):+ mock_sm.return_value.get_all_settings.side_effect = RuntimeError(+ "settings fail"+ )with (patch("local_deep_research.api.research_functions.quick_summary",return_value={"findings": []},- ) as mock_qs,- patch(- "local_deep_research.database.session_context.get_user_db_session",- failing_ctx,),+ patch("local_deep_research.web.api.logger") as mock_logger,):resp = authed_client.post("/api/v1/quick_summary",- json={"query": "test"},+ json={"query": "t", "allow_default_settings": True},)assert resp.status_code == 200- assert mock_qs.call_args[1]["settings_snapshot"] == {}+ # logger.bind(policy_audit=True) was used …+ bind_kwargs = [+ c.kwargs for c in mock_logger.bind.call_args_list+ ]+ assert {"policy_audit": True} in bind_kwargs+ # … and the bound logger emitted a warning naming the opt-in.+ bound = mock_logger.bind.return_value+ assert any(+ "DEFAULT settings" in str(c)+ for c in bound.warning.call_args_list+ )def test_non_dict_body_returns_400(self, authed_client):with _mock_access_control():